| /home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. |
| warnings.warn('Grouped GEMM not available.') |
| wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc. |
| wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login |
| wandb: Tracking run with wandb version 0.28.0 |
| wandb: Run data is saved locally in outputs/healed/grid_math/glean_keep75_s1225/wandb/run-20260716_142818-9x4iij2d |
| wandb: Run `wandb offline` to turn off syncing. |
| wandb: Syncing run glean-math-keep75-s1225 |
| wandb: ⭐️ View project at https://wandb.ai/hbfreed/glean-grid |
| wandb: 🚀 View run at https://wandb.ai/hbfreed/glean-grid/runs/9x4iij2d |
|
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 33%|███▎ | 1/3 [00:01<00:03, 1.53s/it]
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:03<00:01, 1.59s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:03<00:00, 1.04it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:03<00:00, 1.13s/it] |
| resumed student weights from outputs/healed/grid_math/glean_keep75_s1225/step0100 (fresh optimizer, step counter at 0) |
| 12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 5.31B | teacher overlap=False |
| restored optimizer/scheduler state from step 100; rebuilt 260 paged buffers |
| {"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.016109059700369834, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.2041015625, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 63.3, "frames": {"chat": 223}, "mem_gb": 21.95} |
| {"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.016362527111552966, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.21875, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 206}, "mem_gb": 22.1} |
| {"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.013859456993810212, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.228515625, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 213}, "mem_gb": 22.02} |
| {"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018646482431787688, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.22265625, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 223}, "mem_gb": 21.96} |
| {"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.015217417630545484, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.201171875, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 227}, "mem_gb": 22.07} |
| {"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018008409859096478, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.255859375, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 253}, "mem_gb": 22.09} |
| {"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012912440449073135, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.1865234375, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 216}, "mem_gb": 22.1} |
| {"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010693734145350754, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.1748046875, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 205}, "mem_gb": 22.07} |
| {"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.015156030047448197, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.2138671875, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 207}, "mem_gb": 22.16} |
| {"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013672215391354015, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.1982421875, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 215}, "mem_gb": 22.08} |
| The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results. |
| [eval step 110] sample: 'To solve this problem, we need to analyze the spiral pattern of numbers from 1 to 49 arranged on a square grid and identify the four shaded squares that lie on the same diagonal as the number 7. We th' |
| {"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011061214675944453, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.2060546875, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 228}, "mem_gb": 22.1} |
| {"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012901123281742912, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.1982421875, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 221}, "mem_gb": 22.14} |
| {"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009574843731914492, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.15625, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 254}, "mem_gb": 21.93} |
| {"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011826626441131036, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.2392578125, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 210}, "mem_gb": 22.06} |
| {"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010666260581545066, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.166015625, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 226}, "mem_gb": 22.02} |
| {"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01181182961029117, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 212}, "mem_gb": 22.09} |
| {"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.014291961643429628, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.20703125, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.2, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 211}, "mem_gb": 22.02} |
| {"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011632958819546426, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.1640625, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.9, "frames": {"chat": 196}, "mem_gb": 22.07} |
| {"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011126866567115454, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.2119140625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 212}, "mem_gb": 22.09} |
| {"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010413994966812121, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.16015625, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 244}, "mem_gb": 22.0} |
| [eval step 120] sample: "To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers on the same diagonal as the number 7. Let's break down the problem step-by-step:\n\n1. **Underst" |
| {"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010520029380288906, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.166015625, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 222}, "mem_gb": 22.05} |
| {"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011724793996859807, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.1884765625, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 218}, "mem_gb": 22.09} |
| {"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011794728558894713, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.1982421875, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 252}, "mem_gb": 21.97} |
| {"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012576708099556466, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.19140625, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 202}, "mem_gb": 22.14} |
| {"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012525027039841128, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.44140625, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 237}, "mem_gb": 22.1} |
| {"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011847056540916674, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.166015625, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 234}, "mem_gb": 22.08} |
| {"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010580201309620558, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 215}, "mem_gb": 22.1} |
| {"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010746679509648433, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.1796875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 234}, "mem_gb": 22.03} |
| {"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010033142010107016, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.1884765625, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 216}, "mem_gb": 22.08} |
| {"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01059437686605379, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.1787109375, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 210}, "mem_gb": 22.05} |
| [eval step 130] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid starting from the center. We then identify the four shaded squares that lie on the same diagonal' |
| {"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011416423269287528, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.169921875, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 199}, "mem_gb": 22.09} |
| {"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011045920558762736, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.1865234375, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 210}, "mem_gb": 22.11} |
| {"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010189920482278103, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.1689453125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 225}, "mem_gb": 22.05} |
| {"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011348977557197213, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.1728515625, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 253}, "mem_gb": 21.95} |
| {"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010840521045137818, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.1513671875, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 247}, "mem_gb": 22.07} |
| {"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01431039827777228, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.2021484375, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 238}, "mem_gb": 22.07} |
| {"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012493417616401954, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.16796875, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 235}, "mem_gb": 22.09} |
| {"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011775391096648916, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.1640625, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 215}, "mem_gb": 22.06} |
| {"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011272327020104665, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.177734375, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.0, "frames": {"chat": 268}, "mem_gb": 22.06} |
| {"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011195119415794033, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.1689453125, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 228}, "mem_gb": 22.09} |
| [eval step 140] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid and identify the four shaded squares that lie on the same diagonal as the number 7. Then, we wil' |
| {"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01146084621019351, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.1513671875, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 252}, "mem_gb": 22.03} |
| {"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01266345674659824, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.1875, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 223}, "mem_gb": 22.11} |
| {"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012728927766492901, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.1689453125, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 226}, "mem_gb": 22.09} |
| {"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.015653314464636303, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.2421875, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 208}, "mem_gb": 22.14} |
| {"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.00990355064412579, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.154296875, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 240}, "mem_gb": 22.03} |
| {"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011352584307268262, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.171875, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 222}, "mem_gb": 22.02} |
| {"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009270905922904301, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.1396484375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 236}, "mem_gb": 22.09} |
| {"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010650625815118353, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.181640625, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 217}, "mem_gb": 22.06} |
| {"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010312878351037702, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.1611328125, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 252}, "mem_gb": 21.97} |
| {"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009814306650865667, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.14453125, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 222}, "mem_gb": 22.08} |
| [eval step 150] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid and identify the four numbers that lie on the same diagonal as the number 7. We then need to det' |
| checkpoint snapshot queued -> outputs/healed/grid_math/glean_keep75_s1225/step0150 |
| wandb: updating run metadata |
| wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml |
| wandb: |
| wandb: Run history: |
| wandb: comp_len ▅▇▆▅▄▆▇▇▆▄▆▅▆▆█▃▅▅▂▇▄▆▄▆▆▆▅▂▃▃▆▁▂▅▅▃▅▄▅▅ |
| wandb: cumulative_loss_tokens ▁▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇███ |
| wandb: epoch ▁▁▁▁▁███████████████████████████████████ |
| wandb: finish_rate ▃▃▃▅▅▃▂▁▄▂▅▅▄▂▃▆▅▃█▁▆▄▆▄▄▄▇█▇▅▄█▄▆▄▁▅▆▅▅ |
| wandb: forward_topk_kl ▆▆▄█▅▄▂▅▄▂▁▃▂▃▅▂▂▃▃▃▃▂▂▂▂▂▂▃▂▅▃▂▂▃▄▁▃▁▂▁ |
| wandb: grad_norm ▂▃▃▃▂▂▂▃▂▃▁▂▂▂▃▂▂▂▂█▂▂▂▂▂▂▂▁▂▂▂▂▁▂▂▁▂▁▂▁ |
| wandb: lr ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ |
| wandb: mem_gb ▂▆▄▂▅▆▅█▆▆▁▅▄▅▆▅▆▂▇▆▆▄▆▅▆▅▂▅▅▆▅▆▄▆▆▄▄▆▅▆ |
| wandb: step ▁▁▁▁▂▂▂▂▂▂▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇███ |
| wandb: t_data_s ▁▁▁▁▁▁▁▁▁▁▁▁▁▁█▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ |
| wandb: +3 ... |
| wandb: |
| wandb: Run summary: |
| wandb: comp_len 540.5 |
| wandb: cumulative_loss_tokens 18000000 |
| wandb: epoch 2 |
| wandb: finish_rate 0.847 |
| wandb: forward_topk_kl 0.00981 |
| wandb: grad_norm 0.14453 |
| wandb: lr 3e-05 |
| wandb: mem_gb 22.08 |
| wandb: step 150 |
| wandb: t_data_s 0 |
| wandb: +4 ... |
| wandb: |
| wandb: 🚀 View run glean-math-keep75-s1225 at: https://wandb.ai/hbfreed/glean-grid/runs/9x4iij2d |
| wandb: ⭐️ View project at: https://wandb.ai/hbfreed/glean-grid |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) |
| wandb: Find logs at: outputs/healed/grid_math/glean_keep75_s1225/wandb/run-20260716_142818-9x4iij2d/logs |
| { |
| "correct": 910, |
| "accuracy": 0.6899166034874905, |
| "finished": 1315, |
| "finish_rate": 0.9969673995451099, |
| "mean_completion_tokens": 113.55724033358605 |
| } |
| saved item-level results -> outputs/evals/grid_math/glean_keep75_s1225_step100_chat.json |
| { |
| "correct": 909, |
| "accuracy": 0.689158453373768, |
| "finished": 1316, |
| "finish_rate": 0.9977255496588324, |
| "mean_completion_tokens": 114.47536012130402 |
| } |
| saved item-level results -> outputs/evals/grid_math/glean_keep75_s1225_step150_chat.json |
|
|
