| /home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. |
| warnings.warn('Grouped GEMM not available.') |
| wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc. |
| wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login |
| wandb: Tracking run with wandb version 0.28.0 |
| wandb: Run data is saved locally in outputs/healed/grid_math/glean_keep75_s1226/wandb/run-20260716_142818-kru5sldj |
| wandb: Run `wandb offline` to turn off syncing. |
| wandb: Syncing run glean-math-keep75-s1226 |
| wandb: ⭐️ View project at https://wandb.ai/hbfreed/glean-grid |
| wandb: 🚀 View run at https://wandb.ai/hbfreed/glean-grid/runs/kru5sldj |
|
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 33%|███▎ | 1/3 [00:01<00:02, 1.35s/it]
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:02<00:01, 1.51s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:03<00:00, 1.09it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:03<00:00, 1.06s/it] |
| resumed student weights from outputs/healed/grid_math/glean_keep75_s1226/step0100 (fresh optimizer, step counter at 0) |
| 12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 5.31B | teacher overlap=False |
| restored optimizer/scheduler state from step 100; rebuilt 260 paged buffers |
| {"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017711289564648177, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.22265625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 62.0, "frames": {"chat": 232}, "mem_gb": 21.89} |
| {"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017982081608990362, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.832, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 220}, "mem_gb": 22.09} |
| {"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0159690285191716, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.1982421875, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 210}, "mem_gb": 22.14} |
| {"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.014078767126984894, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.1845703125, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 226}, "mem_gb": 22.06} |
| {"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.015146795610602325, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.248046875, "lr": 3e-05, "finish_rate": 0.741, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 212}, "mem_gb": 22.09} |
| {"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.014050660304430251, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.201171875, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 236}, "mem_gb": 22.1} |
| {"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012824523648739948, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.928, "comp_len": 454.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 264}, "mem_gb": 21.97} |
| {"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.017827476510805233, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.2421875, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 229}, "mem_gb": 22.07} |
| {"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01010443297145733, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.16015625, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 465.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 258}, "mem_gb": 21.95} |
| {"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013931514149834403, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.19140625, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 208}, "mem_gb": 22.11} |
| The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results. |
| [eval step 110] sample: 'To solve this problem, we need to understand the geometric properties involved. When the midpoints of the sides of a triangle are connected, the segments joining these midpoints form a smaller triangl' |
| {"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010035461849397204, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.16015625, "lr": 3e-05, "finish_rate": 0.88, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 249}, "mem_gb": 22.02} |
| {"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01025480254034434, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.1796875, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 220}, "mem_gb": 22.09} |
| {"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010523743329072991, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.1708984375, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 223}, "mem_gb": 22.08} |
| {"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01153164215181023, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.17578125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 221}, "mem_gb": 22.09} |
| {"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009408667080748516, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.1552734375, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 230}, "mem_gb": 21.99} |
| {"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011410523822395286, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.1630859375, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 214}, "mem_gb": 22.07} |
| {"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01616842130373698, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.224609375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 214}, "mem_gb": 22.08} |
| {"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013691102667020944, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.19921875, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 210}, "mem_gb": 22.13} |
| {"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013348402527160942, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.1884765625, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.0, "frames": {"chat": 214}, "mem_gb": 22.09} |
| {"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011879650606094704, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.2001953125, "lr": 3e-05, "finish_rate": 0.791, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.1, "frames": {"chat": 215}, "mem_gb": 22.05} |
| [eval step 120] sample: 'To solve this problem, we need to understand the geometric properties involved when the midpoints of the sides of a triangle are connected by segments. This process creates a new triangle, known as th' |
| {"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013821079181631406, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.1845703125, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.3, "frames": {"chat": 208}, "mem_gb": 22.09} |
| {"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010628624554680815, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.1708984375, "lr": 3e-05, "finish_rate": 0.789, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.7, "frames": {"chat": 218}, "mem_gb": 21.97} |
| {"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010161227444757242, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.1640625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.1, "frames": {"chat": 233}, "mem_gb": 21.99} |
| {"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009278946231456938, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.16015625, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.3, "frames": {"chat": 231}, "mem_gb": 22.03} |
| {"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011464818606327754, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.1611328125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.0, "frames": {"chat": 235}, "mem_gb": 22.22} |
| {"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010276864761835895, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.1484375, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 216}, "mem_gb": 22.08} |
| {"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009667191798787098, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.162109375, "lr": 3e-05, "finish_rate": 0.831, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 237}, "mem_gb": 22.1} |
| {"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.014880005238496233, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.25, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.5, "frames": {"chat": 203}, "mem_gb": 22.1} |
| {"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010798121559165885, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.166015625, "lr": 3e-05, "finish_rate": 0.873, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.5, "frames": {"chat": 236}, "mem_gb": 22.13} |
| {"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009982232672628015, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.1513671875, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.7, "frames": {"chat": 214}, "mem_gb": 22.1} |
| [eval step 130] sample: 'To solve this problem, we need to understand the geometric properties involved when the midpoints of the sides of a triangle are connected by segments. This process creates a new triangle, known as th' |
| {"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01276522857361318, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.1826171875, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.5, "frames": {"chat": 209}, "mem_gb": 22.09} |
| {"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01355358442418122, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.2216796875, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 468.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.2, "frames": {"chat": 256}, "mem_gb": 22.1} |
| {"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.00979031468022537, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.1591796875, "lr": 3e-05, "finish_rate": 0.878, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.9, "frames": {"chat": 230}, "mem_gb": 21.96} |
| {"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012052156168699731, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.4, "frames": {"chat": 230}, "mem_gb": 22.15} |
| {"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011693784629782506, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.1640625, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 227}, "mem_gb": 22.05} |
| {"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011806925121663758, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.150390625, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 208}, "mem_gb": 22.11} |
| {"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011330220682158445, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.166015625, "lr": 3e-05, "finish_rate": 0.699, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.6, "frames": {"chat": 206}, "mem_gb": 22.12} |
| {"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010682711551792455, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.15625, "lr": 3e-05, "finish_rate": 0.82, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.0, "frames": {"chat": 228}, "mem_gb": 22.0} |
| {"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011466035542547858, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.15234375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.1, "frames": {"chat": 224}, "mem_gb": 22.09} |
| {"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012643014844014155, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.236328125, "lr": 3e-05, "finish_rate": 0.66, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 200}, "mem_gb": 22.13} |
| [eval step 140] sample: 'To solve this problem, we need to understand the geometric properties involved when the midpoints of the sides of a triangle are connected by segments. This process creates a new triangle, known as th' |
| {"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01080410843101951, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.1689453125, "lr": 3e-05, "finish_rate": 0.714, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 196}, "mem_gb": 22.11} |
| {"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009731826097378507, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.140625, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 223}, "mem_gb": 22.09} |
| {"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011139519218046916, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.2119140625, "lr": 3e-05, "finish_rate": 0.869, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 213}, "mem_gb": 21.98} |
| {"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009973561418584237, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.146484375, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.8, "frames": {"chat": 232}, "mem_gb": 22.02} |
| {"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009199376441648928, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.16015625, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.9, "frames": {"chat": 223}, "mem_gb": 22.02} |
| {"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0111230931228473, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.1611328125, "lr": 3e-05, "finish_rate": 0.85, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.5, "frames": {"chat": 233}, "mem_gb": 22.11} |
| {"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011176060675464882, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.1474609375, "lr": 3e-05, "finish_rate": 0.816, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 217}, "mem_gb": 22.11} |
| {"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01491896515111827, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.185546875, "lr": 3e-05, "finish_rate": 0.752, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 202}, "mem_gb": 22.17} |
| {"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.00973438565802838, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.1484375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.5, "frames": {"chat": 253}, "mem_gb": 22.03} |
| {"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009605388223500147, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.1552734375, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 231}, "mem_gb": 22.03} |
| [eval step 150] sample: 'To solve this problem, we need to understand the geometric properties involved when the midpoints of the sides of a triangle are connected by segments. This process creates a new triangle, known as th' |
| checkpoint snapshot queued -> outputs/healed/grid_math/glean_keep75_s1226/step0150 |
| wandb: updating run metadata |
| wandb: uploading output.log |
| wandb: |
| wandb: Run history: |
| wandb: comp_len ▃▅▄▆▃▁▆▂▅▄▄▆▆▆▆▆▅▃▄▃▃▇▃▆▆▄▄▄▆▇▄▇█▄▆▄▃▅▇▄ |
| wandb: cumulative_loss_tokens ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▆▆▆▆▆▆▇▇▇▇▇▇███ |
| wandb: epoch ▁▁▁▁████████████████████████████████████ |
| wandb: finish_rate ▆▅▄▅▃█▆▇▃▆▆▇▄▄▄▄▃▄▇▆▆▅▃▇▃▇▇▅▇▃▅▆▁▂▆▆▆▅▃▇ |
| wandb: forward_topk_kl ██▆▅▆▄█▂▅▂▂▃▃▅▄▅▂▂▁▃▁▆▂▂▄▁▃▃▃▃▃▄▂▁▃▁▃▃▆▁ |
| wandb: grad_norm ▆█▄▄▇▄▇▂▄▂▃▂▂▆▅▅▄▃▂▂▁▂█▃▂▆▂▄▂▂▂▇▃▁▅▂▂▁▄▂ |
| wandb: lr ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ |
| wandb: mem_gb ▁▅▆▅▅▃▅▂▆▄▅▅▃▅▅▅▄▅▃▃█▅▅▅▆▅▂▇▄▆▃▅▆▆▅▄▄▆▆▄ |
| wandb: step ▁▁▁▁▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███ |
| wandb: t_data_s ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ |
| wandb: +3 ... |
| wandb: |
| wandb: Run summary: |
| wandb: comp_len 519.5 |
| wandb: cumulative_loss_tokens 18000000 |
| wandb: epoch 2 |
| wandb: finish_rate 0.879 |
| wandb: forward_topk_kl 0.00961 |
| wandb: grad_norm 0.15527 |
| wandb: lr 3e-05 |
| wandb: mem_gb 22.03 |
| wandb: step 150 |
| wandb: t_data_s 0 |
| wandb: +4 ... |
| wandb: |
| wandb: 🚀 View run glean-math-keep75-s1226 at: https://wandb.ai/hbfreed/glean-grid/runs/kru5sldj |
| wandb: ⭐️ View project at: https://wandb.ai/hbfreed/glean-grid |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) |
| wandb: Find logs at: outputs/healed/grid_math/glean_keep75_s1226/wandb/run-20260716_142818-kru5sldj/logs |
| { |
| "correct": 915, |
| "accuracy": 0.6937073540561031, |
| "finished": 1314, |
| "finish_rate": 0.9962092494313874, |
| "mean_completion_tokens": 114.16527672479151 |
| } |
| saved item-level results -> outputs/evals/grid_math/glean_keep75_s1226_step100_chat.json |
| { |
| "correct": 925, |
| "accuracy": 0.7012888551933283, |
| "finished": 1314, |
| "finish_rate": 0.9962092494313874, |
| "mean_completion_tokens": 115.14329037149355 |
| } |
| saved item-level results -> outputs/evals/grid_math/glean_keep75_s1226_step150_chat.json |
| |
