Add files using upload-large-folder tool
Browse filesThis view is limited to 50 files because it contains too many changes. Β See raw diff
- healed/grid_general_fairness/glean_keep50_s1224_long500.console.log +0 -0
- healed/grid_general_fairness/glean_keep50_s1224_long500.eval.log +0 -0
- healed/grid_general_fairness/reap_keep50_s1224_long500.console.log +0 -0
- healed/grid_general_fairness/reap_keep50_s1224_long500.eval.log +0 -0
- healed/grid_general_fairness/reap_keep50_s1224_lr1e5.console.log +213 -0
- healed/grid_general_fairness/reap_keep50_s1224_lr1e5.eval.log +0 -0
- healed/grid_math/glean_keep25_s1225.console.log +230 -0
- healed/grid_math/glean_keep25_s1226.console.log +232 -0
- healed/grid_math/keep75.log +56 -0
- healed/grid_math/reap_keep25_s1224.console.log +232 -0
- healed/grid_math/reap_keep25_s1226.console.log +231 -0
- healed/grid_math/reap_keep50_s1224.console.log +233 -0
- healed/grid_math/reap_keep50_s1225.console.log +232 -0
- healed/grid_math/reap_keep50_s1226.console.log +231 -0
- healed/grid_math/reap_keep75_s1224.console.log +324 -0
- healed/grid_math/uniform_keep25_s1224.console.log +231 -0
- healed/grid_math/uniform_keep25_s1225.console.log +230 -0
- healed/grid_math/uniform_keep25_s1226.console.log +232 -0
- healed/grid_math/uniform_keep50_s1224.console.log +232 -0
- healed/grid_math/uniform_keep50_s1225.console.log +231 -0
- healed/grid_math/uniform_keep75_s1224.console.log +232 -0
- healed/grid_math/uniform_keep75_s1225.console.log +232 -0
- healed/grid_math/uniform_keep75_s1226.console.log +248 -0
- healed/grid_math/worker_s1226.log +22 -0
- healed/liger_mb8/args.json +70 -0
- healed/liger_mb8/train_log.jsonl +4 -0
- healed/liger_mb8/vllm_server.log +309 -0
- healed/liger_smoke/args.json +70 -0
- healed/liger_smoke/train_log.jsonl +6 -0
- healed/liger_smoke/vllm_server.log +447 -0
- healed/mixceonly_keep50/args.json +71 -0
- healed/mixceonly_keep50/train_log.jsonl +49 -0
- healed/mixceonly_keep50/vllm_server.log +0 -0
- healed/mixceonly_keep50/wandb_sync.log +2 -0
- healed/mixonly_keep50/args.json +70 -0
- healed/mixonly_keep50/train_log.jsonl +61 -0
- healed/mixonly_keep50/vllm_server.log +0 -0
- healed/opd_warm_fixed_keep50/args.json +71 -0
- healed/opd_warm_fixed_keep50/train_log.jsonl +235 -0
- healed/opd_warm_fixed_keep50/vllm_server.log +0 -0
- healed/opd_warm_fixed_keep50/wandb_sync.log +2 -0
- healed/opd_warm_keep50/args.json +71 -0
- healed/opd_warm_keep50/train_log.jsonl +47 -0
- healed/opd_warm_keep50/vllm_server.log +0 -0
- healed/policy_confirm/chain_seed1226.log +46 -0
- healed/policy_confirm/combo.log +35 -0
- healed/policy_confirm/combo_off25_seed1224.console.log +75 -0
- healed/policy_confirm/combo_off25_seed1225.console.log +77 -0
- healed/policy_confirm/combo_off25_seed1226.console.log +77 -0
- healed/policy_confirm/combo_on25_seed1224.console.log +94 -0
healed/grid_general_fairness/glean_keep50_s1224_long500.console.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
healed/grid_general_fairness/glean_keep50_s1224_long500.eval.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
healed/grid_general_fairness/reap_keep50_s1224_long500.console.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
healed/grid_general_fairness/reap_keep50_s1224_long500.eval.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
healed/grid_general_fairness/reap_keep50_s1224_lr1e5.console.log
ADDED
|
@@ -0,0 +1,213 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 6 |
+
wandb: Run data is saved locally in outputs/healed/grid_general_fairness/reap_keep50_s1224_lr1e5/wandb/run-20260719_141738-sgv88kwj
|
| 7 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 8 |
+
wandb: Syncing run reap_keep50_s1224_lr1e5
|
| 9 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-general-grid
|
| 10 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-general-grid/runs/sgv88kwj
|
| 11 |
+
|
| 12 |
+
58360 cached top-128 chat trajectories / 22,295,631 unique tokens | 185 steps/epoch | 150 total steps | student params 3.70B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.8349037809635202, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 9.5625, "lr": 2.0000000000000003e-06, "finish_rate": 0.869, "comp_len": 463.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.9, "frames": {"chat": 259}, "mem_gb": 15.93}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: 'To compute the rank of a matrix, we need to determine the highest exponent of each distinct integer coefficient (including the sign) that divides the given number. This process essentially counts the '
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.1442626529686153, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 10.875, "lr": 3e-06, "finish_rate": 0.997, "comp_len": 344.8, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 61.9, "frames": {"chat": 348}, "mem_gb": 15.64}
|
| 17 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.0448227830218773, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 10.0625, "lr": 4.000000000000001e-06, "finish_rate": 0.997, "comp_len": 324.3, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 64.4, "frames": {"chat": 370}, "mem_gb": 15.79}
|
| 18 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5794252082020044, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 7.40625, "lr": 5e-06, "finish_rate": 0.852, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 244}, "mem_gb": 16.02}
|
| 19 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.9018620710670948, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 8.6875, "lr": 6e-06, "finish_rate": 0.953, "comp_len": 375.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 59.3, "frames": {"chat": 320}, "mem_gb": 15.94}
|
| 20 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.035891845558087, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 9.0625, "lr": 7e-06, "finish_rate": 0.997, "comp_len": 317.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 64.9, "frames": {"chat": 378}, "mem_gb": 15.54}
|
| 21 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.1266093510329724, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 9.8125, "lr": 8.000000000000001e-06, "finish_rate": 0.997, "comp_len": 312.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 66.2, "frames": {"chat": 384}, "mem_gb": 16.03}
|
| 22 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.1088798726183673, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 9.5, "lr": 9e-06, "finish_rate": 0.997, "comp_len": 370.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 61.6, "frames": {"chat": 324}, "mem_gb": 15.88}
|
| 23 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5539724837720394, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 6.0, "lr": 1e-05, "finish_rate": 0.891, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.5, "frames": {"chat": 239}, "mem_gb": 16.05}
|
| 24 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.994879912075152, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 7.34375, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 392.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.1, "frames": {"chat": 306}, "mem_gb": 15.79}
|
| 25 |
+
[eval step 10] sample: "To find the rank of a matrix, we count the number of non-zero rows. Let's apply this rule to each row of the given matrix:\n\n\\[\n\\begin{matrix}\n[12, -16, 4, 16] \\\\\n[-9, 11, -1, -10]"
|
| 26 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.9568112013991922, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 6.65625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 364.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 62.4, "frames": {"chat": 329}, "mem_gb": 15.82}
|
| 27 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5175115125539402, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 3.703125, "lr": 1e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.8, "frames": {"chat": 254}, "mem_gb": 16.05}
|
| 28 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7996729048542678, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 5.0, "lr": 1e-05, "finish_rate": 0.974, "comp_len": 439.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 58.5, "frames": {"chat": 273}, "mem_gb": 16.03}
|
| 29 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.36379497829837104, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 2.921875, "lr": 1e-05, "finish_rate": 0.873, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 236}, "mem_gb": 15.95}
|
| 30 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.9682946861560146, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 4.90625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 315.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 75.2, "frames": {"chat": 381}, "mem_gb": 15.68}
|
| 31 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.40801479325406254, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 2.40625, "lr": 1e-05, "finish_rate": 0.925, "comp_len": 451.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 57.8, "frames": {"chat": 266}, "mem_gb": 15.88}
|
| 32 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5869657141402363, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 2.921875, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 389.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 64.6, "frames": {"chat": 308}, "mem_gb": 15.87}
|
| 33 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4750889514962832, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 2.390625, "lr": 1e-05, "finish_rate": 0.853, "comp_len": 439.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 57.8, "frames": {"chat": 273}, "mem_gb": 16.05}
|
| 34 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7465602072453748, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 3.078125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 326.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 76.5, "frames": {"chat": 368}, "mem_gb": 15.68}
|
| 35 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.8740993097143869, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 3.15625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 354.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 68.9, "frames": {"chat": 339}, "mem_gb": 15.92}
|
| 36 |
+
[eval step 20] sample: 'To compute the rank of a matrix, we need to determine the maximum number of linearly independent rows or columns in the matrix. We can do this by finding the rank of the matrix through its row or colu'
|
| 37 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.49150919830513495, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 1.984375, "lr": 1e-05, "finish_rate": 0.963, "comp_len": 402.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 65.1, "frames": {"chat": 298}, "mem_gb": 15.85}
|
| 38 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7165370549296339, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 2.453125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 342.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 73.0, "frames": {"chat": 350}, "mem_gb": 15.57}
|
| 39 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.42072020039893687, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 1.625, "lr": 1e-05, "finish_rate": 0.858, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 63.1, "frames": {"chat": 246}, "mem_gb": 16.02}
|
| 40 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7176458316420515, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 2.609375, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 320.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 75.9, "frames": {"chat": 375}, "mem_gb": 15.75}
|
| 41 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7302011691461007, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 2.265625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 340.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 74.7, "frames": {"chat": 352}, "mem_gb": 15.8}
|
| 42 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7150813778654983, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 2.109375, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 303.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 83.3, "frames": {"chat": 396}, "mem_gb": 15.76}
|
| 43 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.597729198537146, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 1.734375, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 331.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 73.3, "frames": {"chat": 362}, "mem_gb": 15.61}
|
| 44 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7470703861179451, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 1.9140625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 325.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 79.6, "frames": {"chat": 369}, "mem_gb": 15.82}
|
| 45 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7586639583493272, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 1.953125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 300.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 83.1, "frames": {"chat": 399}, "mem_gb": 15.67}
|
| 46 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.36857507862376515, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 1.1875, "lr": 1e-05, "finish_rate": 0.911, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 60.2, "frames": {"chat": 247}, "mem_gb": 15.94}
|
| 47 |
+
[eval step 30] sample: "To find the rank of a matrix, we need to determine the maximum number of linearly independent rows or columns. We can achieve this by using Python and the `numpy` library. Here's how we can do it step"
|
| 48 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18771142529603094, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.9609375, "lr": 1e-05, "finish_rate": 0.891, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.2, "frames": {"chat": 230}, "mem_gb": 15.88}
|
| 49 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3433096302593127, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 1.28125, "lr": 1e-05, "finish_rate": 0.905, "comp_len": 456.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 63.6, "frames": {"chat": 263}, "mem_gb": 16.05}
|
| 50 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3737842437001566, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 1.3203125, "lr": 1e-05, "finish_rate": 0.848, "comp_len": 454.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 61.0, "frames": {"chat": 264}, "mem_gb": 16.04}
|
| 51 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6250371708575015, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 1.90625, "lr": 1e-05, "finish_rate": 0.991, "comp_len": 365.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 76.9, "frames": {"chat": 328}, "mem_gb": 15.9}
|
| 52 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3337565890327096, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.9609375, "lr": 1e-05, "finish_rate": 0.925, "comp_len": 427.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 63.4, "frames": {"chat": 281}, "mem_gb": 15.97}
|
| 53 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.25203985444543264, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.91796875, "lr": 1e-05, "finish_rate": 0.861, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.6, "frames": {"chat": 237}, "mem_gb": 15.96}
|
| 54 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3094455597920964, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.85546875, "lr": 1e-05, "finish_rate": 0.91, "comp_len": 431.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 60.7, "frames": {"chat": 278}, "mem_gb": 16.05}
|
| 55 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.577444567546683, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 1.3515625, "lr": 1e-05, "finish_rate": 0.955, "comp_len": 387.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 67.1, "frames": {"chat": 310}, "mem_gb": 15.92}
|
| 56 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3719083008861169, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 1.0390625, "lr": 1e-05, "finish_rate": 0.959, "comp_len": 406.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 64.1, "frames": {"chat": 295}, "mem_gb": 15.86}
|
| 57 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.41891379550583663, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 1.078125, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 364.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 68.3, "frames": {"chat": 329}, "mem_gb": 15.87}
|
| 58 |
+
[eval step 40] sample: 'To find the rank of a matrix, we need to determine the maximum number of linearly independent rows or columns in the matrix. This is equivalent to finding the maximum number of linearly independent ve'
|
| 59 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3865857259180397, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.92578125, "lr": 1e-05, "finish_rate": 0.905, "comp_len": 456.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 63.4, "frames": {"chat": 263}, "mem_gb": 15.99}
|
| 60 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4176833015020316, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.921875, "lr": 1e-05, "finish_rate": 0.981, "comp_len": 381.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 68.6, "frames": {"chat": 315}, "mem_gb": 15.87}
|
| 61 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.570690848267451, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 1.1796875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 339.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 77.1, "frames": {"chat": 354}, "mem_gb": 15.82}
|
| 62 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2644139167781298, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.80078125, "lr": 1e-05, "finish_rate": 0.927, "comp_len": 463.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 59.7, "frames": {"chat": 259}, "mem_gb": 16.0}
|
| 63 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3887903780572116, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.8671875, "lr": 1e-05, "finish_rate": 0.92, "comp_len": 436.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 64.0, "frames": {"chat": 275}, "mem_gb": 15.85}
|
| 64 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5482282175154115, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 1.1015625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 359.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 80.7, "frames": {"chat": 334}, "mem_gb": 15.72}
|
| 65 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5852822284188122, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 1.0, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 320.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 82.4, "frames": {"chat": 374}, "mem_gb": 15.57}
|
| 66 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4763524737470473, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.9375, "lr": 1e-05, "finish_rate": 0.974, "comp_len": 385.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 71.8, "frames": {"chat": 311}, "mem_gb": 15.74}
|
| 67 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4198226667688539, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.8515625, "lr": 1e-05, "finish_rate": 0.876, "comp_len": 463.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 61.0, "frames": {"chat": 259}, "mem_gb": 16.05}
|
| 68 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4939211005295937, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.9296875, "lr": 1e-05, "finish_rate": 0.944, "comp_len": 393.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 71.3, "frames": {"chat": 305}, "mem_gb": 16.03}
|
| 69 |
+
[eval step 50] sample: 'To find the rank of a matrix, we need to determine the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -1 & '
|
| 70 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.44463678032072884, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.890625, "lr": 1e-05, "finish_rate": 0.946, "comp_len": 402.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 69.6, "frames": {"chat": 298}, "mem_gb": 16.01}
|
| 71 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6223699848617738, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 1.0, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 329.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 84.2, "frames": {"chat": 364}, "mem_gb": 15.79}
|
| 72 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3217694567616408, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.71484375, "lr": 1e-05, "finish_rate": 0.96, "comp_len": 397.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 67.2, "frames": {"chat": 302}, "mem_gb": 15.89}
|
| 73 |
+
{"step": 54, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6222446533054113, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 1.0625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 379.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 73.6, "frames": {"chat": 316}, "mem_gb": 15.65}
|
| 74 |
+
{"step": 55, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.637269300268963, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 1.09375, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 384.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 76.5, "frames": {"chat": 312}, "mem_gb": 15.75}
|
| 75 |
+
{"step": 56, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4595022053249801, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.83984375, "lr": 1e-05, "finish_rate": 0.979, "comp_len": 363.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 74.4, "frames": {"chat": 330}, "mem_gb": 15.9}
|
| 76 |
+
{"step": 57, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4707850302280858, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.84375, "lr": 1e-05, "finish_rate": 0.966, "comp_len": 373.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 71.3, "frames": {"chat": 321}, "mem_gb": 15.91}
|
| 77 |
+
{"step": 58, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.46776848399098964, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.8046875, "lr": 1e-05, "finish_rate": 0.968, "comp_len": 348.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 74.1, "frames": {"chat": 344}, "mem_gb": 15.94}
|
| 78 |
+
{"step": 59, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5186073666582505, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.8671875, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 339.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 78.7, "frames": {"chat": 353}, "mem_gb": 15.61}
|
| 79 |
+
{"step": 60, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28219543333159136, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.6171875, "lr": 1e-05, "finish_rate": 0.89, "comp_len": 441.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 64.9, "frames": {"chat": 272}, "mem_gb": 16.04}
|
| 80 |
+
[eval step 60] sample: 'To find the rank of a matrix, we need to determine the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -1 & '
|
| 81 |
+
{"step": 61, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5873189728143314, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 1.0, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 378.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 71.0, "frames": {"chat": 317}, "mem_gb": 15.79}
|
| 82 |
+
{"step": 62, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2684716436355064, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.6796875, "lr": 1e-05, "finish_rate": 0.949, "comp_len": 411.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 62.3, "frames": {"chat": 292}, "mem_gb": 15.79}
|
| 83 |
+
{"step": 63, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15185375872186074, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.5546875, "lr": 1e-05, "finish_rate": 0.886, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.0, "frames": {"chat": 229}, "mem_gb": 15.9}
|
| 84 |
+
{"step": 64, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5643590900054202, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.85546875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 334.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 87.3, "frames": {"chat": 359}, "mem_gb": 15.91}
|
| 85 |
+
{"step": 65, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5052390217740089, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.7890625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 335.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 76.0, "frames": {"chat": 358}, "mem_gb": 15.63}
|
| 86 |
+
{"step": 66, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5727251645745709, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.94140625, "lr": 1e-05, "finish_rate": 0.975, "comp_len": 381.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 81.0, "frames": {"chat": 315}, "mem_gb": 16.01}
|
| 87 |
+
{"step": 67, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3711766970043381, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.74609375, "lr": 1e-05, "finish_rate": 0.905, "comp_len": 454.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 65.7, "frames": {"chat": 264}, "mem_gb": 16.09}
|
| 88 |
+
{"step": 68, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5636533653473481, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.88671875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 356.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 75.2, "frames": {"chat": 337}, "mem_gb": 15.76}
|
| 89 |
+
{"step": 69, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.298638894091174, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.65234375, "lr": 1e-05, "finish_rate": 0.873, "comp_len": 478.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 62.7, "frames": {"chat": 251}, "mem_gb": 16.05}
|
| 90 |
+
{"step": 70, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.42447780985000233, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.76953125, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 402.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 72.0, "frames": {"chat": 298}, "mem_gb": 15.78}
|
| 91 |
+
[eval step 70] sample: 'To determine the rank of a matrix, we need to find the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -1 & '
|
| 92 |
+
{"step": 71, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3556813147383121, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.6484375, "lr": 1e-05, "finish_rate": 0.917, "comp_len": 454.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 65.3, "frames": {"chat": 264}, "mem_gb": 15.99}
|
| 93 |
+
{"step": 72, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.35656437207485236, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.69140625, "lr": 1e-05, "finish_rate": 0.948, "comp_len": 393.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 68.1, "frames": {"chat": 305}, "mem_gb": 16.01}
|
| 94 |
+
{"step": 73, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.342289730213893, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.671875, "lr": 1e-05, "finish_rate": 0.953, "comp_len": 372.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 73.3, "frames": {"chat": 322}, "mem_gb": 15.78}
|
| 95 |
+
{"step": 74, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5632880679681898, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.83984375, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 332.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 78.3, "frames": {"chat": 361}, "mem_gb": 15.77}
|
| 96 |
+
{"step": 75, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3835257761793832, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.71875, "lr": 1e-05, "finish_rate": 0.972, "comp_len": 373.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 78.4, "frames": {"chat": 321}, "mem_gb": 15.95}
|
| 97 |
+
{"step": 76, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.45627901047145325, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.8046875, "lr": 1e-05, "finish_rate": 0.967, "comp_len": 390.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 68.4, "frames": {"chat": 307}, "mem_gb": 15.84}
|
| 98 |
+
{"step": 77, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5366780388602366, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 1.0546875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 367.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 79.8, "frames": {"chat": 327}, "mem_gb": 15.94}
|
| 99 |
+
{"step": 78, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6036675174285968, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.8828125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 337.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 84.1, "frames": {"chat": 356}, "mem_gb": 15.56}
|
| 100 |
+
{"step": 79, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.299665022523577, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.59765625, "lr": 1e-05, "finish_rate": 0.915, "comp_len": 424.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 67.8, "frames": {"chat": 283}, "mem_gb": 15.9}
|
| 101 |
+
{"step": 80, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5391059695269291, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.828125, "lr": 1e-05, "finish_rate": 0.998, "comp_len": 299.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 87.0, "frames": {"chat": 401}, "mem_gb": 15.66}
|
| 102 |
+
[eval step 80] sample: 'To determine the rank of a matrix, we need to find the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -1 & '
|
| 103 |
+
{"step": 81, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3527155181619649, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.625, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 416.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 68.6, "frames": {"chat": 288}, "mem_gb": 15.85}
|
| 104 |
+
{"step": 82, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.44629903313796965, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.73828125, "lr": 1e-05, "finish_rate": 0.981, "comp_len": 371.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 71.5, "frames": {"chat": 323}, "mem_gb": 15.89}
|
| 105 |
+
{"step": 83, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6117205785690496, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.92578125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 357.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 77.7, "frames": {"chat": 336}, "mem_gb": 15.78}
|
| 106 |
+
{"step": 84, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5071015296584616, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.8046875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 364.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 74.3, "frames": {"chat": 329}, "mem_gb": 15.69}
|
| 107 |
+
{"step": 85, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.49685094322388373, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.83203125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 349.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 80.2, "frames": {"chat": 343}, "mem_gb": 15.67}
|
| 108 |
+
{"step": 86, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.520678003478547, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.80078125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 357.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 75.6, "frames": {"chat": 336}, "mem_gb": 15.79}
|
| 109 |
+
{"step": 87, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4837478322509676, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.75390625, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 382.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 71.0, "frames": {"chat": 314}, "mem_gb": 16.03}
|
| 110 |
+
{"step": 88, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3054814977413664, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.5703125, "lr": 1e-05, "finish_rate": 0.968, "comp_len": 427.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 60.7, "frames": {"chat": 281}, "mem_gb": 15.86}
|
| 111 |
+
{"step": 89, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3921526405495281, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.64453125, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 401.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 66.4, "frames": {"chat": 299}, "mem_gb": 15.74}
|
| 112 |
+
{"step": 90, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4749718592443193, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.76953125, "lr": 1e-05, "finish_rate": 0.966, "comp_len": 365.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 74.8, "frames": {"chat": 328}, "mem_gb": 15.93}
|
| 113 |
+
[eval step 90] sample: 'To determine the rank of a matrix, we need to find the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -1 & '
|
| 114 |
+
{"step": 91, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3610402118174359, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.62890625, "lr": 1e-05, "finish_rate": 0.919, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 62.8, "frames": {"chat": 271}, "mem_gb": 16.03}
|
| 115 |
+
{"step": 92, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3756434604829798, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.65625, "lr": 1e-05, "finish_rate": 0.874, "comp_len": 446.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 66.2, "frames": {"chat": 269}, "mem_gb": 16.13}
|
| 116 |
+
{"step": 93, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5966137268158918, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.8359375, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 324.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 85.0, "frames": {"chat": 370}, "mem_gb": 15.67}
|
| 117 |
+
{"step": 94, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6140600745432079, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.97265625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 340.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 80.4, "frames": {"chat": 352}, "mem_gb": 15.73}
|
| 118 |
+
{"step": 95, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5368122404473523, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.82421875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 363.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 77.6, "frames": {"chat": 330}, "mem_gb": 15.94}
|
| 119 |
+
{"step": 96, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4360134310909547, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.734375, "lr": 1e-05, "finish_rate": 0.991, "comp_len": 359.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 74.3, "frames": {"chat": 334}, "mem_gb": 15.66}
|
| 120 |
+
{"step": 97, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.42871204929780216, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.7265625, "lr": 1e-05, "finish_rate": 0.967, "comp_len": 357.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 68.0, "frames": {"chat": 336}, "mem_gb": 15.8}
|
| 121 |
+
{"step": 98, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.32529691268876193, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.609375, "lr": 1e-05, "finish_rate": 0.964, "comp_len": 388.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 69.6, "frames": {"chat": 309}, "mem_gb": 15.89}
|
| 122 |
+
{"step": 99, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5433347911451012, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.74609375, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 306.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 86.5, "frames": {"chat": 391}, "mem_gb": 15.77}
|
| 123 |
+
{"step": 100, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.291603554411605, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.5703125, "lr": 1e-05, "finish_rate": 0.83, "comp_len": 454.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 62.9, "frames": {"chat": 264}, "mem_gb": 16.17}
|
| 124 |
+
[eval step 100] sample: 'To determine the rank of a matrix, we need to find the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -1 & '
|
| 125 |
+
{"step": 101, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.46834106676653026, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.7265625, "lr": 1e-05, "finish_rate": 0.994, "comp_len": 343.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 79.9, "frames": {"chat": 349}, "mem_gb": 15.72}
|
| 126 |
+
{"step": 102, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.44036317128607383, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.8125, "lr": 1e-05, "finish_rate": 0.968, "comp_len": 384.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 68.5, "frames": {"chat": 312}, "mem_gb": 15.78}
|
| 127 |
+
{"step": 103, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17561795191069443, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.470703125, "lr": 1e-05, "finish_rate": 0.817, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.0, "frames": {"chat": 229}, "mem_gb": 16.04}
|
| 128 |
+
{"step": 104, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4123180957959344, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.67578125, "lr": 1e-05, "finish_rate": 0.938, "comp_len": 416.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 68.2, "frames": {"chat": 288}, "mem_gb": 16.04}
|
| 129 |
+
{"step": 105, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.555558460294952, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.80078125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 341.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 79.6, "frames": {"chat": 351}, "mem_gb": 15.7}
|
| 130 |
+
{"step": 106, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2160616306680565, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.52734375, "lr": 1e-05, "finish_rate": 0.812, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.4, "frames": {"chat": 223}, "mem_gb": 16.07}
|
| 131 |
+
{"step": 107, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12863720328882336, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.470703125, "lr": 1e-05, "finish_rate": 0.793, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.4, "frames": {"chat": 213}, "mem_gb": 16.03}
|
| 132 |
+
{"step": 108, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5021176442869628, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.78125, "lr": 1e-05, "finish_rate": 0.995, "comp_len": 315.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 84.1, "frames": {"chat": 380}, "mem_gb": 15.88}
|
| 133 |
+
{"step": 109, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3028392949910834, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.59375, "lr": 1e-05, "finish_rate": 0.893, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 63.4, "frames": {"chat": 271}, "mem_gb": 16.07}
|
| 134 |
+
{"step": 110, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.56058387206768, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.81640625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 352.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 77.5, "frames": {"chat": 340}, "mem_gb": 15.96}
|
| 135 |
+
[eval step 110] sample: 'To compute the rank of a matrix, we need to determine the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -'
|
| 136 |
+
{"step": 111, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5761972085774566, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.82421875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 327.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 79.2, "frames": {"chat": 366}, "mem_gb": 15.65}
|
| 137 |
+
{"step": 112, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.21493664028501758, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.52734375, "lr": 1e-05, "finish_rate": 0.854, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 58.4, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 138 |
+
{"step": 113, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3935816221217625, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.66015625, "lr": 1e-05, "finish_rate": 0.968, "comp_len": 384.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 71.2, "frames": {"chat": 312}, "mem_gb": 15.91}
|
| 139 |
+
{"step": 114, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5420919080346823, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.8515625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 330.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 79.6, "frames": {"chat": 363}, "mem_gb": 15.62}
|
| 140 |
+
{"step": 115, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5536963740902021, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.76953125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 371.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 73.1, "frames": {"chat": 323}, "mem_gb": 15.9}
|
| 141 |
+
{"step": 116, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5554547698056326, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.8203125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 346.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 73.4, "frames": {"chat": 346}, "mem_gb": 15.64}
|
| 142 |
+
{"step": 117, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5243664838907619, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.75, "lr": 1e-05, "finish_rate": 0.975, "comp_len": 377.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 69.6, "frames": {"chat": 318}, "mem_gb": 15.79}
|
| 143 |
+
{"step": 118, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.40670796049317964, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 1.1171875, "lr": 1e-05, "finish_rate": 0.935, "comp_len": 436.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 68.1, "frames": {"chat": 275}, "mem_gb": 15.97}
|
| 144 |
+
{"step": 119, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4883209338987867, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.72265625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 347.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 78.9, "frames": {"chat": 345}, "mem_gb": 15.72}
|
| 145 |
+
{"step": 120, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.483824146924975, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.80859375, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 318.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 83.4, "frames": {"chat": 377}, "mem_gb": 15.86}
|
| 146 |
+
[eval step 120] sample: 'To compute the rank of a matrix, we need to determine the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -'
|
| 147 |
+
{"step": 121, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3961287938904638, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.6484375, "lr": 1e-05, "finish_rate": 0.956, "comp_len": 408.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 69.0, "frames": {"chat": 294}, "mem_gb": 16.09}
|
| 148 |
+
{"step": 122, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5481575008081893, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.7421875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 355.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 77.8, "frames": {"chat": 338}, "mem_gb": 15.78}
|
| 149 |
+
{"step": 123, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5607171248226116, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.76171875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 340.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 81.6, "frames": {"chat": 352}, "mem_gb": 15.68}
|
| 150 |
+
{"step": 124, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5120830921638757, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.73828125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 345.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 81.1, "frames": {"chat": 347}, "mem_gb": 15.73}
|
| 151 |
+
{"step": 125, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3293407527307359, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.59375, "lr": 1e-05, "finish_rate": 0.968, "comp_len": 387.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 73.2, "frames": {"chat": 310}, "mem_gb": 15.71}
|
| 152 |
+
{"step": 126, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5882786161034057, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.8671875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 335.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 84.2, "frames": {"chat": 358}, "mem_gb": 15.73}
|
| 153 |
+
{"step": 127, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3442592374465739, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.7734375, "lr": 1e-05, "finish_rate": 0.947, "comp_len": 376.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 70.4, "frames": {"chat": 319}, "mem_gb": 16.01}
|
| 154 |
+
{"step": 128, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.23729529899004848, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.55078125, "lr": 1e-05, "finish_rate": 0.9, "comp_len": 446.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 60.9, "frames": {"chat": 269}, "mem_gb": 16.04}
|
| 155 |
+
{"step": 129, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.41735585475650927, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.703125, "lr": 1e-05, "finish_rate": 0.974, "comp_len": 387.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 74.0, "frames": {"chat": 310}, "mem_gb": 15.9}
|
| 156 |
+
{"step": 130, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29317258806318663, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.6015625, "lr": 1e-05, "finish_rate": 0.834, "comp_len": 463.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 62.5, "frames": {"chat": 259}, "mem_gb": 16.07}
|
| 157 |
+
[eval step 130] sample: 'To determine the rank of a matrix, we need to find the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -1 & '
|
| 158 |
+
{"step": 131, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28399105481548853, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.5390625, "lr": 1e-05, "finish_rate": 0.868, "comp_len": 452.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 65.7, "frames": {"chat": 265}, "mem_gb": 16.05}
|
| 159 |
+
{"step": 132, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5574907290409009, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.84375, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 356.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 80.8, "frames": {"chat": 337}, "mem_gb": 15.6}
|
| 160 |
+
{"step": 133, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.31406076989608506, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.59765625, "lr": 1e-05, "finish_rate": 0.972, "comp_len": 413.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 67.0, "frames": {"chat": 290}, "mem_gb": 15.86}
|
| 161 |
+
{"step": 134, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.48334512109042455, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.70703125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 343.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 76.4, "frames": {"chat": 349}, "mem_gb": 15.79}
|
| 162 |
+
{"step": 135, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3185885916662713, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.56640625, "lr": 1e-05, "finish_rate": 0.928, "comp_len": 412.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 72.0, "frames": {"chat": 291}, "mem_gb": 15.89}
|
| 163 |
+
{"step": 136, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.508149279033641, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.73046875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 350.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 75.3, "frames": {"chat": 342}, "mem_gb": 15.76}
|
| 164 |
+
{"step": 137, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5859596675620725, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.78125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 355.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 82.2, "frames": {"chat": 338}, "mem_gb": 15.7}
|
| 165 |
+
{"step": 138, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.544119408785676, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.7421875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 337.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 88.5, "frames": {"chat": 356}, "mem_gb": 15.67}
|
| 166 |
+
{"step": 139, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.26994493484509485, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.55859375, "lr": 1e-05, "finish_rate": 0.86, "comp_len": 452.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 71.3, "frames": {"chat": 265}, "mem_gb": 16.05}
|
| 167 |
+
{"step": 140, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4654884164984028, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.703125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 339.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 78.5, "frames": {"chat": 353}, "mem_gb": 15.76}
|
| 168 |
+
[eval step 140] sample: 'To compute the rank of a matrix, we need to determine the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -1'
|
| 169 |
+
{"step": 141, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.42584221496824176, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.69140625, "lr": 1e-05, "finish_rate": 0.949, "comp_len": 379.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 72.6, "frames": {"chat": 316}, "mem_gb": 15.95}
|
| 170 |
+
{"step": 142, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3836496248263555, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.640625, "lr": 1e-05, "finish_rate": 0.945, "comp_len": 415.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 72.3, "frames": {"chat": 289}, "mem_gb": 16.1}
|
| 171 |
+
{"step": 143, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.39533769961390647, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.6484375, "lr": 1e-05, "finish_rate": 0.982, "comp_len": 362.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 77.9, "frames": {"chat": 331}, "mem_gb": 15.73}
|
| 172 |
+
{"step": 144, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3865429982875163, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.6171875, "lr": 1e-05, "finish_rate": 0.972, "comp_len": 375.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 74.6, "frames": {"chat": 320}, "mem_gb": 16.05}
|
| 173 |
+
{"step": 145, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.34288982320117456, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.60546875, "lr": 1e-05, "finish_rate": 0.941, "comp_len": 441.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 60.4, "frames": {"chat": 272}, "mem_gb": 15.78}
|
| 174 |
+
{"step": 146, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6008232054781169, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.8046875, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 326.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 85.1, "frames": {"chat": 368}, "mem_gb": 15.7}
|
| 175 |
+
{"step": 147, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.49043334591481835, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.72265625, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 365.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 80.8, "frames": {"chat": 328}, "mem_gb": 15.73}
|
| 176 |
+
{"step": 148, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4927694099167983, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.70703125, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 368.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 75.7, "frames": {"chat": 326}, "mem_gb": 15.94}
|
| 177 |
+
{"step": 149, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5438016358599067, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.7734375, "lr": 1e-05, "finish_rate": 0.997, "comp_len": 329.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 84.7, "frames": {"chat": 364}, "mem_gb": 15.88}
|
| 178 |
+
{"step": 150, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.35079215467702596, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.5703125, "lr": 1e-05, "finish_rate": 0.97, "comp_len": 405.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 69.0, "frames": {"chat": 296}, "mem_gb": 15.93}
|
| 179 |
+
[eval step 150] sample: 'To compute the rank of a matrix, we need to determine the maximum number of linearly independent rows or columns in the matrix. \n\nGiven the matrix:\n\\[\n\\begin{bmatrix}\n12 & -16 & 4 & 16 \\\\\n-9 & 11 & -1'
|
| 180 |
+
checkpoint snapshot queued -> outputs/healed/grid_general_fairness/reap_keep50_s1224_lr1e5/step0150
|
| 181 |
+
wandb: updating run metadata
|
| 182 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 183 |
+
wandb:
|
| 184 |
+
wandb: Run history:
|
| 185 |
+
wandb: comp_len ββββββββββββββββββββ
βββ
ββββββ
ββββββ
ββββ
β
|
| 186 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββββ
β
β
β
ββββββββββββββββ
|
| 187 |
+
wandb: epoch ββββββββββββββββββββββββββββββββββββββββ
|
| 188 |
+
wandb: finish_rate ββββββββββββββββββ
ββββββββββββββββββββββ
|
| 189 |
+
wandb: forward_topk_kl ββββββββββββββββ
βββββββββ
βββββββββββββββ
|
| 190 |
+
wandb: grad_norm ββ
ββββββββββββββββββββββββββββββββββββββ
|
| 191 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 192 |
+
wandb: mem_gb βββ
βββββ
ββ
ββββ
β
βββββββββββββ
ββββββββββββ
|
| 193 |
+
wandb: step βββββββββββββββββββββ
β
β
β
β
βββββββββββββββ
|
| 194 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 195 |
+
wandb: +3 ...
|
| 196 |
+
wandb:
|
| 197 |
+
wandb: Run summary:
|
| 198 |
+
wandb: comp_len 405.4
|
| 199 |
+
wandb: cumulative_loss_tokens 18000000
|
| 200 |
+
wandb: epoch 0
|
| 201 |
+
wandb: finish_rate 0.97
|
| 202 |
+
wandb: forward_topk_kl 0.35079
|
| 203 |
+
wandb: grad_norm 0.57031
|
| 204 |
+
wandb: lr 1e-05
|
| 205 |
+
wandb: mem_gb 15.93
|
| 206 |
+
wandb: step 150
|
| 207 |
+
wandb: t_data_s 0
|
| 208 |
+
wandb: +4 ...
|
| 209 |
+
wandb:
|
| 210 |
+
wandb: π View run reap_keep50_s1224_lr1e5 at: https://wandb.ai/hbfreed/glean-general-grid/runs/sgv88kwj
|
| 211 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-general-grid
|
| 212 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 213 |
+
wandb: Find logs at: outputs/healed/grid_general_fairness/reap_keep50_s1224_lr1e5/wandb/run-20260719_141738-sgv88kwj/logs
|
healed/grid_general_fairness/reap_keep50_s1224_lr1e5.eval.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
healed/grid_math/glean_keep25_s1225.console.log
ADDED
|
@@ -0,0 +1,230 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 6 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/glean_keep25_s1225/wandb/run-20260716_034843-v3mah2z8
|
| 7 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 8 |
+
wandb: Syncing run glean-math-keep25-s1225
|
| 9 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 10 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/v3mah2z8
|
| 11 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 2.09B | teacher overlap=False
|
| 12 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.3958471946775912, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 105.0, "lr": 6e-06, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.4, "frames": {"chat": 191}, "mem_gb": 9.94}
|
| 13 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 14 |
+
[eval step 1] sample: '\nThe answer is **29**\n\nThe numbers are **1**, **3**, **5**, **7**. The four numbers are **1**, **3**, **5**, **7**. The total number of these four numbers is **15**. The number **29** is prime.\n\n**29*'
|
| 15 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.2812339077095192, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 92.5, "lr": 9e-06, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 16 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.22591313500156, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 62.75, "lr": 1.2e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 207}, "mem_gb": 10.0}
|
| 17 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.9309642657568057, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 14.9375, "lr": 1.5e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 208}, "mem_gb": 9.96}
|
| 18 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7706553599588573, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 10.9375, "lr": 1.8e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 19 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6402923999081055, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 10.9375, "lr": 2.1e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 246}, "mem_gb": 9.87}
|
| 20 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5687407882797222, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 3.71875, "lr": 2.4e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.6, "frames": {"chat": 206}, "mem_gb": 10.02}
|
| 21 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.47117147324259084, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 2.46875, "lr": 2.7000000000000002e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 233}, "mem_gb": 10.0}
|
| 22 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.43779878526590765, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 1.875, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 229}, "mem_gb": 9.87}
|
| 23 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3573542028216024, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 1.3515625, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.2, "frames": {"chat": 236}, "mem_gb": 9.9}
|
| 24 |
+
[eval step 10] sample: "To solve this problem, we need to determine which of the four numbers on the diagonal from \\(7\\) to \\(49\\) are prime. Let's break down the steps:\n\n1. **Identify the numbers on the diagonal:**\n The n"
|
| 25 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3389050622591128, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 1.140625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 239}, "mem_gb": 9.79}
|
| 26 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2907669429977735, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.95703125, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.5, "frames": {"chat": 241}, "mem_gb": 9.91}
|
| 27 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2989568690617879, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.98828125, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 226}, "mem_gb": 9.87}
|
| 28 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.25473856463196376, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.77734375, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.4, "frames": {"chat": 234}, "mem_gb": 10.0}
|
| 29 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2544649389738838, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.74609375, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 257}, "mem_gb": 9.99}
|
| 30 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3264326198320836, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.953125, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.5, "frames": {"chat": 208}, "mem_gb": 10.05}
|
| 31 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2732485909540206, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.8515625, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.2, "frames": {"chat": 211}, "mem_gb": 10.02}
|
| 32 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.24720005844794213, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.765625, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 227}, "mem_gb": 10.0}
|
| 33 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.25496282017032307, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.7734375, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 211}, "mem_gb": 9.98}
|
| 34 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.21520358278726537, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.65234375, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 238}, "mem_gb": 10.0}
|
| 35 |
+
[eval step 20] sample: 'To solve this problem, we need to identify the four numbers that lie on the diagonal from the center of the grid to the number \\(7\\) in the given sequence. The sequence is a spiral pattern starting at'
|
| 36 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2216773895036429, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.671875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.5, "frames": {"chat": 237}, "mem_gb": 10.04}
|
| 37 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.24041704051339377, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.68359375, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 208}, "mem_gb": 10.04}
|
| 38 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2146310205236698, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.59375, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 221}, "mem_gb": 10.12}
|
| 39 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20238777070970584, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.5859375, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.6, "frames": {"chat": 232}, "mem_gb": 9.96}
|
| 40 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20690553727087876, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 208}, "mem_gb": 9.99}
|
| 41 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18880456114976357, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.5703125, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.0, "frames": {"chat": 227}, "mem_gb": 9.91}
|
| 42 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20486585594148685, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.5625, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 221}, "mem_gb": 9.94}
|
| 43 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18253140152214717, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.6015625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.2, "frames": {"chat": 232}, "mem_gb": 10.01}
|
| 44 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17699584313053637, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 219}, "mem_gb": 10.01}
|
| 45 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.19838463523512084, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.5, "frames": {"chat": 195}, "mem_gb": 10.1}
|
| 46 |
+
[eval step 30] sample: 'To solve this problem, we need to analyze the arrangement of numbers on a square grid starting from the center and then determine how many of the numbers on the diagonal from \\(7\\) are prime.\n\n### Ste'
|
| 47 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18508145255607863, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.9, "frames": {"chat": 216}, "mem_gb": 10.0}
|
| 48 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15833309386118005, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.9, "frames": {"chat": 208}, "mem_gb": 9.89}
|
| 49 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15909730202872305, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 235}, "mem_gb": 9.88}
|
| 50 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.162696361270609, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.494140625, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 225}, "mem_gb": 9.99}
|
| 51 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.22678135332918414, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.58203125, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 213}, "mem_gb": 10.08}
|
| 52 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15918074906071028, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.515625, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.2, "frames": {"chat": 257}, "mem_gb": 9.76}
|
| 53 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18729867079984397, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.494140625, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 212}, "mem_gb": 10.03}
|
| 54 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15989416230500986, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.67578125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 221}, "mem_gb": 10.0}
|
| 55 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16033389104095597, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.46484375, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.4, "frames": {"chat": 242}, "mem_gb": 10.0}
|
| 56 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14507702404971545, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 220}, "mem_gb": 9.96}
|
| 57 |
+
[eval step 40] sample: 'To solve this problem, we need to identify the four numbers from the set \\(\\{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,'
|
| 58 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15030326494518667, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.484375, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 240}, "mem_gb": 9.86}
|
| 59 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1607294344684109, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 206}, "mem_gb": 9.99}
|
| 60 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1607213422082675, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 226}, "mem_gb": 10.0}
|
| 61 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.201115768094485, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.8046875, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.4, "frames": {"chat": 244}, "mem_gb": 9.79}
|
| 62 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17653826248689244, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.4, "frames": {"chat": 224}, "mem_gb": 10.01}
|
| 63 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1541471158782641, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.0, "frames": {"chat": 271}, "mem_gb": 9.73}
|
| 64 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14582899590842427, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.45703125, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 236}, "mem_gb": 10.01}
|
| 65 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1692912352339054, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.90625, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 232}, "mem_gb": 9.88}
|
| 66 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13167550868373365, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 210}, "mem_gb": 9.94}
|
| 67 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12800594935423384, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.4140625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 217}, "mem_gb": 9.9}
|
| 68 |
+
[eval step 50] sample: 'To solve this problem, we need to analyze the numbers arranged in a spiral pattern on a square grid and determine how many of the four numbers that lie on the same diagonal as the number \\(7\\) are pri'
|
| 69 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/glean_keep25_s1225/step0050
|
| 70 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16889489130682, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.490234375, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 224}, "mem_gb": 10.02}
|
| 71 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.19071834716852754, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.4, "frames": {"chat": 203}, "mem_gb": 9.87}
|
| 72 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15086872272131344, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 239}, "mem_gb": 9.97}
|
| 73 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11240008413158357, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.419921875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 254}, "mem_gb": 9.88}
|
| 74 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11910581262825677, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 241}, "mem_gb": 9.98}
|
| 75 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15324119401192293, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 213}, "mem_gb": 10.01}
|
| 76 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12598269802760334, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 221}, "mem_gb": 10.05}
|
| 77 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1358413405182461, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 196}, "mem_gb": 10.01}
|
| 78 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11328371758495147, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.6, "frames": {"chat": 270}, "mem_gb": 9.82}
|
| 79 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10341172415204346, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.365234375, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 216}, "mem_gb": 9.99}
|
| 80 |
+
[eval step 60] sample: 'To solve this problem, we need to analyze the arrangement of numbers on a square grid and determine how many of the four numbers on the same diagonal as the number \\(7\\) are prime.\n\n### Steps to Solve'
|
| 81 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14843227218265334, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.9, "frames": {"chat": 200}, "mem_gb": 9.96}
|
| 82 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09767647584409764, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 206}, "mem_gb": 9.91}
|
| 83 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09538950520747652, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 234}, "mem_gb": 9.95}
|
| 84 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11610429440625011, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 215}, "mem_gb": 9.96}
|
| 85 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09513266892942289, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 255}, "mem_gb": 9.94}
|
| 86 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11722616833361486, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.5, "frames": {"chat": 250}, "mem_gb": 9.82}
|
| 87 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10852097176260625, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.375, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.2, "frames": {"chat": 242}, "mem_gb": 9.99}
|
| 88 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14525317518096417, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 89 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16521108877193183, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.46484375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 208}, "mem_gb": 10.04}
|
| 90 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1191304967135191, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.447265625, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 208}, "mem_gb": 9.97}
|
| 91 |
+
[eval step 70] sample: 'To solve this problem, we need to analyze the spiral pattern on the square grid and determine the prime numbers among the numbers that appear in the shaded squares.\n\n### Steps to Solve:\n\n1. **Understa'
|
| 92 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12873777522143598, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 209}, "mem_gb": 10.12}
|
| 93 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09841357960260162, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 235}, "mem_gb": 9.96}
|
| 94 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09810730254290005, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 204}, "mem_gb": 9.95}
|
| 95 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.137890988603731, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.41796875, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 96 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10093486693870897, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.34375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.8, "frames": {"chat": 240}, "mem_gb": 10.0}
|
| 97 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11269029565745343, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 246}, "mem_gb": 9.99}
|
| 98 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11716039955631519, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.439453125, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 243}, "mem_gb": 9.81}
|
| 99 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14083528584881375, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 100 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11617199123160293, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 101 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.13478537587194392, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.408203125, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 211}, "mem_gb": 10.01}
|
| 102 |
+
[eval step 80] sample: 'To solve this problem, we need to analyze the arrangement of numbers on a square grid and determine how many of the numbers on the same diagonal as the number \\(7\\) are prime.\n\n### Steps to Solve the '
|
| 103 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10429689287121097, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.3671875, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 232}, "mem_gb": 9.97}
|
| 104 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12193509586900472, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.39453125, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 214}, "mem_gb": 10.01}
|
| 105 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10920041576723258, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 226}, "mem_gb": 9.9}
|
| 106 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1078172554230628, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.38671875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 107 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09039803395358224, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 1.125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 218}, "mem_gb": 9.83}
|
| 108 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09356577665278067, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 109 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12304461509076257, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.4, "frames": {"chat": 215}, "mem_gb": 10.01}
|
| 110 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1115369677213952, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.365234375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 111 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09695998856667429, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.365234375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 209}, "mem_gb": 9.94}
|
| 112 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09343226164858788, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.1, "frames": {"chat": 262}, "mem_gb": 9.87}
|
| 113 |
+
[eval step 90] sample: 'To solve this problem, we need to analyze the arrangement of numbers on a square grid and determine how many of the numbers in the shaded squares on the same diagonal as the number \\(7\\) are prime.\n\nL'
|
| 114 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0932332858821998, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.3, "frames": {"chat": 249}, "mem_gb": 9.96}
|
| 115 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1266877316886559, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.7, "frames": {"chat": 227}, "mem_gb": 10.0}
|
| 116 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10181538004527489, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 221}, "mem_gb": 9.99}
|
| 117 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10009486745695273, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 234}, "mem_gb": 10.01}
|
| 118 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08687792688549185, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.5, "frames": {"chat": 213}, "mem_gb": 9.96}
|
| 119 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08524717839102572, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 213}, "mem_gb": 9.89}
|
| 120 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10703749001209313, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 234}, "mem_gb": 9.92}
|
| 121 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10287342213264977, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.0, "frames": {"chat": 222}, "mem_gb": 9.99}
|
| 122 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11010189882616202, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.3, "frames": {"chat": 227}, "mem_gb": 10.01}
|
| 123 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10967375374240801, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 218}, "mem_gb": 10.04}
|
| 124 |
+
[eval step 100] sample: "To solve this problem, we need to understand the arrangement of numbers on the square grid and how the numbers are placed based on their positions. Here's a step-by-step approach:\n\n1. **Understand the"
|
| 125 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/glean_keep25_s1225/step0100
|
| 126 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11762644656381259, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 223}, "mem_gb": 10.01}
|
| 127 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11568864874821157, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 206}, "mem_gb": 10.01}
|
| 128 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09638762137287607, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 213}, "mem_gb": 9.92}
|
| 129 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.13616388885583727, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 223}, "mem_gb": 9.86}
|
| 130 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1067974968187511, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 227}, "mem_gb": 9.97}
|
| 131 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09873509239495422, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.3671875, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 253}, "mem_gb": 10.0}
|
| 132 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11062951422066739, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 216}, "mem_gb": 10.01}
|
| 133 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08701037775237734, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 205}, "mem_gb": 9.98}
|
| 134 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11230816109282896, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 207}, "mem_gb": 10.06}
|
| 135 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09889225037389746, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 215}, "mem_gb": 9.98}
|
| 136 |
+
[eval step 110] sample: "To solve this problem, we need to analyze the spiral pattern of numbers on a square grid and determine which of the four shaded squares contain prime numbers. Here's a step-by-step approach:\n\n1. **Und"
|
| 137 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07409078115209317, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.38671875, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.0, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 138 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08839880903875455, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.4, "frames": {"chat": 221}, "mem_gb": 10.04}
|
| 139 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06728584335437045, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 254}, "mem_gb": 9.84}
|
| 140 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06614433599614228, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 210}, "mem_gb": 9.97}
|
| 141 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07396190701468537, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 226}, "mem_gb": 9.92}
|
| 142 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08188189537149544, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.30859375, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 212}, "mem_gb": 9.99}
|
| 143 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09220475707668811, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 211}, "mem_gb": 9.93}
|
| 144 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0956091513237295, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.5, "frames": {"chat": 196}, "mem_gb": 9.97}
|
| 145 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07131106523716202, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.28515625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 212}, "mem_gb": 10.0}
|
| 146 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0689470173562877, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 244}, "mem_gb": 9.91}
|
| 147 |
+
[eval step 120] sample: "To solve this problem, we need to analyze the spiral pattern of numbers on a square grid and determine which of the four shaded squares contain prime numbers. Here's a step-by-step approach:\n\n1. **Und"
|
| 148 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07394362696452687, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 1.6484375, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 222}, "mem_gb": 9.95}
|
| 149 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07991834989693015, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 218}, "mem_gb": 10.0}
|
| 150 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08040042725556219, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.2890625, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.3, "frames": {"chat": 252}, "mem_gb": 9.88}
|
| 151 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08868840474116926, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 202}, "mem_gb": 10.05}
|
| 152 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09232083391323685, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.5, "frames": {"chat": 237}, "mem_gb": 10.0}
|
| 153 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08302971795774065, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.298828125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.7, "frames": {"chat": 234}, "mem_gb": 9.99}
|
| 154 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0661788280802158, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 215}, "mem_gb": 10.0}
|
| 155 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0645129962954515, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.0, "frames": {"chat": 234}, "mem_gb": 9.93}
|
| 156 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06428662312957459, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 216}, "mem_gb": 9.99}
|
| 157 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07294256055513397, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 210}, "mem_gb": 9.95}
|
| 158 |
+
[eval step 130] sample: "To solve this problem, we need to analyze the spiral pattern of numbers on a square grid and determine which of the four shaded squares contain prime numbers. Here's a step-by-step approach:\n\n1. **Und"
|
| 159 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08209835358545339, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.306640625, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 160 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07414536922099069, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 161 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07274689665936554, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.30078125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.3, "frames": {"chat": 225}, "mem_gb": 9.95}
|
| 162 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07239032591326783, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.3, "frames": {"chat": 253}, "mem_gb": 9.85}
|
| 163 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0755888467406854, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.2890625, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 247}, "mem_gb": 9.97}
|
| 164 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0824168190576757, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.3125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 238}, "mem_gb": 9.97}
|
| 165 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09111157214685033, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 235}, "mem_gb": 10.0}
|
| 166 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08819460893472035, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 215}, "mem_gb": 9.97}
|
| 167 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07084991472096493, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.298828125, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.4, "frames": {"chat": 268}, "mem_gb": 9.97}
|
| 168 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07839255714469279, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.4, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 169 |
+
[eval step 140] sample: "To solve this problem, we need to analyze the spiral pattern of numbers on the square grid and determine which of the four shaded squares contain prime numbers. Here's a step-by-step approach:\n\n1. **U"
|
| 170 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07923957366729155, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 252}, "mem_gb": 9.93}
|
| 171 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07850046219552556, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.9, "frames": {"chat": 223}, "mem_gb": 10.01}
|
| 172 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0933897839378876, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.6, "frames": {"chat": 226}, "mem_gb": 10.0}
|
| 173 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09283627185517301, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.1, "frames": {"chat": 208}, "mem_gb": 10.05}
|
| 174 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06709443831786824, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.7, "frames": {"chat": 240}, "mem_gb": 9.93}
|
| 175 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08292833760709813, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.28515625, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 222}, "mem_gb": 9.93}
|
| 176 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06971678353363338, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 4.125, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 236}, "mem_gb": 10.0}
|
| 177 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06898172454525096, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.2890625, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 217}, "mem_gb": 9.96}
|
| 178 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07392021829135095, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.345703125, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 252}, "mem_gb": 9.88}
|
| 179 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0710504538111544, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.28515625, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.3, "frames": {"chat": 222}, "mem_gb": 9.99}
|
| 180 |
+
[eval step 150] sample: "To solve this problem, we need to analyze the spiral pattern of numbers on a square grid and determine which of the four shaded squares contain prime numbers. Here's a step-by-step approach:\n\n1. **Und"
|
| 181 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/glean_keep25_s1225/step0150
|
| 182 |
+
wandb: updating run metadata
|
| 183 |
+
wandb: uploading summary, console lines 170-170
|
| 184 |
+
wandb:
|
| 185 |
+
wandb: Run history:
|
| 186 |
+
wandb: comp_len ββββββ
ββββββββββββββββββ
βββ
β
βββ
ββββ
βββββ
|
| 187 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββββββ
β
β
β
ββββββββββββββ
|
| 188 |
+
wandb: epoch βββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
βββββββββββββ
|
| 189 |
+
wandb: finish_rate βββββββ
βββββ
ββββ
βββ
ββββββ
βββββ
ββββββββββ
|
| 190 |
+
wandb: forward_topk_kl ββββββββββββββββββββββββββββββββββββββββ
|
| 191 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 192 |
+
wandb: lr βββ
βββββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: mem_gb ββββββββββββ
ββββββ
βββββββ
ββ
βββββ
βββββββ
β
|
| 194 |
+
wandb: step βββββββββββββββββββ
β
β
β
β
β
β
βββββββββββββββ
|
| 195 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 196 |
+
wandb: +3 ...
|
| 197 |
+
wandb:
|
| 198 |
+
wandb: Run summary:
|
| 199 |
+
wandb: comp_len 540.5
|
| 200 |
+
wandb: cumulative_loss_tokens 18000000
|
| 201 |
+
wandb: epoch 2
|
| 202 |
+
wandb: finish_rate 0.847
|
| 203 |
+
wandb: forward_topk_kl 0.07105
|
| 204 |
+
wandb: grad_norm 0.28516
|
| 205 |
+
wandb: lr 3e-05
|
| 206 |
+
wandb: mem_gb 9.99
|
| 207 |
+
wandb: step 150
|
| 208 |
+
wandb: t_data_s 0
|
| 209 |
+
wandb: +4 ...
|
| 210 |
+
wandb:
|
| 211 |
+
wandb: π View run glean-math-keep25-s1225 at: https://wandb.ai/hbfreed/glean-grid/runs/v3mah2z8
|
| 212 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 213 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 214 |
+
wandb: Find logs at: outputs/healed/grid_math/glean_keep25_s1225/wandb/run-20260716_034843-v3mah2z8/logs
|
| 215 |
+
{
|
| 216 |
+
"correct": 560,
|
| 217 |
+
"accuracy": 0.4245640636846095,
|
| 218 |
+
"finished": 1259,
|
| 219 |
+
"finish_rate": 0.954510993176649,
|
| 220 |
+
"mean_completion_tokens": 173.51023502653524
|
| 221 |
+
}
|
| 222 |
+
saved item-level results -> outputs/evals/grid_math/glean_keep25_s1225_step100_chat.json
|
| 223 |
+
{
|
| 224 |
+
"correct": 564,
|
| 225 |
+
"accuracy": 0.4275966641394996,
|
| 226 |
+
"finished": 1270,
|
| 227 |
+
"finish_rate": 0.9628506444275967,
|
| 228 |
+
"mean_completion_tokens": 173.10007581501137
|
| 229 |
+
}
|
| 230 |
+
saved item-level results -> outputs/evals/grid_math/glean_keep25_s1225_step150_chat.json
|
healed/grid_math/glean_keep25_s1226.console.log
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run vhgtf0ej
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/glean_keep25_s1226/wandb/run-20260716_034735-vhgtf0ej
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run glean-math-keep25-s1226
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/vhgtf0ej
|
| 12 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 2.09B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.4740517740617196, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 121.0, "lr": 6e-06, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 254}, "mem_gb": 9.82}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: 'The value is 28, and the perimeter of the resulting triangle is 28. The problem is to find the perimeter of the resulting triangle, given the conditions mentioned.\n\nThe perimeter of the resulting tria'
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.377854461752375, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 95.0, "lr": 9e-06, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 241}, "mem_gb": 9.98}
|
| 17 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.2003990252320964, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 59.75, "lr": 1.2e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 213}, "mem_gb": 10.01}
|
| 18 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.8684858515068888, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 14.8125, "lr": 1.5e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.0, "frames": {"chat": 221}, "mem_gb": 10.05}
|
| 19 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.841784818589439, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 8.3125, "lr": 1.8e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.3, "frames": {"chat": 196}, "mem_gb": 10.01}
|
| 20 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6113010039225221, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 8.375, "lr": 2.1e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 36.6, "frames": {"chat": 270}, "mem_gb": 9.82}
|
| 21 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5845785550904771, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 4.53125, "lr": 2.4e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.3, "frames": {"chat": 216}, "mem_gb": 9.99}
|
| 22 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.49846355576987067, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 3.390625, "lr": 2.7000000000000002e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.9, "frames": {"chat": 200}, "mem_gb": 9.96}
|
| 23 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4226150450040897, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 2.609375, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 206}, "mem_gb": 9.91}
|
| 24 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3381338079671065, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 1.7421875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 234}, "mem_gb": 9.95}
|
| 25 |
+
[eval step 10] sample: "To solve this problem, we need to determine the lengths of the sides of the triangle given the perimeter and the midpoints of its sides. Let's break down the problem into manageable steps:\n\n1. **Under"
|
| 26 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.34357373327190677, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 1.3828125, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.3, "frames": {"chat": 215}, "mem_gb": 9.96}
|
| 27 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2818106052008768, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 1.1484375, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 255}, "mem_gb": 9.94}
|
| 28 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.27427206053423386, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.91796875, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.4, "frames": {"chat": 250}, "mem_gb": 9.82}
|
| 29 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2693552000547449, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.8359375, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 242}, "mem_gb": 9.99}
|
| 30 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29865186369282504, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.8671875, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 31 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3619642677543064, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.99609375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 208}, "mem_gb": 10.04}
|
| 32 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.25310858338586983, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.81640625, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 208}, "mem_gb": 9.97}
|
| 33 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2819262358199805, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.80078125, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 209}, "mem_gb": 10.12}
|
| 34 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2152011162099739, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.71875, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 235}, "mem_gb": 9.96}
|
| 35 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.21725329664709667, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.7265625, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.6, "frames": {"chat": 204}, "mem_gb": 9.95}
|
| 36 |
+
[eval step 20] sample: 'To solve this problem, we need to understand the properties of a triangle and its midpoints.\n\n1. **Understanding the Properties:**\n - The perimeter of a triangle is given by the sum of its sides.\n '
|
| 37 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2732300681939969, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.87890625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 38 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20615510911339274, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.62109375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 240}, "mem_gb": 10.0}
|
| 39 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.19796610735058784, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.62109375, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 246}, "mem_gb": 9.99}
|
| 40 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.21532276370519152, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.703125, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.4, "frames": {"chat": 243}, "mem_gb": 9.81}
|
| 41 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2322601548684761, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.6875, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.6, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 42 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.21518369818814098, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.6015625, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.3, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 43 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.23441068366014708, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.6953125, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 211}, "mem_gb": 10.01}
|
| 44 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1949662358665218, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.62109375, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.2, "frames": {"chat": 232}, "mem_gb": 9.97}
|
| 45 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.21759224594825258, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.6171875, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 214}, "mem_gb": 10.01}
|
| 46 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18679429283955445, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 226}, "mem_gb": 9.9}
|
| 47 |
+
[eval step 30] sample: 'To solve this problem, we need to understand the properties of a triangle and its midpoints.\n\n1. **Understand the Properties:**\n - The perimeter of a triangle is the sum of its side lengths.\n - Th'
|
| 48 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17500035125799476, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.3, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 49 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16020464946124702, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 218}, "mem_gb": 9.83}
|
| 50 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17072588143019626, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 51 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2061376795306181, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.6015625, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 215}, "mem_gb": 10.01}
|
| 52 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18651440346712866, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.3, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 53 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16298052088283002, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 209}, "mem_gb": 9.94}
|
| 54 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15828031261581926, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 262}, "mem_gb": 9.87}
|
| 55 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1595262515474111, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.466796875, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 249}, "mem_gb": 9.96}
|
| 56 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.19758843879966687, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 227}, "mem_gb": 10.0}
|
| 57 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14780844743441168, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 221}, "mem_gb": 9.99}
|
| 58 |
+
[eval step 40] sample: "To solve this problem, we need to understand the properties of the midpoints of a triangle's sides and how they affect the perimeter.\n\n1. **Midpoints of Sides:**\n - The midpoint of a side of length "
|
| 59 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16207660000277682, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 234}, "mem_gb": 10.01}
|
| 60 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13810028650884826, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.42578125, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.3, "frames": {"chat": 213}, "mem_gb": 9.96}
|
| 61 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13506372714247555, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.40234375, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.5, "frames": {"chat": 213}, "mem_gb": 9.89}
|
| 62 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14545865754342327, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 234}, "mem_gb": 9.92}
|
| 63 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14481995174276333, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 222}, "mem_gb": 9.99}
|
| 64 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17587772664371878, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.498046875, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.1, "frames": {"chat": 227}, "mem_gb": 10.01}
|
| 65 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1550076254642258, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 218}, "mem_gb": 10.04}
|
| 66 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17269171449063966, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 223}, "mem_gb": 10.01}
|
| 67 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17547114912066608, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 206}, "mem_gb": 10.01}
|
| 68 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1537847354217743, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.490234375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 213}, "mem_gb": 9.92}
|
| 69 |
+
[eval step 50] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is given as"
|
| 70 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/glean_keep25_s1226/step0050
|
| 71 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18943238889773686, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 223}, "mem_gb": 9.86}
|
| 72 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16693324066617837, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 227}, "mem_gb": 9.97}
|
| 73 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14277432735928644, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.4453125, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.6, "frames": {"chat": 253}, "mem_gb": 10.0}
|
| 74 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1437762385570444, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.83984375, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.9, "frames": {"chat": 216}, "mem_gb": 10.01}
|
| 75 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1300227014787495, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.412109375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 205}, "mem_gb": 9.98}
|
| 76 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15664660246831674, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.46875, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.8, "frames": {"chat": 207}, "mem_gb": 10.06}
|
| 77 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1445078781637363, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.4140625, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 215}, "mem_gb": 9.98}
|
| 78 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10567276090113446, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.373046875, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 79 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.13750520292868218, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.3, "frames": {"chat": 221}, "mem_gb": 10.04}
|
| 80 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10064160097377996, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 254}, "mem_gb": 9.84}
|
| 81 |
+
[eval step 60] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is given as"
|
| 82 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09463851169059052, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.375, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.9, "frames": {"chat": 210}, "mem_gb": 9.97}
|
| 83 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10595882567154864, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.345703125, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.2, "frames": {"chat": 226}, "mem_gb": 9.92}
|
| 84 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12365473369524503, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.38671875, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.3, "frames": {"chat": 212}, "mem_gb": 9.99}
|
| 85 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12891767050419004, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.365234375, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.7, "frames": {"chat": 211}, "mem_gb": 9.93}
|
| 86 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.13171432805840547, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.4, "frames": {"chat": 196}, "mem_gb": 9.97}
|
| 87 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11064968370975306, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.38671875, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 212}, "mem_gb": 10.0}
|
| 88 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1052026781039002, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 244}, "mem_gb": 9.91}
|
| 89 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10334055133834481, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 222}, "mem_gb": 9.95}
|
| 90 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11475445196203267, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.3, "frames": {"chat": 218}, "mem_gb": 10.0}
|
| 91 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12215668223105992, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.2, "frames": {"chat": 252}, "mem_gb": 9.88}
|
| 92 |
+
[eval step 70] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by the midpoints of the sides of the original triangle.\n\n1. **Identify the Original Triangle:**\n Let the '
|
| 93 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.13357413773555307, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 202}, "mem_gb": 10.05}
|
| 94 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15701979057999949, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.4, "frames": {"chat": 237}, "mem_gb": 10.0}
|
| 95 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11951561769278099, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.359375, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 234}, "mem_gb": 9.99}
|
| 96 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09150583964148536, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 215}, "mem_gb": 10.0}
|
| 97 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08957106544353689, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 234}, "mem_gb": 9.93}
|
| 98 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08833280240970974, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.3125, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 216}, "mem_gb": 9.99}
|
| 99 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1060180736657232, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.34375, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.3, "frames": {"chat": 210}, "mem_gb": 9.95}
|
| 100 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10358950277809054, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.34375, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 101 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09346498899217695, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 102 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10706426500252758, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.40625, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 225}, "mem_gb": 9.95}
|
| 103 |
+
[eval step 80] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and its midpoints.\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is given as 28.\n '
|
| 104 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10597167926300317, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 253}, "mem_gb": 9.85}
|
| 105 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10689509418696785, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.359375, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.6, "frames": {"chat": 247}, "mem_gb": 9.97}
|
| 106 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10978524073281636, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.3671875, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.4, "frames": {"chat": 238}, "mem_gb": 9.97}
|
| 107 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11432554268656919, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 235}, "mem_gb": 10.0}
|
| 108 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11262670917728294, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 215}, "mem_gb": 9.97}
|
| 109 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11482771853323405, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.443359375, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 268}, "mem_gb": 9.97}
|
| 110 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10706278533112878, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 111 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11632729308865965, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.38671875, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 252}, "mem_gb": 9.93}
|
| 112 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09904835979019602, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.365234375, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 223}, "mem_gb": 10.01}
|
| 113 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15139277683890734, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.45703125, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.6, "frames": {"chat": 226}, "mem_gb": 10.0}
|
| 114 |
+
[eval step 90] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by the midpoints of the sides of a triangle with a given perimeter.\n\n### Steps to Solve:\n\n1. **Understand t'
|
| 115 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12696473920823384, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.0, "frames": {"chat": 208}, "mem_gb": 10.05}
|
| 116 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0981225883629794, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.6, "frames": {"chat": 240}, "mem_gb": 9.93}
|
| 117 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11373697855863721, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 222}, "mem_gb": 9.93}
|
| 118 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09599229777337362, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 236}, "mem_gb": 10.0}
|
| 119 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0905342026155442, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.9, "frames": {"chat": 217}, "mem_gb": 9.96}
|
| 120 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10233824915119136, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 252}, "mem_gb": 9.88}
|
| 121 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09226011450669418, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 222}, "mem_gb": 9.99}
|
| 122 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1017231995769466, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.3671875, "lr": 3e-05, "finish_rate": 0.901, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.4, "frames": {"chat": 242}, "mem_gb": 9.87}
|
| 123 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.13698266774720202, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 219}, "mem_gb": 9.93}
|
| 124 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09786459891942019, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.9, "frames": {"chat": 223}, "mem_gb": 9.94}
|
| 125 |
+
[eval step 100] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 126 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/glean_keep25_s1226/step0100
|
| 127 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11257973559337357, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 232}, "mem_gb": 9.95}
|
| 128 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11817761343022187, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.832, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 220}, "mem_gb": 10.0}
|
| 129 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1162180175398166, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 210}, "mem_gb": 10.04}
|
| 130 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10464485003838005, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 226}, "mem_gb": 9.97}
|
| 131 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09739518145142743, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.34375, "lr": 3e-05, "finish_rate": 0.741, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.9, "frames": {"chat": 212}, "mem_gb": 10.0}
|
| 132 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09368765245955438, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.58203125, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 236}, "mem_gb": 10.01}
|
| 133 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07762365770225103, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.928, "comp_len": 454.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 264}, "mem_gb": 9.88}
|
| 134 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1221225647800602, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.52734375, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 229}, "mem_gb": 9.98}
|
| 135 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0774544737749733, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 465.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.3, "frames": {"chat": 258}, "mem_gb": 9.85}
|
| 136 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11255108813242987, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.3984375, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 137 |
+
[eval step 110] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by the midpoints of the sides of the original triangle.\n\n1. **Midpoints of a Triangle:**\n - The midpoint '
|
| 138 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0868233092402108, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.88, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.7, "frames": {"chat": 249}, "mem_gb": 9.93}
|
| 139 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07358991829988858, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.5, "frames": {"chat": 220}, "mem_gb": 9.99}
|
| 140 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08318625353276729, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.31640625, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 223}, "mem_gb": 9.99}
|
| 141 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07259801399639497, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.3, "frames": {"chat": 221}, "mem_gb": 10.0}
|
| 142 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07171069395930196, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.2890625, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 230}, "mem_gb": 9.9}
|
| 143 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08987705074110999, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.3125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 214}, "mem_gb": 9.97}
|
| 144 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11477152344050506, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 214}, "mem_gb": 9.99}
|
| 145 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08955526669428994, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 210}, "mem_gb": 10.04}
|
| 146 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09731406231168657, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.6, "frames": {"chat": 214}, "mem_gb": 10.0}
|
| 147 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0877075838214097, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.302734375, "lr": 3e-05, "finish_rate": 0.791, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.5, "frames": {"chat": 215}, "mem_gb": 9.95}
|
| 148 |
+
[eval step 120] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the triangle is 28.\n - The midp"
|
| 149 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09859609124142056, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 208}, "mem_gb": 9.99}
|
| 150 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07579588066336389, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.291015625, "lr": 3e-05, "finish_rate": 0.789, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 218}, "mem_gb": 9.88}
|
| 151 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06975389175747211, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 233}, "mem_gb": 9.9}
|
| 152 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06521768421179926, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 231}, "mem_gb": 9.94}
|
| 153 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08123537786286324, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.306640625, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 235}, "mem_gb": 10.13}
|
| 154 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07916606919132173, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 216}, "mem_gb": 9.99}
|
| 155 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0735622343561612, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.28125, "lr": 3e-05, "finish_rate": 0.831, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.4, "frames": {"chat": 237}, "mem_gb": 10.01}
|
| 156 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10053933701403439, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.4, "frames": {"chat": 203}, "mem_gb": 10.01}
|
| 157 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07754695314977629, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.873, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 236}, "mem_gb": 10.04}
|
| 158 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07985588553401952, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.2890625, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 214}, "mem_gb": 10.01}
|
| 159 |
+
[eval step 130] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by the midpoints of the sides of a given triangle.\n\n1. **Understand the Midpoints:**\n - The midpoint of a'
|
| 160 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0826984562702477, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.302734375, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 209}, "mem_gb": 10.0}
|
| 161 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07694104558015243, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.306640625, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 468.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 256}, "mem_gb": 10.0}
|
| 162 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07371389015351111, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.878, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 230}, "mem_gb": 9.87}
|
| 163 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07886174646293123, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.6, "frames": {"chat": 230}, "mem_gb": 10.06}
|
| 164 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09088345281931882, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 227}, "mem_gb": 9.95}
|
| 165 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09291109785580387, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.31640625, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 166 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09028420611228793, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.302734375, "lr": 3e-05, "finish_rate": 0.699, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.2, "frames": {"chat": 206}, "mem_gb": 10.03}
|
| 167 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08365592800673718, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.82, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 228}, "mem_gb": 9.9}
|
| 168 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08592219653519181, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.306640625, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 224}, "mem_gb": 10.0}
|
| 169 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07787073799719413, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.2890625, "lr": 3e-05, "finish_rate": 0.66, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 200}, "mem_gb": 10.03}
|
| 170 |
+
[eval step 140] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by the midpoints of the sides of the original triangle.\n\n1. **Understand the Midpoints:**\n The midpoints '
|
| 171 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07876405616238868, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.714, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.8, "frames": {"chat": 196}, "mem_gb": 10.01}
|
| 172 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0733729308873415, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 223}, "mem_gb": 9.99}
|
| 173 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07316665141967435, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.291015625, "lr": 3e-05, "finish_rate": 0.869, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 213}, "mem_gb": 9.89}
|
| 174 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06799636307867865, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 232}, "mem_gb": 9.93}
|
| 175 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06842424086419245, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 223}, "mem_gb": 9.93}
|
| 176 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07442820003066833, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.85, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 233}, "mem_gb": 10.02}
|
| 177 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08168693628491212, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.816, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 217}, "mem_gb": 10.01}
|
| 178 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11692778237918391, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.752, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 202}, "mem_gb": 10.08}
|
| 179 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0718278338014769, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 253}, "mem_gb": 9.93}
|
| 180 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06792547624123593, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.263671875, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 231}, "mem_gb": 9.94}
|
| 181 |
+
[eval step 150] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by the midpoints of the sides of the original triangle.\n\n1. **Midpoints of a Triangle:**\n The midpoint of'
|
| 182 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/glean_keep25_s1226/step0150
|
| 183 |
+
wandb: updating run metadata
|
| 184 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 185 |
+
wandb: uploading summary, console lines 170-170
|
| 186 |
+
wandb:
|
| 187 |
+
wandb: Run history:
|
| 188 |
+
wandb: comp_len ββββββββ
β
βββ
β
β
βββββββββββββ
βββ
βββββ
βββ
ββ
|
| 189 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββββ
β
β
β
β
β
β
βββββββββββββ
|
| 190 |
+
wandb: epoch ββββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
βββββββββββββββ
|
| 191 |
+
wandb: finish_rate βββββββββ
β
β
βββββββββββ
ββββββ
ββββββββββ
ββ
|
| 192 |
+
wandb: forward_topk_kl ββββ
ββββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 195 |
+
wandb: mem_gb ββ
βββββ
β
ββββ
β
ββββ
ββββ
ββ
βββββ
ββ
ββββ
β
β
β
ββ
β
|
| 196 |
+
wandb: step ββββββββββββββββββββββββ
β
β
β
βββββββββββββ
|
| 197 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 198 |
+
wandb: +3 ...
|
| 199 |
+
wandb:
|
| 200 |
+
wandb: Run summary:
|
| 201 |
+
wandb: comp_len 519.5
|
| 202 |
+
wandb: cumulative_loss_tokens 18000000
|
| 203 |
+
wandb: epoch 2
|
| 204 |
+
wandb: finish_rate 0.879
|
| 205 |
+
wandb: forward_topk_kl 0.06793
|
| 206 |
+
wandb: grad_norm 0.26367
|
| 207 |
+
wandb: lr 3e-05
|
| 208 |
+
wandb: mem_gb 9.94
|
| 209 |
+
wandb: step 150
|
| 210 |
+
wandb: t_data_s 0
|
| 211 |
+
wandb: +4 ...
|
| 212 |
+
wandb:
|
| 213 |
+
wandb: π View run glean-math-keep25-s1226 at: https://wandb.ai/hbfreed/glean-grid/runs/vhgtf0ej
|
| 214 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 215 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 216 |
+
wandb: Find logs at: outputs/healed/grid_math/glean_keep25_s1226/wandb/run-20260716_034735-vhgtf0ej/logs
|
| 217 |
+
{
|
| 218 |
+
"correct": 557,
|
| 219 |
+
"accuracy": 0.422289613343442,
|
| 220 |
+
"finished": 1276,
|
| 221 |
+
"finish_rate": 0.9673995451099318,
|
| 222 |
+
"mean_completion_tokens": 191.71645185746777
|
| 223 |
+
}
|
| 224 |
+
saved item-level results -> outputs/evals/grid_math/glean_keep25_s1226_step100_chat.json
|
| 225 |
+
{
|
| 226 |
+
"correct": 575,
|
| 227 |
+
"accuracy": 0.4359363153904473,
|
| 228 |
+
"finished": 1279,
|
| 229 |
+
"finish_rate": 0.9696739954510993,
|
| 230 |
+
"mean_completion_tokens": 193.40788476118271
|
| 231 |
+
}
|
| 232 |
+
saved item-level results -> outputs/evals/grid_math/glean_keep25_s1226_step150_chat.json
|
healed/grid_math/keep75.log
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2026-07-16T14:28:11-07:00 === keep-75 finish, 2 GPUs concurrent (3rd idle for power) ===
|
| 2 |
+
2026-07-16T14:28:11-07:00 RESUMING glean_keep75_s1226 from step0100 (optimizer+scheduler+data position restored)
|
| 3 |
+
2026-07-16T14:28:11-07:00 RESUMING glean_keep75_s1225 from step0100 (optimizer+scheduler+data position restored)
|
| 4 |
+
2026-07-16T15:09:12-07:00 eval glean_keep75_s1225 step100
|
| 5 |
+
2026-07-16T15:10:41-07:00 eval glean_keep75_s1226 step100
|
| 6 |
+
2026-07-16T15:11:18-07:00 eval glean_keep75_s1225 step150
|
| 7 |
+
2026-07-16T15:12:37-07:00 eval glean_keep75_s1226 step150
|
| 8 |
+
2026-07-16T15:13:13-07:00 glean_keep75_s1225 done -> 0.689158453373768
|
| 9 |
+
2026-07-16T15:13:13-07:00 healing uniform_keep75_s1224 on GPU-a6acf07f (port 8392)
|
| 10 |
+
2026-07-16T15:14:35-07:00 glean_keep75_s1226 done -> 0.7012888551933283
|
| 11 |
+
2026-07-16T15:14:35-07:00 healing glean_keep75_s1224 on GPU-8ca70870 (port 8391)
|
| 12 |
+
2026-07-16T17:14:44-07:00 eval glean_keep75_s1224 step100
|
| 13 |
+
2026-07-16T17:16:42-07:00 eval glean_keep75_s1224 step150
|
| 14 |
+
2026-07-16T17:18:39-07:00 glean_keep75_s1224 done -> 0.6914329037149356
|
| 15 |
+
2026-07-16T17:18:39-07:00 healing reap_keep75_s1224 on GPU-8ca70870 (port 8391)
|
| 16 |
+
2026-07-16T17:24:11-07:00 eval uniform_keep75_s1224 step100
|
| 17 |
+
2026-07-16T17:26:08-07:00 eval uniform_keep75_s1224 step150
|
| 18 |
+
2026-07-16T17:28:03-07:00 uniform_keep75_s1224 done -> 0.6338134950720242
|
| 19 |
+
2026-07-16T17:28:03-07:00 healing reap_keep75_s1225 on GPU-a6acf07f (port 8392)
|
| 20 |
+
2026-07-16T19:38:42-07:00 eval reap_keep75_s1225 step100
|
| 21 |
+
2026-07-16T19:40:26-07:00 eval reap_keep75_s1225 step150
|
| 22 |
+
2026-07-16T19:42:01-07:00 reap_keep75_s1225 done -> 0.6755117513267627
|
| 23 |
+
2026-07-16T19:42:01-07:00 healing uniform_keep75_s1226 on GPU-a6acf07f (port 8392)
|
| 24 |
+
2026-07-16T19:46:16-07:00 === keep-75 finish, 2 GPUs concurrent (3rd idle for power) ===
|
| 25 |
+
2026-07-16T19:46:16-07:00 glean_keep75_s1225 already done, skip
|
| 26 |
+
2026-07-16T19:46:16-07:00 glean_keep75_s1226 already done, skip
|
| 27 |
+
2026-07-16T19:46:16-07:00 uniform_keep75_s1224 already done, skip
|
| 28 |
+
2026-07-16T19:46:16-07:00 glean_keep75_s1224 already done, skip
|
| 29 |
+
2026-07-16T19:46:16-07:00 reap_keep75_s1225 already done, skip
|
| 30 |
+
2026-07-16T19:46:16-07:00 RESUMING reap_keep75_s1224 from step0050 (optimizer+scheduler+data position restored)
|
| 31 |
+
2026-07-16T19:46:16-07:00 healing reap_keep75_s1226 on GPU-a6acf07f (port 8392)
|
| 32 |
+
2026-07-16T21:00:24-07:00 === keep-75 finish, 2 GPUs concurrent (3rd idle for power) ===
|
| 33 |
+
2026-07-16T21:00:24-07:00 glean_keep75_s1226 already done, skip
|
| 34 |
+
2026-07-16T21:00:24-07:00 glean_keep75_s1225 already done, skip
|
| 35 |
+
2026-07-16T21:00:24-07:00 glean_keep75_s1224 already done, skip
|
| 36 |
+
2026-07-16T21:00:24-07:00 uniform_keep75_s1224 already done, skip
|
| 37 |
+
2026-07-16T21:00:24-07:00 RESUMING reap_keep75_s1224 from step0050 (optimizer+scheduler+data position restored)
|
| 38 |
+
2026-07-16T21:00:24-07:00 reap_keep75_s1225 already done, skip
|
| 39 |
+
2026-07-16T21:00:24-07:00 RESUMING reap_keep75_s1226 from step0050 (optimizer+scheduler+data position restored)
|
| 40 |
+
2026-07-16T22:27:41-07:00 eval reap_keep75_s1224 step100
|
| 41 |
+
2026-07-16T22:28:47-07:00 eval reap_keep75_s1226 step100
|
| 42 |
+
2026-07-16T22:29:21-07:00 eval reap_keep75_s1224 step150
|
| 43 |
+
2026-07-16T22:30:31-07:00 eval reap_keep75_s1226 step150
|
| 44 |
+
2026-07-16T22:30:57-07:00 reap_keep75_s1224 done -> 0.6846095526914329
|
| 45 |
+
2026-07-16T22:30:57-07:00 healing uniform_keep75_s1225 on GPU-864c54df (port 8391)
|
| 46 |
+
2026-07-16T22:32:08-07:00 reap_keep75_s1226 done -> 0.6732373009855952
|
| 47 |
+
2026-07-16T22:32:08-07:00 healing uniform_keep75_s1226 on GPU-a6acf07f (port 8392)
|
| 48 |
+
2026-07-17T00:28:54-07:00 eval uniform_keep75_s1226 step100
|
| 49 |
+
2026-07-17T00:29:31-07:00 eval uniform_keep75_s1225 step100
|
| 50 |
+
2026-07-17T00:30:52-07:00 eval uniform_keep75_s1226 step150
|
| 51 |
+
2026-07-17T00:31:28-07:00 eval uniform_keep75_s1225 step150
|
| 52 |
+
2026-07-17T00:32:51-07:00 uniform_keep75_s1226 done -> 0.6322971948445792
|
| 53 |
+
2026-07-17T00:32:51-07:00 LANE GPU-a6acf07f complete
|
| 54 |
+
2026-07-17T00:33:23-07:00 uniform_keep75_s1225 done -> 0.640636846095527
|
| 55 |
+
2026-07-17T00:33:23-07:00 LANE GPU-864c54df complete
|
| 56 |
+
2026-07-17T00:33:23-07:00 === KEEP75 COMPLETE ===
|
healed/grid_math/reap_keep25_s1224.console.log
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run q0iymgs1
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/reap_keep25_s1224/wandb/run-20260716_071102-q0iymgs1
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run reap-math-keep25-s1224
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/q0iymgs1
|
| 12 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 2.09B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 5.414990429035822, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 92.0, "lr": 6e-06, "finish_rate": 0.907, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 236}, "mem_gb": 9.77}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: '\nisk= \n\\\na <\n\n1. \n.ile = \n formula, eq.subsistry-nals,\n equr. ome,m. -Te \\\'\n dolect{for["cey"$-\ndiscabJβd'
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 5.4810665441672, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 73.0, "lr": 9e-06, "finish_rate": 0.781, "comp_len": 558.1, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 41.2, "frames": {"chat": 215}, "mem_gb": 10.0}
|
| 17 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 5.1878687764207525, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 56.0, "lr": 1.2e-05, "finish_rate": 0.825, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.6, "frames": {"chat": 217}, "mem_gb": 9.88}
|
| 18 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 4.475190736228227, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 48.5, "lr": 1.5e-05, "finish_rate": 0.8, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.4, "frames": {"chat": 205}, "mem_gb": 9.94}
|
| 19 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 3.5818144115626813, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 44.5, "lr": 1.8e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 229}, "mem_gb": 9.91}
|
| 20 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 3.2216264387488365, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 29.75, "lr": 2.1e-05, "finish_rate": 0.812, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 223}, "mem_gb": 9.98}
|
| 21 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 2.623535114067793, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 28.75, "lr": 2.4e-05, "finish_rate": 0.708, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 202}, "mem_gb": 10.02}
|
| 22 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 2.0890392102817694, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 21.375, "lr": 2.7000000000000002e-05, "finish_rate": 0.77, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 209}, "mem_gb": 9.99}
|
| 23 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.5289459430545569, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 12.3125, "lr": 3e-05, "finish_rate": 0.885, "comp_len": 528.6, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 227}, "mem_gb": 9.96}
|
| 24 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.3420937741284569, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 18.375, "lr": 3e-05, "finish_rate": 0.848, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 230}, "mem_gb": 10.04}
|
| 25 |
+
[eval step 10] sample: "To solve the problem, we need to determine the possible values of \\(a\\), \\(b\\), and \\(c\\) that satisfy the given equation \\(a + b + m + r = 18\\).\n\nLet's break down the problem into the steps:\n\n1. **Un"
|
| 26 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.0470961780856054, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 8.5, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 231}, "mem_gb": 9.89}
|
| 27 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.8973094273423156, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 5.5625, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 245}, "mem_gb": 9.96}
|
| 28 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7729372168297569, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 3.1875, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.7, "frames": {"chat": 210}, "mem_gb": 9.97}
|
| 29 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7965288292273879, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 3.171875, "lr": 3e-05, "finish_rate": 0.758, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 211}, "mem_gb": 9.97}
|
| 30 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5997185650259257, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 2.03125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 221}, "mem_gb": 10.02}
|
| 31 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5663720039173961, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 1.8359375, "lr": 3e-05, "finish_rate": 0.912, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 250}, "mem_gb": 9.84}
|
| 32 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5484086360000074, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 1.703125, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 229}, "mem_gb": 10.01}
|
| 33 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4377009954464932, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.888, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 250}, "mem_gb": 9.99}
|
| 34 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.47375443490048247, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 1.28125, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 231}, "mem_gb": 9.86}
|
| 35 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4232544344509641, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 1.09375, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 224}, "mem_gb": 9.9}
|
| 36 |
+
[eval step 20] sample: "To solve the problem, we need to determine the values of \\(a\\), \\(b\\), and \\(m\\) such that the given equations are satisfied.\n\nLet's break down the problem step-by-step:\n\n1. **Understand the Equations"
|
| 37 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.38542554356232284, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.98828125, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 212}, "mem_gb": 9.95}
|
| 38 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.34992314758387705, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.91796875, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 238}, "mem_gb": 9.9}
|
| 39 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.37193605013415215, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 1.0078125, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 257}, "mem_gb": 9.78}
|
| 40 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.32420069123518963, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.796875, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 227}, "mem_gb": 9.97}
|
| 41 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.36334449760143955, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.83203125, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 42 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3422913500872751, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.89453125, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 43 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3060648350310822, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.71484375, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 44 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4031517359685153, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 26.5, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 609.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 197}, "mem_gb": 10.08}
|
| 45 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3455044850113491, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.80859375, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 239}, "mem_gb": 9.83}
|
| 46 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3500898130904883, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.8828125, "lr": 3e-05, "finish_rate": 0.83, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 224}, "mem_gb": 9.88}
|
| 47 |
+
[eval step 30] sample: 'To solve the problem, we need to determine the values of \\(a\\), \\(b\\), and \\(p\\) that satisfy the given equations:\n\n1. \\(a + b = k\\)\n2. \\(k + m = p\\)\n3. \\(p + a = r\\)\n4.'
|
| 48 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2807131997364263, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.671875, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 217}, "mem_gb": 9.99}
|
| 49 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.27056356236139933, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.703125, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 241}, "mem_gb": 9.99}
|
| 50 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28164530578665437, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.68359375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 218}, "mem_gb": 9.97}
|
| 51 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2762345147125423, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.64453125, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 206}, "mem_gb": 9.98}
|
| 52 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2955992032624781, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.71484375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 232}, "mem_gb": 10.02}
|
| 53 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29579542716468377, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.69140625, "lr": 3e-05, "finish_rate": 0.771, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.4, "frames": {"chat": 218}, "mem_gb": 10.04}
|
| 54 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28874229360707104, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.63671875, "lr": 3e-05, "finish_rate": 0.779, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 213}, "mem_gb": 10.0}
|
| 55 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2895699510930727, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.70703125, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 483.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 248}, "mem_gb": 9.97}
|
| 56 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2556835312332958, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.640625, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 218}, "mem_gb": 10.03}
|
| 57 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2271388866957277, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.851, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 221}, "mem_gb": 9.98}
|
| 58 |
+
[eval step 40] sample: 'To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(m\\), and \\(p\\) that satisfy the given equations:\n\n1. \\(a + b = k\\)\n2. \\(k + m = p\\)\n3. \\(p + a = r'
|
| 59 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2487573687321196, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.59765625, "lr": 3e-05, "finish_rate": 0.894, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 236}, "mem_gb": 9.92}
|
| 60 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2528259696563085, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.62109375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 246}, "mem_gb": 9.84}
|
| 61 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.24962496552144486, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.61328125, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.0, "frames": {"chat": 234}, "mem_gb": 10.09}
|
| 62 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20244326410479843, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.748, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 202}, "mem_gb": 9.97}
|
| 63 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2265090825246026, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 217}, "mem_gb": 9.99}
|
| 64 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.23977290795333683, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.62109375, "lr": 3e-05, "finish_rate": 0.866, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 224}, "mem_gb": 9.99}
|
| 65 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.26650513206385074, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.59765625, "lr": 3e-05, "finish_rate": 0.753, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 215}, "mem_gb": 10.0}
|
| 66 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.216942871193029, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 1.2265625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 463.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 259}, "mem_gb": 9.92}
|
| 67 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.24050438385202239, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.59765625, "lr": 3e-05, "finish_rate": 0.829, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 210}, "mem_gb": 9.99}
|
| 68 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2729782617907971, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.59375, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 213}, "mem_gb": 10.04}
|
| 69 |
+
[eval step 50] sample: 'To solve the given system of equations, we need to determine the values of \\(a\\), \\(b\\), \\(m\\), and \\(p\\) such that:\n\n\\[\na + b = k\n\\]\n\\[\nk + m = p\n\\]\n\\[\np + a = r\n'
|
| 70 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep25_s1224/step0050
|
| 71 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.23127874396915238, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.0, "frames": {"chat": 222}, "mem_gb": 9.95}
|
| 72 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2022097536571324, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 235}, "mem_gb": 10.0}
|
| 73 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.22392032471460602, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 208}, "mem_gb": 9.96}
|
| 74 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21173793127896884, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.5234375, "lr": 3e-05, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.9, "frames": {"chat": 191}, "mem_gb": 10.0}
|
| 75 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1773922070093453, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.484375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 219}, "mem_gb": 9.99}
|
| 76 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1603636117445305, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.447265625, "lr": 3e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 207}, "mem_gb": 10.0}
|
| 77 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.24646623886699479, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.671875, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 208}, "mem_gb": 9.95}
|
| 78 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16445816849029313, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 219}, "mem_gb": 9.99}
|
| 79 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18552401562519372, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 246}, "mem_gb": 9.87}
|
| 80 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23117222544706117, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 206}, "mem_gb": 10.02}
|
| 81 |
+
[eval step 60] sample: 'To solve the given system of equations, we need to determine the values of \\(a\\), \\(b\\), \\(m\\), and \\(p\\) such that:\n\n\\[\na + b = k\n\\]\n\\[\nk + m = p\n\\]\n\\[\np + a = r\n'
|
| 82 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18189995611303797, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.466796875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 233}, "mem_gb": 10.0}
|
| 83 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18372100273153436, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.8671875, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 229}, "mem_gb": 9.86}
|
| 84 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1487495786678667, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 236}, "mem_gb": 9.9}
|
| 85 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18451065494281552, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.44921875, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 239}, "mem_gb": 9.78}
|
| 86 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16504728367341062, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 241}, "mem_gb": 9.9}
|
| 87 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18468811580395947, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 226}, "mem_gb": 9.87}
|
| 88 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15604329239850243, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 234}, "mem_gb": 10.0}
|
| 89 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16107819736519208, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.435546875, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 257}, "mem_gb": 9.99}
|
| 90 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23071295691871394, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.52734375, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.7, "frames": {"chat": 208}, "mem_gb": 10.04}
|
| 91 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19346359119676054, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 211}, "mem_gb": 10.02}
|
| 92 |
+
[eval step 70] sample: 'To solve the given system of equations, we need to determine the values of \\(a\\), \\(b\\), \\(m\\), and \\(p\\) such that:\n\n\\[\na + b = k\n\\]\n\\[\nk + m = p\n\\]\n\\[\np + a = r\n'
|
| 93 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20979610983723154, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 227}, "mem_gb": 10.0}
|
| 94 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18511814717464148, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 211}, "mem_gb": 9.98}
|
| 95 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15541142247617246, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.455078125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 238}, "mem_gb": 9.99}
|
| 96 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16142152046722671, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.9, "frames": {"chat": 237}, "mem_gb": 10.03}
|
| 97 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18345416961008063, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 208}, "mem_gb": 10.03}
|
| 98 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16612481657912334, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 221}, "mem_gb": 10.12}
|
| 99 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18476293021012097, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.52734375, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 232}, "mem_gb": 9.96}
|
| 100 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1682081102202336, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 208}, "mem_gb": 9.99}
|
| 101 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16280597681732228, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.4453125, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 227}, "mem_gb": 9.91}
|
| 102 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18280604339229564, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 221}, "mem_gb": 9.94}
|
| 103 |
+
[eval step 80] sample: 'To solve the given system of equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe can follow these steps:\n\n'
|
| 104 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14743654915646962, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.388671875, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 232}, "mem_gb": 10.0}
|
| 105 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15533407865427434, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 106 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17884458175196002, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 195}, "mem_gb": 10.09}
|
| 107 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17335635773831357, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.435546875, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 216}, "mem_gb": 10.0}
|
| 108 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15658746059002976, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.421875, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 208}, "mem_gb": 9.88}
|
| 109 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15384809637721628, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.5, "frames": {"chat": 235}, "mem_gb": 9.88}
|
| 110 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17239943368534247, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.46875, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 225}, "mem_gb": 9.99}
|
| 111 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20018711051177235, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 213}, "mem_gb": 10.08}
|
| 112 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1512549991114065, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.0, "frames": {"chat": 257}, "mem_gb": 9.75}
|
| 113 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19405477244090288, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.470703125, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 212}, "mem_gb": 10.02}
|
| 114 |
+
[eval step 90] sample: 'To solve the given system of equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe can follow these steps:\n\n'
|
| 115 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16499492380482456, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.42578125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 221}, "mem_gb": 10.0}
|
| 116 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15074481266401707, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.408203125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 242}, "mem_gb": 9.99}
|
| 117 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12890028902329503, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 220}, "mem_gb": 9.96}
|
| 118 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1444535345125012, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.40234375, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 240}, "mem_gb": 9.85}
|
| 119 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15116780090536922, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 206}, "mem_gb": 9.98}
|
| 120 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16743213449095687, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.44140625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 226}, "mem_gb": 10.0}
|
| 121 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21188729687035085, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.5703125, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 244}, "mem_gb": 9.78}
|
| 122 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20142290921670694, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 224}, "mem_gb": 10.0}
|
| 123 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1621253117416054, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.470703125, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 271}, "mem_gb": 9.72}
|
| 124 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15016368210408837, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.412109375, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 236}, "mem_gb": 10.01}
|
| 125 |
+
[eval step 100] sample: 'To solve this problem, we need to translate the given equations into a system of linear equations and solve for the unknowns \\(a\\), \\(b\\), \\(m\\), and \\(p\\).\n\nGiven:\n\\[\n\\begin{align*}\na + b &= k \\\\\nk +'
|
| 126 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep25_s1224/step0100
|
| 127 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1557210696899332, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 232}, "mem_gb": 9.88}
|
| 128 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1422160825269297, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.421875, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.6, "frames": {"chat": 210}, "mem_gb": 9.93}
|
| 129 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1296106172949386, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.2, "t_rollout_s": 0.0, "t_step_s": 41.4, "frames": {"chat": 217}, "mem_gb": 9.9}
|
| 130 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16544860142044102, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 224}, "mem_gb": 10.02}
|
| 131 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20553272317995627, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 203}, "mem_gb": 9.87}
|
| 132 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15951158448743324, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.46875, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 239}, "mem_gb": 9.97}
|
| 133 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11594521766655767, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 254}, "mem_gb": 9.88}
|
| 134 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10944677127550045, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 241}, "mem_gb": 9.97}
|
| 135 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17688703423095867, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.435546875, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 213}, "mem_gb": 10.0}
|
| 136 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13732703933753074, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 221}, "mem_gb": 10.05}
|
| 137 |
+
[eval step 110] sample: 'To solve the given system of equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe can follow these steps:\n\n'
|
| 138 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.153877969346568, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.4, "frames": {"chat": 196}, "mem_gb": 10.01}
|
| 139 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1227314798528639, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 270}, "mem_gb": 9.81}
|
| 140 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1115176025235715, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 216}, "mem_gb": 9.99}
|
| 141 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1313754518270803, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.4, "frames": {"chat": 200}, "mem_gb": 9.96}
|
| 142 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1082449041414385, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 206}, "mem_gb": 9.91}
|
| 143 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10967711655922854, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.357421875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 234}, "mem_gb": 9.94}
|
| 144 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1271924709101518, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 215}, "mem_gb": 9.95}
|
| 145 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11125657115193704, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.3984375, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 255}, "mem_gb": 9.94}
|
| 146 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12195842446442694, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 250}, "mem_gb": 9.82}
|
| 147 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1218588234690018, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 242}, "mem_gb": 9.99}
|
| 148 |
+
[eval step 120] sample: 'To solve the system of equations given:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe can follow these steps:\n\n'
|
| 149 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13956055214075994, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 150 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1709896477163459, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 208}, "mem_gb": 10.03}
|
| 151 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11171058443682268, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 208}, "mem_gb": 9.97}
|
| 152 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13555439033694566, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 209}, "mem_gb": 10.12}
|
| 153 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10593995610407243, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 235}, "mem_gb": 9.95}
|
| 154 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11018521934015056, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 204}, "mem_gb": 9.94}
|
| 155 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1523059667794034, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 208}, "mem_gb": 10.0}
|
| 156 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11395702697544668, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 240}, "mem_gb": 10.0}
|
| 157 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10960895141505947, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.345703125, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 246}, "mem_gb": 9.99}
|
| 158 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12840980574265123, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 243}, "mem_gb": 9.81}
|
| 159 |
+
[eval step 130] sample: 'To solve the given system of equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe need to determine the values of \\('
|
| 160 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1445476036771511, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.4, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 161 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12889366007174055, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 162 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14989680531652022, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.384765625, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 211}, "mem_gb": 10.01}
|
| 163 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12166979752990106, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 232}, "mem_gb": 9.97}
|
| 164 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1422418523649064, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.408203125, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 214}, "mem_gb": 10.0}
|
| 165 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1125547682431837, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 226}, "mem_gb": 9.89}
|
| 166 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11254967547400545, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 167 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10425090258192891, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 218}, "mem_gb": 9.83}
|
| 168 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10396384794348851, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 233}, "mem_gb": 9.98}
|
| 169 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1503090637408197, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 215}, "mem_gb": 10.0}
|
| 170 |
+
[eval step 140] sample: 'To solve the given system of equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe can follow these steps:\n\n'
|
| 171 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12397584599244098, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 172 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1101376080014122, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 209}, "mem_gb": 9.94}
|
| 173 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10867827801605065, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 262}, "mem_gb": 9.87}
|
| 174 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11113408046951517, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 249}, "mem_gb": 9.96}
|
| 175 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1363239465509231, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 227}, "mem_gb": 9.99}
|
| 176 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.102995293696442, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 221}, "mem_gb": 9.99}
|
| 177 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10967131499343862, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.435546875, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 234}, "mem_gb": 10.01}
|
| 178 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10105065601477399, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 213}, "mem_gb": 9.95}
|
| 179 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10403523254335548, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 213}, "mem_gb": 9.89}
|
| 180 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10564743132308746, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 234}, "mem_gb": 9.92}
|
| 181 |
+
[eval step 150] sample: "To solve the given system of equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe'll follow these steps:\n\n"
|
| 182 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep25_s1224/step0150
|
| 183 |
+
wandb: updating run metadata
|
| 184 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 185 |
+
wandb: uploading summary, console lines 169-170
|
| 186 |
+
wandb:
|
| 187 |
+
wandb: Run history:
|
| 188 |
+
wandb: comp_len ββββ
ββββββββ
ββββββ
β
ββ
ββββββ
β
βββ
ββ
βββββββ
|
| 189 |
+
wandb: cumulative_loss_tokens ββββββββββββββββββββββββββ
β
β
β
β
β
β
ββββββββ
|
| 190 |
+
wandb: epoch ββββββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββββββββββ
|
| 191 |
+
wandb: finish_rate β
ββ
βββ
βββββ
ββββ
ββ
ββ
ββββββ
βββ
ββββββββββββ
|
| 192 |
+
wandb: forward_topk_kl ββββββββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 195 |
+
wandb: mem_gb ββββ
ββ
β
β
β
β
ββββ
β
β
ββββββ
ββββββ
ββ
ββββ
ββββ
β
β
|
| 196 |
+
wandb: step ββββββββββββββββββββ
β
β
β
βββββββββββββββββ
|
| 197 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 198 |
+
wandb: +3 ...
|
| 199 |
+
wandb:
|
| 200 |
+
wandb: Run summary:
|
| 201 |
+
wandb: comp_len 512.8
|
| 202 |
+
wandb: cumulative_loss_tokens 18000000
|
| 203 |
+
wandb: epoch 2
|
| 204 |
+
wandb: finish_rate 0.906
|
| 205 |
+
wandb: forward_topk_kl 0.10565
|
| 206 |
+
wandb: grad_norm 0.34961
|
| 207 |
+
wandb: lr 3e-05
|
| 208 |
+
wandb: mem_gb 9.92
|
| 209 |
+
wandb: step 150
|
| 210 |
+
wandb: t_data_s 0
|
| 211 |
+
wandb: +4 ...
|
| 212 |
+
wandb:
|
| 213 |
+
wandb: π View run reap-math-keep25-s1224 at: https://wandb.ai/hbfreed/glean-grid/runs/q0iymgs1
|
| 214 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 215 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 216 |
+
wandb: Find logs at: outputs/healed/grid_math/reap_keep25_s1224/wandb/run-20260716_071102-q0iymgs1/logs
|
| 217 |
+
{
|
| 218 |
+
"correct": 134,
|
| 219 |
+
"accuracy": 0.10159211523881728,
|
| 220 |
+
"finished": 1084,
|
| 221 |
+
"finish_rate": 0.8218347232752085,
|
| 222 |
+
"mean_completion_tokens": 193.42304776345716
|
| 223 |
+
}
|
| 224 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep25_s1224_step100_chat.json
|
| 225 |
+
{
|
| 226 |
+
"correct": 156,
|
| 227 |
+
"accuracy": 0.11827141774071266,
|
| 228 |
+
"finished": 1076,
|
| 229 |
+
"finish_rate": 0.8157695223654283,
|
| 230 |
+
"mean_completion_tokens": 191.31084154662622
|
| 231 |
+
}
|
| 232 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep25_s1224_step150_chat.json
|
healed/grid_math/reap_keep25_s1226.console.log
ADDED
|
@@ -0,0 +1,231 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run ep759fa6
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/reap_keep25_s1226/wandb/run-20260716_063351-ep759fa6
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run reap-math-keep25-s1226
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/ep759fa6
|
| 12 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 2.09B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 5.3948620602846145, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 83.5, "lr": 6e-06, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.4, "frames": {"chat": 254}, "mem_gb": 9.81}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: '| 16:\nSnrd.\n/VA_#(t_i\n\nTo break n-day::a\n\nas\n\nisntABent\n\nThe a * a bente\n\n|A2 a\n| a.b,b\nfor [\n{U'
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 5.492758668692907, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 68.5, "lr": 9e-06, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 241}, "mem_gb": 9.97}
|
| 17 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 5.288580652968089, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 50.25, "lr": 1.2e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 213}, "mem_gb": 10.0}
|
| 18 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 4.361503725457191, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 47.75, "lr": 1.5e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.7, "frames": {"chat": 221}, "mem_gb": 10.05}
|
| 19 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 3.849147217363119, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 39.5, "lr": 1.8e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.9, "frames": {"chat": 196}, "mem_gb": 10.01}
|
| 20 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 3.022541018138329, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 28.0, "lr": 2.1e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 270}, "mem_gb": 9.81}
|
| 21 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 2.5903835578064123, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 36.5, "lr": 2.4e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 216}, "mem_gb": 9.99}
|
| 22 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.989209920862317, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 22.375, "lr": 2.7000000000000002e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.3, "frames": {"chat": 200}, "mem_gb": 9.96}
|
| 23 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.5486149408777554, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 13.375, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.5, "frames": {"chat": 206}, "mem_gb": 9.91}
|
| 24 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.1493749193057419, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 12.875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 234}, "mem_gb": 9.94}
|
| 25 |
+
[eval step 10] sample: 'To solve the problem, we need to determine the perimeter of the triangle. We are given the the perimeter of the triangle is 28. The perimeter of a triangle is the sum of the distances of the sides.\n\nL'
|
| 26 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.0132465209826826, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 8.375, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 215}, "mem_gb": 9.95}
|
| 27 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.79206941087991, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 4.59375, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 255}, "mem_gb": 9.94}
|
| 28 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7040475417902072, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 7.375, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 250}, "mem_gb": 9.82}
|
| 29 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6600783367355665, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 10.9375, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 242}, "mem_gb": 9.99}
|
| 30 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6760948089194795, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 8.0, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 31 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.735672302259008, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 4.78125, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 208}, "mem_gb": 10.03}
|
| 32 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5825303116974732, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 4.6875, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 208}, "mem_gb": 9.97}
|
| 33 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6073473408266902, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 2.203125, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 209}, "mem_gb": 10.12}
|
| 34 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.45263020926391084, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 1.359375, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 235}, "mem_gb": 9.95}
|
| 35 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4413063018143177, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 3.25, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.4, "frames": {"chat": 204}, "mem_gb": 9.94}
|
| 36 |
+
[eval step 20] sample: 'To solve this problem, we need to determine the perimeter of the resulting triangle given the conditions:\n\n1. The perimeter of the triangle is 28.\n2. The midpoints of the sides of the triangle are con'
|
| 37 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5014240723443528, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 1.453125, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 208}, "mem_gb": 10.0}
|
| 38 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.39647342192269863, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 1.0234375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 240}, "mem_gb": 10.0}
|
| 39 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3701785091145585, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 1.03125, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 246}, "mem_gb": 9.99}
|
| 40 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.38433438787100216, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 1.03125, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.7, "frames": {"chat": 243}, "mem_gb": 9.81}
|
| 41 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.39559759280495344, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 1.2109375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.4, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 42 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.35509972168381015, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 1.3203125, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.4, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 43 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.39220244832945367, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.91796875, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 211}, "mem_gb": 10.01}
|
| 44 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.32253637241298955, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.83203125, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 232}, "mem_gb": 9.97}
|
| 45 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3454374578539282, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.84375, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.4, "frames": {"chat": 214}, "mem_gb": 10.0}
|
| 46 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.30258261669923864, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.73828125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 226}, "mem_gb": 9.89}
|
| 47 |
+
[eval step 30] sample: 'To solve this problem, we need to determine the perimeter of the resulting triangle formed by connecting the midpoints of the sides of the given triangle.\n\nHere are the steps to solve the problem:\n\n1.'
|
| 48 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28547664239617687, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.66015625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 49 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2539228688845411, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.65234375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.4, "frames": {"chat": 218}, "mem_gb": 9.83}
|
| 50 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.26798752832549316, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.6640625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 233}, "mem_gb": 9.98}
|
| 51 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3186436773126324, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.69921875, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 215}, "mem_gb": 10.0}
|
| 52 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2920498409892122, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.6484375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.4, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 53 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2473691465223829, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.62109375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 209}, "mem_gb": 9.94}
|
| 54 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.25275046684630215, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.640625, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 262}, "mem_gb": 9.87}
|
| 55 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2435881425558279, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.57421875, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 249}, "mem_gb": 9.96}
|
| 56 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29404913728336496, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.609375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 227}, "mem_gb": 9.99}
|
| 57 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.22125561543889344, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.515625, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 221}, "mem_gb": 9.99}
|
| 58 |
+
[eval step 40] sample: 'To solve this problem, we need to understand the geometric properties and the relationships between the sides of the triangle formed by the midpoints of the original triangle.\n\nGiven:\n- The perimeter '
|
| 59 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2520220772454515, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.61328125, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 234}, "mem_gb": 10.01}
|
| 60 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20081105666371682, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.1, "frames": {"chat": 213}, "mem_gb": 9.95}
|
| 61 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2016371044045935, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.4, "frames": {"chat": 213}, "mem_gb": 9.89}
|
| 62 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.22321325843880574, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.54296875, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 234}, "mem_gb": 9.92}
|
| 63 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.21784419940554847, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 222}, "mem_gb": 9.99}
|
| 64 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2611530060334752, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.59375, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 227}, "mem_gb": 10.0}
|
| 65 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.22217773687231043, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.4, "frames": {"chat": 218}, "mem_gb": 10.04}
|
| 66 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2560874344268193, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.66796875, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 223}, "mem_gb": 10.01}
|
| 67 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2610147045496851, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.546875, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 206}, "mem_gb": 10.0}
|
| 68 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.22958950562837224, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 213}, "mem_gb": 9.92}
|
| 69 |
+
[eval step 50] sample: "To solve this problem, we need to understand the geometric properties and the given conditions. Here's a step-by-step approach:\n\n1. **Understand the Problem:**\n - The perimeter of the triangle is 28"
|
| 70 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep25_s1226/step0050
|
| 71 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2759530112889906, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.5859375, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 223}, "mem_gb": 9.86}
|
| 72 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.24462636410264918, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 227}, "mem_gb": 9.97}
|
| 73 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2153538477362444, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 253}, "mem_gb": 10.0}
|
| 74 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2091132113194093, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.6484375, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.8, "frames": {"chat": 216}, "mem_gb": 10.0}
|
| 75 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2178824202261865, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.7, "frames": {"chat": 205}, "mem_gb": 9.97}
|
| 76 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2252128013719494, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 207}, "mem_gb": 10.06}
|
| 77 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23820067919362337, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.5234375, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 215}, "mem_gb": 9.98}
|
| 78 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1530524639653663, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 79 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2108373320500677, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.54296875, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 221}, "mem_gb": 10.04}
|
| 80 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16624215446698168, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 254}, "mem_gb": 9.84}
|
| 81 |
+
[eval step 60] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and how the midpoints of its sides affect the perimeter.\n\n### Step-by-Step Solution:\n\n1. **Understand the Geometry'
|
| 82 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14189707856637737, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.466796875, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.0, "frames": {"chat": 210}, "mem_gb": 9.96}
|
| 83 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16133386867145696, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 226}, "mem_gb": 9.92}
|
| 84 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.184017719945622, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.439453125, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 212}, "mem_gb": 9.99}
|
| 85 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2009096172561869, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 211}, "mem_gb": 9.92}
|
| 86 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2035785714487235, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.0, "frames": {"chat": 196}, "mem_gb": 9.97}
|
| 87 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15334912759084254, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 212}, "mem_gb": 9.99}
|
| 88 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16262201901270698, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 244}, "mem_gb": 9.91}
|
| 89 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14996965130375078, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.412109375, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 222}, "mem_gb": 9.95}
|
| 90 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.182706143669722, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.466796875, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 218}, "mem_gb": 10.0}
|
| 91 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17859229601956905, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 252}, "mem_gb": 9.87}
|
| 92 |
+
[eval step 70] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and how the midpoints of its sides are connected by segments.\n\n### Step-by-Step Solution:\n\n1. **Understand the Geo'
|
| 93 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19512056777638695, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.498046875, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.7, "frames": {"chat": 202}, "mem_gb": 10.05}
|
| 94 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22794473583027722, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.578125, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 237}, "mem_gb": 10.0}
|
| 95 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1737093073921899, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 234}, "mem_gb": 9.98}
|
| 96 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.13047870258555438, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 215}, "mem_gb": 10.0}
|
| 97 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.13498239639320722, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.388671875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 234}, "mem_gb": 9.93}
|
| 98 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12114300584094599, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.7, "frames": {"chat": 216}, "mem_gb": 9.98}
|
| 99 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1555439574915295, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.3, "frames": {"chat": 210}, "mem_gb": 9.95}
|
| 100 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1471449433473715, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.5, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 101 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1342087133670226, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.373046875, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 102 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1672618577302744, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.49609375, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 225}, "mem_gb": 9.95}
|
| 103 |
+
[eval step 80] sample: 'To solve this problem, we need to understand the geometric properties and the relationships between the sides of the triangle formed by the midpoints of the original triangle.\n\n### Steps to Solve the '
|
| 104 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1618539959486574, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 253}, "mem_gb": 9.85}
|
| 105 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15880980721451343, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 247}, "mem_gb": 9.97}
|
| 106 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15618948619918277, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 238}, "mem_gb": 9.97}
|
| 107 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19067880896230538, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 235}, "mem_gb": 9.99}
|
| 108 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18032733394745737, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 215}, "mem_gb": 9.96}
|
| 109 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1783826366210977, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.5234375, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 268}, "mem_gb": 9.97}
|
| 110 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17707187270099917, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 111 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1674367210490629, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 252}, "mem_gb": 9.93}
|
| 112 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1522629966557026, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.7, "frames": {"chat": 223}, "mem_gb": 10.01}
|
| 113 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2198721208633855, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.6171875, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 226}, "mem_gb": 9.99}
|
| 114 |
+
[eval step 90] sample: 'To solve this problem, we need to understand the geometric properties and the relationships between the sides of the triangle formed by the midpoints of the original triangle.\n\n### Steps to Solve:\n\n1.'
|
| 115 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19896256684847177, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 208}, "mem_gb": 10.04}
|
| 116 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14341324961800128, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.40625, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 240}, "mem_gb": 9.93}
|
| 117 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16562857267570993, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.44140625, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 222}, "mem_gb": 9.92}
|
| 118 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15155938894315624, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.408203125, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 236}, "mem_gb": 9.99}
|
| 119 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12592543063924339, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.373046875, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 217}, "mem_gb": 9.96}
|
| 120 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.15650399845099697, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 252}, "mem_gb": 9.87}
|
| 121 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.13744228080461424, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 222}, "mem_gb": 9.99}
|
| 122 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14708479140317068, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.901, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 242}, "mem_gb": 9.87}
|
| 123 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19175601966977118, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.44140625, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 219}, "mem_gb": 9.93}
|
| 124 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14486887626430642, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 223}, "mem_gb": 9.94}
|
| 125 |
+
[eval step 100] sample: "To solve this problem, we need to understand the geometric properties and the given conditions. Here's how we can break it down:\n\n1. **Understand the Problem:**\n - We have a triangle with a perimete"
|
| 126 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep25_s1226/step0100
|
| 127 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17521546159796417, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.46484375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 232}, "mem_gb": 9.94}
|
| 128 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17657650477966916, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.832, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 220}, "mem_gb": 10.0}
|
| 129 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1642237206614266, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.40234375, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 210}, "mem_gb": 10.04}
|
| 130 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14797473591733723, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 226}, "mem_gb": 9.97}
|
| 131 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.13756006544300667, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.741, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 212}, "mem_gb": 9.99}
|
| 132 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.14728604319685448, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.44140625, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 236}, "mem_gb": 10.01}
|
| 133 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11492285058296596, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.928, "comp_len": 454.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 264}, "mem_gb": 9.88}
|
| 134 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16432044878825544, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.470703125, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 229}, "mem_gb": 9.98}
|
| 135 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11607246005069465, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 465.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 258}, "mem_gb": 9.85}
|
| 136 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16434998966678976, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.42578125, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 137 |
+
[eval step 110] sample: 'To solve this problem, we need to understand the geometric properties and the relationships between the sides of the triangle and the midpoints.\n\nGiven:\n- The perimeter of the triangle is 28.\n- The mi'
|
| 138 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12820218563723998, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.88, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 249}, "mem_gb": 9.92}
|
| 139 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10775780974778656, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.5, "frames": {"chat": 220}, "mem_gb": 9.99}
|
| 140 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13256505108779917, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.40625, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.9, "frames": {"chat": 223}, "mem_gb": 9.98}
|
| 141 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.107142218930802, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.373046875, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 221}, "mem_gb": 9.99}
|
| 142 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10337071840027347, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 230}, "mem_gb": 9.9}
|
| 143 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12823014822658152, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.373046875, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 214}, "mem_gb": 9.97}
|
| 144 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17329245272489885, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.462890625, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 214}, "mem_gb": 9.99}
|
| 145 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12728931551845743, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 210}, "mem_gb": 10.03}
|
| 146 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13437456069858744, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 214}, "mem_gb": 9.99}
|
| 147 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1287463510526344, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.791, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 215}, "mem_gb": 9.95}
|
| 148 |
+
[eval step 120] sample: "To solve this problem, we need to understand the geometric properties and relationships involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the triangle is"
|
| 149 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1430338466726554, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.419921875, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 208}, "mem_gb": 9.99}
|
| 150 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10728711698964859, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.789, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 218}, "mem_gb": 9.87}
|
| 151 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09843137963234136, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 233}, "mem_gb": 9.89}
|
| 152 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.09210390640692785, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 231}, "mem_gb": 9.93}
|
| 153 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12565180675198015, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.3828125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 235}, "mem_gb": 10.13}
|
| 154 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12365243875219797, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 216}, "mem_gb": 9.98}
|
| 155 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11053694897955284, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.369140625, "lr": 3e-05, "finish_rate": 0.831, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 237}, "mem_gb": 10.01}
|
| 156 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15008561470514784, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.4296875, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 203}, "mem_gb": 10.01}
|
| 157 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11617295994066323, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.873, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 236}, "mem_gb": 10.03}
|
| 158 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11612723016667491, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 214}, "mem_gb": 10.0}
|
| 159 |
+
[eval step 130] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and how its midpoints are connected.\n\n### Steps to Solve the Problem:\n\n1. **Understand the Geometry:**\n - Let th'
|
| 160 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11800649179657921, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 209}, "mem_gb": 9.99}
|
| 161 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11760378193684543, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 468.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 256}, "mem_gb": 10.0}
|
| 162 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1103896147995256, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.3671875, "lr": 3e-05, "finish_rate": 0.878, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 230}, "mem_gb": 9.86}
|
| 163 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11834490465267251, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 230}, "mem_gb": 10.05}
|
| 164 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13661176628569763, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 227}, "mem_gb": 9.95}
|
| 165 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1438256380248815, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 166 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13295528639039644, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.699, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 206}, "mem_gb": 10.03}
|
| 167 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13133824970157196, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.82, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 228}, "mem_gb": 9.9}
|
| 168 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12861836654854317, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 224}, "mem_gb": 9.99}
|
| 169 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1111085697865424, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.357421875, "lr": 3e-05, "finish_rate": 0.66, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.9, "frames": {"chat": 200}, "mem_gb": 10.03}
|
| 170 |
+
[eval step 140] sample: "To solve this problem, we need to understand the geometric properties and relationships involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - We have a triangle with a perime"
|
| 171 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12692416681302712, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.714, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.8, "frames": {"chat": 196}, "mem_gb": 10.01}
|
| 172 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10248101542762791, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 223}, "mem_gb": 9.99}
|
| 173 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10177099369396456, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.869, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.6, "frames": {"chat": 213}, "mem_gb": 9.88}
|
| 174 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10673611074338357, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.369140625, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 232}, "mem_gb": 9.93}
|
| 175 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10685085613004243, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 223}, "mem_gb": 9.92}
|
| 176 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11504278169833124, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.373046875, "lr": 3e-05, "finish_rate": 0.85, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 233}, "mem_gb": 10.02}
|
| 177 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.11391995535188665, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.816, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 217}, "mem_gb": 10.01}
|
| 178 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1639873969303444, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.752, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 202}, "mem_gb": 10.07}
|
| 179 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.10495261400962869, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.365234375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 253}, "mem_gb": 9.93}
|
| 180 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0964433067208156, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 231}, "mem_gb": 9.94}
|
| 181 |
+
[eval step 150] sample: "To solve this problem, we need to understand the geometric properties and relationships involved. Here's a step-by-step approach:\n\n1. **Understand the Problem:**\n - We have a triangle with a perimet"
|
| 182 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep25_s1226/step0150
|
| 183 |
+
wandb: updating run metadata
|
| 184 |
+
wandb: uploading summary, console lines 170-170
|
| 185 |
+
wandb:
|
| 186 |
+
wandb: Run history:
|
| 187 |
+
wandb: comp_len ββββββββββ
βββ
βββ
ββββββββββ
β
ββ
βββββ
β
βββββ
|
| 188 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββββββ
β
β
β
β
β
ββββββββββββ
|
| 189 |
+
wandb: epoch ββββββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
βββββββββββ
|
| 190 |
+
wandb: finish_rate βββββ
ββββββββββ
βββ
βββββ
βββ
βββββββ
βββββββ
|
| 191 |
+
wandb: forward_topk_kl ββββββββββββββββββββββββββββββββββββββββ
|
| 192 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: mem_gb βββ
βββββββββ
βββββββ
ββββββββββββ
ββ
βββββββ
|
| 195 |
+
wandb: step βββββββββββββββββββββ
β
β
β
β
βββββββββββββββ
|
| 196 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 197 |
+
wandb: +3 ...
|
| 198 |
+
wandb:
|
| 199 |
+
wandb: Run summary:
|
| 200 |
+
wandb: comp_len 519.5
|
| 201 |
+
wandb: cumulative_loss_tokens 18000000
|
| 202 |
+
wandb: epoch 2
|
| 203 |
+
wandb: finish_rate 0.879
|
| 204 |
+
wandb: forward_topk_kl 0.09644
|
| 205 |
+
wandb: grad_norm 0.32422
|
| 206 |
+
wandb: lr 3e-05
|
| 207 |
+
wandb: mem_gb 9.94
|
| 208 |
+
wandb: step 150
|
| 209 |
+
wandb: t_data_s 0
|
| 210 |
+
wandb: +4 ...
|
| 211 |
+
wandb:
|
| 212 |
+
wandb: π View run reap-math-keep25-s1226 at: https://wandb.ai/hbfreed/glean-grid/runs/ep759fa6
|
| 213 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 214 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 215 |
+
wandb: Find logs at: outputs/healed/grid_math/reap_keep25_s1226/wandb/run-20260716_063351-ep759fa6/logs
|
| 216 |
+
{
|
| 217 |
+
"correct": 158,
|
| 218 |
+
"accuracy": 0.1197877179681577,
|
| 219 |
+
"finished": 923,
|
| 220 |
+
"finish_rate": 0.6997725549658832,
|
| 221 |
+
"mean_completion_tokens": 233.13343442001516
|
| 222 |
+
}
|
| 223 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep25_s1226_step100_chat.json
|
| 224 |
+
{
|
| 225 |
+
"correct": 163,
|
| 226 |
+
"accuracy": 0.12357846853677028,
|
| 227 |
+
"finished": 1005,
|
| 228 |
+
"finish_rate": 0.7619408642911296,
|
| 229 |
+
"mean_completion_tokens": 214.75056861258528
|
| 230 |
+
}
|
| 231 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep25_s1226_step150_chat.json
|
healed/grid_math/reap_keep50_s1224.console.log
ADDED
|
@@ -0,0 +1,233 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run 1c8m5fh6
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/reap_keep50_s1224/wandb/run-20260716_015810-1c8m5fh6
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run reap-math-keep50-s1224
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/1c8m5fh6
|
| 12 |
+
|
| 13 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 3.70B | teacher overlap=False
|
| 14 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.49594468040814005, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 7.9375, "lr": 6e-06, "finish_rate": 0.907, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.6, "frames": {"chat": 236}, "mem_gb": 15.77}
|
| 15 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 16 |
+
[eval step 1] sample: 'To solve this problem, we need to translate the given conditions into equations and then solve for the digits \\(a\\), \\(b\\), \\(p\\), and \\(r\\).\n\nGiven:\n1. \\(a + b = k\\)\n2. \\(k + m = p\\)\n3.'
|
| 17 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5564288554263611, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 8.125, "lr": 9e-06, "finish_rate": 0.781, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 18 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5516090934860209, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 7.75, "lr": 1.2e-05, "finish_rate": 0.825, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 217}, "mem_gb": 15.93}
|
| 19 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4439431126748522, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 5.65625, "lr": 1.5e-05, "finish_rate": 0.8, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 205}, "mem_gb": 15.99}
|
| 20 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3083168001982073, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 3.984375, "lr": 1.8e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 229}, "mem_gb": 15.96}
|
| 21 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3849331200340142, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 3.5, "lr": 2.1e-05, "finish_rate": 0.812, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 223}, "mem_gb": 16.03}
|
| 22 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28037872347844145, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 2.5, "lr": 2.4e-05, "finish_rate": 0.708, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 202}, "mem_gb": 16.07}
|
| 23 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2623569235720982, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 1.9609375, "lr": 2.7000000000000002e-05, "finish_rate": 0.77, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 209}, "mem_gb": 16.04}
|
| 24 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18439374280131113, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 1.2734375, "lr": 3e-05, "finish_rate": 0.885, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.7, "frames": {"chat": 227}, "mem_gb": 16.01}
|
| 25 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1886376622683679, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 1.6484375, "lr": 3e-05, "finish_rate": 0.848, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.4, "frames": {"chat": 230}, "mem_gb": 16.09}
|
| 26 |
+
[eval step 10] sample: "To solve this problem, we need to determine the values of \\(a\\), \\(b\\), \\(p\\), and \\(r\\) that satisfy the given equations. Let's break down the problem step-by-step:\n\n1. **Understand the Equations:**\n"
|
| 27 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16658974986529598, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.8515625, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 231}, "mem_gb": 15.94}
|
| 28 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15182305362684032, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.66015625, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.1, "frames": {"chat": 245}, "mem_gb": 16.01}
|
| 29 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14790143515200666, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.62890625, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 210}, "mem_gb": 16.02}
|
| 30 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17545699729180583, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.7109375, "lr": 3e-05, "finish_rate": 0.758, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.5, "frames": {"chat": 211}, "mem_gb": 16.02}
|
| 31 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13955167624531314, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 221}, "mem_gb": 16.07}
|
| 32 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1277982988677919, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.912, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.8, "frames": {"chat": 250}, "mem_gb": 15.89}
|
| 33 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13481735738515854, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.7, "frames": {"chat": 229}, "mem_gb": 16.06}
|
| 34 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10775362585720917, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.462890625, "lr": 3e-05, "finish_rate": 0.888, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.3, "frames": {"chat": 250}, "mem_gb": 16.04}
|
| 35 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12802754533017674, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.494140625, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.4, "frames": {"chat": 231}, "mem_gb": 15.91}
|
| 36 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11459196638700862, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 224}, "mem_gb": 15.95}
|
| 37 |
+
[eval step 20] sample: 'To solve this problem, we need to find the digits \\(a, b, k, p, r\\) such that each letter represents a non-zero digit (0-9) and satisfies the given equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &='
|
| 38 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11341522086306165, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 212}, "mem_gb": 16.0}
|
| 39 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10356971820856754, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.494140625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.5, "frames": {"chat": 238}, "mem_gb": 15.95}
|
| 40 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10695767878911769, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.462890625, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.0, "frames": {"chat": 257}, "mem_gb": 15.83}
|
| 41 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09809987622502571, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 227}, "mem_gb": 16.02}
|
| 42 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11162722278796136, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 228}, "mem_gb": 16.05}
|
| 43 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1120611954228642, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.6, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 44 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09978716309297209, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.466796875, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.6, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 45 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13684565862777331, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 609.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.7, "frames": {"chat": 197}, "mem_gb": 16.13}
|
| 46 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11875445463884, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.9, "frames": {"chat": 239}, "mem_gb": 15.88}
|
| 47 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11370838879613826, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.83, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.0, "frames": {"chat": 224}, "mem_gb": 15.93}
|
| 48 |
+
[eval step 30] sample: 'To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(p\\) given the equations:\n\n1. \\(a + b = k\\)\n2. \\(k + m = p\\)\n3. \\(p + a'
|
| 49 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09682779053676252, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 217}, "mem_gb": 16.04}
|
| 50 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09320009283507243, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.38671875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.7, "frames": {"chat": 241}, "mem_gb": 16.04}
|
| 51 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09364061215973149, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.38671875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.0, "frames": {"chat": 218}, "mem_gb": 16.02}
|
| 52 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09702318099336699, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.3671875, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 206}, "mem_gb": 16.03}
|
| 53 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10040006328290328, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.40234375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.6, "frames": {"chat": 232}, "mem_gb": 16.07}
|
| 54 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10575694893598557, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.38671875, "lr": 3e-05, "finish_rate": 0.771, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.2, "frames": {"chat": 218}, "mem_gb": 16.09}
|
| 55 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10570678343428298, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.779, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 213}, "mem_gb": 16.05}
|
| 56 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10308800597370912, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 483.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.3, "frames": {"chat": 248}, "mem_gb": 16.02}
|
| 57 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0880163347641627, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.40625, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 218}, "mem_gb": 16.08}
|
| 58 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08286121298692499, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.851, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 221}, "mem_gb": 16.03}
|
| 59 |
+
[eval step 40] sample: 'To solve the problem, we need to find the digits \\(a, b, k, m, r\\) such that each letter represents a non-zero digit and satisfies the given equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np'
|
| 60 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08477509343984227, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.894, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.3, "frames": {"chat": 236}, "mem_gb": 15.97}
|
| 61 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08537144795643786, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 57.4, "frames": {"chat": 246}, "mem_gb": 15.89}
|
| 62 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08826947533857699, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.5, "frames": {"chat": 234}, "mem_gb": 16.14}
|
| 63 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07720068273398405, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.748, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 202}, "mem_gb": 16.02}
|
| 64 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08485504340163122, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 217}, "mem_gb": 16.04}
|
| 65 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08374200692667315, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.866, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.3, "frames": {"chat": 224}, "mem_gb": 16.04}
|
| 66 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09966607855204493, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.753, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 67 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07644946041551108, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 463.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.6, "frames": {"chat": 259}, "mem_gb": 15.97}
|
| 68 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08766008586073294, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.829, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 210}, "mem_gb": 16.04}
|
| 69 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10265232941933597, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.40625, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 213}, "mem_gb": 16.09}
|
| 70 |
+
[eval step 50] sample: "To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(m\\), \\(p\\), and \\(r\\) such that each letter represents a non-zero digit and satisfies the given equations.\n\nLet's break down th"
|
| 71 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep50_s1224/step0050
|
| 72 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08243085641801978, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 222}, "mem_gb": 16.0}
|
| 73 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0740578943549345, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.3, "frames": {"chat": 235}, "mem_gb": 16.05}
|
| 74 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08452489547633256, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 208}, "mem_gb": 16.01}
|
| 75 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07001415301486849, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.306640625, "lr": 3e-05, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 191}, "mem_gb": 16.05}
|
| 76 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0545745571903574, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.263671875, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 219}, "mem_gb": 16.04}
|
| 77 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05674678605599329, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 207}, "mem_gb": 16.05}
|
| 78 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07849393064022685, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 208}, "mem_gb": 16.0}
|
| 79 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05212787776181164, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.2431640625, "lr": 3e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 219}, "mem_gb": 16.04}
|
| 80 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05130243318529489, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.8, "frames": {"chat": 246}, "mem_gb": 15.92}
|
| 81 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07500699858136164, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 206}, "mem_gb": 16.07}
|
| 82 |
+
[eval step 60] sample: 'To solve the problem, we need to find the digits \\(a, b, k, m, r\\) such that each digit is a non-zero digit (i.e., between 1 and 9) and satisfies the given equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\'
|
| 83 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05669750292877046, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.271484375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.7, "frames": {"chat": 233}, "mem_gb": 16.05}
|
| 84 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05847539465183703, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 229}, "mem_gb": 15.91}
|
| 85 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04844146788320504, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.2431640625, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.4, "frames": {"chat": 236}, "mem_gb": 15.95}
|
| 86 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05839694087315972, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.3, "frames": {"chat": 239}, "mem_gb": 15.83}
|
| 87 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.049815410684685535, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.240234375, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.0, "frames": {"chat": 241}, "mem_gb": 15.95}
|
| 88 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05642013670879727, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.255859375, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 226}, "mem_gb": 15.92}
|
| 89 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04739125943775289, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.2392578125, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.9, "frames": {"chat": 234}, "mem_gb": 16.05}
|
| 90 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04787641853652894, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.23828125, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.4, "frames": {"chat": 257}, "mem_gb": 16.04}
|
| 91 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07120997395546486, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.291015625, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 208}, "mem_gb": 16.09}
|
| 92 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.063085703232248, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.3, "frames": {"chat": 211}, "mem_gb": 16.07}
|
| 93 |
+
[eval step 70] sample: 'To solve the problem, we need to find the digits \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(p\\) such that each letter represents a non-zero digit and satisfies the given equations:\n\n\\[\n\\begin{align*}\na + b &= '
|
| 94 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06971925978801834, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.359375, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.7, "frames": {"chat": 227}, "mem_gb": 16.05}
|
| 95 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.061769093114696444, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 211}, "mem_gb": 16.03}
|
| 96 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05006525234617293, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.6, "frames": {"chat": 238}, "mem_gb": 16.04}
|
| 97 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05073510147240013, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.2392578125, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.5, "frames": {"chat": 237}, "mem_gb": 16.08}
|
| 98 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0631699871141774, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.9, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 99 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.051907375587942076, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.2451171875, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 221}, "mem_gb": 16.17}
|
| 100 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.056417306477592015, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.76953125, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.6, "frames": {"chat": 232}, "mem_gb": 16.01}
|
| 101 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05626846161660117, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.263671875, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 208}, "mem_gb": 16.04}
|
| 102 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.049228766723753266, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 227}, "mem_gb": 15.96}
|
| 103 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05534716739145418, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 221}, "mem_gb": 15.99}
|
| 104 |
+
[eval step 80] sample: "To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(m\\), \\(p\\), and \\(r\\) such that each letter represents a non-zero digit and the given equations hold true.\n\nLet's break down th"
|
| 105 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.047819643541720386, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.2431640625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 232}, "mem_gb": 16.05}
|
| 106 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.056753374835678064, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.271484375, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 107 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05573002262612184, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.2490234375, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 195}, "mem_gb": 16.14}
|
| 108 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05441774775936889, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 216}, "mem_gb": 16.05}
|
| 109 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06264704996475484, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 208}, "mem_gb": 15.93}
|
| 110 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05151226184510936, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.263671875, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 235}, "mem_gb": 15.93}
|
| 111 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05486625511728538, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.291015625, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.7, "frames": {"chat": 225}, "mem_gb": 16.04}
|
| 112 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06505359720261768, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.9, "frames": {"chat": 213}, "mem_gb": 16.13}
|
| 113 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.044957342780055476, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.0, "frames": {"chat": 257}, "mem_gb": 15.8}
|
| 114 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06199448381081844, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.6, "frames": {"chat": 212}, "mem_gb": 16.07}
|
| 115 |
+
[eval step 90] sample: 'To solve the problem, we need to find the digits \\(a, b, k, m, r\\) such that each digit is a non-zero digit (i.e., 1-9) and satisfies the given equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\n'
|
| 116 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05085727345328778, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.279296875, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 221}, "mem_gb": 16.05}
|
| 117 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.046645166625843074, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.2333984375, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 242}, "mem_gb": 16.04}
|
| 118 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.046637191681253416, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 220}, "mem_gb": 16.01}
|
| 119 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.044484876428404825, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.9, "frames": {"chat": 240}, "mem_gb": 15.9}
|
| 120 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05077935193019609, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.25, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 206}, "mem_gb": 16.03}
|
| 121 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05764281274767903, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 226}, "mem_gb": 16.05}
|
| 122 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07217985147928509, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.6, "frames": {"chat": 244}, "mem_gb": 15.83}
|
| 123 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.062065851740368334, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.302734375, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.9, "frames": {"chat": 224}, "mem_gb": 16.05}
|
| 124 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04666607260017966, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.4, "frames": {"chat": 271}, "mem_gb": 15.77}
|
| 125 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05521788966048819, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.6, "frames": {"chat": 236}, "mem_gb": 16.06}
|
| 126 |
+
[eval step 100] sample: 'To solve this problem, we need to find the digits \\(a, b, k, m, r\\) such that each letter represents a non-zero digit and satisfy the given equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np '
|
| 127 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep50_s1224/step0100
|
| 128 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04969318243367597, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.26171875, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 232}, "mem_gb": 15.93}
|
| 129 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05082408647788689, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 210}, "mem_gb": 15.98}
|
| 130 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05025460629562537, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 217}, "mem_gb": 15.95}
|
| 131 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05321642030389048, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.0, "frames": {"chat": 224}, "mem_gb": 16.07}
|
| 132 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06337893520779908, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.28515625, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 203}, "mem_gb": 15.92}
|
| 133 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.050805029303006205, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.5, "frames": {"chat": 239}, "mem_gb": 16.02}
|
| 134 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.033205773511280616, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.8, "frames": {"chat": 254}, "mem_gb": 15.93}
|
| 135 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.033417627344016605, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.2353515625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.0, "frames": {"chat": 241}, "mem_gb": 16.02}
|
| 136 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04776025845406887, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.216796875, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 213}, "mem_gb": 16.05}
|
| 137 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04037379693359447, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.2119140625, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 221}, "mem_gb": 16.1}
|
| 138 |
+
[eval step 110] sample: 'To solve the problem, we need to find the digits \\(a, b, k, m, r\\) such that each digit is a non-zero digit (i.e., between 1 and 9) and satisfies the given equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\'
|
| 139 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.044434299368970094, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.2412109375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 196}, "mem_gb": 16.06}
|
| 140 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0333152589352103, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.2080078125, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.0, "frames": {"chat": 270}, "mem_gb": 15.86}
|
| 141 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03534065244613836, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.21484375, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 216}, "mem_gb": 16.04}
|
| 142 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.040529147774881376, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.197265625, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 200}, "mem_gb": 16.01}
|
| 143 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03757092032081758, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.2138671875, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 206}, "mem_gb": 15.96}
|
| 144 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03247500213080396, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.1923828125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.5, "frames": {"chat": 234}, "mem_gb": 15.99}
|
| 145 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.037848325089799864, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.2109375, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.0, "frames": {"chat": 215}, "mem_gb": 16.0}
|
| 146 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03213813143230509, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.205078125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 255}, "mem_gb": 15.99}
|
| 147 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.032368304668770484, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.1982421875, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.0, "frames": {"chat": 250}, "mem_gb": 15.87}
|
| 148 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.033178579137373404, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.197265625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 242}, "mem_gb": 16.04}
|
| 149 |
+
[eval step 120] sample: "To solve the problem, we need to find the digits \\(a, b, k, m, p,\\) and \\(r\\) such that each letter represents a non-zero digit and the given equations hold true. Let's break down the problem step-by-"
|
| 150 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04187165923851232, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.2177734375, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 199}, "mem_gb": 16.05}
|
| 151 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.049618725496840974, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.22265625, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 152 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03635383392153308, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.3125, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 208}, "mem_gb": 16.02}
|
| 153 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.043045021270743264, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.21875, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.2, "frames": {"chat": 209}, "mem_gb": 16.17}
|
| 154 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.031225860221870242, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.5, "frames": {"chat": 235}, "mem_gb": 16.0}
|
| 155 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.035118358117295426, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.2138671875, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 204}, "mem_gb": 15.99}
|
| 156 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.045566377886307116, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.232421875, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.7, "frames": {"chat": 208}, "mem_gb": 16.05}
|
| 157 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.033411375429985735, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.2001953125, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.0, "frames": {"chat": 240}, "mem_gb": 16.05}
|
| 158 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03442209031227976, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.2255859375, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 246}, "mem_gb": 16.04}
|
| 159 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03496374331774811, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.1943359375, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 243}, "mem_gb": 15.86}
|
| 160 |
+
[eval step 130] sample: "To solve the given system of equations, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by"
|
| 161 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04256709007731018, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.2236328125, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 208}, "mem_gb": 16.06}
|
| 162 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04353734484561719, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.2216796875, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 163 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.045320879577457285, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.2177734375, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 211}, "mem_gb": 16.06}
|
| 164 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03994098002462027, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.2314453125, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.1, "frames": {"chat": 232}, "mem_gb": 16.02}
|
| 165 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04629031352402332, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.224609375, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 214}, "mem_gb": 16.05}
|
| 166 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.036389342272576564, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 226}, "mem_gb": 15.94}
|
| 167 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03558888955223374, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 210}, "mem_gb": 16.06}
|
| 168 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03412930506222571, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.2041015625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 218}, "mem_gb": 15.88}
|
| 169 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03288377221804112, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 233}, "mem_gb": 16.03}
|
| 170 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04284478505373312, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.2119140625, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 171 |
+
[eval step 140] sample: 'To solve the problem, we need to find the digits \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(p\\) such that each letter represents a non-zero digit and satisfy the given equations:\n\n\\[\n\\begin{align*}\na + b &= k '
|
| 172 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03632894418287712, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.19921875, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 173 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.035847735164438684, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.2294921875, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 209}, "mem_gb": 15.99}
|
| 174 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.029498823017751176, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.1708984375, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.6, "frames": {"chat": 262}, "mem_gb": 15.92}
|
| 175 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03295737003815205, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 249}, "mem_gb": 16.01}
|
| 176 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.041888812201377, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.19921875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 227}, "mem_gb": 16.04}
|
| 177 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03317792658337858, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.1875, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 221}, "mem_gb": 16.04}
|
| 178 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03243056264965174, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.1875, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.7, "frames": {"chat": 234}, "mem_gb": 16.06}
|
| 179 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03404021299170951, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.21484375, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 213}, "mem_gb": 16.0}
|
| 180 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03388322359360754, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.1923828125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 213}, "mem_gb": 15.94}
|
| 181 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.032232557944192865, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.185546875, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 234}, "mem_gb": 15.97}
|
| 182 |
+
[eval step 150] sample: "To solve the problem, we need to find the digits \\(a, b, k, m, r\\) such that each letter represents a non-zero digit and the given equations hold true. Let's break down the problem step-by-step:\n\n1. *"
|
| 183 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep50_s1224/step0150
|
| 184 |
+
wandb: updating run metadata
|
| 185 |
+
wandb: uploading summary, console lines 171-171; uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 186 |
+
wandb: uploading data
|
| 187 |
+
wandb:
|
| 188 |
+
wandb: Run history:
|
| 189 |
+
wandb: comp_len ββββββββββββββββββ
βββββββ
ββββββ
βββββββββ
|
| 190 |
+
wandb: cumulative_loss_tokens ββββββββββββββββββββ
β
β
β
β
β
βββββββββββββββ
|
| 191 |
+
wandb: epoch βββββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
βββββββββββββ
|
| 192 |
+
wandb: finish_rate ββ
ββββ
βββ
β
β
βββββββββββ
ββ
ββ
ββ
ββββββββββββ
|
| 193 |
+
wandb: forward_topk_kl ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 195 |
+
wandb: lr ββββ
ββββββββββββββββββββββββββββββββββββ
|
| 196 |
+
wandb: mem_gb ββββ
βββββ
βββββββββββββββ
β
βββββββββββββββ
|
| 197 |
+
wandb: step βββββββββββββββββββ
β
β
β
β
β
ββββββββββββββββ
|
| 198 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 199 |
+
wandb: +3 ...
|
| 200 |
+
wandb:
|
| 201 |
+
wandb: Run summary:
|
| 202 |
+
wandb: comp_len 512.8
|
| 203 |
+
wandb: cumulative_loss_tokens 18000000
|
| 204 |
+
wandb: epoch 2
|
| 205 |
+
wandb: finish_rate 0.906
|
| 206 |
+
wandb: forward_topk_kl 0.03223
|
| 207 |
+
wandb: grad_norm 0.18555
|
| 208 |
+
wandb: lr 3e-05
|
| 209 |
+
wandb: mem_gb 15.97
|
| 210 |
+
wandb: step 150
|
| 211 |
+
wandb: t_data_s 0
|
| 212 |
+
wandb: +4 ...
|
| 213 |
+
wandb:
|
| 214 |
+
wandb: π View run reap-math-keep50-s1224 at: https://wandb.ai/hbfreed/glean-grid/runs/1c8m5fh6
|
| 215 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 216 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 217 |
+
wandb: Find logs at: outputs/healed/grid_math/reap_keep50_s1224/wandb/run-20260716_015810-1c8m5fh6/logs
|
| 218 |
+
{
|
| 219 |
+
"correct": 748,
|
| 220 |
+
"accuracy": 0.5670962850644428,
|
| 221 |
+
"finished": 1310,
|
| 222 |
+
"finish_rate": 0.9931766489764974,
|
| 223 |
+
"mean_completion_tokens": 121.06141015921152
|
| 224 |
+
}
|
| 225 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep50_s1224_step100_chat.json
|
| 226 |
+
{
|
| 227 |
+
"correct": 777,
|
| 228 |
+
"accuracy": 0.5890826383623957,
|
| 229 |
+
"finished": 1307,
|
| 230 |
+
"finish_rate": 0.9909021986353298,
|
| 231 |
+
"mean_completion_tokens": 120.55724033358605
|
| 232 |
+
}
|
| 233 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep50_s1224_step150_chat.json
|
healed/grid_math/reap_keep50_s1225.console.log
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run 3wq993z8
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/reap_keep50_s1225/wandb/run-20260716_014634-3wq993z8
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run reap-math-keep50-s1225
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/3wq993z8
|
| 12 |
+
|
| 13 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 3.70B | teacher overlap=False
|
| 14 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5545137798349062, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 8.1875, "lr": 6e-06, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 191}, "mem_gb": 15.94}
|
| 15 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 16 |
+
[eval step 1] sample: 'To solve this problem, we need to understand how to arrange the numbers from 1 to 49 in a spiral pattern on a square grid and then identify which of the four numbers in the shaded diagonal (which incl'
|
| 17 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.511051233595113, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 7.90625, "lr": 9e-06, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 219}, "mem_gb": 16.04}
|
| 18 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5664724428529541, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 8.25, "lr": 1.2e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 207}, "mem_gb": 16.05}
|
| 19 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4547356678421299, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 5.625, "lr": 1.5e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 208}, "mem_gb": 16.0}
|
| 20 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.35456649297314385, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 4.3125, "lr": 1.8e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 219}, "mem_gb": 16.04}
|
| 21 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.27364045970203976, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 3.15625, "lr": 2.1e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 246}, "mem_gb": 15.92}
|
| 22 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28186713957662385, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 2.6875, "lr": 2.4e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 206}, "mem_gb": 16.07}
|
| 23 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2126233731985092, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 1.6328125, "lr": 2.7000000000000002e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 233}, "mem_gb": 16.05}
|
| 24 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2143800116557007, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 1.3984375, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 229}, "mem_gb": 15.91}
|
| 25 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1643844889894128, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.859375, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 236}, "mem_gb": 15.95}
|
| 26 |
+
[eval step 10] sample: 'To solve this problem, we need to understand how the numbers are arranged in the spiral pattern on the square grid and identify the numbers that will appear in the shaded squares, specifically those o'
|
| 27 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16748397135113677, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.734375, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 239}, "mem_gb": 15.83}
|
| 28 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13983654592956105, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.6796875, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 241}, "mem_gb": 15.95}
|
| 29 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15315795515781888, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.65234375, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 226}, "mem_gb": 15.92}
|
| 30 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12836425958182662, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.58984375, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 234}, "mem_gb": 16.05}
|
| 31 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12653109082685163, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.5625, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 257}, "mem_gb": 16.04}
|
| 32 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16807576383631675, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.62109375, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 208}, "mem_gb": 16.09}
|
| 33 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1437206761604796, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 211}, "mem_gb": 16.07}
|
| 34 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12487599890523901, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 227}, "mem_gb": 16.05}
|
| 35 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1341195928434531, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.52734375, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 211}, "mem_gb": 16.03}
|
| 36 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10862548385470484, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 238}, "mem_gb": 16.04}
|
| 37 |
+
[eval step 20] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers that lie on the same diagonal as the number \\(7\\). The spiral pattern starts at the center and'
|
| 38 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11279117373302579, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.4453125, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.9, "frames": {"chat": 237}, "mem_gb": 16.08}
|
| 39 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1262405207300248, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.46484375, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 40 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1132457225639373, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.42578125, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 221}, "mem_gb": 16.17}
|
| 41 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10590222140826906, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 232}, "mem_gb": 16.01}
|
| 42 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11071867505262295, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 208}, "mem_gb": 16.04}
|
| 43 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09961375055468331, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.412109375, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.5, "frames": {"chat": 227}, "mem_gb": 15.96}
|
| 44 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10894187485749522, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.421875, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 221}, "mem_gb": 15.99}
|
| 45 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0969536163265196, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 232}, "mem_gb": 16.05}
|
| 46 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09553669060282409, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.373046875, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 47 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10545470687095077, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 195}, "mem_gb": 16.14}
|
| 48 |
+
[eval step 30] sample: 'To solve this problem, we need to identify the numbers in the shaded squares on the same diagonal as the number \\(7\\) in a spiral pattern from \\(1\\) to \\(49\\). The spiral pattern is structured such th'
|
| 49 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09930714209843428, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.3984375, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 216}, "mem_gb": 16.05}
|
| 50 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08630999103264572, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 208}, "mem_gb": 15.93}
|
| 51 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0832340875162743, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 235}, "mem_gb": 15.93}
|
| 52 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08614614320375646, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 225}, "mem_gb": 16.04}
|
| 53 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12326049803669253, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 213}, "mem_gb": 16.13}
|
| 54 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0842804021329619, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.455078125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 257}, "mem_gb": 15.8}
|
| 55 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1019067531695279, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.375, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 212}, "mem_gb": 16.07}
|
| 56 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08904900835699713, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 221}, "mem_gb": 16.05}
|
| 57 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0843076329773292, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 242}, "mem_gb": 16.04}
|
| 58 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08060843795586067, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.345703125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 220}, "mem_gb": 16.01}
|
| 59 |
+
[eval step 40] sample: 'To solve this problem, we need to understand the structure of the spiral pattern on the square grid and identify the numbers that lie on the same diagonal as the number \\(7\\).\n\n### Steps to Solve the '
|
| 60 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07860646998998709, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 240}, "mem_gb": 15.9}
|
| 61 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08989213389915725, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 206}, "mem_gb": 16.03}
|
| 62 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08902159064803272, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 226}, "mem_gb": 16.05}
|
| 63 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10788680763120452, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 244}, "mem_gb": 15.83}
|
| 64 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09422282850540553, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 224}, "mem_gb": 16.05}
|
| 65 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0817956012977908, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.359375, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.8, "frames": {"chat": 271}, "mem_gb": 15.77}
|
| 66 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07737589882897834, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 236}, "mem_gb": 16.06}
|
| 67 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08941993831327806, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 232}, "mem_gb": 15.93}
|
| 68 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07456887081516907, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 210}, "mem_gb": 15.98}
|
| 69 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0743259880459557, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.34375, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 217}, "mem_gb": 15.95}
|
| 70 |
+
[eval step 50] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and how the numbers are placed on the grid. The spiral pattern starts at the center and moves outward, forming a diamon'
|
| 71 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep50_s1225/step0050
|
| 72 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09199335136047254, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 224}, "mem_gb": 16.07}
|
| 73 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10627452008575201, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.39453125, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 203}, "mem_gb": 15.92}
|
| 74 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08060978428038458, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.357421875, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 239}, "mem_gb": 16.02}
|
| 75 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05341216823590609, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.271484375, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.9, "frames": {"chat": 254}, "mem_gb": 15.93}
|
| 76 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.057884399864574276, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 241}, "mem_gb": 16.02}
|
| 77 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07263648351286538, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.3046875, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 213}, "mem_gb": 16.05}
|
| 78 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0615926475533129, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 221}, "mem_gb": 16.1}
|
| 79 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06679541949632888, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 196}, "mem_gb": 16.06}
|
| 80 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05176316303300361, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.6, "frames": {"chat": 270}, "mem_gb": 15.86}
|
| 81 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05212113441683663, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 216}, "mem_gb": 16.04}
|
| 82 |
+
[eval step 60] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers that lie on the same diagonal as the number \\(7\\). The spiral pattern starts at the center and'
|
| 83 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07816804139691716, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.359375, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 200}, "mem_gb": 16.01}
|
| 84 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05035967906449611, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 206}, "mem_gb": 15.96}
|
| 85 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.044304009293485436, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.23828125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 234}, "mem_gb": 15.99}
|
| 86 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05535436973415005, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.271484375, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 215}, "mem_gb": 16.0}
|
| 87 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.044907201238783695, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 255}, "mem_gb": 15.99}
|
| 88 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05459509837126049, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 250}, "mem_gb": 15.87}
|
| 89 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.050126402091002095, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 242}, "mem_gb": 16.04}
|
| 90 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07252040647125492, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 199}, "mem_gb": 16.05}
|
| 91 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08199389099515975, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 92 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06270051176787043, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.41796875, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 208}, "mem_gb": 16.02}
|
| 93 |
+
[eval step 70] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers that lie on the same diagonal as the number 7. The spiral pattern starts at the center and mov'
|
| 94 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06326160759705429, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.30859375, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 209}, "mem_gb": 16.17}
|
| 95 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.047964732579079766, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 235}, "mem_gb": 16.0}
|
| 96 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05000873766591152, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.271484375, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 204}, "mem_gb": 15.99}
|
| 97 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06726836371614288, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.302734375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 208}, "mem_gb": 16.05}
|
| 98 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.049516278548678384, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.255859375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 240}, "mem_gb": 16.05}
|
| 99 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05665072427910442, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.3046875, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 246}, "mem_gb": 16.04}
|
| 100 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05669951267732928, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 243}, "mem_gb": 15.86}
|
| 101 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07075669393978702, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 208}, "mem_gb": 16.06}
|
| 102 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05974680195176043, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 103 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0652965749655074, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 211}, "mem_gb": 16.06}
|
| 104 |
+
[eval step 80] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers that lie on the same diagonal as the number 7. The spiral pattern starts at the center and mov'
|
| 105 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05211935499627143, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 232}, "mem_gb": 16.02}
|
| 106 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.059444162550428885, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 214}, "mem_gb": 16.05}
|
| 107 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.055917505533155054, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.5, "frames": {"chat": 226}, "mem_gb": 15.94}
|
| 108 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.055923757360627255, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 210}, "mem_gb": 16.06}
|
| 109 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.047685957645904276, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.24609375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 218}, "mem_gb": 15.88}
|
| 110 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04715602353129846, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.2421875, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 233}, "mem_gb": 16.03}
|
| 111 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0596792153040568, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 112 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05345027676153307, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 113 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.050646483299819134, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 209}, "mem_gb": 15.99}
|
| 114 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.044942502258066085, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.302734375, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 262}, "mem_gb": 15.92}
|
| 115 |
+
[eval step 90] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers that lie on the same diagonal as the number 7. The spiral pattern starts at the center and mov'
|
| 116 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.045185356042953206, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.232421875, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 249}, "mem_gb": 16.01}
|
| 117 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06120699836833713, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.28125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 227}, "mem_gb": 16.04}
|
| 118 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.053615750029784005, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.263671875, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 221}, "mem_gb": 16.04}
|
| 119 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.048285325485731785, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 234}, "mem_gb": 16.06}
|
| 120 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04676177461682819, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.26171875, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 213}, "mem_gb": 16.0}
|
| 121 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.044436498762403305, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.23046875, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 213}, "mem_gb": 15.94}
|
| 122 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.052153425100895885, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.291015625, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 234}, "mem_gb": 15.97}
|
| 123 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05320698270440723, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 222}, "mem_gb": 16.04}
|
| 124 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.051590269053028895, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.2431640625, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.1, "frames": {"chat": 227}, "mem_gb": 16.05}
|
| 125 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.057174564701706794, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.279296875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 218}, "mem_gb": 16.09}
|
| 126 |
+
[eval step 100] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and how the numbers are placed on the grid. The spiral pattern starts at the center and moves outward, forming a diamon'
|
| 127 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep50_s1225/step0100
|
| 128 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05653364848041286, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 223}, "mem_gb": 16.06}
|
| 129 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05626137313898653, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.248046875, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 206}, "mem_gb": 16.05}
|
| 130 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04629964479301125, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.24609375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 213}, "mem_gb": 15.97}
|
| 131 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06836010164196292, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.30859375, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.9, "frames": {"chat": 223}, "mem_gb": 15.91}
|
| 132 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04960931596377244, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.2333984375, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 227}, "mem_gb": 16.02}
|
| 133 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04606072457487074, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.2431640625, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.0, "frames": {"chat": 253}, "mem_gb": 16.05}
|
| 134 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.055348847734757387, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.388671875, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 216}, "mem_gb": 16.05}
|
| 135 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03895854299830583, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 205}, "mem_gb": 16.02}
|
| 136 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04872952806249571, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.2197265625, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 207}, "mem_gb": 16.11}
|
| 137 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.044403939206223, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.20703125, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 215}, "mem_gb": 16.03}
|
| 138 |
+
[eval step 110] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers that lie on the same diagonal as the number 7. The spiral pattern starts at the center and mov'
|
| 139 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.035836265381332486, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.2275390625, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 228}, "mem_gb": 16.05}
|
| 140 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03942134256685773, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.2109375, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 221}, "mem_gb": 16.09}
|
| 141 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.029532018057101716, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.181640625, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 254}, "mem_gb": 15.89}
|
| 142 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03353456746751132, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.248046875, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 210}, "mem_gb": 16.01}
|
| 143 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.033196155026756845, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.203125, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 226}, "mem_gb": 15.97}
|
| 144 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03837967902193001, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.2177734375, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 212}, "mem_gb": 16.04}
|
| 145 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04279236014199753, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.2158203125, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 211}, "mem_gb": 15.97}
|
| 146 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04495121304591497, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.2314453125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 196}, "mem_gb": 16.02}
|
| 147 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03516720853671432, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.2021484375, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 212}, "mem_gb": 16.04}
|
| 148 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03228002276973954, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.1904296875, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 244}, "mem_gb": 15.95}
|
| 149 |
+
[eval step 120] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers that lie on the same diagonal as the number 7. The spiral pattern starts at the center and mov'
|
| 150 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03490392869187829, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.1845703125, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 222}, "mem_gb": 16.0}
|
| 151 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03468469969631793, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.1943359375, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 218}, "mem_gb": 16.05}
|
| 152 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03472803884683332, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.19921875, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.9, "frames": {"chat": 252}, "mem_gb": 15.92}
|
| 153 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04160250526641806, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 202}, "mem_gb": 16.1}
|
| 154 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04120717598129995, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.212890625, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 237}, "mem_gb": 16.05}
|
| 155 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03709639240807543, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.203125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 234}, "mem_gb": 16.03}
|
| 156 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03265363146445403, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.2138671875, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 157 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02957465929450312, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.181640625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 234}, "mem_gb": 15.98}
|
| 158 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.031586363252205776, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.1806640625, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 216}, "mem_gb": 16.03}
|
| 159 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03465553052170047, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.189453125, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 210}, "mem_gb": 16.0}
|
| 160 |
+
[eval step 130] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers that lie on the same diagonal as the number 7. The spiral pattern starts at the center and mov'
|
| 161 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03964869703074607, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.2177734375, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 199}, "mem_gb": 16.05}
|
| 162 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03579281948882466, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.2099609375, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 210}, "mem_gb": 16.06}
|
| 163 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.033542947825191856, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 225}, "mem_gb": 16.0}
|
| 164 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.032140976130838196, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.1826171875, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 253}, "mem_gb": 15.9}
|
| 165 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03224871880656574, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.1796875, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.0, "frames": {"chat": 247}, "mem_gb": 16.02}
|
| 166 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03395894770209367, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.1806640625, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 238}, "mem_gb": 16.02}
|
| 167 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03957084504778807, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.2353515625, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 235}, "mem_gb": 16.04}
|
| 168 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04128963355836458, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.228515625, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.5, "frames": {"chat": 215}, "mem_gb": 16.01}
|
| 169 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03190090727764182, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.205078125, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 268}, "mem_gb": 16.02}
|
| 170 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.035086253207425276, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.201171875, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 228}, "mem_gb": 16.05}
|
| 171 |
+
[eval step 140] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers that lie on the same diagonal as the number 7. The spiral pattern starts at the center and mov'
|
| 172 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.033764375670044686, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.1982421875, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 252}, "mem_gb": 15.98}
|
| 173 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03700443701595068, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.2119140625, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 223}, "mem_gb": 16.06}
|
| 174 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04124379948658558, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.20703125, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 226}, "mem_gb": 16.04}
|
| 175 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04097800794257006, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 208}, "mem_gb": 16.09}
|
| 176 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0317224430399326, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.1904296875, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 240}, "mem_gb": 15.98}
|
| 177 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.037275905701781936, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.1826171875, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 222}, "mem_gb": 15.97}
|
| 178 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03051341254238505, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.173828125, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 236}, "mem_gb": 16.04}
|
| 179 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03466207155267087, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.205078125, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 217}, "mem_gb": 16.01}
|
| 180 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03411953383701233, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.2314453125, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 252}, "mem_gb": 15.92}
|
| 181 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.034074287171739465, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.2041015625, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 222}, "mem_gb": 16.04}
|
| 182 |
+
[eval step 150] sample: 'To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers that lie on the same diagonal as the number 7. Then, we will determine which of these numbers '
|
| 183 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep50_s1225/step0150
|
| 184 |
+
wandb: updating run metadata
|
| 185 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 186 |
+
wandb:
|
| 187 |
+
wandb: Run history:
|
| 188 |
+
wandb: comp_len βββ
βββββββ
β
β
βββ
β
ββββββββββββ
βββ
βββ
βββ
βββ
|
| 189 |
+
wandb: cumulative_loss_tokens ββββββββββββββββββββββββββββββ
β
β
β
β
β
βββββ
|
| 190 |
+
wandb: epoch ββββββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββββββββββ
|
| 191 |
+
wandb: finish_rate ββ
βββ
ββββ
ββ
ββββββββββββββββ
βββ
ββ
ββββββββ
|
| 192 |
+
wandb: forward_topk_kl ββββββββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 195 |
+
wandb: mem_gb ββββββββ
ββββββββββββββββ
β
ββ
βββ
ββ
ββββ
β
βββ
|
| 196 |
+
wandb: step ββββββββββββββββ
β
β
β
β
β
β
ββββββββββββββββββ
|
| 197 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 198 |
+
wandb: +3 ...
|
| 199 |
+
wandb:
|
| 200 |
+
wandb: Run summary:
|
| 201 |
+
wandb: comp_len 540.5
|
| 202 |
+
wandb: cumulative_loss_tokens 18000000
|
| 203 |
+
wandb: epoch 2
|
| 204 |
+
wandb: finish_rate 0.847
|
| 205 |
+
wandb: forward_topk_kl 0.03407
|
| 206 |
+
wandb: grad_norm 0.2041
|
| 207 |
+
wandb: lr 3e-05
|
| 208 |
+
wandb: mem_gb 16.04
|
| 209 |
+
wandb: step 150
|
| 210 |
+
wandb: t_data_s 0
|
| 211 |
+
wandb: +4 ...
|
| 212 |
+
wandb:
|
| 213 |
+
wandb: π View run reap-math-keep50-s1225 at: https://wandb.ai/hbfreed/glean-grid/runs/3wq993z8
|
| 214 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 215 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 216 |
+
wandb: Find logs at: outputs/healed/grid_math/reap_keep50_s1225/wandb/run-20260716_014634-3wq993z8/logs
|
| 217 |
+
{
|
| 218 |
+
"correct": 768,
|
| 219 |
+
"accuracy": 0.5822592873388931,
|
| 220 |
+
"finished": 1300,
|
| 221 |
+
"finish_rate": 0.9855951478392722,
|
| 222 |
+
"mean_completion_tokens": 123.63381349507202
|
| 223 |
+
}
|
| 224 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep50_s1225_step100_chat.json
|
| 225 |
+
{
|
| 226 |
+
"correct": 769,
|
| 227 |
+
"accuracy": 0.5830174374526156,
|
| 228 |
+
"finished": 1306,
|
| 229 |
+
"finish_rate": 0.9901440485216073,
|
| 230 |
+
"mean_completion_tokens": 124.30326004548901
|
| 231 |
+
}
|
| 232 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep50_s1225_step150_chat.json
|
healed/grid_math/reap_keep50_s1226.console.log
ADDED
|
@@ -0,0 +1,231 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 6 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/reap_keep50_s1226/wandb/run-20260716_014602-qt9aq0ed
|
| 7 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 8 |
+
wandb: Syncing run reap-math-keep50-s1226
|
| 9 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 10 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/qt9aq0ed
|
| 11 |
+
|
| 12 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 3.70B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5060686465984832, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 8.0, "lr": 6e-06, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.8, "frames": {"chat": 254}, "mem_gb": 15.82}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: 'To solve this problem, we need to understand how the given conditions relate to the geometry and algebra involved in constructing the perimeter of the resulting triangle.\n\nGiven:\n1. The perimeter of t'
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5423026348200937, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 8.125, "lr": 9e-06, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 241}, "mem_gb": 16.02}
|
| 17 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5477972302235663, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 7.3125, "lr": 1.2e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 213}, "mem_gb": 16.05}
|
| 18 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4212719477940351, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 5.21875, "lr": 1.5e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 221}, "mem_gb": 16.1}
|
| 19 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.405326781245321, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 4.25, "lr": 1.8e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 196}, "mem_gb": 16.06}
|
| 20 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2711340561737617, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 2.765625, "lr": 2.1e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.1, "frames": {"chat": 270}, "mem_gb": 15.86}
|
| 21 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.25732222653937836, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 2.515625, "lr": 2.4e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 216}, "mem_gb": 16.04}
|
| 22 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.22487505531342078, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 2.21875, "lr": 2.7000000000000002e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 200}, "mem_gb": 16.01}
|
| 23 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18226914721199622, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 1.140625, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 206}, "mem_gb": 15.96}
|
| 24 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15004909853960077, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.87890625, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 234}, "mem_gb": 15.99}
|
| 25 |
+
[eval step 10] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of the original triangle.\n\n1. **Understand the Geometry:**\n - Le'
|
| 26 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1570096647741273, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.73046875, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 215}, "mem_gb": 16.0}
|
| 27 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12968674462232738, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.6484375, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 255}, "mem_gb": 15.99}
|
| 28 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1286253351541236, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.578125, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 250}, "mem_gb": 15.87}
|
| 29 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1314010168324535, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.59765625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 242}, "mem_gb": 16.04}
|
| 30 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15113212025662262, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.58984375, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 199}, "mem_gb": 16.05}
|
| 31 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20640578821363548, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.83203125, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 32 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13163779961920033, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.58984375, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 208}, "mem_gb": 16.02}
|
| 33 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1457826923261086, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.55859375, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 209}, "mem_gb": 16.17}
|
| 34 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10949542039077108, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 235}, "mem_gb": 16.0}
|
| 35 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11166741707672675, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.435546875, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 204}, "mem_gb": 15.99}
|
| 36 |
+
[eval step 20] sample: 'To solve this problem, we need to understand the geometric relationship between the original triangle and the new triangle formed by connecting the midpoints of its sides.\n\n### Steps to Solve:\n\n1. **U'
|
| 37 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14399345871539165, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.59375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 208}, "mem_gb": 16.05}
|
| 38 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10751764197905238, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.443359375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 240}, "mem_gb": 16.05}
|
| 39 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10304307096370806, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.41796875, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 246}, "mem_gb": 16.04}
|
| 40 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11230557647921766, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.48828125, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 243}, "mem_gb": 15.86}
|
| 41 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12358204048875099, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.49609375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 208}, "mem_gb": 16.06}
|
| 42 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11778800500386084, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.447265625, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 43 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12739090577003856, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.515625, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 211}, "mem_gb": 16.06}
|
| 44 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10657288894901673, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 232}, "mem_gb": 16.02}
|
| 45 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12152383521292359, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 214}, "mem_gb": 16.05}
|
| 46 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10047705957808842, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.46875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 226}, "mem_gb": 15.94}
|
| 47 |
+
[eval step 30] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a triangle.\n\n### Steps to Solve:\n\n1. **Understand the Geometry:'
|
| 48 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09486804561313862, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 210}, "mem_gb": 16.06}
|
| 49 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08934559583207592, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 218}, "mem_gb": 15.88}
|
| 50 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0917849869439068, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 233}, "mem_gb": 16.03}
|
| 51 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11323483313173055, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 52 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0996293871806624, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.3828125, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 53 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09028497856548057, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.3671875, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 209}, "mem_gb": 15.99}
|
| 54 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08264376465283955, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 262}, "mem_gb": 15.92}
|
| 55 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08528272054875269, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.345703125, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 249}, "mem_gb": 16.01}
|
| 56 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10894312866926194, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 227}, "mem_gb": 16.04}
|
| 57 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08186319210253035, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 221}, "mem_gb": 16.04}
|
| 58 |
+
[eval step 40] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of the original triangle.\n\n### Steps to Solve:\n\n1. **Understand th'
|
| 59 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08784388510862676, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.3671875, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 234}, "mem_gb": 16.06}
|
| 60 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07941963178347797, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 213}, "mem_gb": 16.0}
|
| 61 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0763591806843256, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 213}, "mem_gb": 15.94}
|
| 62 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07646929621233915, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.5, "frames": {"chat": 234}, "mem_gb": 15.97}
|
| 63 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07922823091962686, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 222}, "mem_gb": 16.04}
|
| 64 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09367436103234067, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.9, "frames": {"chat": 227}, "mem_gb": 16.05}
|
| 65 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08664301028043653, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 218}, "mem_gb": 16.09}
|
| 66 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0917276910001412, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 223}, "mem_gb": 16.06}
|
| 67 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09697089948064337, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 206}, "mem_gb": 16.05}
|
| 68 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08414388034647952, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 213}, "mem_gb": 15.97}
|
| 69 |
+
[eval step 50] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by the midpoints of the original triangle.\n\n1. **Understanding the Problem:**\n - The original triangle ha'
|
| 70 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep50_s1226/step0050
|
| 71 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10459128459574034, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 223}, "mem_gb": 15.91}
|
| 72 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08826250612973235, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 227}, "mem_gb": 16.02}
|
| 73 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07563404675796628, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 253}, "mem_gb": 16.05}
|
| 74 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06886030516719135, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 216}, "mem_gb": 16.05}
|
| 75 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06066073822774924, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.271484375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 205}, "mem_gb": 16.02}
|
| 76 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07448192878928966, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.279296875, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 207}, "mem_gb": 16.11}
|
| 77 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07085500255872806, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.28515625, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 215}, "mem_gb": 16.03}
|
| 78 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.054219987386542684, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 228}, "mem_gb": 16.05}
|
| 79 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06803286743182689, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.306640625, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 221}, "mem_gb": 16.09}
|
| 80 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04641861871878306, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.2392578125, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 254}, "mem_gb": 15.89}
|
| 81 |
+
[eval step 60] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 82 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05023701873199704, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.263671875, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 210}, "mem_gb": 16.01}
|
| 83 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.051724348243310424, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 226}, "mem_gb": 15.97}
|
| 84 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06272758410156239, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 212}, "mem_gb": 16.04}
|
| 85 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06369718516276529, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.271484375, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 211}, "mem_gb": 15.97}
|
| 86 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0647698606271452, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.291015625, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 196}, "mem_gb": 16.02}
|
| 87 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.059465903049598756, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 212}, "mem_gb": 16.04}
|
| 88 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0527960841421814, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 244}, "mem_gb": 15.95}
|
| 89 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05209585945080034, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.2314453125, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 222}, "mem_gb": 16.0}
|
| 90 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05374352757440259, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.26171875, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 218}, "mem_gb": 16.05}
|
| 91 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05794376976617301, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.28125, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 252}, "mem_gb": 15.92}
|
| 92 |
+
[eval step 70] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 93 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06893676832563554, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.3125, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 202}, "mem_gb": 16.1}
|
| 94 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07893477528166647, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.359375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 237}, "mem_gb": 16.05}
|
| 95 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06014306015539914, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.26171875, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 234}, "mem_gb": 16.03}
|
| 96 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.047996336566330865, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 97 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.044697797822409, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.2470703125, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 234}, "mem_gb": 15.98}
|
| 98 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04588704365013788, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.26171875, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 216}, "mem_gb": 16.03}
|
| 99 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05109065430917156, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.2451171875, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 210}, "mem_gb": 16.0}
|
| 100 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05233408396915765, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 199}, "mem_gb": 16.05}
|
| 101 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04801637768183525, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.23828125, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 210}, "mem_gb": 16.06}
|
| 102 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.053485469416653116, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 225}, "mem_gb": 16.0}
|
| 103 |
+
[eval step 80] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 104 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05053567462177016, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 253}, "mem_gb": 15.9}
|
| 105 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04922324699338836, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.2373046875, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 247}, "mem_gb": 16.02}
|
| 106 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.050183612028866384, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.2353515625, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 238}, "mem_gb": 16.02}
|
| 107 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.054622684019478035, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 235}, "mem_gb": 16.04}
|
| 108 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05814677753490396, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.30859375, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 215}, "mem_gb": 16.01}
|
| 109 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05711578755577405, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 268}, "mem_gb": 16.02}
|
| 110 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.051577915780417, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 228}, "mem_gb": 16.05}
|
| 111 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.055417160641759014, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.279296875, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 252}, "mem_gb": 15.98}
|
| 112 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05028329864336799, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 223}, "mem_gb": 16.06}
|
| 113 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07661831421251408, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 226}, "mem_gb": 16.04}
|
| 114 |
+
[eval step 90] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and the midpoints of its sides.\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.'
|
| 115 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.061045037919143216, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.291015625, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 208}, "mem_gb": 16.09}
|
| 116 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04929783342856293, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 240}, "mem_gb": 15.98}
|
| 117 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05484731227552208, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.2470703125, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 222}, "mem_gb": 15.97}
|
| 118 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04571815300624973, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.2333984375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 236}, "mem_gb": 16.04}
|
| 119 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04882919267296481, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.255859375, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 217}, "mem_gb": 16.01}
|
| 120 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05304940091329627, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 252}, "mem_gb": 15.92}
|
| 121 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0453409172443673, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.2431640625, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 222}, "mem_gb": 16.04}
|
| 122 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04914017798041459, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.901, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 242}, "mem_gb": 15.92}
|
| 123 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.06614599686032162, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 219}, "mem_gb": 15.98}
|
| 124 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04687103871277844, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.2412109375, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 223}, "mem_gb": 15.99}
|
| 125 |
+
[eval step 100] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 126 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep50_s1226/step0100
|
| 127 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05508217736965356, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 232}, "mem_gb": 15.99}
|
| 128 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05771918959524482, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.28515625, "lr": 3e-05, "finish_rate": 0.832, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 220}, "mem_gb": 16.05}
|
| 129 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.05670769996464563, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.26171875, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 210}, "mem_gb": 16.09}
|
| 130 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04982835436208795, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.2431640625, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 226}, "mem_gb": 16.02}
|
| 131 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.049105689908145, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.248046875, "lr": 3e-05, "finish_rate": 0.741, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 212}, "mem_gb": 16.04}
|
| 132 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04600744808347275, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.2470703125, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 236}, "mem_gb": 16.06}
|
| 133 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.033932253168631965, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.212890625, "lr": 3e-05, "finish_rate": 0.928, "comp_len": 454.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 264}, "mem_gb": 15.93}
|
| 134 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0591216995248571, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 229}, "mem_gb": 16.03}
|
| 135 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.033564152960277475, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.197265625, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 465.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 258}, "mem_gb": 15.9}
|
| 136 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05125372361148087, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 208}, "mem_gb": 16.06}
|
| 137 |
+
[eval step 110] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 138 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03912771535598052, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.2197265625, "lr": 3e-05, "finish_rate": 0.88, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 249}, "mem_gb": 15.97}
|
| 139 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03329837650329185, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.185546875, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 220}, "mem_gb": 16.04}
|
| 140 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03639425171962939, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.216796875, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 223}, "mem_gb": 16.03}
|
| 141 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03565040662499765, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.2001953125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 221}, "mem_gb": 16.04}
|
| 142 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03295701659352829, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 230}, "mem_gb": 15.95}
|
| 143 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04046579788605062, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.205078125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 214}, "mem_gb": 16.02}
|
| 144 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0523497827064246, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.244140625, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 214}, "mem_gb": 16.04}
|
| 145 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04217318628588691, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.2109375, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 210}, "mem_gb": 16.08}
|
| 146 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04559627010982173, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.22265625, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 214}, "mem_gb": 16.04}
|
| 147 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.039164316506125035, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.791, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 215}, "mem_gb": 16.0}
|
| 148 |
+
[eval step 120] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 149 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04476614243153017, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.2177734375, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 208}, "mem_gb": 16.04}
|
| 150 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03574293609918095, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.185546875, "lr": 3e-05, "finish_rate": 0.789, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 218}, "mem_gb": 15.92}
|
| 151 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03212748550867352, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.177734375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 233}, "mem_gb": 15.94}
|
| 152 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.031179524623261144, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.1845703125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 231}, "mem_gb": 15.98}
|
| 153 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.037280551470660915, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.2109375, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 235}, "mem_gb": 16.18}
|
| 154 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03532052122700649, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.181640625, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 216}, "mem_gb": 16.03}
|
| 155 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03285237047360279, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.1943359375, "lr": 3e-05, "finish_rate": 0.831, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 237}, "mem_gb": 16.06}
|
| 156 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0456035717476625, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.2177734375, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 203}, "mem_gb": 16.06}
|
| 157 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03478093251015525, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.873, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 236}, "mem_gb": 16.08}
|
| 158 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03525485854660316, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.189453125, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 214}, "mem_gb": 16.05}
|
| 159 |
+
[eval step 130] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 160 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.039782691275918235, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.2060546875, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 209}, "mem_gb": 16.04}
|
| 161 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03294830878264426, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 468.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 256}, "mem_gb": 16.05}
|
| 162 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03306894852546975, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.1845703125, "lr": 3e-05, "finish_rate": 0.878, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 230}, "mem_gb": 15.91}
|
| 163 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03653117787890757, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.20703125, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 230}, "mem_gb": 16.1}
|
| 164 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04157050093587798, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.212890625, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 227}, "mem_gb": 16.0}
|
| 165 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04171398106655106, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.2119140625, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 208}, "mem_gb": 16.06}
|
| 166 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.04136487604933791, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.203125, "lr": 3e-05, "finish_rate": 0.699, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 206}, "mem_gb": 16.08}
|
| 167 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0372208833127593, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.1982421875, "lr": 3e-05, "finish_rate": 0.82, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 228}, "mem_gb": 15.95}
|
| 168 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03776092570159817, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.1904296875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 224}, "mem_gb": 16.04}
|
| 169 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.037290410421757646, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.205078125, "lr": 3e-05, "finish_rate": 0.66, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 200}, "mem_gb": 16.08}
|
| 170 |
+
[eval step 140] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 171 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03674508915361172, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.1982421875, "lr": 3e-05, "finish_rate": 0.714, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 196}, "mem_gb": 16.06}
|
| 172 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03390032124063, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.1865234375, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 223}, "mem_gb": 16.04}
|
| 173 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03720896863811649, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.21875, "lr": 3e-05, "finish_rate": 0.869, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 213}, "mem_gb": 15.93}
|
| 174 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03197799935330792, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.1787109375, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 232}, "mem_gb": 15.98}
|
| 175 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03063049944328765, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.173828125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 223}, "mem_gb": 15.97}
|
| 176 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03251715304492973, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.1787109375, "lr": 3e-05, "finish_rate": 0.85, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 233}, "mem_gb": 16.07}
|
| 177 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.038545895068844156, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.1923828125, "lr": 3e-05, "finish_rate": 0.816, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 217}, "mem_gb": 16.06}
|
| 178 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05470696568132068, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.240234375, "lr": 3e-05, "finish_rate": 0.752, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 202}, "mem_gb": 16.12}
|
| 179 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03262044265196504, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.1884765625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 253}, "mem_gb": 15.98}
|
| 180 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03266540290627939, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.1953125, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 231}, "mem_gb": 15.98}
|
| 181 |
+
[eval step 150] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 182 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep50_s1226/step0150
|
| 183 |
+
wandb: updating run metadata
|
| 184 |
+
wandb: uploading summary, console lines 171-171
|
| 185 |
+
wandb:
|
| 186 |
+
wandb: Run history:
|
| 187 |
+
wandb: comp_len β
ββββββ
ββ
βββ
βββ
β
βββββ
βββββ
ββ
ββββββ
β
βββββ
|
| 188 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββββββββ
β
β
β
β
β
ββββββββββ
|
| 189 |
+
wandb: epoch βββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
βββββββββββββββ
|
| 190 |
+
wandb: finish_rate β
βββββ
βββ
ββββ
β
βββ
β
ββββ
ββββββββββββββββ
ββ
|
| 191 |
+
wandb: forward_topk_kl ββ
ββββββββββββββββββββββββββββββββββββββ
|
| 192 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: mem_gb βββββ
βββ
β
β
βββββ
β
βββ
ββ
ββ
β
ββ
βββββββ
β
ββββββ
|
| 195 |
+
wandb: step ββββββββββββββββββββββββββ
β
β
ββββββββββββ
|
| 196 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 197 |
+
wandb: +3 ...
|
| 198 |
+
wandb:
|
| 199 |
+
wandb: Run summary:
|
| 200 |
+
wandb: comp_len 519.5
|
| 201 |
+
wandb: cumulative_loss_tokens 18000000
|
| 202 |
+
wandb: epoch 2
|
| 203 |
+
wandb: finish_rate 0.879
|
| 204 |
+
wandb: forward_topk_kl 0.03267
|
| 205 |
+
wandb: grad_norm 0.19531
|
| 206 |
+
wandb: lr 3e-05
|
| 207 |
+
wandb: mem_gb 15.98
|
| 208 |
+
wandb: step 150
|
| 209 |
+
wandb: t_data_s 0
|
| 210 |
+
wandb: +4 ...
|
| 211 |
+
wandb:
|
| 212 |
+
wandb: π View run reap-math-keep50-s1226 at: https://wandb.ai/hbfreed/glean-grid/runs/qt9aq0ed
|
| 213 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 214 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 215 |
+
wandb: Find logs at: outputs/healed/grid_math/reap_keep50_s1226/wandb/run-20260716_014602-qt9aq0ed/logs
|
| 216 |
+
{
|
| 217 |
+
"correct": 776,
|
| 218 |
+
"accuracy": 0.5883244882486732,
|
| 219 |
+
"finished": 1307,
|
| 220 |
+
"finish_rate": 0.9909021986353298,
|
| 221 |
+
"mean_completion_tokens": 123.34874905231236
|
| 222 |
+
}
|
| 223 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep50_s1226_step100_chat.json
|
| 224 |
+
{
|
| 225 |
+
"correct": 780,
|
| 226 |
+
"accuracy": 0.5913570887035633,
|
| 227 |
+
"finished": 1304,
|
| 228 |
+
"finish_rate": 0.9886277482941622,
|
| 229 |
+
"mean_completion_tokens": 126.97573919636088
|
| 230 |
+
}
|
| 231 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep50_s1226_step150_chat.json
|
healed/grid_math/reap_keep75_s1224.console.log
ADDED
|
@@ -0,0 +1,324 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run clpi70s7
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/reap_keep75_s1224/wandb/run-20260716_171845-clpi70s7
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run reap-math-keep75-s1224
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/clpi70s7
|
| 12 |
+
|
| 13 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 5.31B | teacher overlap=False
|
| 14 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09162069754727806, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 2.234375, "lr": 6e-06, "finish_rate": 0.907, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.5, "frames": {"chat": 236}, "mem_gb": 21.78}
|
| 15 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 16 |
+
[eval step 1] sample: "To solve this problem, we need to determine the digits \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that the given equations hold true. Let's break down the problem step-by-step:\n\n1. **Define the Variab"
|
| 17 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12219403834237406, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 2.046875, "lr": 9e-06, "finish_rate": 0.781, "comp_len": 558.1, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 215}, "mem_gb": 22.1}
|
| 18 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12023399443843713, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 2.921875, "lr": 1.2e-05, "finish_rate": 0.825, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 217}, "mem_gb": 21.98}
|
| 19 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09224013833288724, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 1.421875, "lr": 1.5e-05, "finish_rate": 0.8, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 205}, "mem_gb": 22.04}
|
| 20 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05726573241717803, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 0.85546875, "lr": 1.8e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 229}, "mem_gb": 22.01}
|
| 21 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1705658572295991, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 2.609375, "lr": 2.1e-05, "finish_rate": 0.812, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 223}, "mem_gb": 22.08}
|
| 22 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06747314393582443, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 0.875, "lr": 2.4e-05, "finish_rate": 0.708, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 202}, "mem_gb": 22.12}
|
| 23 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08693188726697117, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 1.0546875, "lr": 2.7000000000000002e-05, "finish_rate": 0.77, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 209}, "mem_gb": 22.09}
|
| 24 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.053070977600958816, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 0.5859375, "lr": 3e-05, "finish_rate": 0.885, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 227}, "mem_gb": 22.06}
|
| 25 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05504633582217308, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.54296875, "lr": 3e-05, "finish_rate": 0.848, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 230}, "mem_gb": 22.14}
|
| 26 |
+
[eval step 10] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) that satisfy all the equations simultaneously. Let's break down the problem step-by-step and"
|
| 27 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.049807043247809636, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.49609375, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 231}, "mem_gb": 21.99}
|
| 28 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.049452713058982044, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.421875, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.0, "frames": {"chat": 245}, "mem_gb": 22.06}
|
| 29 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.054898871352678784, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.484375, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 210}, "mem_gb": 22.07}
|
| 30 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.057008949374003956, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.758, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.8, "frames": {"chat": 211}, "mem_gb": 22.07}
|
| 31 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05027078033344199, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.5, "frames": {"chat": 221}, "mem_gb": 22.12}
|
| 32 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.04042372011306385, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.419921875, "lr": 3e-05, "finish_rate": 0.912, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.0, "frames": {"chat": 250}, "mem_gb": 21.94}
|
| 33 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.04448583659005041, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.0, "frames": {"chat": 229}, "mem_gb": 22.11}
|
| 34 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.033981047066890945, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.888, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.0, "frames": {"chat": 250}, "mem_gb": 22.09}
|
| 35 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.04277556057255715, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.5, "frames": {"chat": 231}, "mem_gb": 21.96}
|
| 36 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03793720846382591, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 224}, "mem_gb": 22.0}
|
| 37 |
+
[eval step 20] sample: "To solve the problem, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that the given equations are satisfied. Let's break down the problem step-by-step:\n\n1. **Define Variables"
|
| 38 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.04686265235301107, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.4296875, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 212}, "mem_gb": 22.05}
|
| 39 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.040147161142108964, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.6, "frames": {"chat": 238}, "mem_gb": 22.0}
|
| 40 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.035644251814763994, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.6, "frames": {"chat": 257}, "mem_gb": 21.88}
|
| 41 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03565622107000866, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.1, "frames": {"chat": 227}, "mem_gb": 22.07}
|
| 42 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0370076399468972, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.34375, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 228}, "mem_gb": 22.1}
|
| 43 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03920954315393853, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.9, "frames": {"chat": 233}, "mem_gb": 22.09}
|
| 44 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.036480761430505666, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 233}, "mem_gb": 22.09}
|
| 45 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.04847283381509284, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.45703125, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 609.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 197}, "mem_gb": 22.18}
|
| 46 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.04301566989741599, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 239}, "mem_gb": 21.93}
|
| 47 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.050470552899735045, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.48828125, "lr": 3e-05, "finish_rate": 0.83, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 224}, "mem_gb": 21.98}
|
| 48 |
+
[eval step 30] sample: 'To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit and satisfy the given equations:\n\n\\[\n\\begin{align*}\na +'
|
| 49 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0385393028866034, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 217}, "mem_gb": 22.09}
|
| 50 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03523599283674266, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 241}, "mem_gb": 22.09}
|
| 51 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03175364285764129, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 218}, "mem_gb": 22.07}
|
| 52 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.036605323085002604, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 206}, "mem_gb": 22.08}
|
| 53 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03494659174247645, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.5, "frames": {"chat": 232}, "mem_gb": 22.12}
|
| 54 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.039398681308484326, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.771, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 218}, "mem_gb": 22.14}
|
| 55 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.039367244467589387, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.30859375, "lr": 3e-05, "finish_rate": 0.779, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 213}, "mem_gb": 22.1}
|
| 56 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.034878414384756855, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.3046875, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 483.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.8, "frames": {"chat": 248}, "mem_gb": 22.07}
|
| 57 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.029691330024398242, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.30859375, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 218}, "mem_gb": 22.13}
|
| 58 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03176840224316499, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.851, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 221}, "mem_gb": 22.08}
|
| 59 |
+
[eval step 40] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 60 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.028261825027534118, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.894, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 236}, "mem_gb": 22.02}
|
| 61 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.029155752924651217, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.9, "frames": {"chat": 246}, "mem_gb": 21.94}
|
| 62 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03122899450864643, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.298828125, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 234}, "mem_gb": 22.19}
|
| 63 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03220423270879158, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.748, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 202}, "mem_gb": 22.07}
|
| 64 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.031760701830374695, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.1, "frames": {"chat": 217}, "mem_gb": 22.09}
|
| 65 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.02747882299865596, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.866, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.5, "frames": {"chat": 224}, "mem_gb": 22.09}
|
| 66 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03830287219239399, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.753, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.8, "frames": {"chat": 215}, "mem_gb": 22.1}
|
| 67 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.026270861617297244, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.271484375, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 463.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.1, "frames": {"chat": 259}, "mem_gb": 22.02}
|
| 68 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.02863236511334932, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.829, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.5, "frames": {"chat": 210}, "mem_gb": 22.09}
|
| 69 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03620157398320734, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.6, "frames": {"chat": 213}, "mem_gb": 22.14}
|
| 70 |
+
[eval step 50] sample: 'To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit and satisfy the given equations:\n\n\\[\n\\begin{align*}\na +'
|
| 71 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep75_s1224/step0050
|
| 72 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.02873803478294673, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.9, "frames": {"chat": 222}, "mem_gb": 22.05}
|
| 73 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.029475814702517044, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 455.7, "frames": {"chat": 235}, "mem_gb": 22.1}
|
| 74 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03140903974018681, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 456.1, "frames": {"chat": 208}, "mem_gb": 22.06}
|
| 75 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01991006130973498, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.2041015625, "lr": 3e-05, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 452.3, "frames": {"chat": 191}, "mem_gb": 22.1}
|
| 76 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017155938783214274, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.1845703125, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 445.7, "frames": {"chat": 219}, "mem_gb": 22.09}
|
| 77 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.022398129164334386, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.248046875, "lr": 3e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 480.8, "frames": {"chat": 207}, "mem_gb": 22.1}
|
| 78 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03531387661455665, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 446.3, "frames": {"chat": 208}, "mem_gb": 22.05}
|
| 79 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.019110731818852946, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.212890625, "lr": 3e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 441.8, "frames": {"chat": 219}, "mem_gb": 22.09}
|
| 80 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.020053759875210624, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.244140625, "lr": 3e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 468.8, "frames": {"chat": 246}, "mem_gb": 21.97}
|
| 81 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0288994662804141, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 460.8, "frames": {"chat": 206}, "mem_gb": 22.12}
|
| 82 |
+
[eval step 60] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 83 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.021922170959234547, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.2353515625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 427.6, "frames": {"chat": 233}, "mem_gb": 22.1}
|
| 84 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02431659371315812, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 436.1, "frames": {"chat": 229}, "mem_gb": 21.96}
|
| 85 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018058950587594883, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.2197265625, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 449.1, "frames": {"chat": 236}, "mem_gb": 22.0}
|
| 86 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.021461314609615752, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.2197265625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 433.3, "frames": {"chat": 239}, "mem_gb": 21.88}
|
| 87 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 88 |
+
warnings.warn('Grouped GEMM not available.')
|
| 89 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 90 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 91 |
+
wandb: setting up run s8ctve75
|
| 92 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 93 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/reap_keep75_s1224/wandb/run-20260716_194622-s8ctve75
|
| 94 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 95 |
+
wandb: Syncing run reap-math-keep75-s1224
|
| 96 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 97 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/s8ctve75
|
| 98 |
+
|
| 99 |
+
resumed student weights from outputs/healed/grid_math/reap_keep75_s1224/step0050 (fresh optimizer, step counter at 0)
|
| 100 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 5.31B | teacher overlap=False
|
| 101 |
+
restored optimizer/scheduler state from step 50; rebuilt 228 paged buffers
|
| 102 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.02873803478294673, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.4, "frames": {"chat": 222}, "mem_gb": 21.89}
|
| 103 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.02945255527611201, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.0, "frames": {"chat": 235}, "mem_gb": 22.1}
|
| 104 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03150169197890597, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 208}, "mem_gb": 22.06}
|
| 105 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.019934639439343784, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.203125, "lr": 3e-05, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 191}, "mem_gb": 22.1}
|
| 106 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01720574751541717, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.185546875, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 219}, "mem_gb": 22.09}
|
| 107 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.022371264208108185, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.2333984375, "lr": 3e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 207}, "mem_gb": 22.1}
|
| 108 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.035211084610790325, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 208}, "mem_gb": 22.05}
|
| 109 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.019093551559342694, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.2138671875, "lr": 3e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 219}, "mem_gb": 22.09}
|
| 110 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0200894000865131, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.240234375, "lr": 3e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.9, "frames": {"chat": 246}, "mem_gb": 21.97}
|
| 111 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.028831689852972825, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.298828125, "lr": 3e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.1, "frames": {"chat": 206}, "mem_gb": 22.12}
|
| 112 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 113 |
+
[eval step 60] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 114 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.021927540845160062, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.2373046875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.8, "frames": {"chat": 233}, "mem_gb": 22.1}
|
| 115 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.024323223485890777, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.212890625, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 229}, "mem_gb": 21.96}
|
| 116 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018064844084038245, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.2177734375, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.0, "frames": {"chat": 236}, "mem_gb": 22.0}
|
| 117 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02147197728054598, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.2197265625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.3, "frames": {"chat": 239}, "mem_gb": 21.88}
|
| 118 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01767948740927192, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.181640625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.8, "frames": {"chat": 241}, "mem_gb": 22.0}
|
| 119 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018793437677482143, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.1953125, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 226}, "mem_gb": 21.97}
|
| 120 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.016427281628238657, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.1953125, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.4, "frames": {"chat": 234}, "mem_gb": 22.1}
|
| 121 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.016445727546311295, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.3, "frames": {"chat": 257}, "mem_gb": 22.09}
|
| 122 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.024617226241024522, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.263671875, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.6, "frames": {"chat": 208}, "mem_gb": 22.14}
|
| 123 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.023580408825771882, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.224609375, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 211}, "mem_gb": 22.12}
|
| 124 |
+
[eval step 70] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 125 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.024089405320902976, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.5, "frames": {"chat": 227}, "mem_gb": 22.1}
|
| 126 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.022334026724263094, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.23046875, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 211}, "mem_gb": 22.08}
|
| 127 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.020418210263860724, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.2392578125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.6, "frames": {"chat": 238}, "mem_gb": 22.09}
|
| 128 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017777149594963217, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.1953125, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.3, "frames": {"chat": 237}, "mem_gb": 22.13}
|
| 129 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02333683302376497, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.224609375, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.4, "frames": {"chat": 208}, "mem_gb": 22.13}
|
| 130 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017551681468969522, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 221}, "mem_gb": 22.22}
|
| 131 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.019310074327540738, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.216796875, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.8, "frames": {"chat": 232}, "mem_gb": 22.06}
|
| 132 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.020564663168624975, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.2109375, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 58.0, "frames": {"chat": 208}, "mem_gb": 22.09}
|
| 133 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.015736952224222476, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.1923828125, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 58.5, "frames": {"chat": 227}, "mem_gb": 22.01}
|
| 134 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018951591900084167, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.201171875, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 58.2, "frames": {"chat": 221}, "mem_gb": 22.04}
|
| 135 |
+
[eval step 80] sample: "To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit and the given equations are satisfied.\n\nLet's break dow"
|
| 136 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.016817548004522297, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 59.1, "frames": {"chat": 232}, "mem_gb": 22.1}
|
| 137 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02058507934139731, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.2119140625, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 57.7, "frames": {"chat": 219}, "mem_gb": 22.1}
|
| 138 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01943913425094603, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.1923828125, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.3, "frames": {"chat": 195}, "mem_gb": 22.19}
|
| 139 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.019781266236553588, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.2080078125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 57.3, "frames": {"chat": 216}, "mem_gb": 22.1}
|
| 140 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.023071246190101372, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.228515625, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.8, "frames": {"chat": 208}, "mem_gb": 21.98}
|
| 141 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01572861527125351, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.173828125, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 58.7, "frames": {"chat": 235}, "mem_gb": 21.98}
|
| 142 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0185933650049769, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 57.2, "frames": {"chat": 225}, "mem_gb": 22.09}
|
| 143 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02186365541140549, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.220703125, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 444.3, "frames": {"chat": 213}, "mem_gb": 22.18}
|
| 144 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.016543877450578535, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.197265625, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 459.2, "frames": {"chat": 257}, "mem_gb": 21.85}
|
| 145 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.023722296302152487, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.236328125, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 439.4, "frames": {"chat": 212}, "mem_gb": 22.12}
|
| 146 |
+
[eval step 90] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 147 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.019525157788489015, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 453.0, "frames": {"chat": 221}, "mem_gb": 22.1}
|
| 148 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.016246650055229354, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.1806640625, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 448.9, "frames": {"chat": 242}, "mem_gb": 22.09}
|
| 149 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 150 |
+
warnings.warn('Grouped GEMM not available.')
|
| 151 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 152 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 153 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 154 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/reap_keep75_s1224/wandb/run-20260716_210030-f1ktljdw
|
| 155 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 156 |
+
wandb: Syncing run reap-math-keep75-s1224
|
| 157 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 158 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/f1ktljdw
|
| 159 |
+
|
| 160 |
+
resumed student weights from outputs/healed/grid_math/reap_keep75_s1224/step0050 (fresh optimizer, step counter at 0)
|
| 161 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 5.31B | teacher overlap=False
|
| 162 |
+
restored optimizer/scheduler state from step 50; rebuilt 228 paged buffers
|
| 163 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.02873803478294673, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 60.6, "frames": {"chat": 222}, "mem_gb": 21.89}
|
| 164 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.029428821796993725, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.31640625, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 235}, "mem_gb": 22.1}
|
| 165 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.03151381067850938, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.294921875, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 208}, "mem_gb": 22.06}
|
| 166 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01996984334345131, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.2021484375, "lr": 3e-05, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 191}, "mem_gb": 22.1}
|
| 167 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01718251699108708, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 219}, "mem_gb": 22.09}
|
| 168 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.022302498650659495, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.234375, "lr": 3e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 207}, "mem_gb": 22.1}
|
| 169 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03536568613767158, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 208}, "mem_gb": 22.05}
|
| 170 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01909338028304434, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.21484375, "lr": 3e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 219}, "mem_gb": 22.09}
|
| 171 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.020003728629935845, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.2392578125, "lr": 3e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.9, "frames": {"chat": 246}, "mem_gb": 21.97}
|
| 172 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02891256563877687, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.1, "frames": {"chat": 206}, "mem_gb": 22.12}
|
| 173 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 174 |
+
[eval step 60] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 175 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02196115782572112, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.234375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.8, "frames": {"chat": 233}, "mem_gb": 22.1}
|
| 176 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02430143336264882, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.4, "frames": {"chat": 229}, "mem_gb": 21.96}
|
| 177 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018058023730665444, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.220703125, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.9, "frames": {"chat": 236}, "mem_gb": 22.0}
|
| 178 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.021555279517305705, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.2216796875, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.2, "frames": {"chat": 239}, "mem_gb": 21.88}
|
| 179 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017681766412151047, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.181640625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.7, "frames": {"chat": 241}, "mem_gb": 22.0}
|
| 180 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0187774019391664, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.1943359375, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.7, "frames": {"chat": 226}, "mem_gb": 21.97}
|
| 181 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0163815077082254, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.2, "frames": {"chat": 234}, "mem_gb": 22.1}
|
| 182 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01645927847517499, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.1982421875, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.1, "frames": {"chat": 257}, "mem_gb": 22.09}
|
| 183 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0245373041816211, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.2451171875, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.2, "frames": {"chat": 208}, "mem_gb": 22.14}
|
| 184 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.023614521142126373, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.2275390625, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.6, "frames": {"chat": 211}, "mem_gb": 22.12}
|
| 185 |
+
[eval step 70] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 186 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.024144808137284902, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.0, "frames": {"chat": 227}, "mem_gb": 22.1}
|
| 187 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02236333406943207, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.23046875, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.7, "frames": {"chat": 211}, "mem_gb": 22.08}
|
| 188 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.020404849486495368, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.2353515625, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.4, "frames": {"chat": 238}, "mem_gb": 22.09}
|
| 189 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017783689643566806, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.1943359375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 57.3, "frames": {"chat": 237}, "mem_gb": 22.13}
|
| 190 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02329465467867752, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.2236328125, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.6, "frames": {"chat": 208}, "mem_gb": 22.13}
|
| 191 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017478233100264334, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.1826171875, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.3, "frames": {"chat": 221}, "mem_gb": 22.22}
|
| 192 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01932025155350178, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.21875, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.9, "frames": {"chat": 232}, "mem_gb": 22.06}
|
| 193 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.020524110600926602, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.20703125, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.2, "frames": {"chat": 208}, "mem_gb": 22.09}
|
| 194 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.015765587536404686, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.189453125, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.7, "frames": {"chat": 227}, "mem_gb": 22.01}
|
| 195 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018989490155184953, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.2060546875, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.3, "frames": {"chat": 221}, "mem_gb": 22.04}
|
| 196 |
+
[eval step 80] sample: "To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit and the given equations are satisfied.\n\nLet's break dow"
|
| 197 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.016829694970479855, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.1845703125, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.1, "frames": {"chat": 232}, "mem_gb": 22.1}
|
| 198 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02062391430619949, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.2138671875, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.0, "frames": {"chat": 219}, "mem_gb": 22.1}
|
| 199 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01944762978713649, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.19140625, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.5, "frames": {"chat": 195}, "mem_gb": 22.19}
|
| 200 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.019793485126597808, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.1, "frames": {"chat": 216}, "mem_gb": 22.1}
|
| 201 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.023139762130150728, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.2275390625, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.0, "frames": {"chat": 208}, "mem_gb": 21.98}
|
| 202 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.015751537270572345, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.1748046875, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.5, "frames": {"chat": 235}, "mem_gb": 21.98}
|
| 203 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018503405291447415, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.0, "frames": {"chat": 225}, "mem_gb": 22.09}
|
| 204 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02186763721885315, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.220703125, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.0, "frames": {"chat": 213}, "mem_gb": 22.18}
|
| 205 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01643971200119704, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.8, "frames": {"chat": 257}, "mem_gb": 21.85}
|
| 206 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.023723872152770248, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.236328125, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.5, "frames": {"chat": 212}, "mem_gb": 22.12}
|
| 207 |
+
[eval step 90] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 208 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.019468736032908784, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.2470703125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.6, "frames": {"chat": 221}, "mem_gb": 22.1}
|
| 209 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.016274857111802946, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.1767578125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.4, "frames": {"chat": 242}, "mem_gb": 22.09}
|
| 210 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017215291073658347, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.203125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.3, "frames": {"chat": 220}, "mem_gb": 22.06}
|
| 211 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.015166709664070126, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.7, "frames": {"chat": 240}, "mem_gb": 21.95}
|
| 212 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017237517472457452, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.7, "frames": {"chat": 206}, "mem_gb": 22.08}
|
| 213 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.024369822064802673, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.228515625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.4, "frames": {"chat": 226}, "mem_gb": 22.1}
|
| 214 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03152263613910569, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.8, "frames": {"chat": 244}, "mem_gb": 21.88}
|
| 215 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02141589429558565, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.232421875, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.6, "frames": {"chat": 224}, "mem_gb": 22.1}
|
| 216 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.016397225668475342, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.1943359375, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 57.8, "frames": {"chat": 271}, "mem_gb": 21.82}
|
| 217 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.01929302072028319, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.1, "frames": {"chat": 236}, "mem_gb": 22.11}
|
| 218 |
+
[eval step 100] sample: "To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that the given equations hold true. Let's break down the problem step-by-step:\n\n1. **Define the Vari"
|
| 219 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep75_s1224/step0100
|
| 220 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018019333380716852, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.2041015625, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 232}, "mem_gb": 21.98}
|
| 221 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.019176917587368128, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.2197265625, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 210}, "mem_gb": 22.03}
|
| 222 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.018526647040364334, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.2197265625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 217}, "mem_gb": 22.0}
|
| 223 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017947978142862364, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.19921875, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 224}, "mem_gb": 22.12}
|
| 224 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.021901778530251857, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.23828125, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 203}, "mem_gb": 21.97}
|
| 225 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.017482746841735206, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 239}, "mem_gb": 22.07}
|
| 226 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010781270754368355, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.142578125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.7, "frames": {"chat": 254}, "mem_gb": 21.98}
|
| 227 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012419780704270427, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.171875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 241}, "mem_gb": 22.07}
|
| 228 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.015673370206107696, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.166015625, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 213}, "mem_gb": 22.1}
|
| 229 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.014789011545067964, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.2119140625, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 221}, "mem_gb": 22.15}
|
| 230 |
+
[eval step 110] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 231 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.015781028156393827, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.1943359375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 196}, "mem_gb": 22.11}
|
| 232 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.010667995535299027, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.1279296875, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.5, "frames": {"chat": 270}, "mem_gb": 21.91}
|
| 233 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012989928167418112, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.1572265625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 216}, "mem_gb": 22.09}
|
| 234 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013924577494691281, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.1513671875, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 200}, "mem_gb": 22.06}
|
| 235 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013322410390495013, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.1572265625, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 206}, "mem_gb": 22.01}
|
| 236 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011119994657541004, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.1396484375, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 234}, "mem_gb": 22.04}
|
| 237 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012775608483022855, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.1416015625, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 215}, "mem_gb": 22.05}
|
| 238 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013810691532116228, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.173828125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.2, "frames": {"chat": 255}, "mem_gb": 22.04}
|
| 239 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012154009584007629, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.1611328125, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 250}, "mem_gb": 21.92}
|
| 240 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011215298393421108, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.140625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 242}, "mem_gb": 22.09}
|
| 241 |
+
[eval step 120] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 242 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01326268654340723, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.14453125, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.5, "frames": {"chat": 199}, "mem_gb": 22.1}
|
| 243 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.018925391417974606, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.1865234375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.8, "frames": {"chat": 208}, "mem_gb": 22.13}
|
| 244 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01369877224289424, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.2001953125, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.2, "frames": {"chat": 208}, "mem_gb": 22.07}
|
| 245 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01566584520537872, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.8, "frames": {"chat": 209}, "mem_gb": 22.22}
|
| 246 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011354163011299292, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.1435546875, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.1, "frames": {"chat": 235}, "mem_gb": 22.05}
|
| 247 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013031885967287235, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.1728515625, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 204}, "mem_gb": 22.04}
|
| 248 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.016365782146974622, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.20703125, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.7, "frames": {"chat": 208}, "mem_gb": 22.1}
|
| 249 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012403703387636536, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.158203125, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.8, "frames": {"chat": 240}, "mem_gb": 22.1}
|
| 250 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01273732632562751, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.1630859375, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 246}, "mem_gb": 22.09}
|
| 251 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012520373641039865, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.1396484375, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 243}, "mem_gb": 21.91}
|
| 252 |
+
[eval step 130] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 253 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01572654613229679, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.15625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 208}, "mem_gb": 22.11}
|
| 254 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.017281583469605538, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.171875, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.9, "frames": {"chat": 219}, "mem_gb": 22.1}
|
| 255 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.015536821740671681, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.173828125, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 211}, "mem_gb": 22.11}
|
| 256 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.016642906039534135, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.2080078125, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 232}, "mem_gb": 22.07}
|
| 257 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.019927484658709728, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.1, "frames": {"chat": 214}, "mem_gb": 22.1}
|
| 258 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012803300328789433, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.146484375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.6, "frames": {"chat": 226}, "mem_gb": 21.99}
|
| 259 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012066151755136282, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.1474609375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.7, "frames": {"chat": 210}, "mem_gb": 22.11}
|
| 260 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012860532379080542, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.1591796875, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.7, "frames": {"chat": 218}, "mem_gb": 21.93}
|
| 261 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012094843646103982, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.1396484375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.0, "frames": {"chat": 233}, "mem_gb": 22.08}
|
| 262 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.015558591982846459, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.169921875, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.4, "frames": {"chat": 215}, "mem_gb": 22.1}
|
| 263 |
+
[eval step 140] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 264 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012454204010121369, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.146484375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.0, "frames": {"chat": 233}, "mem_gb": 22.09}
|
| 265 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013477057802421042, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.2138671875, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 209}, "mem_gb": 22.04}
|
| 266 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.009968187428083426, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.123046875, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.8, "frames": {"chat": 262}, "mem_gb": 21.97}
|
| 267 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.013083611388411373, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.15234375, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.6, "frames": {"chat": 249}, "mem_gb": 22.06}
|
| 268 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.014526255074034756, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.1748046875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.5, "frames": {"chat": 227}, "mem_gb": 22.09}
|
| 269 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011909497406838152, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.14453125, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.1, "frames": {"chat": 221}, "mem_gb": 22.09}
|
| 270 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012006994549774875, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.1533203125, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.1, "frames": {"chat": 234}, "mem_gb": 22.11}
|
| 271 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.012652689880512965, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.177734375, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 213}, "mem_gb": 22.05}
|
| 272 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.011978251870415018, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.1513671875, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 213}, "mem_gb": 21.99}
|
| 273 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01043063024947187, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.13671875, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.7, "frames": {"chat": 234}, "mem_gb": 22.02}
|
| 274 |
+
[eval step 150] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-by-step"
|
| 275 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/reap_keep75_s1224/step0150
|
| 276 |
+
wandb: updating run metadata
|
| 277 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 278 |
+
wandb:
|
| 279 |
+
wandb: Run history:
|
| 280 |
+
wandb: comp_len β
ββββββββ
βββ
βββ
βββββββββββββ
ββββββββββββ
|
| 281 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββββ
β
β
β
β
β
β
βββββββββββββ
|
| 282 |
+
wandb: epoch ββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
βββββββββββββββ
|
| 283 |
+
wandb: finish_rate βββ
ββββββββββ
β
βββββ
βββββββ
ββββββββββββββ
|
| 284 |
+
wandb: forward_topk_kl βββ
β
β
βββββ
ββ
ββββββββββββ
ββββββββββββββββ
|
| 285 |
+
wandb: grad_norm ββββ
β
ββββββ
βββββ
βββββββββ
βββββββββββββββ
|
| 286 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 287 |
+
wandb: mem_gb ββββββββββββββ
βββββββ
ββββββ
β
ββββββββββββ
|
| 288 |
+
wandb: step ββββββββββββββββ
β
β
β
β
β
β
β
β
ββββββββββββββββ
|
| 289 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 290 |
+
wandb: +3 ...
|
| 291 |
+
wandb:
|
| 292 |
+
wandb: Run summary:
|
| 293 |
+
wandb: comp_len 512.8
|
| 294 |
+
wandb: cumulative_loss_tokens 18000000
|
| 295 |
+
wandb: epoch 2
|
| 296 |
+
wandb: finish_rate 0.906
|
| 297 |
+
wandb: forward_topk_kl 0.01043
|
| 298 |
+
wandb: grad_norm 0.13672
|
| 299 |
+
wandb: lr 3e-05
|
| 300 |
+
wandb: mem_gb 22.02
|
| 301 |
+
wandb: step 150
|
| 302 |
+
wandb: t_data_s 0
|
| 303 |
+
wandb: +4 ...
|
| 304 |
+
wandb:
|
| 305 |
+
wandb: π View run reap-math-keep75-s1224 at: https://wandb.ai/hbfreed/glean-grid/runs/f1ktljdw
|
| 306 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 307 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 308 |
+
wandb: Find logs at: outputs/healed/grid_math/reap_keep75_s1224/wandb/run-20260716_210030-f1ktljdw/logs
|
| 309 |
+
{
|
| 310 |
+
"correct": 894,
|
| 311 |
+
"accuracy": 0.6777862016679302,
|
| 312 |
+
"finished": 1315,
|
| 313 |
+
"finish_rate": 0.9969673995451099,
|
| 314 |
+
"mean_completion_tokens": 112.61410159211523
|
| 315 |
+
}
|
| 316 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep75_s1224_step100_chat.json
|
| 317 |
+
{
|
| 318 |
+
"correct": 903,
|
| 319 |
+
"accuracy": 0.6846095526914329,
|
| 320 |
+
"finished": 1316,
|
| 321 |
+
"finish_rate": 0.9977255496588324,
|
| 322 |
+
"mean_completion_tokens": 111.83548142532221
|
| 323 |
+
}
|
| 324 |
+
saved item-level results -> outputs/evals/grid_math/reap_keep75_s1224_step150_chat.json
|
healed/grid_math/uniform_keep25_s1224.console.log
ADDED
|
@@ -0,0 +1,231 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run rm3hkgkc
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/uniform_keep25_s1224/wandb/run-20260716_054248-rm3hkgkc
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run uniform-math-keep25-s1224
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/rm3hkgkc
|
| 12 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 2.09B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.2459722065463663, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 14.625, "lr": 6e-06, "finish_rate": 0.907, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.6, "frames": {"chat": 236}, "mem_gb": 9.78}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: 'The value of $p$ is \\(\\box{5}\\).'
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.3142343253110846, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 15.375, "lr": 9e-06, "finish_rate": 0.781, "comp_len": 558.1, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 27.7, "frames": {"chat": 215}, "mem_gb": 10.01}
|
| 17 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.3116438292915622, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 13.9375, "lr": 1.2e-05, "finish_rate": 0.825, "comp_len": 553.0, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 30.0, "frames": {"chat": 217}, "mem_gb": 9.88}
|
| 18 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.1271550920541087, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 11.1875, "lr": 1.5e-05, "finish_rate": 0.8, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.7, "frames": {"chat": 205}, "mem_gb": 9.94}
|
| 19 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.8921521788805723, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 6.40625, "lr": 1.8e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 229}, "mem_gb": 9.92}
|
| 20 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.8736880722011129, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 4.8125, "lr": 2.1e-05, "finish_rate": 0.812, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 223}, "mem_gb": 9.98}
|
| 21 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7521487558588386, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 3.453125, "lr": 2.4e-05, "finish_rate": 0.708, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 202}, "mem_gb": 10.02}
|
| 22 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7010229941956699, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 3.1875, "lr": 2.7000000000000002e-05, "finish_rate": 0.77, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 209}, "mem_gb": 9.99}
|
| 23 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6002694400727749, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 2.453125, "lr": 3e-05, "finish_rate": 0.885, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 227}, "mem_gb": 9.97}
|
| 24 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5602013669068615, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 1.8203125, "lr": 3e-05, "finish_rate": 0.848, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 230}, "mem_gb": 10.05}
|
| 25 |
+
[eval step 10] sample: "To solve the problem, we need to determine the value of \\(p\\) given the following equations:\n\n\\[\na + b = k \\\\\nk + m = p \\\\\np + a = r \\\\\nb + m + r = 18\n\\]\n\nLet's break down the problem step-"
|
| 26 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5372615832733612, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 1.546875, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 231}, "mem_gb": 9.89}
|
| 27 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5131440531161924, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 1.328125, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.8, "frames": {"chat": 245}, "mem_gb": 9.97}
|
| 28 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.47872232480570676, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 210}, "mem_gb": 9.98}
|
| 29 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.537974064292262, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 1.2578125, "lr": 3e-05, "finish_rate": 0.758, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.0, "frames": {"chat": 211}, "mem_gb": 9.98}
|
| 30 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4437455804655949, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.9296875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 221}, "mem_gb": 10.03}
|
| 31 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4362137271172057, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.984375, "lr": 3e-05, "finish_rate": 0.912, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.1, "frames": {"chat": 250}, "mem_gb": 9.84}
|
| 32 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4411819472976029, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.8984375, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.6, "frames": {"chat": 229}, "mem_gb": 10.02}
|
| 33 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.37396339893067876, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.83984375, "lr": 3e-05, "finish_rate": 0.888, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 250}, "mem_gb": 9.99}
|
| 34 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.40561543378531933, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.79296875, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 231}, "mem_gb": 9.87}
|
| 35 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4002509487184385, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.75390625, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 224}, "mem_gb": 9.91}
|
| 36 |
+
[eval step 20] sample: "To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(r\\). Let's break down the problem step-by-step:\n\n1. **Define Variables:**\n - Let \\(a\\) be the first digit"
|
| 37 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.36296452349474034, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.80078125, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.4, "frames": {"chat": 212}, "mem_gb": 9.95}
|
| 38 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.34317847680710256, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.75390625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 238}, "mem_gb": 9.91}
|
| 39 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.364331378469492, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.81640625, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.8, "frames": {"chat": 257}, "mem_gb": 9.78}
|
| 40 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3275189952706297, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.640625, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 227}, "mem_gb": 9.97}
|
| 41 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3667670934761564, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.67578125, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 42 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3577250474138806, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.67578125, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 43 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.32106263146164515, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.671875, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 44 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.41402777843773364, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.734375, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 609.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 197}, "mem_gb": 10.08}
|
| 45 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3724903223272413, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.68359375, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 239}, "mem_gb": 9.84}
|
| 46 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.35697053606547413, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.6875, "lr": 3e-05, "finish_rate": 0.83, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 224}, "mem_gb": 9.88}
|
| 47 |
+
[eval step 30] sample: 'To solve the problem, we need to determine the value of \\( p \\) given the equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}'
|
| 48 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.31142014599790174, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.640625, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.4, "frames": {"chat": 217}, "mem_gb": 10.0}
|
| 49 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29815615977446236, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.609375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.6, "frames": {"chat": 241}, "mem_gb": 10.0}
|
| 50 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3220009417420874, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.609375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 218}, "mem_gb": 9.97}
|
| 51 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.31028348484759527, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.578125, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 206}, "mem_gb": 9.98}
|
| 52 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3256828921566407, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.69140625, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.3, "frames": {"chat": 232}, "mem_gb": 10.03}
|
| 53 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3209710501977553, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.6171875, "lr": 3e-05, "finish_rate": 0.771, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.0, "frames": {"chat": 218}, "mem_gb": 10.05}
|
| 54 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3221005113951862, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.5703125, "lr": 3e-05, "finish_rate": 0.779, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.0, "frames": {"chat": 213}, "mem_gb": 10.01}
|
| 55 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.32981064673662186, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.59765625, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 483.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 248}, "mem_gb": 9.98}
|
| 56 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.30937803875269987, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.6171875, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 218}, "mem_gb": 10.04}
|
| 57 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2680865074227254, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.54296875, "lr": 3e-05, "finish_rate": 0.851, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 221}, "mem_gb": 9.99}
|
| 58 |
+
[eval step 40] sample: 'To solve the system of equations given by:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe need to determine the value of'
|
| 59 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29091000881840784, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.5546875, "lr": 3e-05, "finish_rate": 0.894, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 236}, "mem_gb": 9.92}
|
| 60 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29280417794175445, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.5625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.0, "frames": {"chat": 246}, "mem_gb": 9.84}
|
| 61 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2934912766356021, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.58203125, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.1, "frames": {"chat": 234}, "mem_gb": 10.1}
|
| 62 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.23609974780020615, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.55859375, "lr": 3e-05, "finish_rate": 0.748, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.0, "frames": {"chat": 202}, "mem_gb": 9.98}
|
| 63 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.26752842073862754, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.498046875, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.7, "frames": {"chat": 217}, "mem_gb": 9.99}
|
| 64 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28959610045862694, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.58203125, "lr": 3e-05, "finish_rate": 0.866, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 224}, "mem_gb": 9.99}
|
| 65 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3118727909699082, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.753, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 215}, "mem_gb": 10.01}
|
| 66 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.26413657319024203, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.5625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 463.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.7, "frames": {"chat": 259}, "mem_gb": 9.93}
|
| 67 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29531795247793197, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.829, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 210}, "mem_gb": 9.99}
|
| 68 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.32368664257427054, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.6171875, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.1, "frames": {"chat": 213}, "mem_gb": 10.05}
|
| 69 |
+
[eval step 50] sample: 'To solve the system of equations given by:\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\nwe need to determine the value of \\(p'
|
| 70 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep25_s1224/step0050
|
| 71 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28201613237416995, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.546875, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.7, "frames": {"chat": 222}, "mem_gb": 9.95}
|
| 72 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.24275245352555067, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 235}, "mem_gb": 10.01}
|
| 73 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.27474053088929506, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.6640625, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 208}, "mem_gb": 9.96}
|
| 74 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.27640588794089854, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 1.671875, "lr": 3e-05, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.6, "frames": {"chat": 191}, "mem_gb": 10.0}
|
| 75 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2337344744838774, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.55859375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 76 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19981578330968816, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 207}, "mem_gb": 10.0}
|
| 77 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2742745844999949, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.578125, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 208}, "mem_gb": 9.96}
|
| 78 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2056781067028021, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.462890625, "lr": 3e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 79 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22058189589908966, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.6, "frames": {"chat": 246}, "mem_gb": 9.87}
|
| 80 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2643624147929872, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.5625, "lr": 3e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 206}, "mem_gb": 10.02}
|
| 81 |
+
[eval step 60] sample: 'To solve the system of equations given by:\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\nwe need to find the value of \\(p'
|
| 82 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21315796338009338, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.546875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.8, "frames": {"chat": 233}, "mem_gb": 10.01}
|
| 83 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23010029078846175, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.52734375, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.5, "frames": {"chat": 229}, "mem_gb": 9.87}
|
| 84 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18971112331363063, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 236}, "mem_gb": 9.9}
|
| 85 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22939992027953268, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 239}, "mem_gb": 9.79}
|
| 86 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19626577739318213, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 241}, "mem_gb": 9.91}
|
| 87 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2318629670790086, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 226}, "mem_gb": 9.87}
|
| 88 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21305266705161582, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.478515625, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.7, "frames": {"chat": 234}, "mem_gb": 10.0}
|
| 89 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2066091762378812, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 257}, "mem_gb": 9.99}
|
| 90 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2726578070071836, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.2, "frames": {"chat": 208}, "mem_gb": 10.05}
|
| 91 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22639581966890643, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.6, "frames": {"chat": 211}, "mem_gb": 10.02}
|
| 92 |
+
[eval step 70] sample: 'To solve the system of equations given by:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe need to determine the value of'
|
| 93 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2497449489728858, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.58203125, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 227}, "mem_gb": 10.01}
|
| 94 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23184565024909873, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.478515625, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.1, "frames": {"chat": 211}, "mem_gb": 9.98}
|
| 95 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19929239737944057, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.46875, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.6, "frames": {"chat": 238}, "mem_gb": 10.0}
|
| 96 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21444108126734693, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.0, "frames": {"chat": 237}, "mem_gb": 10.04}
|
| 97 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.228507239554512, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 208}, "mem_gb": 10.04}
|
| 98 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22201900441398223, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.482421875, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 221}, "mem_gb": 10.12}
|
| 99 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2176680883942172, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.515625, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 232}, "mem_gb": 9.96}
|
| 100 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20230479391242068, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.435546875, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.4, "frames": {"chat": 208}, "mem_gb": 10.0}
|
| 101 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20621175042502582, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 227}, "mem_gb": 9.91}
|
| 102 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23712326520072918, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.5234375, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 221}, "mem_gb": 9.94}
|
| 103 |
+
[eval step 80] sample: 'To solve the system of linear equations given by:\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\nwe need to determine the value of \\('
|
| 104 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18516172153086713, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.4, "frames": {"chat": 232}, "mem_gb": 10.01}
|
| 105 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19767518214893837, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.8, "frames": {"chat": 219}, "mem_gb": 10.01}
|
| 106 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20814777844653776, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 195}, "mem_gb": 10.1}
|
| 107 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21465807686001062, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.470703125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.9, "frames": {"chat": 216}, "mem_gb": 10.0}
|
| 108 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19052286817779143, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.796875, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.6, "frames": {"chat": 208}, "mem_gb": 9.89}
|
| 109 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20568204772435128, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 235}, "mem_gb": 9.88}
|
| 110 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2174682028145219, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 225}, "mem_gb": 9.99}
|
| 111 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.25289742040882507, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.2, "frames": {"chat": 213}, "mem_gb": 10.08}
|
| 112 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18773433045589674, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.44921875, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.5, "frames": {"chat": 257}, "mem_gb": 9.76}
|
| 113 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21751305311309796, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.3, "frames": {"chat": 212}, "mem_gb": 10.03}
|
| 114 |
+
[eval step 90] sample: "To solve this system of equations, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) that satisfy all the given equations. Let's break down the problem step-by-step:\n\n1. "
|
| 115 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19270047811983773, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.6, "frames": {"chat": 221}, "mem_gb": 10.0}
|
| 116 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19187464828671266, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 242}, "mem_gb": 10.0}
|
| 117 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16515993953794242, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.4140625, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.4, "frames": {"chat": 220}, "mem_gb": 9.96}
|
| 118 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19365566024774064, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 240}, "mem_gb": 9.86}
|
| 119 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19843267694873115, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.443359375, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 206}, "mem_gb": 9.99}
|
| 120 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1955687409762914, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.2, "frames": {"chat": 226}, "mem_gb": 10.0}
|
| 121 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23913774186559023, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 244}, "mem_gb": 9.79}
|
| 122 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.24111862684388954, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.0, "frames": {"chat": 224}, "mem_gb": 10.01}
|
| 123 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20014771827943623, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.46875, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.5, "frames": {"chat": 271}, "mem_gb": 9.73}
|
| 124 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18671405559678872, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.470703125, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 236}, "mem_gb": 10.01}
|
| 125 |
+
[eval step 100] sample: 'To solve the system of equations given by:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe need to find the values of'
|
| 126 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep25_s1224/step0100
|
| 127 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20005840366501362, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.2, "frames": {"chat": 232}, "mem_gb": 9.88}
|
| 128 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17052629617406057, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.3, "frames": {"chat": 210}, "mem_gb": 9.94}
|
| 129 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16335184906447928, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.2, "frames": {"chat": 217}, "mem_gb": 9.91}
|
| 130 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21374459005383153, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 224}, "mem_gb": 10.03}
|
| 131 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.24060933916370075, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.54296875, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.2, "frames": {"chat": 203}, "mem_gb": 9.87}
|
| 132 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2103607431170841, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 239}, "mem_gb": 9.97}
|
| 133 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16764343557308117, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.53125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.1, "frames": {"chat": 254}, "mem_gb": 9.88}
|
| 134 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14560104954248915, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.5, "frames": {"chat": 241}, "mem_gb": 9.98}
|
| 135 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.21455759129527335, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.494140625, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.5, "frames": {"chat": 213}, "mem_gb": 10.01}
|
| 136 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17589531892972687, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 221}, "mem_gb": 10.05}
|
| 137 |
+
[eval step 110] sample: 'To solve the system of linear equations given by:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe need to determine the values'
|
| 138 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.18606396657917648, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.3, "frames": {"chat": 196}, "mem_gb": 10.01}
|
| 139 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1608481796350951, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.447265625, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.8, "frames": {"chat": 270}, "mem_gb": 9.82}
|
| 140 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14370967580489813, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.4453125, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.8, "frames": {"chat": 216}, "mem_gb": 10.0}
|
| 141 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.18907757144179196, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.5, "frames": {"chat": 200}, "mem_gb": 9.96}
|
| 142 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1409036660529673, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.3, "frames": {"chat": 206}, "mem_gb": 9.91}
|
| 143 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15345207378479342, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 234}, "mem_gb": 9.95}
|
| 144 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1820456429388995, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.45703125, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.0, "frames": {"chat": 215}, "mem_gb": 9.96}
|
| 145 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1530056454622497, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.9, "frames": {"chat": 255}, "mem_gb": 9.94}
|
| 146 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16267353230559578, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.9, "frames": {"chat": 250}, "mem_gb": 9.82}
|
| 147 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16486497175091255, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 242}, "mem_gb": 10.0}
|
| 148 |
+
[eval step 120] sample: 'To solve the system of equations given by:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe need to determine the values of'
|
| 149 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.19459200162775814, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.494140625, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 150 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.20826055347186823, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.515625, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.4, "frames": {"chat": 208}, "mem_gb": 10.04}
|
| 151 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1550956260437922, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.3, "frames": {"chat": 208}, "mem_gb": 9.97}
|
| 152 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17796036245978128, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.5, "frames": {"chat": 209}, "mem_gb": 10.12}
|
| 153 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14426678808629512, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.5, "frames": {"chat": 235}, "mem_gb": 9.96}
|
| 154 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1457931994455556, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 204}, "mem_gb": 9.95}
|
| 155 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.19706413115250568, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.48828125, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 156 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1500166365404303, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.412109375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.1, "frames": {"chat": 240}, "mem_gb": 10.0}
|
| 157 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15483832397218794, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.455078125, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.0, "frames": {"chat": 246}, "mem_gb": 9.99}
|
| 158 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16828932282235473, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.46484375, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.8, "frames": {"chat": 243}, "mem_gb": 9.82}
|
| 159 |
+
[eval step 130] sample: 'To solve the given system of equations, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) such that:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p'
|
| 160 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.18825837670378387, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.1, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 161 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16242987946247062, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.447265625, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 162 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1970008016841486, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.46875, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.6, "frames": {"chat": 211}, "mem_gb": 10.01}
|
| 163 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14961124420731017, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.439453125, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 232}, "mem_gb": 9.98}
|
| 164 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17444652401488275, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.4453125, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 214}, "mem_gb": 10.01}
|
| 165 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15362181645246845, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.421875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.6, "frames": {"chat": 226}, "mem_gb": 9.9}
|
| 166 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15746455868557097, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 1.0859375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.3, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 167 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.131130507747146, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.39453125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.2, "frames": {"chat": 218}, "mem_gb": 9.83}
|
| 168 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13699651974520335, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 2.265625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.8, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 169 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.18451387127457808, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.498046875, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.3, "frames": {"chat": 215}, "mem_gb": 10.01}
|
| 170 |
+
[eval step 140] sample: 'To solve this problem, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) given the equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\n'
|
| 171 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.171622509614937, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.8, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 172 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1397777112826084, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.40625, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.7, "frames": {"chat": 209}, "mem_gb": 9.94}
|
| 173 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14401639284497747, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.44921875, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.4, "frames": {"chat": 262}, "mem_gb": 9.88}
|
| 174 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14294547926882903, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.484375, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 249}, "mem_gb": 9.96}
|
| 175 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1826381297783926, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.466796875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.3, "frames": {"chat": 227}, "mem_gb": 10.0}
|
| 176 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13961569585020964, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.46484375, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.3, "frames": {"chat": 221}, "mem_gb": 10.0}
|
| 177 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15722946502156557, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.412109375, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.2, "frames": {"chat": 234}, "mem_gb": 10.01}
|
| 178 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12678958315073202, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.8515625, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.1, "frames": {"chat": 213}, "mem_gb": 9.96}
|
| 179 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1364368616912514, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.5234375, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.3, "frames": {"chat": 213}, "mem_gb": 9.9}
|
| 180 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15012081333752722, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 1.0390625, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.7, "frames": {"chat": 234}, "mem_gb": 9.93}
|
| 181 |
+
[eval step 150] sample: "To solve this problem, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) such that the given equations hold true. Let's break down the problem step-by-step:\n\n1. **Understand t"
|
| 182 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep25_s1224/step0150
|
| 183 |
+
wandb: updating run metadata
|
| 184 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 185 |
+
wandb:
|
| 186 |
+
wandb: Run history:
|
| 187 |
+
wandb: comp_len ββββββ
ββ
ββ
ββββββ
βββββ
ββββ
βββββββββ
ββ
ββββ
|
| 188 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββββ
β
β
β
β
βββββββββββββββ
|
| 189 |
+
wandb: epoch βββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
βββββββββββ
|
| 190 |
+
wandb: finish_rate ββ
ββββββ
β
βββ
β
ββββββββββββββ
βββ
ββββββββ
ββ
|
| 191 |
+
wandb: forward_topk_kl βββ
βββββββββββββββββββββββββββββββββββββ
|
| 192 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: mem_gb ββ
ββββββ
βββ
βββββββ
ββββββββββββββ
β
βββββββ
|
| 195 |
+
wandb: step ββββββββββββββββββββββ
β
β
β
βββββββββββββββ
|
| 196 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 197 |
+
wandb: +3 ...
|
| 198 |
+
wandb:
|
| 199 |
+
wandb: Run summary:
|
| 200 |
+
wandb: comp_len 512.8
|
| 201 |
+
wandb: cumulative_loss_tokens 18000000
|
| 202 |
+
wandb: epoch 2
|
| 203 |
+
wandb: finish_rate 0.906
|
| 204 |
+
wandb: forward_topk_kl 0.15012
|
| 205 |
+
wandb: grad_norm 1.03906
|
| 206 |
+
wandb: lr 3e-05
|
| 207 |
+
wandb: mem_gb 9.93
|
| 208 |
+
wandb: step 150
|
| 209 |
+
wandb: t_data_s 0
|
| 210 |
+
wandb: +4 ...
|
| 211 |
+
wandb:
|
| 212 |
+
wandb: π View run uniform-math-keep25-s1224 at: https://wandb.ai/hbfreed/glean-grid/runs/rm3hkgkc
|
| 213 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 214 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 215 |
+
wandb: Find logs at: outputs/healed/grid_math/uniform_keep25_s1224/wandb/run-20260716_054248-rm3hkgkc/logs
|
| 216 |
+
{
|
| 217 |
+
"correct": 279,
|
| 218 |
+
"accuracy": 0.21152388172858225,
|
| 219 |
+
"finished": 1256,
|
| 220 |
+
"finish_rate": 0.9522365428354814,
|
| 221 |
+
"mean_completion_tokens": 158.7035633055345
|
| 222 |
+
}
|
| 223 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep25_s1224_step100_chat.json
|
| 224 |
+
{
|
| 225 |
+
"correct": 277,
|
| 226 |
+
"accuracy": 0.2100075815011372,
|
| 227 |
+
"finished": 1271,
|
| 228 |
+
"finish_rate": 0.9636087945413192,
|
| 229 |
+
"mean_completion_tokens": 150.8013646702047
|
| 230 |
+
}
|
| 231 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep25_s1224_step150_chat.json
|
healed/grid_math/uniform_keep25_s1225.console.log
ADDED
|
@@ -0,0 +1,230 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 6 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/uniform_keep25_s1225/wandb/run-20260716_051726-a8l8mpfr
|
| 7 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 8 |
+
wandb: Syncing run uniform-math-keep25-s1225
|
| 9 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 10 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/a8l8mpfr
|
| 11 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 2.09B | teacher overlap=False
|
| 12 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.319479045200348, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 14.9375, "lr": 6e-06, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.2, "t_rollout_s": 0.0, "t_step_s": 32.6, "frames": {"chat": 191}, "mem_gb": 9.94}
|
| 13 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 14 |
+
[eval step 1] sample: "To solve this problem, first few numbers have been entered into the grid below. Let's start by considering the numbers from $1$ to $49$ arranged in a spiral pattern on a square grid.\n\nThe numbers from"
|
| 15 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.250888225916028, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 14.3125, "lr": 9e-06, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.6, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 16 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.3497639717732866, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 14.5, "lr": 1.2e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 27.5, "frames": {"chat": 207}, "mem_gb": 10.0}
|
| 17 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.0802153058772286, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 9.3125, "lr": 1.5e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.0, "frames": {"chat": 208}, "mem_gb": 9.96}
|
| 18 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.9386221677641073, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 6.65625, "lr": 1.8e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 19 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.8062813847805063, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 4.40625, "lr": 2.1e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 246}, "mem_gb": 9.87}
|
| 20 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7548754439207415, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 3.484375, "lr": 2.4e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 206}, "mem_gb": 10.02}
|
| 21 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6572285013991097, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 3.203125, "lr": 2.7000000000000002e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.3, "frames": {"chat": 233}, "mem_gb": 10.01}
|
| 22 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6164511016746362, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 2.546875, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 229}, "mem_gb": 9.87}
|
| 23 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5470844178607066, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 1.953125, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 236}, "mem_gb": 9.9}
|
| 24 |
+
[eval step 10] sample: "To solve this problem, we need to determine which of the four numbers appear in the shaded squares on the grid.\n\nLet's break down the problem:\n\n1. **Identify the Shaded Squares:**\n The grid is a squ"
|
| 25 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5313714265652001, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 1.40625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.0, "frames": {"chat": 239}, "mem_gb": 9.79}
|
| 26 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4681813031862179, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 1.1953125, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 241}, "mem_gb": 9.91}
|
| 27 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.49307297485172746, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 1.1171875, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 226}, "mem_gb": 9.87}
|
| 28 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.44446369409287967, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.97265625, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 234}, "mem_gb": 10.0}
|
| 29 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4230342352161805, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.90234375, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 257}, "mem_gb": 9.99}
|
| 30 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5012797900413474, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.99609375, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 208}, "mem_gb": 10.05}
|
| 31 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4470518503196538, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.90625, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 211}, "mem_gb": 10.02}
|
| 32 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.41256588681464396, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.83203125, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 227}, "mem_gb": 10.01}
|
| 33 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.42249245348448555, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.77734375, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 211}, "mem_gb": 9.98}
|
| 34 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.36867867480044564, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.72265625, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.8, "frames": {"chat": 238}, "mem_gb": 10.0}
|
| 35 |
+
[eval step 20] sample: "To solve this problem, we need to analyze the spiral pattern of the numbers from 1 to 49 and determine which four numbers appear in the shaded squares on the same diagonal as the number 7. We'll then "
|
| 36 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.38083410925380884, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.67578125, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.8, "frames": {"chat": 237}, "mem_gb": 10.04}
|
| 37 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.39549448682889343, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.765625, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.2, "frames": {"chat": 208}, "mem_gb": 10.04}
|
| 38 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3769813402572026, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.67578125, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 221}, "mem_gb": 10.12}
|
| 39 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.35552544313979645, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 3.53125, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.0, "frames": {"chat": 232}, "mem_gb": 9.96}
|
| 40 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3550635836930325, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.6640625, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 208}, "mem_gb": 10.0}
|
| 41 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.33844906397598484, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.64453125, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.6, "frames": {"chat": 227}, "mem_gb": 9.91}
|
| 42 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3774184838908414, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.6796875, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 221}, "mem_gb": 9.94}
|
| 43 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3146386974407981, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.9, "frames": {"chat": 232}, "mem_gb": 10.01}
|
| 44 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.31184066178612413, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.57421875, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 219}, "mem_gb": 10.01}
|
| 45 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3350962167671571, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.6015625, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.4, "frames": {"chat": 195}, "mem_gb": 10.1}
|
| 46 |
+
[eval step 30] sample: 'To solve this problem, we need to determine the number of prime numbers from 1 to 49 that appear in the shaded squares on a square grid, where the first few numbers are entered into the grid below.\n\n#'
|
| 47 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.33699155798591673, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.62109375, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.7, "frames": {"chat": 216}, "mem_gb": 10.0}
|
| 48 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2785733728256077, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.5859375, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.7, "frames": {"chat": 208}, "mem_gb": 9.89}
|
| 49 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3027569059799115, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.60546875, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 235}, "mem_gb": 9.88}
|
| 50 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3082514542732388, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.6484375, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.9, "frames": {"chat": 225}, "mem_gb": 9.99}
|
| 51 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3752551995801429, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.65625, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 213}, "mem_gb": 10.08}
|
| 52 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2921847006034106, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.85546875, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.7, "frames": {"chat": 257}, "mem_gb": 9.76}
|
| 53 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3090001767206937, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.546875, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 212}, "mem_gb": 10.03}
|
| 54 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28328445645694933, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 1.796875, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 221}, "mem_gb": 10.0}
|
| 55 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2893523990919193, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.5703125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 242}, "mem_gb": 10.0}
|
| 56 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.255670898507225, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.5546875, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 220}, "mem_gb": 9.96}
|
| 57 |
+
[eval step 40] sample: 'To solve this problem, we need to analyze the pattern of the numbers from 1 to 49 arranged on a square grid and identify the four numbers that appear in the shaded squares on the same diagonal as the '
|
| 58 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2903485574451586, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.58203125, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.6, "frames": {"chat": 240}, "mem_gb": 9.86}
|
| 59 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28734269070960583, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.6, "frames": {"chat": 206}, "mem_gb": 9.99}
|
| 60 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.27243882406105596, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.52734375, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.6, "frames": {"chat": 226}, "mem_gb": 10.0}
|
| 61 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3234697731980433, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.609375, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 244}, "mem_gb": 9.79}
|
| 62 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3212854475179066, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.6015625, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 224}, "mem_gb": 10.01}
|
| 63 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28673740869238973, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.55859375, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.5, "frames": {"chat": 271}, "mem_gb": 9.73}
|
| 64 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2532580826393639, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 1.125, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 236}, "mem_gb": 10.01}
|
| 65 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28962034405469894, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 232}, "mem_gb": 9.88}
|
| 66 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.23193988344911487, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.4921875, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 210}, "mem_gb": 9.94}
|
| 67 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.22433082110987354, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.90625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.9, "frames": {"chat": 217}, "mem_gb": 9.91}
|
| 68 |
+
[eval step 50] sample: 'To solve this problem, we need to analyze the pattern of the numbers from 1 to 49 arranged on a square grid and identify the four numbers that appear in the shaded squares, each appearing on the same '
|
| 69 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep25_s1225/step0050
|
| 70 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2995344866629069, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.6171875, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 224}, "mem_gb": 10.03}
|
| 71 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.32907882408003014, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.640625, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.9, "frames": {"chat": 203}, "mem_gb": 9.87}
|
| 72 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28704522055424747, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 2.4375, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.9, "frames": {"chat": 239}, "mem_gb": 9.97}
|
| 73 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23595582261749853, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.57421875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.3, "frames": {"chat": 254}, "mem_gb": 9.88}
|
| 74 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21762469755336641, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.5234375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 241}, "mem_gb": 9.98}
|
| 75 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2846538535839568, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.5703125, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 213}, "mem_gb": 10.01}
|
| 76 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.245288224790742, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.482421875, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 221}, "mem_gb": 10.05}
|
| 77 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2521051613455638, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 1.8046875, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.2, "frames": {"chat": 196}, "mem_gb": 10.01}
|
| 78 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23354704158020517, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 32.0, "frames": {"chat": 270}, "mem_gb": 9.82}
|
| 79 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19478359519572308, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 216}, "mem_gb": 10.0}
|
| 80 |
+
[eval step 60] sample: 'To solve this problem, we need to analyze the pattern of the numbers from 1 to 49 arranged on a square grid and identify the four numbers that appear in the shaded squares, each appearing on the same '
|
| 81 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.26863892504523196, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.6484375, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 26.8, "frames": {"chat": 200}, "mem_gb": 9.96}
|
| 82 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18590027238409967, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.83984375, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.7, "frames": {"chat": 206}, "mem_gb": 9.91}
|
| 83 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2012578780563548, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.474609375, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 234}, "mem_gb": 9.95}
|
| 84 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.247304932789132, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.515625, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.3, "frames": {"chat": 215}, "mem_gb": 9.96}
|
| 85 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20205045403062055, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.71484375, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.3, "frames": {"chat": 255}, "mem_gb": 9.94}
|
| 86 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22750451257377863, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 1.609375, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.2, "frames": {"chat": 250}, "mem_gb": 9.82}
|
| 87 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22274810297029715, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.494140625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 242}, "mem_gb": 10.0}
|
| 88 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.26898174686903753, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 3.96875, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 89 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2861793573357165, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.6171875, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 208}, "mem_gb": 10.04}
|
| 90 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22676605374064918, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 1.1484375, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.4, "frames": {"chat": 208}, "mem_gb": 9.97}
|
| 91 |
+
[eval step 70] sample: 'To solve this problem, we need to analyze the spiral pattern of the numbers from 1 to 49 and identify the four numbers that appear in the shaded squares on the same diagonal as the number 7. We will t'
|
| 92 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2383458670858294, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 209}, "mem_gb": 10.12}
|
| 93 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2040254034437239, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.8, "frames": {"chat": 235}, "mem_gb": 9.96}
|
| 94 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18941694195649275, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.490234375, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.5, "frames": {"chat": 204}, "mem_gb": 9.95}
|
| 95 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.26322059602600834, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.58984375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 96 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20454769928430516, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 1.46875, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.4, "frames": {"chat": 240}, "mem_gb": 10.0}
|
| 97 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22102658995253344, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 246}, "mem_gb": 9.99}
|
| 98 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22660920705248913, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.5234375, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 243}, "mem_gb": 9.82}
|
| 99 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2539451661850015, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.6, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 100 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20485700886597236, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 101 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.254067884727257, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.6953125, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.2, "frames": {"chat": 211}, "mem_gb": 10.01}
|
| 102 |
+
[eval step 80] sample: 'To solve this problem, we need to analyze the spiral pattern of the numbers from 1 to 49 and determine which of the four numbers that appear in the shaded squares on the same diagonal as the number 7 '
|
| 103 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1860976280965532, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.478515625, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 232}, "mem_gb": 9.98}
|
| 104 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21269370155887057, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.49609375, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 214}, "mem_gb": 10.01}
|
| 105 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2044230012240509, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.55859375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.9, "frames": {"chat": 226}, "mem_gb": 9.9}
|
| 106 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2102600726918007, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.466796875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 107 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17095083842401704, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.439453125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 218}, "mem_gb": 9.83}
|
| 108 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18065360787672302, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.7, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 109 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23056014474630357, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 215}, "mem_gb": 10.01}
|
| 110 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22403676771732667, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 111 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1838335989182194, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.0, "frames": {"chat": 209}, "mem_gb": 9.94}
|
| 112 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19289504188615827, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.7, "frames": {"chat": 262}, "mem_gb": 9.88}
|
| 113 |
+
[eval step 90] sample: "To solve this problem, we need to identify the four numbers that appear in the shaded squares on the same diagonal as the number 7, and then determine how many of these numbers are prime.\n\nHere's the "
|
| 114 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1809839127952854, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.42578125, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.0, "frames": {"chat": 249}, "mem_gb": 9.96}
|
| 115 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2376722942231844, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.482421875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.4, "frames": {"chat": 227}, "mem_gb": 10.0}
|
| 116 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.186381211121846, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 221}, "mem_gb": 10.0}
|
| 117 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20532299918190886, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.45703125, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 234}, "mem_gb": 10.01}
|
| 118 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1591838776408384, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.4296875, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.4, "frames": {"chat": 213}, "mem_gb": 9.96}
|
| 119 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16577480874514827, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.6, "frames": {"chat": 213}, "mem_gb": 9.9}
|
| 120 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20456980733238161, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.9, "frames": {"chat": 234}, "mem_gb": 9.93}
|
| 121 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20402369832278539, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.482421875, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 222}, "mem_gb": 10.0}
|
| 122 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21510515817857037, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.5, "frames": {"chat": 227}, "mem_gb": 10.01}
|
| 123 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19600455205359807, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 218}, "mem_gb": 10.04}
|
| 124 |
+
[eval step 100] sample: 'To solve this problem, we need to:\n\n1. Identify the numbers from 1 to 49 that are arranged in a spiral pattern on a square grid.\n2. Determine which of these numbers are on the same diagonal as the num'
|
| 125 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep25_s1225/step0100
|
| 126 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2290196806839357, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 223}, "mem_gb": 10.01}
|
| 127 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22636315025078754, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.9, "frames": {"chat": 206}, "mem_gb": 10.01}
|
| 128 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20191379800935585, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.0, "frames": {"chat": 213}, "mem_gb": 9.92}
|
| 129 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2503080016883711, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.2, "frames": {"chat": 223}, "mem_gb": 9.86}
|
| 130 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21158150815398744, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.482421875, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 227}, "mem_gb": 9.97}
|
| 131 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2013531628333653, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.3, "frames": {"chat": 253}, "mem_gb": 10.0}
|
| 132 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.21000758766736835, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.9, "frames": {"chat": 216}, "mem_gb": 10.01}
|
| 133 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.19588753996851543, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.6, "frames": {"chat": 205}, "mem_gb": 9.98}
|
| 134 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.21358148629758508, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.3, "frames": {"chat": 207}, "mem_gb": 10.06}
|
| 135 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.19724183553326874, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.46875, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.7, "frames": {"chat": 215}, "mem_gb": 9.98}
|
| 136 |
+
[eval step 110] sample: 'To solve this problem, we need to analyze the spiral pattern of the numbers from 1 to 49 and identify the four numbers that appear on the same diagonal as the number 7. We then determine which of thes'
|
| 137 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14767051264047623, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.435546875, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 138 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.19400057307276874, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.474609375, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 221}, "mem_gb": 10.04}
|
| 139 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14793867993044357, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.44921875, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 254}, "mem_gb": 9.84}
|
| 140 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12694251759614175, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.455078125, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.9, "frames": {"chat": 210}, "mem_gb": 9.97}
|
| 141 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.154613794028759, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 226}, "mem_gb": 9.92}
|
| 142 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15858659546474616, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.40625, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 212}, "mem_gb": 9.99}
|
| 143 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17931291315760464, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.4, "frames": {"chat": 211}, "mem_gb": 9.93}
|
| 144 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1916892997973909, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.45703125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 26.6, "frames": {"chat": 196}, "mem_gb": 9.98}
|
| 145 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13504183224166433, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.388671875, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.0, "frames": {"chat": 212}, "mem_gb": 10.0}
|
| 146 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14792166811867308, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 244}, "mem_gb": 9.91}
|
| 147 |
+
[eval step 120] sample: 'To solve this problem, we need to:\n\n1. Identify the numbers from 1 to 49 arranged in a spiral pattern.\n2. Determine which of these numbers appear on the same diagonal as the number 7.\n3. Count how man'
|
| 148 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14660488209525743, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 222}, "mem_gb": 9.95}
|
| 149 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17171216915504386, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 218}, "mem_gb": 10.0}
|
| 150 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17044704577376446, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.9, "frames": {"chat": 252}, "mem_gb": 9.88}
|
| 151 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.18156986663484326, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.58203125, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.7, "frames": {"chat": 202}, "mem_gb": 10.05}
|
| 152 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.193661573850736, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.484375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.1, "frames": {"chat": 237}, "mem_gb": 10.0}
|
| 153 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1703850445672249, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.4, "frames": {"chat": 234}, "mem_gb": 9.99}
|
| 154 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12615562903275712, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.39453125, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.6, "frames": {"chat": 215}, "mem_gb": 10.0}
|
| 155 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1354192927910636, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 1.0859375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 234}, "mem_gb": 9.93}
|
| 156 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.12099138240994264, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.7, "frames": {"chat": 216}, "mem_gb": 9.99}
|
| 157 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14705536963663374, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.419921875, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.4, "frames": {"chat": 210}, "mem_gb": 9.95}
|
| 158 |
+
[eval step 130] sample: 'To solve this problem, we need to analyze the spiral pattern of the numbers from 1 to 49 arranged on a square grid and identify the four numbers that appear on the same diagonal as the number 7. We th'
|
| 159 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1616287318826032, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.412109375, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.4, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 160 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14949506662531445, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.0, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 161 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15277357547314216, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.4921875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 225}, "mem_gb": 9.95}
|
| 162 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15180517331815013, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.40625, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.0, "frames": {"chat": 253}, "mem_gb": 9.85}
|
| 163 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1629465045961241, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.3, "frames": {"chat": 247}, "mem_gb": 9.98}
|
| 164 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17035012210526815, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.4453125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.3, "frames": {"chat": 238}, "mem_gb": 9.98}
|
| 165 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.19480876584922274, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.7, "frames": {"chat": 235}, "mem_gb": 10.0}
|
| 166 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.2049359349505355, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.58203125, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.7, "frames": {"chat": 215}, "mem_gb": 9.97}
|
| 167 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15441036838572472, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.4453125, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.1, "frames": {"chat": 268}, "mem_gb": 9.97}
|
| 168 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17245760981744776, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.466796875, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 169 |
+
[eval step 140] sample: 'To solve this problem, we need to analyze the spiral pattern of the numbers from 1 to 49 arranged on a square grid and identify the four numbers that appear in the shaded squares, each appearing on th'
|
| 170 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1604549546021968, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 252}, "mem_gb": 9.93}
|
| 171 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15443554332548132, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.44921875, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.8, "frames": {"chat": 223}, "mem_gb": 10.01}
|
| 172 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.19541836336503426, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.3, "frames": {"chat": 226}, "mem_gb": 10.0}
|
| 173 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.20092230602238947, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.5703125, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.5, "frames": {"chat": 208}, "mem_gb": 10.05}
|
| 174 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1302992971648462, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.3984375, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.4, "frames": {"chat": 240}, "mem_gb": 9.93}
|
| 175 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16997946729871133, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.4296875, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 222}, "mem_gb": 9.93}
|
| 176 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14522260747464993, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.6, "frames": {"chat": 236}, "mem_gb": 10.0}
|
| 177 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13322943811810886, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.419921875, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.0, "frames": {"chat": 217}, "mem_gb": 9.97}
|
| 178 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15352233099521448, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.53125, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.8, "frames": {"chat": 252}, "mem_gb": 9.88}
|
| 179 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14089195124438653, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.3828125, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 222}, "mem_gb": 9.99}
|
| 180 |
+
[eval step 150] sample: 'To solve this problem, we need to:\n\n1. Identify the numbers from 1 to 49 arranged in a spiral pattern.\n2. Determine the positions of the numbers on the same diagonal as the number 7.\n3. Count how many'
|
| 181 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep25_s1225/step0150
|
| 182 |
+
wandb: updating run metadata
|
| 183 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 184 |
+
wandb:
|
| 185 |
+
wandb: Run history:
|
| 186 |
+
wandb: comp_len ββββββ
ββ
βββββββββββββββ
ββ
β
β
ββββ
ββββ
βββββ
|
| 187 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββββ
β
β
β
β
β
β
β
ββββββββββββ
|
| 188 |
+
wandb: epoch ββββββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
βββββββββββ
|
| 189 |
+
wandb: finish_rate ββ
ββββ
ββ
β
ββββββββ
βββ
ββ
ββ
βββββ
ββ
βββββ
ββ
ββ
|
| 190 |
+
wandb: forward_topk_kl βββ
βββββββββββββββββββββββββββββββββββββ
|
| 191 |
+
wandb: grad_norm ββ
ββββββββββββββββββββββββββββββββββββββ
|
| 192 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: mem_gb β
βββββββ
ββββββββ
β
βββ
βββββββββββ
βββ
ββ
ββββ
|
| 194 |
+
wandb: step ββββββββββββββββββββββ
β
β
β
β
ββββββββββββββ
|
| 195 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 196 |
+
wandb: +3 ...
|
| 197 |
+
wandb:
|
| 198 |
+
wandb: Run summary:
|
| 199 |
+
wandb: comp_len 540.5
|
| 200 |
+
wandb: cumulative_loss_tokens 18000000
|
| 201 |
+
wandb: epoch 2
|
| 202 |
+
wandb: finish_rate 0.847
|
| 203 |
+
wandb: forward_topk_kl 0.14089
|
| 204 |
+
wandb: grad_norm 0.38281
|
| 205 |
+
wandb: lr 3e-05
|
| 206 |
+
wandb: mem_gb 9.99
|
| 207 |
+
wandb: step 150
|
| 208 |
+
wandb: t_data_s 0
|
| 209 |
+
wandb: +4 ...
|
| 210 |
+
wandb:
|
| 211 |
+
wandb: π View run uniform-math-keep25-s1225 at: https://wandb.ai/hbfreed/glean-grid/runs/a8l8mpfr
|
| 212 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 213 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 214 |
+
wandb: Find logs at: outputs/healed/grid_math/uniform_keep25_s1225/wandb/run-20260716_051726-a8l8mpfr/logs
|
| 215 |
+
{
|
| 216 |
+
"correct": 267,
|
| 217 |
+
"accuracy": 0.20242608036391205,
|
| 218 |
+
"finished": 1270,
|
| 219 |
+
"finish_rate": 0.9628506444275967,
|
| 220 |
+
"mean_completion_tokens": 150.4215314632297
|
| 221 |
+
}
|
| 222 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep25_s1225_step100_chat.json
|
| 223 |
+
{
|
| 224 |
+
"correct": 291,
|
| 225 |
+
"accuracy": 0.22062168309325247,
|
| 226 |
+
"finished": 1272,
|
| 227 |
+
"finish_rate": 0.9643669446550417,
|
| 228 |
+
"mean_completion_tokens": 148.76952236542834
|
| 229 |
+
}
|
| 230 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep25_s1225_step150_chat.json
|
healed/grid_math/uniform_keep25_s1226.console.log
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run nzko6jqq
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/uniform_keep25_s1226/wandb/run-20260716_051619-nzko6jqq
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run uniform-math-keep25-s1226
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/nzko6jqq
|
| 12 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 2.09B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.2495578979462385, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 14.5625, "lr": 6e-06, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.2, "t_rollout_s": 0.0, "t_step_s": 36.8, "frames": {"chat": 254}, "mem_gb": 9.82}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: "The perimeter of a triangle is given as 28, and the midpoints of its sides are connected by segments. Let's denote the sides of the triangle as follows: a, b, and c. The midpoints of the sides are con"
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.2842795796026787, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 14.25, "lr": 9e-06, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 29.4, "frames": {"chat": 241}, "mem_gb": 9.98}
|
| 17 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.2860378812308113, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 13.3125, "lr": 1.2e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.7, "frames": {"chat": 213}, "mem_gb": 10.01}
|
| 18 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.0296244965081414, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 8.625, "lr": 1.5e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 221}, "mem_gb": 10.05}
|
| 19 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 1.0173526370272041, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 6.9375, "lr": 1.8e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.0, "frames": {"chat": 196}, "mem_gb": 10.01}
|
| 20 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7652708411070208, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 4.1875, "lr": 2.1e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.7, "frames": {"chat": 270}, "mem_gb": 9.82}
|
| 21 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7675839649726948, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 3.734375, "lr": 2.4e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.0, "frames": {"chat": 216}, "mem_gb": 10.0}
|
| 22 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.7070816349441806, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 3.28125, "lr": 2.7000000000000002e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 26.7, "frames": {"chat": 200}, "mem_gb": 9.96}
|
| 23 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.6169270259405176, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 2.75, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.5, "frames": {"chat": 206}, "mem_gb": 9.91}
|
| 24 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5323196953435739, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 1.9609375, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 234}, "mem_gb": 9.95}
|
| 25 |
+
[eval step 10] sample: "To find the perimeter of a triangle formed by connecting the midpoints of its sides, we need to determine the length of each of the three sides of the triangle.\n\nLet's break down the problem step-by-s"
|
| 26 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.5534217075144251, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 1.640625, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.2, "frames": {"chat": 215}, "mem_gb": 9.96}
|
| 27 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.470129220243295, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 1.1953125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 255}, "mem_gb": 9.94}
|
| 28 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4660646371759474, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 1.09375, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.0, "frames": {"chat": 250}, "mem_gb": 9.82}
|
| 29 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4519369126672546, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 1.0234375, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 242}, "mem_gb": 10.0}
|
| 30 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.48208508322536947, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.97265625, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.1, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 31 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.505666505916665, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.98046875, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.4, "frames": {"chat": 208}, "mem_gb": 10.04}
|
| 32 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.42821578803832333, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 1.0078125, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.3, "frames": {"chat": 208}, "mem_gb": 9.97}
|
| 33 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4488865850756566, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.8359375, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 209}, "mem_gb": 10.12}
|
| 34 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3713087936893105, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 12.4375, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 235}, "mem_gb": 9.96}
|
| 35 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3715890112740298, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.80078125, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.4, "frames": {"chat": 204}, "mem_gb": 9.95}
|
| 36 |
+
[eval step 20] sample: 'To solve this problem, we need to understand the geometry of the triangle and the properties of its midpoints. The perimeter of a triangle is given by the sum of its sides, and the midpoints of its si'
|
| 37 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.44148291802903017, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.9375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 38 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.36412479583832125, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.80859375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.2, "frames": {"chat": 240}, "mem_gb": 10.0}
|
| 39 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.35719255141379935, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.69921875, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 246}, "mem_gb": 9.99}
|
| 40 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3621041040317466, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.67578125, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 243}, "mem_gb": 9.82}
|
| 41 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.38283104023163517, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.68359375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.5, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 42 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3491992857553065, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.67578125, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.1, "frames": {"chat": 219}, "mem_gb": 10.0}
|
| 43 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3949156841669232, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.80859375, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 211}, "mem_gb": 10.01}
|
| 44 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3246795868317286, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.69140625, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 232}, "mem_gb": 9.98}
|
| 45 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3552298259820789, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.73046875, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.1, "frames": {"chat": 214}, "mem_gb": 10.01}
|
| 46 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3321069305093338, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.65625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.7, "frames": {"chat": 226}, "mem_gb": 9.9}
|
| 47 |
+
[eval step 30] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and the connections between its midpoints.\n\n1. **Understand the Geometry:**\n - The perimeter of a triangle is gi'
|
| 48 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.32228971752213936, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.64453125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 49 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2798056041251868, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.58203125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 218}, "mem_gb": 9.83}
|
| 50 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29189739949926735, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.61328125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 51 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.35060337477376063, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.64453125, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.8, "frames": {"chat": 215}, "mem_gb": 10.01}
|
| 52 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.33954473176573713, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.640625, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 233}, "mem_gb": 9.99}
|
| 53 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2843651156159739, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 1.828125, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.9, "frames": {"chat": 209}, "mem_gb": 9.94}
|
| 54 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2947497258403649, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.72265625, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.6, "frames": {"chat": 262}, "mem_gb": 9.88}
|
| 55 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28674538816077016, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 249}, "mem_gb": 9.96}
|
| 56 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.33914187191314993, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 1.4921875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.3, "frames": {"chat": 227}, "mem_gb": 10.0}
|
| 57 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.26380810491734497, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.5234375, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 221}, "mem_gb": 10.0}
|
| 58 |
+
[eval step 40] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and the connections between its midpoints.\n\n1. **Understand the Geometry:**\n - The perimeter \\( P \\) of a triang'
|
| 59 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.302417300863564, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.5546875, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 234}, "mem_gb": 10.01}
|
| 60 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.24419383710200587, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.52734375, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.3, "frames": {"chat": 213}, "mem_gb": 9.96}
|
| 61 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.24537141441050916, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.5, "frames": {"chat": 213}, "mem_gb": 9.9}
|
| 62 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.27504100236129014, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.53125, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 234}, "mem_gb": 9.93}
|
| 63 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2705650013284758, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 222}, "mem_gb": 10.0}
|
| 64 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.31124431811695297, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.59375, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 31.3, "frames": {"chat": 227}, "mem_gb": 10.01}
|
| 65 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2672547916886707, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 218}, "mem_gb": 10.04}
|
| 66 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3164061588189254, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.57421875, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 223}, "mem_gb": 10.01}
|
| 67 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.31090367152442533, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.8, "frames": {"chat": 206}, "mem_gb": 10.01}
|
| 68 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2863034582992395, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.5625, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.9, "frames": {"chat": 213}, "mem_gb": 9.92}
|
| 69 |
+
[eval step 50] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and the connections between its midpoints.\n\n1. **Understand the Perimeter:**\n The perimeter \\( P \\) of a triangl'
|
| 70 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep25_s1226/step0050
|
| 71 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.33238482810370623, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.5859375, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 30.3, "frames": {"chat": 223}, "mem_gb": 9.86}
|
| 72 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.30147026825944584, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.5703125, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 227}, "mem_gb": 9.97}
|
| 73 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2694238721400499, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 1.2109375, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.2, "frames": {"chat": 253}, "mem_gb": 10.0}
|
| 74 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2599986071868489, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.609375, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.7, "frames": {"chat": 216}, "mem_gb": 10.01}
|
| 75 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.26703055294280253, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.53125, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.4, "frames": {"chat": 205}, "mem_gb": 9.98}
|
| 76 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2845016695648432, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.6328125, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.1, "frames": {"chat": 207}, "mem_gb": 10.06}
|
| 77 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2710578287235151, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 215}, "mem_gb": 9.98}
|
| 78 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2025449087051054, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.6, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 79 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2637373869329691, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.5625, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.7, "frames": {"chat": 221}, "mem_gb": 10.04}
|
| 80 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20766283342484385, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.7, "frames": {"chat": 254}, "mem_gb": 9.84}
|
| 81 |
+
[eval step 60] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and the connections between its midpoints.\n\n1. **Understand the Geometry:**\n - Let the sides of the triangle be '
|
| 82 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17684483035219212, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.7, "frames": {"chat": 210}, "mem_gb": 9.97}
|
| 83 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21151079111825674, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 226}, "mem_gb": 9.92}
|
| 84 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22429327983123562, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.1, "frames": {"chat": 212}, "mem_gb": 9.99}
|
| 85 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23901226744391024, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.4921875, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.3, "frames": {"chat": 211}, "mem_gb": 9.93}
|
| 86 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2521503586698324, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.53125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 26.5, "frames": {"chat": 196}, "mem_gb": 9.98}
|
| 87 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19568856958678613, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.447265625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.8, "frames": {"chat": 212}, "mem_gb": 10.0}
|
| 88 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2056147424393023, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.447265625, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 244}, "mem_gb": 9.91}
|
| 89 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.19970010116541137, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.435546875, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 222}, "mem_gb": 9.95}
|
| 90 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2406689625217269, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.1, "frames": {"chat": 218}, "mem_gb": 10.0}
|
| 91 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2351806751595189, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.494140625, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.7, "frames": {"chat": 252}, "mem_gb": 9.88}
|
| 92 |
+
[eval step 70] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and the connections between its midpoints.\n\n1. **Understand the Geometry:**\n - Let the sides of the triangle be '
|
| 93 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23980661761003236, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.5, "frames": {"chat": 202}, "mem_gb": 10.05}
|
| 94 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2756894440931578, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.59765625, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 237}, "mem_gb": 10.0}
|
| 95 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22606769043393432, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.3, "frames": {"chat": 234}, "mem_gb": 9.99}
|
| 96 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17098218618429575, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.2, "frames": {"chat": 215}, "mem_gb": 10.0}
|
| 97 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18496631520005563, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.4375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.7, "frames": {"chat": 234}, "mem_gb": 9.93}
|
| 98 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.16087721765795723, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.6, "frames": {"chat": 216}, "mem_gb": 9.99}
|
| 99 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1994732022792101, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.5, "frames": {"chat": 210}, "mem_gb": 9.95}
|
| 100 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20232079331736702, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.443359375, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.2, "frames": {"chat": 199}, "mem_gb": 10.0}
|
| 101 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18569014142416418, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.9, "frames": {"chat": 210}, "mem_gb": 10.01}
|
| 102 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20507156935961296, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.1, "frames": {"chat": 225}, "mem_gb": 9.95}
|
| 103 |
+
[eval step 80] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and the connections between its midpoints.\n\n1. **Understand the Problem:**\n - We are given the perimeter of the '
|
| 104 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20882732380144298, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 253}, "mem_gb": 9.85}
|
| 105 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22202617851061127, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.2, "frames": {"chat": 247}, "mem_gb": 9.98}
|
| 106 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2211942642432948, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 238}, "mem_gb": 9.98}
|
| 107 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23084418179870894, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.6, "frames": {"chat": 235}, "mem_gb": 10.0}
|
| 108 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2283516202347974, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.498046875, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 215}, "mem_gb": 9.97}
|
| 109 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22033791828608762, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.0, "frames": {"chat": 268}, "mem_gb": 9.97}
|
| 110 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22125197298427424, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.486328125, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 228}, "mem_gb": 10.0}
|
| 111 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21613482079487295, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.49609375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.8, "frames": {"chat": 252}, "mem_gb": 9.93}
|
| 112 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1895898001347358, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.46484375, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.6, "frames": {"chat": 223}, "mem_gb": 10.01}
|
| 113 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.26797097403767206, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.2, "frames": {"chat": 226}, "mem_gb": 10.0}
|
| 114 |
+
[eval step 90] sample: 'To solve this problem, we need to follow these steps:\n\n1. **Understand the Problem:**\n - We are given the perimeter of a triangle, which is 28.\n - The midpoints of the sides of the triangle are co'
|
| 115 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.24367389039595921, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.4, "frames": {"chat": 208}, "mem_gb": 10.05}
|
| 116 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17548983993784836, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.4296875, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.2, "frames": {"chat": 240}, "mem_gb": 9.93}
|
| 117 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21608840946884206, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 222}, "mem_gb": 9.93}
|
| 118 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1903983515452904, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.455078125, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 236}, "mem_gb": 10.0}
|
| 119 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17124010507706552, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.8, "frames": {"chat": 217}, "mem_gb": 9.97}
|
| 120 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21378996573444456, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.59375, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 252}, "mem_gb": 9.88}
|
| 121 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.17583670812180888, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.1, "frames": {"chat": 222}, "mem_gb": 9.99}
|
| 122 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2008310172341764, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.470703125, "lr": 3e-05, "finish_rate": 0.901, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 242}, "mem_gb": 9.87}
|
| 123 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.2504596530464788, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.7, "frames": {"chat": 219}, "mem_gb": 9.93}
|
| 124 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20776792142900327, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.44140625, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.8, "frames": {"chat": 223}, "mem_gb": 9.94}
|
| 125 |
+
[eval step 100] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and the connections between its midpoints.\n\n1. **Understand the Problem:**\n - We are given the perimeter of the '
|
| 126 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep25_s1226/step0100
|
| 127 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.21407153125032782, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 232}, "mem_gb": 9.95}
|
| 128 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.22760161069128662, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.53125, "lr": 3e-05, "finish_rate": 0.832, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.2, "frames": {"chat": 220}, "mem_gb": 10.0}
|
| 129 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.23536558933630586, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.6, "frames": {"chat": 210}, "mem_gb": 10.04}
|
| 130 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.20387262995596975, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.482421875, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.6, "frames": {"chat": 226}, "mem_gb": 9.97}
|
| 131 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1928112811392794, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.466796875, "lr": 3e-05, "finish_rate": 0.741, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.8, "frames": {"chat": 212}, "mem_gb": 10.0}
|
| 132 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.18794621144427606, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.48828125, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 236}, "mem_gb": 10.01}
|
| 133 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16627150794851284, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.928, "comp_len": 454.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.7, "frames": {"chat": 264}, "mem_gb": 9.88}
|
| 134 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.21027346796846638, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 229}, "mem_gb": 9.98}
|
| 135 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16645300974740337, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.5078125, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 465.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 258}, "mem_gb": 9.85}
|
| 136 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.21918750831211606, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.55859375, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 137 |
+
[eval step 110] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and the connections between its midpoints.\n\n1. **Understand the Geometry:**\n - Let the sides of the triangle be '
|
| 138 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.18296561656640842, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.88, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.4, "frames": {"chat": 249}, "mem_gb": 9.93}
|
| 139 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14481682858659575, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.5, "frames": {"chat": 220}, "mem_gb": 9.99}
|
| 140 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1859534092683966, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.56640625, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.8, "frames": {"chat": 223}, "mem_gb": 9.99}
|
| 141 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13867307858555578, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.1, "frames": {"chat": 221}, "mem_gb": 10.0}
|
| 142 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15031419390533118, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 230}, "mem_gb": 9.9}
|
| 143 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17688426749395827, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 214}, "mem_gb": 9.97}
|
| 144 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.2198834235218664, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.7, "frames": {"chat": 214}, "mem_gb": 9.99}
|
| 145 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1655972873053203, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.478515625, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 210}, "mem_gb": 10.04}
|
| 146 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.18060686992689345, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.45703125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.2, "frames": {"chat": 214}, "mem_gb": 10.0}
|
| 147 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.18515299578898897, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.791, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 215}, "mem_gb": 9.96}
|
| 148 |
+
[eval step 120] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - We have a triangle with sides \\(a\\), \\(b\\), and \\("
|
| 149 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.19961161344274878, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.4921875, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.4, "frames": {"chat": 208}, "mem_gb": 9.99}
|
| 150 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.150687341630583, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.789, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.4, "frames": {"chat": 218}, "mem_gb": 9.88}
|
| 151 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1382546880559375, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.388671875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.7, "frames": {"chat": 233}, "mem_gb": 9.9}
|
| 152 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13145577659346164, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.388671875, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.8, "frames": {"chat": 231}, "mem_gb": 9.94}
|
| 153 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16622866188141827, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.435546875, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.8, "frames": {"chat": 235}, "mem_gb": 10.13}
|
| 154 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17218660681688538, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.419921875, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.1, "frames": {"chat": 216}, "mem_gb": 9.99}
|
| 155 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16202461753959457, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.831, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.1, "frames": {"chat": 237}, "mem_gb": 10.01}
|
| 156 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1946208799743404, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.46484375, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.9, "frames": {"chat": 203}, "mem_gb": 10.01}
|
| 157 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16476607479564845, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.873, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.5, "frames": {"chat": 236}, "mem_gb": 10.04}
|
| 158 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16959726118591303, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.42578125, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.0, "frames": {"chat": 214}, "mem_gb": 10.01}
|
| 159 |
+
[eval step 130] sample: 'To solve this problem, we need to understand the geometric properties of the triangle and the connections between its midpoints.\n\n1. **Understanding the Problem:**\n - We are given the perimeter of a'
|
| 160 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15791527282614262, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.421875, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.7, "frames": {"chat": 209}, "mem_gb": 10.0}
|
| 161 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16019434139728547, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.443359375, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 468.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 30.5, "frames": {"chat": 256}, "mem_gb": 10.0}
|
| 162 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1639191758962348, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.455078125, "lr": 3e-05, "finish_rate": 0.878, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.0, "frames": {"chat": 230}, "mem_gb": 9.87}
|
| 163 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16772623434948425, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.439453125, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.3, "frames": {"chat": 230}, "mem_gb": 10.06}
|
| 164 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1819166350507488, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.44921875, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.6, "frames": {"chat": 227}, "mem_gb": 9.95}
|
| 165 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.1837459068759655, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.455078125, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.7, "frames": {"chat": 208}, "mem_gb": 10.01}
|
| 166 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.19119155149323244, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.474609375, "lr": 3e-05, "finish_rate": 0.699, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.9, "frames": {"chat": 206}, "mem_gb": 10.03}
|
| 167 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17735388877652586, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.48046875, "lr": 3e-05, "finish_rate": 0.82, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 228}, "mem_gb": 9.9}
|
| 168 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17459255363605916, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.462890625, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.5, "frames": {"chat": 224}, "mem_gb": 10.0}
|
| 169 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15427039775537948, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.408203125, "lr": 3e-05, "finish_rate": 0.66, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.1, "frames": {"chat": 200}, "mem_gb": 10.03}
|
| 170 |
+
[eval step 140] sample: "To solve this problem, we need to understand the geometric properties involved. Let's break down the problem step-by-step:\n\n1. **Understand the Geometry:**\n - A triangle has three sides, say \\(a\\), "
|
| 171 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.17032653220028926, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.45703125, "lr": 3e-05, "finish_rate": 0.714, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 26.8, "frames": {"chat": 196}, "mem_gb": 10.01}
|
| 172 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14531434423498188, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 223}, "mem_gb": 10.0}
|
| 173 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13336851223480578, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.869, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.4, "frames": {"chat": 213}, "mem_gb": 9.89}
|
| 174 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14154418901971852, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.9, "frames": {"chat": 232}, "mem_gb": 9.93}
|
| 175 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15019642852085333, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.447265625, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 223}, "mem_gb": 9.93}
|
| 176 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.15932164447531105, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.421875, "lr": 3e-05, "finish_rate": 0.85, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.7, "frames": {"chat": 233}, "mem_gb": 10.02}
|
| 177 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.16258757215073952, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.816, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 27.9, "frames": {"chat": 217}, "mem_gb": 10.01}
|
| 178 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.21068223652498175, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.515625, "lr": 3e-05, "finish_rate": 0.752, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.3, "frames": {"chat": 202}, "mem_gb": 10.08}
|
| 179 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.14553914751367023, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.4453125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 29.9, "frames": {"chat": 253}, "mem_gb": 9.93}
|
| 180 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.13818583363940318, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 28.6, "frames": {"chat": 231}, "mem_gb": 9.94}
|
| 181 |
+
[eval step 150] sample: "To solve this problem, we need to understand the geometric properties involved. Let's break down the problem into manageable steps:\n\n1. **Understand the Geometry:**\n - A triangle has a given perimet"
|
| 182 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep25_s1226/step0150
|
| 183 |
+
wandb: updating run metadata
|
| 184 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 185 |
+
wandb: uploading history steps 149-149, summary, console lines 168-170
|
| 186 |
+
wandb:
|
| 187 |
+
wandb: Run history:
|
| 188 |
+
wandb: comp_len β
βββββββββββ
βββ
ββββββ
βββ
β
βββββββββββββ
ββ
|
| 189 |
+
wandb: cumulative_loss_tokens ββββββββββββββββββ
β
β
β
β
β
β
β
βββββββββββββββ
|
| 190 |
+
wandb: epoch ββββββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββββββββββ
|
| 191 |
+
wandb: finish_rate ββββββ
β
ββββ
β
βββββ
βββ
βββββββββββββ
ββ
βββββ
|
| 192 |
+
wandb: forward_topk_kl ββ
ββββββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 195 |
+
wandb: mem_gb βββββββββββββββ
ββββββ
β
βββββ
βββββ
β
βββββββ
|
| 196 |
+
wandb: step βββββββββββββββββββββ
β
β
β
β
βββββββββββββββ
|
| 197 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 198 |
+
wandb: +3 ...
|
| 199 |
+
wandb:
|
| 200 |
+
wandb: Run summary:
|
| 201 |
+
wandb: comp_len 519.5
|
| 202 |
+
wandb: cumulative_loss_tokens 18000000
|
| 203 |
+
wandb: epoch 2
|
| 204 |
+
wandb: finish_rate 0.879
|
| 205 |
+
wandb: forward_topk_kl 0.13819
|
| 206 |
+
wandb: grad_norm 0.39648
|
| 207 |
+
wandb: lr 3e-05
|
| 208 |
+
wandb: mem_gb 9.94
|
| 209 |
+
wandb: step 150
|
| 210 |
+
wandb: t_data_s 0
|
| 211 |
+
wandb: +4 ...
|
| 212 |
+
wandb:
|
| 213 |
+
wandb: π View run uniform-math-keep25-s1226 at: https://wandb.ai/hbfreed/glean-grid/runs/nzko6jqq
|
| 214 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 215 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 216 |
+
wandb: Find logs at: outputs/healed/grid_math/uniform_keep25_s1226/wandb/run-20260716_051619-nzko6jqq/logs
|
| 217 |
+
{
|
| 218 |
+
"correct": 264,
|
| 219 |
+
"accuracy": 0.2001516300227445,
|
| 220 |
+
"finished": 1257,
|
| 221 |
+
"finish_rate": 0.9529946929492039,
|
| 222 |
+
"mean_completion_tokens": 155.3972706595906
|
| 223 |
+
}
|
| 224 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep25_s1226_step100_chat.json
|
| 225 |
+
{
|
| 226 |
+
"correct": 300,
|
| 227 |
+
"accuracy": 0.22744503411675512,
|
| 228 |
+
"finished": 1273,
|
| 229 |
+
"finish_rate": 0.9651250947687642,
|
| 230 |
+
"mean_completion_tokens": 150.46095526914328
|
| 231 |
+
}
|
| 232 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep25_s1226_step150_chat.json
|
healed/grid_math/uniform_keep50_s1224.console.log
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run r9qva0v2
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/uniform_keep50_s1224/wandb/run-20260716_000817-r9qva0v2
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run uniform-math-keep50-s1224
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/r9qva0v2
|
| 12 |
+
|
| 13 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 3.70B | teacher overlap=False
|
| 14 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4277946377269924, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 5.4375, "lr": 6e-06, "finish_rate": 0.907, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 236}, "mem_gb": 15.78}
|
| 15 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 16 |
+
[eval step 1] sample: "To solve the given system of equations, we start by substituting \\(a\\), \\(b\\), and \\(p\\) in terms of each other based on the equations provided. Let's denote the digits as \\(a\\), \\(b\\), and \\(p\\).\n\n1."
|
| 17 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4541956744345526, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 5.09375, "lr": 9e-06, "finish_rate": 0.781, "comp_len": 558.1, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 34.7, "frames": {"chat": 215}, "mem_gb": 16.06}
|
| 18 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4834472433740894, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 4.84375, "lr": 1.2e-05, "finish_rate": 0.825, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 217}, "mem_gb": 15.93}
|
| 19 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4124494322578112, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 3.359375, "lr": 1.5e-05, "finish_rate": 0.8, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 33.7, "frames": {"chat": 205}, "mem_gb": 15.99}
|
| 20 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3112520435221493, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 2.0, "lr": 1.8e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.9, "frames": {"chat": 229}, "mem_gb": 15.96}
|
| 21 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.31875163183857996, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 1.7421875, "lr": 2.1e-05, "finish_rate": 0.812, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.3, "frames": {"chat": 223}, "mem_gb": 16.03}
|
| 22 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29619284563846887, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 1.4453125, "lr": 2.4e-05, "finish_rate": 0.708, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.4, "frames": {"chat": 202}, "mem_gb": 16.07}
|
| 23 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2755744682001571, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 1.2890625, "lr": 2.7000000000000002e-05, "finish_rate": 0.77, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.6, "frames": {"chat": 209}, "mem_gb": 16.04}
|
| 24 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.237358801820005, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 1.046875, "lr": 3e-05, "finish_rate": 0.885, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.8, "frames": {"chat": 227}, "mem_gb": 16.02}
|
| 25 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.23537027373934785, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.92578125, "lr": 3e-05, "finish_rate": 0.848, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.2, "frames": {"chat": 230}, "mem_gb": 16.09}
|
| 26 |
+
[eval step 10] sample: 'To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(p\\) given the following system of equations:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np'
|
| 27 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.22712834878303112, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.78515625, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.0, "frames": {"chat": 231}, "mem_gb": 15.94}
|
| 28 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.21386457304588208, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.74609375, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.8, "frames": {"chat": 245}, "mem_gb": 16.02}
|
| 29 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20496360156921048, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.7734375, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.0, "frames": {"chat": 210}, "mem_gb": 16.02}
|
| 30 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.24294944500830024, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.75, "lr": 3e-05, "finish_rate": 0.758, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.1, "frames": {"chat": 211}, "mem_gb": 16.02}
|
| 31 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20237714038888613, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.609375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.7, "frames": {"chat": 221}, "mem_gb": 16.08}
|
| 32 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18871633486977468, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.671875, "lr": 3e-05, "finish_rate": 0.912, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 250}, "mem_gb": 15.89}
|
| 33 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20165518890159825, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.6171875, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 229}, "mem_gb": 16.06}
|
| 34 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1621489983693386, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.888, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 250}, "mem_gb": 16.04}
|
| 35 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18379500441768518, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.55859375, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.9, "frames": {"chat": 231}, "mem_gb": 15.91}
|
| 36 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17702249435447157, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.8, "frames": {"chat": 224}, "mem_gb": 15.96}
|
| 37 |
+
[eval step 20] sample: 'To solve the system of equations given by:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe need to express \\(a\\'
|
| 38 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16907274161676566, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.59375, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 212}, "mem_gb": 16.0}
|
| 39 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15890724460290123, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 238}, "mem_gb": 15.95}
|
| 40 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1618830188171317, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.53125, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 257}, "mem_gb": 15.83}
|
| 41 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1499397516572227, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 227}, "mem_gb": 16.02}
|
| 42 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16830042061501493, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.47265625, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 228}, "mem_gb": 16.05}
|
| 43 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16847392910371223, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.478515625, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 44 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14656532863068084, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.44921875, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 45 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2008793422088027, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 609.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.9, "frames": {"chat": 197}, "mem_gb": 16.13}
|
| 46 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1749267460513239, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.4609375, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 239}, "mem_gb": 15.88}
|
| 47 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15793729521036148, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.83, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 224}, "mem_gb": 15.93}
|
| 48 |
+
[eval step 30] sample: "To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(m\\), \\(r\\), and \\(p\\) such that the given equations hold true. Let's break down the problem step-by-step:\n\n1. **Understand the "
|
| 49 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14660292085744442, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.8, "frames": {"chat": 217}, "mem_gb": 16.05}
|
| 50 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14080375293462227, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 241}, "mem_gb": 16.05}
|
| 51 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15155704532898961, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.4, "frames": {"chat": 218}, "mem_gb": 16.02}
|
| 52 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14863066531103106, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.41796875, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 206}, "mem_gb": 16.03}
|
| 53 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15601216383427383, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 232}, "mem_gb": 16.07}
|
| 54 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1585212317263397, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.443359375, "lr": 3e-05, "finish_rate": 0.771, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 218}, "mem_gb": 16.09}
|
| 55 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15692651657070963, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.779, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 213}, "mem_gb": 16.05}
|
| 56 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15659624576146403, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.447265625, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 483.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 248}, "mem_gb": 16.02}
|
| 57 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1459468737422799, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.4453125, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.9, "frames": {"chat": 218}, "mem_gb": 16.08}
|
| 58 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12854463345023492, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.38671875, "lr": 3e-05, "finish_rate": 0.851, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 221}, "mem_gb": 16.04}
|
| 59 |
+
[eval step 40] sample: "To solve the given system of equations, we need to express \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) in terms of each other. Let's break down the problem step-by-step:\n\n1. **Express \\(a\\) and"
|
| 60 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1351330191159621, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.40234375, "lr": 3e-05, "finish_rate": 0.894, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.8, "frames": {"chat": 236}, "mem_gb": 15.97}
|
| 61 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13463257048794378, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.9, "frames": {"chat": 246}, "mem_gb": 15.89}
|
| 62 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13854125740490852, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.40625, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 234}, "mem_gb": 16.14}
|
| 63 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11513334581442178, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.748, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 202}, "mem_gb": 16.03}
|
| 64 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12807885064010818, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.1, "frames": {"chat": 217}, "mem_gb": 16.04}
|
| 65 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13458335966536156, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.866, "comp_len": 535.7, "t_data_s": 0.3, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 224}, "mem_gb": 16.04}
|
| 66 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14978185854256154, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.753, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.7, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 67 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1224529140399148, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.40234375, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 463.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 259}, "mem_gb": 15.97}
|
| 68 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13974743304556855, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.419921875, "lr": 3e-05, "finish_rate": 0.829, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.9, "frames": {"chat": 210}, "mem_gb": 16.04}
|
| 69 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1602421885613352, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.462890625, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.9, "frames": {"chat": 213}, "mem_gb": 16.09}
|
| 70 |
+
[eval step 50] sample: "To solve the given system of equations, we need to express \\( p \\) in terms of \\( a \\), \\( b \\), \\( k \\), \\( m \\), and \\( r \\). Let's break down the problem step-by-step:\n\n1. **Express \\( p \\) in term"
|
| 71 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep50_s1224/step0050
|
| 72 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13515835460849726, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.3, "frames": {"chat": 222}, "mem_gb": 16.0}
|
| 73 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11656722308822597, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.9, "frames": {"chat": 235}, "mem_gb": 16.05}
|
| 74 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13514603171708683, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.3984375, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.7, "frames": {"chat": 208}, "mem_gb": 16.01}
|
| 75 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12384081650370111, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.42578125, "lr": 3e-05, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.3, "frames": {"chat": 191}, "mem_gb": 16.05}
|
| 76 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09931092557078228, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.369140625, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 77 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09109834918997561, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.4, "frames": {"chat": 207}, "mem_gb": 16.05}
|
| 78 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11065167818926275, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.37109375, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 208}, "mem_gb": 16.01}
|
| 79 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08927264785487204, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.8, "frames": {"chat": 219}, "mem_gb": 16.04}
|
| 80 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08789662679632505, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 246}, "mem_gb": 15.92}
|
| 81 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11347239079214633, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 206}, "mem_gb": 16.07}
|
| 82 |
+
[eval step 60] sample: "To solve the given system of equations, we will use Python and SymPy to handle the algebraic manipulations. Let's break down the problem step-by-step and write the Python code to find the value of \\( "
|
| 83 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0887840253782769, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 233}, "mem_gb": 16.05}
|
| 84 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09239683715384453, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.5, "frames": {"chat": 229}, "mem_gb": 15.92}
|
| 85 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08232894674045965, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 236}, "mem_gb": 15.95}
|
| 86 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09751218746158605, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 239}, "mem_gb": 15.84}
|
| 87 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08627100243655343, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.302734375, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 241}, "mem_gb": 15.96}
|
| 88 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09800913436881577, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.7, "frames": {"chat": 226}, "mem_gb": 15.92}
|
| 89 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08852198531199247, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 234}, "mem_gb": 16.05}
|
| 90 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08382236090765025, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 257}, "mem_gb": 16.04}
|
| 91 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12105931493146345, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.3828125, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 208}, "mem_gb": 16.1}
|
| 92 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1038044841277413, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.369140625, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 211}, "mem_gb": 16.07}
|
| 93 |
+
[eval step 70] sample: 'To solve the given system of equations, we will use Python and SymPy to handle the algebraic manipulations. The equations are:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &='
|
| 94 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1100768730642274, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.8, "frames": {"chat": 227}, "mem_gb": 16.05}
|
| 95 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10338081272356212, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.345703125, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 211}, "mem_gb": 16.03}
|
| 96 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08453753360584378, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 238}, "mem_gb": 16.04}
|
| 97 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09178873166749253, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.4, "frames": {"chat": 237}, "mem_gb": 16.08}
|
| 98 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10419285486250494, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.1, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 99 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09346181132777905, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 221}, "mem_gb": 16.17}
|
| 100 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08947489535867546, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 232}, "mem_gb": 16.01}
|
| 101 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09227189582098896, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.6, "frames": {"chat": 208}, "mem_gb": 16.04}
|
| 102 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08777518702357387, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 227}, "mem_gb": 15.96}
|
| 103 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10104774889331311, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 221}, "mem_gb": 15.99}
|
| 104 |
+
[eval step 80] sample: 'To solve the given system of equations, we need to express \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) in terms of each other and solve for their values.\n\nThe system of equations is:\n\\[\n\\begin{align*'
|
| 105 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08170304526534553, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.8, "frames": {"chat": 232}, "mem_gb": 16.05}
|
| 106 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08983821267243475, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.6, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 107 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09457728577253098, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.3125, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.6, "frames": {"chat": 195}, "mem_gb": 16.14}
|
| 108 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09356701532015577, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 216}, "mem_gb": 16.05}
|
| 109 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09165108890738338, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.369140625, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.8, "frames": {"chat": 208}, "mem_gb": 15.94}
|
| 110 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08849761248029148, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 235}, "mem_gb": 15.93}
|
| 111 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08910521061414232, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 225}, "mem_gb": 16.04}
|
| 112 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11201870662448928, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.345703125, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.0, "frames": {"chat": 213}, "mem_gb": 16.13}
|
| 113 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07798361969428758, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.2, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 257}, "mem_gb": 15.8}
|
| 114 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09493755235892411, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.1, "frames": {"chat": 212}, "mem_gb": 16.08}
|
| 115 |
+
[eval step 90] sample: "To solve the given system of equations, we need to express each variable \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) in terms of each other using the given equations. Let's break down the problem ste"
|
| 116 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08449767493788773, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.6, "frames": {"chat": 221}, "mem_gb": 16.05}
|
| 117 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08178153519337066, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 242}, "mem_gb": 16.04}
|
| 118 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07604952207386183, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.7, "frames": {"chat": 220}, "mem_gb": 16.01}
|
| 119 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0801545599025519, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 240}, "mem_gb": 15.9}
|
| 120 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08843556518893068, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.30859375, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.8, "frames": {"chat": 206}, "mem_gb": 16.04}
|
| 121 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08580168459989751, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.306640625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.9, "frames": {"chat": 226}, "mem_gb": 16.05}
|
| 122 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09918145173918456, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 244}, "mem_gb": 15.84}
|
| 123 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10544447463347266, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 224}, "mem_gb": 16.05}
|
| 124 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08442806187899163, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 271}, "mem_gb": 15.78}
|
| 125 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0845842156385382, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 236}, "mem_gb": 16.06}
|
| 126 |
+
[eval step 100] sample: "To solve the given system of equations, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem "
|
| 127 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep50_s1224/step0100
|
| 128 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08446701570067865, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.3125, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.9, "frames": {"chat": 232}, "mem_gb": 15.93}
|
| 129 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0794957819807964, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.6, "frames": {"chat": 210}, "mem_gb": 15.99}
|
| 130 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07818544425293804, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.7, "frames": {"chat": 217}, "mem_gb": 15.95}
|
| 131 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09452820008428146, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 224}, "mem_gb": 16.07}
|
| 132 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11034645940602447, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 203}, "mem_gb": 15.92}
|
| 133 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08894736041029294, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.0, "frames": {"chat": 239}, "mem_gb": 16.02}
|
| 134 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06482621920686216, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 254}, "mem_gb": 15.93}
|
| 135 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0596997441777649, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.2890625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.9, "frames": {"chat": 241}, "mem_gb": 16.03}
|
| 136 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08630636698001375, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 213}, "mem_gb": 16.05}
|
| 137 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07119898068271577, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 221}, "mem_gb": 16.1}
|
| 138 |
+
[eval step 110] sample: "To solve the given system of equations, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem "
|
| 139 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07626933450518797, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.2, "frames": {"chat": 196}, "mem_gb": 16.06}
|
| 140 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.062216147240251304, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 270}, "mem_gb": 15.86}
|
| 141 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06098471744398897, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.7, "frames": {"chat": 216}, "mem_gb": 16.04}
|
| 142 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07360812601515403, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.294921875, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.6, "frames": {"chat": 200}, "mem_gb": 16.01}
|
| 143 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06202772317926089, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.5, "frames": {"chat": 206}, "mem_gb": 15.96}
|
| 144 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05959480526245509, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 234}, "mem_gb": 15.99}
|
| 145 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07201705848282824, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.30078125, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.3, "frames": {"chat": 215}, "mem_gb": 16.01}
|
| 146 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0571913227643352, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.31640625, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 255}, "mem_gb": 15.99}
|
| 147 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05931704173743104, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 250}, "mem_gb": 15.87}
|
| 148 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06392476364959342, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.6, "frames": {"chat": 242}, "mem_gb": 16.04}
|
| 149 |
+
[eval step 120] sample: "To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) such that the given equations hold true. Let's break down the problem step-by-step:\n\n1. **Define V"
|
| 150 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07929505948486427, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 199}, "mem_gb": 16.05}
|
| 151 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08340449355278785, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 152 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06475446787502151, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 208}, "mem_gb": 16.02}
|
| 153 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0749057637304999, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.3, "frames": {"chat": 209}, "mem_gb": 16.17}
|
| 154 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05738816611807172, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.26171875, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 235}, "mem_gb": 16.01}
|
| 155 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0597335497791258, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.263671875, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.2, "frames": {"chat": 204}, "mem_gb": 16.0}
|
| 156 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08213897060084467, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.0, "frames": {"chat": 208}, "mem_gb": 16.05}
|
| 157 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05984125637561083, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 240}, "mem_gb": 16.05}
|
| 158 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0629925194332997, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.294921875, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 246}, "mem_gb": 16.04}
|
| 159 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06474258221313357, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 243}, "mem_gb": 15.86}
|
| 160 |
+
[eval step 130] sample: "To solve the problem, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) such that the given equations hold true. Let's break down the problem step-by-step and use Python "
|
| 161 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07210097291904191, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.291015625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.2, "frames": {"chat": 208}, "mem_gb": 16.06}
|
| 162 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06917726617272323, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.279296875, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.7, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 163 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08181537830297214, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.302734375, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.0, "frames": {"chat": 211}, "mem_gb": 16.06}
|
| 164 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06436642667967826, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.8, "frames": {"chat": 232}, "mem_gb": 16.02}
|
| 165 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06862061261056612, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.8, "frames": {"chat": 214}, "mem_gb": 16.05}
|
| 166 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06350242085993911, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.8, "frames": {"chat": 226}, "mem_gb": 15.95}
|
| 167 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06530722830637048, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.271484375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.2, "frames": {"chat": 210}, "mem_gb": 16.06}
|
| 168 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05820518815023824, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.9, "frames": {"chat": 218}, "mem_gb": 15.88}
|
| 169 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05650871475211655, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.2431640625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.2, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 170 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07226877674786374, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.1, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 171 |
+
[eval step 140] sample: "To solve the given system of equations, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem "
|
| 172 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06750662778724606, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.2, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 173 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.061347212908416986, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.9, "frames": {"chat": 209}, "mem_gb": 15.99}
|
| 174 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.055460787860878435, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 262}, "mem_gb": 15.92}
|
| 175 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.058363437791649875, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.6, "frames": {"chat": 249}, "mem_gb": 16.01}
|
| 176 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07328259567250497, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.2, "frames": {"chat": 227}, "mem_gb": 16.05}
|
| 177 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.058721957165344306, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.0, "frames": {"chat": 221}, "mem_gb": 16.04}
|
| 178 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.061519284149631856, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.279296875, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.2, "frames": {"chat": 234}, "mem_gb": 16.06}
|
| 179 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.056329902307611576, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.0, "frames": {"chat": 213}, "mem_gb": 16.0}
|
| 180 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05893662882659895, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.5, "frames": {"chat": 213}, "mem_gb": 15.94}
|
| 181 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05945110498759896, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.3515625, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.1, "frames": {"chat": 234}, "mem_gb": 15.97}
|
| 182 |
+
[eval step 150] sample: "To solve the given system of equations, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) such that each letter represents a non-zero digit. Let's break down the problem step-"
|
| 183 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep50_s1224/step0150
|
| 184 |
+
wandb: updating run metadata
|
| 185 |
+
wandb: uploading data
|
| 186 |
+
wandb:
|
| 187 |
+
wandb: Run history:
|
| 188 |
+
wandb: comp_len βββββββ
βββββ
ββββββββββ
ββ
βββββ
βββββββββ
β
β
|
| 189 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββ
β
β
β
β
β
β
βββββββββββββββ
|
| 190 |
+
wandb: epoch βββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββββββββββββ
|
| 191 |
+
wandb: finish_rate β
β
βββββ
β
ββββββββββ
βββββ
ββββ
ββ
ββ
ββββββββ
β
|
| 192 |
+
wandb: forward_topk_kl ββββββββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 195 |
+
wandb: mem_gb β
ββββββ
βββββββββββ
ββββ
βββββββββ
β
βββ
ββββ
β
|
| 196 |
+
wandb: step ββββββββββββββββββββββββ
β
β
ββββββββββββββ
|
| 197 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 198 |
+
wandb: +3 ...
|
| 199 |
+
wandb:
|
| 200 |
+
wandb: Run summary:
|
| 201 |
+
wandb: comp_len 512.8
|
| 202 |
+
wandb: cumulative_loss_tokens 18000000
|
| 203 |
+
wandb: epoch 2
|
| 204 |
+
wandb: finish_rate 0.906
|
| 205 |
+
wandb: forward_topk_kl 0.05945
|
| 206 |
+
wandb: grad_norm 0.35156
|
| 207 |
+
wandb: lr 3e-05
|
| 208 |
+
wandb: mem_gb 15.97
|
| 209 |
+
wandb: step 150
|
| 210 |
+
wandb: t_data_s 0
|
| 211 |
+
wandb: +4 ...
|
| 212 |
+
wandb:
|
| 213 |
+
wandb: π View run uniform-math-keep50-s1224 at: https://wandb.ai/hbfreed/glean-grid/runs/r9qva0v2
|
| 214 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 215 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 216 |
+
wandb: Find logs at: outputs/healed/grid_math/uniform_keep50_s1224/wandb/run-20260716_000817-r9qva0v2/logs
|
| 217 |
+
{
|
| 218 |
+
"correct": 668,
|
| 219 |
+
"accuracy": 0.5064442759666414,
|
| 220 |
+
"finished": 1306,
|
| 221 |
+
"finish_rate": 0.9901440485216073,
|
| 222 |
+
"mean_completion_tokens": 111.20773313115997
|
| 223 |
+
}
|
| 224 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep50_s1224_step100_chat.json
|
| 225 |
+
{
|
| 226 |
+
"correct": 683,
|
| 227 |
+
"accuracy": 0.5178165276724791,
|
| 228 |
+
"finished": 1312,
|
| 229 |
+
"finish_rate": 0.9946929492039424,
|
| 230 |
+
"mean_completion_tokens": 109.86429112964368
|
| 231 |
+
}
|
| 232 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep50_s1224_step150_chat.json
|
healed/grid_math/uniform_keep50_s1225.console.log
ADDED
|
@@ -0,0 +1,231 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 6 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/uniform_keep50_s1225/wandb/run-20260716_000817-tj9y7tlh
|
| 7 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 8 |
+
wandb: Syncing run uniform-math-keep50-s1225
|
| 9 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 10 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/tj9y7tlh
|
| 11 |
+
|
| 12 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 3.70B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.48177218888526163, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 5.25, "lr": 6e-06, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.5, "frames": {"chat": 191}, "mem_gb": 15.94}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: "To solve this problem, we need to arrange the numbers \\(1\\) through \\(49\\) in a spiral pattern on a square grid, starting at the center. The spiral pattern starts with \\(1, 2, 3, \\ldots, 49\\).\n\nLet's "
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.4492101515820871, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 5.15625, "lr": 9e-06, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.3, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 17 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.49509669360568126, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 4.84375, "lr": 1.2e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.9, "frames": {"chat": 207}, "mem_gb": 16.05}
|
| 18 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.39116144820724924, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 3.03125, "lr": 1.5e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.7, "frames": {"chat": 208}, "mem_gb": 16.01}
|
| 19 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.3340688153398534, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 2.09375, "lr": 1.8e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.3, "frames": {"chat": 219}, "mem_gb": 16.04}
|
| 20 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.29177642477502425, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 1.640625, "lr": 2.1e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.1, "frames": {"chat": 246}, "mem_gb": 15.92}
|
| 21 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.30177181346006693, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 1.4375, "lr": 2.4e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.0, "frames": {"chat": 206}, "mem_gb": 16.07}
|
| 22 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.25002946078342697, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 1.234375, "lr": 2.7000000000000002e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.5, "frames": {"chat": 233}, "mem_gb": 16.05}
|
| 23 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.23929908262031774, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 1.046875, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.5, "frames": {"chat": 229}, "mem_gb": 15.92}
|
| 24 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2197762858423094, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.8515625, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.0, "frames": {"chat": 236}, "mem_gb": 15.95}
|
| 25 |
+
[eval step 10] sample: 'To solve this problem, we need to determine the four numbers in the spiral pattern on a square grid that are on the same diagonal as the number \\(7\\).\n\n### Steps to Solve the Problem:\n\n1. **Understand'
|
| 26 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2261457913633436, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.81640625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.4, "frames": {"chat": 239}, "mem_gb": 15.84}
|
| 27 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20089141601889085, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.72265625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.3, "frames": {"chat": 241}, "mem_gb": 15.96}
|
| 28 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.21661407431277135, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.73046875, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.2, "frames": {"chat": 226}, "mem_gb": 15.92}
|
| 29 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.19101213004241388, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.625, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.2, "frames": {"chat": 234}, "mem_gb": 16.05}
|
| 30 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1818893420693775, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.58203125, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.0, "frames": {"chat": 257}, "mem_gb": 16.04}
|
| 31 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.23477967827705046, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.62109375, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.2, "frames": {"chat": 208}, "mem_gb": 16.1}
|
| 32 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.20768065678669761, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.625, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.8, "frames": {"chat": 211}, "mem_gb": 16.07}
|
| 33 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1849105342424164, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.546875, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.4, "frames": {"chat": 227}, "mem_gb": 16.05}
|
| 34 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1964245485352973, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.53125, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.2, "frames": {"chat": 211}, "mem_gb": 16.03}
|
| 35 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1649578903068478, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.3, "frames": {"chat": 238}, "mem_gb": 16.04}
|
| 36 |
+
[eval step 20] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid and identify the four numbers that lie on the same diagonal as the number 7. Then, we need to de'
|
| 37 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17328402358014136, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.5, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.5, "frames": {"chat": 237}, "mem_gb": 16.08}
|
| 38 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18340800293311477, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.5, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 39 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17245891622739534, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.49609375, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.5, "frames": {"chat": 221}, "mem_gb": 16.17}
|
| 40 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1629328786447644, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.474609375, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.5, "frames": {"chat": 232}, "mem_gb": 16.01}
|
| 41 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16500690919520955, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.45703125, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.5, "frames": {"chat": 208}, "mem_gb": 16.04}
|
| 42 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15608946372761082, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.451171875, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.8, "frames": {"chat": 227}, "mem_gb": 15.96}
|
| 43 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.17273450978553542, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.5, "frames": {"chat": 221}, "mem_gb": 15.99}
|
| 44 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14559424359239637, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.2, "frames": {"chat": 232}, "mem_gb": 16.05}
|
| 45 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14504854150991886, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.4, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 46 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16033332358269642, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.2, "frames": {"chat": 195}, "mem_gb": 16.14}
|
| 47 |
+
[eval step 30] sample: "To solve this problem, we need to identify the four numbers that appear in the shaded squares on the same diagonal as the number \\(7\\) in a spiral pattern of numbers from 1 to 49. Let's break down the"
|
| 48 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1565866457330063, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.443359375, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.6, "frames": {"chat": 216}, "mem_gb": 16.05}
|
| 49 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13086035422962158, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.40625, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.6, "frames": {"chat": 208}, "mem_gb": 15.94}
|
| 50 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1362724322141148, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.0, "frames": {"chat": 235}, "mem_gb": 15.93}
|
| 51 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14016164914164692, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.8, "frames": {"chat": 225}, "mem_gb": 16.04}
|
| 52 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18476179402396084, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.48828125, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.4, "frames": {"chat": 213}, "mem_gb": 16.13}
|
| 53 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13153720762940743, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.421875, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.5, "frames": {"chat": 257}, "mem_gb": 15.8}
|
| 54 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15172348517642045, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.419921875, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.9, "frames": {"chat": 212}, "mem_gb": 16.08}
|
| 55 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1344316146434595, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.3, "frames": {"chat": 221}, "mem_gb": 16.05}
|
| 56 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13561088651213796, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.39453125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.5, "frames": {"chat": 242}, "mem_gb": 16.04}
|
| 57 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1209568001365289, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.7, "frames": {"chat": 220}, "mem_gb": 16.01}
|
| 58 |
+
[eval step 40] sample: "To solve this problem, we need to identify the four numbers that appear in the shaded squares on the same diagonal as the number \\(7\\) in a spiral pattern on a square grid. Let's break down the proble"
|
| 59 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12977896326227734, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.0, "frames": {"chat": 240}, "mem_gb": 15.9}
|
| 60 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14052145098727195, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.4, "frames": {"chat": 206}, "mem_gb": 16.04}
|
| 61 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12913375288397075, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.369140625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.8, "frames": {"chat": 226}, "mem_gb": 16.05}
|
| 62 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1500127704118068, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.6, "frames": {"chat": 244}, "mem_gb": 15.84}
|
| 63 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15241718454652775, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.1, "frames": {"chat": 224}, "mem_gb": 16.05}
|
| 64 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13264826799227547, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.5, "frames": {"chat": 271}, "mem_gb": 15.78}
|
| 65 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11873548210449517, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.4140625, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.0, "frames": {"chat": 236}, "mem_gb": 16.06}
|
| 66 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13592392750435198, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.41015625, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.9, "frames": {"chat": 232}, "mem_gb": 15.93}
|
| 67 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11282420297750893, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.375, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.1, "frames": {"chat": 210}, "mem_gb": 15.99}
|
| 68 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11188526095325749, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.375, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.0, "frames": {"chat": 217}, "mem_gb": 15.95}
|
| 69 |
+
[eval step 50] sample: "To solve this problem, we need to identify the four numbers that appear in the shaded squares on the same diagonal as the number \\(7\\) in a spiral pattern of numbers from 1 to 49. Let's break down the"
|
| 70 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep50_s1225/step0050
|
| 71 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14507852032749602, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.4, "frames": {"chat": 224}, "mem_gb": 16.07}
|
| 72 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16389974264514942, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.466796875, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.0, "frames": {"chat": 203}, "mem_gb": 15.92}
|
| 73 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1358009697439149, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.2, "frames": {"chat": 239}, "mem_gb": 16.02}
|
| 74 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0977568897123759, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.388671875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.9, "frames": {"chat": 254}, "mem_gb": 15.93}
|
| 75 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0903720636972847, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.1, "frames": {"chat": 241}, "mem_gb": 16.03}
|
| 76 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1253400321662426, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.5, "frames": {"chat": 213}, "mem_gb": 16.05}
|
| 77 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1029980695111677, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.7, "frames": {"chat": 221}, "mem_gb": 16.1}
|
| 78 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10780552417521054, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.34375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.0, "frames": {"chat": 196}, "mem_gb": 16.06}
|
| 79 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09320413307485481, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.1, "frames": {"chat": 270}, "mem_gb": 15.86}
|
| 80 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08679414614799122, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.2, "frames": {"chat": 216}, "mem_gb": 16.04}
|
| 81 |
+
[eval step 60] sample: 'To solve this problem, we need to identify the four numbers from the sequence \\(1, 2, 3, 4, 5, 6, 7, \\ldots, 49\\) that lie on the same diagonal as the number \\(7\\) and are prime.\n\n### Steps to Solve t'
|
| 82 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12127362799135347, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.40234375, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.6, "frames": {"chat": 200}, "mem_gb": 16.01}
|
| 83 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08365503143308063, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.6, "frames": {"chat": 206}, "mem_gb": 15.96}
|
| 84 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08216556253097951, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.302734375, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.7, "frames": {"chat": 234}, "mem_gb": 15.99}
|
| 85 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10262446160347512, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.34375, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.3, "frames": {"chat": 215}, "mem_gb": 16.01}
|
| 86 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08079534837898487, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.8, "frames": {"chat": 255}, "mem_gb": 15.99}
|
| 87 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08919197535691782, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.365234375, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.8, "frames": {"chat": 250}, "mem_gb": 15.87}
|
| 88 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09303663084845369, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.3, "frames": {"chat": 242}, "mem_gb": 16.04}
|
| 89 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.12093032057676464, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.2, "frames": {"chat": 199}, "mem_gb": 16.05}
|
| 90 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1223956153758491, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.388671875, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.9, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 91 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09326641459406043, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.2, "frames": {"chat": 208}, "mem_gb": 16.02}
|
| 92 |
+
[eval step 70] sample: 'To solve this problem, we need to understand the structure of the spiral pattern on the grid and identify the numbers that lie on the same diagonal as the number \\(7\\).\n\n### Steps to Solve the Problem'
|
| 93 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1034547137827302, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.345703125, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.9, "frames": {"chat": 209}, "mem_gb": 16.17}
|
| 94 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08132403246794517, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.2, "frames": {"chat": 235}, "mem_gb": 16.01}
|
| 95 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08316841166916614, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.31640625, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.4, "frames": {"chat": 204}, "mem_gb": 16.0}
|
| 96 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11559949020318067, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.9, "frames": {"chat": 208}, "mem_gb": 16.05}
|
| 97 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08412509058462456, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.30859375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.9, "frames": {"chat": 240}, "mem_gb": 16.05}
|
| 98 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09016936350489656, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.0, "frames": {"chat": 246}, "mem_gb": 16.04}
|
| 99 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0944399022025677, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.9, "frames": {"chat": 243}, "mem_gb": 15.86}
|
| 100 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10787633103836948, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.359375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.5, "frames": {"chat": 208}, "mem_gb": 16.06}
|
| 101 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09065167806303749, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.3, "frames": {"chat": 219}, "mem_gb": 16.05}
|
| 102 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1123543589844058, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.4, "frames": {"chat": 211}, "mem_gb": 16.06}
|
| 103 |
+
[eval step 80] sample: "To solve this problem, we need to identify the four numbers on the same diagonal as the number \\(7\\) in a spiral pattern of numbers from 1 to 49 arranged on a square grid. Let's break down the steps:\n"
|
| 104 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08160814283859606, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.4, "frames": {"chat": 232}, "mem_gb": 16.02}
|
| 105 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0911076406781872, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.2, "frames": {"chat": 214}, "mem_gb": 16.05}
|
| 106 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09021442107763142, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.0, "frames": {"chat": 226}, "mem_gb": 15.95}
|
| 107 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09332444871294622, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.3, "frames": {"chat": 210}, "mem_gb": 16.06}
|
| 108 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07827468976057135, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.3046875, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.4, "frames": {"chat": 218}, "mem_gb": 15.88}
|
| 109 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07789079143583465, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.1, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 110 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.1009345404283454, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.4, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 111 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0944027965891175, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.6, "frames": {"chat": 233}, "mem_gb": 16.04}
|
| 112 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08136886405814439, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.9, "frames": {"chat": 209}, "mem_gb": 15.99}
|
| 113 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07810565605591982, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.5, "frames": {"chat": 262}, "mem_gb": 15.92}
|
| 114 |
+
[eval step 90] sample: 'To solve this problem, we need to identify the four numbers on the diagonal that correspond to the number \\(7\\) in the spiral pattern of the grid. The spiral pattern starts at the center and moves clo'
|
| 115 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07821270908623312, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.30859375, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.5, "frames": {"chat": 249}, "mem_gb": 16.01}
|
| 116 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.10085701854179303, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.7, "frames": {"chat": 227}, "mem_gb": 16.05}
|
| 117 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0847742678733853, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.6, "frames": {"chat": 221}, "mem_gb": 16.04}
|
| 118 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08609593580262735, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.6, "frames": {"chat": 234}, "mem_gb": 16.06}
|
| 119 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07459482868239284, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.2, "frames": {"chat": 213}, "mem_gb": 16.0}
|
| 120 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07537244438646983, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.4, "frames": {"chat": 213}, "mem_gb": 15.94}
|
| 121 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08689684214126318, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.1, "frames": {"chat": 234}, "mem_gb": 15.97}
|
| 122 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09049659343931514, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.6, "frames": {"chat": 222}, "mem_gb": 16.04}
|
| 123 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09133215679507703, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.3, "frames": {"chat": 227}, "mem_gb": 16.06}
|
| 124 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08776675813384355, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.4, "frames": {"chat": 218}, "mem_gb": 16.09}
|
| 125 |
+
[eval step 100] sample: 'To solve this problem, we need to understand the structure of the spiral pattern on the grid and how the numbers are arranged. The numbers from 1 to 49 are arranged in a spiral pattern, starting from '
|
| 126 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep50_s1225/step0100
|
| 127 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0949454252282468, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.4, "frames": {"chat": 223}, "mem_gb": 16.06}
|
| 128 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.09777595625078926, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.9, "frames": {"chat": 206}, "mem_gb": 16.05}
|
| 129 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08421979916729033, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.0, "frames": {"chat": 213}, "mem_gb": 15.97}
|
| 130 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.11003809169378753, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.7, "frames": {"chat": 223}, "mem_gb": 15.91}
|
| 131 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.08955485637433205, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.6, "frames": {"chat": 227}, "mem_gb": 16.02}
|
| 132 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.07951079289832463, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.0, "frames": {"chat": 253}, "mem_gb": 16.05}
|
| 133 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08083365453135533, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.337890625, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.8, "frames": {"chat": 216}, "mem_gb": 16.06}
|
| 134 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07531270161882664, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.5, "frames": {"chat": 205}, "mem_gb": 16.02}
|
| 135 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.08617055136881148, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.345703125, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.7, "frames": {"chat": 207}, "mem_gb": 16.11}
|
| 136 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07958229148842705, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.30078125, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.1, "frames": {"chat": 215}, "mem_gb": 16.03}
|
| 137 |
+
[eval step 110] sample: 'To solve this problem, we need to understand the structure of the spiral pattern on the grid and identify the four numbers that lie on the same diagonal as the number \\(7\\).\n\n### Steps to Solve the Pr'
|
| 138 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06058003036542796, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.0, "frames": {"chat": 228}, "mem_gb": 16.05}
|
| 139 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07366450755995077, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.291015625, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.3, "frames": {"chat": 221}, "mem_gb": 16.09}
|
| 140 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05554351498146231, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.255859375, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.6, "frames": {"chat": 254}, "mem_gb": 15.89}
|
| 141 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05564516919666591, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.8, "frames": {"chat": 210}, "mem_gb": 16.01}
|
| 142 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.060165981625051546, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.24609375, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.3, "frames": {"chat": 226}, "mem_gb": 15.97}
|
| 143 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06510323798443812, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.2, "frames": {"chat": 212}, "mem_gb": 16.04}
|
| 144 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0749064519510294, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.296875, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.7, "frames": {"chat": 211}, "mem_gb": 15.97}
|
| 145 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07834269242317726, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.298828125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 34.1, "frames": {"chat": 196}, "mem_gb": 16.02}
|
| 146 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.058797676620120184, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.9, "frames": {"chat": 212}, "mem_gb": 16.04}
|
| 147 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.057397480336111036, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.1, "frames": {"chat": 244}, "mem_gb": 15.96}
|
| 148 |
+
[eval step 120] sample: 'To solve this problem, we need to understand the structure of the spiral pattern on the grid and identify the numbers that lie on the same diagonal as the number \\(7\\).\n\n### Steps to Solve the Problem'
|
| 149 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.061074561254587025, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.5, "frames": {"chat": 222}, "mem_gb": 16.0}
|
| 150 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06682961131685103, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.2890625, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.2, "frames": {"chat": 218}, "mem_gb": 16.05}
|
| 151 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0644194861006923, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.6, "frames": {"chat": 252}, "mem_gb": 15.92}
|
| 152 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07198429885810861, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.28515625, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.4, "frames": {"chat": 202}, "mem_gb": 16.1}
|
| 153 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07482979528958289, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.6, "frames": {"chat": 237}, "mem_gb": 16.05}
|
| 154 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06636300194105134, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.7, "frames": {"chat": 234}, "mem_gb": 16.03}
|
| 155 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0545189349170619, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.4, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 156 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.053207050136492275, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.2431640625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.1, "frames": {"chat": 234}, "mem_gb": 15.98}
|
| 157 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05247051075274746, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.2412109375, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.6, "frames": {"chat": 216}, "mem_gb": 16.03}
|
| 158 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0620192121199177, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.26171875, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.5, "frames": {"chat": 210}, "mem_gb": 16.0}
|
| 159 |
+
[eval step 130] sample: 'To solve this problem, we need to identify the four numbers on the diagonal that include the number \\(7\\) and determine how many of these numbers are prime.\n\n### Steps to Solve the Problem:\n\n1. **Iden'
|
| 160 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06753606330246355, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.1, "frames": {"chat": 199}, "mem_gb": 16.05}
|
| 161 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06196977580211436, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.9, "frames": {"chat": 210}, "mem_gb": 16.06}
|
| 162 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05961425757134954, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.2, "frames": {"chat": 225}, "mem_gb": 16.0}
|
| 163 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.058377123098867015, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.5, "frames": {"chat": 253}, "mem_gb": 15.9}
|
| 164 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05910795236534128, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.9, "frames": {"chat": 247}, "mem_gb": 16.02}
|
| 165 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06105499646520863, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.6, "frames": {"chat": 238}, "mem_gb": 16.02}
|
| 166 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0698598912583664, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.279296875, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.0, "frames": {"chat": 235}, "mem_gb": 16.04}
|
| 167 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0758493947354611, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.7, "frames": {"chat": 215}, "mem_gb": 16.02}
|
| 168 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05895033346662919, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.279296875, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.8, "frames": {"chat": 268}, "mem_gb": 16.02}
|
| 169 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06744063342887287, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.4, "frames": {"chat": 228}, "mem_gb": 16.05}
|
| 170 |
+
[eval step 140] sample: 'To solve this problem, we need to identify the four numbers on the diagonal that include the number \\(7\\) and determine how many of these numbers are prime.\n\n### Steps to Solve the Problem:\n\n1. **Iden'
|
| 171 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05960630811361286, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.255859375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.5, "frames": {"chat": 252}, "mem_gb": 15.98}
|
| 172 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.062250979524493835, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.7, "frames": {"chat": 223}, "mem_gb": 16.06}
|
| 173 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07568410117311093, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.28515625, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.7, "frames": {"chat": 226}, "mem_gb": 16.05}
|
| 174 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.07676944995061494, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.0, "frames": {"chat": 208}, "mem_gb": 16.1}
|
| 175 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05351945217698813, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.8, "frames": {"chat": 240}, "mem_gb": 15.98}
|
| 176 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.06589085518893165, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.0, "frames": {"chat": 222}, "mem_gb": 15.97}
|
| 177 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05625805737182964, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.9, "frames": {"chat": 236}, "mem_gb": 16.05}
|
| 178 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05725338245869304, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 35.9, "frames": {"chat": 217}, "mem_gb": 16.01}
|
| 179 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.060780384954002994, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.2, "frames": {"chat": 252}, "mem_gb": 15.92}
|
| 180 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.05998445832872142, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 36.3, "frames": {"chat": 222}, "mem_gb": 16.04}
|
| 181 |
+
[eval step 150] sample: "To solve this problem, we need to identify the four numbers on the diagonal that are prime and lie on the same diagonal as the number \\(7\\). Let's break down the steps:\n\n1. **Identify the Diagonal Num"
|
| 182 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep50_s1225/step0150
|
| 183 |
+
wandb: updating run metadata
|
| 184 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 185 |
+
wandb:
|
| 186 |
+
wandb: Run history:
|
| 187 |
+
wandb: comp_len β
β
βββββ
ββββββββββββββββββββββ
ββ
βββββββββ
|
| 188 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββββββ
β
β
β
β
β
β
β
ββββββββββ
|
| 189 |
+
wandb: epoch ββββββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
βββββββββββ
|
| 190 |
+
wandb: finish_rate ββββββββββ
β
ββ
βββββββ
βββββββββ
ββββ
βββ
ββ
β
β
|
| 191 |
+
wandb: forward_topk_kl ββββββββββββββββββββββββββββββββββββββββ
|
| 192 |
+
wandb: grad_norm ββββ
β
βββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: mem_gb β
β
β
ββββββ
ββββββββ
ββ
ββ
ββ
ββ
βββββ
β
β
β
β
ββ
β
β
ββ
|
| 195 |
+
wandb: step βββββββββββββββββββββββββ
β
ββββββββββββββ
|
| 196 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 197 |
+
wandb: +3 ...
|
| 198 |
+
wandb:
|
| 199 |
+
wandb: Run summary:
|
| 200 |
+
wandb: comp_len 540.5
|
| 201 |
+
wandb: cumulative_loss_tokens 18000000
|
| 202 |
+
wandb: epoch 2
|
| 203 |
+
wandb: finish_rate 0.847
|
| 204 |
+
wandb: forward_topk_kl 0.05998
|
| 205 |
+
wandb: grad_norm 0.26562
|
| 206 |
+
wandb: lr 3e-05
|
| 207 |
+
wandb: mem_gb 16.04
|
| 208 |
+
wandb: step 150
|
| 209 |
+
wandb: t_data_s 0
|
| 210 |
+
wandb: +4 ...
|
| 211 |
+
wandb:
|
| 212 |
+
wandb: π View run uniform-math-keep50-s1225 at: https://wandb.ai/hbfreed/glean-grid/runs/tj9y7tlh
|
| 213 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 214 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 215 |
+
wandb: Find logs at: outputs/healed/grid_math/uniform_keep50_s1225/wandb/run-20260716_000817-tj9y7tlh/logs
|
| 216 |
+
{
|
| 217 |
+
"correct": 663,
|
| 218 |
+
"accuracy": 0.5026535253980288,
|
| 219 |
+
"finished": 1306,
|
| 220 |
+
"finish_rate": 0.9901440485216073,
|
| 221 |
+
"mean_completion_tokens": 111.62471569370736
|
| 222 |
+
}
|
| 223 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep50_s1225_step100_chat.json
|
| 224 |
+
{
|
| 225 |
+
"correct": 676,
|
| 226 |
+
"accuracy": 0.5125094768764216,
|
| 227 |
+
"finished": 1307,
|
| 228 |
+
"finish_rate": 0.9909021986353298,
|
| 229 |
+
"mean_completion_tokens": 111.23199393479909
|
| 230 |
+
}
|
| 231 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep50_s1225_step150_chat.json
|
healed/grid_math/uniform_keep75_s1224.console.log
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run bknlxtxa
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/uniform_keep75_s1224/wandb/run-20260716_151320-bknlxtxa
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run uniform-math-keep75-s1224
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/bknlxtxa
|
| 12 |
+
|
| 13 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 5.31B | teacher overlap=False
|
| 14 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1149226022735859, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 1.3828125, "lr": 6e-06, "finish_rate": 0.907, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.1, "frames": {"chat": 236}, "mem_gb": 21.78}
|
| 15 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 16 |
+
[eval step 1] sample: 'To solve the system of equations given by:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe need to express each variable in'
|
| 17 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.128331601900359, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 1.5078125, "lr": 9e-06, "finish_rate": 0.781, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 215}, "mem_gb": 22.1}
|
| 18 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14486369753740727, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 1.6875, "lr": 1.2e-05, "finish_rate": 0.825, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 217}, "mem_gb": 21.98}
|
| 19 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12441399632034203, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 1.0859375, "lr": 1.5e-05, "finish_rate": 0.8, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 205}, "mem_gb": 22.03}
|
| 20 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09499152126203601, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 0.73828125, "lr": 1.8e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 229}, "mem_gb": 22.01}
|
| 21 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10277049167873338, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 0.79296875, "lr": 2.1e-05, "finish_rate": 0.812, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 223}, "mem_gb": 22.08}
|
| 22 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1052986216888763, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 0.7890625, "lr": 2.4e-05, "finish_rate": 0.708, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 202}, "mem_gb": 22.12}
|
| 23 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10085103818140924, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 0.7109375, "lr": 2.7000000000000002e-05, "finish_rate": 0.77, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 209}, "mem_gb": 22.09}
|
| 24 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08776206363585468, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 0.59765625, "lr": 3e-05, "finish_rate": 0.885, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 227}, "mem_gb": 22.06}
|
| 25 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08946861902835468, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.55859375, "lr": 3e-05, "finish_rate": 0.848, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 230}, "mem_gb": 22.14}
|
| 26 |
+
[eval step 10] sample: 'To solve the system of equations given by:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe need to find the values of'
|
| 27 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08697895860231171, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 231}, "mem_gb": 21.99}
|
| 28 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08442495001095036, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.474609375, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 245}, "mem_gb": 22.06}
|
| 29 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08964056528400009, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.546875, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 210}, "mem_gb": 22.07}
|
| 30 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09891614242913202, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.758, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 211}, "mem_gb": 22.07}
|
| 31 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08705015947806338, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 221}, "mem_gb": 22.12}
|
| 32 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07347599292082402, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.912, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 250}, "mem_gb": 21.94}
|
| 33 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08027003410557906, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.41796875, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 229}, "mem_gb": 22.11}
|
| 34 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06456616308207934, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.3828125, "lr": 3e-05, "finish_rate": 0.888, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 250}, "mem_gb": 22.08}
|
| 35 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07464320035312946, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 231}, "mem_gb": 21.96}
|
| 36 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07067552212617981, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 224}, "mem_gb": 22.0}
|
| 37 |
+
[eval step 20] sample: 'To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) that satisfy all the'
|
| 38 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07461256283394371, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.46875, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 212}, "mem_gb": 22.04}
|
| 39 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06825448354546292, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 238}, "mem_gb": 22.0}
|
| 40 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06435082765305415, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 257}, "mem_gb": 21.88}
|
| 41 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06289123019057637, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.357421875, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 227}, "mem_gb": 22.07}
|
| 42 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06933577463313316, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.3671875, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 228}, "mem_gb": 22.1}
|
| 43 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07340255697735895, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 233}, "mem_gb": 22.09}
|
| 44 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06148227691013987, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 233}, "mem_gb": 22.08}
|
| 45 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08668746229810641, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.45703125, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 609.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 197}, "mem_gb": 22.18}
|
| 46 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07390619851493586, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.3828125, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 239}, "mem_gb": 21.93}
|
| 47 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.062313925416612376, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.83, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 224}, "mem_gb": 21.98}
|
| 48 |
+
[eval step 30] sample: 'To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) that satisfy all the'
|
| 49 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06587653084291765, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.3984375, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 217}, "mem_gb": 22.09}
|
| 50 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06159497142334779, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.369140625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 241}, "mem_gb": 22.09}
|
| 51 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06306371927051804, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.375, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 218}, "mem_gb": 22.07}
|
| 52 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06510458102393896, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 206}, "mem_gb": 22.08}
|
| 53 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06617356744012795, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 232}, "mem_gb": 22.12}
|
| 54 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07194665032081927, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.470703125, "lr": 3e-05, "finish_rate": 0.771, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 218}, "mem_gb": 22.14}
|
| 55 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06972922030417249, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.779, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 213}, "mem_gb": 22.1}
|
| 56 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06515178998891885, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.35546875, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 483.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 248}, "mem_gb": 22.07}
|
| 57 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06063467749779423, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.803, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 218}, "mem_gb": 22.13}
|
| 58 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05568547325697727, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.30078125, "lr": 3e-05, "finish_rate": 0.851, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 221}, "mem_gb": 22.08}
|
| 59 |
+
[eval step 40] sample: 'To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\) that satisfy all the'
|
| 60 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05548202602540453, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.3125, "lr": 3e-05, "finish_rate": 0.894, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 434.5, "frames": {"chat": 236}, "mem_gb": 22.02}
|
| 61 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05675041470772897, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.318359375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 409.9, "frames": {"chat": 246}, "mem_gb": 21.94}
|
| 62 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06052485575717874, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 234}, "mem_gb": 22.19}
|
| 63 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05369023088729009, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.748, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.8, "frames": {"chat": 202}, "mem_gb": 22.07}
|
| 64 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05677622093005727, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.5, "frames": {"chat": 217}, "mem_gb": 22.09}
|
| 65 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05560202524064419, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.866, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.0, "frames": {"chat": 224}, "mem_gb": 22.09}
|
| 66 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06349040182760607, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.753, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 215}, "mem_gb": 22.1}
|
| 67 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.04919720755852759, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.306640625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 463.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.7, "frames": {"chat": 259}, "mem_gb": 22.02}
|
| 68 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.059408606637998795, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.829, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 210}, "mem_gb": 22.09}
|
| 69 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07042193401288241, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.3984375, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.4, "frames": {"chat": 213}, "mem_gb": 22.14}
|
| 70 |
+
[eval step 50] sample: "To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\).\n\nLet's break down t"
|
| 71 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep75_s1224/step0050
|
| 72 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05692075071053114, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 222}, "mem_gb": 22.05}
|
| 73 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.052173386886234706, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 235}, "mem_gb": 22.1}
|
| 74 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.059462278134546555, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 208}, "mem_gb": 22.06}
|
| 75 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.045240455083704244, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.4, "frames": {"chat": 191}, "mem_gb": 22.1}
|
| 76 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03589170680805886, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.244140625, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 219}, "mem_gb": 22.09}
|
| 77 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03801956171578107, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 207}, "mem_gb": 22.1}
|
| 78 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0399568536445188, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 208}, "mem_gb": 22.05}
|
| 79 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03400595511964833, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.2177734375, "lr": 3e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 219}, "mem_gb": 22.09}
|
| 80 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03301748423215468, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.2216796875, "lr": 3e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 246}, "mem_gb": 21.96}
|
| 81 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.043141082558253156, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.287109375, "lr": 3e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 206}, "mem_gb": 22.12}
|
| 82 |
+
[eval step 60] sample: "To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\).\n\nLet's break down t"
|
| 83 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03528507920033298, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.255859375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 233}, "mem_gb": 22.1}
|
| 84 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03265478485104007, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.224609375, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 229}, "mem_gb": 21.96}
|
| 85 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.031905491882663534, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.2255859375, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 236}, "mem_gb": 22.0}
|
| 86 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03695723408527362, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.2373046875, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 239}, "mem_gb": 21.88}
|
| 87 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.032592856184983005, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.2158203125, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 241}, "mem_gb": 22.0}
|
| 88 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03586128164082766, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.232421875, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 226}, "mem_gb": 21.97}
|
| 89 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03132054034647687, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.2275390625, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 234}, "mem_gb": 22.09}
|
| 90 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.030788104644580743, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.224609375, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 257}, "mem_gb": 22.08}
|
| 91 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.045981417168816555, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.2890625, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 208}, "mem_gb": 22.14}
|
| 92 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04108849198470513, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.255859375, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 211}, "mem_gb": 22.11}
|
| 93 |
+
[eval step 70] sample: "To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\).\n\nLet's break down t"
|
| 94 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03987854701854909, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.28125, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.7, "frames": {"chat": 227}, "mem_gb": 22.1}
|
| 95 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03947833718151475, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 211}, "mem_gb": 22.07}
|
| 96 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.032092607042985034, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.23046875, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 238}, "mem_gb": 22.09}
|
| 97 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03403336244607344, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.2216796875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.4, "frames": {"chat": 237}, "mem_gb": 22.13}
|
| 98 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04188025526891773, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 208}, "mem_gb": 22.13}
|
| 99 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.033074141753333, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.216796875, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 221}, "mem_gb": 22.22}
|
| 100 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.032880204598419366, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.0, "frames": {"chat": 232}, "mem_gb": 22.05}
|
| 101 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03631957043294484, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.23828125, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.5, "frames": {"chat": 208}, "mem_gb": 22.09}
|
| 102 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.031250697735549574, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.2373046875, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 227}, "mem_gb": 22.01}
|
| 103 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03714153531583336, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 221}, "mem_gb": 22.03}
|
| 104 |
+
[eval step 80] sample: "To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\).\n\nLet's break down t"
|
| 105 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03134368731607683, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.2470703125, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 232}, "mem_gb": 22.1}
|
| 106 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03455453577995456, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.2333984375, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.5, "frames": {"chat": 219}, "mem_gb": 22.1}
|
| 107 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.036677736731572076, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.2275390625, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 195}, "mem_gb": 22.19}
|
| 108 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03607522614639408, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.234375, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 216}, "mem_gb": 22.1}
|
| 109 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.038589416616572995, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.28125, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 208}, "mem_gb": 21.98}
|
| 110 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03083962368282955, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.21875, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 235}, "mem_gb": 21.98}
|
| 111 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03220725948303783, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 225}, "mem_gb": 22.08}
|
| 112 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04238495488502085, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.5, "frames": {"chat": 213}, "mem_gb": 22.18}
|
| 113 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.028955379137241593, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.2294921875, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 257}, "mem_gb": 21.85}
|
| 114 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03898017764575779, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 212}, "mem_gb": 22.12}
|
| 115 |
+
[eval step 90] sample: "To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\).\n\nLet's break down t"
|
| 116 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.033426403526705686, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.2451171875, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 221}, "mem_gb": 22.1}
|
| 117 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.030457712451570355, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.2109375, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 242}, "mem_gb": 22.09}
|
| 118 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03081075047897175, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.240234375, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 220}, "mem_gb": 22.06}
|
| 119 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02926128526229877, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.2265625, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 240}, "mem_gb": 21.95}
|
| 120 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03344736107902912, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.228515625, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 206}, "mem_gb": 22.08}
|
| 121 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03418054170239096, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.2294921875, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 226}, "mem_gb": 22.1}
|
| 122 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03584913180077759, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 244}, "mem_gb": 21.88}
|
| 123 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03905867359065451, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.279296875, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 224}, "mem_gb": 22.1}
|
| 124 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.030537177202943713, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.2421875, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.4, "frames": {"chat": 271}, "mem_gb": 21.82}
|
| 125 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03323106580647485, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 236}, "mem_gb": 22.11}
|
| 126 |
+
[eval step 100] sample: 'To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\).\n\nThe equations are:'
|
| 127 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep75_s1224/step0100
|
| 128 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.029700769117233964, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.234375, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 232}, "mem_gb": 21.97}
|
| 129 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03277247970288154, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.2451171875, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 210}, "mem_gb": 22.03}
|
| 130 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03219541934624625, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.2470703125, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 217}, "mem_gb": 22.0}
|
| 131 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.035269686618377455, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.2451171875, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 224}, "mem_gb": 22.12}
|
| 132 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.042453550912012965, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 203}, "mem_gb": 21.97}
|
| 133 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03221358998257201, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.2255859375, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 239}, "mem_gb": 22.06}
|
| 134 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0216342947023067, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.1875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 254}, "mem_gb": 21.98}
|
| 135 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0223819046114649, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.2255859375, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 241}, "mem_gb": 22.07}
|
| 136 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0306436812187545, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.2041015625, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 213}, "mem_gb": 22.1}
|
| 137 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.027198022907366975, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 221}, "mem_gb": 22.15}
|
| 138 |
+
[eval step 110] sample: "To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\).\n\nLet's break down t"
|
| 139 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.028086171217844822, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 196}, "mem_gb": 22.11}
|
| 140 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021727316307936173, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.169921875, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.9, "frames": {"chat": 270}, "mem_gb": 21.91}
|
| 141 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.023765899055164, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.216796875, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 216}, "mem_gb": 22.09}
|
| 142 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.027058855175063946, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 200}, "mem_gb": 22.06}
|
| 143 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02359186636308829, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.1953125, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 206}, "mem_gb": 22.01}
|
| 144 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.020846557306901863, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.1748046875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 234}, "mem_gb": 22.04}
|
| 145 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02425564272745202, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.181640625, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 215}, "mem_gb": 22.05}
|
| 146 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.020192126658698545, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.17578125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 255}, "mem_gb": 22.04}
|
| 147 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021318466197893335, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.177734375, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 250}, "mem_gb": 21.92}
|
| 148 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022364306481989723, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.1728515625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 242}, "mem_gb": 22.09}
|
| 149 |
+
[eval step 120] sample: 'To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(p\\) that satisfy all the equati'
|
| 150 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.027774021465998763, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.1953125, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 199}, "mem_gb": 22.1}
|
| 151 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03040962266094672, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.216796875, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 208}, "mem_gb": 22.13}
|
| 152 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02479745573766219, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.23046875, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.7, "frames": {"chat": 208}, "mem_gb": 22.07}
|
| 153 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.029116933320866276, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.2099609375, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 209}, "mem_gb": 22.22}
|
| 154 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.020989714213639186, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.17578125, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 235}, "mem_gb": 22.05}
|
| 155 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022509423316735774, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.197265625, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.9, "frames": {"chat": 204}, "mem_gb": 22.04}
|
| 156 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03046467731000545, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.25, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 208}, "mem_gb": 22.1}
|
| 157 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022415347762413634, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.185546875, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 240}, "mem_gb": 22.09}
|
| 158 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022870081384774917, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.189453125, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.6, "frames": {"chat": 246}, "mem_gb": 22.09}
|
| 159 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02218571406165914, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.1796875, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 243}, "mem_gb": 21.91}
|
| 160 |
+
[eval step 130] sample: 'To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), \\(p\\), and \\(r\\).\n\nThe equations are:'
|
| 161 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024033155613788403, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.1728515625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 208}, "mem_gb": 22.1}
|
| 162 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.026436940440007797, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 219}, "mem_gb": 22.1}
|
| 163 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03020671685578612, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.7, "frames": {"chat": 211}, "mem_gb": 22.11}
|
| 164 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.025810199808344866, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.220703125, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.7, "frames": {"chat": 232}, "mem_gb": 22.07}
|
| 165 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0242060411726047, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.1884765625, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.3, "frames": {"chat": 214}, "mem_gb": 22.1}
|
| 166 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.023627944999715933, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.1748046875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.8, "frames": {"chat": 226}, "mem_gb": 21.99}
|
| 167 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024266153483674863, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.185546875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.3, "frames": {"chat": 210}, "mem_gb": 22.11}
|
| 168 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.023371724192472174, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.208984375, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 218}, "mem_gb": 21.93}
|
| 169 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02139054679266798, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.1748046875, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 233}, "mem_gb": 22.08}
|
| 170 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02719521118995423, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.1982421875, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.4, "frames": {"chat": 215}, "mem_gb": 22.1}
|
| 171 |
+
[eval step 140] sample: 'To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(p\\) that satisfy all the equati'
|
| 172 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02396822643155853, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.1865234375, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.5, "frames": {"chat": 233}, "mem_gb": 22.08}
|
| 173 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024230880417240162, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.212890625, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 209}, "mem_gb": 22.04}
|
| 174 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.01931162694086476, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.1689453125, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.1, "frames": {"chat": 262}, "mem_gb": 21.97}
|
| 175 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022601187952638914, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.181640625, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.9, "frames": {"chat": 249}, "mem_gb": 22.06}
|
| 176 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.026810239897008675, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.1875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.9, "frames": {"chat": 227}, "mem_gb": 22.09}
|
| 177 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022271136725856924, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 221}, "mem_gb": 22.09}
|
| 178 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02173946086432164, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.1640625, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.7, "frames": {"chat": 234}, "mem_gb": 22.1}
|
| 179 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022538168500425913, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.1923828125, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 213}, "mem_gb": 22.05}
|
| 180 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022282669592516808, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.16796875, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 213}, "mem_gb": 21.99}
|
| 181 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.020306303256377577, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.1689453125, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 234}, "mem_gb": 22.02}
|
| 182 |
+
[eval step 150] sample: 'To solve the given system of equations with the constraint that each letter represents a non-zero digit, we need to find the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(p\\) that satisfy all the equati'
|
| 183 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep75_s1224/step0150
|
| 184 |
+
wandb: uploading console lines 171-171; updating run metadata
|
| 185 |
+
wandb: uploading config.yaml; uploading output.log; uploading wandb-summary.json
|
| 186 |
+
wandb:
|
| 187 |
+
wandb: Run history:
|
| 188 |
+
wandb: comp_len β
βββββββ
β
ββ
ββββββ
βββ
ββ
βββ
βββ
βββββββ
ββββ
β
|
| 189 |
+
wandb: cumulative_loss_tokens ββββββββββββββββββββββββ
β
β
β
β
ββββββββββββ
|
| 190 |
+
wandb: epoch βββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
ββββββββββββββ
|
| 191 |
+
wandb: finish_rate β
β
β
ββββββββββ
ββ
βββ
ββ
βββββββ
ββββ
βββββββββ
|
| 192 |
+
wandb: forward_topk_kl βββ
β
β
βββββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: grad_norm ββββ
β
β
β
β
β
β
β
βββββββββββββββββ
ββββββββββββ
|
| 194 |
+
wandb: lr ββ
ββββββββββββββββββββββββββββββββββββββ
|
| 195 |
+
wandb: mem_gb βββ
βββββ
ββββββ
βββββββ
βββββββββββ
β
β
ββββ
ββ
|
| 196 |
+
wandb: step ββββββββββββββββββββββββ
β
β
ββββββββββββββ
|
| 197 |
+
wandb: t_data_s βββββββββββββββββββββββοΏ½οΏ½ββββββββββββββββ
|
| 198 |
+
wandb: +3 ...
|
| 199 |
+
wandb:
|
| 200 |
+
wandb: Run summary:
|
| 201 |
+
wandb: comp_len 512.8
|
| 202 |
+
wandb: cumulative_loss_tokens 18000000
|
| 203 |
+
wandb: epoch 2
|
| 204 |
+
wandb: finish_rate 0.906
|
| 205 |
+
wandb: forward_topk_kl 0.02031
|
| 206 |
+
wandb: grad_norm 0.16895
|
| 207 |
+
wandb: lr 3e-05
|
| 208 |
+
wandb: mem_gb 22.02
|
| 209 |
+
wandb: step 150
|
| 210 |
+
wandb: t_data_s 0
|
| 211 |
+
wandb: +4 ...
|
| 212 |
+
wandb:
|
| 213 |
+
wandb: π View run uniform-math-keep75-s1224 at: https://wandb.ai/hbfreed/glean-grid/runs/bknlxtxa
|
| 214 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 215 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 216 |
+
wandb: Find logs at: outputs/healed/grid_math/uniform_keep75_s1224/wandb/run-20260716_151320-bknlxtxa/logs
|
| 217 |
+
{
|
| 218 |
+
"correct": 821,
|
| 219 |
+
"accuracy": 0.6224412433661866,
|
| 220 |
+
"finished": 1311,
|
| 221 |
+
"finish_rate": 0.9939347990902199,
|
| 222 |
+
"mean_completion_tokens": 117.59438968915845
|
| 223 |
+
}
|
| 224 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep75_s1224_step100_chat.json
|
| 225 |
+
{
|
| 226 |
+
"correct": 836,
|
| 227 |
+
"accuracy": 0.6338134950720242,
|
| 228 |
+
"finished": 1313,
|
| 229 |
+
"finish_rate": 0.9954510993176648,
|
| 230 |
+
"mean_completion_tokens": 115.02274450341167
|
| 231 |
+
}
|
| 232 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep75_s1224_step150_chat.json
|
healed/grid_math/uniform_keep75_s1225.console.log
ADDED
|
@@ -0,0 +1,232 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 6 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/uniform_keep75_s1225/wandb/run-20260716_223103-vjebg3kl
|
| 7 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 8 |
+
wandb: Syncing run uniform-math-keep75-s1225
|
| 9 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 10 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/vjebg3kl
|
| 11 |
+
|
| 12 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 5.31B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1358247014944131, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 1.5390625, "lr": 6e-06, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 191}, "mem_gb": 21.94}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid starting at the center. The goal is to identify the four numbers that lie on the same diagonal a'
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12545704158656298, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 1.453125, "lr": 9e-06, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 219}, "mem_gb": 22.09}
|
| 17 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1497652667608112, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 1.8515625, "lr": 1.2e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.7, "frames": {"chat": 207}, "mem_gb": 22.1}
|
| 18 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11752520349826663, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 0.91796875, "lr": 1.5e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 208}, "mem_gb": 22.05}
|
| 19 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10433478212691844, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 0.80859375, "lr": 1.8e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 219}, "mem_gb": 22.09}
|
| 20 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09108569042940313, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 0.78515625, "lr": 2.1e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 246}, "mem_gb": 21.96}
|
| 21 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.106326757963188, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 0.73828125, "lr": 2.4e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 206}, "mem_gb": 22.12}
|
| 22 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0902948998101987, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 0.69140625, "lr": 2.7000000000000002e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 233}, "mem_gb": 22.1}
|
| 23 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08570309399059042, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 0.58984375, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 229}, "mem_gb": 21.96}
|
| 24 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08330574528363843, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.5546875, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 236}, "mem_gb": 22.0}
|
| 25 |
+
[eval step 10] sample: 'To solve this problem, we need to analyze the spiral pattern of numbers from 1 to 49 on a square grid and identify the numbers on the same diagonal as the number 7. Then, we will determine how many of'
|
| 26 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08719355923005691, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.4765625, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 239}, "mem_gb": 21.88}
|
| 27 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07907083735729878, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.458984375, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 241}, "mem_gb": 22.0}
|
| 28 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08512322157975286, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.482421875, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 226}, "mem_gb": 21.97}
|
| 29 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0749055631302918, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.439453125, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 234}, "mem_gb": 22.09}
|
| 30 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06968406668367486, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 257}, "mem_gb": 22.08}
|
| 31 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09542547322846949, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.455078125, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 208}, "mem_gb": 22.14}
|
| 32 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0896531540566745, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 211}, "mem_gb": 22.11}
|
| 33 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07490203162863229, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 227}, "mem_gb": 22.1}
|
| 34 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08470211627905568, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.412109375, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 211}, "mem_gb": 22.07}
|
| 35 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06893499421194817, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.3828125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 238}, "mem_gb": 22.09}
|
| 36 |
+
[eval step 20] sample: "To solve this problem, we need to understand the structure of the spiral pattern and identify the numbers on the same diagonal as \\(7\\). Let's break down the problem step-by-step:\n\n1. **Understand the"
|
| 37 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07017010920361305, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 237}, "mem_gb": 22.13}
|
| 38 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07960431989231147, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.3828125, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.6, "frames": {"chat": 208}, "mem_gb": 22.13}
|
| 39 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06916012099149327, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 221}, "mem_gb": 22.22}
|
| 40 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06673377439201189, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 232}, "mem_gb": 22.05}
|
| 41 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07113247835698228, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.365234375, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 208}, "mem_gb": 22.09}
|
| 42 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06350237686486604, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.837, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 227}, "mem_gb": 22.01}
|
| 43 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07054388405587524, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.357421875, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 221}, "mem_gb": 22.03}
|
| 44 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.061323589913050335, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 232}, "mem_gb": 22.1}
|
| 45 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06432824564288991, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.349609375, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 219}, "mem_gb": 22.1}
|
| 46 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07021584103650724, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.713, "comp_len": 615.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 195}, "mem_gb": 22.19}
|
| 47 |
+
[eval step 30] sample: 'To solve this problem, we need to analyze the spiral pattern of numbers from 1 to 49 on a square grid and identify the numbers that lie on the same diagonal as the number 7. Then, we will determine ho'
|
| 48 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0648433039499602, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.7, "frames": {"chat": 216}, "mem_gb": 22.1}
|
| 49 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.058143819925840945, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.31640625, "lr": 3e-05, "finish_rate": 0.788, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 208}, "mem_gb": 21.98}
|
| 50 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05636492688258489, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.919, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 235}, "mem_gb": 21.98}
|
| 51 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.055969286663721626, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 225}, "mem_gb": 22.08}
|
| 52 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07901875382158906, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.77, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 213}, "mem_gb": 22.18}
|
| 53 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05415514450389892, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 257}, "mem_gb": 21.85}
|
| 54 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06761667520503203, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 212}, "mem_gb": 22.12}
|
| 55 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05876178986076266, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.33203125, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 221}, "mem_gb": 22.1}
|
| 56 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05697968485564925, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 242}, "mem_gb": 22.09}
|
| 57 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0541666944550816, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 220}, "mem_gb": 22.06}
|
| 58 |
+
[eval step 40] sample: 'To solve this problem, we need to analyze the spiral pattern of numbers from 1 to 49 on a square grid and identify the numbers that lie on the same diagonal as the number 7. Then, we will determine ho'
|
| 59 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05260880702113112, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.328125, "lr": 3e-05, "finish_rate": 0.896, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 240}, "mem_gb": 21.95}
|
| 60 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.060863188926534105, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 206}, "mem_gb": 22.08}
|
| 61 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.057264025495760144, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.31640625, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 226}, "mem_gb": 22.1}
|
| 62 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0610422637766848, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.6, "frames": {"chat": 244}, "mem_gb": 21.88}
|
| 63 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06439507137689118, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.34375, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 224}, "mem_gb": 22.1}
|
| 64 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05581830503217255, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.345703125, "lr": 3e-05, "finish_rate": 0.923, "comp_len": 442.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 271}, "mem_gb": 21.82}
|
| 65 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05291223348335673, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.856, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 236}, "mem_gb": 22.11}
|
| 66 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0554617004743622, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.31640625, "lr": 3e-05, "finish_rate": 0.841, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 232}, "mem_gb": 21.97}
|
| 67 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05114069694283729, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.31640625, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 210}, "mem_gb": 22.03}
|
| 68 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05137673323115644, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 217}, "mem_gb": 22.0}
|
| 69 |
+
[eval step 50] sample: 'To solve this problem, we need to understand the structure of the spiral pattern on the grid and identify the numbers that lie on the same diagonal as the number \\(7\\).\n\n### Steps to Solve the Problem'
|
| 70 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep75_s1225/step0050
|
| 71 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06234172496208921, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.34765625, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 224}, "mem_gb": 22.12}
|
| 72 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07160587412401413, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.375, "lr": 3e-05, "finish_rate": 0.749, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 203}, "mem_gb": 21.97}
|
| 73 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05557245409963652, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 239}, "mem_gb": 22.06}
|
| 74 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03414652680471384, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 254}, "mem_gb": 21.98}
|
| 75 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.034510691293003035, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 241}, "mem_gb": 22.07}
|
| 76 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04583843435803428, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 213}, "mem_gb": 22.1}
|
| 77 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03947926825817364, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 221}, "mem_gb": 22.15}
|
| 78 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04116059052028383, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 196}, "mem_gb": 22.11}
|
| 79 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03308698924773683, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.224609375, "lr": 3e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.0, "frames": {"chat": 270}, "mem_gb": 21.91}
|
| 80 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.034069254077334576, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.2392578125, "lr": 3e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 216}, "mem_gb": 22.09}
|
| 81 |
+
[eval step 60] sample: 'To solve this problem, we need to understand the structure of the spiral pattern on the grid and identify the numbers that lie on the same diagonal as the number \\(7\\).\n\n### Step-by-Step Solution:\n\n1.'
|
| 82 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04350406211614609, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.28515625, "lr": 3e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 200}, "mem_gb": 22.06}
|
| 83 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.033727901365452756, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.2412109375, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 206}, "mem_gb": 22.01}
|
| 84 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03015216571188066, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.2314453125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 234}, "mem_gb": 22.04}
|
| 85 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03634754524397819, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.24609375, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 215}, "mem_gb": 22.05}
|
| 86 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.029129735330779415, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.2314453125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 255}, "mem_gb": 22.04}
|
| 87 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03348110796995461, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.306640625, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 250}, "mem_gb": 21.92}
|
| 88 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03267918671743634, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.25390625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 242}, "mem_gb": 22.09}
|
| 89 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0429476497222359, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.306640625, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 199}, "mem_gb": 22.1}
|
| 90 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04425143690695986, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.271484375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 208}, "mem_gb": 22.13}
|
| 91 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03657152016861364, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.3203125, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 208}, "mem_gb": 22.07}
|
| 92 |
+
[eval step 70] sample: 'To solve this problem, we need to understand the structure of the spiral pattern on the square grid and identify the numbers that lie on the same diagonal as the number \\(7\\).\n\n### Step-by-Step Soluti'
|
| 93 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04123211079263128, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 209}, "mem_gb": 22.22}
|
| 94 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.030864056057242364, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.244140625, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.3, "frames": {"chat": 235}, "mem_gb": 22.05}
|
| 95 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.033988771005704375, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.291015625, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 204}, "mem_gb": 22.04}
|
| 96 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.043746256581383446, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.298828125, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 208}, "mem_gb": 22.1}
|
| 97 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.031305211680731734, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.228515625, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 57.3, "frames": {"chat": 240}, "mem_gb": 22.09}
|
| 98 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03433587677682129, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.8, "frames": {"chat": 246}, "mem_gb": 22.09}
|
| 99 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03481989274016426, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.6, "frames": {"chat": 243}, "mem_gb": 21.91}
|
| 100 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.036875585224215565, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.0, "frames": {"chat": 208}, "mem_gb": 22.1}
|
| 101 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03534174345984745, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.2392578125, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.8, "frames": {"chat": 219}, "mem_gb": 22.1}
|
| 102 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.042089351415184016, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.9, "frames": {"chat": 211}, "mem_gb": 22.11}
|
| 103 |
+
[eval step 80] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid and identify the four shaded squares that lie on the same diagonal as the number 7. Then, we wil'
|
| 104 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03182697140184076, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.23828125, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.1, "frames": {"chat": 232}, "mem_gb": 22.07}
|
| 105 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.034447449180250986, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.2392578125, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.7, "frames": {"chat": 214}, "mem_gb": 22.1}
|
| 106 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03462826530029997, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.232421875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.6, "frames": {"chat": 226}, "mem_gb": 21.99}
|
| 107 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0355406848211928, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.2490234375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.1, "frames": {"chat": 210}, "mem_gb": 22.11}
|
| 108 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03192530898133603, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.25, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.9, "frames": {"chat": 218}, "mem_gb": 21.93}
|
| 109 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.030032073534412, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.2080078125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.0, "frames": {"chat": 233}, "mem_gb": 22.08}
|
| 110 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0384073818150054, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.2412109375, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.2, "frames": {"chat": 215}, "mem_gb": 22.1}
|
| 111 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03470420025307685, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.25, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.2, "frames": {"chat": 233}, "mem_gb": 22.08}
|
| 112 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03370989526286721, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.3, "frames": {"chat": 209}, "mem_gb": 22.04}
|
| 113 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.027565786793866814, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.197265625, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 57.9, "frames": {"chat": 262}, "mem_gb": 21.97}
|
| 114 |
+
[eval step 90] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid and identify the four shaded squares that lie on the same diagonal as the number 7. Then, we wil'
|
| 115 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03008150718464361, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.21875, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 56.6, "frames": {"chat": 249}, "mem_gb": 22.06}
|
| 116 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03847663177931681, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 55.6, "frames": {"chat": 227}, "mem_gb": 22.09}
|
| 117 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.032657393121114, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.275390625, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.1, "frames": {"chat": 221}, "mem_gb": 22.09}
|
| 118 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03173800096526587, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.2265625, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.1, "frames": {"chat": 234}, "mem_gb": 22.1}
|
| 119 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03087014857486356, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.2470703125, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.7, "frames": {"chat": 213}, "mem_gb": 22.05}
|
| 120 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.029955804504655924, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.2236328125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.6, "frames": {"chat": 213}, "mem_gb": 21.99}
|
| 121 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.030467382055746083, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.24609375, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 55.1, "frames": {"chat": 234}, "mem_gb": 22.02}
|
| 122 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03673412882296058, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.263671875, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.4, "frames": {"chat": 222}, "mem_gb": 22.09}
|
| 123 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.033574912037172666, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.23046875, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 58.6, "frames": {"chat": 227}, "mem_gb": 22.1}
|
| 124 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.034915446859543835, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.2578125, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 53.7, "frames": {"chat": 218}, "mem_gb": 22.14}
|
| 125 |
+
[eval step 100] sample: 'To solve this problem, we need to understand the structure of the spiral pattern formed by the numbers from 1 to 49 on a square grid. The numbers are arranged in a spiral pattern, starting from the ce'
|
| 126 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep75_s1225/step0100
|
| 127 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.033782800041108084, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.21875, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.7, "frames": {"chat": 223}, "mem_gb": 22.1}
|
| 128 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03598464701743796, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.24609375, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 206}, "mem_gb": 22.1}
|
| 129 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.030618002940326308, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.2412109375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 213}, "mem_gb": 22.02}
|
| 130 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04078237585676834, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.26171875, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 223}, "mem_gb": 21.96}
|
| 131 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03272634071402718, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.2197265625, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 227}, "mem_gb": 22.07}
|
| 132 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03005558260572143, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.2421875, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 253}, "mem_gb": 22.09}
|
| 133 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.026308241902609976, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 216}, "mem_gb": 22.1}
|
| 134 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02579004672653197, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.201171875, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.7, "frames": {"chat": 205}, "mem_gb": 22.07}
|
| 135 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.030623514922133957, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.20703125, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.4, "frames": {"chat": 207}, "mem_gb": 22.16}
|
| 136 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02862321507255547, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.2001953125, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 215}, "mem_gb": 22.08}
|
| 137 |
+
[eval step 110] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid starting from the center. We then identify the four shaded squares that lie on the same diagonal'
|
| 138 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.023055778127916468, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.30078125, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 228}, "mem_gb": 22.1}
|
| 139 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.025379221981678468, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.189453125, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 221}, "mem_gb": 22.14}
|
| 140 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.019185258447751402, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.1748046875, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 254}, "mem_gb": 21.93}
|
| 141 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02264149820668778, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.2275390625, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 210}, "mem_gb": 22.06}
|
| 142 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021573141938253926, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.1650390625, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 226}, "mem_gb": 22.02}
|
| 143 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024346172539202963, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.2138671875, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 212}, "mem_gb": 22.09}
|
| 144 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02802123403872053, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.2060546875, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 211}, "mem_gb": 22.02}
|
| 145 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.026772187854287526, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.1845703125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.1, "frames": {"chat": 196}, "mem_gb": 22.07}
|
| 146 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02290341926627637, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.1884765625, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 212}, "mem_gb": 22.09}
|
| 147 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02147745310171352, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.1796875, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.0, "frames": {"chat": 244}, "mem_gb": 22.0}
|
| 148 |
+
[eval step 120] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid starting from the center. We then identify the four shaded squares that lie on the same diagonal'
|
| 149 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02290404658280313, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.173828125, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 222}, "mem_gb": 22.05}
|
| 150 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0237099109623814, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.2021484375, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.7, "frames": {"chat": 218}, "mem_gb": 22.09}
|
| 151 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02240949104845058, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.1708984375, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 252}, "mem_gb": 21.97}
|
| 152 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02601201236327179, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.1904296875, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.7, "frames": {"chat": 202}, "mem_gb": 22.14}
|
| 153 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0266425287119889, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.1923828125, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 237}, "mem_gb": 22.1}
|
| 154 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02409238005452013, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.17578125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 234}, "mem_gb": 22.08}
|
| 155 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021687637067609466, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.201171875, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.9, "frames": {"chat": 215}, "mem_gb": 22.1}
|
| 156 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.019817282370629255, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.1669921875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 234}, "mem_gb": 22.03}
|
| 157 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.020775739161809907, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.1708984375, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 216}, "mem_gb": 22.08}
|
| 158 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.023808946748528008, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.1865234375, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.5, "frames": {"chat": 210}, "mem_gb": 22.05}
|
| 159 |
+
[eval step 130] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid and identify the four shaded squares that lie on the same diagonal as the number 7. Then, we wil'
|
| 160 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02468363624312915, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.189453125, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.3, "frames": {"chat": 199}, "mem_gb": 22.1}
|
| 161 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.023548281814196766, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.2041015625, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.3, "frames": {"chat": 210}, "mem_gb": 22.11}
|
| 162 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021664702919110036, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.173828125, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 225}, "mem_gb": 22.05}
|
| 163 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021577526232468273, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.173828125, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 253}, "mem_gb": 21.95}
|
| 164 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02105618586166917, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.166015625, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 247}, "mem_gb": 22.07}
|
| 165 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02262616546676339, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.177734375, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 238}, "mem_gb": 22.07}
|
| 166 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024266567853023297, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.1787109375, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 235}, "mem_gb": 22.09}
|
| 167 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.025859414496971295, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.2001953125, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 215}, "mem_gb": 22.06}
|
| 168 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021803223936345116, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 268}, "mem_gb": 22.06}
|
| 169 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024701077461933407, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.1953125, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 228}, "mem_gb": 22.1}
|
| 170 |
+
[eval step 140] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid and identify the four shaded squares that lie on the same diagonal as the number 7. Then, we wil'
|
| 171 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.020695444104556614, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.162109375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 252}, "mem_gb": 22.03}
|
| 172 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.023492466158481936, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 223}, "mem_gb": 22.11}
|
| 173 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.026726572511414998, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.205078125, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 226}, "mem_gb": 22.09}
|
| 174 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02891799058924274, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 208}, "mem_gb": 22.14}
|
| 175 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02066739065583485, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.1865234375, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 240}, "mem_gb": 22.03}
|
| 176 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.023900935378763824, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.1806640625, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 222}, "mem_gb": 22.02}
|
| 177 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02020211691574271, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.1552734375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 236}, "mem_gb": 22.09}
|
| 178 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022802902391382184, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.1923828125, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.4, "frames": {"chat": 217}, "mem_gb": 22.06}
|
| 179 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02172313815508193, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.1923828125, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 252}, "mem_gb": 21.97}
|
| 180 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022857091062360755, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.1875, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 222}, "mem_gb": 22.08}
|
| 181 |
+
[eval step 150] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid and identify the four shaded squares that lie on the same diagonal as the number 7. We then need'
|
| 182 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep75_s1225/step0150
|
| 183 |
+
wandb: updating run metadata
|
| 184 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 185 |
+
wandb: uploading config.yaml
|
| 186 |
+
wandb:
|
| 187 |
+
wandb: Run history:
|
| 188 |
+
wandb: comp_len ββ
βββββββ
ββββββββββββ
ββββ
ββββ
βββ
ββββ
ββ
ββ
|
| 189 |
+
wandb: cumulative_loss_tokens ββββββββββββββββββββ
β
β
β
β
β
βββββββββββββββ
|
| 190 |
+
wandb: epoch ββββββββββββββββ
β
β
β
β
β
β
β
β
β
β
β
β
β
β
ββββββββββ
|
| 191 |
+
wandb: finish_rate βββββββ
ββ
ββ
βββ
ββββββββββ
βββ
ββββ
βββ
β
ββ
βββ
|
| 192 |
+
wandb: forward_topk_kl ββββ
βββ
β
ββββββββββββββββββββββββββββββββ
|
| 193 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 194 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 195 |
+
wandb: mem_gb ββββ
βββββ
ββββββββ
β
ββββ
ββ
βββββ
βββββ
ββ
β
βββ
|
| 196 |
+
wandb: step ββββββββββββββββββββββ
β
β
β
β
β
β
β
βββββββββββ
|
| 197 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββοΏ½οΏ½οΏ½βββ
|
| 198 |
+
wandb: +3 ...
|
| 199 |
+
wandb:
|
| 200 |
+
wandb: Run summary:
|
| 201 |
+
wandb: comp_len 540.5
|
| 202 |
+
wandb: cumulative_loss_tokens 18000000
|
| 203 |
+
wandb: epoch 2
|
| 204 |
+
wandb: finish_rate 0.847
|
| 205 |
+
wandb: forward_topk_kl 0.02286
|
| 206 |
+
wandb: grad_norm 0.1875
|
| 207 |
+
wandb: lr 3e-05
|
| 208 |
+
wandb: mem_gb 22.08
|
| 209 |
+
wandb: step 150
|
| 210 |
+
wandb: t_data_s 0
|
| 211 |
+
wandb: +4 ...
|
| 212 |
+
wandb:
|
| 213 |
+
wandb: π View run uniform-math-keep75-s1225 at: https://wandb.ai/hbfreed/glean-grid/runs/vjebg3kl
|
| 214 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 215 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 216 |
+
wandb: Find logs at: outputs/healed/grid_math/uniform_keep75_s1225/wandb/run-20260716_223103-vjebg3kl/logs
|
| 217 |
+
{
|
| 218 |
+
"correct": 840,
|
| 219 |
+
"accuracy": 0.6368460955269143,
|
| 220 |
+
"finished": 1311,
|
| 221 |
+
"finish_rate": 0.9939347990902199,
|
| 222 |
+
"mean_completion_tokens": 116.58377558756634
|
| 223 |
+
}
|
| 224 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep75_s1225_step100_chat.json
|
| 225 |
+
{
|
| 226 |
+
"correct": 845,
|
| 227 |
+
"accuracy": 0.640636846095527,
|
| 228 |
+
"finished": 1312,
|
| 229 |
+
"finish_rate": 0.9946929492039424,
|
| 230 |
+
"mean_completion_tokens": 113.11978771796815
|
| 231 |
+
}
|
| 232 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep75_s1225_step150_chat.json
|
healed/grid_math/uniform_keep75_s1226.console.log
ADDED
|
@@ -0,0 +1,248 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 6 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/uniform_keep75_s1226/wandb/run-20260716_194207-t8hs7m3b
|
| 7 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 8 |
+
wandb: Syncing run uniform-math-keep75-s1226
|
| 9 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 10 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/t8hs7m3b
|
| 11 |
+
|
| 12 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 5.31B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11920493249129505, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 1.4453125, "lr": 6e-06, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.7, "frames": {"chat": 254}, "mem_gb": 21.82}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: 'To solve this problem, we need to understand the geometric properties involved. The perimeter of a triangle is given as 28 units. The midpoints of its sides are connected by segments, which creates a '
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1316654554096361, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 1.640625, "lr": 9e-06, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 241}, "mem_gb": 22.07}
|
| 17 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 18 |
+
warnings.warn('Grouped GEMM not available.')
|
| 19 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 20 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 21 |
+
wandb: setting up run ta2qddpy
|
| 22 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 23 |
+
wandb: Run data is saved locally in outputs/healed/grid_math/uniform_keep75_s1226/wandb/run-20260716_223215-ta2qddpy
|
| 24 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 25 |
+
wandb: Syncing run uniform-math-keep75-s1226
|
| 26 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-grid
|
| 27 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-grid/runs/ta2qddpy
|
| 28 |
+
|
| 29 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 150 total steps | student params 5.31B | teacher overlap=False
|
| 30 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11920493249129505, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 1.4453125, "lr": 6e-06, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 54.8, "frames": {"chat": 254}, "mem_gb": 21.82}
|
| 31 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 32 |
+
[eval step 1] sample: 'To solve this problem, we need to understand the geometric properties involved. When the midpoints of the sides of a triangle are connected by segments, the segments form a new triangle called the med'
|
| 33 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13180598530409238, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 1.640625, "lr": 9e-06, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 241}, "mem_gb": 22.07}
|
| 34 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.14182352375667542, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 1.40625, "lr": 1.2e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 213}, "mem_gb": 22.1}
|
| 35 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11608811415806412, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 0.94921875, "lr": 1.5e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 221}, "mem_gb": 22.15}
|
| 36 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12302177668878188, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 0.8515625, "lr": 1.8e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 196}, "mem_gb": 22.11}
|
| 37 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0868311198878102, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 0.75, "lr": 2.1e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 270}, "mem_gb": 21.91}
|
| 38 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09967558807097375, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 0.859375, "lr": 2.4e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.7, "frames": {"chat": 216}, "mem_gb": 22.09}
|
| 39 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09974337469547366, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 0.73828125, "lr": 2.7000000000000002e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.7, "frames": {"chat": 200}, "mem_gb": 22.06}
|
| 40 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08927157517972713, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 0.64453125, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.9, "frames": {"chat": 206}, "mem_gb": 22.01}
|
| 41 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0773138673888209, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.54296875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 234}, "mem_gb": 22.04}
|
| 42 |
+
[eval step 10] sample: "To solve this problem, we need to understand the geometric properties of the midpoints of a triangle's sides and how they relate to the perimeter of the resulting triangle.\n\n### Steps to Solve:\n\n1. **"
|
| 43 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08695185078081365, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.51953125, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 215}, "mem_gb": 22.05}
|
| 44 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07132623356043671, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.455078125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 255}, "mem_gb": 22.04}
|
| 45 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0732923347460106, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 250}, "mem_gb": 21.92}
|
| 46 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07680565097226451, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.4296875, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 242}, "mem_gb": 22.09}
|
| 47 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09355738684851676, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.484375, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 199}, "mem_gb": 22.1}
|
| 48 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10224050603043287, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.57421875, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 208}, "mem_gb": 22.13}
|
| 49 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0901082269590348, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.6328125, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 208}, "mem_gb": 22.07}
|
| 50 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09369800321385265, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.46484375, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 209}, "mem_gb": 22.22}
|
| 51 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06841314388358345, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 235}, "mem_gb": 22.05}
|
| 52 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07472310523393874, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.7, "frames": {"chat": 204}, "mem_gb": 22.04}
|
| 53 |
+
[eval step 20] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a triangle.\n\n### Steps to Solve:\n\n1. **Understand the Problem:*'
|
| 54 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09344625266479949, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.55859375, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 208}, "mem_gb": 22.1}
|
| 55 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07106327358403554, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 240}, "mem_gb": 22.09}
|
| 56 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06766866139359773, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.40234375, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 246}, "mem_gb": 22.09}
|
| 57 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06455388144083941, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.384765625, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 243}, "mem_gb": 21.91}
|
| 58 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07318340876198684, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.38671875, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.9, "frames": {"chat": 208}, "mem_gb": 22.1}
|
| 59 |
+
{"step": 26, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07118238478129109, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 0.36328125, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 219}, "mem_gb": 22.1}
|
| 60 |
+
{"step": 27, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0817199091798005, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.782, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 211}, "mem_gb": 22.11}
|
| 61 |
+
{"step": 28, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06760180381714988, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 0.3828125, "lr": 3e-05, "finish_rate": 0.862, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 232}, "mem_gb": 22.07}
|
| 62 |
+
{"step": 29, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0715414209818157, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.365234375, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 214}, "mem_gb": 22.1}
|
| 63 |
+
{"step": 30, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06632242755762612, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.3359375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 226}, "mem_gb": 21.99}
|
| 64 |
+
[eval step 30] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a triangle.\n\n1. **Midpoint Formula:**\n If \\( A, B, C \\) are t'
|
| 65 |
+
{"step": 31, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06462202967116609, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 210}, "mem_gb": 22.11}
|
| 66 |
+
{"step": 32, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06298202505940571, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 218}, "mem_gb": 21.93}
|
| 67 |
+
{"step": 33, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05829383838605136, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 233}, "mem_gb": 22.08}
|
| 68 |
+
{"step": 34, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07446961007329325, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.388671875, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 215}, "mem_gb": 22.1}
|
| 69 |
+
{"step": 35, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06408025780933288, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 233}, "mem_gb": 22.08}
|
| 70 |
+
{"step": 36, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06080306692530091, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.353515625, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 209}, "mem_gb": 22.04}
|
| 71 |
+
{"step": 37, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05366904664818818, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.310546875, "lr": 3e-05, "finish_rate": 0.908, "comp_len": 458.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.9, "frames": {"chat": 262}, "mem_gb": 21.97}
|
| 72 |
+
{"step": 38, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05671166869318113, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.326171875, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 249}, "mem_gb": 22.06}
|
| 73 |
+
{"step": 39, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07106226710767174, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.3671875, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 227}, "mem_gb": 22.09}
|
| 74 |
+
{"step": 40, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05462301931890349, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 221}, "mem_gb": 22.09}
|
| 75 |
+
[eval step 40] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a triangle. This new triangle is known as the medial triangle, '
|
| 76 |
+
{"step": 41, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0582238324320099, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.859, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 234}, "mem_gb": 22.1}
|
| 77 |
+
{"step": 42, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05399296633934913, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.330078125, "lr": 3e-05, "finish_rate": 0.817, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 213}, "mem_gb": 22.05}
|
| 78 |
+
{"step": 43, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05386831222480784, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.298828125, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 213}, "mem_gb": 21.99}
|
| 79 |
+
{"step": 44, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05173095208670323, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 0.314453125, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 234}, "mem_gb": 22.02}
|
| 80 |
+
{"step": 45, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05363421202797132, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.322265625, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 222}, "mem_gb": 22.09}
|
| 81 |
+
{"step": 46, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0622261526023969, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.361328125, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 227}, "mem_gb": 22.1}
|
| 82 |
+
{"step": 47, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05994350822979274, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 218}, "mem_gb": 22.14}
|
| 83 |
+
{"step": 48, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06174757863963023, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 0.341796875, "lr": 3e-05, "finish_rate": 0.798, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 223}, "mem_gb": 22.1}
|
| 84 |
+
{"step": 49, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06565356359497333, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.359375, "lr": 3e-05, "finish_rate": 0.772, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 206}, "mem_gb": 22.1}
|
| 85 |
+
{"step": 50, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05670059742469651, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.33984375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.7, "frames": {"chat": 213}, "mem_gb": 22.02}
|
| 86 |
+
[eval step 50] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a triangle. This new triangle is known as the medial triangle, '
|
| 87 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep75_s1226/step0050
|
| 88 |
+
{"step": 51, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07032583207286273, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 0.357421875, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 223}, "mem_gb": 21.96}
|
| 89 |
+
{"step": 52, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.059393302978233746, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 0.333984375, "lr": 3e-05, "finish_rate": 0.828, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 227}, "mem_gb": 22.07}
|
| 90 |
+
{"step": 53, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.05184000949463807, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 0.32421875, "lr": 3e-05, "finish_rate": 0.889, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 253}, "mem_gb": 22.09}
|
| 91 |
+
{"step": 54, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.038828376208990815, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.792, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 216}, "mem_gb": 22.1}
|
| 92 |
+
{"step": 55, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.039238105912537624, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 0.255859375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.0, "frames": {"chat": 205}, "mem_gb": 22.07}
|
| 93 |
+
{"step": 56, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.047572107557269434, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 0.2890625, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 207}, "mem_gb": 22.16}
|
| 94 |
+
{"step": 57, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04661659479457885, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 0.28125, "lr": 3e-05, "finish_rate": 0.814, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 215}, "mem_gb": 22.08}
|
| 95 |
+
{"step": 58, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.034201694558095186, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 0.27734375, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.8, "frames": {"chat": 228}, "mem_gb": 22.1}
|
| 96 |
+
{"step": 59, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04092416759391005, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.747, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.4, "frames": {"chat": 221}, "mem_gb": 22.14}
|
| 97 |
+
{"step": 60, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.029803095571851977, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 0.2138671875, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 254}, "mem_gb": 21.93}
|
| 98 |
+
[eval step 60] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a triangle. This new triangle is known as the medial triangle, '
|
| 99 |
+
{"step": 61, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03530521812000467, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 0.28515625, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 210}, "mem_gb": 22.06}
|
| 100 |
+
{"step": 62, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.033094389010050025, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 0.21484375, "lr": 3e-05, "finish_rate": 0.827, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.1, "frames": {"chat": 226}, "mem_gb": 22.02}
|
| 101 |
+
{"step": 63, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03637227460097832, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 212}, "mem_gb": 22.09}
|
| 102 |
+
{"step": 64, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.041987014663917945, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.754, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 211}, "mem_gb": 22.02}
|
| 103 |
+
{"step": 65, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03841204237483131, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 0.26953125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.3, "frames": {"chat": 196}, "mem_gb": 22.07}
|
| 104 |
+
{"step": 66, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03594996488027585, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 0.267578125, "lr": 3e-05, "finish_rate": 0.811, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 212}, "mem_gb": 22.09}
|
| 105 |
+
{"step": 67, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.032274080974639706, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 0.2177734375, "lr": 3e-05, "finish_rate": 0.877, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 244}, "mem_gb": 22.0}
|
| 106 |
+
{"step": 68, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03448318504060929, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 0.2392578125, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 222}, "mem_gb": 22.05}
|
| 107 |
+
{"step": 69, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.035316669923362014, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 0.232421875, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 218}, "mem_gb": 22.09}
|
| 108 |
+
{"step": 70, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0348655257951313, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 0.2353515625, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 252}, "mem_gb": 21.97}
|
| 109 |
+
[eval step 70] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a given triangle.\n\n### Steps to Solve:\n\n1. **Understand the Pro'
|
| 110 |
+
{"step": 71, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03988810336939059, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 0.26171875, "lr": 3e-05, "finish_rate": 0.728, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 202}, "mem_gb": 22.14}
|
| 111 |
+
{"step": 72, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04227977843079716, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 0.29296875, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.8, "frames": {"chat": 237}, "mem_gb": 22.1}
|
| 112 |
+
{"step": 73, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03656210743497747, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 0.234375, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 234}, "mem_gb": 22.08}
|
| 113 |
+
{"step": 74, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03134735133058081, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 0.25, "lr": 3e-05, "finish_rate": 0.809, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 215}, "mem_gb": 22.1}
|
| 114 |
+
{"step": 75, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03046899615646495, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 0.216796875, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.2, "frames": {"chat": 234}, "mem_gb": 22.03}
|
| 115 |
+
{"step": 76, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.029838484331507546, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 0.2099609375, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.1, "frames": {"chat": 216}, "mem_gb": 22.08}
|
| 116 |
+
{"step": 77, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03392047594260269, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 0.248046875, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 210}, "mem_gb": 22.05}
|
| 117 |
+
{"step": 78, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03507764483672411, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 0.25, "lr": 3e-05, "finish_rate": 0.719, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 199}, "mem_gb": 22.1}
|
| 118 |
+
{"step": 79, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03239439557224202, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 0.2216796875, "lr": 3e-05, "finish_rate": 0.824, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 210}, "mem_gb": 22.11}
|
| 119 |
+
{"step": 80, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03286070772272845, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 533.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.0, "frames": {"chat": 225}, "mem_gb": 22.05}
|
| 120 |
+
[eval step 80] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a given triangle.\n\n### Steps to Solve:\n\n1. **Understand the Mid'
|
| 121 |
+
{"step": 81, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.032128432613688834, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 0.25, "lr": 3e-05, "finish_rate": 0.913, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 253}, "mem_gb": 21.95}
|
| 122 |
+
{"step": 82, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.031100997417598652, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 0.21875, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.2, "frames": {"chat": 247}, "mem_gb": 22.07}
|
| 123 |
+
{"step": 83, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.0324900306135416, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 0.2412109375, "lr": 3e-05, "finish_rate": 0.836, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.5, "frames": {"chat": 238}, "mem_gb": 22.07}
|
| 124 |
+
{"step": 84, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03485131102278829, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 0.2451171875, "lr": 3e-05, "finish_rate": 0.86, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 235}, "mem_gb": 22.09}
|
| 125 |
+
{"step": 85, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03912405168330297, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 0.265625, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 215}, "mem_gb": 22.06}
|
| 126 |
+
{"step": 86, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.033098834516915185, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 0.28125, "lr": 3e-05, "finish_rate": 0.925, "comp_len": 447.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 268}, "mem_gb": 22.06}
|
| 127 |
+
{"step": 87, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.034645479888437934, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 0.232421875, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 228}, "mem_gb": 22.1}
|
| 128 |
+
{"step": 88, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03237436599015103, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 0.240234375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 252}, "mem_gb": 22.03}
|
| 129 |
+
{"step": 89, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03435483168452047, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.821, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.3, "frames": {"chat": 223}, "mem_gb": 22.11}
|
| 130 |
+
{"step": 90, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04248241032837735, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 0.294921875, "lr": 3e-05, "finish_rate": 0.805, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 226}, "mem_gb": 22.09}
|
| 131 |
+
[eval step 90] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a triangle.\n\n### Steps to Solve the Problem:\n\n1. **Understand t'
|
| 132 |
+
{"step": 91, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.04142716025705449, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 0.283203125, "lr": 3e-05, "finish_rate": 0.731, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 208}, "mem_gb": 22.14}
|
| 133 |
+
{"step": 92, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.031166394373402, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 0.251953125, "lr": 3e-05, "finish_rate": 0.883, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.8, "frames": {"chat": 240}, "mem_gb": 22.03}
|
| 134 |
+
{"step": 93, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03443516800537861, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 0.23046875, "lr": 3e-05, "finish_rate": 0.842, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.9, "frames": {"chat": 222}, "mem_gb": 22.02}
|
| 135 |
+
{"step": 94, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.030224586171346407, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 0.2333984375, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 236}, "mem_gb": 22.09}
|
| 136 |
+
{"step": 95, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03236451368537576, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 0.2353515625, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 217}, "mem_gb": 22.06}
|
| 137 |
+
{"step": 96, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.034898156173845445, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 0.263671875, "lr": 3e-05, "finish_rate": 0.921, "comp_len": 476.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.3, "frames": {"chat": 252}, "mem_gb": 21.97}
|
| 138 |
+
{"step": 97, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.02913232143558562, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 0.2255859375, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 540.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.8, "frames": {"chat": 222}, "mem_gb": 22.08}
|
| 139 |
+
{"step": 98, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.029040355308757475, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 0.21484375, "lr": 3e-05, "finish_rate": 0.901, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.6, "frames": {"chat": 242}, "mem_gb": 21.97}
|
| 140 |
+
{"step": 99, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.041233757554925976, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 0.2734375, "lr": 3e-05, "finish_rate": 0.804, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.1, "frames": {"chat": 219}, "mem_gb": 22.03}
|
| 141 |
+
{"step": 100, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.031569313709802614, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 0.2138671875, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 223}, "mem_gb": 22.04}
|
| 142 |
+
[eval step 100] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a given triangle.\n\n### Steps to Solve:\n\n1. **Understand the Pro'
|
| 143 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep75_s1226/step0100
|
| 144 |
+
{"step": 101, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03175070390204589, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 0.228515625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.5, "frames": {"chat": 232}, "mem_gb": 22.04}
|
| 145 |
+
{"step": 102, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03924315899686578, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 0.28125, "lr": 3e-05, "finish_rate": 0.832, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 220}, "mem_gb": 22.09}
|
| 146 |
+
{"step": 103, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03732912681673964, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 0.236328125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 210}, "mem_gb": 22.14}
|
| 147 |
+
{"step": 104, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.03158854953125119, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 0.205078125, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 226}, "mem_gb": 22.06}
|
| 148 |
+
{"step": 105, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.033342002084447694, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 0.259765625, "lr": 3e-05, "finish_rate": 0.741, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 212}, "mem_gb": 22.09}
|
| 149 |
+
{"step": 106, "epoch": 1, "training_mode": "off-policy", "forward_topk_kl": 0.030606075126398354, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 0.232421875, "lr": 3e-05, "finish_rate": 0.839, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.3, "frames": {"chat": 236}, "mem_gb": 22.1}
|
| 150 |
+
{"step": 107, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021876435576969135, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 0.2001953125, "lr": 3e-05, "finish_rate": 0.928, "comp_len": 454.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.7, "frames": {"chat": 264}, "mem_gb": 21.97}
|
| 151 |
+
{"step": 108, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.031198739765095525, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 0.21875, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 229}, "mem_gb": 22.07}
|
| 152 |
+
{"step": 109, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02165067300312221, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 0.1806640625, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 465.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 258}, "mem_gb": 21.95}
|
| 153 |
+
{"step": 110, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.028640677657909692, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 0.193359375, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.1, "frames": {"chat": 208}, "mem_gb": 22.11}
|
| 154 |
+
[eval step 110] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a given triangle.\n\n### Steps to Solve:\n\n1. **Understand the Pro'
|
| 155 |
+
{"step": 111, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022872452891978902, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.88, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.0, "frames": {"chat": 249}, "mem_gb": 22.02}
|
| 156 |
+
{"step": 112, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02100934052404482, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 0.19140625, "lr": 3e-05, "finish_rate": 0.845, "comp_len": 545.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.8, "frames": {"chat": 220}, "mem_gb": 22.09}
|
| 157 |
+
{"step": 113, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024147335090193275, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 0.1806640625, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.2, "frames": {"chat": 223}, "mem_gb": 22.08}
|
| 158 |
+
{"step": 114, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024052943600167055, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 0.201171875, "lr": 3e-05, "finish_rate": 0.833, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.7, "frames": {"chat": 221}, "mem_gb": 22.09}
|
| 159 |
+
{"step": 115, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02166470361133882, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 0.162109375, "lr": 3e-05, "finish_rate": 0.9, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.5, "frames": {"chat": 230}, "mem_gb": 22.0}
|
| 160 |
+
{"step": 116, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02461200471681077, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 0.171875, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.2, "frames": {"chat": 214}, "mem_gb": 22.07}
|
| 161 |
+
{"step": 117, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.03192179344429945, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 0.224609375, "lr": 3e-05, "finish_rate": 0.766, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.2, "frames": {"chat": 214}, "mem_gb": 22.08}
|
| 162 |
+
{"step": 118, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.027875469126327275, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 0.2041015625, "lr": 3e-05, "finish_rate": 0.786, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.4, "frames": {"chat": 210}, "mem_gb": 22.13}
|
| 163 |
+
{"step": 119, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.028064899455732668, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 0.1845703125, "lr": 3e-05, "finish_rate": 0.776, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 45.7, "frames": {"chat": 214}, "mem_gb": 22.09}
|
| 164 |
+
{"step": 120, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02598108633084533, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 0.185546875, "lr": 3e-05, "finish_rate": 0.791, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.0, "frames": {"chat": 215}, "mem_gb": 22.05}
|
| 165 |
+
[eval step 120] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a given triangle.\n\n### Steps to Solve:\n\n1. **Understand the Pro'
|
| 166 |
+
{"step": 121, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.029542491081776097, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 0.2060546875, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.9, "frames": {"chat": 208}, "mem_gb": 22.09}
|
| 167 |
+
{"step": 122, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.023529452410464485, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 0.201171875, "lr": 3e-05, "finish_rate": 0.789, "comp_len": 550.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 218}, "mem_gb": 21.97}
|
| 168 |
+
{"step": 123, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021247040836008577, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 0.16015625, "lr": 3e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 233}, "mem_gb": 21.99}
|
| 169 |
+
{"step": 124, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02059493543021381, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 0.173828125, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 231}, "mem_gb": 22.03}
|
| 170 |
+
{"step": 125, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024373427555907982, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 0.1845703125, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.0, "frames": {"chat": 235}, "mem_gb": 22.22}
|
| 171 |
+
{"step": 126, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02307887886830916, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 0.1650390625, "lr": 3e-05, "finish_rate": 0.843, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 216}, "mem_gb": 22.08}
|
| 172 |
+
{"step": 127, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02126138604179335, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 0.162109375, "lr": 3e-05, "finish_rate": 0.831, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.3, "frames": {"chat": 237}, "mem_gb": 22.1}
|
| 173 |
+
{"step": 128, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.029153408732265233, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 0.2001953125, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 591.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 203}, "mem_gb": 22.11}
|
| 174 |
+
{"step": 129, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02260789417422687, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 0.17578125, "lr": 3e-05, "finish_rate": 0.873, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.6, "frames": {"chat": 236}, "mem_gb": 22.13}
|
| 175 |
+
{"step": 130, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022835769553498055, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 0.1728515625, "lr": 3e-05, "finish_rate": 0.734, "comp_len": 560.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 214}, "mem_gb": 22.1}
|
| 176 |
+
[eval step 130] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a triangle. This new triangle is known as the medial triangle, '
|
| 177 |
+
{"step": 131, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.026547468842783323, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 0.189453125, "lr": 3e-05, "finish_rate": 0.78, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.3, "frames": {"chat": 209}, "mem_gb": 22.09}
|
| 178 |
+
{"step": 132, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022498934920489166, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 0.1826171875, "lr": 3e-05, "finish_rate": 0.906, "comp_len": 468.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 52.3, "frames": {"chat": 256}, "mem_gb": 22.1}
|
| 179 |
+
{"step": 133, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021946324599829193, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 0.1787109375, "lr": 3e-05, "finish_rate": 0.878, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.6, "frames": {"chat": 230}, "mem_gb": 21.96}
|
| 180 |
+
{"step": 134, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024039489145313078, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.822, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.9, "frames": {"chat": 230}, "mem_gb": 22.15}
|
| 181 |
+
{"step": 135, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02589866460277699, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 0.1796875, "lr": 3e-05, "finish_rate": 0.881, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.8, "frames": {"chat": 227}, "mem_gb": 22.05}
|
| 182 |
+
{"step": 136, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.023664263719754913, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 0.166015625, "lr": 3e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 208}, "mem_gb": 22.11}
|
| 183 |
+
{"step": 137, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.027114476467870796, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 0.1796875, "lr": 3e-05, "finish_rate": 0.699, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.0, "frames": {"chat": 206}, "mem_gb": 22.12}
|
| 184 |
+
{"step": 138, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02399630657126351, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 0.1767578125, "lr": 3e-05, "finish_rate": 0.82, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.7, "frames": {"chat": 228}, "mem_gb": 22.0}
|
| 185 |
+
{"step": 139, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02343961157395194, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 0.181640625, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.9, "frames": {"chat": 224}, "mem_gb": 22.09}
|
| 186 |
+
{"step": 140, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.024890781568270178, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.66, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 200}, "mem_gb": 22.13}
|
| 187 |
+
[eval step 140] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a given triangle.\n\n### Steps to Solve the Problem:\n\n1. **Unders'
|
| 188 |
+
{"step": 141, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02478467862996428, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 0.1904296875, "lr": 3e-05, "finish_rate": 0.714, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.1, "frames": {"chat": 196}, "mem_gb": 22.11}
|
| 189 |
+
{"step": 142, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022272795487005108, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 0.169921875, "lr": 3e-05, "finish_rate": 0.834, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 223}, "mem_gb": 22.09}
|
| 190 |
+
{"step": 143, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.022915525561606045, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 0.18359375, "lr": 3e-05, "finish_rate": 0.869, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 213}, "mem_gb": 21.98}
|
| 191 |
+
{"step": 144, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.0212597823954653, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 0.1953125, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.4, "frames": {"chat": 232}, "mem_gb": 22.02}
|
| 192 |
+
{"step": 145, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.020988663152477237, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 0.1796875, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.2, "frames": {"chat": 223}, "mem_gb": 22.02}
|
| 193 |
+
{"step": 146, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021486785067555806, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 0.1630859375, "lr": 3e-05, "finish_rate": 0.85, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.0, "frames": {"chat": 233}, "mem_gb": 22.11}
|
| 194 |
+
{"step": 147, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02550031017503546, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 0.2021484375, "lr": 3e-05, "finish_rate": 0.816, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.8, "frames": {"chat": 217}, "mem_gb": 22.11}
|
| 195 |
+
{"step": 148, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.030582861898102175, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 0.1962890625, "lr": 3e-05, "finish_rate": 0.752, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.1, "frames": {"chat": 202}, "mem_gb": 22.17}
|
| 196 |
+
{"step": 149, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.021075663964031262, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 0.1640625, "lr": 3e-05, "finish_rate": 0.858, "comp_len": 474.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.1, "frames": {"chat": 253}, "mem_gb": 22.03}
|
| 197 |
+
{"step": 150, "epoch": 2, "training_mode": "off-policy", "forward_topk_kl": 0.02149440993195555, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 0.30859375, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.8, "frames": {"chat": 231}, "mem_gb": 22.03}
|
| 198 |
+
[eval step 150] sample: 'To solve this problem, we need to understand the geometric properties of the triangle formed by connecting the midpoints of the sides of a given triangle.\n\n### Steps to Solve:\n\n1. **Understand the Pro'
|
| 199 |
+
checkpoint snapshot queued -> outputs/healed/grid_math/uniform_keep75_s1226/step0150
|
| 200 |
+
wandb: updating run metadata
|
| 201 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml
|
| 202 |
+
wandb:
|
| 203 |
+
wandb: Run history:
|
| 204 |
+
wandb: comp_len ββ
ββββ
ββββββββββββββββββββββββββββββββββ
|
| 205 |
+
wandb: cumulative_loss_tokens βββββββββββββββββββ
β
β
β
β
β
ββββββββββββββββ
|
| 206 |
+
wandb: epoch ββββββββββββββββ
β
β
β
β
β
β
β
β
β
β
ββββββββββββββ
|
| 207 |
+
wandb: finish_rate βββββββββββββββββββββββββββ
βββ
β
ββββββ
β
ββ
|
| 208 |
+
wandb: forward_topk_kl βββ
βββββββββββββββββββββββββββββββββββββ
|
| 209 |
+
wandb: grad_norm ββββββββββββββββββββββββββββββββββββββββ
|
| 210 |
+
wandb: lr ββββββββββββββββββββββββββββββββββββββββ
|
| 211 |
+
wandb: mem_gb β
ββ
βββ
ββ
β
β
β
β
β
ββ
β
ββββ
β
βββ
β
βββββ
ββ
ββ
ββββ
ββ
|
| 212 |
+
wandb: step ββββββββββββββββββββββββββ
β
β
ββββββββββββ
|
| 213 |
+
wandb: t_data_s ββββββββββββββββββββββββββββββββββββββββ
|
| 214 |
+
wandb: +3 ...
|
| 215 |
+
wandb:
|
| 216 |
+
wandb: Run summary:
|
| 217 |
+
wandb: comp_len 519.5
|
| 218 |
+
wandb: cumulative_loss_tokens 18000000
|
| 219 |
+
wandb: epoch 2
|
| 220 |
+
wandb: finish_rate 0.879
|
| 221 |
+
wandb: forward_topk_kl 0.02149
|
| 222 |
+
wandb: grad_norm 0.30859
|
| 223 |
+
wandb: lr 3e-05
|
| 224 |
+
wandb: mem_gb 22.03
|
| 225 |
+
wandb: step 150
|
| 226 |
+
wandb: t_data_s 0
|
| 227 |
+
wandb: +4 ...
|
| 228 |
+
wandb:
|
| 229 |
+
wandb: π View run uniform-math-keep75-s1226 at: https://wandb.ai/hbfreed/glean-grid/runs/ta2qddpy
|
| 230 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-grid
|
| 231 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 232 |
+
wandb: Find logs at: outputs/healed/grid_math/uniform_keep75_s1226/wandb/run-20260716_223215-ta2qddpy/logs
|
| 233 |
+
{
|
| 234 |
+
"correct": 826,
|
| 235 |
+
"accuracy": 0.6262319939347991,
|
| 236 |
+
"finished": 1314,
|
| 237 |
+
"finish_rate": 0.9962092494313874,
|
| 238 |
+
"mean_completion_tokens": 114.73464746019712
|
| 239 |
+
}
|
| 240 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep75_s1226_step100_chat.json
|
| 241 |
+
{
|
| 242 |
+
"correct": 834,
|
| 243 |
+
"accuracy": 0.6322971948445792,
|
| 244 |
+
"finished": 1317,
|
| 245 |
+
"finish_rate": 0.9984836997725549,
|
| 246 |
+
"mean_completion_tokens": 114.99090219863533
|
| 247 |
+
}
|
| 248 |
+
saved item-level results -> outputs/evals/grid_math/uniform_keep75_s1226_step150_chat.json
|
healed/grid_math/worker_s1226.log
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2026-07-16T00:08:10-07:00 [s1226] glean_keep50_s1226 already done, skip
|
| 2 |
+
2026-07-16T00:08:10-07:00 [s1226] healing uniform_keep50_s1226 on GPU-864c54df
|
| 3 |
+
2026-07-16T01:41:59-07:00 [s1226] eval uniform_keep50_s1226 step100
|
| 4 |
+
2026-07-16T01:44:01-07:00 [s1226] eval uniform_keep50_s1226 step150
|
| 5 |
+
2026-07-16T01:45:55-07:00 [s1226] uniform_keep50_s1226 done -> 0.5049279757391963
|
| 6 |
+
2026-07-16T01:45:55-07:00 [s1226] healing reap_keep50_s1226 on GPU-864c54df
|
| 7 |
+
2026-07-16T03:44:03-07:00 [s1226] eval reap_keep50_s1226 step100
|
| 8 |
+
2026-07-16T03:45:49-07:00 [s1226] eval reap_keep50_s1226 step150
|
| 9 |
+
2026-07-16T03:47:29-07:00 [s1226] reap_keep50_s1226 done -> 0.5913570887035633
|
| 10 |
+
2026-07-16T03:47:29-07:00 [s1226] healing glean_keep25_s1226 on GPU-864c54df
|
| 11 |
+
2026-07-16T05:11:36-07:00 [s1226] eval glean_keep25_s1226 step100
|
| 12 |
+
2026-07-16T05:13:54-07:00 [s1226] eval glean_keep25_s1226 step150
|
| 13 |
+
2026-07-16T05:16:13-07:00 [s1226] glean_keep25_s1226 done -> 0.4359363153904473
|
| 14 |
+
2026-07-16T05:16:13-07:00 [s1226] healing uniform_keep25_s1226 on GPU-864c54df
|
| 15 |
+
2026-07-16T06:29:43-07:00 [s1226] eval uniform_keep25_s1226 step100
|
| 16 |
+
2026-07-16T06:31:48-07:00 [s1226] eval uniform_keep25_s1226 step150
|
| 17 |
+
2026-07-16T06:33:45-07:00 [s1226] uniform_keep25_s1226 done -> 0.22744503411675512
|
| 18 |
+
2026-07-16T06:33:45-07:00 [s1226] healing reap_keep25_s1226 on GPU-864c54df
|
| 19 |
+
2026-07-16T08:23:55-07:00 [s1226] eval reap_keep25_s1226 step100
|
| 20 |
+
2026-07-16T08:26:19-07:00 [s1226] eval reap_keep25_s1226 step150
|
| 21 |
+
2026-07-16T08:28:37-07:00 [s1226] reap_keep25_s1226 done -> 0.12357846853677028
|
| 22 |
+
2026-07-16T08:28:38-07:00 [s1226] healing glean_keep75_s1226 on GPU-864c54df
|
healed/liger_mb8/args.json
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"student": "outputs/pruned/glean-0125inst-math-keep50",
|
| 3 |
+
"teacher": "allenai/OLMoE-1B-7B-0125-Instruct",
|
| 4 |
+
"training_mode": "on-policy",
|
| 5 |
+
"kl_direction": "reverse",
|
| 6 |
+
"dataset": "allenai/Dolci-Instruct-RL",
|
| 7 |
+
"dataset_sources": null,
|
| 8 |
+
"max_difficulty": null,
|
| 9 |
+
"trajectories": "outputs/teacher_trajectories/dolci_math_curated.jsonl",
|
| 10 |
+
"trajectory_dataset": "allenai/Dolci-Instruct-RL",
|
| 11 |
+
"off_policy_frames": "chat",
|
| 12 |
+
"off_policy_max_seq_len": 2048,
|
| 13 |
+
"topk_targets": null,
|
| 14 |
+
"max_loss_tokens": null,
|
| 15 |
+
"loss_tokens_per_step": null,
|
| 16 |
+
"teacher_device": "cuda:0",
|
| 17 |
+
"student_device": "cuda:1",
|
| 18 |
+
"lr": 3e-05,
|
| 19 |
+
"optimizer": "adamw8bit",
|
| 20 |
+
"weight_decay": 0.1,
|
| 21 |
+
"epochs": 1,
|
| 22 |
+
"prompts_per_step": 64,
|
| 23 |
+
"group_size": 4,
|
| 24 |
+
"rollout_batch": 64,
|
| 25 |
+
"micro_batch": 8,
|
| 26 |
+
"max_new_tokens": 2048,
|
| 27 |
+
"max_prompt_len": 1024,
|
| 28 |
+
"warmup_steps": 10,
|
| 29 |
+
"max_grad_norm": 1.0,
|
| 30 |
+
"eval_every": 10,
|
| 31 |
+
"gsm8k_every": 0,
|
| 32 |
+
"gsm8k_n": 256,
|
| 33 |
+
"gsm8k_batch": 16,
|
| 34 |
+
"gsm8k_max_new_tokens": 512,
|
| 35 |
+
"gsm8k_frames": "chat",
|
| 36 |
+
"save_every": 1000,
|
| 37 |
+
"out_dir": "outputs/healed/liger_mb8",
|
| 38 |
+
"sweep": 4,
|
| 39 |
+
"wandb": false,
|
| 40 |
+
"wandb_project": "glean-heal",
|
| 41 |
+
"wandb_run_name": null,
|
| 42 |
+
"wandb_run_id": null,
|
| 43 |
+
"wandb_resume": null,
|
| 44 |
+
"wandb_mode": "offline",
|
| 45 |
+
"no_wandb_sync": true,
|
| 46 |
+
"debug": false,
|
| 47 |
+
"resume_from": null,
|
| 48 |
+
"start_step": 0,
|
| 49 |
+
"no_grad_checkpointing": false,
|
| 50 |
+
"seed": 1223,
|
| 51 |
+
"no_teacher_overlap": false,
|
| 52 |
+
"sync_checkpoints": false,
|
| 53 |
+
"rollout_engine": "vllm",
|
| 54 |
+
"vllm_gpu": "2",
|
| 55 |
+
"vllm_port": 8377,
|
| 56 |
+
"vllm_refresh_every": 1,
|
| 57 |
+
"vllm_serve_bin": "vllm-plugin/.venv/bin/python",
|
| 58 |
+
"vllm_gpu_mem_util": 0.85,
|
| 59 |
+
"liger_loss": true,
|
| 60 |
+
"gold_mix_lambda": 0.5,
|
| 61 |
+
"gold_topk_targets": "outputs/teacher_trajectories/dolci_combined_top128",
|
| 62 |
+
"gold_mix_decay": 0.0,
|
| 63 |
+
"fast_teacher": true,
|
| 64 |
+
"reference_kl_beta": 0.05,
|
| 65 |
+
"drop_truncated_rollouts": false,
|
| 66 |
+
"vllm_max_model_len": null,
|
| 67 |
+
"vllm_refresh_mode": "reload",
|
| 68 |
+
"vllm_live_dir": null,
|
| 69 |
+
"resolved_kl_direction": "reverse"
|
| 70 |
+
}
|
healed/liger_mb8/train_log.jsonl
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"step": 1, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6749841483754777, "tokens": 39094, "cumulative_loss_tokens": 39094, "grad_norm": 5.34375, "lr": 6e-06, "finish_rate": 1.0, "comp_len": 610.8, "dropped_truncated": 0, "gold_loss": 0.2627, "gold_lambda": 0.5, "rep_ratio": 2.53, "t_data_s": 0.0, "t_rollout_s": 25.4, "t_step_s": 70.5, "t_refresh_s": 0.3, "mem_gb": 12.14, "mem_gb_teacher": 20.77}
|
| 2 |
+
{"step": 2, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7909456714061366, "tokens": 36996, "cumulative_loss_tokens": 76090, "grad_norm": 6.625, "lr": 9e-06, "finish_rate": 0.984, "comp_len": 578.1, "dropped_truncated": 0, "gold_loss": 0.3092, "gold_lambda": 0.5, "rep_ratio": 2.287, "t_data_s": 0.0, "t_rollout_s": 29.7, "t_step_s": 57.0, "t_refresh_s": 0.3, "mem_gb": 11.99, "mem_gb_teacher": 21.07}
|
| 3 |
+
{"step": 3, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7638091775618511, "tokens": 50596, "cumulative_loss_tokens": 126686, "grad_norm": 5.5, "lr": 1.2e-05, "finish_rate": 1.0, "comp_len": 790.6, "dropped_truncated": 0, "gold_loss": 0.2744, "gold_lambda": 0.5, "rep_ratio": 2.382, "t_data_s": 0.0, "t_rollout_s": 29.5, "t_step_s": 59.7, "t_refresh_s": 0.3, "mem_gb": 12.08, "mem_gb_teacher": 20.96}
|
| 4 |
+
{"step": 4, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3501273521456831, "tokens": 46928, "cumulative_loss_tokens": 173614, "grad_norm": 1.8828125, "lr": 1.5e-05, "finish_rate": 0.984, "comp_len": 733.2, "dropped_truncated": 0, "gold_loss": 0.2477, "gold_lambda": 0.5, "rep_ratio": 2.585, "t_data_s": 0.0, "t_rollout_s": 30.6, "t_step_s": 49.4, "t_refresh_s": 0.0, "mem_gb": 12.07, "mem_gb_teacher": 21.07}
|
healed/liger_mb8/vllm_server.log
ADDED
|
@@ -0,0 +1,309 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Skipping import of cpp extensions due to incompatible torch version 2.10.0+cu128 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
|
| 2 |
+
WARNING 07-30 21:52:23 [registry.py:915] Model architecture OlmoeForCausalLM is already registered, and will be overwritten by the new model class glean_vllm.pruned_olmoe:PrunedOlmoeForCausalLM.
|
| 3 |
+
(APIServer pid=2023926) INFO 07-30 21:52:23 [utils.py:299]
|
| 4 |
+
(APIServer pid=2023926) INFO 07-30 21:52:23 [utils.py:299] β β ββ ββ
|
| 5 |
+
(APIServer pid=2023926) INFO 07-30 21:52:23 [utils.py:299] ββ ββ β β β βββ β version 0.19.0
|
| 6 |
+
(APIServer pid=2023926) INFO 07-30 21:52:23 [utils.py:299] ββββ β β β β model outputs/pruned/glean-0125inst-math-keep50
|
| 7 |
+
(APIServer pid=2023926) INFO 07-30 21:52:23 [utils.py:299] ββ βββββ βββββ β β
|
| 8 |
+
(APIServer pid=2023926) INFO 07-30 21:52:23 [utils.py:299]
|
| 9 |
+
(APIServer pid=2023926) INFO 07-30 21:52:23 [utils.py:233] non-default args: {'model_tag': 'outputs/pruned/glean-0125inst-math-keep50', 'host': '127.0.0.1', 'port': 8377, 'model': 'outputs/pruned/glean-0125inst-math-keep50', 'max_model_len': 3200, 'enforce_eager': True, 'served_model_name': ['student'], 'gpu_memory_utilization': 0.85}
|
| 10 |
+
(APIServer pid=2023926) INFO 07-30 21:52:31 [model.py:549] Resolved architecture: OlmoeForCausalLM
|
| 11 |
+
(APIServer pid=2023926) INFO 07-30 21:52:31 [model.py:1678] Using max model len 3200
|
| 12 |
+
(APIServer pid=2023926) INFO 07-30 21:52:31 [vllm.py:790] Asynchronous scheduling is enabled.
|
| 13 |
+
(APIServer pid=2023926) WARNING 07-30 21:52:31 [vllm.py:848] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
| 14 |
+
(APIServer pid=2023926) WARNING 07-30 21:52:31 [vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
| 15 |
+
(APIServer pid=2023926) INFO 07-30 21:52:31 [vllm.py:1025] Cudagraph is disabled under eager mode
|
| 16 |
+
(APIServer pid=2023926) INFO 07-30 21:52:31 [compilation.py:290] Enabled custom fusions: norm_quant, act_quant
|
| 17 |
+
Skipping import of cpp extensions due to incompatible torch version 2.10.0+cu128 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
|
| 18 |
+
(EngineCore pid=2024236) WARNING 07-30 21:52:39 [registry.py:915] Model architecture OlmoeForCausalLM is already registered, and will be overwritten by the new model class glean_vllm.pruned_olmoe:PrunedOlmoeForCausalLM.
|
| 19 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:39 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='outputs/pruned/glean-0125inst-math-keep50', speculative_config=None, tokenizer='outputs/pruned/glean-0125inst-math-keep50', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=3200, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=student, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
|
| 20 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:39 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.0.15:35469 backend=nccl
|
| 21 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:39 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
|
| 22 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:40 [gpu_model_runner.py:4735] Starting to load model outputs/pruned/glean-0125inst-math-keep50...
|
| 23 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:40 [cuda.py:334] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
| 24 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:40 [flash_attn.py:596] Using FlashAttention version 2
|
| 25 |
+
(EngineCore pid=2024236)
|
| 26 |
+
(EngineCore pid=2024236)
|
| 27 |
+
(EngineCore pid=2024236)
|
| 28 |
+
(EngineCore pid=2024236)
|
| 29 |
+
(EngineCore pid=2024236)
|
| 30 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:46 [default_loader.py:384] Loading weights took 5.30 seconds
|
| 31 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:46 [gpu_model_runner.py:4820] Model loading took 6.89 GiB memory and 5.883005 seconds
|
| 32 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:48 [gpu_worker.py:436] Available KV cache memory: 12.86 GiB
|
| 33 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:48 [kv_cache_utils.py:1319] GPU KV cache size: 105,344 tokens
|
| 34 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:48 [kv_cache_utils.py:1324] Maximum concurrency for 3,200 tokens per request: 32.92x
|
| 35 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:48 [core.py:283] init engine (profile, create kv cache, warmup model) took 1.68 seconds
|
| 36 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:49 [vllm.py:790] Asynchronous scheduling is enabled.
|
| 37 |
+
(EngineCore pid=2024236) WARNING 07-30 21:52:49 [vllm.py:848] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
| 38 |
+
(EngineCore pid=2024236) WARNING 07-30 21:52:49 [vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
| 39 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:49 [vllm.py:1025] Cudagraph is disabled under eager mode
|
| 40 |
+
(EngineCore pid=2024236) INFO 07-30 21:52:49 [compilation.py:290] Enabled custom fusions: norm_quant, act_quant
|
| 41 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [api_server.py:590] Supported tasks: ['generate']
|
| 42 |
+
(APIServer pid=2023926) WARNING 07-30 21:52:49 [__init__.py:14] SECURITY WARNING: Development endpoints are enabled! This should NOT be used in production!
|
| 43 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
| 44 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [api_server.py:594] Starting vLLM server on http://127.0.0.1:8377
|
| 45 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:37] Available routes are:
|
| 46 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
|
| 47 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /docs, Methods: HEAD, GET
|
| 48 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
|
| 49 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
|
| 50 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /sleep, Methods: POST
|
| 51 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /wake_up, Methods: POST
|
| 52 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /is_sleeping, Methods: GET
|
| 53 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /collective_rpc, Methods: POST
|
| 54 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /reset_prefix_cache, Methods: POST
|
| 55 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /reset_mm_cache, Methods: POST
|
| 56 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /reset_encoder_cache, Methods: POST
|
| 57 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /tokenize, Methods: POST
|
| 58 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /detokenize, Methods: POST
|
| 59 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /load, Methods: GET
|
| 60 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /version, Methods: GET
|
| 61 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /health, Methods: GET
|
| 62 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /metrics, Methods: GET
|
| 63 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /server_info, Methods: GET
|
| 64 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/models, Methods: GET
|
| 65 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /ping, Methods: GET
|
| 66 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /ping, Methods: POST
|
| 67 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /invocations, Methods: POST
|
| 68 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
| 69 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
|
| 70 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/responses, Methods: POST
|
| 71 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
| 72 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
| 73 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/completions, Methods: POST
|
| 74 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/messages, Methods: POST
|
| 75 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
|
| 76 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
| 77 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /pause, Methods: POST
|
| 78 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /resume, Methods: POST
|
| 79 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /is_paused, Methods: GET
|
| 80 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /init_weight_transfer_engine, Methods: POST
|
| 81 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /update_weights, Methods: POST
|
| 82 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /get_world_size, Methods: GET
|
| 83 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
| 84 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
| 85 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
|
| 86 |
+
(APIServer pid=2023926) INFO 07-30 21:52:49 [launcher.py:46] Route: /v1/completions/render, Methods: POST
|
| 87 |
+
(APIServer pid=2023926) INFO: Started server process [2023926]
|
| 88 |
+
(APIServer pid=2023926) INFO: Waiting for application startup.
|
| 89 |
+
(APIServer pid=2023926) INFO: Application startup complete.
|
| 90 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:49758 - "GET /health HTTP/1.1" 200 OK
|
| 91 |
+
(APIServer pid=2023926) INFO 07-30 21:53:29 [loggers.py:259] Engine 000: Avg prompt throughput: 227.6 tokens/s, Avg generation throughput: 2542.0 tokens/s, Running: 32 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.8%, Prefix cache hit rate: 68.5%
|
| 92 |
+
(APIServer pid=2023926) INFO 07-30 21:53:39 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1247.2 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.6%, Prefix cache hit rate: 68.5%
|
| 93 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:50568 - "POST /v1/completions HTTP/1.1" 200 OK
|
| 94 |
+
(APIServer pid=2023926) INFO 07-30 21:53:49 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 119.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 68.5%
|
| 95 |
+
(APIServer pid=2023926) INFO 07-30 21:53:59 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 68.5%
|
| 96 |
+
(EngineCore pid=2024236) INFO 07-30 21:54:30 [gpu_model_runner.py:4957] Reloading weights inplace...
|
| 97 |
+
(EngineCore pid=2024236)
|
| 98 |
+
(EngineCore pid=2024236)
|
| 99 |
+
(EngineCore pid=2024236)
|
| 100 |
+
(EngineCore pid=2024236)
|
| 101 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeForCausalLM: Failed to load weights
|
| 102 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeModel: Failed to load weights
|
| 103 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] ModuleList: Failed to load weights
|
| 104 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 105 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 106 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] RotaryEmbedding: Failed to load weights
|
| 107 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] ApplyRotaryEmb: Failed to load weights
|
| 108 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 109 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 110 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 111 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 112 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 113 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 114 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 115 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 116 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 117 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 118 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 119 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 120 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 121 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 122 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 123 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 124 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 125 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 126 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 127 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 128 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 129 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 130 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 131 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 132 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 133 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 134 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 135 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 136 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 137 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 138 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 139 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 140 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 141 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 142 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 143 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 144 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 145 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 146 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 147 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 148 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 149 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 150 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 151 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 152 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 153 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 154 |
+
(EngineCore pid=2024236) WARNING 07-30 21:54:31 [layerwise.py:230] LogitsProcessor: Failed to load weights
|
| 155 |
+
(EngineCore pid=2024236) INFO 07-30 21:54:31 [gpu_model_runner.py:4980] Reloading and processing weights took 0.29 seconds
|
| 156 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:35576 - "POST /collective_rpc HTTP/1.1" 200 OK
|
| 157 |
+
(APIServer pid=2023926) INFO 07-30 21:54:31 [api_router.py:39] Resetting prefix cache...
|
| 158 |
+
(EngineCore pid=2024236) INFO 07-30 21:54:31 [block_pool.py:472] Successfully reset prefix cache
|
| 159 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:35586 - "POST /reset_prefix_cache HTTP/1.1" 200 OK
|
| 160 |
+
(APIServer pid=2023926) INFO 07-30 21:54:39 [loggers.py:259] Engine 000: Avg prompt throughput: 230.8 tokens/s, Avg generation throughput: 2472.1 tokens/s, Running: 34 reqs, Waiting: 0 reqs, GPU KV cache usage: 18.2%, Prefix cache hit rate: 69.6%
|
| 161 |
+
(APIServer pid=2023926) INFO 07-30 21:54:49 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 982.5 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 69.6%
|
| 162 |
+
(APIServer pid=2023926) INFO 07-30 21:54:59 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 231.2 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.4%, Prefix cache hit rate: 69.6%
|
| 163 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:35596 - "POST /v1/completions HTTP/1.1" 200 OK
|
| 164 |
+
(APIServer pid=2023926) INFO 07-30 21:55:09 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.4 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.6%
|
| 165 |
+
(APIServer pid=2023926) INFO 07-30 21:55:19 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.6%
|
| 166 |
+
(EngineCore pid=2024236)
|
| 167 |
+
(EngineCore pid=2024236)
|
| 168 |
+
(EngineCore pid=2024236)
|
| 169 |
+
(EngineCore pid=2024236)
|
| 170 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeForCausalLM: Failed to load weights
|
| 171 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeModel: Failed to load weights
|
| 172 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] ModuleList: Failed to load weights
|
| 173 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 174 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 175 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] RotaryEmbedding: Failed to load weights
|
| 176 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] ApplyRotaryEmb: Failed to load weights
|
| 177 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 178 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 179 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 180 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 181 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 182 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 183 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 184 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 185 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 186 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 187 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 188 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 189 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 190 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 191 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 192 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 193 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 194 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 195 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 196 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 197 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 198 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 199 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 200 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 201 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 202 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 203 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 204 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 205 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 206 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 207 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 208 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 209 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 210 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 211 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 212 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 213 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 214 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 215 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 216 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 217 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 218 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 219 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 220 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 221 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 222 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 223 |
+
(EngineCore pid=2024236) WARNING 07-30 21:55:28 [layerwise.py:230] LogitsProcessor: Failed to load weights
|
| 224 |
+
(EngineCore pid=2024236) INFO 07-30 21:55:28 [gpu_model_runner.py:4980] Reloading and processing weights took 0.28 seconds
|
| 225 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:56816 - "POST /collective_rpc HTTP/1.1" 200 OK
|
| 226 |
+
(APIServer pid=2023926) INFO 07-30 21:55:28 [api_router.py:39] Resetting prefix cache...
|
| 227 |
+
(EngineCore pid=2024236) INFO 07-30 21:55:28 [block_pool.py:472] Successfully reset prefix cache
|
| 228 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:56824 - "POST /reset_prefix_cache HTTP/1.1" 200 OK
|
| 229 |
+
(APIServer pid=2023926) INFO 07-30 21:55:29 [loggers.py:259] Engine 000: Avg prompt throughput: 321.6 tokens/s, Avg generation throughput: 350.1 tokens/s, Running: 64 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.0%, Prefix cache hit rate: 71.2%
|
| 230 |
+
(APIServer pid=2023926) INFO 07-30 21:55:39 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2986.2 tokens/s, Running: 37 reqs, Waiting: 0 reqs, GPU KV cache usage: 24.6%, Prefix cache hit rate: 71.2%
|
| 231 |
+
(APIServer pid=2023926) INFO 07-30 21:55:49 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1458.6 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.9%, Prefix cache hit rate: 71.2%
|
| 232 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:56828 - "POST /v1/completions HTTP/1.1" 200 OK
|
| 233 |
+
(APIServer pid=2023926) INFO 07-30 21:55:59 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 264.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 71.2%
|
| 234 |
+
(APIServer pid=2023926) INFO 07-30 21:56:09 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 71.2%
|
| 235 |
+
(EngineCore pid=2024236)
|
| 236 |
+
(EngineCore pid=2024236)
|
| 237 |
+
(EngineCore pid=2024236)
|
| 238 |
+
(EngineCore pid=2024236)
|
| 239 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeForCausalLM: Failed to load weights
|
| 240 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeModel: Failed to load weights
|
| 241 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] ModuleList: Failed to load weights
|
| 242 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 243 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 244 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] RotaryEmbedding: Failed to load weights
|
| 245 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] ApplyRotaryEmb: Failed to load weights
|
| 246 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 247 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 248 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 249 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 250 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 251 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 252 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 253 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 254 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 255 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 256 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 257 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 258 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 259 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 260 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 261 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 262 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 263 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 264 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 265 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 266 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 267 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 268 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 269 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 270 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 271 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 272 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 273 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 274 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 275 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 276 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 277 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 278 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 279 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 280 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 281 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 282 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 283 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 284 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 285 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 286 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 287 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 288 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 289 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 290 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 291 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 292 |
+
(EngineCore pid=2024236) WARNING 07-30 21:56:28 [layerwise.py:230] LogitsProcessor: Failed to load weights
|
| 293 |
+
(EngineCore pid=2024236) INFO 07-30 21:56:28 [gpu_model_runner.py:4980] Reloading and processing weights took 0.29 seconds
|
| 294 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:45308 - "POST /collective_rpc HTTP/1.1" 200 OK
|
| 295 |
+
(APIServer pid=2023926) INFO 07-30 21:56:28 [api_router.py:39] Resetting prefix cache...
|
| 296 |
+
(EngineCore pid=2024236) INFO 07-30 21:56:28 [block_pool.py:472] Successfully reset prefix cache
|
| 297 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:45312 - "POST /reset_prefix_cache HTTP/1.1" 200 OK
|
| 298 |
+
(APIServer pid=2023926) INFO 07-30 21:56:29 [loggers.py:259] Engine 000: Avg prompt throughput: 210.8 tokens/s, Avg generation throughput: 503.3 tokens/s, Running: 62 reqs, Waiting: 0 reqs, GPU KV cache usage: 7.0%, Prefix cache hit rate: 67.4%
|
| 299 |
+
(APIServer pid=2023926) INFO 07-30 21:56:39 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2969.1 tokens/s, Running: 30 reqs, Waiting: 0 reqs, GPU KV cache usage: 21.3%, Prefix cache hit rate: 67.4%
|
| 300 |
+
(APIServer pid=2023926) INFO 07-30 21:56:49 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1059.4 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.8%, Prefix cache hit rate: 67.4%
|
| 301 |
+
(APIServer pid=2023926) INFO: 127.0.0.1:45320 - "POST /v1/completions HTTP/1.1" 200 OK
|
| 302 |
+
(APIServer pid=2023926) INFO 07-30 21:56:59 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 160.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 67.4%
|
| 303 |
+
(APIServer pid=2023926) INFO 07-30 21:57:09 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 67.4%
|
| 304 |
+
(EngineCore pid=2024236) INFO 07-30 21:57:28 [core.py:1210] Shutdown initiated (timeout=0)
|
| 305 |
+
(EngineCore pid=2024236) INFO 07-30 21:57:28 [core.py:1233] Shutdown complete
|
| 306 |
+
(APIServer pid=2023926) INFO: Shutting down
|
| 307 |
+
(APIServer pid=2023926) INFO: Waiting for application shutdown.
|
| 308 |
+
(APIServer pid=2023926) INFO: Application shutdown complete.
|
| 309 |
+
(APIServer pid=2023926) INFO: Finished server process [2023926]
|
healed/liger_smoke/args.json
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"student": "outputs/pruned/glean-0125inst-math-keep50",
|
| 3 |
+
"teacher": "allenai/OLMoE-1B-7B-0125-Instruct",
|
| 4 |
+
"training_mode": "on-policy",
|
| 5 |
+
"kl_direction": "reverse",
|
| 6 |
+
"dataset": "allenai/Dolci-Instruct-RL",
|
| 7 |
+
"dataset_sources": null,
|
| 8 |
+
"max_difficulty": null,
|
| 9 |
+
"trajectories": "outputs/teacher_trajectories/dolci_math_curated.jsonl",
|
| 10 |
+
"trajectory_dataset": "allenai/Dolci-Instruct-RL",
|
| 11 |
+
"off_policy_frames": "chat",
|
| 12 |
+
"off_policy_max_seq_len": 2048,
|
| 13 |
+
"topk_targets": null,
|
| 14 |
+
"max_loss_tokens": null,
|
| 15 |
+
"loss_tokens_per_step": null,
|
| 16 |
+
"teacher_device": "cuda:0",
|
| 17 |
+
"student_device": "cuda:1",
|
| 18 |
+
"lr": 3e-05,
|
| 19 |
+
"optimizer": "adamw8bit",
|
| 20 |
+
"weight_decay": 0.1,
|
| 21 |
+
"epochs": 1,
|
| 22 |
+
"prompts_per_step": 64,
|
| 23 |
+
"group_size": 4,
|
| 24 |
+
"rollout_batch": 64,
|
| 25 |
+
"micro_batch": 4,
|
| 26 |
+
"max_new_tokens": 2048,
|
| 27 |
+
"max_prompt_len": 1024,
|
| 28 |
+
"warmup_steps": 10,
|
| 29 |
+
"max_grad_norm": 1.0,
|
| 30 |
+
"eval_every": 10,
|
| 31 |
+
"gsm8k_every": 0,
|
| 32 |
+
"gsm8k_n": 256,
|
| 33 |
+
"gsm8k_batch": 16,
|
| 34 |
+
"gsm8k_max_new_tokens": 512,
|
| 35 |
+
"gsm8k_frames": "chat",
|
| 36 |
+
"save_every": 1000,
|
| 37 |
+
"out_dir": "outputs/healed/liger_smoke",
|
| 38 |
+
"sweep": 6,
|
| 39 |
+
"wandb": false,
|
| 40 |
+
"wandb_project": "glean-heal",
|
| 41 |
+
"wandb_run_name": null,
|
| 42 |
+
"wandb_run_id": null,
|
| 43 |
+
"wandb_resume": null,
|
| 44 |
+
"wandb_mode": "offline",
|
| 45 |
+
"no_wandb_sync": true,
|
| 46 |
+
"debug": false,
|
| 47 |
+
"resume_from": null,
|
| 48 |
+
"start_step": 0,
|
| 49 |
+
"no_grad_checkpointing": false,
|
| 50 |
+
"seed": 1223,
|
| 51 |
+
"no_teacher_overlap": false,
|
| 52 |
+
"sync_checkpoints": false,
|
| 53 |
+
"rollout_engine": "vllm",
|
| 54 |
+
"vllm_gpu": "2",
|
| 55 |
+
"vllm_port": 8377,
|
| 56 |
+
"vllm_refresh_every": 1,
|
| 57 |
+
"vllm_serve_bin": "vllm-plugin/.venv/bin/python",
|
| 58 |
+
"vllm_gpu_mem_util": 0.85,
|
| 59 |
+
"liger_loss": true,
|
| 60 |
+
"gold_mix_lambda": 0.5,
|
| 61 |
+
"gold_topk_targets": "outputs/teacher_trajectories/dolci_combined_top128",
|
| 62 |
+
"gold_mix_decay": 0.0,
|
| 63 |
+
"fast_teacher": true,
|
| 64 |
+
"reference_kl_beta": 0.05,
|
| 65 |
+
"drop_truncated_rollouts": false,
|
| 66 |
+
"vllm_max_model_len": null,
|
| 67 |
+
"vllm_refresh_mode": "reload",
|
| 68 |
+
"vllm_live_dir": null,
|
| 69 |
+
"resolved_kl_direction": "reverse"
|
| 70 |
+
}
|
healed/liger_smoke/train_log.jsonl
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"step": 1, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6858642641264342, "tokens": 43978, "cumulative_loss_tokens": 43978, "grad_norm": 5.0625, "lr": 6e-06, "finish_rate": 0.984, "comp_len": 687.2, "dropped_truncated": 0, "gold_loss": 0.2626, "gold_lambda": 0.5, "rep_ratio": 2.514, "t_data_s": 0.0, "t_rollout_s": 31.1, "t_step_s": 74.1, "t_refresh_s": 0.3, "mem_gb": 10.19, "mem_gb_teacher": 20.57}
|
| 2 |
+
{"step": 2, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7085006050567664, "tokens": 38599, "cumulative_loss_tokens": 82577, "grad_norm": 5.28125, "lr": 9e-06, "finish_rate": 1.0, "comp_len": 603.1, "dropped_truncated": 0, "gold_loss": 0.3081, "gold_lambda": 0.5, "rep_ratio": 2.303, "t_data_s": 0.0, "t_rollout_s": 28.1, "t_step_s": 54.7, "t_refresh_s": 0.3, "mem_gb": 10.42, "mem_gb_teacher": 20.43}
|
| 3 |
+
{"step": 3, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8436827146367599, "tokens": 55709, "cumulative_loss_tokens": 138286, "grad_norm": 6.3125, "lr": 1.2e-05, "finish_rate": 0.984, "comp_len": 870.5, "dropped_truncated": 0, "gold_loss": 0.275, "gold_lambda": 0.5, "rep_ratio": 2.41, "t_data_s": 0.0, "t_rollout_s": 32.9, "t_step_s": 63.3, "t_refresh_s": 0.3, "mem_gb": 10.16, "mem_gb_teacher": 20.41}
|
| 4 |
+
{"step": 4, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34322795073287565, "tokens": 47483, "cumulative_loss_tokens": 185769, "grad_norm": 1.7109375, "lr": 1.5e-05, "finish_rate": 0.984, "comp_len": 741.9, "dropped_truncated": 0, "gold_loss": 0.245, "gold_lambda": 0.5, "rep_ratio": 2.659, "t_data_s": 0.0, "t_rollout_s": 30.7, "t_step_s": 59.3, "t_refresh_s": 0.3, "mem_gb": 10.46, "mem_gb_teacher": 20.44}
|
| 5 |
+
{"step": 5, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8553304715360663, "tokens": 41777, "cumulative_loss_tokens": 227546, "grad_norm": 6.4375, "lr": 1.8e-05, "finish_rate": 0.984, "comp_len": 652.8, "dropped_truncated": 0, "gold_loss": 0.274, "gold_lambda": 0.5, "rep_ratio": 2.447, "t_data_s": 0.0, "t_rollout_s": 30.7, "t_step_s": 58.6, "t_refresh_s": 0.3, "mem_gb": 10.37, "mem_gb_teacher": 20.39}
|
| 6 |
+
{"step": 6, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3281100414068494, "tokens": 44463, "cumulative_loss_tokens": 272009, "grad_norm": 1.4375, "lr": 2.1e-05, "finish_rate": 1.0, "comp_len": 694.7, "dropped_truncated": 0, "gold_loss": 0.2263, "gold_lambda": 0.5, "rep_ratio": 2.555, "t_data_s": 0.0, "t_rollout_s": 24.5, "t_step_s": 39.6, "t_refresh_s": 0.0, "mem_gb": 10.05, "mem_gb_teacher": 20.26}
|
healed/liger_smoke/vllm_server.log
ADDED
|
@@ -0,0 +1,447 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Skipping import of cpp extensions due to incompatible torch version 2.10.0+cu128 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
|
| 2 |
+
WARNING 07-30 20:41:37 [registry.py:915] Model architecture OlmoeForCausalLM is already registered, and will be overwritten by the new model class glean_vllm.pruned_olmoe:PrunedOlmoeForCausalLM.
|
| 3 |
+
(APIServer pid=2018548) INFO 07-30 20:41:37 [utils.py:299]
|
| 4 |
+
(APIServer pid=2018548) INFO 07-30 20:41:37 [utils.py:299] β β ββ ββ
|
| 5 |
+
(APIServer pid=2018548) INFO 07-30 20:41:37 [utils.py:299] ββ ββ β β β βββ β version 0.19.0
|
| 6 |
+
(APIServer pid=2018548) INFO 07-30 20:41:37 [utils.py:299] ββββ β β β β model outputs/pruned/glean-0125inst-math-keep50
|
| 7 |
+
(APIServer pid=2018548) INFO 07-30 20:41:37 [utils.py:299] ββ βββββ βββββ β β
|
| 8 |
+
(APIServer pid=2018548) INFO 07-30 20:41:37 [utils.py:299]
|
| 9 |
+
(APIServer pid=2018548) INFO 07-30 20:41:37 [utils.py:233] non-default args: {'model_tag': 'outputs/pruned/glean-0125inst-math-keep50', 'host': '127.0.0.1', 'port': 8377, 'model': 'outputs/pruned/glean-0125inst-math-keep50', 'max_model_len': 3200, 'enforce_eager': True, 'served_model_name': ['student'], 'gpu_memory_utilization': 0.85}
|
| 10 |
+
(APIServer pid=2018548) INFO 07-30 20:41:45 [model.py:549] Resolved architecture: OlmoeForCausalLM
|
| 11 |
+
(APIServer pid=2018548) INFO 07-30 20:41:45 [model.py:1678] Using max model len 3200
|
| 12 |
+
(APIServer pid=2018548) INFO 07-30 20:41:45 [vllm.py:790] Asynchronous scheduling is enabled.
|
| 13 |
+
(APIServer pid=2018548) WARNING 07-30 20:41:45 [vllm.py:848] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
| 14 |
+
(APIServer pid=2018548) WARNING 07-30 20:41:45 [vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
| 15 |
+
(APIServer pid=2018548) INFO 07-30 20:41:45 [vllm.py:1025] Cudagraph is disabled under eager mode
|
| 16 |
+
(APIServer pid=2018548) INFO 07-30 20:41:45 [compilation.py:290] Enabled custom fusions: norm_quant, act_quant
|
| 17 |
+
Skipping import of cpp extensions due to incompatible torch version 2.10.0+cu128 for torchao version 0.15.0 Please see https://github.com/pytorch/ao/issues/2919 for more info
|
| 18 |
+
(EngineCore pid=2018864) WARNING 07-30 20:41:53 [registry.py:915] Model architecture OlmoeForCausalLM is already registered, and will be overwritten by the new model class glean_vllm.pruned_olmoe:PrunedOlmoeForCausalLM.
|
| 19 |
+
(EngineCore pid=2018864) INFO 07-30 20:41:53 [core.py:105] Initializing a V1 LLM engine (v0.19.0) with config: model='outputs/pruned/glean-0125inst-math-keep50', speculative_config=None, tokenizer='outputs/pruned/glean-0125inst-math-keep50', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=3200, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, enforce_eager=True, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=student, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.NONE: 0>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['all'], 'splitting_ops': [], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_images_per_batch': 0, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.NONE: 0>, 'cudagraph_num_of_warmups': 0, 'cudagraph_capture_sizes': [], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 0, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}
|
| 20 |
+
(EngineCore pid=2018864) INFO 07-30 20:41:53 [parallel_state.py:1400] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.0.15:40655 backend=nccl
|
| 21 |
+
(EngineCore pid=2018864) INFO 07-30 20:41:53 [parallel_state.py:1716] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
|
| 22 |
+
(EngineCore pid=2018864) INFO 07-30 20:41:54 [gpu_model_runner.py:4735] Starting to load model outputs/pruned/glean-0125inst-math-keep50...
|
| 23 |
+
(EngineCore pid=2018864) INFO 07-30 20:41:55 [cuda.py:334] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
|
| 24 |
+
(EngineCore pid=2018864) INFO 07-30 20:41:55 [flash_attn.py:596] Using FlashAttention version 2
|
| 25 |
+
(EngineCore pid=2018864)
|
| 26 |
+
(EngineCore pid=2018864)
|
| 27 |
+
(EngineCore pid=2018864)
|
| 28 |
+
(EngineCore pid=2018864)
|
| 29 |
+
(EngineCore pid=2018864)
|
| 30 |
+
(EngineCore pid=2018864) INFO 07-30 20:42:00 [default_loader.py:384] Loading weights took 4.98 seconds
|
| 31 |
+
(EngineCore pid=2018864) INFO 07-30 20:42:00 [gpu_model_runner.py:4820] Model loading took 6.89 GiB memory and 5.559024 seconds
|
| 32 |
+
(EngineCore pid=2018864) INFO 07-30 20:42:02 [gpu_worker.py:436] Available KV cache memory: 12.86 GiB
|
| 33 |
+
(EngineCore pid=2018864) INFO 07-30 20:42:02 [kv_cache_utils.py:1319] GPU KV cache size: 105,344 tokens
|
| 34 |
+
(EngineCore pid=2018864) INFO 07-30 20:42:02 [kv_cache_utils.py:1324] Maximum concurrency for 3,200 tokens per request: 32.92x
|
| 35 |
+
(EngineCore pid=2018864) INFO 07-30 20:42:02 [core.py:283] init engine (profile, create kv cache, warmup model) took 1.67 seconds
|
| 36 |
+
(EngineCore pid=2018864) INFO 07-30 20:42:02 [vllm.py:790] Asynchronous scheduling is enabled.
|
| 37 |
+
(EngineCore pid=2018864) WARNING 07-30 20:42:02 [vllm.py:848] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
|
| 38 |
+
(EngineCore pid=2018864) WARNING 07-30 20:42:02 [vllm.py:859] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
|
| 39 |
+
(EngineCore pid=2018864) INFO 07-30 20:42:02 [vllm.py:1025] Cudagraph is disabled under eager mode
|
| 40 |
+
(EngineCore pid=2018864) INFO 07-30 20:42:02 [compilation.py:290] Enabled custom fusions: norm_quant, act_quant
|
| 41 |
+
(APIServer pid=2018548) INFO 07-30 20:42:02 [api_server.py:590] Supported tasks: ['generate']
|
| 42 |
+
(APIServer pid=2018548) WARNING 07-30 20:42:02 [__init__.py:14] SECURITY WARNING: Development endpoints are enabled! This should NOT be used in production!
|
| 43 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [hf.py:314] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
|
| 44 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [api_server.py:594] Starting vLLM server on http://127.0.0.1:8377
|
| 45 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:37] Available routes are:
|
| 46 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
|
| 47 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /docs, Methods: GET, HEAD
|
| 48 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
|
| 49 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
|
| 50 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /sleep, Methods: POST
|
| 51 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /wake_up, Methods: POST
|
| 52 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /is_sleeping, Methods: GET
|
| 53 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /collective_rpc, Methods: POST
|
| 54 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /reset_prefix_cache, Methods: POST
|
| 55 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /reset_mm_cache, Methods: POST
|
| 56 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /reset_encoder_cache, Methods: POST
|
| 57 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /tokenize, Methods: POST
|
| 58 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /detokenize, Methods: POST
|
| 59 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /load, Methods: GET
|
| 60 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /version, Methods: GET
|
| 61 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /health, Methods: GET
|
| 62 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /metrics, Methods: GET
|
| 63 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /server_info, Methods: GET
|
| 64 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/models, Methods: GET
|
| 65 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /ping, Methods: GET
|
| 66 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /ping, Methods: POST
|
| 67 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /invocations, Methods: POST
|
| 68 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
|
| 69 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
|
| 70 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/responses, Methods: POST
|
| 71 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
|
| 72 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
|
| 73 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/completions, Methods: POST
|
| 74 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/messages, Methods: POST
|
| 75 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
|
| 76 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
|
| 77 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /pause, Methods: POST
|
| 78 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /resume, Methods: POST
|
| 79 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /is_paused, Methods: GET
|
| 80 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /init_weight_transfer_engine, Methods: POST
|
| 81 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /update_weights, Methods: POST
|
| 82 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /get_world_size, Methods: GET
|
| 83 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
|
| 84 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
|
| 85 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
|
| 86 |
+
(APIServer pid=2018548) INFO 07-30 20:42:03 [launcher.py:46] Route: /v1/completions/render, Methods: POST
|
| 87 |
+
(APIServer pid=2018548) INFO: Started server process [2018548]
|
| 88 |
+
(APIServer pid=2018548) INFO: Waiting for application startup.
|
| 89 |
+
(APIServer pid=2018548) INFO: Application startup complete.
|
| 90 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:44630 - "GET /health HTTP/1.1" 200 OK
|
| 91 |
+
(APIServer pid=2018548) INFO 07-30 20:42:43 [loggers.py:259] Engine 000: Avg prompt throughput: 227.6 tokens/s, Avg generation throughput: 2709.9 tokens/s, Running: 31 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 68.5%
|
| 92 |
+
(APIServer pid=2018548) INFO 07-30 20:42:53 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1257.7 tokens/s, Running: 13 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.2%, Prefix cache hit rate: 68.5%
|
| 93 |
+
(APIServer pid=2018548) INFO 07-30 20:43:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 414.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.1%, Prefix cache hit rate: 68.5%
|
| 94 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:56364 - "POST /v1/completions HTTP/1.1" 200 OK
|
| 95 |
+
(APIServer pid=2018548) INFO 07-30 20:43:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 68.5%
|
| 96 |
+
(APIServer pid=2018548) INFO 07-30 20:43:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 68.5%
|
| 97 |
+
(EngineCore pid=2018864) INFO 07-30 20:43:47 [gpu_model_runner.py:4957] Reloading weights inplace...
|
| 98 |
+
(EngineCore pid=2018864)
|
| 99 |
+
(EngineCore pid=2018864)
|
| 100 |
+
(EngineCore pid=2018864)
|
| 101 |
+
(EngineCore pid=2018864)
|
| 102 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeForCausalLM: Failed to load weights
|
| 103 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeModel: Failed to load weights
|
| 104 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] ModuleList: Failed to load weights
|
| 105 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 106 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 107 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] RotaryEmbedding: Failed to load weights
|
| 108 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] ApplyRotaryEmb: Failed to load weights
|
| 109 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 110 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 111 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 112 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 113 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 114 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 115 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 116 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 117 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 118 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 119 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 120 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 121 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 122 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 123 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 124 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 125 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 126 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 127 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 128 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 129 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 130 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 131 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 132 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 133 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 134 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 135 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 136 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 137 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 138 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 139 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 140 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 141 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 142 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 143 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 144 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 145 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 146 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 147 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 148 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 149 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 150 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 151 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 152 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 153 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 154 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 155 |
+
(EngineCore pid=2018864) WARNING 07-30 20:43:47 [layerwise.py:230] LogitsProcessor: Failed to load weights
|
| 156 |
+
(EngineCore pid=2018864) INFO 07-30 20:43:47 [gpu_model_runner.py:4980] Reloading and processing weights took 0.30 seconds
|
| 157 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:51956 - "POST /collective_rpc HTTP/1.1" 200 OK
|
| 158 |
+
(APIServer pid=2018548) INFO 07-30 20:43:47 [api_router.py:39] Resetting prefix cache...
|
| 159 |
+
(EngineCore pid=2018864) INFO 07-30 20:43:47 [block_pool.py:472] Successfully reset prefix cache
|
| 160 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:51972 - "POST /reset_prefix_cache HTTP/1.1" 200 OK
|
| 161 |
+
(APIServer pid=2018548) INFO 07-30 20:43:53 [loggers.py:259] Engine 000: Avg prompt throughput: 230.8 tokens/s, Avg generation throughput: 853.3 tokens/s, Running: 56 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.2%, Prefix cache hit rate: 69.6%
|
| 162 |
+
(APIServer pid=2018548) INFO 07-30 20:44:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2343.6 tokens/s, Running: 15 reqs, Waiting: 0 reqs, GPU KV cache usage: 12.4%, Prefix cache hit rate: 69.6%
|
| 163 |
+
(APIServer pid=2018548) INFO 07-30 20:44:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 573.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 69.6%
|
| 164 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:58198 - "POST /v1/completions HTTP/1.1" 200 OK
|
| 165 |
+
(APIServer pid=2018548) INFO 07-30 20:44:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 89.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.6%
|
| 166 |
+
(APIServer pid=2018548) INFO 07-30 20:44:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.6%
|
| 167 |
+
(EngineCore pid=2018864)
|
| 168 |
+
(EngineCore pid=2018864)
|
| 169 |
+
(EngineCore pid=2018864)
|
| 170 |
+
(EngineCore pid=2018864)
|
| 171 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeForCausalLM: Failed to load weights
|
| 172 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeModel: Failed to load weights
|
| 173 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] ModuleList: Failed to load weights
|
| 174 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 175 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 176 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] RotaryEmbedding: Failed to load weights
|
| 177 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] ApplyRotaryEmb: Failed to load weights
|
| 178 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 179 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 180 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 181 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 182 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 183 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 184 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 185 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 186 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 187 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 188 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 189 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 190 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 191 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 192 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 193 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 194 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 195 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 196 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 197 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 198 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 199 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 200 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 201 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 202 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 203 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 204 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 205 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 206 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 207 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 208 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 209 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 210 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 211 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 212 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 213 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 214 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 215 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 216 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 217 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 218 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 219 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 220 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 221 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 222 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 223 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 224 |
+
(EngineCore pid=2018864) WARNING 07-30 20:44:45 [layerwise.py:230] LogitsProcessor: Failed to load weights
|
| 225 |
+
(EngineCore pid=2018864) INFO 07-30 20:44:45 [gpu_model_runner.py:4980] Reloading and processing weights took 0.29 seconds
|
| 226 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:46224 - "POST /collective_rpc HTTP/1.1" 200 OK
|
| 227 |
+
(APIServer pid=2018548) INFO 07-30 20:44:45 [api_router.py:39] Resetting prefix cache...
|
| 228 |
+
(EngineCore pid=2018864) INFO 07-30 20:44:45 [block_pool.py:472] Successfully reset prefix cache
|
| 229 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:46232 - "POST /reset_prefix_cache HTTP/1.1" 200 OK
|
| 230 |
+
(APIServer pid=2018548) INFO 07-30 20:44:53 [loggers.py:259] Engine 000: Avg prompt throughput: 321.6 tokens/s, Avg generation throughput: 2539.9 tokens/s, Running: 52 reqs, Waiting: 0 reqs, GPU KV cache usage: 25.1%, Prefix cache hit rate: 71.2%
|
| 231 |
+
(APIServer pid=2018548) INFO 07-30 20:45:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2042.0 tokens/s, Running: 21 reqs, Waiting: 0 reqs, GPU KV cache usage: 22.3%, Prefix cache hit rate: 71.2%
|
| 232 |
+
(APIServer pid=2018548) INFO 07-30 20:45:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 889.4 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.6%, Prefix cache hit rate: 71.2%
|
| 233 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:46234 - "POST /v1/completions HTTP/1.1" 200 OK
|
| 234 |
+
(APIServer pid=2018548) INFO 07-30 20:45:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 99.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 71.2%
|
| 235 |
+
(APIServer pid=2018548) INFO 07-30 20:45:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 71.2%
|
| 236 |
+
(EngineCore pid=2018864)
|
| 237 |
+
(EngineCore pid=2018864)
|
| 238 |
+
(EngineCore pid=2018864)
|
| 239 |
+
(EngineCore pid=2018864)
|
| 240 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeForCausalLM: Failed to load weights
|
| 241 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeModel: Failed to load weights
|
| 242 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] ModuleList: Failed to load weights
|
| 243 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 244 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 245 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] RotaryEmbedding: Failed to load weights
|
| 246 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] ApplyRotaryEmb: Failed to load weights
|
| 247 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 248 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 249 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 250 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 251 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 252 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 253 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 254 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 255 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 256 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 257 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 258 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 259 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 260 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 261 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 262 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 263 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 264 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 265 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 266 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 267 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 268 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 269 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 270 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 271 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 272 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 273 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 274 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 275 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 276 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 277 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 278 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 279 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 280 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 281 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 282 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 283 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 284 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 285 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 286 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 287 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 288 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 289 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 290 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 291 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 292 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 293 |
+
(EngineCore pid=2018864) WARNING 07-30 20:45:49 [layerwise.py:230] LogitsProcessor: Failed to load weights
|
| 294 |
+
(EngineCore pid=2018864) INFO 07-30 20:45:49 [gpu_model_runner.py:4980] Reloading and processing weights took 0.29 seconds
|
| 295 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:33700 - "POST /collective_rpc HTTP/1.1" 200 OK
|
| 296 |
+
(APIServer pid=2018548) INFO 07-30 20:45:49 [api_router.py:39] Resetting prefix cache...
|
| 297 |
+
(EngineCore pid=2018864) INFO 07-30 20:45:49 [block_pool.py:472] Successfully reset prefix cache
|
| 298 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:33712 - "POST /reset_prefix_cache HTTP/1.1" 200 OK
|
| 299 |
+
(APIServer pid=2018548) INFO 07-30 20:45:53 [loggers.py:259] Engine 000: Avg prompt throughput: 210.8 tokens/s, Avg generation throughput: 1632.7 tokens/s, Running: 58 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.0%, Prefix cache hit rate: 67.4%
|
| 300 |
+
(APIServer pid=2018548) INFO 07-30 20:46:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 2308.8 tokens/s, Running: 18 reqs, Waiting: 0 reqs, GPU KV cache usage: 16.2%, Prefix cache hit rate: 67.4%
|
| 301 |
+
(APIServer pid=2018548) INFO 07-30 20:46:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 684.4 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 67.4%
|
| 302 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:33724 - "POST /v1/completions HTTP/1.1" 200 OK
|
| 303 |
+
(APIServer pid=2018548) INFO 07-30 20:46:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 122.1 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 67.4%
|
| 304 |
+
(APIServer pid=2018548) INFO 07-30 20:46:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 67.4%
|
| 305 |
+
(EngineCore pid=2018864)
|
| 306 |
+
(EngineCore pid=2018864)
|
| 307 |
+
(EngineCore pid=2018864)
|
| 308 |
+
(EngineCore pid=2018864)
|
| 309 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeForCausalLM: Failed to load weights
|
| 310 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeModel: Failed to load weights
|
| 311 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] ModuleList: Failed to load weights
|
| 312 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 313 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 314 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] RotaryEmbedding: Failed to load weights
|
| 315 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] ApplyRotaryEmb: Failed to load weights
|
| 316 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 317 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 318 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 319 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 320 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 321 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 322 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 323 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 324 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 325 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 326 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 327 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 328 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 329 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 330 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 331 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 332 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 333 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 334 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 335 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 336 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 337 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 338 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 339 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 340 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 341 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 342 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 343 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 344 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 345 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 346 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 347 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 348 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 349 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 350 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 351 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 352 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 353 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 354 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 355 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 356 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 357 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 358 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 359 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 360 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 361 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 362 |
+
(EngineCore pid=2018864) WARNING 07-30 20:46:48 [layerwise.py:230] LogitsProcessor: Failed to load weights
|
| 363 |
+
(EngineCore pid=2018864) INFO 07-30 20:46:48 [gpu_model_runner.py:4980] Reloading and processing weights took 0.29 seconds
|
| 364 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:58584 - "POST /collective_rpc HTTP/1.1" 200 OK
|
| 365 |
+
(APIServer pid=2018548) INFO 07-30 20:46:48 [api_router.py:39] Resetting prefix cache...
|
| 366 |
+
(EngineCore pid=2018864) INFO 07-30 20:46:48 [block_pool.py:472] Successfully reset prefix cache
|
| 367 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:58592 - "POST /reset_prefix_cache HTTP/1.1" 200 OK
|
| 368 |
+
(APIServer pid=2018548) INFO 07-30 20:46:53 [loggers.py:259] Engine 000: Avg prompt throughput: 271.6 tokens/s, Avg generation throughput: 1760.6 tokens/s, Running: 51 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.5%, Prefix cache hit rate: 69.8%
|
| 369 |
+
(APIServer pid=2018548) INFO 07-30 20:47:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1980.5 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.9%, Prefix cache hit rate: 69.8%
|
| 370 |
+
(APIServer pid=2018548) INFO 07-30 20:47:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 383.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 69.8%
|
| 371 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:58602 - "POST /v1/completions HTTP/1.1" 200 OK
|
| 372 |
+
(APIServer pid=2018548) INFO 07-30 20:47:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 52.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.8%
|
| 373 |
+
(APIServer pid=2018548) INFO 07-30 20:47:33 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 69.8%
|
| 374 |
+
(EngineCore pid=2018864)
|
| 375 |
+
(EngineCore pid=2018864)
|
| 376 |
+
(EngineCore pid=2018864)
|
| 377 |
+
(EngineCore pid=2018864)
|
| 378 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeForCausalLM: Failed to load weights
|
| 379 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeModel: Failed to load weights
|
| 380 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] ModuleList: Failed to load weights
|
| 381 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 382 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 383 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] RotaryEmbedding: Failed to load weights
|
| 384 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] ApplyRotaryEmb: Failed to load weights
|
| 385 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 386 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 387 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 388 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 389 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 390 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 391 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 392 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 393 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 394 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 395 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 396 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 397 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 398 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 399 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 400 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 401 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 402 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 403 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 404 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 405 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 406 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 407 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 408 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 409 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 410 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 411 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 412 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 413 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 414 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 415 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 416 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 417 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 418 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 419 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 420 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 421 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 422 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 423 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 424 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 425 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 426 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 427 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 428 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] PrunedOlmoeDecoderLayer: Failed to load weights
|
| 429 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] OlmoeAttention: Failed to load weights
|
| 430 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] VllmVariableOlmoeMoE: Failed to load weights
|
| 431 |
+
(EngineCore pid=2018864) WARNING 07-30 20:47:47 [layerwise.py:230] LogitsProcessor: Failed to load weights
|
| 432 |
+
(EngineCore pid=2018864) INFO 07-30 20:47:47 [gpu_model_runner.py:4980] Reloading and processing weights took 0.29 seconds
|
| 433 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:40328 - "POST /collective_rpc HTTP/1.1" 200 OK
|
| 434 |
+
(APIServer pid=2018548) INFO 07-30 20:47:47 [api_router.py:39] Resetting prefix cache...
|
| 435 |
+
(EngineCore pid=2018864) INFO 07-30 20:47:47 [block_pool.py:472] Successfully reset prefix cache
|
| 436 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:40334 - "POST /reset_prefix_cache HTTP/1.1" 200 OK
|
| 437 |
+
(APIServer pid=2018548) INFO 07-30 20:47:53 [loggers.py:259] Engine 000: Avg prompt throughput: 267.6 tokens/s, Avg generation throughput: 2273.8 tokens/s, Running: 49 reqs, Waiting: 0 reqs, GPU KV cache usage: 20.2%, Prefix cache hit rate: 70.0%
|
| 438 |
+
(APIServer pid=2018548) INFO 07-30 20:48:03 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 1862.2 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 14.7%, Prefix cache hit rate: 70.0%
|
| 439 |
+
(APIServer pid=2018548) INFO: 127.0.0.1:40340 - "POST /v1/completions HTTP/1.1" 200 OK
|
| 440 |
+
(APIServer pid=2018548) INFO 07-30 20:48:13 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 309.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 70.0%
|
| 441 |
+
(APIServer pid=2018548) INFO 07-30 20:48:23 [loggers.py:259] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 70.0%
|
| 442 |
+
(EngineCore pid=2018864) INFO 07-30 20:48:36 [core.py:1210] Shutdown initiated (timeout=0)
|
| 443 |
+
(EngineCore pid=2018864) INFO 07-30 20:48:36 [core.py:1233] Shutdown complete
|
| 444 |
+
(APIServer pid=2018548) INFO: Shutting down
|
| 445 |
+
(APIServer pid=2018548) INFO: Waiting for application shutdown.
|
| 446 |
+
(APIServer pid=2018548) INFO: Application shutdown complete.
|
| 447 |
+
(APIServer pid=2018548) INFO: Finished server process [2018548]
|
healed/mixceonly_keep50/args.json
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"student": "outputs/pruned/glean-0125inst-math-keep50",
|
| 3 |
+
"teacher": "allenai/OLMoE-1B-7B-0125-Instruct",
|
| 4 |
+
"training_mode": "on-policy",
|
| 5 |
+
"kl_direction": "reverse",
|
| 6 |
+
"dataset": "allenai/Dolci-Instruct-RL",
|
| 7 |
+
"dataset_sources": null,
|
| 8 |
+
"max_difficulty": null,
|
| 9 |
+
"trajectories": "outputs/teacher_trajectories/dolci_math_curated.jsonl",
|
| 10 |
+
"trajectory_dataset": "allenai/Dolci-Instruct-RL",
|
| 11 |
+
"off_policy_frames": "chat",
|
| 12 |
+
"off_policy_max_seq_len": 2048,
|
| 13 |
+
"topk_targets": null,
|
| 14 |
+
"max_loss_tokens": null,
|
| 15 |
+
"loss_tokens_per_step": null,
|
| 16 |
+
"teacher_device": "cuda:0",
|
| 17 |
+
"student_device": "cuda:1",
|
| 18 |
+
"lr": 3e-05,
|
| 19 |
+
"optimizer": "adamw8bit",
|
| 20 |
+
"weight_decay": 0.1,
|
| 21 |
+
"epochs": 2,
|
| 22 |
+
"prompts_per_step": 256,
|
| 23 |
+
"group_size": 4,
|
| 24 |
+
"rollout_batch": 64,
|
| 25 |
+
"micro_batch": 4,
|
| 26 |
+
"max_new_tokens": 2048,
|
| 27 |
+
"max_prompt_len": 1024,
|
| 28 |
+
"warmup_steps": 10,
|
| 29 |
+
"max_grad_norm": 1.0,
|
| 30 |
+
"eval_every": 10,
|
| 31 |
+
"gsm8k_every": 20,
|
| 32 |
+
"gsm8k_n": 256,
|
| 33 |
+
"gsm8k_batch": 16,
|
| 34 |
+
"gsm8k_max_new_tokens": 1024,
|
| 35 |
+
"gsm8k_frames": "chat",
|
| 36 |
+
"save_every": 1000,
|
| 37 |
+
"out_dir": "outputs/healed/mixceonly_keep50",
|
| 38 |
+
"sweep": 60,
|
| 39 |
+
"wandb": true,
|
| 40 |
+
"wandb_project": "glean-heal",
|
| 41 |
+
"wandb_run_name": "mixce-keep50-s1223",
|
| 42 |
+
"wandb_run_id": null,
|
| 43 |
+
"wandb_resume": null,
|
| 44 |
+
"wandb_mode": "offline",
|
| 45 |
+
"no_wandb_sync": false,
|
| 46 |
+
"debug": false,
|
| 47 |
+
"resume_from": null,
|
| 48 |
+
"start_step": 0,
|
| 49 |
+
"no_grad_checkpointing": false,
|
| 50 |
+
"seed": 1223,
|
| 51 |
+
"no_teacher_overlap": false,
|
| 52 |
+
"sync_checkpoints": false,
|
| 53 |
+
"rollout_engine": "vllm",
|
| 54 |
+
"vllm_gpu": "2",
|
| 55 |
+
"vllm_port": 8377,
|
| 56 |
+
"vllm_refresh_every": 1,
|
| 57 |
+
"vllm_serve_bin": "vllm-plugin/.venv/bin/python",
|
| 58 |
+
"vllm_gpu_mem_util": 0.85,
|
| 59 |
+
"liger_loss": true,
|
| 60 |
+
"gold_mix_lambda": 0.5,
|
| 61 |
+
"gold_topk_targets": "outputs/teacher_trajectories/dolci_combined_top128",
|
| 62 |
+
"gold_loss": "ce",
|
| 63 |
+
"gold_mix_decay": 0.0,
|
| 64 |
+
"fast_teacher": true,
|
| 65 |
+
"reference_kl_beta": 0.0,
|
| 66 |
+
"drop_truncated_rollouts": false,
|
| 67 |
+
"vllm_max_model_len": null,
|
| 68 |
+
"vllm_refresh_mode": "reload",
|
| 69 |
+
"vllm_live_dir": null,
|
| 70 |
+
"resolved_kl_direction": "reverse"
|
| 71 |
+
}
|
healed/mixceonly_keep50/train_log.jsonl
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"step": 1, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6070861224798003, "tokens": 174656, "cumulative_loss_tokens": 174656, "grad_norm": 4.1875, "lr": 6e-06, "finish_rate": 0.98, "comp_len": 682.2, "dropped_truncated": 0, "gold_loss": 0.4598, "gold_lambda": 0.5, "rep_ratio": 2.51, "t_data_s": 0.0, "t_rollout_s": 52.1, "t_step_s": 127.5, "t_refresh_s": 0.3, "mem_gb": 10.13, "mem_gb_teacher": 13.64}
|
| 2 |
+
{"step": 1, "gsm8k_n": 256, "gsm8k_quick_chat": 0.59375, "t_eval_s": 16.7}
|
| 3 |
+
{"step": 2, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.589608316678511, "tokens": 185370, "cumulative_loss_tokens": 360026, "grad_norm": 4.53125, "lr": 9e-06, "finish_rate": 0.984, "comp_len": 724.1, "dropped_truncated": 0, "gold_loss": 0.4409, "gold_lambda": 0.5, "rep_ratio": 2.349, "t_data_s": 0.0, "t_rollout_s": 55.0, "t_step_s": 121.6, "t_refresh_s": 0.3, "mem_gb": 9.93, "mem_gb_teacher": 13.51}
|
| 4 |
+
{"step": 3, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6362397829683438, "tokens": 190611, "cumulative_loss_tokens": 550637, "grad_norm": 3.921875, "lr": 1.2e-05, "finish_rate": 0.957, "comp_len": 744.6, "dropped_truncated": 0, "gold_loss": 0.4903, "gold_lambda": 0.5, "rep_ratio": 2.642, "t_data_s": 0.0, "t_rollout_s": 56.9, "t_step_s": 124.6, "t_refresh_s": 0.3, "mem_gb": 10.23, "mem_gb_teacher": 13.62}
|
| 5 |
+
{"step": 4, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5276193276248334, "tokens": 160132, "cumulative_loss_tokens": 710769, "grad_norm": 3.15625, "lr": 1.5e-05, "finish_rate": 1.0, "comp_len": 625.5, "dropped_truncated": 0, "gold_loss": 0.4785, "gold_lambda": 0.5, "rep_ratio": 2.597, "t_data_s": 0.0, "t_rollout_s": 46.6, "t_step_s": 109.9, "t_refresh_s": 0.3, "mem_gb": 9.98, "mem_gb_teacher": 13.53}
|
| 6 |
+
{"step": 5, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5429938330516391, "tokens": 162385, "cumulative_loss_tokens": 873154, "grad_norm": 4.21875, "lr": 1.8e-05, "finish_rate": 0.992, "comp_len": 634.3, "dropped_truncated": 0, "gold_loss": 1.5549, "gold_lambda": 0.5, "rep_ratio": 2.847, "t_data_s": 0.0, "t_rollout_s": 49.4, "t_step_s": 110.2, "t_refresh_s": 0.3, "mem_gb": 10.26, "mem_gb_teacher": 13.62}
|
| 7 |
+
{"step": 6, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5009330182323017, "tokens": 161557, "cumulative_loss_tokens": 1034711, "grad_norm": 2.875, "lr": 2.1e-05, "finish_rate": 0.984, "comp_len": 631.1, "dropped_truncated": 0, "gold_loss": 1.2, "gold_lambda": 0.5, "rep_ratio": 2.285, "t_data_s": 0.0, "t_rollout_s": 51.3, "t_step_s": 110.1, "t_refresh_s": 0.3, "mem_gb": 10.08, "mem_gb_teacher": 13.56}
|
| 8 |
+
{"step": 7, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4394795861535276, "tokens": 179710, "cumulative_loss_tokens": 1214421, "grad_norm": 2.40625, "lr": 2.4e-05, "finish_rate": 0.996, "comp_len": 702.0, "dropped_truncated": 0, "gold_loss": 1.56, "gold_lambda": 0.5, "rep_ratio": 2.334, "t_data_s": 0.0, "t_rollout_s": 54.2, "t_step_s": 118.6, "t_refresh_s": 0.3, "mem_gb": 9.93, "mem_gb_teacher": 13.53}
|
| 9 |
+
{"step": 8, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.46311661079864236, "tokens": 172672, "cumulative_loss_tokens": 1387093, "grad_norm": 2.375, "lr": 2.7000000000000002e-05, "finish_rate": 0.992, "comp_len": 674.5, "dropped_truncated": 0, "gold_loss": 1.6159, "gold_lambda": 0.5, "rep_ratio": 2.589, "t_data_s": 0.0, "t_rollout_s": 50.6, "t_step_s": 112.0, "t_refresh_s": 0.3, "mem_gb": 10.01, "mem_gb_teacher": 13.54}
|
| 10 |
+
{"step": 9, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5894883901328609, "tokens": 177365, "cumulative_loss_tokens": 1564458, "grad_norm": 1.46875, "lr": 3e-05, "finish_rate": 0.98, "comp_len": 692.8, "dropped_truncated": 0, "gold_loss": 0.49, "gold_lambda": 0.5, "rep_ratio": 2.318, "t_data_s": 0.0, "t_rollout_s": 53.0, "t_step_s": 116.8, "t_refresh_s": 0.3, "mem_gb": 10.16, "mem_gb_teacher": 13.59}
|
| 11 |
+
{"step": 10, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5003410960365811, "tokens": 179180, "cumulative_loss_tokens": 1743638, "grad_norm": 1.2109375, "lr": 3e-05, "finish_rate": 0.98, "comp_len": 699.9, "dropped_truncated": 0, "gold_loss": 0.4765, "gold_lambda": 0.5, "rep_ratio": 2.69, "t_data_s": 0.0, "t_rollout_s": 53.2, "t_step_s": 116.4, "t_refresh_s": 0.3, "mem_gb": 9.91, "mem_gb_teacher": 13.5}
|
| 12 |
+
{"step": 11, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4788465677366256, "tokens": 169222, "cumulative_loss_tokens": 1912860, "grad_norm": 7.65625, "lr": 3e-05, "finish_rate": 0.988, "comp_len": 661.0, "dropped_truncated": 0, "gold_loss": 1.8904, "gold_lambda": 0.5, "rep_ratio": 2.661, "t_data_s": 0.0, "t_rollout_s": 50.4, "t_step_s": 112.0, "t_refresh_s": 0.3, "mem_gb": 9.92, "mem_gb_teacher": 13.51}
|
| 13 |
+
{"step": 12, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4898799755617264, "tokens": 190041, "cumulative_loss_tokens": 2102901, "grad_norm": 5.0, "lr": 3e-05, "finish_rate": 0.984, "comp_len": 742.3, "dropped_truncated": 0, "gold_loss": 1.6883, "gold_lambda": 0.5, "rep_ratio": 2.72, "t_data_s": 0.0, "t_rollout_s": 55.1, "t_step_s": 121.7, "t_refresh_s": 0.3, "mem_gb": 10.29, "mem_gb_teacher": 13.63}
|
| 14 |
+
{"step": 13, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6191565852231934, "tokens": 179426, "cumulative_loss_tokens": 2282327, "grad_norm": 3.453125, "lr": 3e-05, "finish_rate": 0.973, "comp_len": 700.9, "dropped_truncated": 0, "gold_loss": 1.7587, "gold_lambda": 0.5, "rep_ratio": 2.516, "t_data_s": 0.0, "t_rollout_s": 51.9, "t_step_s": 114.8, "t_refresh_s": 0.3, "mem_gb": 9.91, "mem_gb_teacher": 13.49}
|
| 15 |
+
{"step": 14, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5307344244249613, "tokens": 173105, "cumulative_loss_tokens": 2455432, "grad_norm": 4.125, "lr": 3e-05, "finish_rate": 0.973, "comp_len": 676.2, "dropped_truncated": 0, "gold_loss": 1.681, "gold_lambda": 0.5, "rep_ratio": 2.729, "t_data_s": 0.0, "t_rollout_s": 51.0, "t_step_s": 114.6, "t_refresh_s": 0.3, "mem_gb": 9.86, "mem_gb_teacher": 13.48}
|
| 16 |
+
{"step": 15, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6075845100702332, "tokens": 182124, "cumulative_loss_tokens": 2637556, "grad_norm": 2.34375, "lr": 3e-05, "finish_rate": 0.992, "comp_len": 711.4, "dropped_truncated": 0, "gold_loss": 0.4579, "gold_lambda": 0.5, "rep_ratio": 2.474, "t_data_s": 0.0, "t_rollout_s": 53.4, "t_step_s": 117.2, "t_refresh_s": 0.3, "mem_gb": 9.92, "mem_gb_teacher": 13.51}
|
| 17 |
+
{"step": 16, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5922139764446475, "tokens": 180837, "cumulative_loss_tokens": 2818393, "grad_norm": 2.078125, "lr": 3e-05, "finish_rate": 0.973, "comp_len": 706.4, "dropped_truncated": 0, "gold_loss": 0.4464, "gold_lambda": 0.5, "rep_ratio": 2.593, "t_data_s": 0.0, "t_rollout_s": 52.7, "t_step_s": 119.9, "t_refresh_s": 0.3, "mem_gb": 9.91, "mem_gb_teacher": 13.51}
|
| 18 |
+
{"step": 17, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6971930358045317, "tokens": 168901, "cumulative_loss_tokens": 2987294, "grad_norm": 2.09375, "lr": 3e-05, "finish_rate": 0.969, "comp_len": 659.8, "dropped_truncated": 0, "gold_loss": 1.3389, "gold_lambda": 0.5, "rep_ratio": 2.484, "t_data_s": 0.0, "t_rollout_s": 51.5, "t_step_s": 116.2, "t_refresh_s": 0.3, "mem_gb": 10.15, "mem_gb_teacher": 13.59}
|
| 19 |
+
{"step": 18, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6623637577042752, "tokens": 178358, "cumulative_loss_tokens": 3165652, "grad_norm": 1.875, "lr": 3e-05, "finish_rate": 0.984, "comp_len": 696.7, "dropped_truncated": 0, "gold_loss": 1.3371, "gold_lambda": 0.5, "rep_ratio": 2.515, "t_data_s": 0.0, "t_rollout_s": 55.5, "t_step_s": 122.1, "t_refresh_s": 0.3, "mem_gb": 10.43, "mem_gb_teacher": 13.68}
|
| 20 |
+
{"step": 19, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6558139320164235, "tokens": 180673, "cumulative_loss_tokens": 3346325, "grad_norm": 1.671875, "lr": 3e-05, "finish_rate": 0.965, "comp_len": 705.8, "dropped_truncated": 0, "gold_loss": 1.593, "gold_lambda": 0.5, "rep_ratio": 2.497, "t_data_s": 0.0, "t_rollout_s": 54.9, "t_step_s": 121.6, "t_refresh_s": 0.3, "mem_gb": 10.12, "mem_gb_teacher": 13.57}
|
| 21 |
+
{"step": 20, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5545611710471838, "tokens": 198365, "cumulative_loss_tokens": 3544690, "grad_norm": 2.609375, "lr": 3e-05, "finish_rate": 0.965, "comp_len": 774.9, "dropped_truncated": 0, "gold_loss": 1.6895, "gold_lambda": 0.5, "rep_ratio": 2.682, "t_data_s": 0.0, "t_rollout_s": 59.5, "t_step_s": 130.4, "t_refresh_s": 0.3, "mem_gb": 10.21, "mem_gb_teacher": 13.61}
|
| 22 |
+
{"step": 20, "gsm8k_n": 256, "gsm8k_quick_chat": 0.57421875, "t_eval_s": 21.4}
|
| 23 |
+
{"step": 21, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7081326860186451, "tokens": 153659, "cumulative_loss_tokens": 3698349, "grad_norm": 2.59375, "lr": 3e-05, "finish_rate": 0.969, "comp_len": 600.2, "dropped_truncated": 0, "gold_loss": 0.3889, "gold_lambda": 0.5, "rep_ratio": 2.61, "t_data_s": 0.0, "t_rollout_s": 47.0, "t_step_s": 108.7, "t_refresh_s": 0.3, "mem_gb": 10.0, "mem_gb_teacher": 13.54}
|
| 24 |
+
{"step": 22, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6475236291024463, "tokens": 175549, "cumulative_loss_tokens": 3873898, "grad_norm": 1.8671875, "lr": 3e-05, "finish_rate": 0.984, "comp_len": 685.7, "dropped_truncated": 0, "gold_loss": 0.3969, "gold_lambda": 0.5, "rep_ratio": 2.524, "t_data_s": 0.0, "t_rollout_s": 54.1, "t_step_s": 123.2, "t_refresh_s": 0.3, "mem_gb": 10.13, "mem_gb_teacher": 13.58}
|
| 25 |
+
{"step": 23, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7012895279056457, "tokens": 156583, "cumulative_loss_tokens": 4030481, "grad_norm": 1.9921875, "lr": 3e-05, "finish_rate": 0.973, "comp_len": 611.7, "dropped_truncated": 0, "gold_loss": 0.9875, "gold_lambda": 0.5, "rep_ratio": 2.474, "t_data_s": 0.0, "t_rollout_s": 49.0, "t_step_s": 112.0, "t_refresh_s": 0.3, "mem_gb": 9.95, "mem_gb_teacher": 13.51}
|
| 26 |
+
{"step": 24, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8830423523515076, "tokens": 142499, "cumulative_loss_tokens": 4172980, "grad_norm": 9.5625, "lr": 3e-05, "finish_rate": 0.988, "comp_len": 556.6, "dropped_truncated": 0, "gold_loss": 1.126, "gold_lambda": 0.5, "rep_ratio": 2.431, "t_data_s": 0.0, "t_rollout_s": 45.3, "t_step_s": 103.6, "t_refresh_s": 0.3, "mem_gb": 10.08, "mem_gb_teacher": 13.57}
|
| 27 |
+
{"step": 25, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6655911572447009, "tokens": 185008, "cumulative_loss_tokens": 4357988, "grad_norm": 2.921875, "lr": 3e-05, "finish_rate": 0.953, "comp_len": 722.7, "dropped_truncated": 0, "gold_loss": 1.9257, "gold_lambda": 0.5, "rep_ratio": 2.694, "t_data_s": 0.0, "t_rollout_s": 56.8, "t_step_s": 125.6, "t_refresh_s": 0.3, "mem_gb": 10.33, "mem_gb_teacher": 13.65}
|
| 28 |
+
{"step": 26, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5897857546141282, "tokens": 172974, "cumulative_loss_tokens": 4530962, "grad_norm": 2.03125, "lr": 3e-05, "finish_rate": 0.973, "comp_len": 675.7, "dropped_truncated": 0, "gold_loss": 1.6761, "gold_lambda": 0.5, "rep_ratio": 2.771, "t_data_s": 0.0, "t_rollout_s": 52.4, "t_step_s": 118.5, "t_refresh_s": 0.3, "mem_gb": 10.27, "mem_gb_teacher": 13.63}
|
| 29 |
+
{"step": 27, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6517713041402011, "tokens": 172075, "cumulative_loss_tokens": 4703037, "grad_norm": 2.171875, "lr": 3e-05, "finish_rate": 0.977, "comp_len": 672.2, "dropped_truncated": 0, "gold_loss": 1.2191, "gold_lambda": 0.5, "rep_ratio": 2.356, "t_data_s": 0.0, "t_rollout_s": 51.2, "t_step_s": 115.7, "t_refresh_s": 0.3, "mem_gb": 10.03, "mem_gb_teacher": 13.54}
|
| 30 |
+
{"step": 28, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8185856890662986, "tokens": 129102, "cumulative_loss_tokens": 4832139, "grad_norm": 4.03125, "lr": 3e-05, "finish_rate": 0.988, "comp_len": 504.3, "dropped_truncated": 0, "gold_loss": 1.333, "gold_lambda": 0.5, "rep_ratio": 2.367, "t_data_s": 0.0, "t_rollout_s": 42.8, "t_step_s": 99.2, "t_refresh_s": 0.3, "mem_gb": 10.03, "mem_gb_teacher": 13.54}
|
| 31 |
+
{"step": 29, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7672473138648033, "tokens": 144369, "cumulative_loss_tokens": 4976508, "grad_norm": 4.65625, "lr": 3e-05, "finish_rate": 0.961, "comp_len": 563.9, "dropped_truncated": 0, "gold_loss": 0.5047, "gold_lambda": 0.5, "rep_ratio": 2.525, "t_data_s": 0.0, "t_rollout_s": 46.2, "t_step_s": 110.5, "t_refresh_s": 0.3, "mem_gb": 10.15, "mem_gb_teacher": 13.59}
|
| 32 |
+
{"step": 30, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7387997474785865, "tokens": 146453, "cumulative_loss_tokens": 5122961, "grad_norm": 3.546875, "lr": 3e-05, "finish_rate": 0.988, "comp_len": 572.1, "dropped_truncated": 0, "gold_loss": 0.4805, "gold_lambda": 0.5, "rep_ratio": 2.373, "t_data_s": 0.0, "t_rollout_s": 46.5, "t_step_s": 109.7, "t_refresh_s": 0.3, "mem_gb": 10.26, "mem_gb_teacher": 13.62}
|
| 33 |
+
{"step": 31, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6948283791479429, "tokens": 184172, "cumulative_loss_tokens": 5307133, "grad_norm": 4.15625, "lr": 3e-05, "finish_rate": 0.945, "comp_len": 719.4, "dropped_truncated": 0, "gold_loss": 1.624, "gold_lambda": 0.5, "rep_ratio": 2.974, "t_data_s": 0.0, "t_rollout_s": 54.5, "t_step_s": 123.4, "t_refresh_s": 0.3, "mem_gb": 9.99, "mem_gb_teacher": 13.53}
|
| 34 |
+
{"step": 32, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7520270073016101, "tokens": 161634, "cumulative_loss_tokens": 5468767, "grad_norm": 3.125, "lr": 3e-05, "finish_rate": 0.941, "comp_len": 631.4, "dropped_truncated": 0, "gold_loss": 1.385, "gold_lambda": 0.5, "rep_ratio": 2.371, "t_data_s": 0.0, "t_rollout_s": 51.0, "t_step_s": 122.5, "t_refresh_s": 0.3, "mem_gb": 10.15, "mem_gb_teacher": 13.59}
|
| 35 |
+
{"step": 33, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7959626553265123, "tokens": 174953, "cumulative_loss_tokens": 5643720, "grad_norm": 6.625, "lr": 3e-05, "finish_rate": 0.941, "comp_len": 683.4, "dropped_truncated": 0, "gold_loss": 1.7494, "gold_lambda": 0.5, "rep_ratio": 2.384, "t_data_s": 0.0, "t_rollout_s": 54.0, "t_step_s": 125.1, "t_refresh_s": 0.3, "mem_gb": 10.12, "mem_gb_teacher": 13.57}
|
| 36 |
+
{"step": 34, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.683204715838983, "tokens": 197798, "cumulative_loss_tokens": 5841518, "grad_norm": 12.4375, "lr": 3e-05, "finish_rate": 0.941, "comp_len": 772.6, "dropped_truncated": 0, "gold_loss": 2.2089, "gold_lambda": 0.5, "rep_ratio": 2.54, "t_data_s": 0.0, "t_rollout_s": 59.9, "t_step_s": 132.0, "t_refresh_s": 0.3, "mem_gb": 9.99, "mem_gb_teacher": 13.53}
|
| 37 |
+
{"step": 35, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7359022620276259, "tokens": 224154, "cumulative_loss_tokens": 6065672, "grad_norm": 2.53125, "lr": 3e-05, "finish_rate": 0.895, "comp_len": 875.6, "dropped_truncated": 0, "gold_loss": 0.3891, "gold_lambda": 0.5, "rep_ratio": 2.576, "t_data_s": 0.0, "t_rollout_s": 66.7, "t_step_s": 146.6, "t_refresh_s": 0.3, "mem_gb": 10.4, "mem_gb_teacher": 13.67}
|
| 38 |
+
{"step": 36, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.995374442305058, "tokens": 238556, "cumulative_loss_tokens": 6304228, "grad_norm": 3.0, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 931.9, "dropped_truncated": 0, "gold_loss": 0.4685, "gold_lambda": 0.5, "rep_ratio": 3.499, "t_data_s": 0.0, "t_rollout_s": 75.3, "t_step_s": 159.4, "t_refresh_s": 0.3, "mem_gb": 10.2, "mem_gb_teacher": 13.6}
|
| 39 |
+
{"step": 37, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8644908860862507, "tokens": 221896, "cumulative_loss_tokens": 6526124, "grad_norm": 10.8125, "lr": 3e-05, "finish_rate": 0.871, "comp_len": 866.8, "dropped_truncated": 0, "gold_loss": 1.5978, "gold_lambda": 0.5, "rep_ratio": 2.379, "t_data_s": 0.0, "t_rollout_s": 66.9, "t_step_s": 148.7, "t_refresh_s": 0.3, "mem_gb": 9.9, "mem_gb_teacher": 13.5}
|
| 40 |
+
{"step": 38, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7681629302413664, "tokens": 270641, "cumulative_loss_tokens": 6796765, "grad_norm": 15.0, "lr": 3e-05, "finish_rate": 0.84, "comp_len": 1057.2, "dropped_truncated": 0, "gold_loss": 1.5569, "gold_lambda": 0.5, "rep_ratio": 2.879, "t_data_s": 0.0, "t_rollout_s": 99.2, "t_step_s": 188.3, "t_refresh_s": 0.3, "mem_gb": 10.18, "mem_gb_teacher": 13.59}
|
| 41 |
+
{"step": 39, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1499663096357071, "tokens": 262364, "cumulative_loss_tokens": 7059129, "grad_norm": 9.0, "lr": 3e-05, "finish_rate": 0.781, "comp_len": 1024.9, "dropped_truncated": 0, "gold_loss": 0.4787, "gold_lambda": 0.5, "rep_ratio": 4.284, "t_data_s": 0.0, "t_rollout_s": 96.1, "t_step_s": 181.6, "t_refresh_s": 0.3, "mem_gb": 10.0, "mem_gb_teacher": 13.53}
|
| 42 |
+
{"step": 40, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1472117115308085, "tokens": 270179, "cumulative_loss_tokens": 7329308, "grad_norm": 9.6875, "lr": 3e-05, "finish_rate": 0.777, "comp_len": 1055.4, "dropped_truncated": 0, "gold_loss": 0.484, "gold_lambda": 0.5, "rep_ratio": 3.411, "t_data_s": 0.0, "t_rollout_s": 101.5, "t_step_s": 192.6, "t_refresh_s": 0.3, "mem_gb": 10.21, "mem_gb_teacher": 13.6}
|
| 43 |
+
{"step": 40, "gsm8k_n": 256, "gsm8k_quick_chat": 0.5546875, "t_eval_s": 42.1}
|
| 44 |
+
{"step": 41, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.289868695305495, "tokens": 307658, "cumulative_loss_tokens": 7636966, "grad_norm": 22.125, "lr": 3e-05, "finish_rate": 0.707, "comp_len": 1201.8, "dropped_truncated": 0, "gold_loss": 1.7162, "gold_lambda": 0.5, "rep_ratio": 7.153, "t_data_s": 0.0, "t_rollout_s": 111.7, "t_step_s": 202.1, "t_refresh_s": 0.3, "mem_gb": 10.13, "mem_gb_teacher": 13.57}
|
| 45 |
+
{"step": 42, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.597719869015183, "tokens": 325402, "cumulative_loss_tokens": 7962368, "grad_norm": 34.25, "lr": 3e-05, "finish_rate": 0.684, "comp_len": 1271.1, "dropped_truncated": 0, "gold_loss": 1.8621, "gold_lambda": 0.5, "rep_ratio": 3.49, "t_data_s": 0.0, "t_rollout_s": 117.8, "t_step_s": 209.5, "t_refresh_s": 0.3, "mem_gb": 10.22, "mem_gb_teacher": 13.69}
|
| 46 |
+
{"step": 43, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 3.4306665449676728, "tokens": 468403, "cumulative_loss_tokens": 8430771, "grad_norm": 736.0, "lr": 3e-05, "finish_rate": 0.168, "comp_len": 1829.7, "dropped_truncated": 0, "gold_loss": 5.2979, "gold_lambda": 0.5, "rep_ratio": 1.949, "t_data_s": 0.0, "t_rollout_s": 190.4, "t_step_s": 289.5, "t_refresh_s": 0.3, "mem_gb": 10.43, "mem_gb_teacher": 13.68}
|
| 47 |
+
{"step": 44, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 4.171818237882794, "tokens": 505709, "cumulative_loss_tokens": 8936480, "grad_norm": 1064.0, "lr": 3e-05, "finish_rate": 0.066, "comp_len": 1975.4, "dropped_truncated": 0, "gold_loss": 7.5318, "gold_lambda": 0.5, "rep_ratio": 1.675, "t_data_s": 0.0, "t_rollout_s": 204.4, "t_step_s": 303.7, "t_refresh_s": 0.3, "mem_gb": 10.22, "mem_gb_teacher": 13.6}
|
| 48 |
+
{"step": 45, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 4.106944052248165, "tokens": 489030, "cumulative_loss_tokens": 9425510, "grad_norm": 980.0, "lr": 3e-05, "finish_rate": 0.074, "comp_len": 1910.3, "dropped_truncated": 0, "gold_loss": 6.428, "gold_lambda": 0.5, "rep_ratio": 1.691, "t_data_s": 0.0, "t_rollout_s": 196.6, "t_step_s": 296.7, "t_refresh_s": 0.3, "mem_gb": 10.15, "mem_gb_teacher": 13.58}
|
| 49 |
+
{"step": 46, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 3.450120545435666, "tokens": 459101, "cumulative_loss_tokens": 9884611, "grad_norm": 466.0, "lr": 3e-05, "finish_rate": 0.195, "comp_len": 1793.4, "dropped_truncated": 0, "gold_loss": 3.4515, "gold_lambda": 0.5, "rep_ratio": 1.592, "t_data_s": 0.0, "t_rollout_s": 185.3, "t_step_s": 285.1, "t_refresh_s": 0.3, "mem_gb": 10.43, "mem_gb_teacher": 13.67}
|
healed/mixceonly_keep50/vllm_server.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
healed/mixceonly_keep50/wandb_sync.log
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Find logs at: /tmp/debug-cli.henry.log
|
| 2 |
+
Syncing: https://wandb.ai/hbfreed/glean-heal/runs/009xr5f3 ... done.
|
healed/mixonly_keep50/args.json
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"student": "outputs/pruned/glean-0125inst-math-keep50",
|
| 3 |
+
"teacher": "allenai/OLMoE-1B-7B-0125-Instruct",
|
| 4 |
+
"training_mode": "on-policy",
|
| 5 |
+
"kl_direction": "reverse",
|
| 6 |
+
"dataset": "allenai/Dolci-Instruct-RL",
|
| 7 |
+
"dataset_sources": null,
|
| 8 |
+
"max_difficulty": null,
|
| 9 |
+
"trajectories": "outputs/teacher_trajectories/dolci_math_curated.jsonl",
|
| 10 |
+
"trajectory_dataset": "allenai/Dolci-Instruct-RL",
|
| 11 |
+
"off_policy_frames": "chat",
|
| 12 |
+
"off_policy_max_seq_len": 2048,
|
| 13 |
+
"topk_targets": null,
|
| 14 |
+
"max_loss_tokens": null,
|
| 15 |
+
"loss_tokens_per_step": null,
|
| 16 |
+
"teacher_device": "cuda:0",
|
| 17 |
+
"student_device": "cuda:1",
|
| 18 |
+
"lr": 3e-05,
|
| 19 |
+
"optimizer": "adamw8bit",
|
| 20 |
+
"weight_decay": 0.1,
|
| 21 |
+
"epochs": 2,
|
| 22 |
+
"prompts_per_step": 256,
|
| 23 |
+
"group_size": 4,
|
| 24 |
+
"rollout_batch": 64,
|
| 25 |
+
"micro_batch": 4,
|
| 26 |
+
"max_new_tokens": 2048,
|
| 27 |
+
"max_prompt_len": 1024,
|
| 28 |
+
"warmup_steps": 10,
|
| 29 |
+
"max_grad_norm": 1.0,
|
| 30 |
+
"eval_every": 10,
|
| 31 |
+
"gsm8k_every": 20,
|
| 32 |
+
"gsm8k_n": 256,
|
| 33 |
+
"gsm8k_batch": 16,
|
| 34 |
+
"gsm8k_max_new_tokens": 1024,
|
| 35 |
+
"gsm8k_frames": "chat",
|
| 36 |
+
"save_every": 1000,
|
| 37 |
+
"out_dir": "outputs/healed/mixonly_keep50",
|
| 38 |
+
"sweep": 60,
|
| 39 |
+
"wandb": true,
|
| 40 |
+
"wandb_project": "glean-heal",
|
| 41 |
+
"wandb_run_name": "mixonly_keep50-s1223",
|
| 42 |
+
"wandb_run_id": null,
|
| 43 |
+
"wandb_resume": null,
|
| 44 |
+
"wandb_mode": "offline",
|
| 45 |
+
"no_wandb_sync": false,
|
| 46 |
+
"debug": false,
|
| 47 |
+
"resume_from": null,
|
| 48 |
+
"start_step": 0,
|
| 49 |
+
"no_grad_checkpointing": false,
|
| 50 |
+
"seed": 1223,
|
| 51 |
+
"no_teacher_overlap": false,
|
| 52 |
+
"sync_checkpoints": false,
|
| 53 |
+
"rollout_engine": "vllm",
|
| 54 |
+
"vllm_gpu": "2",
|
| 55 |
+
"vllm_port": 8377,
|
| 56 |
+
"vllm_refresh_every": 1,
|
| 57 |
+
"vllm_serve_bin": "vllm-plugin/.venv/bin/python",
|
| 58 |
+
"vllm_gpu_mem_util": 0.85,
|
| 59 |
+
"liger_loss": true,
|
| 60 |
+
"gold_mix_lambda": 0.5,
|
| 61 |
+
"gold_topk_targets": "outputs/teacher_trajectories/dolci_combined_top128",
|
| 62 |
+
"gold_mix_decay": 0.0,
|
| 63 |
+
"fast_teacher": true,
|
| 64 |
+
"reference_kl_beta": 0.0,
|
| 65 |
+
"drop_truncated_rollouts": false,
|
| 66 |
+
"vllm_max_model_len": null,
|
| 67 |
+
"vllm_refresh_mode": "reload",
|
| 68 |
+
"vllm_live_dir": null,
|
| 69 |
+
"resolved_kl_direction": "reverse"
|
| 70 |
+
}
|
healed/mixonly_keep50/train_log.jsonl
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"step": 1, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.64160251680288, "tokens": 180076, "cumulative_loss_tokens": 180076, "grad_norm": 4.375, "lr": 6e-06, "finish_rate": 0.988, "comp_len": 703.4, "dropped_truncated": 0, "gold_loss": 0.2766, "gold_lambda": 0.5, "rep_ratio": 2.441, "t_data_s": 0.0, "t_rollout_s": 54.0, "t_step_s": 132.5, "t_refresh_s": 0.3, "mem_gb": 10.07, "mem_gb_teacher": 13.64}
|
| 2 |
+
{"step": 1, "gsm8k_n": 256, "gsm8k_quick_chat": 0.59765625, "t_eval_s": 23.8}
|
| 3 |
+
{"step": 2, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5804529535180645, "tokens": 184371, "cumulative_loss_tokens": 364447, "grad_norm": 4.34375, "lr": 9e-06, "finish_rate": 0.98, "comp_len": 720.2, "dropped_truncated": 0, "gold_loss": 0.2539, "gold_lambda": 0.5, "rep_ratio": 2.415, "t_data_s": 0.0, "t_rollout_s": 55.2, "t_step_s": 120.0, "t_refresh_s": 0.3, "mem_gb": 10.74, "mem_gb_teacher": 13.51}
|
| 4 |
+
{"step": 3, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6017302746850163, "tokens": 179285, "cumulative_loss_tokens": 543732, "grad_norm": 3.53125, "lr": 1.2e-05, "finish_rate": 0.957, "comp_len": 700.3, "dropped_truncated": 0, "gold_loss": 0.271, "gold_lambda": 0.5, "rep_ratio": 2.491, "t_data_s": 0.0, "t_rollout_s": 53.3, "t_step_s": 118.2, "t_refresh_s": 0.3, "mem_gb": 10.62, "mem_gb_teacher": 13.57}
|
| 5 |
+
{"step": 4, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5312556731148038, "tokens": 160812, "cumulative_loss_tokens": 704544, "grad_norm": 3.359375, "lr": 1.5e-05, "finish_rate": 0.992, "comp_len": 628.2, "dropped_truncated": 0, "gold_loss": 0.2614, "gold_lambda": 0.5, "rep_ratio": 2.538, "t_data_s": 0.0, "t_rollout_s": 48.5, "t_step_s": 109.0, "t_refresh_s": 0.3, "mem_gb": 10.62, "mem_gb_teacher": 13.53}
|
| 6 |
+
{"step": 5, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5771709610503701, "tokens": 162424, "cumulative_loss_tokens": 866968, "grad_norm": 4.21875, "lr": 1.8e-05, "finish_rate": 0.992, "comp_len": 634.5, "dropped_truncated": 0, "gold_loss": 1.0453, "gold_lambda": 0.5, "rep_ratio": 2.573, "t_data_s": 0.0, "t_rollout_s": 49.0, "t_step_s": 111.2, "t_refresh_s": 0.3, "mem_gb": 9.91, "mem_gb_teacher": 13.5}
|
| 7 |
+
{"step": 6, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.49112553297794564, "tokens": 167277, "cumulative_loss_tokens": 1034245, "grad_norm": 2.765625, "lr": 2.1e-05, "finish_rate": 0.969, "comp_len": 653.4, "dropped_truncated": 0, "gold_loss": 0.7951, "gold_lambda": 0.5, "rep_ratio": 2.475, "t_data_s": 0.0, "t_rollout_s": 52.6, "t_step_s": 114.7, "t_refresh_s": 0.3, "mem_gb": 10.07, "mem_gb_teacher": 13.55}
|
| 8 |
+
{"step": 7, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.46942076250366277, "tokens": 182789, "cumulative_loss_tokens": 1217034, "grad_norm": 2.359375, "lr": 2.4e-05, "finish_rate": 0.98, "comp_len": 714.0, "dropped_truncated": 0, "gold_loss": 1.0811, "gold_lambda": 0.5, "rep_ratio": 2.421, "t_data_s": 0.0, "t_rollout_s": 54.6, "t_step_s": 120.7, "t_refresh_s": 0.3, "mem_gb": 10.17, "mem_gb_teacher": 13.51}
|
| 9 |
+
{"step": 8, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.49086518251754685, "tokens": 178960, "cumulative_loss_tokens": 1395994, "grad_norm": 2.265625, "lr": 2.7000000000000002e-05, "finish_rate": 0.988, "comp_len": 699.1, "dropped_truncated": 0, "gold_loss": 1.1405, "gold_lambda": 0.5, "rep_ratio": 2.698, "t_data_s": 0.0, "t_rollout_s": 52.1, "t_step_s": 113.6, "t_refresh_s": 0.3, "mem_gb": 10.21, "mem_gb_teacher": 13.61}
|
| 10 |
+
{"step": 9, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6112907274088395, "tokens": 181928, "cumulative_loss_tokens": 1577922, "grad_norm": 1.4375, "lr": 3e-05, "finish_rate": 0.988, "comp_len": 710.7, "dropped_truncated": 0, "gold_loss": 0.2885, "gold_lambda": 0.5, "rep_ratio": 2.417, "t_data_s": 0.0, "t_rollout_s": 54.8, "t_step_s": 119.8, "t_refresh_s": 0.3, "mem_gb": 10.33, "mem_gb_teacher": 13.55}
|
| 11 |
+
{"step": 10, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5313742167280878, "tokens": 181629, "cumulative_loss_tokens": 1759551, "grad_norm": 1.21875, "lr": 3e-05, "finish_rate": 0.977, "comp_len": 709.5, "dropped_truncated": 0, "gold_loss": 0.2864, "gold_lambda": 0.5, "rep_ratio": 2.739, "t_data_s": 0.0, "t_rollout_s": 53.8, "t_step_s": 118.0, "t_refresh_s": 0.3, "mem_gb": 10.57, "mem_gb_teacher": 13.54}
|
| 12 |
+
{"step": 11, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4908617648609082, "tokens": 170172, "cumulative_loss_tokens": 1929723, "grad_norm": 8.0, "lr": 3e-05, "finish_rate": 0.996, "comp_len": 664.7, "dropped_truncated": 0, "gold_loss": 1.4057, "gold_lambda": 0.5, "rep_ratio": 2.715, "t_data_s": 0.0, "t_rollout_s": 50.5, "t_step_s": 112.2, "t_refresh_s": 0.3, "mem_gb": 10.09, "mem_gb_teacher": 13.57}
|
| 13 |
+
{"step": 12, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5093631678122801, "tokens": 195821, "cumulative_loss_tokens": 2125544, "grad_norm": 5.28125, "lr": 3e-05, "finish_rate": 0.977, "comp_len": 764.9, "dropped_truncated": 0, "gold_loss": 1.2099, "gold_lambda": 0.5, "rep_ratio": 2.709, "t_data_s": 0.0, "t_rollout_s": 57.1, "t_step_s": 127.1, "t_refresh_s": 0.3, "mem_gb": 10.28, "mem_gb_teacher": 13.63}
|
| 14 |
+
{"step": 13, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5889715160570573, "tokens": 176993, "cumulative_loss_tokens": 2302537, "grad_norm": 3.640625, "lr": 3e-05, "finish_rate": 0.977, "comp_len": 691.4, "dropped_truncated": 0, "gold_loss": 1.2551, "gold_lambda": 0.5, "rep_ratio": 2.421, "t_data_s": 0.0, "t_rollout_s": 51.6, "t_step_s": 115.7, "t_refresh_s": 0.3, "mem_gb": 9.99, "mem_gb_teacher": 13.49}
|
| 15 |
+
{"step": 14, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5329977462907185, "tokens": 172763, "cumulative_loss_tokens": 2475300, "grad_norm": 3.921875, "lr": 3e-05, "finish_rate": 0.977, "comp_len": 674.9, "dropped_truncated": 0, "gold_loss": 1.1983, "gold_lambda": 0.5, "rep_ratio": 2.534, "t_data_s": 0.0, "t_rollout_s": 51.4, "t_step_s": 113.2, "t_refresh_s": 0.3, "mem_gb": 9.99, "mem_gb_teacher": 13.53}
|
| 16 |
+
{"step": 15, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5466928472964669, "tokens": 189465, "cumulative_loss_tokens": 2664765, "grad_norm": 2.25, "lr": 3e-05, "finish_rate": 0.973, "comp_len": 740.1, "dropped_truncated": 0, "gold_loss": 0.2453, "gold_lambda": 0.5, "rep_ratio": 2.76, "t_data_s": 0.0, "t_rollout_s": 54.9, "t_step_s": 121.2, "t_refresh_s": 0.3, "mem_gb": 10.52, "mem_gb_teacher": 13.55}
|
| 17 |
+
{"step": 16, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5614842639313343, "tokens": 185066, "cumulative_loss_tokens": 2849831, "grad_norm": 1.921875, "lr": 3e-05, "finish_rate": 0.969, "comp_len": 722.9, "dropped_truncated": 0, "gold_loss": 0.2391, "gold_lambda": 0.5, "rep_ratio": 2.488, "t_data_s": 0.0, "t_rollout_s": 54.9, "t_step_s": 122.9, "t_refresh_s": 0.3, "mem_gb": 10.66, "mem_gb_teacher": 13.5}
|
| 18 |
+
{"step": 17, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6893614743820268, "tokens": 168385, "cumulative_loss_tokens": 3018216, "grad_norm": 1.9296875, "lr": 3e-05, "finish_rate": 0.98, "comp_len": 657.8, "dropped_truncated": 0, "gold_loss": 0.9408, "gold_lambda": 0.5, "rep_ratio": 2.476, "t_data_s": 0.0, "t_rollout_s": 51.0, "t_step_s": 113.6, "t_refresh_s": 0.3, "mem_gb": 10.23, "mem_gb_teacher": 13.58}
|
| 19 |
+
{"step": 18, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.614498667786979, "tokens": 175240, "cumulative_loss_tokens": 3193456, "grad_norm": 1.3671875, "lr": 3e-05, "finish_rate": 0.973, "comp_len": 684.5, "dropped_truncated": 0, "gold_loss": 0.9207, "gold_lambda": 0.5, "rep_ratio": 2.421, "t_data_s": 0.0, "t_rollout_s": 54.5, "t_step_s": 118.6, "t_refresh_s": 0.3, "mem_gb": 10.42, "mem_gb_teacher": 13.68}
|
| 20 |
+
{"step": 19, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.6265725124825673, "tokens": 174073, "cumulative_loss_tokens": 3367529, "grad_norm": 1.46875, "lr": 3e-05, "finish_rate": 0.969, "comp_len": 680.0, "dropped_truncated": 0, "gold_loss": 1.1071, "gold_lambda": 0.5, "rep_ratio": 2.444, "t_data_s": 0.0, "t_rollout_s": 53.1, "t_step_s": 119.2, "t_refresh_s": 0.3, "mem_gb": 10.12, "mem_gb_teacher": 13.57}
|
| 21 |
+
{"step": 20, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5855338332263215, "tokens": 192790, "cumulative_loss_tokens": 3560319, "grad_norm": 2.03125, "lr": 3e-05, "finish_rate": 0.965, "comp_len": 753.1, "dropped_truncated": 0, "gold_loss": 1.224, "gold_lambda": 0.5, "rep_ratio": 2.746, "t_data_s": 0.0, "t_rollout_s": 57.2, "t_step_s": 128.1, "t_refresh_s": 0.3, "mem_gb": 10.2, "mem_gb_teacher": 13.61}
|
| 22 |
+
{"step": 20, "gsm8k_n": 256, "gsm8k_quick_chat": 0.5625, "t_eval_s": 21.6}
|
| 23 |
+
{"step": 21, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7211202926032049, "tokens": 153388, "cumulative_loss_tokens": 3713707, "grad_norm": 3.21875, "lr": 3e-05, "finish_rate": 0.977, "comp_len": 599.2, "dropped_truncated": 0, "gold_loss": 0.2009, "gold_lambda": 0.5, "rep_ratio": 2.488, "t_data_s": 0.0, "t_rollout_s": 47.3, "t_step_s": 108.8, "t_refresh_s": 0.3, "mem_gb": 10.62, "mem_gb_teacher": 13.65}
|
| 24 |
+
{"step": 22, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.067914567337345, "tokens": 201926, "cumulative_loss_tokens": 3915633, "grad_norm": 29.875, "lr": 3e-05, "finish_rate": 0.91, "comp_len": 788.8, "dropped_truncated": 0, "gold_loss": 0.331, "gold_lambda": 0.5, "rep_ratio": 2.649, "t_data_s": 0.0, "t_rollout_s": 60.8, "t_step_s": 138.6, "t_refresh_s": 0.3, "mem_gb": 10.67, "mem_gb_teacher": 13.65}
|
| 25 |
+
{"step": 23, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1885152613010703, "tokens": 75419, "cumulative_loss_tokens": 3991052, "grad_norm": 16.75, "lr": 3e-05, "finish_rate": 0.992, "comp_len": 294.6, "dropped_truncated": 0, "gold_loss": 0.702, "gold_lambda": 0.5, "rep_ratio": 2.069, "t_data_s": 0.0, "t_rollout_s": 34.8, "t_step_s": 84.7, "t_refresh_s": 0.3, "mem_gb": 10.1, "mem_gb_teacher": 13.57}
|
| 26 |
+
{"step": 24, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.0524406387251954, "tokens": 108957, "cumulative_loss_tokens": 4100009, "grad_norm": 12.625, "lr": 3e-05, "finish_rate": 0.992, "comp_len": 425.6, "dropped_truncated": 0, "gold_loss": 0.7187, "gold_lambda": 0.5, "rep_ratio": 2.242, "t_data_s": 0.0, "t_rollout_s": 39.7, "t_step_s": 92.6, "t_refresh_s": 0.3, "mem_gb": 9.88, "mem_gb_teacher": 13.5}
|
| 27 |
+
{"step": 25, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8443096130165939, "tokens": 117861, "cumulative_loss_tokens": 4217870, "grad_norm": 6.28125, "lr": 3e-05, "finish_rate": 0.977, "comp_len": 460.4, "dropped_truncated": 0, "gold_loss": 1.3743, "gold_lambda": 0.5, "rep_ratio": 2.388, "t_data_s": 0.0, "t_rollout_s": 41.9, "t_step_s": 98.2, "t_refresh_s": 0.3, "mem_gb": 10.32, "mem_gb_teacher": 13.65}
|
| 28 |
+
{"step": 26, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8987009880359061, "tokens": 109466, "cumulative_loss_tokens": 4327336, "grad_norm": 6.59375, "lr": 3e-05, "finish_rate": 0.992, "comp_len": 427.6, "dropped_truncated": 0, "gold_loss": 1.1959, "gold_lambda": 0.5, "rep_ratio": 2.193, "t_data_s": 0.0, "t_rollout_s": 40.9, "t_step_s": 95.5, "t_refresh_s": 0.3, "mem_gb": 10.15, "mem_gb_teacher": 13.59}
|
| 29 |
+
{"step": 27, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8460174916677944, "tokens": 107677, "cumulative_loss_tokens": 4435013, "grad_norm": 5.375, "lr": 3e-05, "finish_rate": 0.992, "comp_len": 420.6, "dropped_truncated": 0, "gold_loss": 0.8485, "gold_lambda": 0.5, "rep_ratio": 2.235, "t_data_s": 0.0, "t_rollout_s": 40.2, "t_step_s": 95.6, "t_refresh_s": 0.3, "mem_gb": 10.08, "mem_gb_teacher": 13.57}
|
| 30 |
+
{"step": 28, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7708598505660351, "tokens": 144090, "cumulative_loss_tokens": 4579103, "grad_norm": 3.0625, "lr": 3e-05, "finish_rate": 0.977, "comp_len": 562.9, "dropped_truncated": 0, "gold_loss": 0.8888, "gold_lambda": 0.5, "rep_ratio": 2.31, "t_data_s": 0.0, "t_rollout_s": 45.6, "t_step_s": 106.4, "t_refresh_s": 0.3, "mem_gb": 10.03, "mem_gb_teacher": 13.54}
|
| 31 |
+
{"step": 29, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7552366074717857, "tokens": 148802, "cumulative_loss_tokens": 4727905, "grad_norm": 3.4375, "lr": 3e-05, "finish_rate": 0.949, "comp_len": 581.3, "dropped_truncated": 0, "gold_loss": 0.3015, "gold_lambda": 0.5, "rep_ratio": 2.271, "t_data_s": 0.0, "t_rollout_s": 47.7, "t_step_s": 110.9, "t_refresh_s": 0.3, "mem_gb": 10.37, "mem_gb_teacher": 13.59}
|
| 32 |
+
{"step": 30, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7948331056197779, "tokens": 194950, "cumulative_loss_tokens": 4922855, "grad_norm": 4.15625, "lr": 3e-05, "finish_rate": 0.941, "comp_len": 761.5, "dropped_truncated": 0, "gold_loss": 0.3372, "gold_lambda": 0.5, "rep_ratio": 2.294, "t_data_s": 0.0, "t_rollout_s": 58.6, "t_step_s": 130.6, "t_refresh_s": 0.3, "mem_gb": 10.67, "mem_gb_teacher": 13.58}
|
| 33 |
+
{"step": 31, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.727050904920988, "tokens": 242212, "cumulative_loss_tokens": 5165067, "grad_norm": 10.375, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 946.1, "dropped_truncated": 0, "gold_loss": 1.2321, "gold_lambda": 0.5, "rep_ratio": 3.08, "t_data_s": 0.0, "t_rollout_s": 81.0, "t_step_s": 162.4, "t_refresh_s": 0.3, "mem_gb": 10.1, "mem_gb_teacher": 13.57}
|
| 34 |
+
{"step": 32, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7908006863976065, "tokens": 223097, "cumulative_loss_tokens": 5388164, "grad_norm": 12.3125, "lr": 3e-05, "finish_rate": 0.898, "comp_len": 871.5, "dropped_truncated": 0, "gold_loss": 1.0474, "gold_lambda": 0.5, "rep_ratio": 2.355, "t_data_s": 0.0, "t_rollout_s": 68.3, "t_step_s": 150.8, "t_refresh_s": 0.3, "mem_gb": 10.21, "mem_gb_teacher": 13.61}
|
| 35 |
+
{"step": 33, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8169354122986394, "tokens": 286531, "cumulative_loss_tokens": 5674695, "grad_norm": 20.875, "lr": 3e-05, "finish_rate": 0.758, "comp_len": 1119.3, "dropped_truncated": 0, "gold_loss": 1.3302, "gold_lambda": 0.5, "rep_ratio": 2.819, "t_data_s": 0.0, "t_rollout_s": 103.9, "t_step_s": 195.8, "t_refresh_s": 0.3, "mem_gb": 10.33, "mem_gb_teacher": 13.64}
|
| 36 |
+
{"step": 34, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7717946813290117, "tokens": 322535, "cumulative_loss_tokens": 5997230, "grad_norm": 28.25, "lr": 3e-05, "finish_rate": 0.727, "comp_len": 1259.9, "dropped_truncated": 0, "gold_loss": 1.9537, "gold_lambda": 0.5, "rep_ratio": 3.725, "t_data_s": 0.0, "t_rollout_s": 118.8, "t_step_s": 213.6, "t_refresh_s": 0.3, "mem_gb": 10.4, "mem_gb_teacher": 13.67}
|
| 37 |
+
{"step": 35, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9259877295891465, "tokens": 396632, "cumulative_loss_tokens": 6393862, "grad_norm": 6.875, "lr": 3e-05, "finish_rate": 0.422, "comp_len": 1549.3, "dropped_truncated": 0, "gold_loss": 0.3645, "gold_lambda": 0.5, "rep_ratio": 6.038, "t_data_s": 0.0, "t_rollout_s": 157.8, "t_step_s": 257.5, "t_refresh_s": 0.3, "mem_gb": 10.42, "mem_gb_teacher": 13.67}
|
| 38 |
+
{"step": 36, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1080191085919773, "tokens": 444782, "cumulative_loss_tokens": 6838644, "grad_norm": 8.6875, "lr": 3e-05, "finish_rate": 0.262, "comp_len": 1737.4, "dropped_truncated": 0, "gold_loss": 0.5514, "gold_lambda": 0.5, "rep_ratio": 4.155, "t_data_s": 0.0, "t_rollout_s": 177.5, "t_step_s": 289.2, "t_refresh_s": 0.3, "mem_gb": 10.21, "mem_gb_teacher": 13.6}
|
| 39 |
+
{"step": 37, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.0517108480146504, "tokens": 419060, "cumulative_loss_tokens": 7257704, "grad_norm": 10.375, "lr": 3e-05, "finish_rate": 0.328, "comp_len": 1637.0, "dropped_truncated": 0, "gold_loss": 1.485, "gold_lambda": 0.5, "rep_ratio": 5.223, "t_data_s": 0.0, "t_rollout_s": 167.3, "t_step_s": 268.9, "t_refresh_s": 0.3, "mem_gb": 10.43, "mem_gb_teacher": 13.68}
|
| 40 |
+
{"step": 38, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.6416851874601466, "tokens": 367687, "cumulative_loss_tokens": 7625391, "grad_norm": 48.5, "lr": 3e-05, "finish_rate": 0.453, "comp_len": 1436.3, "dropped_truncated": 0, "gold_loss": 1.8324, "gold_lambda": 0.5, "rep_ratio": 2.234, "t_data_s": 0.0, "t_rollout_s": 147.2, "t_step_s": 247.5, "t_refresh_s": 0.3, "mem_gb": 10.28, "mem_gb_teacher": 13.63}
|
| 41 |
+
{"step": 39, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.5387520026285295, "tokens": 228164, "cumulative_loss_tokens": 7853555, "grad_norm": 45.5, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 891.3, "dropped_truncated": 0, "gold_loss": 0.6286, "gold_lambda": 0.5, "rep_ratio": 1.432, "t_data_s": 0.0, "t_rollout_s": 72.4, "t_step_s": 157.0, "t_refresh_s": 0.3, "mem_gb": 10.36, "mem_gb_teacher": 13.64}
|
| 42 |
+
{"step": 40, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.6039773274506475, "tokens": 132404, "cumulative_loss_tokens": 7985959, "grad_norm": 13.25, "lr": 3e-05, "finish_rate": 0.945, "comp_len": 517.2, "dropped_truncated": 0, "gold_loss": 0.4578, "gold_lambda": 0.5, "rep_ratio": 1.992, "t_data_s": 0.0, "t_rollout_s": 42.2, "t_step_s": 110.0, "t_refresh_s": 0.3, "mem_gb": 10.1, "mem_gb_teacher": 13.57}
|
| 43 |
+
{"step": 40, "gsm8k_n": 256, "gsm8k_quick_chat": 0.39453125, "t_eval_s": 25.1}
|
| 44 |
+
{"step": 41, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.4861478830602124, "tokens": 106912, "cumulative_loss_tokens": 8092871, "grad_norm": 19.375, "lr": 3e-05, "finish_rate": 0.969, "comp_len": 417.6, "dropped_truncated": 0, "gold_loss": 1.5294, "gold_lambda": 0.5, "rep_ratio": 5.931, "t_data_s": 0.0, "t_rollout_s": 40.0, "t_step_s": 99.7, "t_refresh_s": 0.3, "mem_gb": 10.1, "mem_gb_teacher": 13.57}
|
| 45 |
+
{"step": 42, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.5399118802486043, "tokens": 142363, "cumulative_loss_tokens": 8235234, "grad_norm": 10.75, "lr": 3e-05, "finish_rate": 0.984, "comp_len": 556.1, "dropped_truncated": 0, "gold_loss": 1.2452, "gold_lambda": 0.5, "rep_ratio": 1.479, "t_data_s": 0.0, "t_rollout_s": 45.7, "t_step_s": 106.3, "t_refresh_s": 0.3, "mem_gb": 9.95, "mem_gb_teacher": 13.52}
|
| 46 |
+
{"step": 43, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.416146864755438, "tokens": 166726, "cumulative_loss_tokens": 8401960, "grad_norm": 7.96875, "lr": 3e-05, "finish_rate": 0.891, "comp_len": 651.3, "dropped_truncated": 0, "gold_loss": 1.5954, "gold_lambda": 0.5, "rep_ratio": 2.054, "t_data_s": 0.0, "t_rollout_s": 55.3, "t_step_s": 132.5, "t_refresh_s": 0.3, "mem_gb": 10.07, "mem_gb_teacher": 13.56}
|
| 47 |
+
{"step": 44, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.3857274030623108, "tokens": 134182, "cumulative_loss_tokens": 8536142, "grad_norm": 8.125, "lr": 3e-05, "finish_rate": 0.953, "comp_len": 524.1, "dropped_truncated": 0, "gold_loss": 1.7352, "gold_lambda": 0.5, "rep_ratio": 4.157, "t_data_s": 0.0, "t_rollout_s": 45.6, "t_step_s": 106.6, "t_refresh_s": 0.3, "mem_gb": 10.1, "mem_gb_teacher": 13.57}
|
| 48 |
+
{"step": 45, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.6115993951621064, "tokens": 142849, "cumulative_loss_tokens": 8678991, "grad_norm": 5.75, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 558.0, "dropped_truncated": 0, "gold_loss": 1.1847, "gold_lambda": 0.5, "rep_ratio": 3.488, "t_data_s": 0.0, "t_rollout_s": 50.0, "t_step_s": 123.0, "t_refresh_s": 0.3, "mem_gb": 10.08, "mem_gb_teacher": 13.57}
|
| 49 |
+
{"step": 46, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.3081135477824917, "tokens": 246696, "cumulative_loss_tokens": 8925687, "grad_norm": 4.8125, "lr": 3e-05, "finish_rate": 0.91, "comp_len": 963.7, "dropped_truncated": 0, "gold_loss": 1.1953, "gold_lambda": 0.5, "rep_ratio": 2.181, "t_data_s": 0.0, "t_rollout_s": 80.8, "t_step_s": 164.8, "t_refresh_s": 0.3, "mem_gb": 10.08, "mem_gb_teacher": 13.56}
|
| 50 |
+
{"step": 47, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.3434679277120278, "tokens": 251881, "cumulative_loss_tokens": 9177568, "grad_norm": 4.53125, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 983.9, "dropped_truncated": 0, "gold_loss": 0.4438, "gold_lambda": 0.5, "rep_ratio": 5.83, "t_data_s": 0.0, "t_rollout_s": 99.2, "t_step_s": 187.6, "t_refresh_s": 0.3, "mem_gb": 10.63, "mem_gb_teacher": 13.57}
|
| 51 |
+
{"step": 48, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.0025027087925982, "tokens": 260401, "cumulative_loss_tokens": 9437969, "grad_norm": 3.78125, "lr": 3e-05, "finish_rate": 0.777, "comp_len": 1017.2, "dropped_truncated": 0, "gold_loss": 0.4541, "gold_lambda": 0.5, "rep_ratio": 2.332, "t_data_s": 0.0, "t_rollout_s": 101.7, "t_step_s": 188.0, "t_refresh_s": 0.3, "mem_gb": 10.55, "mem_gb_teacher": 13.61}
|
| 52 |
+
{"step": 49, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1418893655655513, "tokens": 280455, "cumulative_loss_tokens": 9718424, "grad_norm": 11.5625, "lr": 3e-05, "finish_rate": 0.793, "comp_len": 1095.5, "dropped_truncated": 0, "gold_loss": 1.4994, "gold_lambda": 0.5, "rep_ratio": 2.672, "t_data_s": 0.0, "t_rollout_s": 105.5, "t_step_s": 194.5, "t_refresh_s": 0.3, "mem_gb": 10.26, "mem_gb_teacher": 13.62}
|
| 53 |
+
{"step": 50, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1426357563978717, "tokens": 276410, "cumulative_loss_tokens": 9994834, "grad_norm": 8.5, "lr": 3e-05, "finish_rate": 0.785, "comp_len": 1079.7, "dropped_truncated": 0, "gold_loss": 1.3239, "gold_lambda": 0.5, "rep_ratio": 2.225, "t_data_s": 0.0, "t_rollout_s": 105.7, "t_step_s": 197.7, "t_refresh_s": 0.3, "mem_gb": 10.14, "mem_gb_teacher": 13.61}
|
| 54 |
+
{"step": 51, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1355144874474041, "tokens": 258456, "cumulative_loss_tokens": 10253290, "grad_norm": 13.8125, "lr": 3e-05, "finish_rate": 0.75, "comp_len": 1009.6, "dropped_truncated": 0, "gold_loss": 1.3571, "gold_lambda": 0.5, "rep_ratio": 2.504, "t_data_s": 0.0, "t_rollout_s": 96.5, "t_step_s": 183.9, "t_refresh_s": 0.3, "mem_gb": 10.15, "mem_gb_teacher": 13.58}
|
| 55 |
+
{"step": 52, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1035922887892957, "tokens": 240934, "cumulative_loss_tokens": 10494224, "grad_norm": 16.125, "lr": 3e-05, "finish_rate": 0.812, "comp_len": 941.1, "dropped_truncated": 0, "gold_loss": 1.0417, "gold_lambda": 0.5, "rep_ratio": 2.275, "t_data_s": 0.0, "t_rollout_s": 79.7, "t_step_s": 163.8, "t_refresh_s": 0.3, "mem_gb": 10.17, "mem_gb_teacher": 13.6}
|
| 56 |
+
{"step": 53, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.0594787928429006, "tokens": 204779, "cumulative_loss_tokens": 10699003, "grad_norm": 7.09375, "lr": 3e-05, "finish_rate": 0.891, "comp_len": 799.9, "dropped_truncated": 0, "gold_loss": 1.142, "gold_lambda": 0.5, "rep_ratio": 2.822, "t_data_s": 0.0, "t_rollout_s": 67.6, "t_step_s": 145.8, "t_refresh_s": 0.3, "mem_gb": 10.43, "mem_gb_teacher": 13.69}
|
| 57 |
+
{"step": 54, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9060865840831999, "tokens": 196367, "cumulative_loss_tokens": 10895370, "grad_norm": 7.59375, "lr": 3e-05, "finish_rate": 0.887, "comp_len": 767.1, "dropped_truncated": 0, "gold_loss": 1.3938, "gold_lambda": 0.5, "rep_ratio": 3.417, "t_data_s": 0.0, "t_rollout_s": 62.0, "t_step_s": 137.1, "t_refresh_s": 0.3, "mem_gb": 10.07, "mem_gb_teacher": 13.56}
|
| 58 |
+
{"step": 55, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9142444251972912, "tokens": 170261, "cumulative_loss_tokens": 11065631, "grad_norm": 8.8125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 665.1, "dropped_truncated": 0, "gold_loss": 2.0206, "gold_lambda": 0.5, "rep_ratio": 2.457, "t_data_s": 0.0, "t_rollout_s": 56.3, "t_step_s": 126.4, "t_refresh_s": 0.3, "mem_gb": 10.09, "mem_gb_teacher": 13.57}
|
| 59 |
+
{"step": 56, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.863587859787581, "tokens": 174080, "cumulative_loss_tokens": 11239711, "grad_norm": 6.1875, "lr": 3e-05, "finish_rate": 0.93, "comp_len": 680.0, "dropped_truncated": 0, "gold_loss": 1.3418, "gold_lambda": 0.5, "rep_ratio": 2.256, "t_data_s": 0.0, "t_rollout_s": 56.5, "t_step_s": 126.7, "t_refresh_s": 0.3, "mem_gb": 10.16, "mem_gb_teacher": 13.59}
|
| 60 |
+
{"step": 57, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8766434817449779, "tokens": 161951, "cumulative_loss_tokens": 11401662, "grad_norm": 4.375, "lr": 3e-05, "finish_rate": 0.945, "comp_len": 632.6, "dropped_truncated": 0, "gold_loss": 1.0576, "gold_lambda": 0.5, "rep_ratio": 3.024, "t_data_s": 0.0, "t_rollout_s": 51.8, "t_step_s": 118.4, "t_refresh_s": 0.3, "mem_gb": 10.11, "mem_gb_teacher": 13.58}
|
| 61 |
+
{"step": 58, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8401733050490829, "tokens": 210930, "cumulative_loss_tokens": 11612592, "grad_norm": 4.125, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 823.9, "dropped_truncated": 0, "gold_loss": 1.2873, "gold_lambda": 0.5, "rep_ratio": 2.37, "t_data_s": 0.0, "t_rollout_s": 64.8, "t_step_s": 141.1, "t_refresh_s": 0.3, "mem_gb": 10.07, "mem_gb_teacher": 13.56}
|
healed/mixonly_keep50/vllm_server.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
healed/opd_warm_fixed_keep50/args.json
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"student": "outputs/healed/keep50_warmup_fixed_s1224/step0150",
|
| 3 |
+
"teacher": "allenai/OLMoE-1B-7B-0125-Instruct",
|
| 4 |
+
"training_mode": "on-policy",
|
| 5 |
+
"kl_direction": "reverse",
|
| 6 |
+
"dataset": "allenai/Dolci-Instruct-RL",
|
| 7 |
+
"dataset_sources": null,
|
| 8 |
+
"max_difficulty": null,
|
| 9 |
+
"trajectories": "outputs/teacher_trajectories/dolci_math_curated.jsonl",
|
| 10 |
+
"trajectory_dataset": "allenai/Dolci-Instruct-RL",
|
| 11 |
+
"off_policy_frames": "chat",
|
| 12 |
+
"off_policy_max_seq_len": 2048,
|
| 13 |
+
"topk_targets": null,
|
| 14 |
+
"max_loss_tokens": null,
|
| 15 |
+
"loss_tokens_per_step": null,
|
| 16 |
+
"teacher_device": "cuda:0",
|
| 17 |
+
"student_device": "cuda:1",
|
| 18 |
+
"lr": 1e-05,
|
| 19 |
+
"optimizer": "adamw8bit",
|
| 20 |
+
"weight_decay": 0.1,
|
| 21 |
+
"epochs": 2,
|
| 22 |
+
"prompts_per_step": 256,
|
| 23 |
+
"group_size": 4,
|
| 24 |
+
"rollout_batch": 64,
|
| 25 |
+
"micro_batch": 4,
|
| 26 |
+
"max_new_tokens": 2048,
|
| 27 |
+
"max_prompt_len": 1024,
|
| 28 |
+
"warmup_steps": 10,
|
| 29 |
+
"max_grad_norm": 1.0,
|
| 30 |
+
"eval_every": 10,
|
| 31 |
+
"gsm8k_every": 20,
|
| 32 |
+
"gsm8k_n": 256,
|
| 33 |
+
"gsm8k_batch": 16,
|
| 34 |
+
"gsm8k_max_new_tokens": 1024,
|
| 35 |
+
"gsm8k_frames": "chat",
|
| 36 |
+
"save_every": 100,
|
| 37 |
+
"out_dir": "outputs/healed/opd_warm_fixed_keep50",
|
| 38 |
+
"sweep": 500,
|
| 39 |
+
"wandb": true,
|
| 40 |
+
"wandb_project": "glean-heal",
|
| 41 |
+
"wandb_run_name": "opd-warm-long-keep50-s1223",
|
| 42 |
+
"wandb_run_id": null,
|
| 43 |
+
"wandb_resume": null,
|
| 44 |
+
"wandb_mode": "offline",
|
| 45 |
+
"no_wandb_sync": false,
|
| 46 |
+
"debug": false,
|
| 47 |
+
"resume_from": "outputs/healed/opd_warm_fixed_keep50/step0120",
|
| 48 |
+
"start_step": 120,
|
| 49 |
+
"no_grad_checkpointing": false,
|
| 50 |
+
"seed": 1223,
|
| 51 |
+
"no_teacher_overlap": false,
|
| 52 |
+
"sync_checkpoints": false,
|
| 53 |
+
"rollout_engine": "vllm",
|
| 54 |
+
"vllm_gpu": "2",
|
| 55 |
+
"vllm_port": 8377,
|
| 56 |
+
"vllm_refresh_every": 1,
|
| 57 |
+
"vllm_serve_bin": "vllm-plugin/.venv/bin/python",
|
| 58 |
+
"vllm_gpu_mem_util": 0.85,
|
| 59 |
+
"liger_loss": true,
|
| 60 |
+
"gold_mix_lambda": 0.5,
|
| 61 |
+
"gold_topk_targets": "outputs/teacher_trajectories/dolci_combined_top128",
|
| 62 |
+
"gold_loss": "ce",
|
| 63 |
+
"gold_mix_decay": 0.0,
|
| 64 |
+
"fast_teacher": false,
|
| 65 |
+
"reference_kl_beta": 0.05,
|
| 66 |
+
"drop_truncated_rollouts": false,
|
| 67 |
+
"vllm_max_model_len": null,
|
| 68 |
+
"vllm_refresh_mode": "reload",
|
| 69 |
+
"vllm_live_dir": null,
|
| 70 |
+
"resolved_kl_direction": "reverse"
|
| 71 |
+
}
|
healed/opd_warm_fixed_keep50/train_log.jsonl
ADDED
|
@@ -0,0 +1,235 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"step": 1, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.44963827088755237, "tokens": 194827, "cumulative_loss_tokens": 194827, "grad_norm": 2.0625, "lr": 2.0000000000000003e-06, "finish_rate": 0.957, "comp_len": 761.0, "dropped_truncated": 0, "gold_loss": 0.2115, "gold_lambda": 0.5, "rep_ratio": 2.401, "t_data_s": 0.0, "t_rollout_s": 56.4, "t_step_s": 182.8, "t_refresh_s": 0.3, "mem_gb": 17.24, "mem_gb_teacher": 20.26}
|
| 2 |
+
{"step": 1, "gsm8k_n": 256, "gsm8k_quick_chat": 0.65234375, "t_eval_s": 23.8}
|
| 3 |
+
{"step": 2, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3940014742786295, "tokens": 187987, "cumulative_loss_tokens": 382814, "grad_norm": 2.0, "lr": 3e-06, "finish_rate": 0.992, "comp_len": 734.3, "dropped_truncated": 0, "gold_loss": 0.2068, "gold_lambda": 0.5, "rep_ratio": 2.369, "t_data_s": 0.0, "t_rollout_s": 56.2, "t_step_s": 157.9, "t_refresh_s": 0.3, "mem_gb": 17.22, "mem_gb_teacher": 20.24}
|
| 4 |
+
{"step": 3, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3842378264528005, "tokens": 175719, "cumulative_loss_tokens": 558533, "grad_norm": 1.9296875, "lr": 4.000000000000001e-06, "finish_rate": 0.969, "comp_len": 686.4, "dropped_truncated": 0, "gold_loss": 0.2386, "gold_lambda": 0.5, "rep_ratio": 2.451, "t_data_s": 0.0, "t_rollout_s": 52.3, "t_step_s": 150.2, "t_refresh_s": 0.3, "mem_gb": 17.33, "mem_gb_teacher": 20.27}
|
| 5 |
+
{"step": 4, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3169176116394187, "tokens": 183284, "cumulative_loss_tokens": 741817, "grad_norm": 1.703125, "lr": 5e-06, "finish_rate": 0.973, "comp_len": 716.0, "dropped_truncated": 0, "gold_loss": 0.2391, "gold_lambda": 0.5, "rep_ratio": 2.682, "t_data_s": 0.0, "t_rollout_s": 53.4, "t_step_s": 156.8, "t_refresh_s": 0.3, "mem_gb": 17.1, "mem_gb_teacher": 20.22}
|
| 6 |
+
{"step": 5, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.31469852179097824, "tokens": 169650, "cumulative_loss_tokens": 911467, "grad_norm": 2.625, "lr": 6e-06, "finish_rate": 0.988, "comp_len": 662.7, "dropped_truncated": 0, "gold_loss": 1.3211, "gold_lambda": 0.5, "rep_ratio": 2.526, "t_data_s": 0.0, "t_rollout_s": 50.1, "t_step_s": 144.7, "t_refresh_s": 0.3, "mem_gb": 17.17, "mem_gb_teacher": 20.24}
|
| 7 |
+
{"step": 6, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3012831195711605, "tokens": 182143, "cumulative_loss_tokens": 1093610, "grad_norm": 2.203125, "lr": 7e-06, "finish_rate": 0.949, "comp_len": 711.5, "dropped_truncated": 0, "gold_loss": 0.9775, "gold_lambda": 0.5, "rep_ratio": 2.403, "t_data_s": 0.0, "t_rollout_s": 56.4, "t_step_s": 154.9, "t_refresh_s": 0.3, "mem_gb": 17.26, "mem_gb_teacher": 20.26}
|
| 8 |
+
{"step": 7, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2884359588736265, "tokens": 182618, "cumulative_loss_tokens": 1276228, "grad_norm": 2.640625, "lr": 8.000000000000001e-06, "finish_rate": 0.977, "comp_len": 713.4, "dropped_truncated": 0, "gold_loss": 1.3168, "gold_lambda": 0.5, "rep_ratio": 2.266, "t_data_s": 0.0, "t_rollout_s": 53.8, "t_step_s": 154.1, "t_refresh_s": 0.3, "mem_gb": 17.47, "mem_gb_teacher": 20.28}
|
| 9 |
+
{"step": 8, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25864222868998754, "tokens": 182352, "cumulative_loss_tokens": 1458580, "grad_norm": 2.9375, "lr": 9e-06, "finish_rate": 0.969, "comp_len": 712.3, "dropped_truncated": 0, "gold_loss": 1.3305, "gold_lambda": 0.5, "rep_ratio": 2.607, "t_data_s": 0.0, "t_rollout_s": 53.3, "t_step_s": 153.4, "t_refresh_s": 0.3, "mem_gb": 17.53, "mem_gb_teacher": 20.29}
|
| 10 |
+
{"step": 9, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4240309239163299, "tokens": 161654, "cumulative_loss_tokens": 1620234, "grad_norm": 1.8046875, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 631.5, "dropped_truncated": 0, "gold_loss": 0.2139, "gold_lambda": 0.5, "rep_ratio": 2.185, "t_data_s": 0.0, "t_rollout_s": 49.1, "t_step_s": 143.8, "t_refresh_s": 0.3, "mem_gb": 17.52, "mem_gb_teacher": 20.28}
|
| 11 |
+
{"step": 10, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3422506856649129, "tokens": 182466, "cumulative_loss_tokens": 1802700, "grad_norm": 1.4453125, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 712.8, "dropped_truncated": 0, "gold_loss": 0.2024, "gold_lambda": 0.5, "rep_ratio": 2.523, "t_data_s": 0.0, "t_rollout_s": 53.5, "t_step_s": 153.2, "t_refresh_s": 0.3, "mem_gb": 17.34, "mem_gb_teacher": 20.25}
|
| 12 |
+
{"step": 11, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3194563395594054, "tokens": 162668, "cumulative_loss_tokens": 1965368, "grad_norm": 2.875, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 635.4, "dropped_truncated": 0, "gold_loss": 1.4128, "gold_lambda": 0.5, "rep_ratio": 2.353, "t_data_s": 0.0, "t_rollout_s": 49.0, "t_step_s": 144.6, "t_refresh_s": 0.3, "mem_gb": 17.31, "mem_gb_teacher": 20.25}
|
| 13 |
+
{"step": 12, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.30020437081658174, "tokens": 172670, "cumulative_loss_tokens": 2138038, "grad_norm": 2.5, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 674.5, "dropped_truncated": 0, "gold_loss": 1.3403, "gold_lambda": 0.5, "rep_ratio": 2.41, "t_data_s": 0.0, "t_rollout_s": 52.5, "t_step_s": 152.4, "t_refresh_s": 0.3, "mem_gb": 17.45, "mem_gb_teacher": 20.3}
|
| 14 |
+
{"step": 13, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4090317845232767, "tokens": 160408, "cumulative_loss_tokens": 2298446, "grad_norm": 3.203125, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 626.6, "dropped_truncated": 0, "gold_loss": 1.4213, "gold_lambda": 0.5, "rep_ratio": 2.23, "t_data_s": 0.0, "t_rollout_s": 47.7, "t_step_s": 141.4, "t_refresh_s": 0.3, "mem_gb": 17.07, "mem_gb_teacher": 20.25}
|
| 15 |
+
{"step": 14, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34899632673731384, "tokens": 159410, "cumulative_loss_tokens": 2457856, "grad_norm": 2.6875, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 622.7, "dropped_truncated": 0, "gold_loss": 1.3901, "gold_lambda": 0.5, "rep_ratio": 2.422, "t_data_s": 0.0, "t_rollout_s": 47.6, "t_step_s": 141.7, "t_refresh_s": 0.3, "mem_gb": 17.12, "mem_gb_teacher": 20.25}
|
| 16 |
+
{"step": 15, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34543543075337274, "tokens": 166862, "cumulative_loss_tokens": 2624718, "grad_norm": 1.15625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 651.8, "dropped_truncated": 0, "gold_loss": 0.2286, "gold_lambda": 0.5, "rep_ratio": 2.459, "t_data_s": 0.0, "t_rollout_s": 48.7, "t_step_s": 145.8, "t_refresh_s": 0.3, "mem_gb": 17.25, "mem_gb_teacher": 20.24}
|
| 17 |
+
{"step": 16, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3638701808236742, "tokens": 154124, "cumulative_loss_tokens": 2778842, "grad_norm": 1.2109375, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 602.0, "dropped_truncated": 0, "gold_loss": 0.2253, "gold_lambda": 0.5, "rep_ratio": 2.369, "t_data_s": 0.0, "t_rollout_s": 47.2, "t_step_s": 141.5, "t_refresh_s": 0.3, "mem_gb": 17.09, "mem_gb_teacher": 20.23}
|
| 18 |
+
{"step": 17, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3916856637959853, "tokens": 153093, "cumulative_loss_tokens": 2931935, "grad_norm": 2.0625, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 598.0, "dropped_truncated": 0, "gold_loss": 1.0651, "gold_lambda": 0.5, "rep_ratio": 2.331, "t_data_s": 0.0, "t_rollout_s": 47.5, "t_step_s": 149.3, "t_refresh_s": 0.3, "mem_gb": 17.26, "mem_gb_teacher": 20.27}
|
| 19 |
+
{"step": 18, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3789892320845085, "tokens": 167800, "cumulative_loss_tokens": 3099735, "grad_norm": 1.96875, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 655.5, "dropped_truncated": 0, "gold_loss": 1.0761, "gold_lambda": 0.5, "rep_ratio": 2.334, "t_data_s": 0.0, "t_rollout_s": 52.0, "t_step_s": 153.1, "t_refresh_s": 0.3, "mem_gb": 17.18, "mem_gb_teacher": 20.24}
|
| 20 |
+
{"step": 19, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.35074403341776794, "tokens": 167173, "cumulative_loss_tokens": 3266908, "grad_norm": 2.390625, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 653.0, "dropped_truncated": 0, "gold_loss": 1.3153, "gold_lambda": 0.5, "rep_ratio": 2.337, "t_data_s": 0.0, "t_rollout_s": 50.2, "t_step_s": 143.5, "t_refresh_s": 0.3, "mem_gb": 17.32, "mem_gb_teacher": 20.25}
|
| 21 |
+
{"step": 20, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2626116644125829, "tokens": 190018, "cumulative_loss_tokens": 3456926, "grad_norm": 2.375, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 742.3, "dropped_truncated": 0, "gold_loss": 1.4092, "gold_lambda": 0.5, "rep_ratio": 2.615, "t_data_s": 0.0, "t_rollout_s": 55.8, "t_step_s": 158.7, "t_refresh_s": 0.3, "mem_gb": 17.61, "mem_gb_teacher": 20.29}
|
| 22 |
+
{"step": 20, "gsm8k_n": 256, "gsm8k_quick_chat": 0.6171875, "t_eval_s": 21.4}
|
| 23 |
+
{"step": 21, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36320468789413923, "tokens": 161040, "cumulative_loss_tokens": 3617966, "grad_norm": 1.4921875, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 629.1, "dropped_truncated": 0, "gold_loss": 0.2034, "gold_lambda": 0.5, "rep_ratio": 2.491, "t_data_s": 0.0, "t_rollout_s": 47.5, "t_step_s": 141.8, "t_refresh_s": 0.3, "mem_gb": 17.31, "mem_gb_teacher": 20.25}
|
| 24 |
+
{"step": 22, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3560513313296494, "tokens": 186196, "cumulative_loss_tokens": 3804162, "grad_norm": 1.140625, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 727.3, "dropped_truncated": 0, "gold_loss": 0.2012, "gold_lambda": 0.5, "rep_ratio": 2.621, "t_data_s": 0.0, "t_rollout_s": 56.3, "t_step_s": 157.9, "t_refresh_s": 0.3, "mem_gb": 17.48, "mem_gb_teacher": 20.28}
|
| 25 |
+
{"step": 23, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4002737975893586, "tokens": 172291, "cumulative_loss_tokens": 3976453, "grad_norm": 1.5546875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 673.0, "dropped_truncated": 0, "gold_loss": 0.7825, "gold_lambda": 0.5, "rep_ratio": 2.324, "t_data_s": 0.0, "t_rollout_s": 50.2, "t_step_s": 146.6, "t_refresh_s": 0.3, "mem_gb": 17.11, "mem_gb_teacher": 20.25}
|
| 26 |
+
{"step": 24, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3151660050172998, "tokens": 164852, "cumulative_loss_tokens": 4141305, "grad_norm": 1.6875, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 644.0, "dropped_truncated": 0, "gold_loss": 0.801, "gold_lambda": 0.5, "rep_ratio": 2.331, "t_data_s": 0.0, "t_rollout_s": 49.6, "t_step_s": 143.3, "t_refresh_s": 0.3, "mem_gb": 17.59, "mem_gb_teacher": 20.31}
|
| 27 |
+
{"step": 25, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.313829974284698, "tokens": 196304, "cumulative_loss_tokens": 4337609, "grad_norm": 2.234375, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 766.8, "dropped_truncated": 0, "gold_loss": 1.6377, "gold_lambda": 0.5, "rep_ratio": 2.688, "t_data_s": 0.0, "t_rollout_s": 57.5, "t_step_s": 156.7, "t_refresh_s": 0.3, "mem_gb": 17.76, "mem_gb_teacher": 20.32}
|
| 28 |
+
{"step": 26, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27340404820149083, "tokens": 174461, "cumulative_loss_tokens": 4512070, "grad_norm": 1.9375, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 681.5, "dropped_truncated": 0, "gold_loss": 1.3934, "gold_lambda": 0.5, "rep_ratio": 2.549, "t_data_s": 0.0, "t_rollout_s": 52.1, "t_step_s": 146.5, "t_refresh_s": 0.3, "mem_gb": 17.47, "mem_gb_teacher": 20.28}
|
| 29 |
+
{"step": 27, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3913407735489407, "tokens": 181479, "cumulative_loss_tokens": 4693549, "grad_norm": 1.8671875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 708.9, "dropped_truncated": 0, "gold_loss": 1.0072, "gold_lambda": 0.5, "rep_ratio": 2.356, "t_data_s": 0.0, "t_rollout_s": 52.5, "t_step_s": 146.6, "t_refresh_s": 0.3, "mem_gb": 17.33, "mem_gb_teacher": 20.26}
|
| 30 |
+
{"step": 28, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3790620736299349, "tokens": 177593, "cumulative_loss_tokens": 4871142, "grad_norm": 1.6015625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 693.7, "dropped_truncated": 0, "gold_loss": 1.04, "gold_lambda": 0.5, "rep_ratio": 2.369, "t_data_s": 0.0, "t_rollout_s": 51.6, "t_step_s": 147.2, "t_refresh_s": 0.3, "mem_gb": 17.29, "mem_gb_teacher": 20.25}
|
| 31 |
+
{"step": 29, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3367005885269224, "tokens": 169115, "cumulative_loss_tokens": 5040257, "grad_norm": 1.34375, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 660.6, "dropped_truncated": 0, "gold_loss": 0.2224, "gold_lambda": 0.5, "rep_ratio": 2.308, "t_data_s": 0.0, "t_rollout_s": 52.3, "t_step_s": 150.0, "t_refresh_s": 0.3, "mem_gb": 17.5, "mem_gb_teacher": 20.28}
|
| 32 |
+
{"step": 30, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3335959742327791, "tokens": 192932, "cumulative_loss_tokens": 5233189, "grad_norm": 1.765625, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 753.6, "dropped_truncated": 0, "gold_loss": 0.1972, "gold_lambda": 0.5, "rep_ratio": 2.476, "t_data_s": 0.0, "t_rollout_s": 57.4, "t_step_s": 161.6, "t_refresh_s": 0.3, "mem_gb": 17.65, "mem_gb_teacher": 20.3}
|
| 33 |
+
{"step": 31, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3171257981787659, "tokens": 191095, "cumulative_loss_tokens": 5424284, "grad_norm": 1.8046875, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 746.5, "dropped_truncated": 0, "gold_loss": 1.2779, "gold_lambda": 0.5, "rep_ratio": 2.497, "t_data_s": 0.0, "t_rollout_s": 56.9, "t_step_s": 156.9, "t_refresh_s": 0.3, "mem_gb": 17.18, "mem_gb_teacher": 20.26}
|
| 34 |
+
{"step": 32, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3311199295843264, "tokens": 187659, "cumulative_loss_tokens": 5611943, "grad_norm": 1.6171875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 733.0, "dropped_truncated": 0, "gold_loss": 1.0749, "gold_lambda": 0.5, "rep_ratio": 2.448, "t_data_s": 0.0, "t_rollout_s": 56.4, "t_step_s": 161.5, "t_refresh_s": 0.3, "mem_gb": 17.37, "mem_gb_teacher": 20.29}
|
| 35 |
+
{"step": 33, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3653596817669372, "tokens": 162176, "cumulative_loss_tokens": 5774119, "grad_norm": 1.96875, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 633.5, "dropped_truncated": 0, "gold_loss": 1.3587, "gold_lambda": 0.5, "rep_ratio": 2.488, "t_data_s": 0.0, "t_rollout_s": 50.0, "t_step_s": 147.9, "t_refresh_s": 0.3, "mem_gb": 17.77, "mem_gb_teacher": 20.31}
|
| 36 |
+
{"step": 34, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3303874825869901, "tokens": 175637, "cumulative_loss_tokens": 5949756, "grad_norm": 2.125, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 686.1, "dropped_truncated": 0, "gold_loss": 1.6698, "gold_lambda": 0.5, "rep_ratio": 2.444, "t_data_s": 0.0, "t_rollout_s": 53.0, "t_step_s": 150.9, "t_refresh_s": 0.3, "mem_gb": 17.3, "mem_gb_teacher": 20.27}
|
| 37 |
+
{"step": 35, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.338662115701657, "tokens": 164097, "cumulative_loss_tokens": 6113853, "grad_norm": 1.2578125, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 641.0, "dropped_truncated": 0, "gold_loss": 0.1611, "gold_lambda": 0.5, "rep_ratio": 2.39, "t_data_s": 0.0, "t_rollout_s": 48.7, "t_step_s": 140.7, "t_refresh_s": 0.3, "mem_gb": 17.31, "mem_gb_teacher": 20.25}
|
| 38 |
+
{"step": 36, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3760627886982097, "tokens": 180353, "cumulative_loss_tokens": 6294206, "grad_norm": 1.171875, "lr": 1e-05, "finish_rate": 0.949, "comp_len": 704.5, "dropped_truncated": 0, "gold_loss": 0.1814, "gold_lambda": 0.5, "rep_ratio": 2.598, "t_data_s": 0.0, "t_rollout_s": 54.3, "t_step_s": 154.0, "t_refresh_s": 0.3, "mem_gb": 17.34, "mem_gb_teacher": 20.27}
|
| 39 |
+
{"step": 37, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3777555677864465, "tokens": 179099, "cumulative_loss_tokens": 6473305, "grad_norm": 1.734375, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 699.6, "dropped_truncated": 0, "gold_loss": 1.1694, "gold_lambda": 0.5, "rep_ratio": 2.291, "t_data_s": 0.0, "t_rollout_s": 52.4, "t_step_s": 147.9, "t_refresh_s": 0.3, "mem_gb": 17.28, "mem_gb_teacher": 20.25}
|
| 40 |
+
{"step": 38, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2655141159261545, "tokens": 183837, "cumulative_loss_tokens": 6657142, "grad_norm": 1.7890625, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 718.1, "dropped_truncated": 0, "gold_loss": 1.0682, "gold_lambda": 0.5, "rep_ratio": 2.283, "t_data_s": 0.0, "t_rollout_s": 52.4, "t_step_s": 149.0, "t_refresh_s": 0.3, "mem_gb": 16.97, "mem_gb_teacher": 20.22}
|
| 41 |
+
{"step": 39, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26126282006497425, "tokens": 181345, "cumulative_loss_tokens": 6838487, "grad_norm": 1.140625, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 708.4, "dropped_truncated": 0, "gold_loss": 0.1805, "gold_lambda": 0.5, "rep_ratio": 2.463, "t_data_s": 0.0, "t_rollout_s": 53.9, "t_step_s": 152.2, "t_refresh_s": 0.3, "mem_gb": 17.13, "mem_gb_teacher": 20.23}
|
| 42 |
+
{"step": 40, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4208335484820406, "tokens": 166853, "cumulative_loss_tokens": 7005340, "grad_norm": 1.3125, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 651.8, "dropped_truncated": 0, "gold_loss": 0.1644, "gold_lambda": 0.5, "rep_ratio": 2.677, "t_data_s": 0.0, "t_rollout_s": 48.8, "t_step_s": 140.2, "t_refresh_s": 0.3, "mem_gb": 17.35, "mem_gb_teacher": 20.29}
|
| 43 |
+
{"step": 40, "gsm8k_n": 256, "gsm8k_quick_chat": 0.62890625, "t_eval_s": 21.6}
|
| 44 |
+
{"step": 41, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.33376682261101664, "tokens": 161263, "cumulative_loss_tokens": 7166603, "grad_norm": 1.765625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 629.9, "dropped_truncated": 0, "gold_loss": 1.0706, "gold_lambda": 0.5, "rep_ratio": 2.454, "t_data_s": 0.0, "t_rollout_s": 47.2, "t_step_s": 136.0, "t_refresh_s": 0.3, "mem_gb": 17.22, "mem_gb_teacher": 20.26}
|
| 45 |
+
{"step": 42, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34152158943635763, "tokens": 171806, "cumulative_loss_tokens": 7338409, "grad_norm": 1.8125, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 671.1, "dropped_truncated": 0, "gold_loss": 1.0227, "gold_lambda": 0.5, "rep_ratio": 2.453, "t_data_s": 0.0, "t_rollout_s": 50.1, "t_step_s": 141.3, "t_refresh_s": 0.3, "mem_gb": 17.12, "mem_gb_teacher": 20.23}
|
| 46 |
+
{"step": 43, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.30709751060902213, "tokens": 173000, "cumulative_loss_tokens": 7511409, "grad_norm": 1.765625, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 675.8, "dropped_truncated": 0, "gold_loss": 1.2339, "gold_lambda": 0.5, "rep_ratio": 2.397, "t_data_s": 0.0, "t_rollout_s": 51.4, "t_step_s": 144.4, "t_refresh_s": 0.3, "mem_gb": 17.07, "mem_gb_teacher": 20.24}
|
| 47 |
+
{"step": 44, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.38481537799269605, "tokens": 160690, "cumulative_loss_tokens": 7672099, "grad_norm": 2.046875, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 627.7, "dropped_truncated": 0, "gold_loss": 1.3762, "gold_lambda": 0.5, "rep_ratio": 2.414, "t_data_s": 0.0, "t_rollout_s": 47.9, "t_step_s": 138.1, "t_refresh_s": 0.3, "mem_gb": 17.33, "mem_gb_teacher": 20.25}
|
| 48 |
+
{"step": 45, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3474364483801316, "tokens": 163851, "cumulative_loss_tokens": 7835950, "grad_norm": 1.609375, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 640.0, "dropped_truncated": 0, "gold_loss": 1.1411, "gold_lambda": 0.5, "rep_ratio": 2.204, "t_data_s": 0.0, "t_rollout_s": 49.1, "t_step_s": 143.4, "t_refresh_s": 0.3, "mem_gb": 17.42, "mem_gb_teacher": 20.27}
|
| 49 |
+
{"step": 46, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2480004573858833, "tokens": 174070, "cumulative_loss_tokens": 8010020, "grad_norm": 1.796875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 680.0, "dropped_truncated": 0, "gold_loss": 1.1273, "gold_lambda": 0.5, "rep_ratio": 2.398, "t_data_s": 0.0, "t_rollout_s": 50.5, "t_step_s": 144.5, "t_refresh_s": 0.3, "mem_gb": 17.11, "mem_gb_teacher": 20.24}
|
| 50 |
+
{"step": 47, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3238008658347637, "tokens": 175419, "cumulative_loss_tokens": 8185439, "grad_norm": 1.15625, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 685.2, "dropped_truncated": 0, "gold_loss": 0.2272, "gold_lambda": 0.5, "rep_ratio": 2.395, "t_data_s": 0.0, "t_rollout_s": 52.7, "t_step_s": 151.6, "t_refresh_s": 0.3, "mem_gb": 17.06, "mem_gb_teacher": 20.24}
|
| 51 |
+
{"step": 48, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26486295279158406, "tokens": 174191, "cumulative_loss_tokens": 8359630, "grad_norm": 1.0078125, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 680.4, "dropped_truncated": 0, "gold_loss": 0.2168, "gold_lambda": 0.5, "rep_ratio": 2.41, "t_data_s": 0.0, "t_rollout_s": 51.6, "t_step_s": 148.9, "t_refresh_s": 0.3, "mem_gb": 17.0, "mem_gb_teacher": 20.23}
|
| 52 |
+
{"step": 49, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3201779053440231, "tokens": 175865, "cumulative_loss_tokens": 8535495, "grad_norm": 1.84375, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 687.0, "dropped_truncated": 0, "gold_loss": 1.2798, "gold_lambda": 0.5, "rep_ratio": 2.35, "t_data_s": 0.0, "t_rollout_s": 52.2, "t_step_s": 147.4, "t_refresh_s": 0.3, "mem_gb": 17.22, "mem_gb_teacher": 20.26}
|
| 53 |
+
{"step": 50, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3491487042176898, "tokens": 173904, "cumulative_loss_tokens": 8709399, "grad_norm": 1.609375, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 679.3, "dropped_truncated": 0, "gold_loss": 1.1223, "gold_lambda": 0.5, "rep_ratio": 2.19, "t_data_s": 0.0, "t_rollout_s": 52.0, "t_step_s": 146.4, "t_refresh_s": 0.3, "mem_gb": 17.47, "mem_gb_teacher": 20.27}
|
| 54 |
+
{"step": 51, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4068168763832476, "tokens": 149202, "cumulative_loss_tokens": 8858601, "grad_norm": 1.84375, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 582.8, "dropped_truncated": 0, "gold_loss": 0.9763, "gold_lambda": 0.5, "rep_ratio": 2.411, "t_data_s": 0.0, "t_rollout_s": 44.8, "t_step_s": 130.7, "t_refresh_s": 0.3, "mem_gb": 17.24, "mem_gb_teacher": 20.26}
|
| 55 |
+
{"step": 52, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3949036657737358, "tokens": 157801, "cumulative_loss_tokens": 9016402, "grad_norm": 1.5390625, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 616.4, "dropped_truncated": 0, "gold_loss": 0.7635, "gold_lambda": 0.5, "rep_ratio": 2.429, "t_data_s": 0.0, "t_rollout_s": 48.4, "t_step_s": 137.8, "t_refresh_s": 0.3, "mem_gb": 16.94, "mem_gb_teacher": 20.22}
|
| 56 |
+
{"step": 53, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3542952813312953, "tokens": 173066, "cumulative_loss_tokens": 9189468, "grad_norm": 1.375, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 676.0, "dropped_truncated": 0, "gold_loss": 0.888, "gold_lambda": 0.5, "rep_ratio": 2.359, "t_data_s": 0.0, "t_rollout_s": 52.3, "t_step_s": 149.8, "t_refresh_s": 0.3, "mem_gb": 17.61, "mem_gb_teacher": 20.31}
|
| 57 |
+
{"step": 54, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3200471835924996, "tokens": 183743, "cumulative_loss_tokens": 9373211, "grad_norm": 1.6796875, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 717.7, "dropped_truncated": 0, "gold_loss": 1.0623, "gold_lambda": 0.5, "rep_ratio": 2.51, "t_data_s": 0.0, "t_rollout_s": 54.3, "t_step_s": 153.0, "t_refresh_s": 0.3, "mem_gb": 17.3, "mem_gb_teacher": 20.26}
|
| 58 |
+
{"step": 55, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3881143610330633, "tokens": 161627, "cumulative_loss_tokens": 9534838, "grad_norm": 2.515625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 631.4, "dropped_truncated": 0, "gold_loss": 1.6456, "gold_lambda": 0.5, "rep_ratio": 2.152, "t_data_s": 0.0, "t_rollout_s": 50.7, "t_step_s": 143.3, "t_refresh_s": 0.3, "mem_gb": 17.43, "mem_gb_teacher": 20.29}
|
| 59 |
+
{"step": 56, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.37364923851747267, "tokens": 168945, "cumulative_loss_tokens": 9703783, "grad_norm": 1.6953125, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 659.9, "dropped_truncated": 0, "gold_loss": 1.1163, "gold_lambda": 0.5, "rep_ratio": 2.215, "t_data_s": 0.0, "t_rollout_s": 51.2, "t_step_s": 148.9, "t_refresh_s": 0.3, "mem_gb": 17.32, "mem_gb_teacher": 20.25}
|
| 60 |
+
{"step": 57, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4587748102189034, "tokens": 168237, "cumulative_loss_tokens": 9872020, "grad_norm": 1.5390625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 657.2, "dropped_truncated": 0, "gold_loss": 0.9107, "gold_lambda": 0.5, "rep_ratio": 2.531, "t_data_s": 0.0, "t_rollout_s": 49.2, "t_step_s": 144.3, "t_refresh_s": 0.3, "mem_gb": 17.16, "mem_gb_teacher": 20.23}
|
| 61 |
+
{"step": 58, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26425971459508646, "tokens": 184409, "cumulative_loss_tokens": 10056429, "grad_norm": 1.5, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 720.3, "dropped_truncated": 0, "gold_loss": 1.125, "gold_lambda": 0.5, "rep_ratio": 2.684, "t_data_s": 0.0, "t_rollout_s": 54.7, "t_step_s": 153.4, "t_refresh_s": 0.3, "mem_gb": 17.21, "mem_gb_teacher": 20.25}
|
| 62 |
+
{"step": 59, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2719602728308547, "tokens": 189369, "cumulative_loss_tokens": 10245798, "grad_norm": 1.671875, "lr": 1e-05, "finish_rate": 0.945, "comp_len": 739.7, "dropped_truncated": 0, "gold_loss": 0.9619, "gold_lambda": 0.5, "rep_ratio": 2.581, "t_data_s": 0.0, "t_rollout_s": 56.5, "t_step_s": 154.0, "t_refresh_s": 0.3, "mem_gb": 17.25, "mem_gb_teacher": 20.24}
|
| 63 |
+
{"step": 60, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.35498727428496857, "tokens": 166983, "cumulative_loss_tokens": 10412781, "grad_norm": 1.84375, "lr": 1e-05, "finish_rate": 0.953, "comp_len": 652.3, "dropped_truncated": 0, "gold_loss": 1.1915, "gold_lambda": 0.5, "rep_ratio": 2.391, "t_data_s": 0.0, "t_rollout_s": 49.4, "t_step_s": 142.7, "t_refresh_s": 0.3, "mem_gb": 17.2, "mem_gb_teacher": 20.25}
|
| 64 |
+
{"step": 60, "gsm8k_n": 256, "gsm8k_quick_chat": 0.6171875, "t_eval_s": 21.5}
|
| 65 |
+
{"step": 61, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36016367042313613, "tokens": 195255, "cumulative_loss_tokens": 10608036, "grad_norm": 1.1640625, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 762.7, "dropped_truncated": 0, "gold_loss": 0.2184, "gold_lambda": 0.5, "rep_ratio": 2.441, "t_data_s": 0.0, "t_rollout_s": 56.4, "t_step_s": 157.9, "t_refresh_s": 0.3, "mem_gb": 17.14, "mem_gb_teacher": 20.25}
|
| 66 |
+
{"step": 62, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2672880692024946, "tokens": 183249, "cumulative_loss_tokens": 10791285, "grad_norm": 1.0625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 715.8, "dropped_truncated": 0, "gold_loss": 0.2236, "gold_lambda": 0.5, "rep_ratio": 2.47, "t_data_s": 0.0, "t_rollout_s": 54.1, "t_step_s": 151.8, "t_refresh_s": 0.3, "mem_gb": 17.37, "mem_gb_teacher": 20.26}
|
| 67 |
+
{"step": 63, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.272046200843007, "tokens": 169354, "cumulative_loss_tokens": 10960639, "grad_norm": 1.078125, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 661.5, "dropped_truncated": 0, "gold_loss": 0.2677, "gold_lambda": 0.5, "rep_ratio": 2.399, "t_data_s": 0.0, "t_rollout_s": 50.5, "t_step_s": 146.3, "t_refresh_s": 0.3, "mem_gb": 16.99, "mem_gb_teacher": 20.22}
|
| 68 |
+
{"step": 64, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36498770492525573, "tokens": 162379, "cumulative_loss_tokens": 11123018, "grad_norm": 1.3359375, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 634.3, "dropped_truncated": 0, "gold_loss": 0.2643, "gold_lambda": 0.5, "rep_ratio": 2.482, "t_data_s": 0.0, "t_rollout_s": 50.7, "t_step_s": 148.6, "t_refresh_s": 0.3, "mem_gb": 17.24, "mem_gb_teacher": 20.25}
|
| 69 |
+
{"step": 65, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3866863870528368, "tokens": 155402, "cumulative_loss_tokens": 11278420, "grad_norm": 1.5390625, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 607.0, "dropped_truncated": 0, "gold_loss": 0.8323, "gold_lambda": 0.5, "rep_ratio": 2.378, "t_data_s": 0.0, "t_rollout_s": 47.0, "t_step_s": 136.8, "t_refresh_s": 0.3, "mem_gb": 17.26, "mem_gb_teacher": 20.27}
|
| 70 |
+
{"step": 66, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2787221595843288, "tokens": 183694, "cumulative_loss_tokens": 11462114, "grad_norm": 1.875, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 717.6, "dropped_truncated": 0, "gold_loss": 1.1385, "gold_lambda": 0.5, "rep_ratio": 2.538, "t_data_s": 0.0, "t_rollout_s": 53.9, "t_step_s": 152.2, "t_refresh_s": 0.3, "mem_gb": 17.04, "mem_gb_teacher": 20.22}
|
| 71 |
+
{"step": 67, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27035813788552, "tokens": 176040, "cumulative_loss_tokens": 11638154, "grad_norm": 1.71875, "lr": 1e-05, "finish_rate": 0.949, "comp_len": 687.7, "dropped_truncated": 0, "gold_loss": 0.9716, "gold_lambda": 0.5, "rep_ratio": 2.298, "t_data_s": 0.0, "t_rollout_s": 51.3, "t_step_s": 142.2, "t_refresh_s": 0.3, "mem_gb": 17.37, "mem_gb_teacher": 20.28}
|
| 72 |
+
{"step": 68, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.28064265166714214, "tokens": 163139, "cumulative_loss_tokens": 11801293, "grad_norm": 1.53125, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 637.3, "dropped_truncated": 0, "gold_loss": 0.9747, "gold_lambda": 0.5, "rep_ratio": 2.236, "t_data_s": 0.0, "t_rollout_s": 48.4, "t_step_s": 137.7, "t_refresh_s": 0.3, "mem_gb": 17.12, "mem_gb_teacher": 20.22}
|
| 73 |
+
{"step": 69, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26903990990510324, "tokens": 171609, "cumulative_loss_tokens": 11972902, "grad_norm": 2.0625, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 670.3, "dropped_truncated": 0, "gold_loss": 1.3383, "gold_lambda": 0.5, "rep_ratio": 2.405, "t_data_s": 0.0, "t_rollout_s": 51.2, "t_step_s": 145.0, "t_refresh_s": 0.3, "mem_gb": 17.11, "mem_gb_teacher": 20.23}
|
| 74 |
+
{"step": 70, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3467570015864296, "tokens": 158831, "cumulative_loss_tokens": 12131733, "grad_norm": 2.0625, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 620.4, "dropped_truncated": 0, "gold_loss": 1.2858, "gold_lambda": 0.5, "rep_ratio": 2.309, "t_data_s": 0.0, "t_rollout_s": 48.3, "t_step_s": 136.2, "t_refresh_s": 0.3, "mem_gb": 17.13, "mem_gb_teacher": 20.26}
|
| 75 |
+
{"step": 71, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3942991111679687, "tokens": 131738, "cumulative_loss_tokens": 12263471, "grad_norm": 1.671875, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 514.6, "dropped_truncated": 0, "gold_loss": 0.9927, "gold_lambda": 0.5, "rep_ratio": 2.479, "t_data_s": 0.0, "t_rollout_s": 42.0, "t_step_s": 123.0, "t_refresh_s": 0.3, "mem_gb": 17.01, "mem_gb_teacher": 20.22}
|
| 76 |
+
{"step": 72, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2885391192831585, "tokens": 177469, "cumulative_loss_tokens": 12440940, "grad_norm": 1.71875, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 693.2, "dropped_truncated": 0, "gold_loss": 1.2827, "gold_lambda": 0.5, "rep_ratio": 2.388, "t_data_s": 0.0, "t_rollout_s": 52.2, "t_step_s": 148.6, "t_refresh_s": 0.3, "mem_gb": 17.1, "mem_gb_teacher": 20.24}
|
| 77 |
+
{"step": 73, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3441034620278951, "tokens": 153979, "cumulative_loss_tokens": 12594919, "grad_norm": 1.1875, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 601.5, "dropped_truncated": 0, "gold_loss": 0.2327, "gold_lambda": 0.5, "rep_ratio": 2.452, "t_data_s": 0.0, "t_rollout_s": 47.3, "t_step_s": 140.2, "t_refresh_s": 0.3, "mem_gb": 16.98, "mem_gb_teacher": 20.22}
|
| 78 |
+
{"step": 74, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2929858381382192, "tokens": 160418, "cumulative_loss_tokens": 12755337, "grad_norm": 1.0703125, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 626.6, "dropped_truncated": 0, "gold_loss": 0.2401, "gold_lambda": 0.5, "rep_ratio": 2.408, "t_data_s": 0.0, "t_rollout_s": 48.7, "t_step_s": 144.0, "t_refresh_s": 0.3, "mem_gb": 17.21, "mem_gb_teacher": 20.24}
|
| 79 |
+
{"step": 75, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.35599221217074245, "tokens": 160043, "cumulative_loss_tokens": 12915380, "grad_norm": 1.5703125, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 625.2, "dropped_truncated": 0, "gold_loss": 1.0205, "gold_lambda": 0.5, "rep_ratio": 2.571, "t_data_s": 0.0, "t_rollout_s": 47.8, "t_step_s": 136.3, "t_refresh_s": 0.3, "mem_gb": 17.06, "mem_gb_teacher": 20.22}
|
| 80 |
+
{"step": 76, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24107641524497922, "tokens": 196064, "cumulative_loss_tokens": 13111444, "grad_norm": 1.5078125, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 765.9, "dropped_truncated": 0, "gold_loss": 0.945, "gold_lambda": 0.5, "rep_ratio": 2.558, "t_data_s": 0.0, "t_rollout_s": 56.1, "t_step_s": 154.5, "t_refresh_s": 0.3, "mem_gb": 17.26, "mem_gb_teacher": 20.27}
|
| 81 |
+
{"step": 77, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2918642508978692, "tokens": 168701, "cumulative_loss_tokens": 13280145, "grad_norm": 1.484375, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 659.0, "dropped_truncated": 0, "gold_loss": 0.9558, "gold_lambda": 0.5, "rep_ratio": 2.496, "t_data_s": 0.0, "t_rollout_s": 51.7, "t_step_s": 146.1, "t_refresh_s": 0.3, "mem_gb": 17.23, "mem_gb_teacher": 20.24}
|
| 82 |
+
{"step": 78, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3819451671292889, "tokens": 133293, "cumulative_loss_tokens": 13413438, "grad_norm": 1.5625, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 520.7, "dropped_truncated": 0, "gold_loss": 1.0728, "gold_lambda": 0.5, "rep_ratio": 2.506, "t_data_s": 0.0, "t_rollout_s": 43.8, "t_step_s": 128.9, "t_refresh_s": 0.3, "mem_gb": 17.52, "mem_gb_teacher": 20.28}
|
| 83 |
+
{"step": 79, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2689037877672512, "tokens": 178569, "cumulative_loss_tokens": 13592007, "grad_norm": 1.6953125, "lr": 1e-05, "finish_rate": 0.934, "comp_len": 697.5, "dropped_truncated": 0, "gold_loss": 1.2626, "gold_lambda": 0.5, "rep_ratio": 2.574, "t_data_s": 0.0, "t_rollout_s": 53.3, "t_step_s": 150.3, "t_refresh_s": 0.3, "mem_gb": 17.2, "mem_gb_teacher": 20.24}
|
| 84 |
+
{"step": 80, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.31916272058224376, "tokens": 170669, "cumulative_loss_tokens": 13762676, "grad_norm": 1.7109375, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 666.7, "dropped_truncated": 0, "gold_loss": 1.2327, "gold_lambda": 0.5, "rep_ratio": 2.53, "t_data_s": 0.0, "t_rollout_s": 51.5, "t_step_s": 145.4, "t_refresh_s": 0.3, "mem_gb": 17.48, "mem_gb_teacher": 20.28}
|
| 85 |
+
{"step": 80, "gsm8k_n": 256, "gsm8k_quick_chat": 0.59765625, "t_eval_s": 21.4}
|
| 86 |
+
{"step": 81, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2485308842504299, "tokens": 186465, "cumulative_loss_tokens": 13949141, "grad_norm": 1.953125, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 728.4, "dropped_truncated": 0, "gold_loss": 1.1764, "gold_lambda": 0.5, "rep_ratio": 2.547, "t_data_s": 0.0, "t_rollout_s": 55.5, "t_step_s": 152.9, "t_refresh_s": 0.3, "mem_gb": 17.07, "mem_gb_teacher": 20.23}
|
| 87 |
+
{"step": 82, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25967463855320316, "tokens": 190239, "cumulative_loss_tokens": 14139380, "grad_norm": 1.6484375, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 743.1, "dropped_truncated": 0, "gold_loss": 1.3093, "gold_lambda": 0.5, "rep_ratio": 2.526, "t_data_s": 0.0, "t_rollout_s": 57.3, "t_step_s": 158.5, "t_refresh_s": 0.3, "mem_gb": 17.07, "mem_gb_teacher": 20.24}
|
| 88 |
+
{"step": 83, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.33700879973768977, "tokens": 182259, "cumulative_loss_tokens": 14321639, "grad_norm": 1.6875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 711.9, "dropped_truncated": 0, "gold_loss": 1.0788, "gold_lambda": 0.5, "rep_ratio": 2.505, "t_data_s": 0.0, "t_rollout_s": 55.0, "t_step_s": 152.8, "t_refresh_s": 0.3, "mem_gb": 17.47, "mem_gb_teacher": 20.27}
|
| 89 |
+
{"step": 84, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2682194592400478, "tokens": 170072, "cumulative_loss_tokens": 14491711, "grad_norm": 1.3984375, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 664.3, "dropped_truncated": 0, "gold_loss": 0.7509, "gold_lambda": 0.5, "rep_ratio": 2.597, "t_data_s": 0.0, "t_rollout_s": 51.4, "t_step_s": 144.0, "t_refresh_s": 0.3, "mem_gb": 17.44, "mem_gb_teacher": 20.27}
|
| 90 |
+
{"step": 85, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23743917622813884, "tokens": 199223, "cumulative_loss_tokens": 14690934, "grad_norm": 1.8359375, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 778.2, "dropped_truncated": 0, "gold_loss": 0.8562, "gold_lambda": 0.5, "rep_ratio": 2.627, "t_data_s": 0.0, "t_rollout_s": 58.4, "t_step_s": 159.8, "t_refresh_s": 0.3, "mem_gb": 17.48, "mem_gb_teacher": 20.28}
|
| 91 |
+
{"step": 86, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.40081178832628034, "tokens": 179491, "cumulative_loss_tokens": 14870425, "grad_norm": 1.96875, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 701.1, "dropped_truncated": 0, "gold_loss": 0.7619, "gold_lambda": 0.5, "rep_ratio": 2.334, "t_data_s": 0.0, "t_rollout_s": 55.1, "t_step_s": 154.6, "t_refresh_s": 0.3, "mem_gb": 17.68, "mem_gb_teacher": 20.32}
|
| 92 |
+
{"step": 87, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.31119211183407003, "tokens": 161086, "cumulative_loss_tokens": 15031511, "grad_norm": 1.609375, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 629.2, "dropped_truncated": 0, "gold_loss": 1.1431, "gold_lambda": 0.5, "rep_ratio": 2.341, "t_data_s": 0.0, "t_rollout_s": 48.7, "t_step_s": 139.4, "t_refresh_s": 0.3, "mem_gb": 17.28, "mem_gb_teacher": 20.25}
|
| 93 |
+
{"step": 88, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21312830959642717, "tokens": 172945, "cumulative_loss_tokens": 15204456, "grad_norm": 1.53125, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 675.6, "dropped_truncated": 0, "gold_loss": 1.0969, "gold_lambda": 0.5, "rep_ratio": 2.558, "t_data_s": 0.0, "t_rollout_s": 52.2, "t_step_s": 148.0, "t_refresh_s": 0.3, "mem_gb": 17.49, "mem_gb_teacher": 20.28}
|
| 94 |
+
{"step": 89, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.32083614396197874, "tokens": 197781, "cumulative_loss_tokens": 15402237, "grad_norm": 1.796875, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 772.6, "dropped_truncated": 0, "gold_loss": 1.1581, "gold_lambda": 0.5, "rep_ratio": 2.371, "t_data_s": 0.0, "t_rollout_s": 59.1, "t_step_s": 158.0, "t_refresh_s": 0.3, "mem_gb": 17.13, "mem_gb_teacher": 20.24}
|
| 95 |
+
{"step": 90, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3539003647110563, "tokens": 177862, "cumulative_loss_tokens": 15580099, "grad_norm": 1.8046875, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 694.8, "dropped_truncated": 0, "gold_loss": 1.1751, "gold_lambda": 0.5, "rep_ratio": 2.484, "t_data_s": 0.0, "t_rollout_s": 53.6, "t_step_s": 152.0, "t_refresh_s": 0.3, "mem_gb": 17.37, "mem_gb_teacher": 20.27}
|
| 96 |
+
{"step": 91, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3003133533208709, "tokens": 190730, "cumulative_loss_tokens": 15770829, "grad_norm": 1.1328125, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 745.0, "dropped_truncated": 0, "gold_loss": 0.2162, "gold_lambda": 0.5, "rep_ratio": 2.401, "t_data_s": 0.0, "t_rollout_s": 56.0, "t_step_s": 156.9, "t_refresh_s": 0.3, "mem_gb": 17.01, "mem_gb_teacher": 20.25}
|
| 97 |
+
{"step": 92, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3466024366871271, "tokens": 169647, "cumulative_loss_tokens": 15940476, "grad_norm": 1.34375, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 662.7, "dropped_truncated": 0, "gold_loss": 0.2275, "gold_lambda": 0.5, "rep_ratio": 2.384, "t_data_s": 0.0, "t_rollout_s": 50.4, "t_step_s": 144.5, "t_refresh_s": 0.3, "mem_gb": 17.16, "mem_gb_teacher": 20.24}
|
| 98 |
+
{"step": 93, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23514000853877456, "tokens": 167638, "cumulative_loss_tokens": 16108114, "grad_norm": 1.8359375, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 654.8, "dropped_truncated": 0, "gold_loss": 1.6188, "gold_lambda": 0.5, "rep_ratio": 2.248, "t_data_s": 0.0, "t_rollout_s": 50.4, "t_step_s": 145.5, "t_refresh_s": 0.3, "mem_gb": 17.14, "mem_gb_teacher": 20.25}
|
| 99 |
+
{"step": 94, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.31567410076907493, "tokens": 176303, "cumulative_loss_tokens": 16284417, "grad_norm": 1.6484375, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 688.7, "dropped_truncated": 0, "gold_loss": 1.4047, "gold_lambda": 0.5, "rep_ratio": 2.276, "t_data_s": 0.0, "t_rollout_s": 52.5, "t_step_s": 150.1, "t_refresh_s": 0.3, "mem_gb": 17.19, "mem_gb_teacher": 20.23}
|
| 100 |
+
{"step": 95, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3306478749962856, "tokens": 173098, "cumulative_loss_tokens": 16457515, "grad_norm": 1.6796875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 676.2, "dropped_truncated": 0, "gold_loss": 1.3459, "gold_lambda": 0.5, "rep_ratio": 2.403, "t_data_s": 0.0, "t_rollout_s": 51.7, "t_step_s": 145.0, "t_refresh_s": 0.3, "mem_gb": 17.25, "mem_gb_teacher": 20.24}
|
| 101 |
+
{"step": 96, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2879815189713935, "tokens": 176694, "cumulative_loss_tokens": 16634209, "grad_norm": 1.4140625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 690.2, "dropped_truncated": 0, "gold_loss": 1.2006, "gold_lambda": 0.5, "rep_ratio": 2.383, "t_data_s": 0.0, "t_rollout_s": 51.7, "t_step_s": 148.5, "t_refresh_s": 0.3, "mem_gb": 17.54, "mem_gb_teacher": 20.28}
|
| 102 |
+
{"step": 97, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23541706224584372, "tokens": 193571, "cumulative_loss_tokens": 16827780, "grad_norm": 2.34375, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 756.1, "dropped_truncated": 0, "gold_loss": 1.2615, "gold_lambda": 0.5, "rep_ratio": 2.438, "t_data_s": 0.0, "t_rollout_s": 57.9, "t_step_s": 158.2, "t_refresh_s": 0.3, "mem_gb": 17.75, "mem_gb_teacher": 20.32}
|
| 103 |
+
{"step": 98, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34905908556710274, "tokens": 149760, "cumulative_loss_tokens": 16977540, "grad_norm": 1.75, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 585.0, "dropped_truncated": 0, "gold_loss": 1.1712, "gold_lambda": 0.5, "rep_ratio": 2.325, "t_data_s": 0.0, "t_rollout_s": 47.2, "t_step_s": 139.0, "t_refresh_s": 0.3, "mem_gb": 17.11, "mem_gb_teacher": 20.28}
|
| 104 |
+
{"step": 99, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2776076079423005, "tokens": 183255, "cumulative_loss_tokens": 17160795, "grad_norm": 0.9765625, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 715.8, "dropped_truncated": 0, "gold_loss": 0.2535, "gold_lambda": 0.5, "rep_ratio": 2.47, "t_data_s": 0.0, "t_rollout_s": 53.3, "t_step_s": 153.1, "t_refresh_s": 0.3, "mem_gb": 17.33, "mem_gb_teacher": 20.25}
|
| 105 |
+
{"step": 100, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.33610000682790214, "tokens": 155488, "cumulative_loss_tokens": 17316283, "grad_norm": 1.125, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 607.4, "dropped_truncated": 0, "gold_loss": 0.2397, "gold_lambda": 0.5, "rep_ratio": 2.429, "t_data_s": 0.0, "t_rollout_s": 46.8, "t_step_s": 138.1, "t_refresh_s": 0.3, "mem_gb": 17.13, "mem_gb_teacher": 20.23}
|
| 106 |
+
{"step": 100, "gsm8k_n": 256, "gsm8k_quick_chat": 0.6171875, "t_eval_s": 12.9}
|
| 107 |
+
{"step": 101, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2934597294326003, "tokens": 180157, "cumulative_loss_tokens": 17496440, "grad_norm": 1.6328125, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 703.7, "dropped_truncated": 0, "gold_loss": 1.1732, "gold_lambda": 0.5, "rep_ratio": 2.543, "t_data_s": 0.0, "t_rollout_s": 53.7, "t_step_s": 152.7, "t_refresh_s": 0.3, "mem_gb": 17.36, "mem_gb_teacher": 20.26}
|
| 108 |
+
{"step": 102, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.38382341174255885, "tokens": 152575, "cumulative_loss_tokens": 17649015, "grad_norm": 1.8203125, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 596.0, "dropped_truncated": 0, "gold_loss": 1.3037, "gold_lambda": 0.5, "rep_ratio": 2.116, "t_data_s": 0.0, "t_rollout_s": 46.9, "t_step_s": 137.3, "t_refresh_s": 0.3, "mem_gb": 17.14, "mem_gb_teacher": 20.25}
|
| 109 |
+
{"step": 103, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3232517142888692, "tokens": 178507, "cumulative_loss_tokens": 17827522, "grad_norm": 1.109375, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 697.3, "dropped_truncated": 0, "gold_loss": 0.2206, "gold_lambda": 0.5, "rep_ratio": 2.433, "t_data_s": 0.0, "t_rollout_s": 52.6, "t_step_s": 148.0, "t_refresh_s": 0.3, "mem_gb": 17.63, "mem_gb_teacher": 20.3}
|
| 110 |
+
{"step": 104, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.258444819787859, "tokens": 163920, "cumulative_loss_tokens": 17991442, "grad_norm": 1.0, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 640.3, "dropped_truncated": 0, "gold_loss": 0.2333, "gold_lambda": 0.5, "rep_ratio": 2.313, "t_data_s": 0.0, "t_rollout_s": 49.4, "t_step_s": 142.5, "t_refresh_s": 0.3, "mem_gb": 17.27, "mem_gb_teacher": 20.27}
|
| 111 |
+
{"step": 105, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3201432822026658, "tokens": 134492, "cumulative_loss_tokens": 18125934, "grad_norm": 1.4921875, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 525.4, "dropped_truncated": 0, "gold_loss": 0.8977, "gold_lambda": 0.5, "rep_ratio": 2.268, "t_data_s": 0.0, "t_rollout_s": 43.1, "t_step_s": 129.5, "t_refresh_s": 0.3, "mem_gb": 17.43, "mem_gb_teacher": 20.27}
|
| 112 |
+
{"step": 106, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.38415758914838366, "tokens": 173745, "cumulative_loss_tokens": 18299679, "grad_norm": 1.703125, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 678.7, "dropped_truncated": 0, "gold_loss": 1.2116, "gold_lambda": 0.5, "rep_ratio": 2.414, "t_data_s": 0.0, "t_rollout_s": 53.3, "t_step_s": 151.0, "t_refresh_s": 0.3, "mem_gb": 17.51, "mem_gb_teacher": 20.3}
|
| 113 |
+
{"step": 107, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27767968833353346, "tokens": 150360, "cumulative_loss_tokens": 18450039, "grad_norm": 1.625, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 587.3, "dropped_truncated": 0, "gold_loss": 1.1874, "gold_lambda": 0.5, "rep_ratio": 2.226, "t_data_s": 0.0, "t_rollout_s": 46.6, "t_step_s": 135.4, "t_refresh_s": 0.3, "mem_gb": 17.1, "mem_gb_teacher": 20.25}
|
| 114 |
+
{"step": 108, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2810955602886575, "tokens": 212108, "cumulative_loss_tokens": 18662147, "grad_norm": 1.890625, "lr": 1e-05, "finish_rate": 0.953, "comp_len": 828.5, "dropped_truncated": 0, "gold_loss": 1.2019, "gold_lambda": 0.5, "rep_ratio": 2.377, "t_data_s": 0.0, "t_rollout_s": 63.5, "t_step_s": 167.3, "t_refresh_s": 0.3, "mem_gb": 17.55, "mem_gb_teacher": 20.3}
|
| 115 |
+
{"step": 109, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3388609603948928, "tokens": 163795, "cumulative_loss_tokens": 18825942, "grad_norm": 1.515625, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 639.8, "dropped_truncated": 0, "gold_loss": 0.8848, "gold_lambda": 0.5, "rep_ratio": 2.577, "t_data_s": 0.0, "t_rollout_s": 49.4, "t_step_s": 142.1, "t_refresh_s": 0.3, "mem_gb": 17.91, "mem_gb_teacher": 20.35}
|
| 116 |
+
{"step": 110, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.354563303726803, "tokens": 163003, "cumulative_loss_tokens": 18988945, "grad_norm": 1.4765625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 636.7, "dropped_truncated": 0, "gold_loss": 0.99, "gold_lambda": 0.5, "rep_ratio": 2.477, "t_data_s": 0.0, "t_rollout_s": 48.9, "t_step_s": 142.6, "t_refresh_s": 0.3, "mem_gb": 17.17, "mem_gb_teacher": 20.24}
|
| 117 |
+
{"step": 111, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3537407948279594, "tokens": 195368, "cumulative_loss_tokens": 19184313, "grad_norm": 1.6015625, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 763.2, "dropped_truncated": 0, "gold_loss": 1.1307, "gold_lambda": 0.5, "rep_ratio": 2.286, "t_data_s": 0.0, "t_rollout_s": 57.3, "t_step_s": 157.0, "t_refresh_s": 0.3, "mem_gb": 17.21, "mem_gb_teacher": 20.25}
|
| 118 |
+
{"step": 112, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2772862368117265, "tokens": 204542, "cumulative_loss_tokens": 19388855, "grad_norm": 2.03125, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 799.0, "dropped_truncated": 0, "gold_loss": 1.339, "gold_lambda": 0.5, "rep_ratio": 2.278, "t_data_s": 0.0, "t_rollout_s": 59.7, "t_step_s": 161.2, "t_refresh_s": 0.3, "mem_gb": 17.17, "mem_gb_teacher": 20.26}
|
| 119 |
+
{"step": 113, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.310700205462893, "tokens": 153520, "cumulative_loss_tokens": 19542375, "grad_norm": 1.515625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 599.7, "dropped_truncated": 0, "gold_loss": 0.8762, "gold_lambda": 0.5, "rep_ratio": 2.643, "t_data_s": 0.0, "t_rollout_s": 46.8, "t_step_s": 137.2, "t_refresh_s": 0.3, "mem_gb": 16.91, "mem_gb_teacher": 20.22}
|
| 120 |
+
{"step": 114, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.41260987902655, "tokens": 157613, "cumulative_loss_tokens": 19699988, "grad_norm": 1.4453125, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 615.7, "dropped_truncated": 0, "gold_loss": 0.8989, "gold_lambda": 0.5, "rep_ratio": 2.193, "t_data_s": 0.0, "t_rollout_s": 47.3, "t_step_s": 133.6, "t_refresh_s": 0.3, "mem_gb": 17.36, "mem_gb_teacher": 20.26}
|
| 121 |
+
{"step": 115, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4028610580933015, "tokens": 147120, "cumulative_loss_tokens": 19847108, "grad_norm": 1.28125, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 574.7, "dropped_truncated": 0, "gold_loss": 0.1728, "gold_lambda": 0.5, "rep_ratio": 2.412, "t_data_s": 0.0, "t_rollout_s": 46.1, "t_step_s": 130.4, "t_refresh_s": 0.3, "mem_gb": 16.95, "mem_gb_teacher": 20.22}
|
| 122 |
+
{"step": 116, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34066545782229457, "tokens": 173292, "cumulative_loss_tokens": 20020400, "grad_norm": 1.0625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 676.9, "dropped_truncated": 0, "gold_loss": 0.1759, "gold_lambda": 0.5, "rep_ratio": 2.363, "t_data_s": 0.0, "t_rollout_s": 51.2, "t_step_s": 144.9, "t_refresh_s": 0.3, "mem_gb": 17.43, "mem_gb_teacher": 20.28}
|
| 123 |
+
{"step": 117, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3317595209318613, "tokens": 152431, "cumulative_loss_tokens": 20172831, "grad_norm": 1.46875, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 595.4, "dropped_truncated": 0, "gold_loss": 1.0952, "gold_lambda": 0.5, "rep_ratio": 2.472, "t_data_s": 0.0, "t_rollout_s": 47.2, "t_step_s": 134.1, "t_refresh_s": 0.3, "mem_gb": 17.4, "mem_gb_teacher": 20.27}
|
| 124 |
+
{"step": 118, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.30969671645999375, "tokens": 141862, "cumulative_loss_tokens": 20314693, "grad_norm": 1.734375, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 554.1, "dropped_truncated": 0, "gold_loss": 1.3548, "gold_lambda": 0.5, "rep_ratio": 2.267, "t_data_s": 0.0, "t_rollout_s": 44.3, "t_step_s": 131.8, "t_refresh_s": 0.3, "mem_gb": 17.6, "mem_gb_teacher": 20.29}
|
| 125 |
+
{"step": 119, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24082254176214968, "tokens": 194590, "cumulative_loss_tokens": 20509283, "grad_norm": 1.4921875, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 760.1, "dropped_truncated": 0, "gold_loss": 1.0994, "gold_lambda": 0.5, "rep_ratio": 2.633, "t_data_s": 0.0, "t_rollout_s": 56.9, "t_step_s": 154.8, "t_refresh_s": 0.3, "mem_gb": 17.21, "mem_gb_teacher": 20.24}
|
| 126 |
+
{"step": 120, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3365587571631956, "tokens": 160555, "cumulative_loss_tokens": 20669838, "grad_norm": 1.5, "lr": 1e-05, "finish_rate": 1.0, "comp_len": 627.2, "dropped_truncated": 0, "gold_loss": 1.0218, "gold_lambda": 0.5, "rep_ratio": 2.389, "t_data_s": 0.0, "t_rollout_s": 46.6, "t_step_s": 126.9, "t_refresh_s": 0.0, "mem_gb": 16.8, "mem_gb_teacher": 20.2}
|
| 127 |
+
{"step": 120, "gsm8k_n": 256, "gsm8k_quick_chat": 0.625, "t_eval_s": 31.8}
|
| 128 |
+
{"step": 121, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.39473937900511763, "tokens": 161799, "cumulative_loss_tokens": 20831637, "grad_norm": 1.3515625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 632.0, "dropped_truncated": 0, "gold_loss": 0.2047, "gold_lambda": 0.5, "rep_ratio": 2.446, "t_data_s": 0.0, "t_rollout_s": 48.2, "t_step_s": 156.0, "t_refresh_s": 0.3, "mem_gb": 16.97, "mem_gb_teacher": 20.24}
|
| 129 |
+
{"step": 122, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.30440131709249874, "tokens": 166617, "cumulative_loss_tokens": 20998254, "grad_norm": 1.0390625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 650.8, "dropped_truncated": 0, "gold_loss": 0.2003, "gold_lambda": 0.5, "rep_ratio": 2.285, "t_data_s": 0.0, "t_rollout_s": 50.6, "t_step_s": 145.0, "t_refresh_s": 0.3, "mem_gb": 17.63, "mem_gb_teacher": 20.32}
|
| 130 |
+
{"step": 123, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3542209245783803, "tokens": 176166, "cumulative_loss_tokens": 21174420, "grad_norm": 1.1875, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 688.1, "dropped_truncated": 0, "gold_loss": 0.229, "gold_lambda": 0.5, "rep_ratio": 2.173, "t_data_s": 0.0, "t_rollout_s": 52.9, "t_step_s": 151.4, "t_refresh_s": 0.3, "mem_gb": 17.19, "mem_gb_teacher": 20.24}
|
| 131 |
+
{"step": 124, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3309741401607386, "tokens": 175597, "cumulative_loss_tokens": 21350017, "grad_norm": 1.09375, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 685.9, "dropped_truncated": 0, "gold_loss": 0.2282, "gold_lambda": 0.5, "rep_ratio": 2.603, "t_data_s": 0.0, "t_rollout_s": 51.7, "t_step_s": 147.2, "t_refresh_s": 0.3, "mem_gb": 17.39, "mem_gb_teacher": 20.26}
|
| 132 |
+
{"step": 125, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3333785288495929, "tokens": 171066, "cumulative_loss_tokens": 21521083, "grad_norm": 1.5, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 668.2, "dropped_truncated": 0, "gold_loss": 1.1147, "gold_lambda": 0.5, "rep_ratio": 2.306, "t_data_s": 0.0, "t_rollout_s": 52.9, "t_step_s": 150.2, "t_refresh_s": 0.3, "mem_gb": 17.62, "mem_gb_teacher": 20.32}
|
| 133 |
+
{"step": 126, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3157572144692479, "tokens": 169961, "cumulative_loss_tokens": 21691044, "grad_norm": 1.40625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 663.9, "dropped_truncated": 0, "gold_loss": 0.8063, "gold_lambda": 0.5, "rep_ratio": 2.355, "t_data_s": 0.0, "t_rollout_s": 50.7, "t_step_s": 145.7, "t_refresh_s": 0.3, "mem_gb": 17.37, "mem_gb_teacher": 20.28}
|
| 134 |
+
{"step": 127, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27284040474848087, "tokens": 140128, "cumulative_loss_tokens": 21831172, "grad_norm": 1.46875, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 547.4, "dropped_truncated": 0, "gold_loss": 1.1113, "gold_lambda": 0.5, "rep_ratio": 2.248, "t_data_s": 0.0, "t_rollout_s": 44.4, "t_step_s": 130.9, "t_refresh_s": 0.3, "mem_gb": 17.21, "mem_gb_teacher": 20.25}
|
| 135 |
+
{"step": 128, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36759839778598735, "tokens": 165466, "cumulative_loss_tokens": 21996638, "grad_norm": 1.4453125, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 646.4, "dropped_truncated": 0, "gold_loss": 1.1112, "gold_lambda": 0.5, "rep_ratio": 2.328, "t_data_s": 0.0, "t_rollout_s": 47.9, "t_step_s": 140.0, "t_refresh_s": 0.3, "mem_gb": 17.27, "mem_gb_teacher": 20.27}
|
| 136 |
+
{"step": 129, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3560167171387749, "tokens": 147422, "cumulative_loss_tokens": 22144060, "grad_norm": 1.2109375, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 575.9, "dropped_truncated": 0, "gold_loss": 0.2043, "gold_lambda": 0.5, "rep_ratio": 2.391, "t_data_s": 0.0, "t_rollout_s": 46.3, "t_step_s": 134.0, "t_refresh_s": 0.3, "mem_gb": 16.94, "mem_gb_teacher": 20.22}
|
| 137 |
+
{"step": 130, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.37849711662147584, "tokens": 176001, "cumulative_loss_tokens": 22320061, "grad_norm": 1.0546875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 687.5, "dropped_truncated": 0, "gold_loss": 0.1924, "gold_lambda": 0.5, "rep_ratio": 2.274, "t_data_s": 0.0, "t_rollout_s": 51.1, "t_step_s": 145.5, "t_refresh_s": 0.3, "mem_gb": 17.15, "mem_gb_teacher": 20.26}
|
| 138 |
+
{"step": 131, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.5230161204257552, "tokens": 122077, "cumulative_loss_tokens": 22442138, "grad_norm": 1.9375, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 476.9, "dropped_truncated": 0, "gold_loss": 1.188, "gold_lambda": 0.5, "rep_ratio": 2.262, "t_data_s": 0.0, "t_rollout_s": 41.4, "t_step_s": 126.2, "t_refresh_s": 0.3, "mem_gb": 17.01, "mem_gb_teacher": 20.23}
|
| 139 |
+
{"step": 132, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2914811503083664, "tokens": 153907, "cumulative_loss_tokens": 22596045, "grad_norm": 1.671875, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 601.2, "dropped_truncated": 0, "gold_loss": 1.1376, "gold_lambda": 0.5, "rep_ratio": 2.206, "t_data_s": 0.0, "t_rollout_s": 47.6, "t_step_s": 140.4, "t_refresh_s": 0.3, "mem_gb": 17.67, "mem_gb_teacher": 20.31}
|
| 140 |
+
{"step": 133, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3968975455559935, "tokens": 160860, "cumulative_loss_tokens": 22756905, "grad_norm": 1.953125, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 628.4, "dropped_truncated": 0, "gold_loss": 1.1765, "gold_lambda": 0.5, "rep_ratio": 2.271, "t_data_s": 0.0, "t_rollout_s": 49.6, "t_step_s": 142.4, "t_refresh_s": 0.3, "mem_gb": 18.04, "mem_gb_teacher": 20.34}
|
| 141 |
+
{"step": 134, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34955939524991836, "tokens": 160464, "cumulative_loss_tokens": 22917369, "grad_norm": 1.65625, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 626.8, "dropped_truncated": 0, "gold_loss": 1.1657, "gold_lambda": 0.5, "rep_ratio": 2.334, "t_data_s": 0.0, "t_rollout_s": 49.5, "t_step_s": 140.7, "t_refresh_s": 0.3, "mem_gb": 17.31, "mem_gb_teacher": 20.27}
|
| 142 |
+
{"step": 135, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3188193332602094, "tokens": 147417, "cumulative_loss_tokens": 23064786, "grad_norm": 1.0703125, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 575.8, "dropped_truncated": 0, "gold_loss": 0.2166, "gold_lambda": 0.5, "rep_ratio": 2.564, "t_data_s": 0.0, "t_rollout_s": 45.8, "t_step_s": 135.5, "t_refresh_s": 0.3, "mem_gb": 16.98, "mem_gb_teacher": 20.23}
|
| 143 |
+
{"step": 136, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.35583786295102454, "tokens": 165023, "cumulative_loss_tokens": 23229809, "grad_norm": 1.0390625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 644.6, "dropped_truncated": 0, "gold_loss": 0.2156, "gold_lambda": 0.5, "rep_ratio": 2.309, "t_data_s": 0.0, "t_rollout_s": 49.6, "t_step_s": 145.1, "t_refresh_s": 0.3, "mem_gb": 17.11, "mem_gb_teacher": 20.24}
|
| 144 |
+
{"step": 137, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4206642100727087, "tokens": 171534, "cumulative_loss_tokens": 23401343, "grad_norm": 1.4609375, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 670.1, "dropped_truncated": 0, "gold_loss": 0.9229, "gold_lambda": 0.5, "rep_ratio": 2.577, "t_data_s": 0.0, "t_rollout_s": 52.2, "t_step_s": 148.4, "t_refresh_s": 0.3, "mem_gb": 17.56, "mem_gb_teacher": 20.3}
|
| 145 |
+
{"step": 138, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36283362583526685, "tokens": 165703, "cumulative_loss_tokens": 23567046, "grad_norm": 1.3984375, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 647.3, "dropped_truncated": 0, "gold_loss": 0.9425, "gold_lambda": 0.5, "rep_ratio": 2.474, "t_data_s": 0.0, "t_rollout_s": 48.2, "t_step_s": 139.7, "t_refresh_s": 0.3, "mem_gb": 17.31, "mem_gb_teacher": 20.25}
|
| 146 |
+
{"step": 139, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36461856598141684, "tokens": 153711, "cumulative_loss_tokens": 23720757, "grad_norm": 1.6953125, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 600.4, "dropped_truncated": 0, "gold_loss": 1.1359, "gold_lambda": 0.5, "rep_ratio": 2.415, "t_data_s": 0.0, "t_rollout_s": 48.5, "t_step_s": 140.9, "t_refresh_s": 0.3, "mem_gb": 17.39, "mem_gb_teacher": 20.26}
|
| 147 |
+
{"step": 140, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.334692141021399, "tokens": 168255, "cumulative_loss_tokens": 23889012, "grad_norm": 1.65625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 657.2, "dropped_truncated": 0, "gold_loss": 1.213, "gold_lambda": 0.5, "rep_ratio": 2.454, "t_data_s": 0.0, "t_rollout_s": 50.5, "t_step_s": 144.3, "t_refresh_s": 0.3, "mem_gb": 17.8, "mem_gb_teacher": 20.32}
|
| 148 |
+
{"step": 140, "gsm8k_n": 256, "gsm8k_quick_chat": 0.60546875, "t_eval_s": 23.7}
|
| 149 |
+
{"step": 141, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.31522672475747, "tokens": 176793, "cumulative_loss_tokens": 24065805, "grad_norm": 1.09375, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 690.6, "dropped_truncated": 0, "gold_loss": 0.1946, "gold_lambda": 0.5, "rep_ratio": 2.322, "t_data_s": 0.0, "t_rollout_s": 51.9, "t_step_s": 148.5, "t_refresh_s": 0.3, "mem_gb": 17.05, "mem_gb_teacher": 20.24}
|
| 150 |
+
{"step": 142, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3055049458263928, "tokens": 167092, "cumulative_loss_tokens": 24232897, "grad_norm": 0.94921875, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 652.7, "dropped_truncated": 0, "gold_loss": 0.1903, "gold_lambda": 0.5, "rep_ratio": 2.48, "t_data_s": 0.0, "t_rollout_s": 50.6, "t_step_s": 145.5, "t_refresh_s": 0.3, "mem_gb": 17.37, "mem_gb_teacher": 20.26}
|
| 151 |
+
{"step": 143, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.41086978236292004, "tokens": 138916, "cumulative_loss_tokens": 24371813, "grad_norm": 1.40625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 542.6, "dropped_truncated": 0, "gold_loss": 0.6694, "gold_lambda": 0.5, "rep_ratio": 2.348, "t_data_s": 0.0, "t_rollout_s": 44.3, "t_step_s": 132.8, "t_refresh_s": 0.3, "mem_gb": 17.48, "mem_gb_teacher": 20.28}
|
| 152 |
+
{"step": 144, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.31509188192000326, "tokens": 158022, "cumulative_loss_tokens": 24529835, "grad_norm": 1.359375, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 617.3, "dropped_truncated": 0, "gold_loss": 0.6787, "gold_lambda": 0.5, "rep_ratio": 2.632, "t_data_s": 0.0, "t_rollout_s": 48.5, "t_step_s": 137.8, "t_refresh_s": 0.3, "mem_gb": 17.13, "mem_gb_teacher": 20.25}
|
| 153 |
+
{"step": 145, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36162450281834124, "tokens": 180233, "cumulative_loss_tokens": 24710068, "grad_norm": 1.6484375, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 704.0, "dropped_truncated": 0, "gold_loss": 1.4542, "gold_lambda": 0.5, "rep_ratio": 3.537, "t_data_s": 0.0, "t_rollout_s": 52.9, "t_step_s": 148.3, "t_refresh_s": 0.3, "mem_gb": 17.84, "mem_gb_teacher": 20.33}
|
| 154 |
+
{"step": 146, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.30904037311862825, "tokens": 174184, "cumulative_loss_tokens": 24884252, "grad_norm": 1.578125, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 680.4, "dropped_truncated": 0, "gold_loss": 1.2217, "gold_lambda": 0.5, "rep_ratio": 2.481, "t_data_s": 0.0, "t_rollout_s": 51.2, "t_step_s": 144.9, "t_refresh_s": 0.3, "mem_gb": 17.53, "mem_gb_teacher": 20.3}
|
| 155 |
+
{"step": 147, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3328264856027079, "tokens": 161901, "cumulative_loss_tokens": 25046153, "grad_norm": 1.2890625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 632.4, "dropped_truncated": 0, "gold_loss": 0.8878, "gold_lambda": 0.5, "rep_ratio": 2.361, "t_data_s": 0.0, "t_rollout_s": 47.7, "t_step_s": 136.8, "t_refresh_s": 0.3, "mem_gb": 17.29, "mem_gb_teacher": 20.25}
|
| 156 |
+
{"step": 148, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3381848439622244, "tokens": 172441, "cumulative_loss_tokens": 25218594, "grad_norm": 1.3828125, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 673.6, "dropped_truncated": 0, "gold_loss": 0.9029, "gold_lambda": 0.5, "rep_ratio": 2.238, "t_data_s": 0.0, "t_rollout_s": 52.1, "t_step_s": 145.1, "t_refresh_s": 0.3, "mem_gb": 17.09, "mem_gb_teacher": 20.23}
|
| 157 |
+
{"step": 149, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25333612187819804, "tokens": 168625, "cumulative_loss_tokens": 25387219, "grad_norm": 1.0, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 658.7, "dropped_truncated": 0, "gold_loss": 0.2142, "gold_lambda": 0.5, "rep_ratio": 2.441, "t_data_s": 0.0, "t_rollout_s": 50.6, "t_step_s": 145.4, "t_refresh_s": 0.3, "mem_gb": 17.38, "mem_gb_teacher": 20.26}
|
| 158 |
+
{"step": 150, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4731657869966873, "tokens": 145284, "cumulative_loss_tokens": 25532503, "grad_norm": 1.5390625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 567.5, "dropped_truncated": 0, "gold_loss": 0.1908, "gold_lambda": 0.5, "rep_ratio": 2.201, "t_data_s": 0.0, "t_rollout_s": 44.7, "t_step_s": 135.1, "t_refresh_s": 0.3, "mem_gb": 17.52, "mem_gb_teacher": 20.28}
|
| 159 |
+
{"step": 151, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3933305354915603, "tokens": 156446, "cumulative_loss_tokens": 25688949, "grad_norm": 1.65625, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 611.1, "dropped_truncated": 0, "gold_loss": 1.1245, "gold_lambda": 0.5, "rep_ratio": 2.291, "t_data_s": 0.0, "t_rollout_s": 47.9, "t_step_s": 138.4, "t_refresh_s": 0.3, "mem_gb": 17.65, "mem_gb_teacher": 20.29}
|
| 160 |
+
{"step": 152, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.37683931713473384, "tokens": 180670, "cumulative_loss_tokens": 25869619, "grad_norm": 1.3671875, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 705.7, "dropped_truncated": 0, "gold_loss": 0.9474, "gold_lambda": 0.5, "rep_ratio": 2.159, "t_data_s": 0.0, "t_rollout_s": 53.7, "t_step_s": 151.6, "t_refresh_s": 0.3, "mem_gb": 17.06, "mem_gb_teacher": 20.27}
|
| 161 |
+
{"step": 153, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.38866299738107646, "tokens": 184874, "cumulative_loss_tokens": 26054493, "grad_norm": 1.5703125, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 722.2, "dropped_truncated": 0, "gold_loss": 1.2021, "gold_lambda": 0.5, "rep_ratio": 2.233, "t_data_s": 0.0, "t_rollout_s": 54.7, "t_step_s": 155.1, "t_refresh_s": 0.3, "mem_gb": 17.54, "mem_gb_teacher": 20.29}
|
| 162 |
+
{"step": 154, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3472696636471152, "tokens": 148352, "cumulative_loss_tokens": 26202845, "grad_norm": 1.828125, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 579.5, "dropped_truncated": 0, "gold_loss": 1.4754, "gold_lambda": 0.5, "rep_ratio": 2.406, "t_data_s": 0.0, "t_rollout_s": 46.9, "t_step_s": 136.7, "t_refresh_s": 0.3, "mem_gb": 18.03, "mem_gb_teacher": 20.36}
|
| 163 |
+
{"step": 155, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4225933315944912, "tokens": 165462, "cumulative_loss_tokens": 26368307, "grad_norm": 1.1484375, "lr": 1e-05, "finish_rate": 0.953, "comp_len": 646.3, "dropped_truncated": 0, "gold_loss": 0.1532, "gold_lambda": 0.5, "rep_ratio": 2.439, "t_data_s": 0.0, "t_rollout_s": 48.8, "t_step_s": 140.6, "t_refresh_s": 0.3, "mem_gb": 17.12, "mem_gb_teacher": 20.23}
|
| 164 |
+
{"step": 156, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.376902294824839, "tokens": 179733, "cumulative_loss_tokens": 26548040, "grad_norm": 1.1640625, "lr": 1e-05, "finish_rate": 0.949, "comp_len": 702.1, "dropped_truncated": 0, "gold_loss": 0.1693, "gold_lambda": 0.5, "rep_ratio": 2.239, "t_data_s": 0.0, "t_rollout_s": 53.0, "t_step_s": 148.4, "t_refresh_s": 0.3, "mem_gb": 17.25, "mem_gb_teacher": 20.26}
|
| 165 |
+
{"step": 157, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4335031142711173, "tokens": 173783, "cumulative_loss_tokens": 26721823, "grad_norm": 1.5703125, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 678.8, "dropped_truncated": 0, "gold_loss": 1.0377, "gold_lambda": 0.5, "rep_ratio": 2.156, "t_data_s": 0.0, "t_rollout_s": 52.7, "t_step_s": 151.0, "t_refresh_s": 0.3, "mem_gb": 16.95, "mem_gb_teacher": 20.22}
|
| 166 |
+
{"step": 158, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3546720364810063, "tokens": 153490, "cumulative_loss_tokens": 26875313, "grad_norm": 1.4140625, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 599.6, "dropped_truncated": 0, "gold_loss": 0.9327, "gold_lambda": 0.5, "rep_ratio": 2.211, "t_data_s": 0.0, "t_rollout_s": 46.5, "t_step_s": 133.3, "t_refresh_s": 0.3, "mem_gb": 16.95, "mem_gb_teacher": 20.22}
|
| 167 |
+
{"step": 159, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34111932526074734, "tokens": 155177, "cumulative_loss_tokens": 27030490, "grad_norm": 0.9921875, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 606.2, "dropped_truncated": 0, "gold_loss": 0.1699, "gold_lambda": 0.5, "rep_ratio": 2.439, "t_data_s": 0.0, "t_rollout_s": 46.9, "t_step_s": 140.1, "t_refresh_s": 0.3, "mem_gb": 17.28, "mem_gb_teacher": 20.25}
|
| 168 |
+
{"step": 160, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34542942264660215, "tokens": 158378, "cumulative_loss_tokens": 27188868, "grad_norm": 0.9765625, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 618.7, "dropped_truncated": 0, "gold_loss": 0.1558, "gold_lambda": 0.5, "rep_ratio": 2.444, "t_data_s": 0.0, "t_rollout_s": 48.0, "t_step_s": 137.0, "t_refresh_s": 0.3, "mem_gb": 17.06, "mem_gb_teacher": 20.24}
|
| 169 |
+
{"step": 160, "gsm8k_n": 256, "gsm8k_quick_chat": 0.60546875, "t_eval_s": 21.3}
|
| 170 |
+
{"step": 161, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3408296214623561, "tokens": 164277, "cumulative_loss_tokens": 27353145, "grad_norm": 1.5078125, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 641.7, "dropped_truncated": 0, "gold_loss": 0.9431, "gold_lambda": 0.5, "rep_ratio": 2.547, "t_data_s": 0.0, "t_rollout_s": 48.5, "t_step_s": 139.0, "t_refresh_s": 0.3, "mem_gb": 17.32, "mem_gb_teacher": 20.25}
|
| 171 |
+
{"step": 162, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2568380696480312, "tokens": 202087, "cumulative_loss_tokens": 27555232, "grad_norm": 1.4296875, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 789.4, "dropped_truncated": 0, "gold_loss": 0.8863, "gold_lambda": 0.5, "rep_ratio": 5.6, "t_data_s": 0.0, "t_rollout_s": 60.8, "t_step_s": 162.7, "t_refresh_s": 0.3, "mem_gb": 17.49, "mem_gb_teacher": 20.31}
|
| 172 |
+
{"step": 163, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.41024620194135997, "tokens": 155952, "cumulative_loss_tokens": 27711184, "grad_norm": 1.5390625, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 609.2, "dropped_truncated": 0, "gold_loss": 1.0859, "gold_lambda": 0.5, "rep_ratio": 2.566, "t_data_s": 0.0, "t_rollout_s": 48.3, "t_step_s": 134.9, "t_refresh_s": 0.3, "mem_gb": 16.93, "mem_gb_teacher": 20.22}
|
| 173 |
+
{"step": 164, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.30812923730058833, "tokens": 152153, "cumulative_loss_tokens": 27863337, "grad_norm": 1.703125, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 594.3, "dropped_truncated": 0, "gold_loss": 1.2045, "gold_lambda": 0.5, "rep_ratio": 2.509, "t_data_s": 0.0, "t_rollout_s": 46.6, "t_step_s": 135.8, "t_refresh_s": 0.3, "mem_gb": 17.4, "mem_gb_teacher": 20.28}
|
| 174 |
+
{"step": 165, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3024375501106136, "tokens": 181882, "cumulative_loss_tokens": 28045219, "grad_norm": 1.34375, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 710.5, "dropped_truncated": 0, "gold_loss": 1.0263, "gold_lambda": 0.5, "rep_ratio": 2.197, "t_data_s": 0.0, "t_rollout_s": 53.8, "t_step_s": 149.8, "t_refresh_s": 0.3, "mem_gb": 17.2, "mem_gb_teacher": 20.24}
|
| 175 |
+
{"step": 166, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3207656219341267, "tokens": 176203, "cumulative_loss_tokens": 28221422, "grad_norm": 1.4921875, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 688.3, "dropped_truncated": 0, "gold_loss": 1.0071, "gold_lambda": 0.5, "rep_ratio": 2.647, "t_data_s": 0.0, "t_rollout_s": 52.6, "t_step_s": 146.9, "t_refresh_s": 0.3, "mem_gb": 17.77, "mem_gb_teacher": 20.32}
|
| 176 |
+
{"step": 167, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2939028003884097, "tokens": 182128, "cumulative_loss_tokens": 28403550, "grad_norm": 1.0859375, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 711.4, "dropped_truncated": 0, "gold_loss": 0.2184, "gold_lambda": 0.5, "rep_ratio": 2.332, "t_data_s": 0.0, "t_rollout_s": 55.4, "t_step_s": 156.6, "t_refresh_s": 0.3, "mem_gb": 17.47, "mem_gb_teacher": 20.27}
|
| 177 |
+
{"step": 168, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3914614871089253, "tokens": 160405, "cumulative_loss_tokens": 28563955, "grad_norm": 1.1015625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 626.6, "dropped_truncated": 0, "gold_loss": 0.2073, "gold_lambda": 0.5, "rep_ratio": 2.731, "t_data_s": 0.0, "t_rollout_s": 48.8, "t_step_s": 143.4, "t_refresh_s": 0.3, "mem_gb": 17.21, "mem_gb_teacher": 20.25}
|
| 178 |
+
{"step": 169, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.42604499969692455, "tokens": 141392, "cumulative_loss_tokens": 28705347, "grad_norm": 1.6875, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 552.3, "dropped_truncated": 0, "gold_loss": 1.1482, "gold_lambda": 0.5, "rep_ratio": 2.344, "t_data_s": 0.0, "t_rollout_s": 44.3, "t_step_s": 132.5, "t_refresh_s": 0.3, "mem_gb": 17.08, "mem_gb_teacher": 20.23}
|
| 179 |
+
{"step": 170, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.30282570564244016, "tokens": 168999, "cumulative_loss_tokens": 28874346, "grad_norm": 1.3671875, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 660.2, "dropped_truncated": 0, "gold_loss": 1.0061, "gold_lambda": 0.5, "rep_ratio": 2.6, "t_data_s": 0.0, "t_rollout_s": 52.2, "t_step_s": 150.0, "t_refresh_s": 0.3, "mem_gb": 17.16, "mem_gb_teacher": 20.25}
|
| 180 |
+
{"step": 171, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.32433426110505087, "tokens": 171462, "cumulative_loss_tokens": 29045808, "grad_norm": 1.421875, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 669.8, "dropped_truncated": 0, "gold_loss": 0.8578, "gold_lambda": 0.5, "rep_ratio": 2.481, "t_data_s": 0.0, "t_rollout_s": 50.6, "t_step_s": 141.3, "t_refresh_s": 0.3, "mem_gb": 16.97, "mem_gb_teacher": 20.23}
|
| 181 |
+
{"step": 172, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2515257881250364, "tokens": 150671, "cumulative_loss_tokens": 29196479, "grad_norm": 1.3359375, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 588.6, "dropped_truncated": 0, "gold_loss": 0.6706, "gold_lambda": 0.5, "rep_ratio": 2.243, "t_data_s": 0.0, "t_rollout_s": 47.5, "t_step_s": 139.3, "t_refresh_s": 0.3, "mem_gb": 17.33, "mem_gb_teacher": 20.26}
|
| 182 |
+
{"step": 173, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.28828223970889105, "tokens": 153462, "cumulative_loss_tokens": 29349941, "grad_norm": 1.1796875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 599.5, "dropped_truncated": 0, "gold_loss": 0.7977, "gold_lambda": 0.5, "rep_ratio": 2.5, "t_data_s": 0.0, "t_rollout_s": 46.7, "t_step_s": 138.3, "t_refresh_s": 0.3, "mem_gb": 17.26, "mem_gb_teacher": 20.25}
|
| 183 |
+
{"step": 174, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3212347760944004, "tokens": 153249, "cumulative_loss_tokens": 29503190, "grad_norm": 1.3984375, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 598.6, "dropped_truncated": 0, "gold_loss": 0.9575, "gold_lambda": 0.5, "rep_ratio": 2.404, "t_data_s": 0.0, "t_rollout_s": 48.6, "t_step_s": 139.9, "t_refresh_s": 0.3, "mem_gb": 17.18, "mem_gb_teacher": 20.26}
|
| 184 |
+
{"step": 175, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.274831599589699, "tokens": 174416, "cumulative_loss_tokens": 29677606, "grad_norm": 1.6015625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 681.3, "dropped_truncated": 0, "gold_loss": 1.4904, "gold_lambda": 0.5, "rep_ratio": 2.286, "t_data_s": 0.0, "t_rollout_s": 50.9, "t_step_s": 146.4, "t_refresh_s": 0.3, "mem_gb": 16.99, "mem_gb_teacher": 20.23}
|
| 185 |
+
{"step": 176, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2922794361090287, "tokens": 162432, "cumulative_loss_tokens": 29840038, "grad_norm": 1.40625, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 634.5, "dropped_truncated": 0, "gold_loss": 0.9979, "gold_lambda": 0.5, "rep_ratio": 2.399, "t_data_s": 0.0, "t_rollout_s": 48.8, "t_step_s": 142.4, "t_refresh_s": 0.3, "mem_gb": 17.23, "mem_gb_teacher": 20.24}
|
| 186 |
+
{"step": 177, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2864210852720016, "tokens": 158705, "cumulative_loss_tokens": 29998743, "grad_norm": 1.2421875, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 619.9, "dropped_truncated": 0, "gold_loss": 0.8183, "gold_lambda": 0.5, "rep_ratio": 2.391, "t_data_s": 0.0, "t_rollout_s": 48.2, "t_step_s": 139.7, "t_refresh_s": 0.3, "mem_gb": 16.99, "mem_gb_teacher": 20.22}
|
| 187 |
+
{"step": 178, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27929797398222117, "tokens": 193832, "cumulative_loss_tokens": 30192575, "grad_norm": 1.328125, "lr": 1e-05, "finish_rate": 0.953, "comp_len": 757.2, "dropped_truncated": 0, "gold_loss": 1.0017, "gold_lambda": 0.5, "rep_ratio": 2.575, "t_data_s": 0.0, "t_rollout_s": 57.7, "t_step_s": 158.2, "t_refresh_s": 0.3, "mem_gb": 17.42, "mem_gb_teacher": 20.27}
|
| 188 |
+
{"step": 179, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.357833825747071, "tokens": 152899, "cumulative_loss_tokens": 30345474, "grad_norm": 1.4609375, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 597.3, "dropped_truncated": 0, "gold_loss": 0.8395, "gold_lambda": 0.5, "rep_ratio": 2.238, "t_data_s": 0.0, "t_rollout_s": 46.9, "t_step_s": 136.9, "t_refresh_s": 0.3, "mem_gb": 17.3, "mem_gb_teacher": 20.25}
|
| 189 |
+
{"step": 180, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34215263686785763, "tokens": 156673, "cumulative_loss_tokens": 30502147, "grad_norm": 2.203125, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 612.0, "dropped_truncated": 0, "gold_loss": 1.0484, "gold_lambda": 0.5, "rep_ratio": 2.341, "t_data_s": 0.0, "t_rollout_s": 47.1, "t_step_s": 140.1, "t_refresh_s": 0.3, "mem_gb": 17.49, "mem_gb_teacher": 20.29}
|
| 190 |
+
{"step": 180, "gsm8k_n": 256, "gsm8k_quick_chat": 0.5859375, "t_eval_s": 21.4}
|
| 191 |
+
{"step": 181, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26608461892176444, "tokens": 202356, "cumulative_loss_tokens": 30704503, "grad_norm": 0.97265625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 790.5, "dropped_truncated": 0, "gold_loss": 0.2087, "gold_lambda": 0.5, "rep_ratio": 2.579, "t_data_s": 0.0, "t_rollout_s": 59.7, "t_step_s": 160.7, "t_refresh_s": 0.3, "mem_gb": 17.14, "mem_gb_teacher": 20.25}
|
| 192 |
+
{"step": 182, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.45915434895015195, "tokens": 168394, "cumulative_loss_tokens": 30872897, "grad_norm": 1.1875, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 657.8, "dropped_truncated": 0, "gold_loss": 0.2126, "gold_lambda": 0.5, "rep_ratio": 2.71, "t_data_s": 0.0, "t_rollout_s": 50.3, "t_step_s": 141.5, "t_refresh_s": 0.3, "mem_gb": 17.0, "mem_gb_teacher": 20.25}
|
| 193 |
+
{"step": 183, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3733151632443131, "tokens": 195527, "cumulative_loss_tokens": 31068424, "grad_norm": 0.98046875, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 763.8, "dropped_truncated": 0, "gold_loss": 0.2549, "gold_lambda": 0.5, "rep_ratio": 2.523, "t_data_s": 0.0, "t_rollout_s": 58.9, "t_step_s": 163.2, "t_refresh_s": 0.3, "mem_gb": 17.32, "mem_gb_teacher": 20.28}
|
| 194 |
+
{"step": 184, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34833771082801096, "tokens": 165364, "cumulative_loss_tokens": 31233788, "grad_norm": 1.1640625, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 646.0, "dropped_truncated": 0, "gold_loss": 0.2519, "gold_lambda": 0.5, "rep_ratio": 2.327, "t_data_s": 0.0, "t_rollout_s": 50.2, "t_step_s": 145.2, "t_refresh_s": 0.3, "mem_gb": 17.07, "mem_gb_teacher": 20.24}
|
| 195 |
+
{"step": 185, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.33309874653422017, "tokens": 172956, "cumulative_loss_tokens": 31406744, "grad_norm": 1.2890625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 675.6, "dropped_truncated": 0, "gold_loss": 0.7426, "gold_lambda": 0.5, "rep_ratio": 2.473, "t_data_s": 0.0, "t_rollout_s": 50.6, "t_step_s": 143.9, "t_refresh_s": 0.3, "mem_gb": 17.33, "mem_gb_teacher": 20.26}
|
| 196 |
+
{"step": 186, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3097168772122742, "tokens": 185109, "cumulative_loss_tokens": 31591853, "grad_norm": 1.5390625, "lr": 1e-05, "finish_rate": 0.949, "comp_len": 723.1, "dropped_truncated": 0, "gold_loss": 1.0151, "gold_lambda": 0.5, "rep_ratio": 2.499, "t_data_s": 0.0, "t_rollout_s": 54.1, "t_step_s": 147.8, "t_refresh_s": 0.3, "mem_gb": 17.38, "mem_gb_teacher": 20.27}
|
| 197 |
+
{"step": 187, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.37370173848453153, "tokens": 159109, "cumulative_loss_tokens": 31750962, "grad_norm": 1.4765625, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 621.5, "dropped_truncated": 0, "gold_loss": 0.8549, "gold_lambda": 0.5, "rep_ratio": 2.46, "t_data_s": 0.0, "t_rollout_s": 48.0, "t_step_s": 136.1, "t_refresh_s": 0.3, "mem_gb": 17.09, "mem_gb_teacher": 20.24}
|
| 198 |
+
{"step": 188, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.37454756275697315, "tokens": 152034, "cumulative_loss_tokens": 31902996, "grad_norm": 1.4921875, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 593.9, "dropped_truncated": 0, "gold_loss": 0.8653, "gold_lambda": 0.5, "rep_ratio": 2.284, "t_data_s": 0.0, "t_rollout_s": 48.1, "t_step_s": 136.8, "t_refresh_s": 0.3, "mem_gb": 17.07, "mem_gb_teacher": 20.23}
|
| 199 |
+
{"step": 189, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2785147800779869, "tokens": 195136, "cumulative_loss_tokens": 32098132, "grad_norm": 1.546875, "lr": 1e-05, "finish_rate": 0.941, "comp_len": 762.2, "dropped_truncated": 0, "gold_loss": 1.2027, "gold_lambda": 0.5, "rep_ratio": 2.493, "t_data_s": 0.0, "t_rollout_s": 58.6, "t_step_s": 158.7, "t_refresh_s": 0.3, "mem_gb": 17.45, "mem_gb_teacher": 20.27}
|
| 200 |
+
{"step": 190, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.31850948415368513, "tokens": 138949, "cumulative_loss_tokens": 32237081, "grad_norm": 1.625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 542.8, "dropped_truncated": 0, "gold_loss": 1.1625, "gold_lambda": 0.5, "rep_ratio": 2.509, "t_data_s": 0.0, "t_rollout_s": 43.3, "t_step_s": 126.0, "t_refresh_s": 0.3, "mem_gb": 17.15, "mem_gb_teacher": 20.23}
|
| 201 |
+
{"step": 191, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3427866812707038, "tokens": 160415, "cumulative_loss_tokens": 32397496, "grad_norm": 1.375, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 626.6, "dropped_truncated": 0, "gold_loss": 0.8919, "gold_lambda": 0.5, "rep_ratio": 2.495, "t_data_s": 0.0, "t_rollout_s": 49.2, "t_step_s": 140.0, "t_refresh_s": 0.3, "mem_gb": 17.53, "mem_gb_teacher": 20.28}
|
| 202 |
+
{"step": 192, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26732967083055503, "tokens": 166069, "cumulative_loss_tokens": 32563565, "grad_norm": 1.4375, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 648.7, "dropped_truncated": 0, "gold_loss": 1.1761, "gold_lambda": 0.5, "rep_ratio": 2.546, "t_data_s": 0.0, "t_rollout_s": 48.9, "t_step_s": 140.8, "t_refresh_s": 0.3, "mem_gb": 17.12, "mem_gb_teacher": 20.24}
|
| 203 |
+
{"step": 193, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.31801907909245924, "tokens": 153597, "cumulative_loss_tokens": 32717162, "grad_norm": 1.03125, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 600.0, "dropped_truncated": 0, "gold_loss": 0.2223, "gold_lambda": 0.5, "rep_ratio": 2.577, "t_data_s": 0.0, "t_rollout_s": 46.6, "t_step_s": 137.7, "t_refresh_s": 0.3, "mem_gb": 17.08, "mem_gb_teacher": 20.24}
|
| 204 |
+
{"step": 194, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.33985951355520294, "tokens": 175902, "cumulative_loss_tokens": 32893064, "grad_norm": 1.1484375, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 687.1, "dropped_truncated": 0, "gold_loss": 0.2286, "gold_lambda": 0.5, "rep_ratio": 2.409, "t_data_s": 0.0, "t_rollout_s": 52.1, "t_step_s": 148.2, "t_refresh_s": 0.3, "mem_gb": 17.23, "mem_gb_teacher": 20.25}
|
| 205 |
+
{"step": 195, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3251325274948627, "tokens": 172561, "cumulative_loss_tokens": 33065625, "grad_norm": 1.375, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 674.1, "dropped_truncated": 0, "gold_loss": 0.9158, "gold_lambda": 0.5, "rep_ratio": 2.501, "t_data_s": 0.0, "t_rollout_s": 51.2, "t_step_s": 144.2, "t_refresh_s": 0.3, "mem_gb": 17.61, "mem_gb_teacher": 20.29}
|
| 206 |
+
{"step": 196, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36415145249221653, "tokens": 134283, "cumulative_loss_tokens": 33199908, "grad_norm": 1.359375, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 524.5, "dropped_truncated": 0, "gold_loss": 0.8467, "gold_lambda": 0.5, "rep_ratio": 2.191, "t_data_s": 0.0, "t_rollout_s": 43.3, "t_step_s": 128.5, "t_refresh_s": 0.3, "mem_gb": 17.15, "mem_gb_teacher": 20.25}
|
| 207 |
+
{"step": 197, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3355495961174737, "tokens": 180667, "cumulative_loss_tokens": 33380575, "grad_norm": 1.2265625, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 705.7, "dropped_truncated": 0, "gold_loss": 0.8572, "gold_lambda": 0.5, "rep_ratio": 2.602, "t_data_s": 0.0, "t_rollout_s": 52.7, "t_step_s": 146.5, "t_refresh_s": 0.3, "mem_gb": 17.53, "mem_gb_teacher": 20.28}
|
| 208 |
+
{"step": 198, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.35168284546899287, "tokens": 159395, "cumulative_loss_tokens": 33539970, "grad_norm": 1.34375, "lr": 1e-05, "finish_rate": 0.961, "comp_len": 622.6, "dropped_truncated": 0, "gold_loss": 0.9759, "gold_lambda": 0.5, "rep_ratio": 2.528, "t_data_s": 0.0, "t_rollout_s": 48.8, "t_step_s": 138.8, "t_refresh_s": 0.3, "mem_gb": 17.4, "mem_gb_teacher": 20.28}
|
| 209 |
+
{"step": 199, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3245107139904741, "tokens": 164163, "cumulative_loss_tokens": 33704133, "grad_norm": 1.5234375, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 641.3, "dropped_truncated": 0, "gold_loss": 1.1404, "gold_lambda": 0.5, "rep_ratio": 2.434, "t_data_s": 0.0, "t_rollout_s": 49.0, "t_step_s": 138.6, "t_refresh_s": 0.3, "mem_gb": 17.48, "mem_gb_teacher": 20.28}
|
| 210 |
+
{"step": 200, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.291937393281781, "tokens": 168800, "cumulative_loss_tokens": 33872933, "grad_norm": 1.5625, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 659.4, "dropped_truncated": 0, "gold_loss": 1.112, "gold_lambda": 0.5, "rep_ratio": 2.579, "t_data_s": 0.0, "t_rollout_s": 51.3, "t_step_s": 144.2, "t_refresh_s": 0.3, "mem_gb": 17.27, "mem_gb_teacher": 20.27}
|
| 211 |
+
{"step": 200, "gsm8k_n": 256, "gsm8k_quick_chat": 0.59765625, "t_eval_s": 21.3}
|
| 212 |
+
{"step": 201, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.41485607102408806, "tokens": 158986, "cumulative_loss_tokens": 34031919, "grad_norm": 1.6015625, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 621.0, "dropped_truncated": 0, "gold_loss": 1.054, "gold_lambda": 0.5, "rep_ratio": 2.344, "t_data_s": 0.0, "t_rollout_s": 47.8, "t_step_s": 137.9, "t_refresh_s": 0.3, "mem_gb": 17.31, "mem_gb_teacher": 20.27}
|
| 213 |
+
{"step": 202, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.35614293253047014, "tokens": 162105, "cumulative_loss_tokens": 34194024, "grad_norm": 1.5625, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 633.2, "dropped_truncated": 0, "gold_loss": 1.2005, "gold_lambda": 0.5, "rep_ratio": 2.578, "t_data_s": 0.0, "t_rollout_s": 49.3, "t_step_s": 142.8, "t_refresh_s": 0.3, "mem_gb": 17.19, "mem_gb_teacher": 20.26}
|
| 214 |
+
{"step": 203, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2048144059971256, "tokens": 187822, "cumulative_loss_tokens": 34381846, "grad_norm": 1.3671875, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 733.7, "dropped_truncated": 0, "gold_loss": 0.9802, "gold_lambda": 0.5, "rep_ratio": 2.567, "t_data_s": 0.0, "t_rollout_s": 54.9, "t_step_s": 154.9, "t_refresh_s": 0.3, "mem_gb": 17.1, "mem_gb_teacher": 20.23}
|
| 215 |
+
{"step": 204, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4261384146695686, "tokens": 151644, "cumulative_loss_tokens": 34533490, "grad_norm": 1.4140625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 592.4, "dropped_truncated": 0, "gold_loss": 0.6612, "gold_lambda": 0.5, "rep_ratio": 2.415, "t_data_s": 0.0, "t_rollout_s": 46.4, "t_step_s": 135.2, "t_refresh_s": 0.3, "mem_gb": 17.06, "mem_gb_teacher": 20.23}
|
| 216 |
+
{"step": 205, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.40458391981911457, "tokens": 156841, "cumulative_loss_tokens": 34690331, "grad_norm": 1.4296875, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 612.7, "dropped_truncated": 0, "gold_loss": 0.7533, "gold_lambda": 0.5, "rep_ratio": 2.393, "t_data_s": 0.0, "t_rollout_s": 48.0, "t_step_s": 141.8, "t_refresh_s": 0.3, "mem_gb": 17.08, "mem_gb_teacher": 20.24}
|
| 217 |
+
{"step": 206, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.32073098364184105, "tokens": 177406, "cumulative_loss_tokens": 34867737, "grad_norm": 1.3203125, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 693.0, "dropped_truncated": 0, "gold_loss": 0.6748, "gold_lambda": 0.5, "rep_ratio": 2.475, "t_data_s": 0.0, "t_rollout_s": 53.5, "t_step_s": 150.1, "t_refresh_s": 0.3, "mem_gb": 17.42, "mem_gb_teacher": 20.27}
|
| 218 |
+
{"step": 207, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.30331866068701224, "tokens": 179407, "cumulative_loss_tokens": 35047144, "grad_norm": 1.328125, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 700.8, "dropped_truncated": 0, "gold_loss": 1.0473, "gold_lambda": 0.5, "rep_ratio": 2.451, "t_data_s": 0.0, "t_rollout_s": 51.8, "t_step_s": 143.5, "t_refresh_s": 0.3, "mem_gb": 17.04, "mem_gb_teacher": 20.23}
|
| 219 |
+
{"step": 208, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2920594240087867, "tokens": 165842, "cumulative_loss_tokens": 35212986, "grad_norm": 1.3515625, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 647.8, "dropped_truncated": 0, "gold_loss": 0.994, "gold_lambda": 0.5, "rep_ratio": 2.51, "t_data_s": 0.0, "t_rollout_s": 49.0, "t_step_s": 139.6, "t_refresh_s": 0.3, "mem_gb": 17.08, "mem_gb_teacher": 20.23}
|
| 220 |
+
{"step": 209, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.33393017593075786, "tokens": 166060, "cumulative_loss_tokens": 35379046, "grad_norm": 1.4296875, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 648.7, "dropped_truncated": 0, "gold_loss": 1.0433, "gold_lambda": 0.5, "rep_ratio": 2.487, "t_data_s": 0.0, "t_rollout_s": 49.6, "t_step_s": 138.4, "t_refresh_s": 0.3, "mem_gb": 16.98, "mem_gb_teacher": 20.22}
|
| 221 |
+
{"step": 210, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3979693316027235, "tokens": 143251, "cumulative_loss_tokens": 35522297, "grad_norm": 1.578125, "lr": 1e-05, "finish_rate": 0.992, "comp_len": 559.6, "dropped_truncated": 0, "gold_loss": 1.0529, "gold_lambda": 0.5, "rep_ratio": 2.204, "t_data_s": 0.0, "t_rollout_s": 46.0, "t_step_s": 135.8, "t_refresh_s": 0.3, "mem_gb": 17.14, "mem_gb_teacher": 20.25}
|
| 222 |
+
{"step": 211, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3822666124768588, "tokens": 153854, "cumulative_loss_tokens": 35676151, "grad_norm": 1.171875, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 601.0, "dropped_truncated": 0, "gold_loss": 0.208, "gold_lambda": 0.5, "rep_ratio": 2.247, "t_data_s": 0.0, "t_rollout_s": 47.5, "t_step_s": 139.1, "t_refresh_s": 0.3, "mem_gb": 17.14, "mem_gb_teacher": 20.23}
|
| 223 |
+
{"step": 212, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4683672417548319, "tokens": 155158, "cumulative_loss_tokens": 35831309, "grad_norm": 1.1640625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 606.1, "dropped_truncated": 0, "gold_loss": 0.2194, "gold_lambda": 0.5, "rep_ratio": 2.419, "t_data_s": 0.0, "t_rollout_s": 48.5, "t_step_s": 141.3, "t_refresh_s": 0.3, "mem_gb": 17.42, "mem_gb_teacher": 20.26}
|
| 224 |
+
{"step": 213, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.4032089717409549, "tokens": 159676, "cumulative_loss_tokens": 35990985, "grad_norm": 1.8828125, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 623.7, "dropped_truncated": 0, "gold_loss": 1.4951, "gold_lambda": 0.5, "rep_ratio": 2.362, "t_data_s": 0.0, "t_rollout_s": 47.6, "t_step_s": 139.2, "t_refresh_s": 0.3, "mem_gb": 17.09, "mem_gb_teacher": 20.22}
|
| 225 |
+
{"step": 214, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3410087909027437, "tokens": 179372, "cumulative_loss_tokens": 36170357, "grad_norm": 1.515625, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 700.7, "dropped_truncated": 0, "gold_loss": 1.2847, "gold_lambda": 0.5, "rep_ratio": 2.327, "t_data_s": 0.0, "t_rollout_s": 52.5, "t_step_s": 145.0, "t_refresh_s": 0.3, "mem_gb": 17.51, "mem_gb_teacher": 20.3}
|
| 226 |
+
{"step": 215, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3329530498687338, "tokens": 178024, "cumulative_loss_tokens": 36348381, "grad_norm": 1.6015625, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 695.4, "dropped_truncated": 0, "gold_loss": 1.2281, "gold_lambda": 0.5, "rep_ratio": 2.88, "t_data_s": 0.0, "t_rollout_s": 54.6, "t_step_s": 150.8, "t_refresh_s": 0.3, "mem_gb": 17.83, "mem_gb_teacher": 20.33}
|
| 227 |
+
{"step": 216, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34099034257406186, "tokens": 158663, "cumulative_loss_tokens": 36507044, "grad_norm": 1.3671875, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 619.8, "dropped_truncated": 0, "gold_loss": 1.1019, "gold_lambda": 0.5, "rep_ratio": 2.335, "t_data_s": 0.0, "t_rollout_s": 48.0, "t_step_s": 136.8, "t_refresh_s": 0.3, "mem_gb": 17.27, "mem_gb_teacher": 20.24}
|
| 228 |
+
{"step": 217, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.38961163875816524, "tokens": 171705, "cumulative_loss_tokens": 36678749, "grad_norm": 1.9453125, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 670.7, "dropped_truncated": 0, "gold_loss": 1.1446, "gold_lambda": 0.5, "rep_ratio": 2.401, "t_data_s": 0.0, "t_rollout_s": 50.3, "t_step_s": 142.3, "t_refresh_s": 0.3, "mem_gb": 17.19, "mem_gb_teacher": 20.25}
|
| 229 |
+
{"step": 218, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.38725802709332435, "tokens": 135912, "cumulative_loss_tokens": 36814661, "grad_norm": 1.5859375, "lr": 1e-05, "finish_rate": 0.984, "comp_len": 530.9, "dropped_truncated": 0, "gold_loss": 1.05, "gold_lambda": 0.5, "rep_ratio": 2.297, "t_data_s": 0.0, "t_rollout_s": 43.1, "t_step_s": 126.3, "t_refresh_s": 0.3, "mem_gb": 17.14, "mem_gb_teacher": 20.23}
|
| 230 |
+
{"step": 219, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3388758097317682, "tokens": 157159, "cumulative_loss_tokens": 36971820, "grad_norm": 1.1015625, "lr": 1e-05, "finish_rate": 0.996, "comp_len": 613.9, "dropped_truncated": 0, "gold_loss": 0.2417, "gold_lambda": 0.5, "rep_ratio": 2.276, "t_data_s": 0.0, "t_rollout_s": 46.9, "t_step_s": 138.3, "t_refresh_s": 0.3, "mem_gb": 17.08, "mem_gb_teacher": 20.24}
|
| 231 |
+
{"step": 220, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.29886958715946443, "tokens": 165992, "cumulative_loss_tokens": 37137812, "grad_norm": 1.0234375, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 648.4, "dropped_truncated": 0, "gold_loss": 0.2285, "gold_lambda": 0.5, "rep_ratio": 2.418, "t_data_s": 0.0, "t_rollout_s": 50.3, "t_step_s": 145.0, "t_refresh_s": 0.3, "mem_gb": 17.68, "mem_gb_teacher": 20.31}
|
| 232 |
+
{"step": 220, "gsm8k_n": 256, "gsm8k_quick_chat": 0.6171875, "t_eval_s": 21.3}
|
| 233 |
+
{"step": 221, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3517129271357706, "tokens": 152770, "cumulative_loss_tokens": 37290582, "grad_norm": 1.4453125, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 596.8, "dropped_truncated": 0, "gold_loss": 1.0769, "gold_lambda": 0.5, "rep_ratio": 2.233, "t_data_s": 0.0, "t_rollout_s": 46.0, "t_step_s": 136.6, "t_refresh_s": 0.3, "mem_gb": 17.02, "mem_gb_teacher": 20.23}
|
| 234 |
+
{"step": 222, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2916604653136137, "tokens": 173179, "cumulative_loss_tokens": 37463761, "grad_norm": 1.6015625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 676.5, "dropped_truncated": 0, "gold_loss": 1.2028, "gold_lambda": 0.5, "rep_ratio": 2.849, "t_data_s": 0.0, "t_rollout_s": 50.8, "t_step_s": 142.9, "t_refresh_s": 0.3, "mem_gb": 17.04, "mem_gb_teacher": 20.23}
|
| 235 |
+
{"step": 223, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.37753672219781415, "tokens": 164152, "cumulative_loss_tokens": 37627913, "grad_norm": 1.046875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 641.2, "dropped_truncated": 0, "gold_loss": 0.2096, "gold_lambda": 0.5, "rep_ratio": 2.378, "t_data_s": 0.0, "t_rollout_s": 49.1, "t_step_s": 142.5, "t_refresh_s": 0.3, "mem_gb": 17.24, "mem_gb_teacher": 20.24}
|
healed/opd_warm_fixed_keep50/vllm_server.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
healed/opd_warm_fixed_keep50/wandb_sync.log
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
Find logs at: /tmp/debug-cli.henry.log
|
| 2 |
+
Syncing: https://wandb.ai/hbfreed/glean-heal/runs/ubll0rd3 ... done.
|
healed/opd_warm_keep50/args.json
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"student": "outputs/healed/keep50_offpolicy_warmup_s1224/step0150",
|
| 3 |
+
"teacher": "allenai/OLMoE-1B-7B-0125-Instruct",
|
| 4 |
+
"training_mode": "on-policy",
|
| 5 |
+
"kl_direction": "reverse",
|
| 6 |
+
"dataset": "allenai/Dolci-Instruct-RL",
|
| 7 |
+
"dataset_sources": null,
|
| 8 |
+
"max_difficulty": null,
|
| 9 |
+
"trajectories": "outputs/teacher_trajectories/dolci_math_curated.jsonl",
|
| 10 |
+
"trajectory_dataset": "allenai/Dolci-Instruct-RL",
|
| 11 |
+
"off_policy_frames": "chat",
|
| 12 |
+
"off_policy_max_seq_len": 2048,
|
| 13 |
+
"topk_targets": null,
|
| 14 |
+
"max_loss_tokens": null,
|
| 15 |
+
"loss_tokens_per_step": null,
|
| 16 |
+
"teacher_device": "cuda:0",
|
| 17 |
+
"student_device": "cuda:1",
|
| 18 |
+
"lr": 1e-05,
|
| 19 |
+
"optimizer": "adamw8bit",
|
| 20 |
+
"weight_decay": 0.1,
|
| 21 |
+
"epochs": 2,
|
| 22 |
+
"prompts_per_step": 256,
|
| 23 |
+
"group_size": 4,
|
| 24 |
+
"rollout_batch": 64,
|
| 25 |
+
"micro_batch": 4,
|
| 26 |
+
"max_new_tokens": 2048,
|
| 27 |
+
"max_prompt_len": 1024,
|
| 28 |
+
"warmup_steps": 10,
|
| 29 |
+
"max_grad_norm": 1.0,
|
| 30 |
+
"eval_every": 10,
|
| 31 |
+
"gsm8k_every": 20,
|
| 32 |
+
"gsm8k_n": 256,
|
| 33 |
+
"gsm8k_batch": 16,
|
| 34 |
+
"gsm8k_max_new_tokens": 1024,
|
| 35 |
+
"gsm8k_frames": "chat",
|
| 36 |
+
"save_every": 1000,
|
| 37 |
+
"out_dir": "outputs/healed/opd_warm_keep50",
|
| 38 |
+
"sweep": 120,
|
| 39 |
+
"wandb": true,
|
| 40 |
+
"wandb_project": "glean-heal",
|
| 41 |
+
"wandb_run_name": "opd-warm-keep50-s1223",
|
| 42 |
+
"wandb_run_id": null,
|
| 43 |
+
"wandb_resume": null,
|
| 44 |
+
"wandb_mode": "offline",
|
| 45 |
+
"no_wandb_sync": false,
|
| 46 |
+
"debug": false,
|
| 47 |
+
"resume_from": null,
|
| 48 |
+
"start_step": 0,
|
| 49 |
+
"no_grad_checkpointing": false,
|
| 50 |
+
"seed": 1223,
|
| 51 |
+
"no_teacher_overlap": false,
|
| 52 |
+
"sync_checkpoints": false,
|
| 53 |
+
"rollout_engine": "vllm",
|
| 54 |
+
"vllm_gpu": "2",
|
| 55 |
+
"vllm_port": 8377,
|
| 56 |
+
"vllm_refresh_every": 1,
|
| 57 |
+
"vllm_serve_bin": "vllm-plugin/.venv/bin/python",
|
| 58 |
+
"vllm_gpu_mem_util": 0.85,
|
| 59 |
+
"liger_loss": true,
|
| 60 |
+
"gold_mix_lambda": 0.5,
|
| 61 |
+
"gold_topk_targets": "outputs/teacher_trajectories/dolci_combined_top128",
|
| 62 |
+
"gold_loss": "ce",
|
| 63 |
+
"gold_mix_decay": 0.0,
|
| 64 |
+
"fast_teacher": true,
|
| 65 |
+
"reference_kl_beta": 0.05,
|
| 66 |
+
"drop_truncated_rollouts": false,
|
| 67 |
+
"vllm_max_model_len": null,
|
| 68 |
+
"vllm_refresh_mode": "reload",
|
| 69 |
+
"vllm_live_dir": null,
|
| 70 |
+
"resolved_kl_direction": "reverse"
|
| 71 |
+
}
|
healed/opd_warm_keep50/train_log.jsonl
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"step": 1, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.0715529867857179, "tokens": 330677, "cumulative_loss_tokens": 330677, "grad_norm": 18.0, "lr": 2.0000000000000003e-06, "finish_rate": 0.66, "comp_len": 1291.7, "dropped_truncated": 0, "gold_loss": 0.4816, "gold_lambda": 0.5, "rep_ratio": 7.359, "t_data_s": 0.0, "t_rollout_s": 117.2, "t_step_s": 230.0, "t_refresh_s": 0.3, "mem_gb": 10.4, "mem_gb_teacher": 20.7}
|
| 2 |
+
{"step": 1, "gsm8k_n": 256, "gsm8k_quick_chat": 0.46875, "t_eval_s": 40.3}
|
| 3 |
+
{"step": 2, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.0476409321224727, "tokens": 305425, "cumulative_loss_tokens": 636102, "grad_norm": 19.25, "lr": 3e-06, "finish_rate": 0.699, "comp_len": 1193.1, "dropped_truncated": 0, "gold_loss": 0.4636, "gold_lambda": 0.5, "rep_ratio": 6.842, "t_data_s": 0.0, "t_rollout_s": 113.7, "t_step_s": 214.3, "t_refresh_s": 0.3, "mem_gb": 10.55, "mem_gb_teacher": 20.67}
|
| 4 |
+
{"step": 3, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.2240935077801478, "tokens": 302635, "cumulative_loss_tokens": 938737, "grad_norm": 16.75, "lr": 4.000000000000001e-06, "finish_rate": 0.711, "comp_len": 1182.2, "dropped_truncated": 0, "gold_loss": 0.4989, "gold_lambda": 0.5, "rep_ratio": 8.206, "t_data_s": 0.0, "t_rollout_s": 109.6, "t_step_s": 211.3, "t_refresh_s": 0.3, "mem_gb": 10.46, "mem_gb_teacher": 20.63}
|
| 5 |
+
{"step": 4, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1232984279951, "tokens": 314442, "cumulative_loss_tokens": 1253179, "grad_norm": 13.625, "lr": 5e-06, "finish_rate": 0.723, "comp_len": 1228.3, "dropped_truncated": 0, "gold_loss": 0.4989, "gold_lambda": 0.5, "rep_ratio": 4.007, "t_data_s": 0.0, "t_rollout_s": 112.2, "t_step_s": 215.0, "t_refresh_s": 0.3, "mem_gb": 10.24, "mem_gb_teacher": 20.57}
|
| 6 |
+
{"step": 5, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.196489499796483, "tokens": 314169, "cumulative_loss_tokens": 1567348, "grad_norm": 12.0625, "lr": 6e-06, "finish_rate": 0.715, "comp_len": 1227.2, "dropped_truncated": 0, "gold_loss": 1.7581, "gold_lambda": 0.5, "rep_ratio": 7.457, "t_data_s": 0.0, "t_rollout_s": 113.5, "t_step_s": 213.6, "t_refresh_s": 0.3, "mem_gb": 10.48, "mem_gb_teacher": 20.64}
|
| 7 |
+
{"step": 6, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.174327937240284, "tokens": 302861, "cumulative_loss_tokens": 1870209, "grad_norm": 10.125, "lr": 7e-06, "finish_rate": 0.715, "comp_len": 1183.1, "dropped_truncated": 0, "gold_loss": 1.4087, "gold_lambda": 0.5, "rep_ratio": 6.804, "t_data_s": 0.0, "t_rollout_s": 112.3, "t_step_s": 214.1, "t_refresh_s": 0.3, "mem_gb": 10.49, "mem_gb_teacher": 20.64}
|
| 8 |
+
{"step": 7, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.142888064626811, "tokens": 259225, "cumulative_loss_tokens": 2129434, "grad_norm": 8.0625, "lr": 8.000000000000001e-06, "finish_rate": 0.875, "comp_len": 1012.6, "dropped_truncated": 0, "gold_loss": 1.7253, "gold_lambda": 0.5, "rep_ratio": 2.291, "t_data_s": 0.0, "t_rollout_s": 85.8, "t_step_s": 180.9, "t_refresh_s": 0.3, "mem_gb": 10.5, "mem_gb_teacher": 20.66}
|
| 9 |
+
{"step": 8, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.2367885471050992, "tokens": 275165, "cumulative_loss_tokens": 2404599, "grad_norm": 10.0625, "lr": 9e-06, "finish_rate": 0.777, "comp_len": 1074.9, "dropped_truncated": 0, "gold_loss": 1.791, "gold_lambda": 0.5, "rep_ratio": 4.278, "t_data_s": 0.0, "t_rollout_s": 101.9, "t_step_s": 198.9, "t_refresh_s": 0.3, "mem_gb": 10.37, "mem_gb_teacher": 20.6}
|
| 10 |
+
{"step": 9, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.2960635659850794, "tokens": 287956, "cumulative_loss_tokens": 2692555, "grad_norm": 10.5, "lr": 1e-05, "finish_rate": 0.777, "comp_len": 1124.8, "dropped_truncated": 0, "gold_loss": 0.4427, "gold_lambda": 0.5, "rep_ratio": 3.539, "t_data_s": 0.0, "t_rollout_s": 107.3, "t_step_s": 217.4, "t_refresh_s": 0.3, "mem_gb": 10.37, "mem_gb_teacher": 20.61}
|
| 11 |
+
{"step": 10, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1268998827677594, "tokens": 256196, "cumulative_loss_tokens": 2948751, "grad_norm": 8.1875, "lr": 1e-05, "finish_rate": 0.852, "comp_len": 1000.8, "dropped_truncated": 0, "gold_loss": 0.4179, "gold_lambda": 0.5, "rep_ratio": 5.761, "t_data_s": 0.0, "t_rollout_s": 83.3, "t_step_s": 177.5, "t_refresh_s": 0.3, "mem_gb": 10.63, "mem_gb_teacher": 20.7}
|
| 12 |
+
{"step": 11, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1852989046363867, "tokens": 249794, "cumulative_loss_tokens": 3198545, "grad_norm": 9.4375, "lr": 1e-05, "finish_rate": 0.879, "comp_len": 975.8, "dropped_truncated": 0, "gold_loss": 1.8732, "gold_lambda": 0.5, "rep_ratio": 2.592, "t_data_s": 0.0, "t_rollout_s": 79.1, "t_step_s": 170.9, "t_refresh_s": 0.3, "mem_gb": 10.17, "mem_gb_teacher": 20.53}
|
| 13 |
+
{"step": 12, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.0672082473087046, "tokens": 235329, "cumulative_loss_tokens": 3433874, "grad_norm": 6.40625, "lr": 1e-05, "finish_rate": 0.918, "comp_len": 919.3, "dropped_truncated": 0, "gold_loss": 1.7626, "gold_lambda": 0.5, "rep_ratio": 2.465, "t_data_s": 0.0, "t_rollout_s": 72.1, "t_step_s": 161.9, "t_refresh_s": 0.3, "mem_gb": 10.51, "mem_gb_teacher": 20.65}
|
| 14 |
+
{"step": 13, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.1896850641550707, "tokens": 196953, "cumulative_loss_tokens": 3630827, "grad_norm": 6.78125, "lr": 1e-05, "finish_rate": 0.945, "comp_len": 769.3, "dropped_truncated": 0, "gold_loss": 1.8544, "gold_lambda": 0.5, "rep_ratio": 2.39, "t_data_s": 0.0, "t_rollout_s": 57.2, "t_step_s": 138.9, "t_refresh_s": 0.3, "mem_gb": 10.31, "mem_gb_teacher": 20.59}
|
| 15 |
+
{"step": 14, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.110568549902207, "tokens": 195354, "cumulative_loss_tokens": 3826181, "grad_norm": 5.25, "lr": 1e-05, "finish_rate": 0.938, "comp_len": 763.1, "dropped_truncated": 0, "gold_loss": 1.7907, "gold_lambda": 0.5, "rep_ratio": 2.662, "t_data_s": 0.0, "t_rollout_s": 57.3, "t_step_s": 140.7, "t_refresh_s": 0.3, "mem_gb": 10.2, "mem_gb_teacher": 20.55}
|
| 16 |
+
{"step": 15, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.0880019511972423, "tokens": 211631, "cumulative_loss_tokens": 4037812, "grad_norm": 4.78125, "lr": 1e-05, "finish_rate": 0.938, "comp_len": 826.7, "dropped_truncated": 0, "gold_loss": 0.4573, "gold_lambda": 0.5, "rep_ratio": 4.91, "t_data_s": 0.0, "t_rollout_s": 61.5, "t_step_s": 147.1, "t_refresh_s": 0.3, "mem_gb": 10.26, "mem_gb_teacher": 20.53}
|
| 17 |
+
{"step": 16, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9728942313239658, "tokens": 194169, "cumulative_loss_tokens": 4231981, "grad_norm": 7.90625, "lr": 1e-05, "finish_rate": 0.953, "comp_len": 758.5, "dropped_truncated": 0, "gold_loss": 0.4647, "gold_lambda": 0.5, "rep_ratio": 12.498, "t_data_s": 0.0, "t_rollout_s": 57.5, "t_step_s": 140.4, "t_refresh_s": 0.3, "mem_gb": 10.38, "mem_gb_teacher": 20.56}
|
| 18 |
+
{"step": 17, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.0340790669212048, "tokens": 182968, "cumulative_loss_tokens": 4414949, "grad_norm": 8.5625, "lr": 1e-05, "finish_rate": 0.945, "comp_len": 714.7, "dropped_truncated": 0, "gold_loss": 1.5083, "gold_lambda": 0.5, "rep_ratio": 2.658, "t_data_s": 0.0, "t_rollout_s": 55.6, "t_step_s": 137.3, "t_refresh_s": 0.3, "mem_gb": 10.54, "mem_gb_teacher": 20.66}
|
| 19 |
+
{"step": 18, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9951681053661686, "tokens": 192973, "cumulative_loss_tokens": 4607922, "grad_norm": 4.09375, "lr": 1e-05, "finish_rate": 0.965, "comp_len": 753.8, "dropped_truncated": 0, "gold_loss": 1.5254, "gold_lambda": 0.5, "rep_ratio": 2.551, "t_data_s": 0.0, "t_rollout_s": 58.0, "t_step_s": 139.2, "t_refresh_s": 0.3, "mem_gb": 10.28, "mem_gb_teacher": 20.58}
|
| 20 |
+
{"step": 19, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.941273376354514, "tokens": 172849, "cumulative_loss_tokens": 4780771, "grad_norm": 3.65625, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 675.2, "dropped_truncated": 0, "gold_loss": 1.7476, "gold_lambda": 0.5, "rep_ratio": 2.772, "t_data_s": 0.0, "t_rollout_s": 52.5, "t_step_s": 129.2, "t_refresh_s": 0.3, "mem_gb": 10.44, "mem_gb_teacher": 20.63}
|
| 21 |
+
{"step": 20, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8855722967075427, "tokens": 178588, "cumulative_loss_tokens": 4959359, "grad_norm": 3.34375, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 697.6, "dropped_truncated": 0, "gold_loss": 1.8831, "gold_lambda": 0.5, "rep_ratio": 2.626, "t_data_s": 0.0, "t_rollout_s": 53.1, "t_step_s": 129.8, "t_refresh_s": 0.3, "mem_gb": 10.64, "mem_gb_teacher": 20.63}
|
| 22 |
+
{"step": 20, "gsm8k_n": 256, "gsm8k_quick_chat": 0.49609375, "t_eval_s": 43.9}
|
| 23 |
+
{"step": 21, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 1.0167943671227804, "tokens": 170868, "cumulative_loss_tokens": 5130227, "grad_norm": 4.125, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 667.5, "dropped_truncated": 0, "gold_loss": 0.4302, "gold_lambda": 0.5, "rep_ratio": 2.259, "t_data_s": 0.0, "t_rollout_s": 50.9, "t_step_s": 126.6, "t_refresh_s": 0.3, "mem_gb": 10.41, "mem_gb_teacher": 20.55}
|
| 24 |
+
{"step": 22, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9240483859857905, "tokens": 186012, "cumulative_loss_tokens": 5316239, "grad_norm": 3.453125, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 726.6, "dropped_truncated": 0, "gold_loss": 0.4496, "gold_lambda": 0.5, "rep_ratio": 2.576, "t_data_s": 0.0, "t_rollout_s": 54.9, "t_step_s": 133.4, "t_refresh_s": 0.3, "mem_gb": 10.37, "mem_gb_teacher": 20.58}
|
| 25 |
+
{"step": 23, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9889739579658967, "tokens": 161101, "cumulative_loss_tokens": 5477340, "grad_norm": 3.6875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 629.3, "dropped_truncated": 0, "gold_loss": 1.1295, "gold_lambda": 0.5, "rep_ratio": 2.562, "t_data_s": 0.0, "t_rollout_s": 49.2, "t_step_s": 126.1, "t_refresh_s": 0.3, "mem_gb": 10.19, "mem_gb_teacher": 20.54}
|
| 26 |
+
{"step": 24, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9001673450655785, "tokens": 172167, "cumulative_loss_tokens": 5649507, "grad_norm": 2.84375, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 672.5, "dropped_truncated": 0, "gold_loss": 1.184, "gold_lambda": 0.5, "rep_ratio": 2.392, "t_data_s": 0.0, "t_rollout_s": 51.2, "t_step_s": 125.6, "t_refresh_s": 0.3, "mem_gb": 10.42, "mem_gb_teacher": 20.61}
|
| 27 |
+
{"step": 25, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8294563550778241, "tokens": 193603, "cumulative_loss_tokens": 5843110, "grad_norm": 3.203125, "lr": 1e-05, "finish_rate": 0.949, "comp_len": 756.3, "dropped_truncated": 0, "gold_loss": 2.1916, "gold_lambda": 0.5, "rep_ratio": 2.737, "t_data_s": 0.0, "t_rollout_s": 57.4, "t_step_s": 137.7, "t_refresh_s": 0.3, "mem_gb": 10.55, "mem_gb_teacher": 20.67}
|
| 28 |
+
{"step": 26, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8520757204397251, "tokens": 184343, "cumulative_loss_tokens": 6027453, "grad_norm": 2.6875, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 720.1, "dropped_truncated": 0, "gold_loss": 1.8995, "gold_lambda": 0.5, "rep_ratio": 2.86, "t_data_s": 0.0, "t_rollout_s": 54.2, "t_step_s": 131.3, "t_refresh_s": 0.3, "mem_gb": 10.35, "mem_gb_teacher": 20.59}
|
| 29 |
+
{"step": 27, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9107435231664042, "tokens": 190215, "cumulative_loss_tokens": 6217668, "grad_norm": 2.625, "lr": 1e-05, "finish_rate": 0.953, "comp_len": 743.0, "dropped_truncated": 0, "gold_loss": 1.4011, "gold_lambda": 0.5, "rep_ratio": 2.559, "t_data_s": 0.0, "t_rollout_s": 55.9, "t_step_s": 134.6, "t_refresh_s": 0.3, "mem_gb": 10.41, "mem_gb_teacher": 20.62}
|
| 30 |
+
{"step": 28, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9077256934040764, "tokens": 178659, "cumulative_loss_tokens": 6396327, "grad_norm": 2.1875, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 697.9, "dropped_truncated": 0, "gold_loss": 1.4574, "gold_lambda": 0.5, "rep_ratio": 2.687, "t_data_s": 0.0, "t_rollout_s": 51.5, "t_step_s": 126.5, "t_refresh_s": 0.3, "mem_gb": 10.37, "mem_gb_teacher": 20.54}
|
| 31 |
+
{"step": 29, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9122361920726376, "tokens": 179195, "cumulative_loss_tokens": 6575522, "grad_norm": 1.7109375, "lr": 1e-05, "finish_rate": 0.949, "comp_len": 700.0, "dropped_truncated": 0, "gold_loss": 0.4425, "gold_lambda": 0.5, "rep_ratio": 2.463, "t_data_s": 0.0, "t_rollout_s": 54.0, "t_step_s": 131.5, "t_refresh_s": 0.3, "mem_gb": 10.38, "mem_gb_teacher": 20.6}
|
| 32 |
+
{"step": 30, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8639218128417365, "tokens": 196883, "cumulative_loss_tokens": 6772405, "grad_norm": 1.625, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 769.1, "dropped_truncated": 0, "gold_loss": 0.4074, "gold_lambda": 0.5, "rep_ratio": 2.545, "t_data_s": 0.0, "t_rollout_s": 57.7, "t_step_s": 138.1, "t_refresh_s": 0.3, "mem_gb": 10.48, "mem_gb_teacher": 20.64}
|
| 33 |
+
{"step": 31, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8510895792700558, "tokens": 203228, "cumulative_loss_tokens": 6975633, "grad_norm": 2.484375, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 793.9, "dropped_truncated": 0, "gold_loss": 1.7625, "gold_lambda": 0.5, "rep_ratio": 2.537, "t_data_s": 0.0, "t_rollout_s": 59.8, "t_step_s": 139.0, "t_refresh_s": 0.3, "mem_gb": 10.1, "mem_gb_teacher": 20.51}
|
| 34 |
+
{"step": 32, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.799281217758318, "tokens": 186923, "cumulative_loss_tokens": 7162556, "grad_norm": 2.125, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 730.2, "dropped_truncated": 0, "gold_loss": 1.4991, "gold_lambda": 0.5, "rep_ratio": 2.655, "t_data_s": 0.0, "t_rollout_s": 55.1, "t_step_s": 134.5, "t_refresh_s": 0.3, "mem_gb": 10.43, "mem_gb_teacher": 20.6}
|
| 35 |
+
{"step": 33, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.9108344851361305, "tokens": 206685, "cumulative_loss_tokens": 7369241, "grad_norm": 2.609375, "lr": 1e-05, "finish_rate": 0.941, "comp_len": 807.4, "dropped_truncated": 0, "gold_loss": 1.8247, "gold_lambda": 0.5, "rep_ratio": 2.723, "t_data_s": 0.0, "t_rollout_s": 61.8, "t_step_s": 145.9, "t_refresh_s": 0.3, "mem_gb": 10.34, "mem_gb_teacher": 20.6}
|
| 36 |
+
{"step": 34, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8392004683972054, "tokens": 200636, "cumulative_loss_tokens": 7569877, "grad_norm": 5.21875, "lr": 1e-05, "finish_rate": 0.953, "comp_len": 783.7, "dropped_truncated": 0, "gold_loss": 2.3876, "gold_lambda": 0.5, "rep_ratio": 2.624, "t_data_s": 0.0, "t_rollout_s": 59.5, "t_step_s": 140.6, "t_refresh_s": 0.3, "mem_gb": 10.38, "mem_gb_teacher": 20.56}
|
| 37 |
+
{"step": 35, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8846296758133513, "tokens": 180440, "cumulative_loss_tokens": 7750317, "grad_norm": 2.015625, "lr": 1e-05, "finish_rate": 0.98, "comp_len": 704.8, "dropped_truncated": 0, "gold_loss": 0.3641, "gold_lambda": 0.5, "rep_ratio": 2.604, "t_data_s": 0.0, "t_rollout_s": 51.9, "t_step_s": 127.8, "t_refresh_s": 0.3, "mem_gb": 10.24, "mem_gb_teacher": 20.53}
|
| 38 |
+
{"step": 36, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8721849152298049, "tokens": 194494, "cumulative_loss_tokens": 7944811, "grad_norm": 1.859375, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 759.7, "dropped_truncated": 0, "gold_loss": 0.4256, "gold_lambda": 0.5, "rep_ratio": 2.591, "t_data_s": 0.0, "t_rollout_s": 56.4, "t_step_s": 136.2, "t_refresh_s": 0.3, "mem_gb": 10.27, "mem_gb_teacher": 20.57}
|
| 39 |
+
{"step": 37, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.849565157371755, "tokens": 186626, "cumulative_loss_tokens": 8131437, "grad_norm": 2.390625, "lr": 1e-05, "finish_rate": 0.977, "comp_len": 729.0, "dropped_truncated": 0, "gold_loss": 1.6035, "gold_lambda": 0.5, "rep_ratio": 2.528, "t_data_s": 0.0, "t_rollout_s": 54.0, "t_step_s": 132.0, "t_refresh_s": 0.3, "mem_gb": 10.26, "mem_gb_teacher": 20.53}
|
| 40 |
+
{"step": 38, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.7732366005655155, "tokens": 210982, "cumulative_loss_tokens": 8342419, "grad_norm": 4.96875, "lr": 1e-05, "finish_rate": 0.969, "comp_len": 824.1, "dropped_truncated": 0, "gold_loss": 1.5962, "gold_lambda": 0.5, "rep_ratio": 2.4, "t_data_s": 0.0, "t_rollout_s": 63.0, "t_step_s": 146.4, "t_refresh_s": 0.3, "mem_gb": 10.31, "mem_gb_teacher": 20.56}
|
| 41 |
+
{"step": 39, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8038227905917273, "tokens": 187841, "cumulative_loss_tokens": 8530260, "grad_norm": 1.6328125, "lr": 1e-05, "finish_rate": 0.988, "comp_len": 733.8, "dropped_truncated": 0, "gold_loss": 0.3985, "gold_lambda": 0.5, "rep_ratio": 2.407, "t_data_s": 0.0, "t_rollout_s": 53.9, "t_step_s": 129.2, "t_refresh_s": 0.3, "mem_gb": 10.3, "mem_gb_teacher": 20.55}
|
| 42 |
+
{"step": 40, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.886597216392579, "tokens": 225830, "cumulative_loss_tokens": 8756090, "grad_norm": 1.6328125, "lr": 1e-05, "finish_rate": 0.93, "comp_len": 882.1, "dropped_truncated": 0, "gold_loss": 0.3814, "gold_lambda": 0.5, "rep_ratio": 3.385, "t_data_s": 0.0, "t_rollout_s": 65.4, "t_step_s": 152.9, "t_refresh_s": 0.3, "mem_gb": 10.41, "mem_gb_teacher": 20.62}
|
| 43 |
+
{"step": 40, "gsm8k_n": 256, "gsm8k_quick_chat": 0.47265625, "t_eval_s": 79.9}
|
| 44 |
+
{"step": 41, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8627100001349794, "tokens": 193111, "cumulative_loss_tokens": 8949201, "grad_norm": 2.421875, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 754.3, "dropped_truncated": 0, "gold_loss": 1.5705, "gold_lambda": 0.5, "rep_ratio": 2.466, "t_data_s": 0.0, "t_rollout_s": 54.0, "t_step_s": 130.6, "t_refresh_s": 0.3, "mem_gb": 10.18, "mem_gb_teacher": 20.55}
|
| 45 |
+
{"step": 42, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8723323843721557, "tokens": 194522, "cumulative_loss_tokens": 9143723, "grad_norm": 3.125, "lr": 1e-05, "finish_rate": 0.973, "comp_len": 759.9, "dropped_truncated": 0, "gold_loss": 1.5079, "gold_lambda": 0.5, "rep_ratio": 2.692, "t_data_s": 0.0, "t_rollout_s": 55.4, "t_step_s": 130.9, "t_refresh_s": 0.3, "mem_gb": 10.3, "mem_gb_teacher": 20.58}
|
| 46 |
+
{"step": 43, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.844180815874755, "tokens": 201531, "cumulative_loss_tokens": 9345254, "grad_norm": 3.4375, "lr": 1e-05, "finish_rate": 0.957, "comp_len": 787.2, "dropped_truncated": 0, "gold_loss": 1.8123, "gold_lambda": 0.5, "rep_ratio": 2.897, "t_data_s": 0.0, "t_rollout_s": 57.7, "t_step_s": 138.0, "t_refresh_s": 0.3, "mem_gb": 10.22, "mem_gb_teacher": 20.56}
|
| 47 |
+
{"step": 44, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.8408065198205802, "tokens": 199613, "cumulative_loss_tokens": 9544867, "grad_norm": 4.1875, "lr": 1e-05, "finish_rate": 0.949, "comp_len": 779.7, "dropped_truncated": 0, "gold_loss": 1.9436, "gold_lambda": 0.5, "rep_ratio": 2.812, "t_data_s": 0.0, "t_rollout_s": 57.5, "t_step_s": 139.0, "t_refresh_s": 0.3, "mem_gb": 10.42, "mem_gb_teacher": 20.58}
|
healed/opd_warm_keep50/vllm_server.log
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
healed/policy_confirm/chain_seed1226.log
ADDED
|
@@ -0,0 +1,46 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2026-07-14T14:22:51-07:00 phase1: waiting for tmux policy-on-rkl-1225 to end
|
| 2 |
+
2026-07-14T14:37:51-07:00 GPU GPU-8ca70870-ddf2-d274-bcc5-182c2075bced released (33 MiB)
|
| 3 |
+
2026-07-14T14:37:51-07:00 phase1: starting off-policy seed 1226 on GPU-8ca70870-ddf2-d274-bcc5-182c2075bced
|
| 4 |
+
2026-07-14T15:15:49-07:00 phase1: validated outputs/healed/policy_confirm/off_forward_seed1226/step0050
|
| 5 |
+
2026-07-14T15:15:49-07:00 GPU GPU-8ca70870-ddf2-d274-bcc5-182c2075bced released (33 MiB)
|
| 6 |
+
{
|
| 7 |
+
"correct": 850,
|
| 8 |
+
"accuracy": 0.6444275966641395,
|
| 9 |
+
"finished": 1294,
|
| 10 |
+
"finish_rate": 0.9810462471569371,
|
| 11 |
+
"mean_completion_tokens": 115.54738438210765
|
| 12 |
+
}
|
| 13 |
+
saved item-level results -> outputs/evals/policy_confirm/off_forward_seed1226.json
|
| 14 |
+
2026-07-14T15:17:37-07:00 phase1 complete -> outputs/evals/policy_confirm/off_forward_seed1226.json
|
| 15 |
+
2026-07-14T15:17:37-07:00 phase2: waiting for tmux policy-pair-1225 to end
|
| 16 |
+
2026-07-14T15:17:37-07:00 GPU GPU-8ca70870-ddf2-d274-bcc5-182c2075bced released (33 MiB)
|
| 17 |
+
2026-07-14T15:17:37-07:00 GPU GPU-a6acf07f-31f5-618f-a5d0-c0017e7e2e27 released (46 MiB)
|
| 18 |
+
2026-07-14T15:17:37-07:00 GPU GPU-864c54df-0130-7780-e271-8a5551d1733f released (15 MiB)
|
| 19 |
+
2026-07-14T15:17:37-07:00 phase2: launching on-policy seed 1226 (teacher GPU-8ca70870-ddf2-d274-bcc5-182c2075bced, student GPU-a6acf07f-31f5-618f-a5d0-c0017e7e2e27, vLLM GPU-864c54df-0130-7780-e271-8a5551d1733f)
|
| 20 |
+
2026-07-14T16:37:38-07:00 phase2: validated outputs/healed/policy_confirm/on_reverse_seed1226/step0050
|
| 21 |
+
2026-07-14T16:37:38-07:00 GPU GPU-864c54df-0130-7780-e271-8a5551d1733f released (15 MiB)
|
| 22 |
+
{
|
| 23 |
+
"correct": 852,
|
| 24 |
+
"accuracy": 0.6459438968915845,
|
| 25 |
+
"finished": 1259,
|
| 26 |
+
"finish_rate": 0.954510993176649,
|
| 27 |
+
"mean_completion_tokens": 119.7407126611069
|
| 28 |
+
}
|
| 29 |
+
saved item-level results -> outputs/evals/policy_confirm/on_reverse_seed1226.json
|
| 30 |
+
{
|
| 31 |
+
"on_policy": "on_reverse_seed1226",
|
| 32 |
+
"off_policy": "off_forward_seed1226",
|
| 33 |
+
"frame": "chat",
|
| 34 |
+
"n": 1319,
|
| 35 |
+
"on_accuracy": 0.6459438968915845,
|
| 36 |
+
"off_accuracy": 0.6444275966641395,
|
| 37 |
+
"off_minus_on": -0.001516300227445034,
|
| 38 |
+
"paired_counts": {
|
| 39 |
+
"both_correct": 755,
|
| 40 |
+
"on_only": 97,
|
| 41 |
+
"off_only": 95,
|
| 42 |
+
"both_wrong": 372
|
| 43 |
+
},
|
| 44 |
+
"mcnemar_exact_p": 0.9424925696532238
|
| 45 |
+
}
|
| 46 |
+
2026-07-14T16:39:29-07:00 seed 1226 pair complete -> outputs/evals/policy_confirm/pair_seed1226.json
|
healed/policy_confirm/combo.log
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2026-07-14T14:58:20-07:00 waiting for tmux policy-chain-1226 to end (confirmatory set first)
|
| 2 |
+
2026-07-14T16:40:21-07:00 GPU GPU-8ca70870-ddf2-d274-bcc5-182c2075bced released (33 MiB)
|
| 3 |
+
2026-07-14T16:40:21-07:00 GPU GPU-a6acf07f-31f5-618f-a5d0-c0017e7e2e27 released (46 MiB)
|
| 4 |
+
2026-07-14T16:40:21-07:00 GPU GPU-864c54df-0130-7780-e271-8a5551d1733f released (15 MiB)
|
| 5 |
+
2026-07-14T16:40:21-07:00 combo phase1: off-policy 25 steps, seed 1225 on GPU-864c54df-0130-7780-e271-8a5551d1733f
|
| 6 |
+
2026-07-14T16:40:21-07:00 combo phase1: off-policy 25 steps, seed 1224 on GPU-8ca70870-ddf2-d274-bcc5-182c2075bced
|
| 7 |
+
2026-07-14T16:58:18-07:00 combo phase1 validated: outputs/healed/policy_confirm/combo_off25_seed1225/step0025
|
| 8 |
+
2026-07-14T16:59:26-07:00 combo phase1 validated: outputs/healed/policy_confirm/combo_off25_seed1224/step0025
|
| 9 |
+
2026-07-14T16:59:26-07:00 GPU GPU-8ca70870-ddf2-d274-bcc5-182c2075bced released (33 MiB)
|
| 10 |
+
2026-07-14T16:59:26-07:00 combo phase1: off-policy 25 steps, seed 1226 on GPU-8ca70870-ddf2-d274-bcc5-182c2075bced
|
| 11 |
+
2026-07-14T17:16:54-07:00 combo phase1 validated: outputs/healed/policy_confirm/combo_off25_seed1226/step0025
|
| 12 |
+
2026-07-14T17:16:54-07:00 GPU GPU-8ca70870-ddf2-d274-bcc5-182c2075bced released (33 MiB)
|
| 13 |
+
2026-07-14T17:16:54-07:00 GPU GPU-864c54df-0130-7780-e271-8a5551d1733f released (15 MiB)
|
| 14 |
+
2026-07-14T17:16:54-07:00 combo phase2: on-policy steps 26-50, seed 1224
|
| 15 |
+
2026-07-14T17:57:46-07:00 combo phase2 validated: outputs/healed/policy_confirm/combo_on25_seed1224/step0050
|
| 16 |
+
2026-07-14T17:57:46-07:00 GPU GPU-8ca70870-ddf2-d274-bcc5-182c2075bced released (33 MiB)
|
| 17 |
+
2026-07-14T17:57:46-07:00 GPU GPU-a6acf07f-31f5-618f-a5d0-c0017e7e2e27 released (46 MiB)
|
| 18 |
+
2026-07-14T17:57:46-07:00 GPU GPU-864c54df-0130-7780-e271-8a5551d1733f released (15 MiB)
|
| 19 |
+
2026-07-14T17:59:35-07:00 combo seed 1224 complete -> outputs/evals/policy_confirm/combo_off25on25_seed1224.json
|
| 20 |
+
2026-07-14T17:59:35-07:00 GPU GPU-864c54df-0130-7780-e271-8a5551d1733f released (15 MiB)
|
| 21 |
+
2026-07-14T17:59:35-07:00 combo phase2: on-policy steps 26-50, seed 1225
|
| 22 |
+
2026-07-14T18:40:52-07:00 combo phase2 validated: outputs/healed/policy_confirm/combo_on25_seed1225/step0050
|
| 23 |
+
2026-07-14T18:40:52-07:00 GPU GPU-8ca70870-ddf2-d274-bcc5-182c2075bced released (33 MiB)
|
| 24 |
+
2026-07-14T18:40:52-07:00 GPU GPU-a6acf07f-31f5-618f-a5d0-c0017e7e2e27 released (46 MiB)
|
| 25 |
+
2026-07-14T18:40:52-07:00 GPU GPU-864c54df-0130-7780-e271-8a5551d1733f released (15 MiB)
|
| 26 |
+
2026-07-14T18:42:42-07:00 combo seed 1225 complete -> outputs/evals/policy_confirm/combo_off25on25_seed1225.json
|
| 27 |
+
2026-07-14T18:42:42-07:00 GPU GPU-864c54df-0130-7780-e271-8a5551d1733f released (15 MiB)
|
| 28 |
+
2026-07-14T18:42:42-07:00 combo phase2: on-policy steps 26-50, seed 1226
|
| 29 |
+
2026-07-14T19:24:14-07:00 combo phase2 validated: outputs/healed/policy_confirm/combo_on25_seed1226/step0050
|
| 30 |
+
2026-07-14T19:24:14-07:00 GPU GPU-8ca70870-ddf2-d274-bcc5-182c2075bced released (33 MiB)
|
| 31 |
+
2026-07-14T19:24:14-07:00 GPU GPU-a6acf07f-31f5-618f-a5d0-c0017e7e2e27 released (46 MiB)
|
| 32 |
+
2026-07-14T19:24:14-07:00 GPU GPU-864c54df-0130-7780-e271-8a5551d1733f released (15 MiB)
|
| 33 |
+
2026-07-14T19:26:03-07:00 combo seed 1226 complete -> outputs/evals/policy_confirm/combo_off25on25_seed1226.json
|
| 34 |
+
2026-07-14T19:26:03-07:00 GPU GPU-864c54df-0130-7780-e271-8a5551d1733f released (15 MiB)
|
| 35 |
+
2026-07-14T19:26:03-07:00 combo arm complete for all seeds
|
healed/policy_confirm/combo_off25_seed1224.console.log
ADDED
|
@@ -0,0 +1,75 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 6 |
+
wandb: Run data is saved locally in outputs/healed/policy_confirm/combo_off25_seed1224/wandb/run-20260714_164028-fnb5m9as
|
| 7 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 8 |
+
wandb: Syncing run policy-combo-off25-seed1224
|
| 9 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-heal
|
| 10 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-heal/runs/fnb5m9as
|
| 11 |
+
|
| 12 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 25 total steps | student params 3.70B | teacher overlap=False
|
| 13 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2185505145077904, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 4.65625, "lr": 6e-06, "finish_rate": 0.907, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.5, "frames": {"chat": 236}, "mem_gb": 15.78}
|
| 14 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 15 |
+
[eval step 1] sample: 'To solve the given system of equations:\n\\[\n\\begin{align*}\na + b &= k, \\\\\nk + m &= p, \\\\\np + a &= r, \\\\\nb + m + r &= 18,\n\\end{align*}\n\\]\nwe need to determine the values'
|
| 16 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2723684337724, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 4.84375, "lr": 9e-06, "finish_rate": 0.781, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.9, "frames": {"chat": 215}, "mem_gb": 16.05}
|
| 17 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.27742008934554957, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 4.0625, "lr": 1.2e-05, "finish_rate": 0.825, "comp_len": 553.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.0, "frames": {"chat": 217}, "mem_gb": 15.93}
|
| 18 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.21144348942749203, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 2.484375, "lr": 1.5e-05, "finish_rate": 0.8, "comp_len": 585.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.8, "frames": {"chat": 205}, "mem_gb": 15.99}
|
| 19 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13916345189710458, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 1.59375, "lr": 1.8e-05, "finish_rate": 0.834, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.1, "frames": {"chat": 229}, "mem_gb": 15.96}
|
| 20 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2522294740737726, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 2.5, "lr": 2.1e-05, "finish_rate": 0.812, "comp_len": 538.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.5, "frames": {"chat": 223}, "mem_gb": 16.03}
|
| 21 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1405846708993738, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 1.1875, "lr": 2.4e-05, "finish_rate": 0.708, "comp_len": 594.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.8, "frames": {"chat": 202}, "mem_gb": 16.07}
|
| 22 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.16176770092062653, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 1.1640625, "lr": 2.7000000000000002e-05, "finish_rate": 0.77, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.8, "frames": {"chat": 209}, "mem_gb": 16.04}
|
| 23 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10622600217446064, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 0.68359375, "lr": 3e-05, "finish_rate": 0.885, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.5, "frames": {"chat": 227}, "mem_gb": 16.02}
|
| 24 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11151557442427923, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.6640625, "lr": 3e-05, "finish_rate": 0.848, "comp_len": 521.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.9, "frames": {"chat": 230}, "mem_gb": 16.09}
|
| 25 |
+
[eval step 10] sample: "To solve this system of equations, we need to determine the values of \\(a\\), \\(b\\), \\(k\\), \\(m\\), and \\(p\\) given the constraints that each letter represents a non-zero digit.\n\nLet's break down the pr"
|
| 26 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09873726044449334, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.75390625, "lr": 3e-05, "finish_rate": 0.879, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 231}, "mem_gb": 15.94}
|
| 27 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09630266562933103, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.882, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.6, "frames": {"chat": 245}, "mem_gb": 16.02}
|
| 28 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09670859327483922, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.5703125, "lr": 3e-05, "finish_rate": 0.81, "comp_len": 571.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.6, "frames": {"chat": 210}, "mem_gb": 16.02}
|
| 29 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11307675357162952, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.578125, "lr": 3e-05, "finish_rate": 0.758, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.0, "frames": {"chat": 211}, "mem_gb": 16.02}
|
| 30 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09364407028704881, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.455078125, "lr": 3e-05, "finish_rate": 0.819, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.9, "frames": {"chat": 221}, "mem_gb": 16.08}
|
| 31 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08135358470983338, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.484375, "lr": 3e-05, "finish_rate": 0.912, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.0, "frames": {"chat": 250}, "mem_gb": 15.89}
|
| 32 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08961299906177446, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.50390625, "lr": 3e-05, "finish_rate": 0.79, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 50.0, "frames": {"chat": 229}, "mem_gb": 16.06}
|
| 33 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06862492963150144, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.404296875, "lr": 3e-05, "finish_rate": 0.888, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 47.4, "frames": {"chat": 250}, "mem_gb": 16.04}
|
| 34 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08355683204904199, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.423828125, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 519.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 231}, "mem_gb": 15.91}
|
| 35 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0749407821027562, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.419921875, "lr": 3e-05, "finish_rate": 0.844, "comp_len": 535.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 48.3, "frames": {"chat": 224}, "mem_gb": 15.96}
|
| 36 |
+
[eval step 20] sample: 'To solve the system of equations given:\n\n\\[\n\\begin{align*}\na + b &= k \\\\\nk + m &= p \\\\\np + a &= r \\\\\nb + m + r &= 18\n\\end{align*}\n\\]\n\nwe need to determine the values of \\('
|
| 37 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08050858488449206, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.484375, "lr": 3e-05, "finish_rate": 0.802, "comp_len": 566.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 212}, "mem_gb": 16.0}
|
| 38 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0722364233136798, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.427734375, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 46.9, "frames": {"chat": 238}, "mem_gb": 15.95}
|
| 39 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07254953771786143, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.4453125, "lr": 3e-05, "finish_rate": 0.903, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 51.0, "frames": {"chat": 257}, "mem_gb": 15.83}
|
| 40 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06550259976585707, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.359375, "lr": 3e-05, "finish_rate": 0.868, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.7, "frames": {"chat": 227}, "mem_gb": 16.02}
|
| 41 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07561354989591054, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.838, "comp_len": 526.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 228}, "mem_gb": 16.05}
|
| 42 |
+
checkpoint snapshot queued -> outputs/healed/policy_confirm/combo_off25_seed1224/step0025
|
| 43 |
+
wandb: updating run metadata
|
| 44 |
+
wandb: uploading summary, console lines 31-31
|
| 45 |
+
wandb:
|
| 46 |
+
wandb: Run history:
|
| 47 |
+
wandb: comp_len ββββββ
βββββββββ
βββββ
βββββ
|
| 48 |
+
wandb: cumulative_loss_tokens βββββββββββββ
β
β
β
βββββββββ
|
| 49 |
+
wandb: epoch βββββββββββββββββββββββββ
|
| 50 |
+
wandb: finish_rate βββ
ββ
β
βββββββ
ββ
ββββββββββ
|
| 51 |
+
wandb: forward_topk_kl βββββββββββββββββββββββββ
|
| 52 |
+
wandb: grad_norm βββββββββββββββββββββββββ
|
| 53 |
+
wandb: lr βββββ
β
βββββββββββββββββββ
|
| 54 |
+
wandb: mem_gb βββββ
ββββββ
βββββββββ
ββ
βββ
|
| 55 |
+
wandb: step βββββββββββββ
β
β
β
βββββββββ
|
| 56 |
+
wandb: t_data_s βββββββββββββββββββββββββ
|
| 57 |
+
wandb: +3 ...
|
| 58 |
+
wandb:
|
| 59 |
+
wandb: Run summary:
|
| 60 |
+
wandb: comp_len 526.3
|
| 61 |
+
wandb: cumulative_loss_tokens 3000000
|
| 62 |
+
wandb: epoch 0
|
| 63 |
+
wandb: finish_rate 0.838
|
| 64 |
+
wandb: forward_topk_kl 0.07561
|
| 65 |
+
wandb: grad_norm 0.38086
|
| 66 |
+
wandb: lr 3e-05
|
| 67 |
+
wandb: mem_gb 16.05
|
| 68 |
+
wandb: step 25
|
| 69 |
+
wandb: t_data_s 0
|
| 70 |
+
wandb: +4 ...
|
| 71 |
+
wandb:
|
| 72 |
+
wandb: π View run policy-combo-off25-seed1224 at: https://wandb.ai/hbfreed/glean-heal/runs/fnb5m9as
|
| 73 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-heal
|
| 74 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 75 |
+
wandb: Find logs at: outputs/healed/policy_confirm/combo_off25_seed1224/wandb/run-20260714_164028-fnb5m9as/logs
|
healed/policy_confirm/combo_off25_seed1225.console.log
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run 5fsq79h5
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/policy_confirm/combo_off25_seed1225/wandb/run-20260714_164028-5fsq79h5
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run policy-combo-off25-seed1225
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-heal
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-heal/runs/5fsq79h5
|
| 12 |
+
|
| 13 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 25 total steps | student params 3.70B | teacher overlap=False
|
| 14 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2507890793694804, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 4.65625, "lr": 6e-06, "finish_rate": 0.733, "comp_len": 628.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.4, "frames": {"chat": 191}, "mem_gb": 15.94}
|
| 15 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 16 |
+
[eval step 1] sample: 'To solve this problem, we need to arrange the numbers from 1 to 49 in a spiral pattern on a square grid and identify the four shaded numbers that lie on the same diagonal as the number 7. Finally, we '
|
| 17 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2288356147143369, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 4.28125, "lr": 9e-06, "finish_rate": 0.845, "comp_len": 547.9, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 39.2, "frames": {"chat": 219}, "mem_gb": 16.04}
|
| 18 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.28311064375787975, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 4.125, "lr": 1.2e-05, "finish_rate": 0.778, "comp_len": 579.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.6, "frames": {"chat": 207}, "mem_gb": 16.05}
|
| 19 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.23871733199569087, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 2.421875, "lr": 1.5e-05, "finish_rate": 0.755, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.5, "frames": {"chat": 208}, "mem_gb": 16.01}
|
| 20 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1616144200786948, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 1.6484375, "lr": 1.8e-05, "finish_rate": 0.799, "comp_len": 547.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.2, "frames": {"chat": 219}, "mem_gb": 16.04}
|
| 21 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13260349893815196, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 1.3046875, "lr": 2.1e-05, "finish_rate": 0.915, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.4, "frames": {"chat": 246}, "mem_gb": 15.92}
|
| 22 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13819025703755516, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 1.0078125, "lr": 2.4e-05, "finish_rate": 0.704, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.8, "frames": {"chat": 206}, "mem_gb": 16.07}
|
| 23 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.1139759704611885, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 0.796875, "lr": 2.7000000000000002e-05, "finish_rate": 0.876, "comp_len": 515.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.3, "frames": {"chat": 233}, "mem_gb": 16.05}
|
| 24 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12906688051999857, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 0.94140625, "lr": 3e-05, "finish_rate": 0.847, "comp_len": 524.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.2, "frames": {"chat": 229}, "mem_gb": 15.92}
|
| 25 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0961639422957475, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.62109375, "lr": 3e-05, "finish_rate": 0.864, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.7, "frames": {"chat": 236}, "mem_gb": 15.95}
|
| 26 |
+
[eval step 10] sample: "To solve this problem, we need to identify the four numbers on the diagonal of the grid that contain the number \\(7\\) and determine how many of these numbers are prime.\n\nHere's a step-by-step approach"
|
| 27 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10284294348070398, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.57421875, "lr": 3e-05, "finish_rate": 0.87, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.4, "frames": {"chat": 239}, "mem_gb": 15.84}
|
| 28 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08901896035398046, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.63671875, "lr": 3e-05, "finish_rate": 0.867, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 241}, "mem_gb": 15.96}
|
| 29 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09528991474263991, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.5390625, "lr": 3e-05, "finish_rate": 0.863, "comp_len": 531.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 226}, "mem_gb": 15.92}
|
| 30 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08106174418615798, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.453125, "lr": 3e-05, "finish_rate": 0.893, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.1, "frames": {"chat": 234}, "mem_gb": 16.05}
|
| 31 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07995114640053362, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.43359375, "lr": 3e-05, "finish_rate": 0.914, "comp_len": 466.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.9, "frames": {"chat": 257}, "mem_gb": 16.04}
|
| 32 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.11156130492898325, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.5625, "lr": 3e-05, "finish_rate": 0.76, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.2, "frames": {"chat": 208}, "mem_gb": 16.1}
|
| 33 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10080602069403977, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.53125, "lr": 3e-05, "finish_rate": 0.763, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.5, "frames": {"chat": 211}, "mem_gb": 16.07}
|
| 34 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08554738140531505, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.4296875, "lr": 3e-05, "finish_rate": 0.806, "comp_len": 528.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.2, "frames": {"chat": 227}, "mem_gb": 16.05}
|
| 35 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09060717803711693, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.462890625, "lr": 3e-05, "finish_rate": 0.796, "comp_len": 568.7, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 211}, "mem_gb": 16.03}
|
| 36 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07435101296724751, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.384765625, "lr": 3e-05, "finish_rate": 0.861, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.1, "frames": {"chat": 238}, "mem_gb": 16.04}
|
| 37 |
+
[eval step 20] sample: 'To solve this problem, we need to identify the four numbers on the diagonal from the center of the spiral pattern that contain the number \\(7\\). Then, we will determine how many of these four numbers '
|
| 38 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07675727263481046, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.400390625, "lr": 3e-05, "finish_rate": 0.835, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 43.6, "frames": {"chat": 237}, "mem_gb": 16.08}
|
| 39 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08794694169480353, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.416015625, "lr": 3e-05, "finish_rate": 0.721, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.3, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 40 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07553549966296802, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.376953125, "lr": 3e-05, "finish_rate": 0.801, "comp_len": 543.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.3, "frames": {"chat": 221}, "mem_gb": 16.17}
|
| 41 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.0735723745985888, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.37890625, "lr": 3e-05, "finish_rate": 0.853, "comp_len": 517.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 42.5, "frames": {"chat": 232}, "mem_gb": 16.01}
|
| 42 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07792395336097106, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.380859375, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.2, "frames": {"chat": 208}, "mem_gb": 16.04}
|
| 43 |
+
checkpoint snapshot queued -> outputs/healed/policy_confirm/combo_off25_seed1225/step0025
|
| 44 |
+
wandb: updating run metadata
|
| 45 |
+
wandb: uploading output.log; uploading wandb-summary.json; uploading config.yaml; uploading summary, console lines 31-31
|
| 46 |
+
wandb: uploading data
|
| 47 |
+
wandb:
|
| 48 |
+
wandb: Run history:
|
| 49 |
+
wandb: comp_len ββ
βββ
ββββββββββββ
ββ
ββββββ
|
| 50 |
+
wandb: cumulative_loss_tokens βββββββββββββ
β
β
β
βββββββββ
|
| 51 |
+
wandb: epoch βββββββββββββββββββββββββ
|
| 52 |
+
wandb: finish_rate βββββββββββββββββββββ
ββββ
|
| 53 |
+
wandb: forward_topk_kl βββββββββββββββββββββββββ
|
| 54 |
+
wandb: grad_norm βββββββββββββββββββββββββ
|
| 55 |
+
wandb: lr βββββ
β
βββββββββββββββββββ
|
| 56 |
+
wandb: mem_gb ββ
β
β
β
βββ
ββββββ
β
βββ
β
β
ββββ
β
|
| 57 |
+
wandb: step βββββββββββββ
β
β
β
βββββββββ
|
| 58 |
+
wandb: t_data_s βββββββββββββββββββββββββ
|
| 59 |
+
wandb: +3 ...
|
| 60 |
+
wandb:
|
| 61 |
+
wandb: Run summary:
|
| 62 |
+
wandb: comp_len 576.9
|
| 63 |
+
wandb: cumulative_loss_tokens 3000000
|
| 64 |
+
wandb: epoch 0
|
| 65 |
+
wandb: finish_rate 0.764
|
| 66 |
+
wandb: forward_topk_kl 0.07792
|
| 67 |
+
wandb: grad_norm 0.38086
|
| 68 |
+
wandb: lr 3e-05
|
| 69 |
+
wandb: mem_gb 16.04
|
| 70 |
+
wandb: step 25
|
| 71 |
+
wandb: t_data_s 0
|
| 72 |
+
wandb: +4 ...
|
| 73 |
+
wandb:
|
| 74 |
+
wandb: π View run policy-combo-off25-seed1225 at: https://wandb.ai/hbfreed/glean-heal/runs/5fsq79h5
|
| 75 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-heal
|
| 76 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 77 |
+
wandb: Find logs at: outputs/healed/policy_confirm/combo_off25_seed1225/wandb/run-20260714_164028-5fsq79h5/logs
|
healed/policy_confirm/combo_off25_seed1226.console.log
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: setting up run jh6zgcxt
|
| 6 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 7 |
+
wandb: Run data is saved locally in outputs/healed/policy_confirm/combo_off25_seed1226/wandb/run-20260714_165932-jh6zgcxt
|
| 8 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 9 |
+
wandb: Syncing run policy-combo-off25-seed1226
|
| 10 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-heal
|
| 11 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-heal/runs/jh6zgcxt
|
| 12 |
+
|
| 13 |
+
12115 cached top-128 chat trajectories / 6,476,634 unique tokens | 53 steps/epoch | 25 total steps | student params 3.70B | teacher overlap=False
|
| 14 |
+
{"step": 1, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2265280557061235, "tokens": 120000, "cumulative_loss_tokens": 120000, "grad_norm": 4.9375, "lr": 6e-06, "finish_rate": 0.902, "comp_len": 472.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 49.1, "frames": {"chat": 254}, "mem_gb": 15.82}
|
| 15 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 16 |
+
[eval step 1] sample: 'To solve this problem, we need to understand the properties of the triangle and the segments formed by connecting the midpoints of its sides.\n\n1. **Understanding the Problem:**\n - The perimeter of t'
|
| 17 |
+
{"step": 2, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.2571202766161412, "tokens": 120000, "cumulative_loss_tokens": 240000, "grad_norm": 4.5625, "lr": 9e-06, "finish_rate": 0.876, "comp_len": 497.9, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 41.3, "frames": {"chat": 241}, "mem_gb": 16.03}
|
| 18 |
+
{"step": 3, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.24384070302322508, "tokens": 120000, "cumulative_loss_tokens": 360000, "grad_norm": 3.6875, "lr": 1.2e-05, "finish_rate": 0.746, "comp_len": 563.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.2, "frames": {"chat": 213}, "mem_gb": 16.05}
|
| 19 |
+
{"step": 4, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.18487517377516877, "tokens": 120000, "cumulative_loss_tokens": 480000, "grad_norm": 2.21875, "lr": 1.5e-05, "finish_rate": 0.864, "comp_len": 543.0, "t_data_s": 0.1, "t_rollout_s": 0.0, "t_step_s": 39.6, "frames": {"chat": 221}, "mem_gb": 16.1}
|
| 20 |
+
{"step": 5, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.19795994342279932, "tokens": 120000, "cumulative_loss_tokens": 600000, "grad_norm": 1.859375, "lr": 1.8e-05, "finish_rate": 0.745, "comp_len": 612.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.6, "frames": {"chat": 196}, "mem_gb": 16.06}
|
| 21 |
+
{"step": 6, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12876303391549737, "tokens": 120000, "cumulative_loss_tokens": 720000, "grad_norm": 1.359375, "lr": 2.1e-05, "finish_rate": 0.926, "comp_len": 444.4, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 44.3, "frames": {"chat": 270}, "mem_gb": 15.86}
|
| 22 |
+
{"step": 7, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.13868847138527782, "tokens": 120000, "cumulative_loss_tokens": 840000, "grad_norm": 1.171875, "lr": 2.4e-05, "finish_rate": 0.815, "comp_len": 555.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.9, "frames": {"chat": 216}, "mem_gb": 16.04}
|
| 23 |
+
{"step": 8, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.12091609455912064, "tokens": 120000, "cumulative_loss_tokens": 960000, "grad_norm": 0.71875, "lr": 2.7000000000000002e-05, "finish_rate": 0.775, "comp_len": 600.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.1, "frames": {"chat": 200}, "mem_gb": 16.01}
|
| 24 |
+
{"step": 9, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10763629353487243, "tokens": 120000, "cumulative_loss_tokens": 1080000, "grad_norm": 0.75390625, "lr": 3e-05, "finish_rate": 0.767, "comp_len": 582.5, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.2, "frames": {"chat": 206}, "mem_gb": 15.96}
|
| 25 |
+
{"step": 10, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08658104830309749, "tokens": 120000, "cumulative_loss_tokens": 1200000, "grad_norm": 0.60546875, "lr": 3e-05, "finish_rate": 0.902, "comp_len": 512.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 39.3, "frames": {"chat": 234}, "mem_gb": 15.99}
|
| 26 |
+
[eval step 10] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Geometry:**\n - The perimeter of the original triangle is 28.\n "
|
| 27 |
+
{"step": 11, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09525315140672028, "tokens": 120000, "cumulative_loss_tokens": 1320000, "grad_norm": 0.55078125, "lr": 3e-05, "finish_rate": 0.823, "comp_len": 558.1, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.8, "frames": {"chat": 215}, "mem_gb": 16.01}
|
| 28 |
+
{"step": 12, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07842937783297772, "tokens": 120000, "cumulative_loss_tokens": 1440000, "grad_norm": 0.515625, "lr": 3e-05, "finish_rate": 0.922, "comp_len": 470.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.5, "frames": {"chat": 255}, "mem_gb": 15.99}
|
| 29 |
+
{"step": 13, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08281613596221432, "tokens": 120000, "cumulative_loss_tokens": 1560000, "grad_norm": 0.51171875, "lr": 3e-05, "finish_rate": 0.892, "comp_len": 480.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.6, "frames": {"chat": 250}, "mem_gb": 15.87}
|
| 30 |
+
{"step": 14, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08376936465194448, "tokens": 120000, "cumulative_loss_tokens": 1680000, "grad_norm": 0.478515625, "lr": 3e-05, "finish_rate": 0.884, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.0, "frames": {"chat": 242}, "mem_gb": 16.04}
|
| 31 |
+
{"step": 15, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09975259796567261, "tokens": 120000, "cumulative_loss_tokens": 1800000, "grad_norm": 0.53515625, "lr": 3e-05, "finish_rate": 0.729, "comp_len": 603.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.7, "frames": {"chat": 199}, "mem_gb": 16.05}
|
| 32 |
+
{"step": 16, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.15622244120972853, "tokens": 120000, "cumulative_loss_tokens": 1920000, "grad_norm": 0.96484375, "lr": 3e-05, "finish_rate": 0.784, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.5, "frames": {"chat": 208}, "mem_gb": 16.08}
|
| 33 |
+
{"step": 17, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.09498789354876305, "tokens": 120000, "cumulative_loss_tokens": 2040000, "grad_norm": 0.5703125, "lr": 3e-05, "finish_rate": 0.764, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.5, "frames": {"chat": 208}, "mem_gb": 16.02}
|
| 34 |
+
{"step": 18, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10344938092194497, "tokens": 120000, "cumulative_loss_tokens": 2160000, "grad_norm": 0.52734375, "lr": 3e-05, "finish_rate": 0.732, "comp_len": 574.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.4, "frames": {"chat": 209}, "mem_gb": 16.17}
|
| 35 |
+
{"step": 19, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07368671357665832, "tokens": 120000, "cumulative_loss_tokens": 2280000, "grad_norm": 0.392578125, "lr": 3e-05, "finish_rate": 0.855, "comp_len": 510.6, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.0, "frames": {"chat": 235}, "mem_gb": 16.0}
|
| 36 |
+
{"step": 20, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08098533270259699, "tokens": 120000, "cumulative_loss_tokens": 2400000, "grad_norm": 0.431640625, "lr": 3e-05, "finish_rate": 0.74, "comp_len": 588.2, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 37.9, "frames": {"chat": 204}, "mem_gb": 16.0}
|
| 37 |
+
[eval step 20] sample: "To solve this problem, we need to understand the geometric properties involved. Here's a step-by-step breakdown:\n\n1. **Understand the Problem:**\n - The perimeter of the original triangle is 28.\n -"
|
| 38 |
+
{"step": 21, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.10646473311164106, "tokens": 120000, "cumulative_loss_tokens": 2520000, "grad_norm": 0.60546875, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.6, "frames": {"chat": 208}, "mem_gb": 16.05}
|
| 39 |
+
{"step": 22, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07540695022086924, "tokens": 120000, "cumulative_loss_tokens": 2640000, "grad_norm": 0.390625, "lr": 3e-05, "finish_rate": 0.825, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 41.9, "frames": {"chat": 240}, "mem_gb": 16.05}
|
| 40 |
+
{"step": 23, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.06904787591006607, "tokens": 120000, "cumulative_loss_tokens": 2760000, "grad_norm": 0.396484375, "lr": 3e-05, "finish_rate": 0.89, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.8, "frames": {"chat": 246}, "mem_gb": 16.04}
|
| 41 |
+
{"step": 24, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.07999428922263906, "tokens": 120000, "cumulative_loss_tokens": 2880000, "grad_norm": 0.5703125, "lr": 3e-05, "finish_rate": 0.909, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 40.3, "frames": {"chat": 243}, "mem_gb": 15.86}
|
| 42 |
+
{"step": 25, "epoch": 0, "training_mode": "off-policy", "forward_topk_kl": 0.08892722107529019, "tokens": 120000, "cumulative_loss_tokens": 3000000, "grad_norm": 0.44140625, "lr": 3e-05, "finish_rate": 0.745, "comp_len": 576.9, "t_data_s": 0.0, "t_rollout_s": 0.0, "t_step_s": 38.0, "frames": {"chat": 208}, "mem_gb": 16.06}
|
| 43 |
+
checkpoint snapshot queued -> outputs/healed/policy_confirm/combo_off25_seed1226/step0025
|
| 44 |
+
wandb: updating run metadata
|
| 45 |
+
wandb: uploading config.yaml; uploading output.log; uploading wandb-summary.json
|
| 46 |
+
wandb: uploading summary
|
| 47 |
+
wandb:
|
| 48 |
+
wandb: Run history:
|
| 49 |
+
wandb: comp_len ββββ
βββββββββββββββββββββ
|
| 50 |
+
wandb: cumulative_loss_tokens βββββββββββββ
β
β
β
βββββββββ
|
| 51 |
+
wandb: epoch βββββββββββββββββββββββββ
|
| 52 |
+
wandb: finish_rate βββββββββββββββββββ
ββββββ
|
| 53 |
+
wandb: forward_topk_kl ββββ
βββββββββββββββββββββ
|
| 54 |
+
wandb: grad_norm βββββββββββββββββββββββββ
|
| 55 |
+
wandb: lr βββββ
β
βββββββββββββββββββ
|
| 56 |
+
wandb: mem_gb ββ
βββββ
β
βββ
βββ
βββ
ββ
β
βββ
ββ
|
| 57 |
+
wandb: step βββββββββββββ
β
β
β
βββββββββ
|
| 58 |
+
wandb: t_data_s βββββββββββββββββββββββββ
|
| 59 |
+
wandb: +3 ...
|
| 60 |
+
wandb:
|
| 61 |
+
wandb: Run summary:
|
| 62 |
+
wandb: comp_len 576.9
|
| 63 |
+
wandb: cumulative_loss_tokens 3000000
|
| 64 |
+
wandb: epoch 0
|
| 65 |
+
wandb: finish_rate 0.745
|
| 66 |
+
wandb: forward_topk_kl 0.08893
|
| 67 |
+
wandb: grad_norm 0.44141
|
| 68 |
+
wandb: lr 3e-05
|
| 69 |
+
wandb: mem_gb 16.06
|
| 70 |
+
wandb: step 25
|
| 71 |
+
wandb: t_data_s 0
|
| 72 |
+
wandb: +4 ...
|
| 73 |
+
wandb:
|
| 74 |
+
wandb: π View run policy-combo-off25-seed1226 at: https://wandb.ai/hbfreed/glean-heal/runs/jh6zgcxt
|
| 75 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-heal
|
| 76 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 77 |
+
wandb: Find logs at: outputs/healed/policy_confirm/combo_off25_seed1226/wandb/run-20260714_165932-jh6zgcxt/logs
|
healed/policy_confirm/combo_on25_seed1224.console.log
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
|
| 2 |
+
warnings.warn('Grouped GEMM not available.')
|
| 3 |
+
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
|
| 4 |
+
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
|
| 5 |
+
wandb: Tracking run with wandb version 0.28.0
|
| 6 |
+
wandb: Run data is saved locally in outputs/healed/policy_confirm/combo_on25_seed1224/wandb/run-20260714_171701-62dayi8c
|
| 7 |
+
wandb: Run `wandb offline` to turn off syncing.
|
| 8 |
+
wandb: Syncing run policy-combo-on25-seed1224
|
| 9 |
+
wandb: βοΈ View project at https://wandb.ai/hbfreed/glean-heal
|
| 10 |
+
wandb: π View run at https://wandb.ai/hbfreed/glean-heal/runs/62dayi8c
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
resumed student weights from outputs/healed/policy_confirm/combo_off25_seed1224/step0025 (fresh optimizer, step counter at 0)
|
| 14 |
+
starting vllm rollout server on GPU GPU-864c54df-0130-7780-e271-8a5551d1733f (port 8377) ...
|
| 15 |
+
vllm server healthy in 26s
|
| 16 |
+
dataset_source filter ['omega', 'polaris', 'orz_math', 'mathsub', 'dapo-math'] -> 63998 prompts
|
| 17 |
+
|
| 18 |
+
difficulty <= 4 -> 63998 prompts
|
| 19 |
+
WARNING: difficulty filter removed nothing β the slice likely carries difficulty=None (Dolci math sources do), so the teacher-competence guard is NOT in effect
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
63977 prompts | 940 steps/epoch | 50 total steps | student params 3.70B | teacher overlap=True
|
| 23 |
+
restored optimizer/scheduler state from step 25; rebuilt 252 paged buffers
|
| 24 |
+
{"step": 26, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2147833664106205, "tokens": 120000, "cumulative_loss_tokens": 3120000, "grad_norm": 1.5625, "lr": 3e-05, "finish_rate": 0.139, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 45.1, "t_step_s": 101.0, "t_refresh_s": 0.4, "mem_gb": 16.49}
|
| 25 |
+
{"step": 27, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.19520521579974642, "tokens": 120000, "cumulative_loss_tokens": 3240000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.129, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 46.9, "t_step_s": 95.9, "t_refresh_s": 0.4, "mem_gb": 16.65}
|
| 26 |
+
{"step": 28, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.16118225911688058, "tokens": 120000, "cumulative_loss_tokens": 3360000, "grad_norm": 1.046875, "lr": 3e-05, "finish_rate": 0.139, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 42.9, "t_step_s": 91.4, "t_refresh_s": 0.4, "mem_gb": 16.56}
|
| 27 |
+
{"step": 29, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.17453759159445761, "tokens": 120000, "cumulative_loss_tokens": 3480000, "grad_norm": 0.99609375, "lr": 3e-05, "finish_rate": 0.171, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 45.4, "t_step_s": 94.7, "t_refresh_s": 0.4, "mem_gb": 16.55}
|
| 28 |
+
{"step": 30, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.12905135469088952, "tokens": 120000, "cumulative_loss_tokens": 3600000, "grad_norm": 0.99609375, "lr": 3e-05, "finish_rate": 0.227, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 43.8, "t_step_s": 91.5, "t_refresh_s": 0.4, "mem_gb": 16.54}
|
| 29 |
+
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
|
| 30 |
+
[eval step 30] sample: 'To solve this problem, we need to determine the maximum number of maples that can be planted along an alley given the constraints:\n\n1. There are a total of 75 trees.\n2. There are no two maples that ar'
|
| 31 |
+
{"step": 31, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.10643706315976258, "tokens": 120000, "cumulative_loss_tokens": 3720000, "grad_norm": 0.92578125, "lr": 3e-05, "finish_rate": 0.189, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 45.5, "t_step_s": 93.2, "t_refresh_s": 0.4, "mem_gb": 16.51}
|
| 32 |
+
{"step": 32, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.1212659154509815, "tokens": 120000, "cumulative_loss_tokens": 3840000, "grad_norm": 1.09375, "lr": 3e-05, "finish_rate": 0.159, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 45.0, "t_step_s": 93.8, "t_refresh_s": 0.4, "mem_gb": 16.56}
|
| 33 |
+
{"step": 33, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20737513309875502, "tokens": 120000, "cumulative_loss_tokens": 3960000, "grad_norm": 1.0546875, "lr": 3e-05, "finish_rate": 0.193, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 43.2, "t_step_s": 92.5, "t_refresh_s": 0.4, "mem_gb": 16.56}
|
| 34 |
+
{"step": 34, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.11765108014475555, "tokens": 120000, "cumulative_loss_tokens": 4080000, "grad_norm": 0.875, "lr": 3e-05, "finish_rate": 0.228, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 43.9, "t_step_s": 92.3, "t_refresh_s": 0.4, "mem_gb": 16.59}
|
| 35 |
+
{"step": 35, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.16500413158191368, "tokens": 120000, "cumulative_loss_tokens": 4200000, "grad_norm": 0.93359375, "lr": 3e-05, "finish_rate": 0.189, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 42.1, "t_step_s": 91.5, "t_refresh_s": 0.4, "mem_gb": 16.55}
|
| 36 |
+
{"step": 36, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.10690366555958365, "tokens": 120000, "cumulative_loss_tokens": 4320000, "grad_norm": 0.9453125, "lr": 3e-05, "finish_rate": 0.18, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 45.7, "t_step_s": 94.8, "t_refresh_s": 0.4, "mem_gb": 16.56}
|
| 37 |
+
{"step": 37, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.11145049199406058, "tokens": 120000, "cumulative_loss_tokens": 4440000, "grad_norm": 0.9375, "lr": 3e-05, "finish_rate": 0.177, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 46.0, "t_step_s": 94.2, "t_refresh_s": 0.4, "mem_gb": 16.48}
|
| 38 |
+
{"step": 38, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.11367983870146176, "tokens": 120000, "cumulative_loss_tokens": 4560000, "grad_norm": 0.96875, "lr": 3e-05, "finish_rate": 0.119, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 46.2, "t_step_s": 94.0, "t_refresh_s": 0.4, "mem_gb": 16.59}
|
| 39 |
+
{"step": 39, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.1883416127125422, "tokens": 120000, "cumulative_loss_tokens": 4680000, "grad_norm": 0.91796875, "lr": 3e-05, "finish_rate": 0.171, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 43.2, "t_step_s": 91.6, "t_refresh_s": 0.4, "mem_gb": 16.53}
|
| 40 |
+
{"step": 40, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.10533649440019702, "tokens": 120000, "cumulative_loss_tokens": 4800000, "grad_norm": 0.859375, "lr": 3e-05, "finish_rate": 0.181, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 46.4, "t_step_s": 94.6, "t_refresh_s": 0.4, "mem_gb": 16.53}
|
| 41 |
+
[eval step 40] sample: 'To solve this problem, we need to determine the maximum number of maples that can be planted along an alley given the constraints:\n\n1. There are a total of 75 trees.\n2. There are no two maples that ar'
|
| 42 |
+
{"step": 41, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.1024044312520884, "tokens": 120000, "cumulative_loss_tokens": 4920000, "grad_norm": 1.1640625, "lr": 3e-05, "finish_rate": 0.174, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 43.6, "t_step_s": 93.2, "t_refresh_s": 0.4, "mem_gb": 16.67}
|
| 43 |
+
{"step": 42, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.1360514337644602, "tokens": 120000, "cumulative_loss_tokens": 5040000, "grad_norm": 0.9453125, "lr": 3e-05, "finish_rate": 0.152, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 46.1, "t_step_s": 94.5, "t_refresh_s": 0.4, "mem_gb": 16.52}
|
| 44 |
+
{"step": 43, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.1632226786027973, "tokens": 120000, "cumulative_loss_tokens": 5160000, "grad_norm": 0.87109375, "lr": 3e-05, "finish_rate": 0.209, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 45.3, "t_step_s": 93.2, "t_refresh_s": 0.4, "mem_gb": 16.45}
|
| 45 |
+
{"step": 44, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.10166807023091241, "tokens": 120000, "cumulative_loss_tokens": 5280000, "grad_norm": 1.015625, "lr": 3e-05, "finish_rate": 0.226, "comp_len": 483.9, "t_data_s": 0.0, "t_rollout_s": 42.7, "t_step_s": 90.9, "t_refresh_s": 0.4, "mem_gb": 16.52}
|
| 46 |
+
{"step": 45, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.10869865408443535, "tokens": 120000, "cumulative_loss_tokens": 5400000, "grad_norm": 0.95703125, "lr": 3e-05, "finish_rate": 0.15, "comp_len": 487.8, "t_data_s": 0.0, "t_rollout_s": 44.3, "t_step_s": 92.2, "t_refresh_s": 0.4, "mem_gb": 16.42}
|
| 47 |
+
{"step": 46, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.14331271203849463, "tokens": 120000, "cumulative_loss_tokens": 5520000, "grad_norm": 0.86328125, "lr": 3e-05, "finish_rate": 0.165, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 46.2, "t_step_s": 94.5, "t_refresh_s": 0.4, "mem_gb": 16.5}
|
| 48 |
+
{"step": 47, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.11096431756988168, "tokens": 120000, "cumulative_loss_tokens": 5640000, "grad_norm": 1.0078125, "lr": 3e-05, "finish_rate": 0.192, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 43.2, "t_step_s": 91.2, "t_refresh_s": 0.4, "mem_gb": 16.45}
|
| 49 |
+
{"step": 48, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.13337831716419507, "tokens": 120000, "cumulative_loss_tokens": 5760000, "grad_norm": 1.0234375, "lr": 3e-05, "finish_rate": 0.166, "comp_len": 485.8, "t_data_s": 0.0, "t_rollout_s": 45.1, "t_step_s": 94.1, "t_refresh_s": 0.4, "mem_gb": 16.49}
|
| 50 |
+
{"step": 49, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.10467844488921886, "tokens": 120000, "cumulative_loss_tokens": 5880000, "grad_norm": 0.87890625, "lr": 3e-05, "finish_rate": 0.112, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 46.0, "t_step_s": 93.9, "t_refresh_s": 0.4, "mem_gb": 16.58}
|
| 51 |
+
{"step": 50, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.10983106163193782, "tokens": 120000, "cumulative_loss_tokens": 6000000, "grad_norm": 0.98828125, "lr": 3e-05, "finish_rate": 0.147, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 46.4, "t_step_s": 83.1, "t_refresh_s": 0.0, "mem_gb": 16.53}
|
| 52 |
+
[eval step 50] sample: 'To solve this problem, we need to maximize the number of maples \\( m \\) planted along an alley such that there are no two maples separated by exactly 5 trees. This means that if \\( m \\) maples are pla'
|
| 53 |
+
checkpoint snapshot queued -> outputs/healed/policy_confirm/combo_on25_seed1224/step0050
|
| 54 |
+
wandb: updating run metadata
|
| 55 |
+
wandb: uploading wandb-summary.json; uploading config.yaml; uploading output.log
|
| 56 |
+
wandb:
|
| 57 |
+
wandb: Run history:
|
| 58 |
+
wandb: comp_len βββ
ββββββββββββββ
β
βββββββ
|
| 59 |
+
wandb: cumulative_loss_tokens βββββββββββββ
β
β
β
βββββββββ
|
| 60 |
+
wandb: epoch βββββββββββββββββββββββββ
|
| 61 |
+
wandb: finish_rate ββββ
βββββββ
β
ββ
β
β
βββββββββ
|
| 62 |
+
wandb: grad_norm βββββββββββββββββββββββββ
|
| 63 |
+
wandb: lr βββββββββββββββββββββββββ
|
| 64 |
+
wandb: mem_gb βββ
β
βββ
β
ββ
β
βββββββββββββ
β
|
| 65 |
+
wandb: reverse_kl βββ
βββββββ
ββββββββ
βββββββ
|
| 66 |
+
wandb: step βββββββββββββ
β
β
β
βββββββββ
|
| 67 |
+
wandb: t_data_s βββββββββββββββββββββββββ
|
| 68 |
+
wandb: +4 ...
|
| 69 |
+
wandb:
|
| 70 |
+
wandb: Run summary:
|
| 71 |
+
wandb: comp_len 489.8
|
| 72 |
+
wandb: cumulative_loss_tokens 6000000
|
| 73 |
+
wandb: epoch 0
|
| 74 |
+
wandb: finish_rate 0.147
|
| 75 |
+
wandb: grad_norm 0.98828
|
| 76 |
+
wandb: lr 3e-05
|
| 77 |
+
wandb: mem_gb 16.53
|
| 78 |
+
wandb: reverse_kl 0.10983
|
| 79 |
+
wandb: step 50
|
| 80 |
+
wandb: t_data_s 0
|
| 81 |
+
wandb: +5 ...
|
| 82 |
+
wandb:
|
| 83 |
+
wandb: π View run policy-combo-on25-seed1224 at: https://wandb.ai/hbfreed/glean-heal/runs/62dayi8c
|
| 84 |
+
wandb: βοΈ View project at: https://wandb.ai/hbfreed/glean-heal
|
| 85 |
+
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
|
| 86 |
+
wandb: Find logs at: outputs/healed/policy_confirm/combo_on25_seed1224/wandb/run-20260714_171701-62dayi8c/logs
|
| 87 |
+
{
|
| 88 |
+
"correct": 851,
|
| 89 |
+
"accuracy": 0.645185746777862,
|
| 90 |
+
"finished": 1290,
|
| 91 |
+
"finish_rate": 0.978013646702047,
|
| 92 |
+
"mean_completion_tokens": 115.40864291129644
|
| 93 |
+
}
|
| 94 |
+
saved item-level results -> outputs/evals/policy_confirm/combo_off25on25_seed1224.json
|