/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
  warnings.warn('Grouped GEMM not available.')
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: setting up run vpov0urn
wandb: Tracking run with wandb version 0.28.0
wandb: Run data is saved locally in outputs/healed/healing_breadth/glean_math_keep25_seed1224_long/wandb/run-20260715_063332-vpov0urn
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run heal-glean-math-keep25-seed1224-long-step500
wandb: ⭐️ View project at https://wandb.ai/hbfreed/glean-heal
wandb: 🚀 View run at https://wandb.ai/hbfreed/glean-heal/runs/vpov0urn

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:00<00:00,  2.78it/s]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:00<00:00,  2.69it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00,  2.90it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00,  2.85it/s]
resumed student weights from outputs/healed/healing_breadth/glean_math_keep25_seed1224/step0050 (fresh optimizer, step counter at 50)
starting vllm rollout server on GPU GPU-864c54df-0130-7780-e271-8a5551d1733f (port 8377) ...
vllm server healthy in 24s
dataset_source filter ['omega', 'polaris', 'orz_math', 'mathsub', 'dapo-math'] -> 63998 prompts

Filter:   0%|          | 0/63998 [00:00<?, ? examples/s]
Filter:   2%|▏         | 1000/63998 [00:00<00:08, 7679.54 examples/s]
Filter:   3%|▎         | 2000/63998 [00:00<00:07, 8027.50 examples/s]
Filter:   5%|▍         | 3000/63998 [00:00<00:07, 8324.27 examples/s]
Filter:   6%|▋         | 4000/63998 [00:00<00:07, 8473.20 examples/s]
Filter:   8%|▊         | 5000/63998 [00:00<00:06, 8526.58 examples/s]
Filter:   9%|▉         | 6000/63998 [00:00<00:06, 8666.31 examples/s]
Filter:  11%|█         | 7000/63998 [00:00<00:06, 8673.75 examples/s]
Filter:  13%|█▎        | 8000/63998 [00:00<00:06, 8704.81 examples/s]
Filter:  14%|█▍        | 9000/63998 [00:01<00:06, 8625.62 examples/s]
Filter:  16%|█▌        | 10000/63998 [00:01<00:06, 8706.01 examples/s]
Filter:  17%|█▋        | 11000/63998 [00:01<00:06, 8649.18 examples/s]
Filter:  19%|█▉        | 12000/63998 [00:01<00:05, 8698.80 examples/s]
Filter:  20%|██        | 13000/63998 [00:01<00:05, 8701.21 examples/s]
Filter:  22%|██▏       | 14000/63998 [00:01<00:05, 8711.37 examples/s]
Filter:  23%|██▎       | 15000/63998 [00:01<00:05, 8630.47 examples/s]
Filter:  25%|██▌       | 16000/63998 [00:01<00:05, 8554.94 examples/s]
Filter:  27%|██▋       | 17000/63998 [00:01<00:05, 8498.99 examples/s]
Filter:  28%|██▊       | 18000/63998 [00:02<00:05, 8517.17 examples/s]
Filter:  30%|██▉       | 19000/63998 [00:02<00:05, 8598.56 examples/s]
Filter:  31%|███▏      | 20000/63998 [00:02<00:05, 8547.10 examples/s]
Filter:  33%|███▎      | 21000/63998 [00:02<00:04, 8728.18 examples/s]
Filter:  34%|███▍      | 22000/63998 [00:02<00:04, 8878.51 examples/s]
Filter:  36%|███▌      | 23000/63998 [00:02<00:04, 9017.68 examples/s]
Filter:  38%|███▊      | 24000/63998 [00:02<00:04, 9097.53 examples/s]
Filter:  39%|███▉      | 25000/63998 [00:02<00:04, 9131.81 examples/s]
Filter:  41%|████      | 26000/63998 [00:02<00:04, 9165.42 examples/s]
Filter:  42%|████▏     | 27000/63998 [00:03<00:04, 9179.28 examples/s]
Filter:  44%|████▍     | 28000/63998 [00:03<00:03, 9210.82 examples/s]
Filter:  45%|████▌     | 29000/63998 [00:03<00:03, 9207.85 examples/s]
Filter:  47%|████▋     | 30000/63998 [00:03<00:03, 9201.69 examples/s]
Filter:  48%|████▊     | 31000/63998 [00:03<00:03, 9196.81 examples/s]
Filter:  50%|█████     | 32000/63998 [00:03<00:03, 9192.89 examples/s]
Filter:  52%|█████▏    | 33000/63998 [00:03<00:03, 9182.84 examples/s]
Filter:  53%|█████▎    | 34000/63998 [00:03<00:03, 9176.81 examples/s]
Filter:  55%|█████▍    | 35000/63998 [00:03<00:03, 8713.49 examples/s]
Filter:  56%|█████▋    | 36000/63998 [00:04<00:03, 8462.70 examples/s]
Filter:  58%|█████▊    | 37000/63998 [00:04<00:03, 8236.16 examples/s]
Filter:  59%|█████▉    | 38000/63998 [00:04<00:03, 8079.71 examples/s]
Filter:  61%|██████    | 39000/63998 [00:04<00:03, 7950.19 examples/s]
Filter:  63%|██████▎   | 40000/63998 [00:04<00:03, 7863.02 examples/s]
Filter:  64%|██████▍   | 41000/63998 [00:04<00:02, 7850.10 examples/s]
Filter:  66%|██████▌   | 42000/63998 [00:05<00:04, 5336.26 examples/s]
Filter:  67%|██████▋   | 43000/63998 [00:05<00:03, 5865.95 examples/s]
Filter:  69%|██████▉   | 44000/63998 [00:05<00:03, 6349.64 examples/s]
Filter:  70%|███████   | 45000/63998 [00:05<00:02, 6684.66 examples/s]
Filter:  72%|███████▏  | 46000/63998 [00:05<00:02, 6945.00 examples/s]
Filter:  73%|███████▎  | 47000/63998 [00:05<00:02, 7140.39 examples/s]
Filter:  75%|███████▌  | 48000/63998 [00:05<00:02, 7293.02 examples/s]
Filter:  77%|███████▋  | 49000/63998 [00:05<00:01, 7690.42 examples/s]
Filter:  78%|███████▊  | 50000/63998 [00:06<00:01, 7965.63 examples/s]
Filter:  80%|███████▉  | 51000/63998 [00:06<00:01, 8197.57 examples/s]
Filter:  81%|████████▏ | 52000/63998 [00:06<00:01, 8381.47 examples/s]
Filter:  83%|████████▎ | 53000/63998 [00:06<00:01, 8495.06 examples/s]
Filter:  84%|████████▍ | 54000/63998 [00:06<00:01, 8618.97 examples/s]
Filter:  86%|████████▌ | 55000/63998 [00:06<00:01, 8671.38 examples/s]
Filter:  88%|████████▊ | 56000/63998 [00:06<00:00, 8713.31 examples/s]
Filter:  89%|████████▉ | 57000/63998 [00:06<00:00, 8755.50 examples/s]
Filter:  91%|█████████ | 58000/63998 [00:07<00:00, 8437.83 examples/s]
Filter:  92%|█████████▏| 59000/63998 [00:07<00:00, 8215.57 examples/s]
Filter:  94%|█████████▍| 60000/63998 [00:07<00:00, 8069.09 examples/s]
Filter:  95%|█████████▌| 61000/63998 [00:07<00:00, 7999.23 examples/s]
Filter:  97%|█████████▋| 62000/63998 [00:07<00:00, 7901.28 examples/s]
Filter:  98%|█████████▊| 63000/63998 [00:07<00:00, 7886.19 examples/s]
Filter: 100%|██████████| 63998/63998 [00:07<00:00, 7867.82 examples/s]
Filter: 100%|██████████| 63998/63998 [00:07<00:00, 8208.87 examples/s]
difficulty <= 4 -> 63998 prompts
WARNING: difficulty filter removed nothing — the slice likely carries difficulty=None (Dolci math sources do), so the teacher-competence guard is NOT in effect

Map:   0%|          | 0/63998 [00:00<?, ? examples/s]
Map:   0%|          | 290/63998 [00:00<00:22, 2869.81 examples/s]
Map:   1%|          | 618/63998 [00:00<00:20, 3107.07 examples/s]
Map:   1%|▏         | 941/63998 [00:00<00:19, 3160.39 examples/s]
Map:   2%|▏         | 1368/63998 [00:00<00:20, 2999.13 examples/s]
Map:   3%|▎         | 1699/63998 [00:00<00:20, 3097.22 examples/s]
Map:   3%|▎         | 2165/63998 [00:00<00:19, 3096.05 examples/s]
Map:   4%|▍         | 2501/63998 [00:00<00:19, 3166.87 examples/s]
Map:   4%|▍         | 2837/63998 [00:00<00:18, 3219.53 examples/s]
Map:   5%|▍         | 3164/63998 [00:01<00:19, 3183.23 examples/s]
Map:   5%|▌         | 3497/63998 [00:01<00:18, 3224.09 examples/s]
Map:   6%|▌         | 3832/63998 [00:01<00:18, 3256.34 examples/s]
Map:   7%|▋         | 4166/63998 [00:01<00:18, 3216.03 examples/s]
Map:   7%|▋         | 4500/63998 [00:01<00:18, 3248.00 examples/s]
Map:   8%|▊         | 4836/63998 [00:01<00:18, 3276.38 examples/s]
Map:   8%|▊         | 5170/63998 [00:01<00:18, 3232.46 examples/s]
Map:   9%|▊         | 5509/63998 [00:01<00:17, 3274.01 examples/s]
Map:   9%|▉         | 5839/63998 [00:01<00:17, 3280.60 examples/s]
Map:  10%|▉         | 6341/63998 [00:01<00:17, 3246.64 examples/s]
Map:  10%|█         | 6684/63998 [00:02<00:17, 3290.38 examples/s]
Map:  11%|█         | 7170/63998 [00:02<00:17, 3268.17 examples/s]
Map:  12%|█▏        | 7501/63998 [00:02<00:17, 3276.67 examples/s]
Map:  12%|█▏        | 7989/63998 [00:02<00:17, 3265.93 examples/s]
Map:  13%|█▎        | 8465/63998 [00:02<00:17, 3232.81 examples/s]
Map:  14%|█▎        | 8793/63998 [00:02<00:17, 3242.89 examples/s]
Map:  14%|█▍        | 9263/63998 [00:02<00:17, 3203.19 examples/s]
Map:  15%|█▍        | 9596/63998 [00:02<00:16, 3231.64 examples/s]
Map:  16%|█▌        | 9930/63998 [00:03<00:16, 3256.49 examples/s]
Map:  16%|█▋        | 10407/63998 [00:03<00:16, 3225.13 examples/s]
Map:  17%|█▋        | 10743/63998 [00:03<00:16, 3256.98 examples/s]
Map:  18%|█▊        | 11226/63998 [00:03<00:16, 3237.18 examples/s]
Map:  18%|█▊        | 11561/63998 [00:03<00:16, 3265.07 examples/s]
Map:  19%|█▊        | 11899/63998 [00:03<00:15, 3294.10 examples/s]
Map:  19%|█▉        | 12375/63998 [00:03<00:15, 3246.31 examples/s]
Map:  20%|█▉        | 12709/63998 [00:03<00:15, 3269.89 examples/s]
Map:  21%|██        | 13190/63998 [00:04<00:15, 3245.67 examples/s]
Map:  21%|██        | 13523/63998 [00:04<00:15, 3263.40 examples/s]
Map:  22%|██▏       | 13859/63998 [00:04<00:15, 3284.69 examples/s]
Map:  22%|██▏       | 14329/63998 [00:04<00:15, 3225.52 examples/s]
Map:  23%|██▎       | 14668/63998 [00:04<00:15, 3265.35 examples/s]
Map:  23%|██▎       | 15000/63998 [00:04<00:15, 3241.10 examples/s]
Map:  24%|██▍       | 15332/63998 [00:04<00:14, 3259.47 examples/s]
Map:  24%|██▍       | 15677/63998 [00:04<00:14, 3310.71 examples/s]
Map:  25%|██▌       | 16166/63998 [00:05<00:14, 3241.17 examples/s]
Map:  26%|██▌       | 16505/63998 [00:05<00:14, 3275.59 examples/s]
Map:  26%|██▋       | 16842/63998 [00:05<00:14, 3296.92 examples/s]
Map:  27%|██▋       | 17326/63998 [00:05<00:14, 3233.60 examples/s]
Map:  28%|██▊       | 17655/63998 [00:05<00:14, 3245.95 examples/s]
Map:  28%|██▊       | 17987/63998 [00:05<00:14, 3260.41 examples/s]
Map:  29%|██▉       | 18446/63998 [00:05<00:14, 3185.80 examples/s]
Map:  29%|██▉       | 18779/63998 [00:05<00:14, 3217.71 examples/s]
Map:  30%|███       | 19250/63998 [00:05<00:14, 3183.04 examples/s]
Map:  31%|███       | 19583/63998 [00:06<00:13, 3216.47 examples/s]
Map:  31%|███       | 19915/63998 [00:06<00:13, 3239.19 examples/s]
Map:  32%|███▏      | 20388/63998 [00:06<00:13, 3206.34 examples/s]
Map:  32%|███▏      | 20743/63998 [00:06<00:13, 3290.03 examples/s]
Map:  33%|███▎      | 21232/63998 [00:06<00:13, 3275.69 examples/s]
Map:  34%|███▎      | 21593/63998 [00:06<00:12, 3358.23 examples/s]
Map:  34%|███▍      | 21940/63998 [00:06<00:12, 3385.53 examples/s]
Map:  35%|███▌      | 22445/63998 [00:06<00:12, 3373.49 examples/s]
Map:  36%|███▌      | 22802/63998 [00:07<00:12, 3421.52 examples/s]
Map:  36%|███▋      | 23311/63998 [00:07<00:11, 3407.90 examples/s]
Map:  37%|███▋      | 23836/63998 [00:07<00:11, 3431.14 examples/s]
Map:  38%|███▊      | 24347/63998 [00:07<00:11, 3400.01 examples/s]
Map:  39%|███▊      | 24693/63998 [00:07<00:11, 3410.97 examples/s]
Map:  39%|███▉      | 25209/63998 [00:07<00:11, 3416.16 examples/s]
Map:  40%|███▉      | 25566/63998 [00:07<00:11, 3450.36 examples/s]
Map:  40%|████      | 25916/63998 [00:07<00:11, 3459.38 examples/s]
Map:  41%|████▏     | 26435/63998 [00:08<00:10, 3457.14 examples/s]
Map:  42%|████▏     | 26782/63998 [00:08<00:10, 3458.85 examples/s]
Map:  43%|████▎     | 27287/63998 [00:08<00:10, 3422.58 examples/s]
Map:  43%|████▎     | 27652/63998 [00:08<00:10, 3477.07 examples/s]
Map:  44%|████▍     | 28186/63998 [00:08<00:10, 3492.32 examples/s]
Map:  45%|████▍     | 28702/63998 [00:08<00:10, 3470.93 examples/s]
Map:  46%|████▌     | 29226/63998 [00:08<00:10, 3474.06 examples/s]
Map:  46%|████▌     | 29580/63998 [00:08<00:09, 3487.33 examples/s]
Map:  47%|████▋     | 30078/63998 [00:09<00:09, 3429.26 examples/s]
Map:  48%|████▊     | 30427/63998 [00:09<00:09, 3438.35 examples/s]
Map:  48%|████▊     | 30949/63998 [00:09<00:09, 3450.87 examples/s]
Map:  49%|████▉     | 31471/63998 [00:09<00:09, 3454.55 examples/s]
Map:  50%|████▉     | 31819/63998 [00:09<00:09, 3457.87 examples/s]
Map:  50%|█████     | 32177/63998 [00:09<00:09, 3440.76 examples/s]
Map:  51%|█████     | 32701/63998 [00:09<00:09, 3456.15 examples/s]
Map:  52%|█████▏    | 33192/63998 [00:10<00:09, 3392.07 examples/s]
Map:  52%|█████▏    | 33551/63998 [00:10<00:08, 3437.50 examples/s]
Map:  53%|█████▎    | 33916/63998 [00:10<00:08, 3489.39 examples/s]
Map:  54%|█████▎    | 34300/63998 [00:10<00:09, 3156.10 examples/s]
Map:  54%|█████▍    | 34683/63998 [00:10<00:09, 2945.67 examples/s]
Map:  55%|█████▍    | 35035/63998 [00:10<00:10, 2750.03 examples/s]
Map:  55%|█████▌    | 35415/63998 [00:10<00:10, 2680.95 examples/s]
Map:  56%|█████▌    | 35817/63998 [00:10<00:10, 2672.96 examples/s]
Map:  57%|█████▋    | 36186/63998 [00:11<00:10, 2604.64 examples/s]
Map:  57%|█████▋    | 36551/63998 [00:11<00:10, 2549.85 examples/s]
Map:  58%|█████▊    | 36810/63998 [00:11<00:10, 2557.70 examples/s]
Map:  58%|█████▊    | 37151/63998 [00:11<00:10, 2463.98 examples/s]
Map:  59%|█████▊    | 37522/63998 [00:11<00:10, 2460.20 examples/s]
Map:  59%|█████▉    | 37890/63998 [00:11<00:10, 2456.28 examples/s]
Map:  60%|█████▉    | 38234/63998 [00:11<00:10, 2385.21 examples/s]
Map:  60%|██████    | 38475/63998 [00:12<00:10, 2383.28 examples/s]
Map:  61%|██████    | 38726/63998 [00:12<00:10, 2406.42 examples/s]
Map:  61%|██████    | 38979/63998 [00:12<00:10, 2437.37 examples/s]
Map:  61%|██████▏   | 39316/63998 [00:12<00:10, 2363.25 examples/s]
Map:  62%|██████▏   | 39564/63998 [00:12<00:10, 2390.61 examples/s]
Map:  62%|██████▏   | 39814/63998 [00:12<00:10, 2417.51 examples/s]
Map:  63%|██████▎   | 40166/63998 [00:12<00:09, 2390.10 examples/s]
Map:  63%|██████▎   | 40426/63998 [00:12<00:09, 2438.03 examples/s]
Map:  64%|██████▎   | 40679/63998 [00:13<00:09, 2452.26 examples/s]
Map:  64%|██████▍   | 40951/63998 [00:13<00:09, 2523.64 examples/s]
Map:  65%|██████▍   | 41318/63998 [00:13<00:09, 2487.39 examples/s]
Map:  65%|██████▌   | 41686/63998 [00:13<00:09, 2472.20 examples/s]
Map:  66%|██████▌   | 42032/63998 [00:13<00:09, 2414.98 examples/s]
Map:  66%|██████▌   | 42393/63998 [00:13<00:08, 2410.55 examples/s]
Map:  67%|██████▋   | 42639/63998 [00:13<00:08, 2419.42 examples/s]
Map:  67%|██████▋   | 42890/63998 [00:13<00:08, 2435.30 examples/s]
Map:  68%|██████▊   | 43252/63998 [00:14<00:08, 2403.11 examples/s]
Map:  68%|██████▊   | 43498/63998 [00:14<00:08, 2410.16 examples/s]
Map:  68%|██████▊   | 43763/63998 [00:14<00:08, 2469.74 examples/s]
Map:  69%|██████▉   | 44131/63998 [00:14<00:08, 2442.90 examples/s]
Map:  70%|██████▉   | 44492/63998 [00:14<00:08, 2423.69 examples/s]
Map:  70%|██████▉   | 44738/63998 [00:14<00:07, 2430.87 examples/s]
Map:  70%|███████   | 45000/63998 [00:14<00:07, 2438.84 examples/s]
Map:  71%|███████   | 45270/63998 [00:14<00:07, 2494.99 examples/s]
Map:  71%|███████▏  | 45617/63998 [00:15<00:07, 2427.18 examples/s]
Map:  72%|███████▏  | 45865/63998 [00:15<00:07, 2428.44 examples/s]
Map:  72%|███████▏  | 46221/63998 [00:15<00:07, 2403.05 examples/s]
Map:  73%|███████▎  | 46487/63998 [00:15<00:07, 2463.45 examples/s]
Map:  73%|███████▎  | 46809/63998 [00:15<00:07, 2350.76 examples/s]
Map:  74%|███████▎  | 47158/63998 [00:15<00:07, 2334.27 examples/s]
Map:  74%|███████▍  | 47421/63998 [00:15<00:06, 2402.27 examples/s]
Map:  74%|███████▍  | 47668/63998 [00:15<00:06, 2413.79 examples/s]
Map:  75%|███████▌  | 48001/63998 [00:16<00:06, 2342.95 examples/s]
Map:  76%|███████▌  | 48331/63998 [00:16<00:06, 2582.02 examples/s]
Map:  76%|███████▌  | 48658/63998 [00:16<00:05, 2760.84 examples/s]
Map:  77%|███████▋  | 48999/63998 [00:16<00:05, 2934.09 examples/s]
Map:  77%|███████▋  | 49458/63998 [00:16<00:04, 2976.54 examples/s]
Map:  78%|███████▊  | 49774/63998 [00:16<00:04, 3016.66 examples/s]
Map:  79%|███████▊  | 50242/63998 [00:16<00:04, 3048.83 examples/s]
Map:  79%|███████▉  | 50570/63998 [00:16<00:04, 3104.58 examples/s]
Map:  80%|███████▉  | 50901/63998 [00:16<00:04, 3155.95 examples/s]
Map:  80%|████████  | 51362/63998 [00:17<00:04, 3120.84 examples/s]
Map:  81%|████████  | 51692/63998 [00:17<00:03, 3161.39 examples/s]
Map:  82%|████████▏ | 52173/63998 [00:17<00:03, 3173.32 examples/s]
Map:  82%|████████▏ | 52509/63998 [00:17<00:03, 3215.78 examples/s]
Map:  83%|████████▎ | 52841/63998 [00:17<00:03, 3239.31 examples/s]
Map:  83%|████████▎ | 53169/63998 [00:17<00:03, 3192.12 examples/s]
Map:  84%|████████▎ | 53511/63998 [00:17<00:03, 3254.57 examples/s]
Map:  84%|████████▍ | 53853/63998 [00:17<00:03, 3298.38 examples/s]
Map:  85%|████████▍ | 54338/63998 [00:17<00:02, 3269.76 examples/s]
Map:  86%|████████▌ | 54827/63998 [00:18<00:02, 3265.55 examples/s]
Map:  86%|████████▋ | 55302/63998 [00:18<00:02, 3221.53 examples/s]
Map:  87%|████████▋ | 55633/63998 [00:18<00:02, 3240.51 examples/s]
Map:  87%|████████▋ | 55959/63998 [00:18<00:02, 3240.25 examples/s]
Map:  88%|████████▊ | 56433/63998 [00:18<00:02, 3208.07 examples/s]
Map:  89%|████████▊ | 56761/63998 [00:18<00:02, 3224.77 examples/s]
Map:  89%|████████▉ | 57227/63998 [00:18<00:02, 3177.68 examples/s]
Map:  90%|████████▉ | 57555/63998 [00:19<00:02, 3201.90 examples/s]
Map:  90%|█████████ | 57883/63998 [00:19<00:01, 3218.58 examples/s]
Map:  91%|█████████ | 58344/63998 [00:19<00:01, 3163.61 examples/s]
Map:  92%|█████████▏| 58671/63998 [00:19<00:01, 3188.70 examples/s]
Map:  92%|█████████▏| 59000/63998 [00:19<00:01, 3172.52 examples/s]
Map:  93%|█████████▎| 59328/63998 [00:19<00:01, 3201.22 examples/s]
Map:  93%|█████████▎| 59808/63998 [00:19<00:01, 3199.92 examples/s]
Map:  94%|█████████▍| 60271/63998 [00:19<00:01, 3158.02 examples/s]
Map:  95%|█████████▍| 60596/63998 [00:19<00:01, 3177.05 examples/s]
Map:  95%|█████████▌| 60935/63998 [00:20<00:00, 3230.49 examples/s]
Map:  96%|█████████▌| 61387/63998 [00:20<00:00, 3146.34 examples/s]
Map:  96%|█████████▋| 61711/63998 [00:20<00:00, 3166.29 examples/s]
Map:  97%|█████████▋| 62174/63998 [00:20<00:00, 3133.44 examples/s]
Map:  98%|█████████▊| 62504/63998 [00:20<00:00, 3172.40 examples/s]
Map:  98%|█████████▊| 62827/63998 [00:20<00:00, 3185.25 examples/s]
Map:  99%|█████████▉| 63285/63998 [00:20<00:00, 3136.41 examples/s]
Map:  99%|█████████▉| 63608/63998 [00:20<00:00, 3156.23 examples/s]
Map: 100%|█████████▉| 63929/63998 [00:21<00:00, 3166.93 examples/s]
Map: 100%|██████████| 63998/63998 [00:21<00:00, 3040.89 examples/s]

Filter:   0%|          | 0/63998 [00:00<?, ? examples/s]
Filter:   6%|▋         | 4000/63998 [00:00<00:02, 27755.62 examples/s]
Filter:  13%|█▎        | 8000/63998 [00:00<00:01, 28121.37 examples/s]
Filter:  19%|█▉        | 12000/63998 [00:00<00:01, 28078.54 examples/s]
Filter:  25%|██▌       | 16000/63998 [00:00<00:01, 28102.05 examples/s]
Filter:  31%|███▏      | 20000/63998 [00:00<00:01, 28219.55 examples/s]
Filter:  38%|███▊      | 24000/63998 [00:00<00:01, 29194.66 examples/s]
Filter:  44%|████▍     | 28000/63998 [00:00<00:01, 29945.46 examples/s]
Filter:  50%|█████     | 32000/63998 [00:01<00:01, 30325.26 examples/s]
Filter:  58%|█████▊    | 37000/63998 [00:01<00:00, 27232.20 examples/s]
Filter:  63%|██████▎   | 40000/63998 [00:01<00:00, 24765.25 examples/s]
Filter:  67%|██████▋   | 43000/63998 [00:01<00:00, 23208.72 examples/s]
Filter:  72%|███████▏  | 46000/63998 [00:01<00:00, 22162.35 examples/s]
Filter:  78%|███████▊  | 50000/63998 [00:01<00:00, 22462.85 examples/s]
Filter:  84%|████████▍ | 54000/63998 [00:02<00:00, 24068.96 examples/s]
Filter:  91%|█████████ | 58000/63998 [00:02<00:00, 25159.84 examples/s]
Filter:  97%|█████████▋| 62000/63998 [00:02<00:00, 25655.08 examples/s]
Filter: 100%|██████████| 63998/63998 [00:02<00:00, 26072.90 examples/s]
63977 prompts | 940 steps/epoch | 500 total steps | student params 2.09B | teacher overlap=True
restored optimizer/scheduler state from step 50; rebuilt 228 paged buffers
{"step": 51, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36993047905315957, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 2.203125, "lr": 3e-05, "finish_rate": 0.025, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 37.6, "t_step_s": 80.2, "t_refresh_s": 0.3, "mem_gb": 10.4}
wandb: updating run metadata
wandb: uploading summary
wandb: 
wandb: Run history:
wandb:               comp_len ▁
wandb: cumulative_loss_tokens ▁
wandb:                  epoch ▁
wandb:            finish_rate ▁
wandb:              grad_norm ▁
wandb:                     lr ▁
wandb:                 mem_gb ▁
wandb:             reverse_kl ▁
wandb:                   step ▁
wandb:               t_data_s ▁
wandb:                     +4 ...
wandb: 
wandb: Run summary:
wandb:               comp_len 508.5
wandb: cumulative_loss_tokens 6120000
wandb:                  epoch 0
wandb:            finish_rate 0.025
wandb:              grad_norm 2.20312
wandb:                     lr 3e-05
wandb:                 mem_gb 10.4
wandb:             reverse_kl 0.36993
wandb:                   step 51
wandb:               t_data_s 0
wandb:                     +5 ...
wandb: 
wandb: 🚀 View run heal-glean-math-keep25-seed1224-long-step500 at: https://wandb.ai/hbfreed/glean-heal/runs/vpov0urn
wandb: ⭐️ View project at: https://wandb.ai/hbfreed/glean-heal
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: outputs/healed/healing_breadth/glean_math_keep25_seed1224_long/wandb/run-20260715_063332-vpov0urn/logs
Traceback (most recent call last):
  File "/home/henry/Documents/PythonProjects/variable-reap/scripts/11_distill_on_policy.py", line 1017, in <module>
    main()
  File "/home/henry/Documents/PythonProjects/variable-reap/scripts/11_distill_on_policy.py", line 860, in main
    save_student(student, tokenizer, live)
  File "/home/henry/Documents/PythonProjects/variable-reap/scripts/11_distill_on_policy.py", line 974, in save_student
    write_student_snapshot(student, tokenizer, path, snapshot_student(student))
  File "/home/henry/Documents/PythonProjects/variable-reap/scripts/11_distill_on_policy.py", line 961, in write_student_snapshot
    student.save_pretrained(path, state_dict=state_dict)
  File "/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4173, in save_pretrained
    safe_save_file(shard, os.path.join(save_directory, shard_file), metadata=metadata)
  File "/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/safetensors/torch.py", line 323, in save_file
    serialize_file(
safetensors._safetensors_rust.SafetensorError: Error while serializing: I/O error: No space left on device (os error 28)
/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available.
  warnings.warn('Grouped GEMM not available.')
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc.
wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: setting up run i5fg3xzq
wandb: Tracking run with wandb version 0.28.0
wandb: Run data is saved locally in outputs/healed/healing_breadth/glean_math_keep25_seed1224_long/wandb/run-20260715_071701-i5fg3xzq
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run heal-glean-math-keep25-seed1224-long-step500
wandb: ⭐️ View project at https://wandb.ai/hbfreed/glean-heal
wandb: 🚀 View run at https://wandb.ai/hbfreed/glean-heal/runs/i5fg3xzq

Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:00<00:00,  8.78it/s]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:00<00:00,  8.80it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00,  9.50it/s]
resumed student weights from outputs/healed/healing_breadth/glean_math_keep25_seed1224/step0050 (fresh optimizer, step counter at 50)
starting vllm rollout server on GPU GPU-864c54df-0130-7780-e271-8a5551d1733f (port 8377) ...
vllm server healthy in 24s
dataset_source filter ['omega', 'polaris', 'orz_math', 'mathsub', 'dapo-math'] -> 63998 prompts
difficulty <= 4 -> 63998 prompts
WARNING: difficulty filter removed nothing — the slice likely carries difficulty=None (Dolci math sources do), so the teacher-competence guard is NOT in effect
63977 prompts | 940 steps/epoch | 500 total steps | student params 2.09B | teacher overlap=True
restored optimizer/scheduler state from step 50; rebuilt 228 paged buffers
{"step": 51, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36993047905315957, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 2.203125, "lr": 3e-05, "finish_rate": 0.025, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 37.4, "t_step_s": 79.7, "t_refresh_s": 0.4, "mem_gb": 10.4}
{"step": 52, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3874280411116779, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 1.765625, "lr": 3e-05, "finish_rate": 0.021, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 37.0, "t_step_s": 72.9, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 53, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3746817006888489, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 1.703125, "lr": 3e-05, "finish_rate": 0.046, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.1, "t_step_s": 73.6, "t_refresh_s": 0.3, "mem_gb": 10.45}
{"step": 54, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.35510130622684954, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 2.140625, "lr": 3e-05, "finish_rate": 0.021, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 37.7, "t_step_s": 74.6, "t_refresh_s": 0.3, "mem_gb": 10.49}
{"step": 55, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34786954234316947, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 1.8359375, "lr": 3e-05, "finish_rate": 0.025, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 36.9, "t_step_s": 72.5, "t_refresh_s": 0.3, "mem_gb": 10.37}
{"step": 56, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3465809899068127, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 1.6796875, "lr": 3e-05, "finish_rate": 0.038, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 36.9, "t_step_s": 72.6, "t_refresh_s": 0.3, "mem_gb": 10.43}
{"step": 57, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.37013316290453074, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 2.34375, "lr": 3e-05, "finish_rate": 0.059, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 36.9, "t_step_s": 72.9, "t_refresh_s": 0.3, "mem_gb": 10.51}
{"step": 58, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3522711561133464, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 1.6328125, "lr": 3e-05, "finish_rate": 0.104, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.5, "t_step_s": 72.8, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 59, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3397159261740744, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 1.8203125, "lr": 3e-05, "finish_rate": 0.055, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 36.8, "t_step_s": 72.5, "t_refresh_s": 0.3, "mem_gb": 10.37}
{"step": 60, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3382764852608244, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 1.7265625, "lr": 3e-05, "finish_rate": 0.08, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.7, "t_step_s": 72.7, "t_refresh_s": 0.3, "mem_gb": 10.42}
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
[eval step 60] sample: "To solve this problem, we need to consider the constraints and the total number of trees. Let's denote:\n- \\( m \\) as the number of maples.\n- \\( l \\) as the number of larks.\n\nGiven:\n1. The total number"
{"step": 60, "gsm8k_n": 64, "gsm8k_quick_chat": 0.390625, "t_eval_s": 46.9}
{"step": 61, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.31868271032124756, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 1.5859375, "lr": 3e-05, "finish_rate": 0.059, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.5, "t_step_s": 74.1, "t_refresh_s": 0.3, "mem_gb": 12.44}
{"step": 62, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3191785570286214, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 1.5, "lr": 3e-05, "finish_rate": 0.063, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 36.9, "t_step_s": 73.1, "t_refresh_s": 0.3, "mem_gb": 10.43}
{"step": 63, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3180096957164506, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 1.5625, "lr": 3e-05, "finish_rate": 0.092, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 35.3, "t_step_s": 70.9, "t_refresh_s": 0.3, "mem_gb": 10.37}
{"step": 64, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.288780878906697, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 1.7578125, "lr": 3e-05, "finish_rate": 0.046, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 37.6, "t_step_s": 74.6, "t_refresh_s": 0.3, "mem_gb": 10.51}
{"step": 65, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.29233540159290033, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 1.4921875, "lr": 3e-05, "finish_rate": 0.084, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.2, "t_step_s": 71.9, "t_refresh_s": 0.3, "mem_gb": 10.5}
{"step": 66, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2769135721528282, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 1.3515625, "lr": 3e-05, "finish_rate": 0.105, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 35.6, "t_step_s": 71.3, "t_refresh_s": 0.3, "mem_gb": 10.53}
{"step": 67, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3056252569064498, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 1.4375, "lr": 3e-05, "finish_rate": 0.181, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.5, "t_step_s": 69.5, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 68, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.30779550124146043, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 1.5, "lr": 3e-05, "finish_rate": 0.129, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 71.2, "t_refresh_s": 0.3, "mem_gb": 10.42}
{"step": 69, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2875990556370467, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 1.4609375, "lr": 3e-05, "finish_rate": 0.132, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.8, "t_step_s": 69.7, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 70, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.28901881557305653, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 1.3828125, "lr": 3e-05, "finish_rate": 0.108, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 36.2, "t_step_s": 73.7, "t_refresh_s": 0.3, "mem_gb": 10.54}
[eval step 70] sample: 'To solve this problem, we need to determine the maximum number of maples that can be planted along the alley given the constraints:\n\n1. The total number of trees is 75.\n2. There are no two maples betw'
{"step": 70, "gsm8k_n": 64, "gsm8k_quick_chat": 0.28125, "t_eval_s": 44.7}
{"step": 71, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.281591778382659, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 1.53125, "lr": 3e-05, "finish_rate": 0.144, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.2, "t_step_s": 69.5, "t_refresh_s": 0.3, "mem_gb": 12.44}
{"step": 72, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27427094464078544, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 1.4375, "lr": 3e-05, "finish_rate": 0.107, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 35.8, "t_step_s": 73.1, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 73, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25415447360488275, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 1.3359375, "lr": 3e-05, "finish_rate": 0.116, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.4, "t_step_s": 71.6, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 74, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2598834279651443, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 1.1875, "lr": 3e-05, "finish_rate": 0.109, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.8, "t_step_s": 72.9, "t_refresh_s": 0.3, "mem_gb": 10.46}
{"step": 75, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3036417054095616, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 1.40625, "lr": 3e-05, "finish_rate": 0.133, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 10.41}
{"step": 76, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.28468160825570427, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 1.40625, "lr": 3e-05, "finish_rate": 0.124, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.7, "t_step_s": 71.7, "t_refresh_s": 0.3, "mem_gb": 10.45}
{"step": 77, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26907755986650783, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 1.2109375, "lr": 3e-05, "finish_rate": 0.137, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.0, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 10.46}
{"step": 78, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23572471514294546, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 1.3125, "lr": 3e-05, "finish_rate": 0.113, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.0, "t_step_s": 72.0, "t_refresh_s": 0.3, "mem_gb": 10.45}
{"step": 79, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2453982294137279, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 1.296875, "lr": 3e-05, "finish_rate": 0.156, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 33.0, "t_step_s": 69.6, "t_refresh_s": 0.3, "mem_gb": 10.43}
{"step": 80, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2668868999974181, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 2.328125, "lr": 3e-05, "finish_rate": 0.149, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.2, "t_step_s": 70.1, "t_refresh_s": 0.3, "mem_gb": 10.4}
[eval step 80] sample: 'To solve this problem, we need to determine the maximum number of maples that can be planted along the alley given the constraints:\n\n1. The total number of trees is 75.\n2. There are no two maples betw'
{"step": 80, "gsm8k_n": 64, "gsm8k_quick_chat": 0.390625, "t_eval_s": 44.8}
{"step": 81, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27245304357651623, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 1.46875, "lr": 3e-05, "finish_rate": 0.097, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 35.7, "t_step_s": 71.9, "t_refresh_s": 0.3, "mem_gb": 12.44}
{"step": 82, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26285181757311027, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 1.1875, "lr": 3e-05, "finish_rate": 0.117, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.5, "t_step_s": 71.6, "t_refresh_s": 0.3, "mem_gb": 10.42}
{"step": 83, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2504500327867145, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 1.390625, "lr": 3e-05, "finish_rate": 0.157, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.4, "t_step_s": 71.2, "t_refresh_s": 0.3, "mem_gb": 10.46}
{"step": 84, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25419983249952394, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 1.640625, "lr": 3e-05, "finish_rate": 0.165, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 10.43}
{"step": 85, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24969746678496402, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 1.3203125, "lr": 3e-05, "finish_rate": 0.117, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.8, "t_step_s": 73.1, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 86, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26418633902855215, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 1.3984375, "lr": 3e-05, "finish_rate": 0.172, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 34.2, "t_step_s": 71.4, "t_refresh_s": 0.3, "mem_gb": 10.48}
{"step": 87, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25853364124981065, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 1.3515625, "lr": 3e-05, "finish_rate": 0.137, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.3, "t_step_s": 70.2, "t_refresh_s": 0.3, "mem_gb": 10.4}
{"step": 88, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26424587182166676, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 1.328125, "lr": 3e-05, "finish_rate": 0.132, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.2, "t_step_s": 70.9, "t_refresh_s": 0.3, "mem_gb": 10.48}
{"step": 89, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.247395279224962, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 1.2109375, "lr": 3e-05, "finish_rate": 0.12, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.5, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 10.37}
{"step": 90, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27036167939615746, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 1.3359375, "lr": 3e-05, "finish_rate": 0.112, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 34.6, "t_step_s": 71.2, "t_refresh_s": 0.3, "mem_gb": 10.53}
[eval step 90] sample: "To solve this problem, we need to maximize the number of maples \\( M \\) such that the total number of trees \\( T \\) is 75, and there are no two maples between which there are exactly 5 trees.\n\nLet's b"
{"step": 90, "gsm8k_n": 64, "gsm8k_quick_chat": 0.359375, "t_eval_s": 44.7}
{"step": 91, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2520888489575436, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.216, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 33.6, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 12.44}
{"step": 92, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23066797492839397, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.088, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 34.6, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 10.43}
{"step": 93, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23549754046387972, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 1.3125, "lr": 3e-05, "finish_rate": 0.088, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 37.7, "t_step_s": 74.4, "t_refresh_s": 0.3, "mem_gb": 10.54}
{"step": 94, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2587053025153776, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 1.40625, "lr": 3e-05, "finish_rate": 0.126, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.1, "t_step_s": 71.5, "t_refresh_s": 0.3, "mem_gb": 10.45}
{"step": 95, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25772884955219927, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 1.2265625, "lr": 3e-05, "finish_rate": 0.059, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.7, "t_step_s": 74.7, "t_refresh_s": 0.3, "mem_gb": 10.52}
{"step": 96, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2583792527654519, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 1.296875, "lr": 3e-05, "finish_rate": 0.145, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 33.8, "t_step_s": 70.6, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 97, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2516133760511875, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 1.3984375, "lr": 3e-05, "finish_rate": 0.174, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.8, "t_step_s": 71.0, "t_refresh_s": 0.3, "mem_gb": 10.56}
{"step": 98, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2511387197598815, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 1.25, "lr": 3e-05, "finish_rate": 0.083, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 36.0, "t_step_s": 72.5, "t_refresh_s": 0.3, "mem_gb": 10.5}
{"step": 99, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25845928863280765, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 1.328125, "lr": 3e-05, "finish_rate": 0.121, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.0, "t_step_s": 71.0, "t_refresh_s": 0.3, "mem_gb": 10.4}
{"step": 100, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2377076770828416, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.177, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 32.8, "t_step_s": 69.4, "t_refresh_s": 0.3, "mem_gb": 10.38}
[eval step 100] sample: 'To solve this problem, we need to maximize the number of maples (M) that can be planted along the alley such that the total number of trees (T) is 75, and there are no two maples between which there a'
{"step": 100, "gsm8k_n": 64, "gsm8k_quick_chat": 0.328125, "t_eval_s": 44.8}
checkpoint snapshot queued -> outputs/healed/healing_breadth/glean_math_keep25_seed1224_long/step0100
{"step": 101, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2684837412368506, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 1.296875, "lr": 3e-05, "finish_rate": 0.196, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 33.0, "t_step_s": 69.4, "t_refresh_s": 0.3, "mem_gb": 12.44}
{"step": 102, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24094757018235202, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 1.3046875, "lr": 3e-05, "finish_rate": 0.141, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.0, "t_step_s": 70.5, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 103, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2634418107945472, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 1.2265625, "lr": 3e-05, "finish_rate": 0.181, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 70.0, "t_refresh_s": 0.3, "mem_gb": 10.4}
{"step": 104, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22899044912308456, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 1.171875, "lr": 3e-05, "finish_rate": 0.136, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.8, "t_step_s": 70.1, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 105, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23377793765577176, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.128, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 72.2, "t_refresh_s": 0.3, "mem_gb": 10.52}
{"step": 106, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23661676298119128, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 1.1484375, "lr": 3e-05, "finish_rate": 0.121, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.6, "t_step_s": 72.4, "t_refresh_s": 0.3, "mem_gb": 10.46}
{"step": 107, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22338223065932591, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 1.1328125, "lr": 3e-05, "finish_rate": 0.145, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.3, "t_step_s": 71.2, "t_refresh_s": 0.3, "mem_gb": 10.49}
{"step": 108, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24060181738672157, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 1.34375, "lr": 3e-05, "finish_rate": 0.168, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 70.5, "t_refresh_s": 0.3, "mem_gb": 10.54}
{"step": 109, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22999099592032532, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 1.1796875, "lr": 3e-05, "finish_rate": 0.161, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 70.3, "t_refresh_s": 0.3, "mem_gb": 10.37}
{"step": 110, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24095743914954365, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 1.578125, "lr": 3e-05, "finish_rate": 0.112, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 71.6, "t_refresh_s": 0.3, "mem_gb": 10.48}
[eval step 110] sample: 'To solve this problem, we need to determine the maximum number of maples (\\(M\\)) that can be planted along the alley such that the total number of trees (\\(T\\)) is 75, and there are no two maples betw'
{"step": 110, "gsm8k_n": 64, "gsm8k_quick_chat": 0.390625, "t_eval_s": 45.3}
{"step": 111, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21752187181736032, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 1.0859375, "lr": 3e-05, "finish_rate": 0.217, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 30.9, "t_step_s": 68.1, "t_refresh_s": 0.3, "mem_gb": 12.44}
{"step": 112, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2314429316678395, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 1.3359375, "lr": 3e-05, "finish_rate": 0.12, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.8, "t_step_s": 71.5, "t_refresh_s": 0.3, "mem_gb": 10.52}
{"step": 113, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2235619456699739, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 1.1484375, "lr": 3e-05, "finish_rate": 0.137, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 10.55}
{"step": 114, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23131392294329903, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.124, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.3, "t_step_s": 71.2, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 115, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22863815203296642, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 1.2578125, "lr": 3e-05, "finish_rate": 0.104, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 71.7, "t_refresh_s": 0.3, "mem_gb": 10.4}
{"step": 116, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22345312169020376, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 1.3203125, "lr": 3e-05, "finish_rate": 0.125, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 72.0, "t_refresh_s": 0.3, "mem_gb": 10.48}
{"step": 117, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22978670983004074, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 1.1328125, "lr": 3e-05, "finish_rate": 0.108, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 35.8, "t_step_s": 73.0, "t_refresh_s": 0.3, "mem_gb": 10.52}
{"step": 118, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2331024925402055, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 1.28125, "lr": 3e-05, "finish_rate": 0.076, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 37.3, "t_step_s": 73.9, "t_refresh_s": 0.3, "mem_gb": 10.46}
{"step": 119, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22393636475646247, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.13, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.4, "t_step_s": 71.5, "t_refresh_s": 0.3, "mem_gb": 10.44}
{"step": 120, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2355032395929719, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.16, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 33.1, "t_step_s": 68.8, "t_refresh_s": 0.3, "mem_gb": 10.36}
[eval step 120] sample: 'To solve this problem, we need to determine the maximum number of maples that can be planted along the alley given the constraints:\n\n1. The total number of trees is 75.\n2. There are no two maples betw'
{"step": 120, "gsm8k_n": 64, "gsm8k_quick_chat": 0.34375, "t_eval_s": 44.9}
{"step": 121, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24267969185064237, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 1.171875, "lr": 3e-05, "finish_rate": 0.113, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 12.43}
{"step": 122, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23425059191025793, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 1.1640625, "lr": 3e-05, "finish_rate": 0.093, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.0, "t_step_s": 74.2, "t_refresh_s": 0.3, "mem_gb": 10.53}
{"step": 123, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26844764651320874, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 1.359375, "lr": 3e-05, "finish_rate": 0.096, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.0, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 124, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21723629867484173, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 1.2578125, "lr": 3e-05, "finish_rate": 0.116, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 33.6, "t_step_s": 69.0, "t_refresh_s": 0.3, "mem_gb": 10.37}
{"step": 125, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21943003199926267, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 1.3515625, "lr": 3e-05, "finish_rate": 0.14, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.5, "t_step_s": 70.8, "t_refresh_s": 0.3, "mem_gb": 10.49}
{"step": 126, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.214982470301042, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 1.171875, "lr": 3e-05, "finish_rate": 0.13, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 70.3, "t_refresh_s": 0.3, "mem_gb": 10.37}
{"step": 127, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22647308003318806, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 1.34375, "lr": 3e-05, "finish_rate": 0.105, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.0, "t_step_s": 71.0, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 128, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23893264475651085, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 1.4140625, "lr": 3e-05, "finish_rate": 0.104, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.3, "t_step_s": 71.4, "t_refresh_s": 0.3, "mem_gb": 10.5}
{"step": 129, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23907007528046767, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 1.1796875, "lr": 3e-05, "finish_rate": 0.127, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 31.7, "t_step_s": 69.0, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 130, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23370486757767697, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.139, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 32.2, "t_step_s": 68.9, "t_refresh_s": 0.3, "mem_gb": 10.43}
[eval step 130] sample: 'To solve this problem, we need to determine the maximum number of maples that can be planted along the alley given the constraints:\n\n1. The total number of trees is 75.\n2. There are no two maples betw'
{"step": 130, "gsm8k_n": 64, "gsm8k_quick_chat": 0.375, "t_eval_s": 45.3}
{"step": 131, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22123020526878537, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.088, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.5, "t_step_s": 73.1, "t_refresh_s": 0.3, "mem_gb": 12.44}
{"step": 132, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22262640608406314, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.129, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 33.3, "t_step_s": 69.3, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 133, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23642245450820773, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 1.3125, "lr": 3e-05, "finish_rate": 0.141, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 70.1, "t_refresh_s": 0.3, "mem_gb": 10.5}
{"step": 134, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22181539568305014, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 1.1484375, "lr": 3e-05, "finish_rate": 0.124, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 35.1, "t_step_s": 71.9, "t_refresh_s": 0.3, "mem_gb": 10.49}
{"step": 135, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21886845049690457, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 1.265625, "lr": 3e-05, "finish_rate": 0.088, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.4, "t_step_s": 71.3, "t_refresh_s": 0.3, "mem_gb": 10.43}
{"step": 136, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23528965907134117, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.117, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.0, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 10.45}
{"step": 137, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22028850772418082, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 1.3828125, "lr": 3e-05, "finish_rate": 0.076, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.0, "t_step_s": 71.8, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 138, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22388356751191119, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 1.171875, "lr": 3e-05, "finish_rate": 0.123, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 69.5, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 139, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2186046968769282, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 1.1171875, "lr": 3e-05, "finish_rate": 0.084, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.2, "t_step_s": 72.8, "t_refresh_s": 0.3, "mem_gb": 10.46}
{"step": 140, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21518618610985576, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 1.1171875, "lr": 3e-05, "finish_rate": 0.121, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 69.8, "t_refresh_s": 0.3, "mem_gb": 10.44}
[eval step 140] sample: 'To solve this problem, we need to carefully analyze the constraints given:\n\n1. **Total number of trees:** 75 trees.\n2. **No two maples between which there are exactly 5 trees:** This means that no two'
{"step": 140, "gsm8k_n": 64, "gsm8k_quick_chat": 0.359375, "t_eval_s": 44.8}
{"step": 141, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21726824556073795, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 1.1953125, "lr": 3e-05, "finish_rate": 0.072, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.1, "t_step_s": 73.1, "t_refresh_s": 0.3, "mem_gb": 12.43}
{"step": 142, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21776566527485847, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 1.2109375, "lr": 3e-05, "finish_rate": 0.104, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 70.1, "t_refresh_s": 0.3, "mem_gb": 10.4}
{"step": 143, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21226181217512738, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.128, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.6, "t_step_s": 70.3, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 144, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20745828753014406, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 1.09375, "lr": 3e-05, "finish_rate": 0.141, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 71.5, "t_refresh_s": 0.3, "mem_gb": 10.46}
{"step": 145, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20095169744286687, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 1.1796875, "lr": 3e-05, "finish_rate": 0.067, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 37.0, "t_step_s": 72.8, "t_refresh_s": 0.3, "mem_gb": 10.5}
{"step": 146, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21430944070120653, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 1.15625, "lr": 3e-05, "finish_rate": 0.092, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 71.4, "t_refresh_s": 0.3, "mem_gb": 10.41}
{"step": 147, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20074411552796761, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 1.1015625, "lr": 3e-05, "finish_rate": 0.092, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.3, "t_step_s": 71.3, "t_refresh_s": 0.3, "mem_gb": 10.51}
{"step": 148, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2208368306187292, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 1.1328125, "lr": 3e-05, "finish_rate": 0.148, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 69.9, "t_refresh_s": 0.3, "mem_gb": 10.4}
{"step": 149, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22132619506145518, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 1.265625, "lr": 3e-05, "finish_rate": 0.108, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 36.8, "t_step_s": 74.1, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 150, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2195085643796871, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 1.1328125, "lr": 3e-05, "finish_rate": 0.112, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 36.1, "t_step_s": 73.3, "t_refresh_s": 0.3, "mem_gb": 10.53}
[eval step 150] sample: "To solve this problem, we need to determine the maximum number of maples that can be placed along the alley such that there are no two maples between which there are exactly 5 trees.\n\nLet's break down"
{"step": 150, "gsm8k_n": 64, "gsm8k_quick_chat": 0.3125, "t_eval_s": 44.9}
checkpoint snapshot queued -> outputs/healed/healing_breadth/glean_math_keep25_seed1224_long/step0150
{"step": 151, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21052663061742982, "tokens": 120000, "cumulative_loss_tokens": 18120000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.1, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.0, "t_step_s": 70.8, "t_refresh_s": 0.3, "mem_gb": 12.43}
{"step": 152, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23573231500076752, "tokens": 120000, "cumulative_loss_tokens": 18240000, "grad_norm": 1.140625, "lr": 3e-05, "finish_rate": 0.076, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 37.6, "t_step_s": 74.7, "t_refresh_s": 0.3, "mem_gb": 10.54}
{"step": 153, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21262744003695747, "tokens": 120000, "cumulative_loss_tokens": 18360000, "grad_norm": 1.125, "lr": 3e-05, "finish_rate": 0.152, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.7, "t_step_s": 70.2, "t_refresh_s": 0.3, "mem_gb": 10.46}
{"step": 154, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20737704386965683, "tokens": 120000, "cumulative_loss_tokens": 18480000, "grad_norm": 1.125, "lr": 3e-05, "finish_rate": 0.155, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 32.1, "t_step_s": 69.1, "t_refresh_s": 0.3, "mem_gb": 10.4}
{"step": 155, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20515705520591387, "tokens": 120000, "cumulative_loss_tokens": 18600000, "grad_norm": 1.125, "lr": 3e-05, "finish_rate": 0.125, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 34.8, "t_step_s": 72.9, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 156, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.227416008350874, "tokens": 120000, "cumulative_loss_tokens": 18720000, "grad_norm": 1.1640625, "lr": 3e-05, "finish_rate": 0.143, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 33.7, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 10.46}
{"step": 157, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2179569326767077, "tokens": 120000, "cumulative_loss_tokens": 18840000, "grad_norm": 1.1953125, "lr": 3e-05, "finish_rate": 0.116, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.7, "t_step_s": 70.6, "t_refresh_s": 0.3, "mem_gb": 10.37}
{"step": 158, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21790637281499803, "tokens": 120000, "cumulative_loss_tokens": 18960000, "grad_norm": 1.1953125, "lr": 3e-05, "finish_rate": 0.12, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.5, "t_step_s": 69.6, "t_refresh_s": 0.3, "mem_gb": 10.37}
{"step": 159, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21037111974650374, "tokens": 120000, "cumulative_loss_tokens": 19080000, "grad_norm": 1.25, "lr": 3e-05, "finish_rate": 0.124, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.3, "t_step_s": 70.5, "t_refresh_s": 0.3, "mem_gb": 10.39}
{"step": 160, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22135015249016385, "tokens": 120000, "cumulative_loss_tokens": 19200000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.1, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 72.0, "t_refresh_s": 0.3, "mem_gb": 10.44}
[eval step 160] sample: "To solve this problem, we need to determine the maximum number of maples that can be placed along the alley such that there are no two maples between which there are exactly 5 trees.\n\nLet's break down"
{"step": 160, "gsm8k_n": 64, "gsm8k_quick_chat": 0.375, "t_eval_s": 44.8}
{"step": 161, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.19312054982806245, "tokens": 120000, "cumulative_loss_tokens": 19320000, "grad_norm": 1.078125, "lr": 3e-05, "finish_rate": 0.076, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.0, "t_step_s": 73.0, "t_refresh_s": 0.3, "mem_gb": 12.43}
{"step": 162, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21525976048012574, "tokens": 120000, "cumulative_loss_tokens": 19440000, "grad_norm": 1.2734375, "lr": 3e-05, "finish_rate": 0.177, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 69.6, "t_refresh_s": 0.3, "mem_gb": 10.38}
{"step": 163, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21099251879341901, "tokens": 120000, "cumulative_loss_tokens": 19560000, "grad_norm": 1.171875, "lr": 3e-05, "finish_rate": 0.108, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.3, "t_step_s": 71.9, "t_refresh_s": 0.3, "mem_gb": 10.41}
{"step": 164, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2188319661433498, "tokens": 120000, "cumulative_loss_tokens": 19680000, "grad_norm": 1.1953125, "lr": 3e-05, "finish_rate": 0.136, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.7, "t_step_s": 70.1, "t_refresh_s": 0.3, "mem_gb": 10.47}
{"step": 165, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21699987841347854, "tokens": 120000, "cumulative_loss_tokens": 19800000, "grad_norm": 1.1640625, "lr": 3e-05, "finish_rate": 0.137, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.7, "t_step_s": 72.0, "t_refresh_s": 0.3, "mem_gb": 10.54}
{"step": 166, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20422210597346227, "tokens": 120000, "cumulative_loss_tokens": 19920000, "grad_norm": 1.1875, "lr": 3e-05, "finish_rate": 0.088, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 34.4, "t_step_s": 69.9, "t_refresh_s": 0.3, "mem_gb": 10.36}
{"step": 167, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21616086491048336, "tokens": 120000, "cumulative_loss_tokens": 20040000, "grad_norm": 1.109375, "lr": 3e-05, "finish_rate": 0.108, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.9, "t_step_s": 72.8, "t_refresh_s": 0.3, "mem_gb": 10.49}
{"step": 168, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2041427251155178, "tokens": 120000, "cumulative_loss_tokens": 20160000, "grad_norm": 1.1796875, "lr": 3e-05, "finish_rate": 0.117, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 10.39}
{"step": 169, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2007420184203113, "tokens": 120000, "cumulative_loss_tokens": 20280000, "grad_norm": 1.0546875, "lr": 3e-05, "finish_rate": 0.113, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.2, "t_step_s": 72.4, "t_refresh_s": 0.3, "mem_gb": 10.51}
{"step": 170, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22800080033155778, "tokens": 120000, "cumulative_loss_tokens": 20400000, "grad_norm": 1.40625, "lr": 3e-05, "finish_rate": 0.153, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.6, "t_step_s": 69.9, "t_refresh_s": 0.3, "mem_gb": 10.42}
[eval step 170] sample: 'To solve this problem, we need to carefully analyze the constraints and use reasoning to determine the maximum number of maples that can be planted.\n\n### Problem Breakdown:\n\n1. **Total Number of Trees'
{"step": 170, "gsm8k_n": 64, "gsm8k_quick_chat": 0.34375, "t_eval_s": 44.8}
{"step": 171, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20762226390354335, "tokens": 120000, "cumulative_loss_tokens": 20520000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.112, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 72.1, "t_refresh_s": 0.3, "mem_gb": 12.43}
{"step": 172, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.19745559071643898, "tokens": 120000, "cumulative_loss_tokens": 20640000, "grad_norm": 1.1875, "lr": 3e-05, "finish_rate": 0.145, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 69.5, "t_refresh_s": 0.3, "mem_gb": 10.38}