| /home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. |
| warnings.warn('Grouped GEMM not available.') |
| wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc. |
| wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin |
| wandb: setting up run vpov0urn |
| wandb: Tracking run with wandb version 0.28.0 |
| wandb: Run data is saved locally in outputs/healed/healing_breadth/glean_math_keep25_seed1224_long/wandb/run-20260715_063332-vpov0urn |
| wandb: Run `wandb offline` to turn off syncing. |
| wandb: Syncing run heal-glean-math-keep25-seed1224-long-step500 |
| wandb: ⭐️ View project at https://wandb.ai/hbfreed/glean-heal |
| wandb: 🚀 View run at https://wandb.ai/hbfreed/glean-heal/runs/vpov0urn |
|
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 33%|███▎ | 1/3 [00:00<00:00, 2.78it/s]
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:00<00:00, 2.69it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00, 2.90it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:01<00:00, 2.85it/s] |
| resumed student weights from outputs/healed/healing_breadth/glean_math_keep25_seed1224/step0050 (fresh optimizer, step counter at 50) |
| starting vllm rollout server on GPU GPU-864c54df-0130-7780-e271-8a5551d1733f (port 8377) ... |
| vllm server healthy in 24s |
| dataset_source filter ['omega', 'polaris', 'orz_math', 'mathsub', 'dapo-math'] -> 63998 prompts |
|
Filter: 0%| | 0/63998 [00:00<?, ? examples/s]
Filter: 2%|▏ | 1000/63998 [00:00<00:08, 7679.54 examples/s]
Filter: 3%|▎ | 2000/63998 [00:00<00:07, 8027.50 examples/s]
Filter: 5%|▍ | 3000/63998 [00:00<00:07, 8324.27 examples/s]
Filter: 6%|▋ | 4000/63998 [00:00<00:07, 8473.20 examples/s]
Filter: 8%|▊ | 5000/63998 [00:00<00:06, 8526.58 examples/s]
Filter: 9%|▉ | 6000/63998 [00:00<00:06, 8666.31 examples/s]
Filter: 11%|█ | 7000/63998 [00:00<00:06, 8673.75 examples/s]
Filter: 13%|█▎ | 8000/63998 [00:00<00:06, 8704.81 examples/s]
Filter: 14%|█▍ | 9000/63998 [00:01<00:06, 8625.62 examples/s]
Filter: 16%|█▌ | 10000/63998 [00:01<00:06, 8706.01 examples/s]
Filter: 17%|█▋ | 11000/63998 [00:01<00:06, 8649.18 examples/s]
Filter: 19%|█▉ | 12000/63998 [00:01<00:05, 8698.80 examples/s]
Filter: 20%|██ | 13000/63998 [00:01<00:05, 8701.21 examples/s]
Filter: 22%|██▏ | 14000/63998 [00:01<00:05, 8711.37 examples/s]
Filter: 23%|██▎ | 15000/63998 [00:01<00:05, 8630.47 examples/s]
Filter: 25%|██▌ | 16000/63998 [00:01<00:05, 8554.94 examples/s]
Filter: 27%|██▋ | 17000/63998 [00:01<00:05, 8498.99 examples/s]
Filter: 28%|██▊ | 18000/63998 [00:02<00:05, 8517.17 examples/s]
Filter: 30%|██▉ | 19000/63998 [00:02<00:05, 8598.56 examples/s]
Filter: 31%|███▏ | 20000/63998 [00:02<00:05, 8547.10 examples/s]
Filter: 33%|███▎ | 21000/63998 [00:02<00:04, 8728.18 examples/s]
Filter: 34%|███▍ | 22000/63998 [00:02<00:04, 8878.51 examples/s]
Filter: 36%|███▌ | 23000/63998 [00:02<00:04, 9017.68 examples/s]
Filter: 38%|███▊ | 24000/63998 [00:02<00:04, 9097.53 examples/s]
Filter: 39%|███▉ | 25000/63998 [00:02<00:04, 9131.81 examples/s]
Filter: 41%|████ | 26000/63998 [00:02<00:04, 9165.42 examples/s]
Filter: 42%|████▏ | 27000/63998 [00:03<00:04, 9179.28 examples/s]
Filter: 44%|████▍ | 28000/63998 [00:03<00:03, 9210.82 examples/s]
Filter: 45%|████▌ | 29000/63998 [00:03<00:03, 9207.85 examples/s]
Filter: 47%|████▋ | 30000/63998 [00:03<00:03, 9201.69 examples/s]
Filter: 48%|████▊ | 31000/63998 [00:03<00:03, 9196.81 examples/s]
Filter: 50%|█████ | 32000/63998 [00:03<00:03, 9192.89 examples/s]
Filter: 52%|█████▏ | 33000/63998 [00:03<00:03, 9182.84 examples/s]
Filter: 53%|█████▎ | 34000/63998 [00:03<00:03, 9176.81 examples/s]
Filter: 55%|█████▍ | 35000/63998 [00:03<00:03, 8713.49 examples/s]
Filter: 56%|█████▋ | 36000/63998 [00:04<00:03, 8462.70 examples/s]
Filter: 58%|█████▊ | 37000/63998 [00:04<00:03, 8236.16 examples/s]
Filter: 59%|█████▉ | 38000/63998 [00:04<00:03, 8079.71 examples/s]
Filter: 61%|██████ | 39000/63998 [00:04<00:03, 7950.19 examples/s]
Filter: 63%|██████▎ | 40000/63998 [00:04<00:03, 7863.02 examples/s]
Filter: 64%|██████▍ | 41000/63998 [00:04<00:02, 7850.10 examples/s]
Filter: 66%|██████▌ | 42000/63998 [00:05<00:04, 5336.26 examples/s]
Filter: 67%|██████▋ | 43000/63998 [00:05<00:03, 5865.95 examples/s]
Filter: 69%|██████▉ | 44000/63998 [00:05<00:03, 6349.64 examples/s]
Filter: 70%|███████ | 45000/63998 [00:05<00:02, 6684.66 examples/s]
Filter: 72%|███████▏ | 46000/63998 [00:05<00:02, 6945.00 examples/s]
Filter: 73%|███████▎ | 47000/63998 [00:05<00:02, 7140.39 examples/s]
Filter: 75%|███████▌ | 48000/63998 [00:05<00:02, 7293.02 examples/s]
Filter: 77%|███████▋ | 49000/63998 [00:05<00:01, 7690.42 examples/s]
Filter: 78%|███████▊ | 50000/63998 [00:06<00:01, 7965.63 examples/s]
Filter: 80%|███████▉ | 51000/63998 [00:06<00:01, 8197.57 examples/s]
Filter: 81%|████████▏ | 52000/63998 [00:06<00:01, 8381.47 examples/s]
Filter: 83%|████████▎ | 53000/63998 [00:06<00:01, 8495.06 examples/s]
Filter: 84%|████████▍ | 54000/63998 [00:06<00:01, 8618.97 examples/s]
Filter: 86%|████████▌ | 55000/63998 [00:06<00:01, 8671.38 examples/s]
Filter: 88%|████████▊ | 56000/63998 [00:06<00:00, 8713.31 examples/s]
Filter: 89%|████████▉ | 57000/63998 [00:06<00:00, 8755.50 examples/s]
Filter: 91%|█████████ | 58000/63998 [00:07<00:00, 8437.83 examples/s]
Filter: 92%|█████████▏| 59000/63998 [00:07<00:00, 8215.57 examples/s]
Filter: 94%|█████████▍| 60000/63998 [00:07<00:00, 8069.09 examples/s]
Filter: 95%|█████████▌| 61000/63998 [00:07<00:00, 7999.23 examples/s]
Filter: 97%|█████████▋| 62000/63998 [00:07<00:00, 7901.28 examples/s]
Filter: 98%|█████████▊| 63000/63998 [00:07<00:00, 7886.19 examples/s]
Filter: 100%|██████████| 63998/63998 [00:07<00:00, 7867.82 examples/s]
Filter: 100%|██████████| 63998/63998 [00:07<00:00, 8208.87 examples/s] |
| difficulty <= 4 -> 63998 prompts |
| WARNING: difficulty filter removed nothing — the slice likely carries difficulty=None (Dolci math sources do), so the teacher-competence guard is NOT in effect |
|
Map: 0%| | 0/63998 [00:00<?, ? examples/s]
Map: 0%| | 290/63998 [00:00<00:22, 2869.81 examples/s]
Map: 1%| | 618/63998 [00:00<00:20, 3107.07 examples/s]
Map: 1%|▏ | 941/63998 [00:00<00:19, 3160.39 examples/s]
Map: 2%|▏ | 1368/63998 [00:00<00:20, 2999.13 examples/s]
Map: 3%|▎ | 1699/63998 [00:00<00:20, 3097.22 examples/s]
Map: 3%|▎ | 2165/63998 [00:00<00:19, 3096.05 examples/s]
Map: 4%|▍ | 2501/63998 [00:00<00:19, 3166.87 examples/s]
Map: 4%|▍ | 2837/63998 [00:00<00:18, 3219.53 examples/s]
Map: 5%|▍ | 3164/63998 [00:01<00:19, 3183.23 examples/s]
Map: 5%|▌ | 3497/63998 [00:01<00:18, 3224.09 examples/s]
Map: 6%|▌ | 3832/63998 [00:01<00:18, 3256.34 examples/s]
Map: 7%|▋ | 4166/63998 [00:01<00:18, 3216.03 examples/s]
Map: 7%|▋ | 4500/63998 [00:01<00:18, 3248.00 examples/s]
Map: 8%|▊ | 4836/63998 [00:01<00:18, 3276.38 examples/s]
Map: 8%|▊ | 5170/63998 [00:01<00:18, 3232.46 examples/s]
Map: 9%|▊ | 5509/63998 [00:01<00:17, 3274.01 examples/s]
Map: 9%|▉ | 5839/63998 [00:01<00:17, 3280.60 examples/s]
Map: 10%|▉ | 6341/63998 [00:01<00:17, 3246.64 examples/s]
Map: 10%|█ | 6684/63998 [00:02<00:17, 3290.38 examples/s]
Map: 11%|█ | 7170/63998 [00:02<00:17, 3268.17 examples/s]
Map: 12%|█▏ | 7501/63998 [00:02<00:17, 3276.67 examples/s]
Map: 12%|█▏ | 7989/63998 [00:02<00:17, 3265.93 examples/s]
Map: 13%|█▎ | 8465/63998 [00:02<00:17, 3232.81 examples/s]
Map: 14%|█▎ | 8793/63998 [00:02<00:17, 3242.89 examples/s]
Map: 14%|█▍ | 9263/63998 [00:02<00:17, 3203.19 examples/s]
Map: 15%|█▍ | 9596/63998 [00:02<00:16, 3231.64 examples/s]
Map: 16%|█▌ | 9930/63998 [00:03<00:16, 3256.49 examples/s]
Map: 16%|█▋ | 10407/63998 [00:03<00:16, 3225.13 examples/s]
Map: 17%|█▋ | 10743/63998 [00:03<00:16, 3256.98 examples/s]
Map: 18%|█▊ | 11226/63998 [00:03<00:16, 3237.18 examples/s]
Map: 18%|█▊ | 11561/63998 [00:03<00:16, 3265.07 examples/s]
Map: 19%|█▊ | 11899/63998 [00:03<00:15, 3294.10 examples/s]
Map: 19%|█▉ | 12375/63998 [00:03<00:15, 3246.31 examples/s]
Map: 20%|█▉ | 12709/63998 [00:03<00:15, 3269.89 examples/s]
Map: 21%|██ | 13190/63998 [00:04<00:15, 3245.67 examples/s]
Map: 21%|██ | 13523/63998 [00:04<00:15, 3263.40 examples/s]
Map: 22%|██▏ | 13859/63998 [00:04<00:15, 3284.69 examples/s]
Map: 22%|██▏ | 14329/63998 [00:04<00:15, 3225.52 examples/s]
Map: 23%|██▎ | 14668/63998 [00:04<00:15, 3265.35 examples/s]
Map: 23%|██▎ | 15000/63998 [00:04<00:15, 3241.10 examples/s]
Map: 24%|██▍ | 15332/63998 [00:04<00:14, 3259.47 examples/s]
Map: 24%|██▍ | 15677/63998 [00:04<00:14, 3310.71 examples/s]
Map: 25%|██▌ | 16166/63998 [00:05<00:14, 3241.17 examples/s]
Map: 26%|██▌ | 16505/63998 [00:05<00:14, 3275.59 examples/s]
Map: 26%|██▋ | 16842/63998 [00:05<00:14, 3296.92 examples/s]
Map: 27%|██▋ | 17326/63998 [00:05<00:14, 3233.60 examples/s]
Map: 28%|██▊ | 17655/63998 [00:05<00:14, 3245.95 examples/s]
Map: 28%|██▊ | 17987/63998 [00:05<00:14, 3260.41 examples/s]
Map: 29%|██▉ | 18446/63998 [00:05<00:14, 3185.80 examples/s]
Map: 29%|██▉ | 18779/63998 [00:05<00:14, 3217.71 examples/s]
Map: 30%|███ | 19250/63998 [00:05<00:14, 3183.04 examples/s]
Map: 31%|███ | 19583/63998 [00:06<00:13, 3216.47 examples/s]
Map: 31%|███ | 19915/63998 [00:06<00:13, 3239.19 examples/s]
Map: 32%|███▏ | 20388/63998 [00:06<00:13, 3206.34 examples/s]
Map: 32%|███▏ | 20743/63998 [00:06<00:13, 3290.03 examples/s]
Map: 33%|███▎ | 21232/63998 [00:06<00:13, 3275.69 examples/s]
Map: 34%|███▎ | 21593/63998 [00:06<00:12, 3358.23 examples/s]
Map: 34%|███▍ | 21940/63998 [00:06<00:12, 3385.53 examples/s]
Map: 35%|███▌ | 22445/63998 [00:06<00:12, 3373.49 examples/s]
Map: 36%|███▌ | 22802/63998 [00:07<00:12, 3421.52 examples/s]
Map: 36%|███▋ | 23311/63998 [00:07<00:11, 3407.90 examples/s]
Map: 37%|███▋ | 23836/63998 [00:07<00:11, 3431.14 examples/s]
Map: 38%|███▊ | 24347/63998 [00:07<00:11, 3400.01 examples/s]
Map: 39%|███▊ | 24693/63998 [00:07<00:11, 3410.97 examples/s]
Map: 39%|███▉ | 25209/63998 [00:07<00:11, 3416.16 examples/s]
Map: 40%|███▉ | 25566/63998 [00:07<00:11, 3450.36 examples/s]
Map: 40%|████ | 25916/63998 [00:07<00:11, 3459.38 examples/s]
Map: 41%|████▏ | 26435/63998 [00:08<00:10, 3457.14 examples/s]
Map: 42%|████▏ | 26782/63998 [00:08<00:10, 3458.85 examples/s]
Map: 43%|████▎ | 27287/63998 [00:08<00:10, 3422.58 examples/s]
Map: 43%|████▎ | 27652/63998 [00:08<00:10, 3477.07 examples/s]
Map: 44%|████▍ | 28186/63998 [00:08<00:10, 3492.32 examples/s]
Map: 45%|████▍ | 28702/63998 [00:08<00:10, 3470.93 examples/s]
Map: 46%|████▌ | 29226/63998 [00:08<00:10, 3474.06 examples/s]
Map: 46%|████▌ | 29580/63998 [00:08<00:09, 3487.33 examples/s]
Map: 47%|████▋ | 30078/63998 [00:09<00:09, 3429.26 examples/s]
Map: 48%|████▊ | 30427/63998 [00:09<00:09, 3438.35 examples/s]
Map: 48%|████▊ | 30949/63998 [00:09<00:09, 3450.87 examples/s]
Map: 49%|████▉ | 31471/63998 [00:09<00:09, 3454.55 examples/s]
Map: 50%|████▉ | 31819/63998 [00:09<00:09, 3457.87 examples/s]
Map: 50%|█████ | 32177/63998 [00:09<00:09, 3440.76 examples/s]
Map: 51%|█████ | 32701/63998 [00:09<00:09, 3456.15 examples/s]
Map: 52%|█████▏ | 33192/63998 [00:10<00:09, 3392.07 examples/s]
Map: 52%|█████▏ | 33551/63998 [00:10<00:08, 3437.50 examples/s]
Map: 53%|█████▎ | 33916/63998 [00:10<00:08, 3489.39 examples/s]
Map: 54%|█████▎ | 34300/63998 [00:10<00:09, 3156.10 examples/s]
Map: 54%|█████▍ | 34683/63998 [00:10<00:09, 2945.67 examples/s]
Map: 55%|█████▍ | 35035/63998 [00:10<00:10, 2750.03 examples/s]
Map: 55%|█████▌ | 35415/63998 [00:10<00:10, 2680.95 examples/s]
Map: 56%|█████▌ | 35817/63998 [00:10<00:10, 2672.96 examples/s]
Map: 57%|█████▋ | 36186/63998 [00:11<00:10, 2604.64 examples/s]
Map: 57%|█████▋ | 36551/63998 [00:11<00:10, 2549.85 examples/s]
Map: 58%|█████▊ | 36810/63998 [00:11<00:10, 2557.70 examples/s]
Map: 58%|█████▊ | 37151/63998 [00:11<00:10, 2463.98 examples/s]
Map: 59%|█████▊ | 37522/63998 [00:11<00:10, 2460.20 examples/s]
Map: 59%|█████▉ | 37890/63998 [00:11<00:10, 2456.28 examples/s]
Map: 60%|█████▉ | 38234/63998 [00:11<00:10, 2385.21 examples/s]
Map: 60%|██████ | 38475/63998 [00:12<00:10, 2383.28 examples/s]
Map: 61%|██████ | 38726/63998 [00:12<00:10, 2406.42 examples/s]
Map: 61%|██████ | 38979/63998 [00:12<00:10, 2437.37 examples/s]
Map: 61%|██████▏ | 39316/63998 [00:12<00:10, 2363.25 examples/s]
Map: 62%|██████▏ | 39564/63998 [00:12<00:10, 2390.61 examples/s]
Map: 62%|██████▏ | 39814/63998 [00:12<00:10, 2417.51 examples/s]
Map: 63%|██████▎ | 40166/63998 [00:12<00:09, 2390.10 examples/s]
Map: 63%|██████▎ | 40426/63998 [00:12<00:09, 2438.03 examples/s]
Map: 64%|██████▎ | 40679/63998 [00:13<00:09, 2452.26 examples/s]
Map: 64%|██████▍ | 40951/63998 [00:13<00:09, 2523.64 examples/s]
Map: 65%|██████▍ | 41318/63998 [00:13<00:09, 2487.39 examples/s]
Map: 65%|██████▌ | 41686/63998 [00:13<00:09, 2472.20 examples/s]
Map: 66%|██████▌ | 42032/63998 [00:13<00:09, 2414.98 examples/s]
Map: 66%|██████▌ | 42393/63998 [00:13<00:08, 2410.55 examples/s]
Map: 67%|██████▋ | 42639/63998 [00:13<00:08, 2419.42 examples/s]
Map: 67%|██████▋ | 42890/63998 [00:13<00:08, 2435.30 examples/s]
Map: 68%|██████▊ | 43252/63998 [00:14<00:08, 2403.11 examples/s]
Map: 68%|██████▊ | 43498/63998 [00:14<00:08, 2410.16 examples/s]
Map: 68%|██████▊ | 43763/63998 [00:14<00:08, 2469.74 examples/s]
Map: 69%|██████▉ | 44131/63998 [00:14<00:08, 2442.90 examples/s]
Map: 70%|██████▉ | 44492/63998 [00:14<00:08, 2423.69 examples/s]
Map: 70%|██████▉ | 44738/63998 [00:14<00:07, 2430.87 examples/s]
Map: 70%|███████ | 45000/63998 [00:14<00:07, 2438.84 examples/s]
Map: 71%|███████ | 45270/63998 [00:14<00:07, 2494.99 examples/s]
Map: 71%|███████▏ | 45617/63998 [00:15<00:07, 2427.18 examples/s]
Map: 72%|███████▏ | 45865/63998 [00:15<00:07, 2428.44 examples/s]
Map: 72%|███████▏ | 46221/63998 [00:15<00:07, 2403.05 examples/s]
Map: 73%|███████▎ | 46487/63998 [00:15<00:07, 2463.45 examples/s]
Map: 73%|███████▎ | 46809/63998 [00:15<00:07, 2350.76 examples/s]
Map: 74%|███████▎ | 47158/63998 [00:15<00:07, 2334.27 examples/s]
Map: 74%|███████▍ | 47421/63998 [00:15<00:06, 2402.27 examples/s]
Map: 74%|███████▍ | 47668/63998 [00:15<00:06, 2413.79 examples/s]
Map: 75%|███████▌ | 48001/63998 [00:16<00:06, 2342.95 examples/s]
Map: 76%|███████▌ | 48331/63998 [00:16<00:06, 2582.02 examples/s]
Map: 76%|███████▌ | 48658/63998 [00:16<00:05, 2760.84 examples/s]
Map: 77%|███████▋ | 48999/63998 [00:16<00:05, 2934.09 examples/s]
Map: 77%|███████▋ | 49458/63998 [00:16<00:04, 2976.54 examples/s]
Map: 78%|███████▊ | 49774/63998 [00:16<00:04, 3016.66 examples/s]
Map: 79%|███████▊ | 50242/63998 [00:16<00:04, 3048.83 examples/s]
Map: 79%|███████▉ | 50570/63998 [00:16<00:04, 3104.58 examples/s]
Map: 80%|███████▉ | 50901/63998 [00:16<00:04, 3155.95 examples/s]
Map: 80%|████████ | 51362/63998 [00:17<00:04, 3120.84 examples/s]
Map: 81%|████████ | 51692/63998 [00:17<00:03, 3161.39 examples/s]
Map: 82%|████████▏ | 52173/63998 [00:17<00:03, 3173.32 examples/s]
Map: 82%|████████▏ | 52509/63998 [00:17<00:03, 3215.78 examples/s]
Map: 83%|████████▎ | 52841/63998 [00:17<00:03, 3239.31 examples/s]
Map: 83%|████████▎ | 53169/63998 [00:17<00:03, 3192.12 examples/s]
Map: 84%|████████▎ | 53511/63998 [00:17<00:03, 3254.57 examples/s]
Map: 84%|████████▍ | 53853/63998 [00:17<00:03, 3298.38 examples/s]
Map: 85%|████████▍ | 54338/63998 [00:17<00:02, 3269.76 examples/s]
Map: 86%|████████▌ | 54827/63998 [00:18<00:02, 3265.55 examples/s]
Map: 86%|████████▋ | 55302/63998 [00:18<00:02, 3221.53 examples/s]
Map: 87%|████████▋ | 55633/63998 [00:18<00:02, 3240.51 examples/s]
Map: 87%|████████▋ | 55959/63998 [00:18<00:02, 3240.25 examples/s]
Map: 88%|████████▊ | 56433/63998 [00:18<00:02, 3208.07 examples/s]
Map: 89%|████████▊ | 56761/63998 [00:18<00:02, 3224.77 examples/s]
Map: 89%|████████▉ | 57227/63998 [00:18<00:02, 3177.68 examples/s]
Map: 90%|████████▉ | 57555/63998 [00:19<00:02, 3201.90 examples/s]
Map: 90%|█████████ | 57883/63998 [00:19<00:01, 3218.58 examples/s]
Map: 91%|█████████ | 58344/63998 [00:19<00:01, 3163.61 examples/s]
Map: 92%|█████████▏| 58671/63998 [00:19<00:01, 3188.70 examples/s]
Map: 92%|█████████▏| 59000/63998 [00:19<00:01, 3172.52 examples/s]
Map: 93%|█████████▎| 59328/63998 [00:19<00:01, 3201.22 examples/s]
Map: 93%|█████████▎| 59808/63998 [00:19<00:01, 3199.92 examples/s]
Map: 94%|█████████▍| 60271/63998 [00:19<00:01, 3158.02 examples/s]
Map: 95%|█████████▍| 60596/63998 [00:19<00:01, 3177.05 examples/s]
Map: 95%|█████████▌| 60935/63998 [00:20<00:00, 3230.49 examples/s]
Map: 96%|█████████▌| 61387/63998 [00:20<00:00, 3146.34 examples/s]
Map: 96%|█████████▋| 61711/63998 [00:20<00:00, 3166.29 examples/s]
Map: 97%|█████████▋| 62174/63998 [00:20<00:00, 3133.44 examples/s]
Map: 98%|█████████▊| 62504/63998 [00:20<00:00, 3172.40 examples/s]
Map: 98%|█████████▊| 62827/63998 [00:20<00:00, 3185.25 examples/s]
Map: 99%|█████████▉| 63285/63998 [00:20<00:00, 3136.41 examples/s]
Map: 99%|█████████▉| 63608/63998 [00:20<00:00, 3156.23 examples/s]
Map: 100%|█████████▉| 63929/63998 [00:21<00:00, 3166.93 examples/s]
Map: 100%|██████████| 63998/63998 [00:21<00:00, 3040.89 examples/s] |
|
Filter: 0%| | 0/63998 [00:00<?, ? examples/s]
Filter: 6%|▋ | 4000/63998 [00:00<00:02, 27755.62 examples/s]
Filter: 13%|█▎ | 8000/63998 [00:00<00:01, 28121.37 examples/s]
Filter: 19%|█▉ | 12000/63998 [00:00<00:01, 28078.54 examples/s]
Filter: 25%|██▌ | 16000/63998 [00:00<00:01, 28102.05 examples/s]
Filter: 31%|███▏ | 20000/63998 [00:00<00:01, 28219.55 examples/s]
Filter: 38%|███▊ | 24000/63998 [00:00<00:01, 29194.66 examples/s]
Filter: 44%|████▍ | 28000/63998 [00:00<00:01, 29945.46 examples/s]
Filter: 50%|█████ | 32000/63998 [00:01<00:01, 30325.26 examples/s]
Filter: 58%|█████▊ | 37000/63998 [00:01<00:00, 27232.20 examples/s]
Filter: 63%|██████▎ | 40000/63998 [00:01<00:00, 24765.25 examples/s]
Filter: 67%|██████▋ | 43000/63998 [00:01<00:00, 23208.72 examples/s]
Filter: 72%|███████▏ | 46000/63998 [00:01<00:00, 22162.35 examples/s]
Filter: 78%|███████▊ | 50000/63998 [00:01<00:00, 22462.85 examples/s]
Filter: 84%|████████▍ | 54000/63998 [00:02<00:00, 24068.96 examples/s]
Filter: 91%|█████████ | 58000/63998 [00:02<00:00, 25159.84 examples/s]
Filter: 97%|█████████▋| 62000/63998 [00:02<00:00, 25655.08 examples/s]
Filter: 100%|██████████| 63998/63998 [00:02<00:00, 26072.90 examples/s] |
| 63977 prompts | 940 steps/epoch | 500 total steps | student params 2.09B | teacher overlap=True |
| restored optimizer/scheduler state from step 50; rebuilt 228 paged buffers |
| {"step": 51, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36993047905315957, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 2.203125, "lr": 3e-05, "finish_rate": 0.025, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 37.6, "t_step_s": 80.2, "t_refresh_s": 0.3, "mem_gb": 10.4} |
| wandb: updating run metadata |
| wandb: uploading summary |
| wandb: |
| wandb: Run history: |
| wandb: comp_len ▁ |
| wandb: cumulative_loss_tokens ▁ |
| wandb: epoch ▁ |
| wandb: finish_rate ▁ |
| wandb: grad_norm ▁ |
| wandb: lr ▁ |
| wandb: mem_gb ▁ |
| wandb: reverse_kl ▁ |
| wandb: step ▁ |
| wandb: t_data_s ▁ |
| wandb: +4 ... |
| wandb: |
| wandb: Run summary: |
| wandb: comp_len 508.5 |
| wandb: cumulative_loss_tokens 6120000 |
| wandb: epoch 0 |
| wandb: finish_rate 0.025 |
| wandb: grad_norm 2.20312 |
| wandb: lr 3e-05 |
| wandb: mem_gb 10.4 |
| wandb: reverse_kl 0.36993 |
| wandb: step 51 |
| wandb: t_data_s 0 |
| wandb: +5 ... |
| wandb: |
| wandb: 🚀 View run heal-glean-math-keep25-seed1224-long-step500 at: https://wandb.ai/hbfreed/glean-heal/runs/vpov0urn |
| wandb: ⭐️ View project at: https://wandb.ai/hbfreed/glean-heal |
| wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) |
| wandb: Find logs at: outputs/healed/healing_breadth/glean_math_keep25_seed1224_long/wandb/run-20260715_063332-vpov0urn/logs |
| Traceback (most recent call last): |
| File "/home/henry/Documents/PythonProjects/variable-reap/scripts/11_distill_on_policy.py", line 1017, in <module> |
| main() |
| File "/home/henry/Documents/PythonProjects/variable-reap/scripts/11_distill_on_policy.py", line 860, in main |
| save_student(student, tokenizer, live) |
| File "/home/henry/Documents/PythonProjects/variable-reap/scripts/11_distill_on_policy.py", line 974, in save_student |
| write_student_snapshot(student, tokenizer, path, snapshot_student(student)) |
| File "/home/henry/Documents/PythonProjects/variable-reap/scripts/11_distill_on_policy.py", line 961, in write_student_snapshot |
| student.save_pretrained(path, state_dict=state_dict) |
| File "/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py", line 4173, in save_pretrained |
| safe_save_file(shard, os.path.join(save_directory, shard_file), metadata=metadata) |
| File "/home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/safetensors/torch.py", line 323, in save_file |
| serialize_file( |
| safetensors._safetensors_rust.SafetensorError: Error while serializing: I/O error: No space left on device (os error 28) |
| /home/henry/Documents/PythonProjects/variable-reap/.venv/lib/python3.12/site-packages/megablocks/grouped_gemm_util.py:10: UserWarning: Grouped GEMM not available. |
| warnings.warn('Grouped GEMM not available.') |
| wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /home/henry/.netrc. |
| wandb: Currently logged in as: hbfreed to https://api.wandb.ai. Use `wandb login --relogin` to force relogin |
| wandb: setting up run i5fg3xzq |
| wandb: Tracking run with wandb version 0.28.0 |
| wandb: Run data is saved locally in outputs/healed/healing_breadth/glean_math_keep25_seed1224_long/wandb/run-20260715_071701-i5fg3xzq |
| wandb: Run `wandb offline` to turn off syncing. |
| wandb: Syncing run heal-glean-math-keep25-seed1224-long-step500 |
| wandb: ⭐️ View project at https://wandb.ai/hbfreed/glean-heal |
| wandb: 🚀 View run at https://wandb.ai/hbfreed/glean-heal/runs/i5fg3xzq |
|
Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards: 33%|███▎ | 1/3 [00:00<00:00, 8.78it/s]
Loading checkpoint shards: 67%|██████▋ | 2/3 [00:00<00:00, 8.80it/s]
Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 9.50it/s] |
| resumed student weights from outputs/healed/healing_breadth/glean_math_keep25_seed1224/step0050 (fresh optimizer, step counter at 50) |
| starting vllm rollout server on GPU GPU-864c54df-0130-7780-e271-8a5551d1733f (port 8377) ... |
| vllm server healthy in 24s |
| dataset_source filter ['omega', 'polaris', 'orz_math', 'mathsub', 'dapo-math'] -> 63998 prompts |
| difficulty <= 4 -> 63998 prompts |
| WARNING: difficulty filter removed nothing — the slice likely carries difficulty=None (Dolci math sources do), so the teacher-competence guard is NOT in effect |
| 63977 prompts | 940 steps/epoch | 500 total steps | student params 2.09B | teacher overlap=True |
| restored optimizer/scheduler state from step 50; rebuilt 228 paged buffers |
| {"step": 51, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.36993047905315957, "tokens": 120000, "cumulative_loss_tokens": 6120000, "grad_norm": 2.203125, "lr": 3e-05, "finish_rate": 0.025, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 37.4, "t_step_s": 79.7, "t_refresh_s": 0.4, "mem_gb": 10.4} |
| {"step": 52, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3874280411116779, "tokens": 120000, "cumulative_loss_tokens": 6240000, "grad_norm": 1.765625, "lr": 3e-05, "finish_rate": 0.021, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 37.0, "t_step_s": 72.9, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 53, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3746817006888489, "tokens": 120000, "cumulative_loss_tokens": 6360000, "grad_norm": 1.703125, "lr": 3e-05, "finish_rate": 0.046, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.1, "t_step_s": 73.6, "t_refresh_s": 0.3, "mem_gb": 10.45} |
| {"step": 54, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.35510130622684954, "tokens": 120000, "cumulative_loss_tokens": 6480000, "grad_norm": 2.140625, "lr": 3e-05, "finish_rate": 0.021, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 37.7, "t_step_s": 74.6, "t_refresh_s": 0.3, "mem_gb": 10.49} |
| {"step": 55, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.34786954234316947, "tokens": 120000, "cumulative_loss_tokens": 6600000, "grad_norm": 1.8359375, "lr": 3e-05, "finish_rate": 0.025, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 36.9, "t_step_s": 72.5, "t_refresh_s": 0.3, "mem_gb": 10.37} |
| {"step": 56, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3465809899068127, "tokens": 120000, "cumulative_loss_tokens": 6720000, "grad_norm": 1.6796875, "lr": 3e-05, "finish_rate": 0.038, "comp_len": 508.5, "t_data_s": 0.0, "t_rollout_s": 36.9, "t_step_s": 72.6, "t_refresh_s": 0.3, "mem_gb": 10.43} |
| {"step": 57, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.37013316290453074, "tokens": 120000, "cumulative_loss_tokens": 6840000, "grad_norm": 2.34375, "lr": 3e-05, "finish_rate": 0.059, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 36.9, "t_step_s": 72.9, "t_refresh_s": 0.3, "mem_gb": 10.51} |
| {"step": 58, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3522711561133464, "tokens": 120000, "cumulative_loss_tokens": 6960000, "grad_norm": 1.6328125, "lr": 3e-05, "finish_rate": 0.104, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.5, "t_step_s": 72.8, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 59, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3397159261740744, "tokens": 120000, "cumulative_loss_tokens": 7080000, "grad_norm": 1.8203125, "lr": 3e-05, "finish_rate": 0.055, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 36.8, "t_step_s": 72.5, "t_refresh_s": 0.3, "mem_gb": 10.37} |
| {"step": 60, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3382764852608244, "tokens": 120000, "cumulative_loss_tokens": 7200000, "grad_norm": 1.7265625, "lr": 3e-05, "finish_rate": 0.08, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.7, "t_step_s": 72.7, "t_refresh_s": 0.3, "mem_gb": 10.42} |
| The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results. |
| [eval step 60] sample: "To solve this problem, we need to consider the constraints and the total number of trees. Let's denote:\n- \\( m \\) as the number of maples.\n- \\( l \\) as the number of larks.\n\nGiven:\n1. The total number" |
| {"step": 60, "gsm8k_n": 64, "gsm8k_quick_chat": 0.390625, "t_eval_s": 46.9} |
| {"step": 61, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.31868271032124756, "tokens": 120000, "cumulative_loss_tokens": 7320000, "grad_norm": 1.5859375, "lr": 3e-05, "finish_rate": 0.059, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.5, "t_step_s": 74.1, "t_refresh_s": 0.3, "mem_gb": 12.44} |
| {"step": 62, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3191785570286214, "tokens": 120000, "cumulative_loss_tokens": 7440000, "grad_norm": 1.5, "lr": 3e-05, "finish_rate": 0.063, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 36.9, "t_step_s": 73.1, "t_refresh_s": 0.3, "mem_gb": 10.43} |
| {"step": 63, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3180096957164506, "tokens": 120000, "cumulative_loss_tokens": 7560000, "grad_norm": 1.5625, "lr": 3e-05, "finish_rate": 0.092, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 35.3, "t_step_s": 70.9, "t_refresh_s": 0.3, "mem_gb": 10.37} |
| {"step": 64, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.288780878906697, "tokens": 120000, "cumulative_loss_tokens": 7680000, "grad_norm": 1.7578125, "lr": 3e-05, "finish_rate": 0.046, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 37.6, "t_step_s": 74.6, "t_refresh_s": 0.3, "mem_gb": 10.51} |
| {"step": 65, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.29233540159290033, "tokens": 120000, "cumulative_loss_tokens": 7800000, "grad_norm": 1.4921875, "lr": 3e-05, "finish_rate": 0.084, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.2, "t_step_s": 71.9, "t_refresh_s": 0.3, "mem_gb": 10.5} |
| {"step": 66, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2769135721528282, "tokens": 120000, "cumulative_loss_tokens": 7920000, "grad_norm": 1.3515625, "lr": 3e-05, "finish_rate": 0.105, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 35.6, "t_step_s": 71.3, "t_refresh_s": 0.3, "mem_gb": 10.53} |
| {"step": 67, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3056252569064498, "tokens": 120000, "cumulative_loss_tokens": 8040000, "grad_norm": 1.4375, "lr": 3e-05, "finish_rate": 0.181, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.5, "t_step_s": 69.5, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 68, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.30779550124146043, "tokens": 120000, "cumulative_loss_tokens": 8160000, "grad_norm": 1.5, "lr": 3e-05, "finish_rate": 0.129, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 71.2, "t_refresh_s": 0.3, "mem_gb": 10.42} |
| {"step": 69, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2875990556370467, "tokens": 120000, "cumulative_loss_tokens": 8280000, "grad_norm": 1.4609375, "lr": 3e-05, "finish_rate": 0.132, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.8, "t_step_s": 69.7, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 70, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.28901881557305653, "tokens": 120000, "cumulative_loss_tokens": 8400000, "grad_norm": 1.3828125, "lr": 3e-05, "finish_rate": 0.108, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 36.2, "t_step_s": 73.7, "t_refresh_s": 0.3, "mem_gb": 10.54} |
| [eval step 70] sample: 'To solve this problem, we need to determine the maximum number of maples that can be planted along the alley given the constraints:\n\n1. The total number of trees is 75.\n2. There are no two maples betw' |
| {"step": 70, "gsm8k_n": 64, "gsm8k_quick_chat": 0.28125, "t_eval_s": 44.7} |
| {"step": 71, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.281591778382659, "tokens": 120000, "cumulative_loss_tokens": 8520000, "grad_norm": 1.53125, "lr": 3e-05, "finish_rate": 0.144, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.2, "t_step_s": 69.5, "t_refresh_s": 0.3, "mem_gb": 12.44} |
| {"step": 72, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27427094464078544, "tokens": 120000, "cumulative_loss_tokens": 8640000, "grad_norm": 1.4375, "lr": 3e-05, "finish_rate": 0.107, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 35.8, "t_step_s": 73.1, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 73, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25415447360488275, "tokens": 120000, "cumulative_loss_tokens": 8760000, "grad_norm": 1.3359375, "lr": 3e-05, "finish_rate": 0.116, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.4, "t_step_s": 71.6, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 74, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2598834279651443, "tokens": 120000, "cumulative_loss_tokens": 8880000, "grad_norm": 1.1875, "lr": 3e-05, "finish_rate": 0.109, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.8, "t_step_s": 72.9, "t_refresh_s": 0.3, "mem_gb": 10.46} |
| {"step": 75, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.3036417054095616, "tokens": 120000, "cumulative_loss_tokens": 9000000, "grad_norm": 1.40625, "lr": 3e-05, "finish_rate": 0.133, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 10.41} |
| {"step": 76, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.28468160825570427, "tokens": 120000, "cumulative_loss_tokens": 9120000, "grad_norm": 1.40625, "lr": 3e-05, "finish_rate": 0.124, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.7, "t_step_s": 71.7, "t_refresh_s": 0.3, "mem_gb": 10.45} |
| {"step": 77, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26907755986650783, "tokens": 120000, "cumulative_loss_tokens": 9240000, "grad_norm": 1.2109375, "lr": 3e-05, "finish_rate": 0.137, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.0, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 10.46} |
| {"step": 78, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23572471514294546, "tokens": 120000, "cumulative_loss_tokens": 9360000, "grad_norm": 1.3125, "lr": 3e-05, "finish_rate": 0.113, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.0, "t_step_s": 72.0, "t_refresh_s": 0.3, "mem_gb": 10.45} |
| {"step": 79, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2453982294137279, "tokens": 120000, "cumulative_loss_tokens": 9480000, "grad_norm": 1.296875, "lr": 3e-05, "finish_rate": 0.156, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 33.0, "t_step_s": 69.6, "t_refresh_s": 0.3, "mem_gb": 10.43} |
| {"step": 80, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2668868999974181, "tokens": 120000, "cumulative_loss_tokens": 9600000, "grad_norm": 2.328125, "lr": 3e-05, "finish_rate": 0.149, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.2, "t_step_s": 70.1, "t_refresh_s": 0.3, "mem_gb": 10.4} |
| [eval step 80] sample: 'To solve this problem, we need to determine the maximum number of maples that can be planted along the alley given the constraints:\n\n1. The total number of trees is 75.\n2. There are no two maples betw' |
| {"step": 80, "gsm8k_n": 64, "gsm8k_quick_chat": 0.390625, "t_eval_s": 44.8} |
| {"step": 81, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27245304357651623, "tokens": 120000, "cumulative_loss_tokens": 9720000, "grad_norm": 1.46875, "lr": 3e-05, "finish_rate": 0.097, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 35.7, "t_step_s": 71.9, "t_refresh_s": 0.3, "mem_gb": 12.44} |
| {"step": 82, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26285181757311027, "tokens": 120000, "cumulative_loss_tokens": 9840000, "grad_norm": 1.1875, "lr": 3e-05, "finish_rate": 0.117, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.5, "t_step_s": 71.6, "t_refresh_s": 0.3, "mem_gb": 10.42} |
| {"step": 83, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2504500327867145, "tokens": 120000, "cumulative_loss_tokens": 9960000, "grad_norm": 1.390625, "lr": 3e-05, "finish_rate": 0.157, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.4, "t_step_s": 71.2, "t_refresh_s": 0.3, "mem_gb": 10.46} |
| {"step": 84, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25419983249952394, "tokens": 120000, "cumulative_loss_tokens": 10080000, "grad_norm": 1.640625, "lr": 3e-05, "finish_rate": 0.165, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 10.43} |
| {"step": 85, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24969746678496402, "tokens": 120000, "cumulative_loss_tokens": 10200000, "grad_norm": 1.3203125, "lr": 3e-05, "finish_rate": 0.117, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.8, "t_step_s": 73.1, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 86, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26418633902855215, "tokens": 120000, "cumulative_loss_tokens": 10320000, "grad_norm": 1.3984375, "lr": 3e-05, "finish_rate": 0.172, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 34.2, "t_step_s": 71.4, "t_refresh_s": 0.3, "mem_gb": 10.48} |
| {"step": 87, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25853364124981065, "tokens": 120000, "cumulative_loss_tokens": 10440000, "grad_norm": 1.3515625, "lr": 3e-05, "finish_rate": 0.137, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.3, "t_step_s": 70.2, "t_refresh_s": 0.3, "mem_gb": 10.4} |
| {"step": 88, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26424587182166676, "tokens": 120000, "cumulative_loss_tokens": 10560000, "grad_norm": 1.328125, "lr": 3e-05, "finish_rate": 0.132, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.2, "t_step_s": 70.9, "t_refresh_s": 0.3, "mem_gb": 10.48} |
| {"step": 89, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.247395279224962, "tokens": 120000, "cumulative_loss_tokens": 10680000, "grad_norm": 1.2109375, "lr": 3e-05, "finish_rate": 0.12, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.5, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 10.37} |
| {"step": 90, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.27036167939615746, "tokens": 120000, "cumulative_loss_tokens": 10800000, "grad_norm": 1.3359375, "lr": 3e-05, "finish_rate": 0.112, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 34.6, "t_step_s": 71.2, "t_refresh_s": 0.3, "mem_gb": 10.53} |
| [eval step 90] sample: "To solve this problem, we need to maximize the number of maples \\( M \\) such that the total number of trees \\( T \\) is 75, and there are no two maples between which there are exactly 5 trees.\n\nLet's b" |
| {"step": 90, "gsm8k_n": 64, "gsm8k_quick_chat": 0.359375, "t_eval_s": 44.7} |
| {"step": 91, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2520888489575436, "tokens": 120000, "cumulative_loss_tokens": 10920000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.216, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 33.6, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 12.44} |
| {"step": 92, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23066797492839397, "tokens": 120000, "cumulative_loss_tokens": 11040000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.088, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 34.6, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 10.43} |
| {"step": 93, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23549754046387972, "tokens": 120000, "cumulative_loss_tokens": 11160000, "grad_norm": 1.3125, "lr": 3e-05, "finish_rate": 0.088, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 37.7, "t_step_s": 74.4, "t_refresh_s": 0.3, "mem_gb": 10.54} |
| {"step": 94, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2587053025153776, "tokens": 120000, "cumulative_loss_tokens": 11280000, "grad_norm": 1.40625, "lr": 3e-05, "finish_rate": 0.126, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.1, "t_step_s": 71.5, "t_refresh_s": 0.3, "mem_gb": 10.45} |
| {"step": 95, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25772884955219927, "tokens": 120000, "cumulative_loss_tokens": 11400000, "grad_norm": 1.2265625, "lr": 3e-05, "finish_rate": 0.059, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.7, "t_step_s": 74.7, "t_refresh_s": 0.3, "mem_gb": 10.52} |
| {"step": 96, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2583792527654519, "tokens": 120000, "cumulative_loss_tokens": 11520000, "grad_norm": 1.296875, "lr": 3e-05, "finish_rate": 0.145, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 33.8, "t_step_s": 70.6, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 97, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2516133760511875, "tokens": 120000, "cumulative_loss_tokens": 11640000, "grad_norm": 1.3984375, "lr": 3e-05, "finish_rate": 0.174, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.8, "t_step_s": 71.0, "t_refresh_s": 0.3, "mem_gb": 10.56} |
| {"step": 98, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2511387197598815, "tokens": 120000, "cumulative_loss_tokens": 11760000, "grad_norm": 1.25, "lr": 3e-05, "finish_rate": 0.083, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 36.0, "t_step_s": 72.5, "t_refresh_s": 0.3, "mem_gb": 10.5} |
| {"step": 99, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.25845928863280765, "tokens": 120000, "cumulative_loss_tokens": 11880000, "grad_norm": 1.328125, "lr": 3e-05, "finish_rate": 0.121, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.0, "t_step_s": 71.0, "t_refresh_s": 0.3, "mem_gb": 10.4} |
| {"step": 100, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2377076770828416, "tokens": 120000, "cumulative_loss_tokens": 12000000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.177, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 32.8, "t_step_s": 69.4, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| [eval step 100] sample: 'To solve this problem, we need to maximize the number of maples (M) that can be planted along the alley such that the total number of trees (T) is 75, and there are no two maples between which there a' |
| {"step": 100, "gsm8k_n": 64, "gsm8k_quick_chat": 0.328125, "t_eval_s": 44.8} |
| checkpoint snapshot queued -> outputs/healed/healing_breadth/glean_math_keep25_seed1224_long/step0100 |
| {"step": 101, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2684837412368506, "tokens": 120000, "cumulative_loss_tokens": 12120000, "grad_norm": 1.296875, "lr": 3e-05, "finish_rate": 0.196, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 33.0, "t_step_s": 69.4, "t_refresh_s": 0.3, "mem_gb": 12.44} |
| {"step": 102, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24094757018235202, "tokens": 120000, "cumulative_loss_tokens": 12240000, "grad_norm": 1.3046875, "lr": 3e-05, "finish_rate": 0.141, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.0, "t_step_s": 70.5, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 103, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2634418107945472, "tokens": 120000, "cumulative_loss_tokens": 12360000, "grad_norm": 1.2265625, "lr": 3e-05, "finish_rate": 0.181, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 70.0, "t_refresh_s": 0.3, "mem_gb": 10.4} |
| {"step": 104, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22899044912308456, "tokens": 120000, "cumulative_loss_tokens": 12480000, "grad_norm": 1.171875, "lr": 3e-05, "finish_rate": 0.136, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.8, "t_step_s": 70.1, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 105, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23377793765577176, "tokens": 120000, "cumulative_loss_tokens": 12600000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.128, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 72.2, "t_refresh_s": 0.3, "mem_gb": 10.52} |
| {"step": 106, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23661676298119128, "tokens": 120000, "cumulative_loss_tokens": 12720000, "grad_norm": 1.1484375, "lr": 3e-05, "finish_rate": 0.121, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.6, "t_step_s": 72.4, "t_refresh_s": 0.3, "mem_gb": 10.46} |
| {"step": 107, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22338223065932591, "tokens": 120000, "cumulative_loss_tokens": 12840000, "grad_norm": 1.1328125, "lr": 3e-05, "finish_rate": 0.145, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.3, "t_step_s": 71.2, "t_refresh_s": 0.3, "mem_gb": 10.49} |
| {"step": 108, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24060181738672157, "tokens": 120000, "cumulative_loss_tokens": 12960000, "grad_norm": 1.34375, "lr": 3e-05, "finish_rate": 0.168, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 70.5, "t_refresh_s": 0.3, "mem_gb": 10.54} |
| {"step": 109, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22999099592032532, "tokens": 120000, "cumulative_loss_tokens": 13080000, "grad_norm": 1.1796875, "lr": 3e-05, "finish_rate": 0.161, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 70.3, "t_refresh_s": 0.3, "mem_gb": 10.37} |
| {"step": 110, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24095743914954365, "tokens": 120000, "cumulative_loss_tokens": 13200000, "grad_norm": 1.578125, "lr": 3e-05, "finish_rate": 0.112, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 71.6, "t_refresh_s": 0.3, "mem_gb": 10.48} |
| [eval step 110] sample: 'To solve this problem, we need to determine the maximum number of maples (\\(M\\)) that can be planted along the alley such that the total number of trees (\\(T\\)) is 75, and there are no two maples betw' |
| {"step": 110, "gsm8k_n": 64, "gsm8k_quick_chat": 0.390625, "t_eval_s": 45.3} |
| {"step": 111, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21752187181736032, "tokens": 120000, "cumulative_loss_tokens": 13320000, "grad_norm": 1.0859375, "lr": 3e-05, "finish_rate": 0.217, "comp_len": 481.9, "t_data_s": 0.0, "t_rollout_s": 30.9, "t_step_s": 68.1, "t_refresh_s": 0.3, "mem_gb": 12.44} |
| {"step": 112, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2314429316678395, "tokens": 120000, "cumulative_loss_tokens": 13440000, "grad_norm": 1.3359375, "lr": 3e-05, "finish_rate": 0.12, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.8, "t_step_s": 71.5, "t_refresh_s": 0.3, "mem_gb": 10.52} |
| {"step": 113, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2235619456699739, "tokens": 120000, "cumulative_loss_tokens": 13560000, "grad_norm": 1.1484375, "lr": 3e-05, "finish_rate": 0.137, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 10.55} |
| {"step": 114, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23131392294329903, "tokens": 120000, "cumulative_loss_tokens": 13680000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.124, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.3, "t_step_s": 71.2, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 115, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22863815203296642, "tokens": 120000, "cumulative_loss_tokens": 13800000, "grad_norm": 1.2578125, "lr": 3e-05, "finish_rate": 0.104, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 71.7, "t_refresh_s": 0.3, "mem_gb": 10.4} |
| {"step": 116, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22345312169020376, "tokens": 120000, "cumulative_loss_tokens": 13920000, "grad_norm": 1.3203125, "lr": 3e-05, "finish_rate": 0.125, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 72.0, "t_refresh_s": 0.3, "mem_gb": 10.48} |
| {"step": 117, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22978670983004074, "tokens": 120000, "cumulative_loss_tokens": 14040000, "grad_norm": 1.1328125, "lr": 3e-05, "finish_rate": 0.108, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 35.8, "t_step_s": 73.0, "t_refresh_s": 0.3, "mem_gb": 10.52} |
| {"step": 118, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2331024925402055, "tokens": 120000, "cumulative_loss_tokens": 14160000, "grad_norm": 1.28125, "lr": 3e-05, "finish_rate": 0.076, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 37.3, "t_step_s": 73.9, "t_refresh_s": 0.3, "mem_gb": 10.46} |
| {"step": 119, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22393636475646247, "tokens": 120000, "cumulative_loss_tokens": 14280000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.13, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.4, "t_step_s": 71.5, "t_refresh_s": 0.3, "mem_gb": 10.44} |
| {"step": 120, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2355032395929719, "tokens": 120000, "cumulative_loss_tokens": 14400000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.16, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 33.1, "t_step_s": 68.8, "t_refresh_s": 0.3, "mem_gb": 10.36} |
| [eval step 120] sample: 'To solve this problem, we need to determine the maximum number of maples that can be planted along the alley given the constraints:\n\n1. The total number of trees is 75.\n2. There are no two maples betw' |
| {"step": 120, "gsm8k_n": 64, "gsm8k_quick_chat": 0.34375, "t_eval_s": 44.9} |
| {"step": 121, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.24267969185064237, "tokens": 120000, "cumulative_loss_tokens": 14520000, "grad_norm": 1.171875, "lr": 3e-05, "finish_rate": 0.113, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 12.43} |
| {"step": 122, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23425059191025793, "tokens": 120000, "cumulative_loss_tokens": 14640000, "grad_norm": 1.1640625, "lr": 3e-05, "finish_rate": 0.093, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.0, "t_step_s": 74.2, "t_refresh_s": 0.3, "mem_gb": 10.53} |
| {"step": 123, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.26844764651320874, "tokens": 120000, "cumulative_loss_tokens": 14760000, "grad_norm": 1.359375, "lr": 3e-05, "finish_rate": 0.096, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.0, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 124, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21723629867484173, "tokens": 120000, "cumulative_loss_tokens": 14880000, "grad_norm": 1.2578125, "lr": 3e-05, "finish_rate": 0.116, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 33.6, "t_step_s": 69.0, "t_refresh_s": 0.3, "mem_gb": 10.37} |
| {"step": 125, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21943003199926267, "tokens": 120000, "cumulative_loss_tokens": 15000000, "grad_norm": 1.3515625, "lr": 3e-05, "finish_rate": 0.14, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 34.5, "t_step_s": 70.8, "t_refresh_s": 0.3, "mem_gb": 10.49} |
| {"step": 126, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.214982470301042, "tokens": 120000, "cumulative_loss_tokens": 15120000, "grad_norm": 1.171875, "lr": 3e-05, "finish_rate": 0.13, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 70.3, "t_refresh_s": 0.3, "mem_gb": 10.37} |
| {"step": 127, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22647308003318806, "tokens": 120000, "cumulative_loss_tokens": 15240000, "grad_norm": 1.34375, "lr": 3e-05, "finish_rate": 0.105, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.0, "t_step_s": 71.0, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 128, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23893264475651085, "tokens": 120000, "cumulative_loss_tokens": 15360000, "grad_norm": 1.4140625, "lr": 3e-05, "finish_rate": 0.104, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.3, "t_step_s": 71.4, "t_refresh_s": 0.3, "mem_gb": 10.5} |
| {"step": 129, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23907007528046767, "tokens": 120000, "cumulative_loss_tokens": 15480000, "grad_norm": 1.1796875, "lr": 3e-05, "finish_rate": 0.127, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 31.7, "t_step_s": 69.0, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 130, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23370486757767697, "tokens": 120000, "cumulative_loss_tokens": 15600000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.139, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 32.2, "t_step_s": 68.9, "t_refresh_s": 0.3, "mem_gb": 10.43} |
| [eval step 130] sample: 'To solve this problem, we need to determine the maximum number of maples that can be planted along the alley given the constraints:\n\n1. The total number of trees is 75.\n2. There are no two maples betw' |
| {"step": 130, "gsm8k_n": 64, "gsm8k_quick_chat": 0.375, "t_eval_s": 45.3} |
| {"step": 131, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22123020526878537, "tokens": 120000, "cumulative_loss_tokens": 15720000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.088, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.5, "t_step_s": 73.1, "t_refresh_s": 0.3, "mem_gb": 12.44} |
| {"step": 132, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22262640608406314, "tokens": 120000, "cumulative_loss_tokens": 15840000, "grad_norm": 1.203125, "lr": 3e-05, "finish_rate": 0.129, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 33.3, "t_step_s": 69.3, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 133, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23642245450820773, "tokens": 120000, "cumulative_loss_tokens": 15960000, "grad_norm": 1.3125, "lr": 3e-05, "finish_rate": 0.141, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 70.1, "t_refresh_s": 0.3, "mem_gb": 10.5} |
| {"step": 134, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22181539568305014, "tokens": 120000, "cumulative_loss_tokens": 16080000, "grad_norm": 1.1484375, "lr": 3e-05, "finish_rate": 0.124, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 35.1, "t_step_s": 71.9, "t_refresh_s": 0.3, "mem_gb": 10.49} |
| {"step": 135, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21886845049690457, "tokens": 120000, "cumulative_loss_tokens": 16200000, "grad_norm": 1.265625, "lr": 3e-05, "finish_rate": 0.088, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.4, "t_step_s": 71.3, "t_refresh_s": 0.3, "mem_gb": 10.43} |
| {"step": 136, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23528965907134117, "tokens": 120000, "cumulative_loss_tokens": 16320000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.117, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.0, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 10.45} |
| {"step": 137, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22028850772418082, "tokens": 120000, "cumulative_loss_tokens": 16440000, "grad_norm": 1.3828125, "lr": 3e-05, "finish_rate": 0.076, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.0, "t_step_s": 71.8, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 138, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22388356751191119, "tokens": 120000, "cumulative_loss_tokens": 16560000, "grad_norm": 1.171875, "lr": 3e-05, "finish_rate": 0.123, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 69.5, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 139, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2186046968769282, "tokens": 120000, "cumulative_loss_tokens": 16680000, "grad_norm": 1.1171875, "lr": 3e-05, "finish_rate": 0.084, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.2, "t_step_s": 72.8, "t_refresh_s": 0.3, "mem_gb": 10.46} |
| {"step": 140, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21518618610985576, "tokens": 120000, "cumulative_loss_tokens": 16800000, "grad_norm": 1.1171875, "lr": 3e-05, "finish_rate": 0.121, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 69.8, "t_refresh_s": 0.3, "mem_gb": 10.44} |
| [eval step 140] sample: 'To solve this problem, we need to carefully analyze the constraints given:\n\n1. **Total number of trees:** 75 trees.\n2. **No two maples between which there are exactly 5 trees:** This means that no two' |
| {"step": 140, "gsm8k_n": 64, "gsm8k_quick_chat": 0.359375, "t_eval_s": 44.8} |
| {"step": 141, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21726824556073795, "tokens": 120000, "cumulative_loss_tokens": 16920000, "grad_norm": 1.1953125, "lr": 3e-05, "finish_rate": 0.072, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.1, "t_step_s": 73.1, "t_refresh_s": 0.3, "mem_gb": 12.43} |
| {"step": 142, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21776566527485847, "tokens": 120000, "cumulative_loss_tokens": 17040000, "grad_norm": 1.2109375, "lr": 3e-05, "finish_rate": 0.104, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 33.9, "t_step_s": 70.1, "t_refresh_s": 0.3, "mem_gb": 10.4} |
| {"step": 143, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21226181217512738, "tokens": 120000, "cumulative_loss_tokens": 17160000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.128, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.6, "t_step_s": 70.3, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 144, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20745828753014406, "tokens": 120000, "cumulative_loss_tokens": 17280000, "grad_norm": 1.09375, "lr": 3e-05, "finish_rate": 0.141, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 71.5, "t_refresh_s": 0.3, "mem_gb": 10.46} |
| {"step": 145, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20095169744286687, "tokens": 120000, "cumulative_loss_tokens": 17400000, "grad_norm": 1.1796875, "lr": 3e-05, "finish_rate": 0.067, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 37.0, "t_step_s": 72.8, "t_refresh_s": 0.3, "mem_gb": 10.5} |
| {"step": 146, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21430944070120653, "tokens": 120000, "cumulative_loss_tokens": 17520000, "grad_norm": 1.15625, "lr": 3e-05, "finish_rate": 0.092, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 71.4, "t_refresh_s": 0.3, "mem_gb": 10.41} |
| {"step": 147, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20074411552796761, "tokens": 120000, "cumulative_loss_tokens": 17640000, "grad_norm": 1.1015625, "lr": 3e-05, "finish_rate": 0.092, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.3, "t_step_s": 71.3, "t_refresh_s": 0.3, "mem_gb": 10.51} |
| {"step": 148, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2208368306187292, "tokens": 120000, "cumulative_loss_tokens": 17760000, "grad_norm": 1.1328125, "lr": 3e-05, "finish_rate": 0.148, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 69.9, "t_refresh_s": 0.3, "mem_gb": 10.4} |
| {"step": 149, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22132619506145518, "tokens": 120000, "cumulative_loss_tokens": 17880000, "grad_norm": 1.265625, "lr": 3e-05, "finish_rate": 0.108, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 36.8, "t_step_s": 74.1, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 150, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2195085643796871, "tokens": 120000, "cumulative_loss_tokens": 18000000, "grad_norm": 1.1328125, "lr": 3e-05, "finish_rate": 0.112, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 36.1, "t_step_s": 73.3, "t_refresh_s": 0.3, "mem_gb": 10.53} |
| [eval step 150] sample: "To solve this problem, we need to determine the maximum number of maples that can be placed along the alley such that there are no two maples between which there are exactly 5 trees.\n\nLet's break down" |
| {"step": 150, "gsm8k_n": 64, "gsm8k_quick_chat": 0.3125, "t_eval_s": 44.9} |
| checkpoint snapshot queued -> outputs/healed/healing_breadth/glean_math_keep25_seed1224_long/step0150 |
| {"step": 151, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21052663061742982, "tokens": 120000, "cumulative_loss_tokens": 18120000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.1, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.0, "t_step_s": 70.8, "t_refresh_s": 0.3, "mem_gb": 12.43} |
| {"step": 152, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.23573231500076752, "tokens": 120000, "cumulative_loss_tokens": 18240000, "grad_norm": 1.140625, "lr": 3e-05, "finish_rate": 0.076, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 37.6, "t_step_s": 74.7, "t_refresh_s": 0.3, "mem_gb": 10.54} |
| {"step": 153, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21262744003695747, "tokens": 120000, "cumulative_loss_tokens": 18360000, "grad_norm": 1.125, "lr": 3e-05, "finish_rate": 0.152, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.7, "t_step_s": 70.2, "t_refresh_s": 0.3, "mem_gb": 10.46} |
| {"step": 154, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20737704386965683, "tokens": 120000, "cumulative_loss_tokens": 18480000, "grad_norm": 1.125, "lr": 3e-05, "finish_rate": 0.155, "comp_len": 489.8, "t_data_s": 0.0, "t_rollout_s": 32.1, "t_step_s": 69.1, "t_refresh_s": 0.3, "mem_gb": 10.4} |
| {"step": 155, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20515705520591387, "tokens": 120000, "cumulative_loss_tokens": 18600000, "grad_norm": 1.125, "lr": 3e-05, "finish_rate": 0.125, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 34.8, "t_step_s": 72.9, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 156, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.227416008350874, "tokens": 120000, "cumulative_loss_tokens": 18720000, "grad_norm": 1.1640625, "lr": 3e-05, "finish_rate": 0.143, "comp_len": 491.8, "t_data_s": 0.0, "t_rollout_s": 33.7, "t_step_s": 70.4, "t_refresh_s": 0.3, "mem_gb": 10.46} |
| {"step": 157, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2179569326767077, "tokens": 120000, "cumulative_loss_tokens": 18840000, "grad_norm": 1.1953125, "lr": 3e-05, "finish_rate": 0.116, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.7, "t_step_s": 70.6, "t_refresh_s": 0.3, "mem_gb": 10.37} |
| {"step": 158, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21790637281499803, "tokens": 120000, "cumulative_loss_tokens": 18960000, "grad_norm": 1.1953125, "lr": 3e-05, "finish_rate": 0.12, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.5, "t_step_s": 69.6, "t_refresh_s": 0.3, "mem_gb": 10.37} |
| {"step": 159, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21037111974650374, "tokens": 120000, "cumulative_loss_tokens": 19080000, "grad_norm": 1.25, "lr": 3e-05, "finish_rate": 0.124, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.3, "t_step_s": 70.5, "t_refresh_s": 0.3, "mem_gb": 10.39} |
| {"step": 160, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22135015249016385, "tokens": 120000, "cumulative_loss_tokens": 19200000, "grad_norm": 1.234375, "lr": 3e-05, "finish_rate": 0.1, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 72.0, "t_refresh_s": 0.3, "mem_gb": 10.44} |
| [eval step 160] sample: "To solve this problem, we need to determine the maximum number of maples that can be placed along the alley such that there are no two maples between which there are exactly 5 trees.\n\nLet's break down" |
| {"step": 160, "gsm8k_n": 64, "gsm8k_quick_chat": 0.375, "t_eval_s": 44.8} |
| {"step": 161, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.19312054982806245, "tokens": 120000, "cumulative_loss_tokens": 19320000, "grad_norm": 1.078125, "lr": 3e-05, "finish_rate": 0.076, "comp_len": 506.3, "t_data_s": 0.0, "t_rollout_s": 37.0, "t_step_s": 73.0, "t_refresh_s": 0.3, "mem_gb": 12.43} |
| {"step": 162, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21525976048012574, "tokens": 120000, "cumulative_loss_tokens": 19440000, "grad_norm": 1.2734375, "lr": 3e-05, "finish_rate": 0.177, "comp_len": 493.8, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 69.6, "t_refresh_s": 0.3, "mem_gb": 10.38} |
| {"step": 163, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21099251879341901, "tokens": 120000, "cumulative_loss_tokens": 19560000, "grad_norm": 1.171875, "lr": 3e-05, "finish_rate": 0.108, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.3, "t_step_s": 71.9, "t_refresh_s": 0.3, "mem_gb": 10.41} |
| {"step": 164, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2188319661433498, "tokens": 120000, "cumulative_loss_tokens": 19680000, "grad_norm": 1.1953125, "lr": 3e-05, "finish_rate": 0.136, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.7, "t_step_s": 70.1, "t_refresh_s": 0.3, "mem_gb": 10.47} |
| {"step": 165, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21699987841347854, "tokens": 120000, "cumulative_loss_tokens": 19800000, "grad_norm": 1.1640625, "lr": 3e-05, "finish_rate": 0.137, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.7, "t_step_s": 72.0, "t_refresh_s": 0.3, "mem_gb": 10.54} |
| {"step": 166, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20422210597346227, "tokens": 120000, "cumulative_loss_tokens": 19920000, "grad_norm": 1.1875, "lr": 3e-05, "finish_rate": 0.088, "comp_len": 502.1, "t_data_s": 0.0, "t_rollout_s": 34.4, "t_step_s": 69.9, "t_refresh_s": 0.3, "mem_gb": 10.36} |
| {"step": 167, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.21616086491048336, "tokens": 120000, "cumulative_loss_tokens": 20040000, "grad_norm": 1.109375, "lr": 3e-05, "finish_rate": 0.108, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.9, "t_step_s": 72.8, "t_refresh_s": 0.3, "mem_gb": 10.49} |
| {"step": 168, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2041427251155178, "tokens": 120000, "cumulative_loss_tokens": 20160000, "grad_norm": 1.1796875, "lr": 3e-05, "finish_rate": 0.117, "comp_len": 500.0, "t_data_s": 0.0, "t_rollout_s": 35.2, "t_step_s": 71.1, "t_refresh_s": 0.3, "mem_gb": 10.39} |
| {"step": 169, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.2007420184203113, "tokens": 120000, "cumulative_loss_tokens": 20280000, "grad_norm": 1.0546875, "lr": 3e-05, "finish_rate": 0.113, "comp_len": 504.2, "t_data_s": 0.0, "t_rollout_s": 36.2, "t_step_s": 72.4, "t_refresh_s": 0.3, "mem_gb": 10.51} |
| {"step": 170, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.22800080033155778, "tokens": 120000, "cumulative_loss_tokens": 20400000, "grad_norm": 1.40625, "lr": 3e-05, "finish_rate": 0.153, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.6, "t_step_s": 69.9, "t_refresh_s": 0.3, "mem_gb": 10.42} |
| [eval step 170] sample: 'To solve this problem, we need to carefully analyze the constraints and use reasoning to determine the maximum number of maples that can be planted.\n\n |
| {"step": 170, "gsm8k_n": 64, "gsm8k_quick_chat": 0.34375, "t_eval_s": 44.8} |
| {"step": 171, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.20762226390354335, "tokens": 120000, "cumulative_loss_tokens": 20520000, "grad_norm": 1.2421875, "lr": 3e-05, "finish_rate": 0.112, "comp_len": 497.9, "t_data_s": 0.0, "t_rollout_s": 34.9, "t_step_s": 72.1, "t_refresh_s": 0.3, "mem_gb": 12.43} |
| {"step": 172, "epoch": 0, "training_mode": "on-policy", "reverse_kl": 0.19745559071643898, "tokens": 120000, "cumulative_loss_tokens": 20640000, "grad_norm": 1.1875, "lr": 3e-05, "finish_rate": 0.145, "comp_len": 495.9, "t_data_s": 0.0, "t_rollout_s": 33.4, "t_step_s": 69.5, "t_refresh_s": 0.3, "mem_gb": 10.38} |
|
|
