| [13:30:32] trial=exp273_puzzle_finetune arch=squares64 device=cuda vocab=1968 |
| [13:30:32] train bs=384 accum=1 grad_ckpt=False compile=True elo_every=0 save_every=250 |
| [13:30:32] model=Squares64RecurrentConfig(encoder_dim=256, hidden_dim=736, num_heads=8, ffn_ratio=4, dropout=0.05, use_swiglu=True, use_qk_norm=True, zero_init_out_proj=True, gradient_checkpointing=False, prefix_layers=4, recurrent_layers=7, recurrent_unrolls=3, suffix_layers=4, policy_head_dim=384, value_hidden=384, n_value_classes=3, use_history=False, use_threat_head=False) |
| [13:30:33] params=98.97M |
| [13:30:33] WEIGHTS-ONLY WARM START /root/transform/outputs/exp273_puzzle_finetune/init.pt from eval/SWA snapshot (optimizer/RNG re-initialized; step reset to 0) |
| [13:30:33] optimizer=PolarNorMuon (97.7M) + AdamW aux (1.3M) compile_polar=True |
| [13:30:33] force_lr muon=0.002 adam=3e-05 |
| [13:30:34] torch.compile enabled (mode=default); will fall back if inductor fails |
| [13:30:34] external eval puzzles n=2,824,449 /root/transform/outputs/exp273_puzzle_finetune/puzzle_eval.pt |
| [13:30:47] live soft dropped 60,418 internal hash dups |
| [13:30:47] soft train=11,134,864 val=0 blocked=0 targets empty=0 unnorm=0 external_eval |
| [13:32:03] batch probe fill_vram bs=384 peak=25.94GB < 40.0; grow bs=528 |
| [13:33:17] batch probe ok bs=528 peak_vram=35.34GB compile=True |
| [13:33:17] average_recurrent_grads() divides bank .grad by unroll count. That does NOT divide optimizer updates by the same amount: Polar-NorMuon is scale-invariant after orthogonalization, and RMS-Adam (m/sqrt(v)) is also approximately scale-invariant for a stationary grad. What does scale is SGD-style updates and the pre-clip global grad norm (so clipping fires less often after averaging). Use explicit param-group LRs if different update sizes are intended. |
| [13:33:17] exposure deep/shallow odds=0.0x (mix=0.0 deep_n=0 shallow_n=11134864) |
| [13:33:25] step 1/25000 | loss=0.9958 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=528/0/0 q=0 p=0 | 59 pos/s | vram=35.34GB |
| [13:33:47] step 25/25000 | loss=0.7251 | lr=2.00e-03,3.00e-05 | clip=24/25 | mix s/d/b=13200/0/0 q=0 p=0 | 431 pos/s | vram=35.48GB |
| [13:34:10] step 50/25000 | loss=0.5213 | lr=2.00e-03,3.00e-05 | clip=4/25 | mix s/d/b=26400/0/0 q=0 p=0 | 494 pos/s | vram=35.48GB |
| [13:34:33] step 75/25000 | loss=0.4780 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=39600/0/0 q=0 p=0 | 520 pos/s | vram=35.48GB |
| [13:34:56] step 100/25000 | loss=0.4653 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=52800/0/0 q=0 p=0 | 533 pos/s | vram=35.48GB |
| [13:35:18] step 125/25000 | loss=0.4490 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=66000/0/0 q=0 p=0 | 542 pos/s | vram=35.48GB |
| [13:35:41] step 150/25000 | loss=0.4426 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=79200/0/0 q=0 p=0 | 548 pos/s | vram=35.48GB |
| [13:36:04] step 175/25000 | loss=0.4205 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=92400/0/0 q=0 p=0 | 552 pos/s | vram=35.48GB |
| [13:36:27] step 200/25000 | loss=0.4196 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=105600/0/0 q=0 p=0 | 556 pos/s | vram=35.48GB |
| [13:36:49] step 225/25000 | loss=0.4091 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=118800/0/0 q=0 p=0 | 558 pos/s | vram=35.48GB |
| [13:37:12] step 250/25000 | loss=0.4259 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=132000/0/0 q=0 p=0 | 560 pos/s | vram=35.48GB |
| [13:37:13] ckpt latest.pt (live weights) steps=250 status=mid |
| [13:37:15] val/puzzles hard_ce=0.3695 soft_ce=0.3695 soft_temp_ce=0.3695 teacher_entropy=0.0000 teacher_kl=0.3695 value_rows=0.0000 |
| [13:37:38] step 275/25000 | loss=0.4060 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=145200/0/0 q=0 p=0 | 556 pos/s | vram=35.48GB |
| [13:38:00] step 300/25000 | loss=0.4065 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=158400/0/0 q=0 p=0 | 558 pos/s | vram=35.48GB |
| [13:38:23] step 325/25000 | loss=0.4078 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=171600/0/0 q=0 p=0 | 559 pos/s | vram=35.48GB |
| [13:38:46] step 350/25000 | loss=0.4021 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=184800/0/0 q=0 p=0 | 561 pos/s | vram=35.48GB |
| [13:39:09] step 375/25000 | loss=0.3947 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=198000/0/0 q=0 p=0 | 562 pos/s | vram=35.48GB |
| [13:39:32] step 400/25000 | loss=0.3842 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=211200/0/0 q=0 p=0 | 563 pos/s | vram=35.48GB |
| [13:39:54] step 425/25000 | loss=0.3865 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=224400/0/0 q=0 p=0 | 564 pos/s | vram=35.48GB |
| [13:40:17] step 450/25000 | loss=0.3908 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=237600/0/0 q=0 p=0 | 565 pos/s | vram=35.48GB |
| [13:40:40] step 475/25000 | loss=0.4010 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=250800/0/0 q=0 p=0 | 566 pos/s | vram=35.48GB |
| [13:41:03] step 500/25000 | loss=0.3888 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=264000/0/0 q=0 p=0 | 566 pos/s | vram=35.48GB |
| [13:41:03] ckpt latest.pt (live weights) steps=500 status=mid |
| [13:41:05] val/puzzles hard_ce=0.3455 soft_ce=0.3455 soft_temp_ce=0.3455 teacher_entropy=0.0000 teacher_kl=0.3455 value_rows=0.0000 |
| [13:41:28] step 525/25000 | loss=0.3806 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=277200/0/0 q=0 p=0 | 564 pos/s | vram=35.48GB |
| [13:41:51] step 550/25000 | loss=0.3844 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=290400/0/0 q=0 p=0 | 564 pos/s | vram=35.48GB |
| [13:42:14] step 575/25000 | loss=0.3872 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=303600/0/0 q=0 p=0 | 565 pos/s | vram=35.48GB |
| [13:42:37] step 600/25000 | loss=0.3948 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=316800/0/0 q=0 p=0 | 566 pos/s | vram=35.48GB |
| [13:42:59] step 625/25000 | loss=0.3849 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=330000/0/0 q=0 p=0 | 566 pos/s | vram=35.48GB |
| [13:43:22] step 650/25000 | loss=0.3793 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=343200/0/0 q=0 p=0 | 567 pos/s | vram=35.48GB |
| [13:43:45] step 675/25000 | loss=0.3702 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=356400/0/0 q=0 p=0 | 567 pos/s | vram=35.48GB |
| [13:44:08] step 700/25000 | loss=0.3823 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=369600/0/0 q=0 p=0 | 568 pos/s | vram=35.48GB |
| [13:44:30] step 725/25000 | loss=0.3730 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=382800/0/0 q=0 p=0 | 568 pos/s | vram=35.48GB |
| [13:44:53] step 750/25000 | loss=0.3811 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=396000/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:44:54] ckpt latest.pt (live weights) steps=750 status=mid |
| [13:44:56] val/puzzles hard_ce=0.3389 soft_ce=0.3389 soft_temp_ce=0.3389 teacher_entropy=0.0000 teacher_kl=0.3389 value_rows=0.0000 |
| [13:45:19] step 775/25000 | loss=0.3713 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=409200/0/0 q=0 p=0 | 567 pos/s | vram=35.48GB |
| [13:45:41] step 800/25000 | loss=0.3711 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=422400/0/0 q=0 p=0 | 567 pos/s | vram=35.48GB |
| [13:46:04] step 825/25000 | loss=0.3672 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=435600/0/0 q=0 p=0 | 567 pos/s | vram=35.48GB |
| [13:46:27] step 850/25000 | loss=0.3746 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=448800/0/0 q=0 p=0 | 568 pos/s | vram=35.48GB |
| [13:46:50] step 875/25000 | loss=0.3620 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=462000/0/0 q=0 p=0 | 568 pos/s | vram=35.48GB |
| [13:47:12] step 900/25000 | loss=0.3616 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=475200/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:47:35] step 925/25000 | loss=0.3628 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=488400/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:47:58] step 950/25000 | loss=0.3661 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=501600/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:48:21] step 975/25000 | loss=0.3710 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=514800/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:48:43] step 1000/25000 | loss=0.3567 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=528000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:48:44] ckpt step_001000.pt + latest.pt (live) steps=1000 status=mid kind=full |
| [13:48:47] val/puzzles hard_ce=0.3181 soft_ce=0.3181 soft_temp_ce=0.3181 teacher_entropy=0.0000 teacher_kl=0.3181 value_rows=0.0000 |
| [13:49:09] step 1025/25000 | loss=0.3531 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=541200/0/0 q=0 p=0 | 568 pos/s | vram=35.48GB |
| [13:49:32] step 1050/25000 | loss=0.3685 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=554400/0/0 q=0 p=0 | 568 pos/s | vram=35.48GB |
| [13:49:55] step 1075/25000 | loss=0.3636 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=567600/0/0 q=0 p=0 | 568 pos/s | vram=35.48GB |
| [13:50:18] step 1100/25000 | loss=0.3561 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=580800/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:50:41] step 1125/25000 | loss=0.3418 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=594000/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:51:03] step 1150/25000 | loss=0.3634 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=607200/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:51:26] step 1175/25000 | loss=0.3587 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=620400/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:51:49] step 1200/25000 | loss=0.3544 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=633600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:52:12] step 1225/25000 | loss=0.3439 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=646800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:52:34] step 1250/25000 | loss=0.3477 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=660000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:52:35] ckpt latest.pt (live weights) steps=1250 status=mid |
| [13:52:37] val/puzzles hard_ce=0.3072 soft_ce=0.3072 soft_temp_ce=0.3072 teacher_entropy=0.0000 teacher_kl=0.3072 value_rows=0.0000 |
| [13:53:00] step 1275/25000 | loss=0.3472 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=673200/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:53:23] step 1300/25000 | loss=0.3454 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=686400/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:53:46] step 1325/25000 | loss=0.3432 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=699600/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:54:08] step 1350/25000 | loss=0.3534 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=712800/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:54:31] step 1375/25000 | loss=0.3471 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=726000/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:54:54] step 1400/25000 | loss=0.3473 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=739200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:55:17] step 1425/25000 | loss=0.3483 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=752400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:55:40] step 1450/25000 | loss=0.3617 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=765600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:56:02] step 1475/25000 | loss=0.3456 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=778800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:56:25] step 1500/25000 | loss=0.3438 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=792000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:56:26] ckpt latest.pt (live weights) steps=1500 status=mid |
| [13:56:28] val/puzzles hard_ce=0.2983 soft_ce=0.2983 soft_temp_ce=0.2983 teacher_entropy=0.0000 teacher_kl=0.2983 value_rows=0.0000 |
| [13:56:51] step 1525/25000 | loss=0.3465 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=805200/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:57:14] step 1550/25000 | loss=0.3474 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=818400/0/0 q=0 p=0 | 569 pos/s | vram=35.48GB |
| [13:57:36] step 1575/25000 | loss=0.3410 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=831600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:57:59] step 1600/25000 | loss=0.3278 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=844800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:58:22] step 1625/25000 | loss=0.3338 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=858000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:58:45] step 1650/25000 | loss=0.3465 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=871200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:59:08] step 1675/25000 | loss=0.3236 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=884400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:59:30] step 1700/25000 | loss=0.3372 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=897600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [13:59:53] step 1725/25000 | loss=0.3315 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=910800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:00:16] step 1750/25000 | loss=0.3401 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=924000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:00:17] ckpt latest.pt (live weights) steps=1750 status=mid |
| [14:00:19] val/puzzles hard_ce=0.3009 soft_ce=0.3009 soft_temp_ce=0.3009 teacher_entropy=0.0000 teacher_kl=0.3009 value_rows=0.0000 |
| [14:00:42] step 1775/25000 | loss=0.3170 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=937200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:01:04] step 1800/25000 | loss=0.3352 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=950400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:01:27] step 1825/25000 | loss=0.3453 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=963600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:01:50] step 1850/25000 | loss=0.3476 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=976800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:02:13] step 1875/25000 | loss=0.3423 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=990000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:02:36] step 1900/25000 | loss=0.3257 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1003200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:02:58] step 1925/25000 | loss=0.3399 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1016400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:03:21] step 1950/25000 | loss=0.3416 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1029600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:03:44] step 1975/25000 | loss=0.3472 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1042800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:04:07] step 2000/25000 | loss=0.3251 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1056000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:04:08] ckpt step_002000.pt + latest.pt (live) steps=2000 status=mid kind=full |
| [14:04:10] val/puzzles hard_ce=0.3034 soft_ce=0.3034 soft_temp_ce=0.3034 teacher_entropy=0.0000 teacher_kl=0.3034 value_rows=0.0000 |
| [14:04:33] step 2025/25000 | loss=0.3217 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1069200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:04:56] step 2050/25000 | loss=0.3233 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1082400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:05:18] step 2075/25000 | loss=0.3218 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1095600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:05:41] step 2100/25000 | loss=0.3284 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1108800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:06:04] step 2125/25000 | loss=0.3370 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1122000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:06:27] step 2150/25000 | loss=0.3277 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1135200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:06:50] step 2175/25000 | loss=0.3250 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1148400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:07:13] step 2200/25000 | loss=0.3353 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1161600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:07:35] step 2225/25000 | loss=0.3145 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1174800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:07:58] step 2250/25000 | loss=0.3223 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1188000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:07:59] ckpt latest.pt (live weights) steps=2250 status=mid |
| [14:08:01] val/puzzles hard_ce=0.2998 soft_ce=0.2998 soft_temp_ce=0.2998 teacher_entropy=0.0000 teacher_kl=0.2998 value_rows=0.0000 |
| [14:08:24] step 2275/25000 | loss=0.3225 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1201200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:08:47] step 2300/25000 | loss=0.3203 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1214400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:09:09] step 2325/25000 | loss=0.3161 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1227600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:09:32] step 2350/25000 | loss=0.3381 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1240800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:09:55] step 2375/25000 | loss=0.3337 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1254000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:10:18] step 2400/25000 | loss=0.3327 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1267200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:10:41] step 2425/25000 | loss=0.3217 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1280400/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:11:03] step 2450/25000 | loss=0.3161 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1293600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:11:26] step 2475/25000 | loss=0.3263 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1306800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:11:49] step 2500/25000 | loss=0.3251 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1320000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:11:50] ckpt latest.pt (live weights) steps=2500 status=mid |
| [14:11:52] val/puzzles hard_ce=0.3009 soft_ce=0.3009 soft_temp_ce=0.3009 teacher_entropy=0.0000 teacher_kl=0.3009 value_rows=0.0000 |
| [14:12:15] step 2525/25000 | loss=0.3216 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1333200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:12:38] step 2550/25000 | loss=0.3239 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1346400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:13:00] step 2575/25000 | loss=0.3234 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1359600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:13:23] step 2600/25000 | loss=0.3207 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1372800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:13:46] step 2625/25000 | loss=0.3264 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1386000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:14:09] step 2650/25000 | loss=0.3238 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1399200/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:14:32] step 2675/25000 | loss=0.3120 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1412400/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:14:55] step 2700/25000 | loss=0.3116 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1425600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:15:17] step 2725/25000 | loss=0.3226 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1438800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:15:40] step 2750/25000 | loss=0.3151 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1452000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:15:41] ckpt latest.pt (live weights) steps=2750 status=mid |
| [14:15:43] val/puzzles hard_ce=0.2811 soft_ce=0.2811 soft_temp_ce=0.2811 teacher_entropy=0.0000 teacher_kl=0.2811 value_rows=0.0000 |
| [14:16:06] step 2775/25000 | loss=0.3035 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1465200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:16:29] step 2800/25000 | loss=0.3226 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1478400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:16:52] step 2825/25000 | loss=0.3182 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1491600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:17:14] step 2850/25000 | loss=0.3241 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1504800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:17:37] step 2875/25000 | loss=0.3122 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1518000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:18:00] step 2900/25000 | loss=0.3269 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1531200/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:18:23] step 2925/25000 | loss=0.3108 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1544400/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:18:46] step 2950/25000 | loss=0.3086 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1557600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:19:08] step 2975/25000 | loss=0.3315 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1570800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:19:31] step 3000/25000 | loss=0.3148 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1584000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:19:32] ckpt step_003000.pt + latest.pt (live) steps=3000 status=mid kind=full |
| [14:19:35] val/puzzles hard_ce=0.2793 soft_ce=0.2793 soft_temp_ce=0.2793 teacher_entropy=0.0000 teacher_kl=0.2793 value_rows=0.0000 |
| [14:19:57] step 3025/25000 | loss=0.3195 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1597200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:20:20] step 3050/25000 | loss=0.3150 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1610400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:20:43] step 3075/25000 | loss=0.3111 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1623600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:21:06] step 3100/25000 | loss=0.3182 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1636800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:21:29] step 3125/25000 | loss=0.3121 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1650000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:21:51] step 3150/25000 | loss=0.3254 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1663200/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:22:14] step 3175/25000 | loss=0.2964 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1676400/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:22:37] step 3200/25000 | loss=0.3249 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1689600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:23:00] step 3225/25000 | loss=0.3204 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1702800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:23:23] step 3250/25000 | loss=0.3038 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1716000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:23:23] ckpt latest.pt (live weights) steps=3250 status=mid |
| [14:23:25] val/puzzles hard_ce=0.2797 soft_ce=0.2797 soft_temp_ce=0.2797 teacher_entropy=0.0000 teacher_kl=0.2797 value_rows=0.0000 |
| [14:23:48] step 3275/25000 | loss=0.3007 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1729200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:24:11] step 3300/25000 | loss=0.3118 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1742400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:24:34] step 3325/25000 | loss=0.3077 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1755600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:24:57] step 3350/25000 | loss=0.3273 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1768800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:25:20] step 3375/25000 | loss=0.3123 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1782000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:25:42] step 3400/25000 | loss=0.3079 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1795200/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:26:05] step 3425/25000 | loss=0.3049 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1808400/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:26:28] step 3450/25000 | loss=0.3075 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1821600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:26:51] step 3475/25000 | loss=0.3041 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1834800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:27:14] step 3500/25000 | loss=0.3077 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1848000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:27:14] ckpt latest.pt (live weights) steps=3500 status=mid |
| [14:27:17] val/puzzles hard_ce=0.2776 soft_ce=0.2776 soft_temp_ce=0.2776 teacher_entropy=0.0000 teacher_kl=0.2776 value_rows=0.0000 |
| [14:27:39] step 3525/25000 | loss=0.3065 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1861200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:28:02] step 3550/25000 | loss=0.3155 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1874400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:28:25] step 3575/25000 | loss=0.3037 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1887600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:28:48] step 3600/25000 | loss=0.3110 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1900800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:29:11] step 3625/25000 | loss=0.3072 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1914000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:29:33] step 3650/25000 | loss=0.3152 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1927200/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:29:56] step 3675/25000 | loss=0.2986 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1940400/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:30:19] step 3700/25000 | loss=0.2906 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1953600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:30:42] step 3725/25000 | loss=0.3137 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1966800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:31:06] step 3750/25000 | loss=0.3062 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1980000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:31:07] ckpt latest.pt (live weights) steps=3750 status=mid |
| [14:31:09] val/puzzles hard_ce=0.2801 soft_ce=0.2801 soft_temp_ce=0.2801 teacher_entropy=0.0000 teacher_kl=0.2801 value_rows=0.0000 |
| [14:31:32] step 3775/25000 | loss=0.3176 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=1993200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:31:55] step 3800/25000 | loss=0.3009 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2006400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:32:18] step 3825/25000 | loss=0.3004 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2019600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:32:40] step 3850/25000 | loss=0.3074 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2032800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:33:03] step 3875/25000 | loss=0.2976 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2046000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:33:26] step 3900/25000 | loss=0.3123 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2059200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:33:49] step 3925/25000 | loss=0.3108 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2072400/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:34:12] step 3950/25000 | loss=0.3077 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2085600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:34:35] step 3975/25000 | loss=0.3036 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2098800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:34:58] step 4000/25000 | loss=0.3050 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2112000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:34:59] ckpt step_004000.pt + latest.pt (live) steps=4000 status=mid kind=full |
| [14:35:01] val/puzzles hard_ce=0.2812 soft_ce=0.2812 soft_temp_ce=0.2812 teacher_entropy=0.0000 teacher_kl=0.2812 value_rows=0.0000 |
| [14:35:24] step 4025/25000 | loss=0.3111 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2125200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:35:47] step 4050/25000 | loss=0.2996 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2138400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:36:09] step 4075/25000 | loss=0.3096 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2151600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:36:32] step 4100/25000 | loss=0.3107 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2164800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:36:55] step 4125/25000 | loss=0.2974 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2178000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:37:18] step 4150/25000 | loss=0.2968 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=2191200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:37:41] step 4175/25000 | loss=0.3016 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2204400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:38:04] step 4200/25000 | loss=0.2995 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2217600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:38:26] step 4225/25000 | loss=0.2980 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2230800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:38:49] step 4250/25000 | loss=0.2934 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2244000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:38:50] ckpt latest.pt (live weights) steps=4250 status=mid |
| [14:38:52] val/puzzles hard_ce=0.2745 soft_ce=0.2745 soft_temp_ce=0.2745 teacher_entropy=0.0000 teacher_kl=0.2745 value_rows=0.0000 |
| [14:39:15] step 4275/25000 | loss=0.2996 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2257200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:39:38] step 4300/25000 | loss=0.3061 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2270400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:40:01] step 4325/25000 | loss=0.2989 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2283600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:40:24] step 4350/25000 | loss=0.3069 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2296800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:40:46] step 4375/25000 | loss=0.2972 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2310000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:41:09] step 4400/25000 | loss=0.2886 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2323200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:41:32] step 4425/25000 | loss=0.3063 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2336400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:41:55] step 4450/25000 | loss=0.3112 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2349600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:42:18] step 4475/25000 | loss=0.2967 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2362800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:42:41] step 4500/25000 | loss=0.2937 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2376000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:42:41] ckpt latest.pt (live weights) steps=4500 status=mid |
| [14:42:43] val/puzzles hard_ce=0.2740 soft_ce=0.2740 soft_temp_ce=0.2740 teacher_entropy=0.0000 teacher_kl=0.2740 value_rows=0.0000 |
| [14:43:06] step 4525/25000 | loss=0.2870 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2389200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:43:29] step 4550/25000 | loss=0.3029 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2402400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:43:52] step 4575/25000 | loss=0.2891 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2415600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:44:15] step 4600/25000 | loss=0.2965 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2428800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:44:38] step 4625/25000 | loss=0.2894 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2442000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:45:01] step 4650/25000 | loss=0.2990 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2455200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:45:23] step 4675/25000 | loss=0.3087 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2468400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:45:46] step 4700/25000 | loss=0.3106 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=2481600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:46:09] step 4725/25000 | loss=0.2917 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2494800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:46:32] step 4750/25000 | loss=0.3088 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2508000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:46:33] ckpt latest.pt (live weights) steps=4750 status=mid |
| [14:46:35] val/puzzles hard_ce=0.2731 soft_ce=0.2731 soft_temp_ce=0.2731 teacher_entropy=0.0000 teacher_kl=0.2731 value_rows=0.0000 |
| [14:46:58] step 4775/25000 | loss=0.2905 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2521200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:47:21] step 4800/25000 | loss=0.3086 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2534400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:47:43] step 4825/25000 | loss=0.2974 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2547600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:48:06] step 4850/25000 | loss=0.2826 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2560800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:48:29] step 4875/25000 | loss=0.3007 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2574000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:48:52] step 4900/25000 | loss=0.2966 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2587200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:49:15] step 4925/25000 | loss=0.2928 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2600400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:49:38] step 4950/25000 | loss=0.2917 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2613600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:50:00] step 4975/25000 | loss=0.2959 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2626800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:50:23] step 5000/25000 | loss=0.2906 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2640000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:50:24] ckpt step_005000.pt + latest.pt (live) steps=5000 status=mid kind=full |
| [14:50:27] val/puzzles hard_ce=0.2654 soft_ce=0.2654 soft_temp_ce=0.2654 teacher_entropy=0.0000 teacher_kl=0.2654 value_rows=0.0000 |
| [14:50:50] step 5025/25000 | loss=0.2976 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2653200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:51:12] step 5050/25000 | loss=0.2870 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2666400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:51:35] step 5075/25000 | loss=0.3023 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2679600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:51:58] step 5100/25000 | loss=0.3015 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2692800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:52:21] step 5125/25000 | loss=0.2870 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2706000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:52:44] step 5150/25000 | loss=0.2791 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2719200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:53:07] step 5175/25000 | loss=0.2855 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2732400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:53:30] step 5200/25000 | loss=0.2940 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2745600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:53:52] step 5225/25000 | loss=0.2922 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2758800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:54:15] step 5250/25000 | loss=0.2970 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2772000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [14:54:16] ckpt latest.pt (live weights) steps=5250 status=mid |
| [14:54:18] val/puzzles hard_ce=0.2742 soft_ce=0.2742 soft_temp_ce=0.2742 teacher_entropy=0.0000 teacher_kl=0.2742 value_rows=0.0000 |
| [14:54:41] step 5275/25000 | loss=0.2920 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2785200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:55:04] step 5300/25000 | loss=0.2877 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2798400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:55:27] step 5325/25000 | loss=0.2826 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2811600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:55:50] step 5350/25000 | loss=0.2952 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2824800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:56:12] step 5375/25000 | loss=0.2905 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2838000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:56:35] step 5400/25000 | loss=0.2788 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2851200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:56:58] step 5425/25000 | loss=0.3050 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2864400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:57:21] step 5450/25000 | loss=0.2985 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2877600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:57:44] step 5475/25000 | loss=0.2878 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2890800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:58:07] step 5500/25000 | loss=0.2798 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2904000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:58:08] ckpt latest.pt (live weights) steps=5500 status=mid |
| [14:58:10] val/puzzles hard_ce=0.2647 soft_ce=0.2647 soft_temp_ce=0.2647 teacher_entropy=0.0000 teacher_kl=0.2647 value_rows=0.0000 |
| [14:58:33] step 5525/25000 | loss=0.2843 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2917200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:58:56] step 5550/25000 | loss=0.2845 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2930400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:59:18] step 5575/25000 | loss=0.2841 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2943600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [14:59:41] step 5600/25000 | loss=0.2849 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2956800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:00:04] step 5625/25000 | loss=0.2873 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2970000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:00:27] step 5650/25000 | loss=0.2963 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2983200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:00:50] step 5675/25000 | loss=0.2904 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=2996400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:01:13] step 5700/25000 | loss=0.2756 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3009600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:01:36] step 5725/25000 | loss=0.2883 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3022800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:01:58] step 5750/25000 | loss=0.2788 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3036000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:01:59] ckpt latest.pt (live weights) steps=5750 status=mid |
| [15:02:01] val/puzzles hard_ce=0.2655 soft_ce=0.2655 soft_temp_ce=0.2655 teacher_entropy=0.0000 teacher_kl=0.2655 value_rows=0.0000 |
| [15:02:24] step 5775/25000 | loss=0.2937 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3049200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:02:47] step 5800/25000 | loss=0.2844 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3062400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:03:10] step 5825/25000 | loss=0.2927 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3075600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:03:32] step 5850/25000 | loss=0.2939 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3088800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:03:55] step 5875/25000 | loss=0.2870 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3102000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:04:18] step 5900/25000 | loss=0.2861 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3115200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:04:41] step 5925/25000 | loss=0.2905 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3128400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:05:04] step 5950/25000 | loss=0.2781 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3141600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:05:27] step 5975/25000 | loss=0.2914 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3154800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:05:50] step 6000/25000 | loss=0.2805 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3168000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:05:51] ckpt step_006000.pt + latest.pt (live) steps=6000 status=mid kind=full |
| [15:05:53] val/puzzles hard_ce=0.2699 soft_ce=0.2699 soft_temp_ce=0.2699 teacher_entropy=0.0000 teacher_kl=0.2699 value_rows=0.0000 |
| [15:06:16] step 6025/25000 | loss=0.2839 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3181200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:06:39] step 6050/25000 | loss=0.2804 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3194400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:07:01] step 6075/25000 | loss=0.2759 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3207600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:07:24] step 6100/25000 | loss=0.2833 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3220800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:07:47] step 6125/25000 | loss=0.2842 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3234000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:08:10] step 6150/25000 | loss=0.2800 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3247200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:08:33] step 6175/25000 | loss=0.2954 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3260400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:08:56] step 6200/25000 | loss=0.2854 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3273600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:09:19] step 6225/25000 | loss=0.2819 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3286800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:09:41] step 6250/25000 | loss=0.2791 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3300000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:09:42] ckpt latest.pt (live weights) steps=6250 status=mid |
| [15:09:44] val/puzzles hard_ce=0.2591 soft_ce=0.2591 soft_temp_ce=0.2591 teacher_entropy=0.0000 teacher_kl=0.2591 value_rows=0.0000 |
| [15:10:07] step 6275/25000 | loss=0.2781 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3313200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:10:30] step 6300/25000 | loss=0.2824 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3326400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:10:53] step 6325/25000 | loss=0.3021 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3339600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:11:16] step 6350/25000 | loss=0.2884 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3352800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:11:39] step 6375/25000 | loss=0.2789 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3366000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:12:01] step 6400/25000 | loss=0.2869 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3379200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:12:24] step 6425/25000 | loss=0.2964 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3392400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:12:47] step 6450/25000 | loss=0.2770 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3405600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:13:10] step 6475/25000 | loss=0.2843 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3418800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:13:33] step 6500/25000 | loss=0.2788 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3432000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:13:33] ckpt latest.pt (live weights) steps=6500 status=mid |
| [15:13:36] val/puzzles hard_ce=0.2591 soft_ce=0.2591 soft_temp_ce=0.2591 teacher_entropy=0.0000 teacher_kl=0.2591 value_rows=0.0000 |
| [15:13:58] step 6525/25000 | loss=0.2817 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3445200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:14:21] step 6550/25000 | loss=0.2793 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3458400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:14:44] step 6575/25000 | loss=0.2687 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3471600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:15:07] step 6600/25000 | loss=0.2747 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3484800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:15:30] step 6625/25000 | loss=0.2758 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3498000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:15:53] step 6650/25000 | loss=0.2713 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3511200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:16:15] step 6675/25000 | loss=0.2765 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3524400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:16:38] step 6700/25000 | loss=0.2892 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3537600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:17:01] step 6725/25000 | loss=0.2728 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3550800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:17:24] step 6750/25000 | loss=0.2749 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3564000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:17:25] ckpt latest.pt (live weights) steps=6750 status=mid |
| [15:17:27] val/puzzles hard_ce=0.2579 soft_ce=0.2579 soft_temp_ce=0.2579 teacher_entropy=0.0000 teacher_kl=0.2579 value_rows=0.0000 |
| [15:17:50] step 6775/25000 | loss=0.2851 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3577200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:18:13] step 6800/25000 | loss=0.2749 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3590400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:18:35] step 6825/25000 | loss=0.2772 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3603600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:18:58] step 6850/25000 | loss=0.2716 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3616800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:19:21] step 6875/25000 | loss=0.2727 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3630000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:19:44] step 6900/25000 | loss=0.2800 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3643200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:20:07] step 6925/25000 | loss=0.2737 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3656400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:20:30] step 6950/25000 | loss=0.2722 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3669600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:20:53] step 6975/25000 | loss=0.2911 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3682800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:21:16] step 7000/25000 | loss=0.2789 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3696000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:21:17] ckpt step_007000.pt + latest.pt (live) steps=7000 status=mid kind=full |
| [15:21:19] val/puzzles hard_ce=0.2623 soft_ce=0.2623 soft_temp_ce=0.2623 teacher_entropy=0.0000 teacher_kl=0.2623 value_rows=0.0000 |
| [15:21:42] step 7025/25000 | loss=0.2715 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3709200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:22:05] step 7050/25000 | loss=0.2780 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3722400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:22:28] step 7075/25000 | loss=0.2737 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3735600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:22:51] step 7100/25000 | loss=0.2893 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3748800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:23:13] step 7125/25000 | loss=0.2700 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3762000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:23:36] step 7150/25000 | loss=0.2735 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3775200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:23:59] step 7175/25000 | loss=0.2758 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3788400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:24:22] step 7200/25000 | loss=0.2738 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3801600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:24:45] step 7225/25000 | loss=0.2803 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3814800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:25:08] step 7250/25000 | loss=0.2723 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3828000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:25:08] ckpt latest.pt (live weights) steps=7250 status=mid |
| [15:25:10] val/puzzles hard_ce=0.2524 soft_ce=0.2524 soft_temp_ce=0.2524 teacher_entropy=0.0000 teacher_kl=0.2524 value_rows=0.0000 |
| [15:25:33] step 7275/25000 | loss=0.2810 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3841200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:25:56] step 7300/25000 | loss=0.2738 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3854400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:26:19] step 7325/25000 | loss=0.2696 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3867600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:26:42] step 7350/25000 | loss=0.2750 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3880800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:27:04] step 7375/25000 | loss=0.2782 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3894000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:27:27] step 7400/25000 | loss=0.2833 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3907200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:27:50] step 7425/25000 | loss=0.2740 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3920400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:28:13] step 7450/25000 | loss=0.2760 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3933600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:28:36] step 7475/25000 | loss=0.2862 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3946800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:28:58] step 7500/25000 | loss=0.2759 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3960000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:28:59] ckpt latest.pt (live weights) steps=7500 status=mid |
| [15:29:01] val/puzzles hard_ce=0.2534 soft_ce=0.2534 soft_temp_ce=0.2534 teacher_entropy=0.0000 teacher_kl=0.2534 value_rows=0.0000 |
| [15:29:24] step 7525/25000 | loss=0.2834 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3973200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:29:47] step 7550/25000 | loss=0.2743 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3986400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:30:09] step 7575/25000 | loss=0.2879 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=3999600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:30:32] step 7600/25000 | loss=0.2785 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4012800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:30:55] step 7625/25000 | loss=0.2823 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4026000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:31:18] step 7650/25000 | loss=0.2724 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4039200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:31:40] step 7675/25000 | loss=0.2704 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4052400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:32:03] step 7700/25000 | loss=0.2794 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4065600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:32:26] step 7725/25000 | loss=0.2687 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4078800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:32:49] step 7750/25000 | loss=0.2710 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4092000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:32:49] ckpt latest.pt (live weights) steps=7750 status=mid |
| [15:32:52] val/puzzles hard_ce=0.2558 soft_ce=0.2558 soft_temp_ce=0.2558 teacher_entropy=0.0000 teacher_kl=0.2558 value_rows=0.0000 |
| [15:33:14] step 7775/25000 | loss=0.2707 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4105200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:33:37] step 7800/25000 | loss=0.2742 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4118400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:34:00] step 7825/25000 | loss=0.2766 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4131600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:34:23] step 7850/25000 | loss=0.2754 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4144800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:34:46] step 7875/25000 | loss=0.2724 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4158000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:35:09] step 7900/25000 | loss=0.2764 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4171200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:35:32] step 7925/25000 | loss=0.2657 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4184400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:35:54] step 7950/25000 | loss=0.2687 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4197600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:36:17] step 7975/25000 | loss=0.2657 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4210800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:36:40] step 8000/25000 | loss=0.2618 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4224000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:36:41] ckpt step_008000.pt + latest.pt (live) steps=8000 status=mid kind=full |
| [15:36:44] val/puzzles hard_ce=0.2555 soft_ce=0.2555 soft_temp_ce=0.2555 teacher_entropy=0.0000 teacher_kl=0.2555 value_rows=0.0000 |
| [15:37:06] step 8025/25000 | loss=0.2676 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4237200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:37:29] step 8050/25000 | loss=0.2706 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4250400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:37:52] step 8075/25000 | loss=0.2769 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4263600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:38:15] step 8100/25000 | loss=0.2787 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4276800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:38:38] step 8125/25000 | loss=0.2637 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4290000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:39:01] step 8150/25000 | loss=0.2710 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4303200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:39:23] step 8175/25000 | loss=0.2734 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4316400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:39:46] step 8200/25000 | loss=0.2697 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4329600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:40:09] step 8225/25000 | loss=0.2769 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4342800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:40:32] step 8250/25000 | loss=0.2658 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4356000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:40:32] ckpt latest.pt (live weights) steps=8250 status=mid |
| [15:40:35] val/puzzles hard_ce=0.2547 soft_ce=0.2547 soft_temp_ce=0.2547 teacher_entropy=0.0000 teacher_kl=0.2547 value_rows=0.0000 |
| [15:40:57] step 8275/25000 | loss=0.2673 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4369200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:41:20] step 8300/25000 | loss=0.2696 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4382400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:41:43] step 8325/25000 | loss=0.2563 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4395600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:42:06] step 8350/25000 | loss=0.2682 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4408800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:42:29] step 8375/25000 | loss=0.2594 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4422000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:42:52] step 8400/25000 | loss=0.2623 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4435200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:43:14] step 8425/25000 | loss=0.2789 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4448400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:43:37] step 8450/25000 | loss=0.2608 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4461600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:44:00] step 8475/25000 | loss=0.2672 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4474800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:44:23] step 8500/25000 | loss=0.2637 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4488000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:44:23] ckpt latest.pt (live weights) steps=8500 status=mid |
| [15:44:26] val/puzzles hard_ce=0.2487 soft_ce=0.2487 soft_temp_ce=0.2487 teacher_entropy=0.0000 teacher_kl=0.2487 value_rows=0.0000 |
| [15:44:48] step 8525/25000 | loss=0.2699 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4501200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:45:11] step 8550/25000 | loss=0.2564 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4514400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:45:34] step 8575/25000 | loss=0.2647 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4527600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:45:57] step 8600/25000 | loss=0.2565 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4540800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:46:20] step 8625/25000 | loss=0.2688 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4554000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:46:43] step 8650/25000 | loss=0.2569 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4567200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:47:05] step 8675/25000 | loss=0.2606 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4580400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:47:28] step 8700/25000 | loss=0.2642 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4593600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:47:51] step 8725/25000 | loss=0.2637 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4606800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:48:14] step 8750/25000 | loss=0.2650 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4620000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:48:15] ckpt latest.pt (live weights) steps=8750 status=mid |
| [15:48:17] val/puzzles hard_ce=0.2543 soft_ce=0.2543 soft_temp_ce=0.2543 teacher_entropy=0.0000 teacher_kl=0.2543 value_rows=0.0000 |
| [15:48:40] step 8775/25000 | loss=0.2657 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4633200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:49:03] step 8800/25000 | loss=0.2663 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4646400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:49:25] step 8825/25000 | loss=0.2649 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4659600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:49:48] step 8850/25000 | loss=0.2654 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4672800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:50:11] step 8875/25000 | loss=0.2643 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4686000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:50:34] step 8900/25000 | loss=0.2617 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4699200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:50:57] step 8925/25000 | loss=0.2684 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4712400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:51:20] step 8950/25000 | loss=0.2677 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4725600/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:51:42] step 8975/25000 | loss=0.2642 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4738800/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:52:05] step 9000/25000 | loss=0.2618 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4752000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [15:52:06] ckpt step_009000.pt + latest.pt (live) steps=9000 status=mid kind=full |
| [15:52:09] val/puzzles hard_ce=0.2463 soft_ce=0.2463 soft_temp_ce=0.2463 teacher_entropy=0.0000 teacher_kl=0.2463 value_rows=0.0000 |
| [15:52:32] step 9025/25000 | loss=0.2541 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4765200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:52:54] step 9050/25000 | loss=0.2644 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4778400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:53:17] step 9075/25000 | loss=0.2676 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4791600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:53:40] step 9100/25000 | loss=0.2627 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4804800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:54:03] step 9125/25000 | loss=0.2479 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4818000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:54:26] step 9150/25000 | loss=0.2633 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4831200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:54:49] step 9175/25000 | loss=0.2700 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4844400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:55:12] step 9200/25000 | loss=0.2614 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4857600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:55:35] step 9225/25000 | loss=0.2568 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4870800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:55:57] step 9250/25000 | loss=0.2553 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=4884000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:55:58] ckpt latest.pt (live weights) steps=9250 status=mid |
| [15:56:00] val/puzzles hard_ce=0.2414 soft_ce=0.2414 soft_temp_ce=0.2414 teacher_entropy=0.0000 teacher_kl=0.2414 value_rows=0.0000 |
| [15:56:23] step 9275/25000 | loss=0.2614 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4897200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:56:46] step 9300/25000 | loss=0.2519 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4910400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:57:09] step 9325/25000 | loss=0.2585 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=4923600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:57:32] step 9350/25000 | loss=0.2574 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4936800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:57:55] step 9375/25000 | loss=0.2604 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4950000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:58:18] step 9400/25000 | loss=0.2687 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4963200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:58:41] step 9425/25000 | loss=0.2584 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4976400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:59:04] step 9450/25000 | loss=0.2763 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=4989600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:59:26] step 9475/25000 | loss=0.2669 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5002800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:59:49] step 9500/25000 | loss=0.2622 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5016000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [15:59:50] ckpt latest.pt (live weights) steps=9500 status=mid |
| [15:59:52] val/puzzles hard_ce=0.2469 soft_ce=0.2469 soft_temp_ce=0.2469 teacher_entropy=0.0000 teacher_kl=0.2469 value_rows=0.0000 |
| [16:00:15] step 9525/25000 | loss=0.2789 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5029200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:00:38] step 9550/25000 | loss=0.2599 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5042400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:01:01] step 9575/25000 | loss=0.2546 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=5055600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:01:24] step 9600/25000 | loss=0.2679 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5068800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:01:47] step 9625/25000 | loss=0.2591 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5082000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:02:09] step 9650/25000 | loss=0.2710 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5095200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:02:32] step 9675/25000 | loss=0.2526 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5108400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:02:55] step 9700/25000 | loss=0.2519 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5121600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:03:18] step 9725/25000 | loss=0.2626 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5134800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:03:41] step 9750/25000 | loss=0.2654 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5148000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:03:41] ckpt latest.pt (live weights) steps=9750 status=mid |
| [16:03:44] val/puzzles hard_ce=0.2471 soft_ce=0.2471 soft_temp_ce=0.2471 teacher_entropy=0.0000 teacher_kl=0.2471 value_rows=0.0000 |
| [16:04:06] step 9775/25000 | loss=0.2532 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5161200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:04:29] step 9800/25000 | loss=0.2524 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5174400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:04:52] step 9825/25000 | loss=0.2533 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5187600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:05:15] step 9850/25000 | loss=0.2690 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5200800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:05:38] step 9875/25000 | loss=0.2607 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5214000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:06:00] step 9900/25000 | loss=0.2754 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5227200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:06:23] step 9925/25000 | loss=0.2596 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5240400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:06:46] step 9950/25000 | loss=0.2527 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5253600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:07:09] step 9975/25000 | loss=0.2595 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5266800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:07:32] step 10000/25000 | loss=0.2611 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5280000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:07:33] ckpt step_010000.pt + latest.pt (live) steps=10000 status=mid kind=full |
| [16:07:35] val/puzzles hard_ce=0.2447 soft_ce=0.2447 soft_temp_ce=0.2447 teacher_entropy=0.0000 teacher_kl=0.2447 value_rows=0.0000 |
| [16:07:58] step 10025/25000 | loss=0.2576 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5293200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:08:21] step 10050/25000 | loss=0.2606 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5306400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:08:44] step 10075/25000 | loss=0.2656 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5319600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:09:07] step 10100/25000 | loss=0.2564 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5332800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:09:29] step 10125/25000 | loss=0.2510 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5346000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:09:52] step 10150/25000 | loss=0.2573 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=5359200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:10:15] step 10175/25000 | loss=0.2499 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5372400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:10:38] step 10200/25000 | loss=0.2503 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5385600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:11:01] step 10225/25000 | loss=0.2520 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5398800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:11:24] step 10250/25000 | loss=0.2579 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5412000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:11:24] ckpt latest.pt (live weights) steps=10250 status=mid |
| [16:11:26] val/puzzles hard_ce=0.2465 soft_ce=0.2465 soft_temp_ce=0.2465 teacher_entropy=0.0000 teacher_kl=0.2465 value_rows=0.0000 |
| [16:11:49] step 10275/25000 | loss=0.2621 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5425200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:12:12] step 10300/25000 | loss=0.2639 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5438400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:12:35] step 10325/25000 | loss=0.2585 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5451600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:12:58] step 10350/25000 | loss=0.2507 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5464800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:13:21] step 10375/25000 | loss=0.2633 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5478000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:13:44] step 10400/25000 | loss=0.2459 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5491200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:14:06] step 10425/25000 | loss=0.2692 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5504400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:14:29] step 10450/25000 | loss=0.2575 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5517600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:14:52] step 10475/25000 | loss=0.2669 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5530800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:15:15] step 10500/25000 | loss=0.2471 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5544000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:15:15] ckpt latest.pt (live weights) steps=10500 status=mid |
| [16:15:18] val/puzzles hard_ce=0.2419 soft_ce=0.2419 soft_temp_ce=0.2419 teacher_entropy=0.0000 teacher_kl=0.2419 value_rows=0.0000 |
| [16:15:41] step 10525/25000 | loss=0.2552 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5557200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:16:03] step 10550/25000 | loss=0.2504 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5570400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:16:26] step 10575/25000 | loss=0.2587 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5583600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:16:49] step 10600/25000 | loss=0.2568 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5596800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:17:12] step 10625/25000 | loss=0.2517 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5610000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:17:35] step 10650/25000 | loss=0.2512 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5623200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:17:57] step 10675/25000 | loss=0.2565 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5636400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:18:20] step 10700/25000 | loss=0.2504 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5649600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:18:43] step 10725/25000 | loss=0.2573 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5662800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:19:06] step 10750/25000 | loss=0.2638 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5676000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:19:06] ckpt latest.pt (live weights) steps=10750 status=mid |
| [16:19:09] val/puzzles hard_ce=0.2386 soft_ce=0.2386 soft_temp_ce=0.2386 teacher_entropy=0.0000 teacher_kl=0.2386 value_rows=0.0000 |
| [16:19:32] step 10775/25000 | loss=0.2461 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5689200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:19:54] step 10800/25000 | loss=0.2499 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5702400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:20:17] step 10825/25000 | loss=0.2633 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5715600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:20:40] step 10850/25000 | loss=0.2549 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5728800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:21:03] step 10875/25000 | loss=0.2581 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5742000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:21:26] step 10900/25000 | loss=0.2634 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5755200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:21:48] step 10925/25000 | loss=0.2523 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5768400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:22:11] step 10950/25000 | loss=0.2565 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5781600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:22:34] step 10975/25000 | loss=0.2602 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5794800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:22:57] step 11000/25000 | loss=0.2635 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5808000/0/0 q=0 p=0 | 571 pos/s | vram=35.48GB |
| [16:22:58] ckpt step_011000.pt + latest.pt (live) steps=11000 status=mid kind=full |
| [16:23:00] val/puzzles hard_ce=0.2443 soft_ce=0.2443 soft_temp_ce=0.2443 teacher_entropy=0.0000 teacher_kl=0.2443 value_rows=0.0000 |
| [16:23:23] step 11025/25000 | loss=0.2604 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5821200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:23:46] step 11050/25000 | loss=0.2465 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5834400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:24:09] step 11075/25000 | loss=0.2462 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5847600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:24:32] step 11100/25000 | loss=0.2501 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5860800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:24:54] step 11125/25000 | loss=0.2426 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5874000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:25:17] step 11150/25000 | loss=0.2496 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5887200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:25:40] step 11175/25000 | loss=0.2582 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5900400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:26:03] step 11200/25000 | loss=0.2642 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5913600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:26:26] step 11225/25000 | loss=0.2494 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5926800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:26:49] step 11250/25000 | loss=0.2530 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5940000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:26:49] ckpt latest.pt (live weights) steps=11250 status=mid |
| [16:26:52] val/puzzles hard_ce=0.2322 soft_ce=0.2322 soft_temp_ce=0.2322 teacher_entropy=0.0000 teacher_kl=0.2322 value_rows=0.0000 |
| [16:27:15] step 11275/25000 | loss=0.2525 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5953200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:27:37] step 11300/25000 | loss=0.2563 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5966400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:28:00] step 11325/25000 | loss=0.2403 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5979600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:28:23] step 11350/25000 | loss=0.2549 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=5992800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:28:46] step 11375/25000 | loss=0.2558 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6006000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:29:09] step 11400/25000 | loss=0.2508 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6019200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:29:32] step 11425/25000 | loss=0.2524 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6032400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:29:55] step 11450/25000 | loss=0.2457 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6045600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:30:18] step 11475/25000 | loss=0.2450 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6058800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:30:41] step 11500/25000 | loss=0.2548 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6072000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:30:41] ckpt latest.pt (live weights) steps=11500 status=mid |
| [16:30:44] val/puzzles hard_ce=0.2368 soft_ce=0.2368 soft_temp_ce=0.2368 teacher_entropy=0.0000 teacher_kl=0.2368 value_rows=0.0000 |
| [16:31:06] step 11525/25000 | loss=0.2482 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6085200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:31:29] step 11550/25000 | loss=0.2504 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6098400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:31:52] step 11575/25000 | loss=0.2488 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6111600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:32:15] step 11600/25000 | loss=0.2584 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6124800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:32:38] step 11625/25000 | loss=0.2458 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6138000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:33:01] step 11650/25000 | loss=0.2536 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6151200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:33:24] step 11675/25000 | loss=0.2497 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6164400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:33:47] step 11700/25000 | loss=0.2532 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6177600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:34:10] step 11725/25000 | loss=0.2396 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6190800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:34:33] step 11750/25000 | loss=0.2459 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6204000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:34:33] ckpt latest.pt (live weights) steps=11750 status=mid |
| [16:34:35] val/puzzles hard_ce=0.2433 soft_ce=0.2433 soft_temp_ce=0.2433 teacher_entropy=0.0000 teacher_kl=0.2433 value_rows=0.0000 |
| [16:34:58] step 11775/25000 | loss=0.2561 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6217200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:35:21] step 11800/25000 | loss=0.2367 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6230400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:35:44] step 11825/25000 | loss=0.2490 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6243600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:36:07] step 11850/25000 | loss=0.2445 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6256800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:36:30] step 11875/25000 | loss=0.2594 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6270000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:36:53] step 11900/25000 | loss=0.2531 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6283200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:37:16] step 11925/25000 | loss=0.2522 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6296400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:37:39] step 11950/25000 | loss=0.2459 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6309600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:38:01] step 11975/25000 | loss=0.2483 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6322800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:38:24] step 12000/25000 | loss=0.2525 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6336000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:38:25] ckpt step_012000.pt + latest.pt (live) steps=12000 status=mid kind=full |
| [16:38:28] val/puzzles hard_ce=0.2247 soft_ce=0.2247 soft_temp_ce=0.2247 teacher_entropy=0.0000 teacher_kl=0.2247 value_rows=0.0000 |
| [16:38:51] step 12025/25000 | loss=0.2424 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6349200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:39:13] step 12050/25000 | loss=0.2412 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6362400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:39:36] step 12075/25000 | loss=0.2544 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6375600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:39:59] step 12100/25000 | loss=0.2492 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6388800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:40:22] step 12125/25000 | loss=0.2481 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6402000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:40:45] step 12150/25000 | loss=0.2428 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6415200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:41:08] step 12175/25000 | loss=0.2640 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6428400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:41:31] step 12200/25000 | loss=0.2445 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6441600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:41:54] step 12225/25000 | loss=0.2438 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6454800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:42:16] step 12250/25000 | loss=0.2395 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6468000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:42:17] ckpt latest.pt (live weights) steps=12250 status=mid |
| [16:42:19] val/puzzles hard_ce=0.2299 soft_ce=0.2299 soft_temp_ce=0.2299 teacher_entropy=0.0000 teacher_kl=0.2299 value_rows=0.0000 |
| [16:42:42] step 12275/25000 | loss=0.2554 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6481200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:43:05] step 12300/25000 | loss=0.2514 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6494400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:43:28] step 12325/25000 | loss=0.2426 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6507600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:43:51] step 12350/25000 | loss=0.2438 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6520800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:44:14] step 12375/25000 | loss=0.2475 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6534000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:44:37] step 12400/25000 | loss=0.2470 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6547200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:44:59] step 12425/25000 | loss=0.2404 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6560400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:45:22] step 12450/25000 | loss=0.2379 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6573600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:45:45] step 12475/25000 | loss=0.2500 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6586800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:46:08] step 12500/25000 | loss=0.2478 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6600000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:46:09] ckpt latest.pt (live weights) steps=12500 status=mid |
| [16:46:11] val/puzzles hard_ce=0.2341 soft_ce=0.2341 soft_temp_ce=0.2341 teacher_entropy=0.0000 teacher_kl=0.2341 value_rows=0.0000 |
| [16:46:34] step 12525/25000 | loss=0.2529 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6613200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:46:57] step 12550/25000 | loss=0.2421 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6626400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:47:20] step 12575/25000 | loss=0.2498 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6639600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:47:43] step 12600/25000 | loss=0.2381 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6652800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:48:05] step 12625/25000 | loss=0.2502 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6666000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:48:28] step 12650/25000 | loss=0.2518 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6679200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:48:51] step 12675/25000 | loss=0.2441 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6692400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:49:14] step 12700/25000 | loss=0.2482 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6705600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:49:37] step 12725/25000 | loss=0.2441 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6718800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:50:00] step 12750/25000 | loss=0.2469 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6732000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:50:00] ckpt latest.pt (live weights) steps=12750 status=mid |
| [16:50:03] val/puzzles hard_ce=0.2289 soft_ce=0.2289 soft_temp_ce=0.2289 teacher_entropy=0.0000 teacher_kl=0.2289 value_rows=0.0000 |
| [16:50:26] step 12775/25000 | loss=0.2540 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6745200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:50:48] step 12800/25000 | loss=0.2463 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6758400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:51:11] step 12825/25000 | loss=0.2338 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6771600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:51:34] step 12850/25000 | loss=0.2489 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6784800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:51:57] step 12875/25000 | loss=0.2398 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6798000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:52:20] step 12900/25000 | loss=0.2450 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6811200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:52:43] step 12925/25000 | loss=0.2521 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6824400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:53:06] step 12950/25000 | loss=0.2466 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6837600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:53:29] step 12975/25000 | loss=0.2485 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6850800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:53:52] step 13000/25000 | loss=0.2467 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6864000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:53:53] ckpt step_013000.pt + latest.pt (live) steps=13000 status=mid kind=full |
| [16:53:55] val/puzzles hard_ce=0.2195 soft_ce=0.2195 soft_temp_ce=0.2195 teacher_entropy=0.0000 teacher_kl=0.2195 value_rows=0.0000 |
| [16:54:18] step 13025/25000 | loss=0.2473 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6877200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:54:41] step 13050/25000 | loss=0.2448 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6890400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:55:04] step 13075/25000 | loss=0.2390 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6903600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:55:26] step 13100/25000 | loss=0.2386 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6916800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:55:49] step 13125/25000 | loss=0.2471 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6930000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:56:12] step 13150/25000 | loss=0.2415 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6943200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:56:35] step 13175/25000 | loss=0.2399 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6956400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:56:58] step 13200/25000 | loss=0.2406 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6969600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:57:21] step 13225/25000 | loss=0.2330 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6982800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:57:44] step 13250/25000 | loss=0.2402 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=6996000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:57:44] ckpt latest.pt (live weights) steps=13250 status=mid |
| [16:57:47] val/puzzles hard_ce=0.2313 soft_ce=0.2313 soft_temp_ce=0.2313 teacher_entropy=0.0000 teacher_kl=0.2313 value_rows=0.0000 |
| [16:58:09] step 13275/25000 | loss=0.2399 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7009200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:58:32] step 13300/25000 | loss=0.2366 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7022400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:58:55] step 13325/25000 | loss=0.2485 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7035600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:59:19] step 13350/25000 | loss=0.2366 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7048800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [16:59:42] step 13375/25000 | loss=0.2390 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7062000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:00:05] step 13400/25000 | loss=0.2467 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7075200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:00:28] step 13425/25000 | loss=0.2328 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7088400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:00:51] step 13450/25000 | loss=0.2413 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7101600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:01:13] step 13475/25000 | loss=0.2417 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7114800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:01:36] step 13500/25000 | loss=0.2460 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7128000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:01:37] ckpt latest.pt (live weights) steps=13500 status=mid |
| [17:01:39] val/puzzles hard_ce=0.2281 soft_ce=0.2281 soft_temp_ce=0.2281 teacher_entropy=0.0000 teacher_kl=0.2281 value_rows=0.0000 |
| [17:02:02] step 13525/25000 | loss=0.2432 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7141200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:02:25] step 13550/25000 | loss=0.2457 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7154400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:02:48] step 13575/25000 | loss=0.2485 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7167600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:03:10] step 13600/25000 | loss=0.2406 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7180800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:03:33] step 13625/25000 | loss=0.2354 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7194000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:03:56] step 13650/25000 | loss=0.2370 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7207200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:04:19] step 13675/25000 | loss=0.2350 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7220400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:04:42] step 13700/25000 | loss=0.2446 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7233600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:05:04] step 13725/25000 | loss=0.2350 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7246800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:05:27] step 13750/25000 | loss=0.2401 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7260000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:05:28] ckpt latest.pt (live weights) steps=13750 status=mid |
| [17:05:30] val/puzzles hard_ce=0.2263 soft_ce=0.2263 soft_temp_ce=0.2263 teacher_entropy=0.0000 teacher_kl=0.2263 value_rows=0.0000 |
| [17:05:53] step 13775/25000 | loss=0.2355 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7273200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:06:16] step 13800/25000 | loss=0.2298 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7286400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:06:39] step 13825/25000 | loss=0.2314 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7299600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:07:02] step 13850/25000 | loss=0.2426 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7312800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:07:25] step 13875/25000 | loss=0.2331 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7326000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:07:47] step 13900/25000 | loss=0.2468 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7339200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:08:10] step 13925/25000 | loss=0.2361 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7352400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:08:33] step 13950/25000 | loss=0.2416 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7365600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:08:56] step 13975/25000 | loss=0.2400 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7378800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:09:19] step 14000/25000 | loss=0.2478 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7392000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:09:20] ckpt step_014000.pt + latest.pt (live) steps=14000 status=mid kind=full |
| [17:09:22] val/puzzles hard_ce=0.2318 soft_ce=0.2318 soft_temp_ce=0.2318 teacher_entropy=0.0000 teacher_kl=0.2318 value_rows=0.0000 |
| [17:09:45] step 14025/25000 | loss=0.2434 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7405200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:10:08] step 14050/25000 | loss=0.2271 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7418400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:10:31] step 14075/25000 | loss=0.2422 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7431600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:10:54] step 14100/25000 | loss=0.2362 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7444800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:11:17] step 14125/25000 | loss=0.2267 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7458000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:11:40] step 14150/25000 | loss=0.2342 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7471200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:12:03] step 14175/25000 | loss=0.2485 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7484400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:12:26] step 14200/25000 | loss=0.2409 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7497600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:12:49] step 14225/25000 | loss=0.2462 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7510800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:13:11] step 14250/25000 | loss=0.2422 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7524000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:13:12] ckpt latest.pt (live weights) steps=14250 status=mid |
| [17:13:14] val/puzzles hard_ce=0.2376 soft_ce=0.2376 soft_temp_ce=0.2376 teacher_entropy=0.0000 teacher_kl=0.2376 value_rows=0.0000 |
| [17:13:37] step 14275/25000 | loss=0.2365 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7537200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:14:00] step 14300/25000 | loss=0.2412 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7550400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:14:23] step 14325/25000 | loss=0.2388 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7563600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:14:46] step 14350/25000 | loss=0.2360 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7576800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:15:09] step 14375/25000 | loss=0.2402 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7590000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:15:31] step 14400/25000 | loss=0.2369 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7603200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:15:54] step 14425/25000 | loss=0.2266 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7616400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:16:17] step 14450/25000 | loss=0.2296 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7629600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:16:40] step 14475/25000 | loss=0.2331 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7642800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:17:03] step 14500/25000 | loss=0.2431 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7656000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:17:03] ckpt latest.pt (live weights) steps=14500 status=mid |
| [17:17:06] val/puzzles hard_ce=0.2277 soft_ce=0.2277 soft_temp_ce=0.2277 teacher_entropy=0.0000 teacher_kl=0.2277 value_rows=0.0000 |
| [17:17:28] step 14525/25000 | loss=0.2249 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7669200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:17:51] step 14550/25000 | loss=0.2380 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7682400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:18:14] step 14575/25000 | loss=0.2377 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7695600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:18:37] step 14600/25000 | loss=0.2310 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7708800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:19:00] step 14625/25000 | loss=0.2354 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7722000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:19:23] step 14650/25000 | loss=0.2447 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7735200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:19:46] step 14675/25000 | loss=0.2319 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7748400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:20:08] step 14700/25000 | loss=0.2430 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7761600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:20:31] step 14725/25000 | loss=0.2379 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7774800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:20:54] step 14750/25000 | loss=0.2382 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7788000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:20:55] ckpt latest.pt (live weights) steps=14750 status=mid |
| [17:20:57] val/puzzles hard_ce=0.2197 soft_ce=0.2197 soft_temp_ce=0.2197 teacher_entropy=0.0000 teacher_kl=0.2197 value_rows=0.0000 |
| [17:21:20] step 14775/25000 | loss=0.2433 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7801200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:21:43] step 14800/25000 | loss=0.2456 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7814400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:22:06] step 14825/25000 | loss=0.2338 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7827600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:22:28] step 14850/25000 | loss=0.2352 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7840800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:22:51] step 14875/25000 | loss=0.2299 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7854000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:23:14] step 14900/25000 | loss=0.2424 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7867200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:23:37] step 14925/25000 | loss=0.2423 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7880400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:24:00] step 14950/25000 | loss=0.2316 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7893600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:24:23] step 14975/25000 | loss=0.2518 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7906800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:24:46] step 15000/25000 | loss=0.2416 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7920000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:24:47] ckpt step_015000.pt + latest.pt (live) steps=15000 status=mid kind=full |
| [17:24:49] val/puzzles hard_ce=0.2312 soft_ce=0.2312 soft_temp_ce=0.2312 teacher_entropy=0.0000 teacher_kl=0.2312 value_rows=0.0000 |
| [17:25:12] step 15025/25000 | loss=0.2289 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7933200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:25:35] step 15050/25000 | loss=0.2419 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7946400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:25:57] step 15075/25000 | loss=0.2354 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7959600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:26:20] step 15100/25000 | loss=0.2397 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7972800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:26:43] step 15125/25000 | loss=0.2308 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7986000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:27:06] step 15150/25000 | loss=0.2362 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=7999200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:27:29] step 15175/25000 | loss=0.2367 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8012400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:27:52] step 15200/25000 | loss=0.2291 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8025600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:28:15] step 15225/25000 | loss=0.2392 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8038800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:28:38] step 15250/25000 | loss=0.2379 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8052000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:28:38] ckpt latest.pt (live weights) steps=15250 status=mid |
| [17:28:41] val/puzzles hard_ce=0.2281 soft_ce=0.2281 soft_temp_ce=0.2281 teacher_entropy=0.0000 teacher_kl=0.2281 value_rows=0.0000 |
| [17:29:03] step 15275/25000 | loss=0.2418 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8065200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:29:26] step 15300/25000 | loss=0.2359 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8078400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:29:49] step 15325/25000 | loss=0.2372 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8091600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:30:12] step 15350/25000 | loss=0.2333 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8104800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:30:35] step 15375/25000 | loss=0.2395 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8118000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:30:58] step 15400/25000 | loss=0.2370 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8131200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:31:20] step 15425/25000 | loss=0.2371 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8144400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:31:43] step 15450/25000 | loss=0.2327 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8157600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:32:06] step 15475/25000 | loss=0.2394 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8170800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:32:29] step 15500/25000 | loss=0.2334 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8184000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:32:30] ckpt latest.pt (live weights) steps=15500 status=mid |
| [17:32:32] val/puzzles hard_ce=0.2319 soft_ce=0.2319 soft_temp_ce=0.2319 teacher_entropy=0.0000 teacher_kl=0.2319 value_rows=0.0000 |
| [17:32:55] step 15525/25000 | loss=0.2333 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8197200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:33:18] step 15550/25000 | loss=0.2346 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8210400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:33:40] step 15575/25000 | loss=0.2351 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8223600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:34:03] step 15600/25000 | loss=0.2432 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8236800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:34:26] step 15625/25000 | loss=0.2388 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8250000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:34:49] step 15650/25000 | loss=0.2348 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8263200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:35:12] step 15675/25000 | loss=0.2337 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8276400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:35:34] step 15700/25000 | loss=0.2263 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8289600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:35:57] step 15725/25000 | loss=0.2283 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8302800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:36:20] step 15750/25000 | loss=0.2402 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8316000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:36:20] ckpt latest.pt (live weights) steps=15750 status=mid |
| [17:36:23] val/puzzles hard_ce=0.2316 soft_ce=0.2316 soft_temp_ce=0.2316 teacher_entropy=0.0000 teacher_kl=0.2316 value_rows=0.0000 |
| [17:36:46] step 15775/25000 | loss=0.2263 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8329200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:37:08] step 15800/25000 | loss=0.2266 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8342400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:37:31] step 15825/25000 | loss=0.2291 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8355600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:37:54] step 15850/25000 | loss=0.2285 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8368800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:38:17] step 15875/25000 | loss=0.2229 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8382000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:38:40] step 15900/25000 | loss=0.2441 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8395200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:39:03] step 15925/25000 | loss=0.2291 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8408400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:39:26] step 15950/25000 | loss=0.2313 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8421600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:39:49] step 15975/25000 | loss=0.2338 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8434800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:40:12] step 16000/25000 | loss=0.2364 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8448000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:40:13] ckpt step_016000.pt + latest.pt (live) steps=16000 status=mid kind=full |
| [17:40:15] val/puzzles hard_ce=0.2224 soft_ce=0.2224 soft_temp_ce=0.2224 teacher_entropy=0.0000 teacher_kl=0.2224 value_rows=0.0000 |
| [17:40:38] step 16025/25000 | loss=0.2342 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8461200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:41:01] step 16050/25000 | loss=0.2378 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8474400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:41:23] step 16075/25000 | loss=0.2395 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8487600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:41:46] step 16100/25000 | loss=0.2252 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8500800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:42:09] step 16125/25000 | loss=0.2245 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8514000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:42:32] step 16150/25000 | loss=0.2243 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8527200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:42:55] step 16175/25000 | loss=0.2384 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8540400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:43:18] step 16200/25000 | loss=0.2340 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8553600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:43:41] step 16225/25000 | loss=0.2307 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8566800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:44:03] step 16250/25000 | loss=0.2354 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8580000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:44:04] ckpt latest.pt (live weights) steps=16250 status=mid |
| [17:44:06] val/puzzles hard_ce=0.2247 soft_ce=0.2247 soft_temp_ce=0.2247 teacher_entropy=0.0000 teacher_kl=0.2247 value_rows=0.0000 |
| [17:44:29] step 16275/25000 | loss=0.2285 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8593200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:44:52] step 16300/25000 | loss=0.2328 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8606400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:45:15] step 16325/25000 | loss=0.2320 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8619600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:45:38] step 16350/25000 | loss=0.2360 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8632800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:46:01] step 16375/25000 | loss=0.2354 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8646000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:46:23] step 16400/25000 | loss=0.2249 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8659200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:46:46] step 16425/25000 | loss=0.2292 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8672400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:47:09] step 16450/25000 | loss=0.2355 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8685600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:47:32] step 16475/25000 | loss=0.2186 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8698800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:47:55] step 16500/25000 | loss=0.2315 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8712000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:47:55] ckpt latest.pt (live weights) steps=16500 status=mid |
| [17:47:58] val/puzzles hard_ce=0.2241 soft_ce=0.2241 soft_temp_ce=0.2241 teacher_entropy=0.0000 teacher_kl=0.2241 value_rows=0.0000 |
| [17:48:21] step 16525/25000 | loss=0.2314 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8725200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:48:43] step 16550/25000 | loss=0.2331 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8738400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:49:06] step 16575/25000 | loss=0.2338 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8751600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:49:29] step 16600/25000 | loss=0.2359 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8764800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:49:52] step 16625/25000 | loss=0.2320 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8778000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:50:15] step 16650/25000 | loss=0.2310 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8791200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:50:38] step 16675/25000 | loss=0.2309 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8804400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:51:00] step 16700/25000 | loss=0.2214 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8817600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:51:23] step 16725/25000 | loss=0.2389 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8830800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:51:46] step 16750/25000 | loss=0.2339 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8844000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:51:47] ckpt latest.pt (live weights) steps=16750 status=mid |
| [17:51:49] val/puzzles hard_ce=0.2207 soft_ce=0.2207 soft_temp_ce=0.2207 teacher_entropy=0.0000 teacher_kl=0.2207 value_rows=0.0000 |
| [17:52:12] step 16775/25000 | loss=0.2290 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8857200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:52:35] step 16800/25000 | loss=0.2247 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=8870400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:52:57] step 16825/25000 | loss=0.2252 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8883600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:53:20] step 16850/25000 | loss=0.2218 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8896800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:53:43] step 16875/25000 | loss=0.2287 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8910000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:54:06] step 16900/25000 | loss=0.2287 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=8923200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:54:29] step 16925/25000 | loss=0.2288 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8936400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:54:52] step 16950/25000 | loss=0.2383 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8949600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:55:14] step 16975/25000 | loss=0.2389 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=8962800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:55:37] step 17000/25000 | loss=0.2239 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8976000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:55:39] ckpt step_017000.pt + latest.pt (live) steps=17000 status=mid kind=full |
| [17:55:41] val/puzzles hard_ce=0.2233 soft_ce=0.2233 soft_temp_ce=0.2233 teacher_entropy=0.0000 teacher_kl=0.2233 value_rows=0.0000 |
| [17:56:04] step 17025/25000 | loss=0.2230 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=8989200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:56:27] step 17050/25000 | loss=0.2309 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=9002400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:56:50] step 17075/25000 | loss=0.2369 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9015600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:57:13] step 17100/25000 | loss=0.2398 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9028800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:57:35] step 17125/25000 | loss=0.2213 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9042000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:57:58] step 17150/25000 | loss=0.2476 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9055200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:58:21] step 17175/25000 | loss=0.2247 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9068400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:58:44] step 17200/25000 | loss=0.2292 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9081600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:59:07] step 17225/25000 | loss=0.2109 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9094800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:59:30] step 17250/25000 | loss=0.2263 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9108000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [17:59:30] ckpt latest.pt (live weights) steps=17250 status=mid |
| [17:59:32] val/puzzles hard_ce=0.2297 soft_ce=0.2297 soft_temp_ce=0.2297 teacher_entropy=0.0000 teacher_kl=0.2297 value_rows=0.0000 |
| [17:59:55] step 17275/25000 | loss=0.2434 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9121200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:00:18] step 17300/25000 | loss=0.2195 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9134400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:00:41] step 17325/25000 | loss=0.2233 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9147600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:01:04] step 17350/25000 | loss=0.2314 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9160800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:01:27] step 17375/25000 | loss=0.2292 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9174000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:01:50] step 17400/25000 | loss=0.2180 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9187200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:02:12] step 17425/25000 | loss=0.2284 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9200400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:02:35] step 17450/25000 | loss=0.2335 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9213600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:02:58] step 17475/25000 | loss=0.2214 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9226800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:03:21] step 17500/25000 | loss=0.2238 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9240000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:03:22] ckpt latest.pt (live weights) steps=17500 status=mid |
| [18:03:24] val/puzzles hard_ce=0.2250 soft_ce=0.2250 soft_temp_ce=0.2250 teacher_entropy=0.0000 teacher_kl=0.2250 value_rows=0.0000 |
| [18:03:47] step 17525/25000 | loss=0.2185 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9253200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:04:10] step 17550/25000 | loss=0.2398 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9266400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:04:32] step 17575/25000 | loss=0.2255 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9279600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:04:55] step 17600/25000 | loss=0.2281 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9292800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:05:18] step 17625/25000 | loss=0.2340 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9306000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:05:41] step 17650/25000 | loss=0.2279 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9319200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:06:04] step 17675/25000 | loss=0.2284 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9332400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:06:27] step 17700/25000 | loss=0.2226 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9345600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:06:50] step 17725/25000 | loss=0.2292 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9358800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:07:12] step 17750/25000 | loss=0.2311 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9372000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:07:13] ckpt latest.pt (live weights) steps=17750 status=mid |
| [18:07:15] val/puzzles hard_ce=0.2250 soft_ce=0.2250 soft_temp_ce=0.2250 teacher_entropy=0.0000 teacher_kl=0.2250 value_rows=0.0000 |
| [18:07:38] step 17775/25000 | loss=0.2355 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9385200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:08:01] step 17800/25000 | loss=0.2296 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9398400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:08:24] step 17825/25000 | loss=0.2217 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9411600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:08:47] step 17850/25000 | loss=0.2191 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9424800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:09:10] step 17875/25000 | loss=0.2256 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9438000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:09:33] step 17900/25000 | loss=0.2181 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9451200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:09:56] step 17925/25000 | loss=0.2207 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9464400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:10:19] step 17950/25000 | loss=0.2169 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9477600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:10:41] step 17975/25000 | loss=0.2288 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9490800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:11:04] step 18000/25000 | loss=0.2251 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9504000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:11:06] ckpt step_018000.pt + latest.pt (live) steps=18000 status=mid kind=full |
| [18:11:08] val/puzzles hard_ce=0.2246 soft_ce=0.2246 soft_temp_ce=0.2246 teacher_entropy=0.0000 teacher_kl=0.2246 value_rows=0.0000 |
| [18:11:31] step 18025/25000 | loss=0.2273 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9517200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:11:54] step 18050/25000 | loss=0.2169 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9530400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:12:17] step 18075/25000 | loss=0.2218 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9543600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:12:40] step 18100/25000 | loss=0.2228 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9556800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:13:03] step 18125/25000 | loss=0.2246 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=9570000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:13:26] step 18150/25000 | loss=0.2325 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9583200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:13:49] step 18175/25000 | loss=0.2302 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9596400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:14:12] step 18200/25000 | loss=0.2352 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9609600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:14:35] step 18225/25000 | loss=0.2231 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9622800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:14:58] step 18250/25000 | loss=0.2327 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9636000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:14:58] ckpt latest.pt (live weights) steps=18250 status=mid |
| [18:15:00] val/puzzles hard_ce=0.2315 soft_ce=0.2315 soft_temp_ce=0.2315 teacher_entropy=0.0000 teacher_kl=0.2315 value_rows=0.0000 |
| [18:15:23] step 18275/25000 | loss=0.2218 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9649200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:15:46] step 18300/25000 | loss=0.2220 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9662400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:16:09] step 18325/25000 | loss=0.2240 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9675600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:16:32] step 18350/25000 | loss=0.2230 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9688800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:16:55] step 18375/25000 | loss=0.2300 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9702000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:17:18] step 18400/25000 | loss=0.2309 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9715200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:17:41] step 18425/25000 | loss=0.2269 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9728400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:18:04] step 18450/25000 | loss=0.2260 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9741600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:18:27] step 18475/25000 | loss=0.2289 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9754800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:18:50] step 18500/25000 | loss=0.2209 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9768000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:18:51] ckpt latest.pt (live weights) steps=18500 status=mid |
| [18:18:53] val/puzzles hard_ce=0.2303 soft_ce=0.2303 soft_temp_ce=0.2303 teacher_entropy=0.0000 teacher_kl=0.2303 value_rows=0.0000 |
| [18:19:16] step 18525/25000 | loss=0.2164 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9781200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:19:39] step 18550/25000 | loss=0.2339 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9794400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:20:02] step 18575/25000 | loss=0.2244 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9807600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:20:25] step 18600/25000 | loss=0.2210 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9820800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:20:48] step 18625/25000 | loss=0.2173 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9834000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:21:10] step 18650/25000 | loss=0.2171 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9847200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:21:33] step 18675/25000 | loss=0.2184 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9860400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:21:56] step 18700/25000 | loss=0.2217 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9873600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:22:19] step 18725/25000 | loss=0.2187 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9886800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:22:42] step 18750/25000 | loss=0.2290 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9900000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:22:43] ckpt latest.pt (live weights) steps=18750 status=mid |
| [18:22:45] val/puzzles hard_ce=0.2211 soft_ce=0.2211 soft_temp_ce=0.2211 teacher_entropy=0.0000 teacher_kl=0.2211 value_rows=0.0000 |
| [18:23:08] step 18775/25000 | loss=0.2208 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9913200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:23:31] step 18800/25000 | loss=0.2261 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9926400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:23:53] step 18825/25000 | loss=0.2341 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9939600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:24:16] step 18850/25000 | loss=0.2258 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9952800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:24:39] step 18875/25000 | loss=0.2144 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9966000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:25:02] step 18900/25000 | loss=0.2269 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9979200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:25:25] step 18925/25000 | loss=0.2223 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=9992400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:25:48] step 18950/25000 | loss=0.2306 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=10005600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:26:11] step 18975/25000 | loss=0.2127 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=10018800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:26:33] step 19000/25000 | loss=0.2229 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=10032000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:26:35] ckpt step_019000.pt + latest.pt (live) steps=19000 status=mid kind=full |
| [18:26:37] val/puzzles hard_ce=0.2317 soft_ce=0.2317 soft_temp_ce=0.2317 teacher_entropy=0.0000 teacher_kl=0.2317 value_rows=0.0000 |
| [18:27:00] step 19025/25000 | loss=0.2228 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=10045200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:27:23] step 19050/25000 | loss=0.2233 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=10058400/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:27:45] step 19075/25000 | loss=0.2229 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=10071600/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:28:08] step 19100/25000 | loss=0.2263 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=10084800/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:28:31] step 19125/25000 | loss=0.2304 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=10098000/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:28:54] step 19150/25000 | loss=0.2239 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=10111200/0/0 q=0 p=0 | 570 pos/s | vram=35.48GB |
| [18:29:00] STOP file seen at step 19156; saving and exiting |
| [18:29:01] timing wall=17743s active=17446s steps=19156 status=interrupted |
| [18:29:01] done steps=19156 pos_s=570.0 params=98.97M status=interrupted kind=full |
|
|
