train.log

W0724 12:29:28.140000 10498 torch/distributed/run.py:852] 
W0724 12:29:28.140000 10498 torch/distributed/run.py:852] *****************************************
W0724 12:29:28.140000 10498 torch/distributed/run.py:852] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0724 12:29:28.140000 10498 torch/distributed/run.py:852] *****************************************
Training GPT
  Device: cuda:0, dtype: torch.bfloat16
  Output: /dev/shm/out/vanilla-small-20B
  Sparsity mode: none
  DDP: world_size=2
  Token budget:
    tokens/iter = batch_size × seq_len × grad_accum_global = 32 × 1024 × 48 = 1,572,864
      (2 ranks × 24 micro-steps × 32 batch per rank)
    max_iters   = 12,716
    total       = 20,000,538,624 tokens (~20.00B)

Loading data from /dev/shm/data/fineweb-edu-25B...
  Train: 24,975,629,888 tokens
  Val: 24,433,651 tokens

Creating model...
  n_layer=12, n_head=8, n_embd=1024
  pos_encoding=learned
number of parameters: 202.53M
Compiling model...
number of parameters: 202.53M
NCCL version 2.27.5+cuda12.9
Muon params: 48 tensors, 150,994,944 parameters
AdamW decay params: 2 tensors, 52,559,872 parameters
AdamW no-decay params: 25 tensors, 25,600 parameters
using Muon + AdamW optimizer (fused AdamW: True)
Muon params: 48 tensors, 150,994,944 parameters
AdamW decay params: 2 tensors, 52,559,872 parameters
AdamW no-decay params: 25 tensors, 25,600 parameters
using Muon + AdamW optimizer (fused AdamW: True)
wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from /root/.netrc.
wandb: Currently logged in as: markhenry-software (markhenrysoftware) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.26.1
wandb: Run data is saved locally in /workspace/markhenry-mutagen/sparse-nanogpt-private/deeptopk/wandb/run-20260724_122939-9iezcto0
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run vanilla-small-20B
wandb: ⭐️ View project at https://wandb.ai/markhenrysoftware/sparse-nanogpt
wandb: 🚀 View run at https://wandb.ai/markhenrysoftware/sparse-nanogpt/runs/9iezcto0

Starting training from iter 0...
iter      0 | loss 11.0540 | lr 6.00e-05 | grad_norm 18.25 | 77476 tok/s
iter     10 | loss 8.4686 | lr 6.60e-04 | grad_norm 13.38 | 688710 tok/s
iter     20 | loss 7.6680 | lr 1.26e-03 | grad_norm 27.12 | 708089 tok/s
iter     30 | loss 7.0983 | lr 1.86e-03 | grad_norm 20.88 | 700854 tok/s
iter     40 | loss 6.5000 | lr 2.46e-03 | grad_norm 9.50 | 696461 tok/s
iter     50 | loss 6.1356 | lr 3.06e-03 | grad_norm 6.66 | 697653 tok/s
iter     60 | loss 5.8697 | lr 3.66e-03 | grad_norm 5.34 | 694462 tok/s
iter     70 | loss 5.7241 | lr 4.26e-03 | grad_norm 3.91 | 698866 tok/s
iter     80 | loss 5.5761 | lr 4.86e-03 | grad_norm 3.59 | 691455 tok/s
iter     90 | loss 5.4801 | lr 5.46e-03 | grad_norm 2.97 | 692340 tok/s
iter    100 | loss 5.4140 | lr 6.06e-03 | grad_norm 2.56 | 696813 tok/s
  eval | train_loss 5.3923 | val_loss 5.3944
Saved checkpoint at iter 100 -> ckpt.pt
Saved checkpoint at iter 100 -> ckpt_best.pt
  New best val_loss: 5.3944
iter    110 | loss 5.2727 | lr 6.66e-03 | grad_norm 2.16 | 424927 tok/s
iter    120 | loss 5.2040 | lr 7.26e-03 | grad_norm 2.03 | 696049 tok/s
iter    130 | loss 5.0535 | lr 7.86e-03 | grad_norm 1.41 | 693579 tok/s
iter    140 | loss 4.9408 | lr 8.46e-03 | grad_norm 1.41 | 696818 tok/s
iter    150 | loss 4.8185 | lr 9.06e-03 | grad_norm 1.38 | 696543 tok/s
iter    160 | loss 4.7056 | lr 9.66e-03 | grad_norm 1.09 | 700567 tok/s
iter    170 | loss 4.6376 | lr 1.03e-02 | grad_norm 1.21 | 699554 tok/s
iter    180 | loss 4.5670 | lr 1.09e-02 | grad_norm 1.18 | 694146 tok/s
iter    190 | loss 4.4424 | lr 1.15e-02 | grad_norm 1.16 | 694543 tok/s
iter    200 | loss 4.4108 | lr 1.20e-02 | grad_norm 1.24 | 696636 tok/s
  eval | train_loss 4.4256 | val_loss 4.4123
Saved checkpoint at iter 200 -> ckpt.pt
Saved checkpoint at iter 200 -> ckpt_best.pt
  New best val_loss: 4.4123
iter    210 | loss 4.3307 | lr 1.20e-02 | grad_norm 1.01 | 426702 tok/s
iter    220 | loss 4.2424 | lr 1.20e-02 | grad_norm 1.22 | 694370 tok/s
iter    230 | loss 4.1775 | lr 1.20e-02 | grad_norm 0.95 | 697236 tok/s
iter    240 | loss 4.1460 | lr 1.20e-02 | grad_norm 0.91 | 694919 tok/s
iter    250 | loss 4.1004 | lr 1.20e-02 | grad_norm 0.87 | 691555 tok/s
iter    260 | loss 4.0631 | lr 1.20e-02 | grad_norm 0.82 | 693950 tok/s
iter    270 | loss 4.0329 | lr 1.20e-02 | grad_norm 0.78 | 689725 tok/s
iter    280 | loss 3.9906 | lr 1.20e-02 | grad_norm 0.75 | 688356 tok/s
iter    290 | loss 3.9808 | lr 1.20e-02 | grad_norm 0.79 | 688424 tok/s
iter    300 | loss 3.9251 | lr 1.20e-02 | grad_norm 0.73 | 690322 tok/s
  eval | train_loss 3.9328 | val_loss 3.9326
Saved checkpoint at iter 300 -> ckpt.pt
Saved checkpoint at iter 300 -> ckpt_best.pt
  New best val_loss: 3.9326
iter    310 | loss 3.9237 | lr 1.20e-02 | grad_norm 0.72 | 421641 tok/s
iter    320 | loss 3.9068 | lr 1.20e-02 | grad_norm 0.66 | 691082 tok/s
iter    330 | loss 3.8567 | lr 1.20e-02 | grad_norm 0.67 | 687952 tok/s
iter    340 | loss 3.8213 | lr 1.20e-02 | grad_norm 0.63 | 690774 tok/s
iter    350 | loss 3.7992 | lr 1.20e-02 | grad_norm 0.65 | 693421 tok/s
iter    360 | loss 3.7596 | lr 1.20e-02 | grad_norm 0.61 | 693952 tok/s
iter    370 | loss 3.7780 | lr 1.20e-02 | grad_norm 0.66 | 691120 tok/s
iter    380 | loss 3.7434 | lr 1.20e-02 | grad_norm 0.63 | 690508 tok/s
iter    390 | loss 3.7297 | lr 1.20e-02 | grad_norm 0.56 | 691191 tok/s
iter    400 | loss 3.7173 | lr 1.20e-02 | grad_norm 0.57 | 690360 tok/s
  eval | train_loss 3.7044 | val_loss 3.7130
Saved checkpoint at iter 400 -> ckpt.pt
Saved checkpoint at iter 400 -> ckpt_best.pt
  New best val_loss: 3.7130
iter    410 | loss 3.6964 | lr 1.20e-02 | grad_norm 0.59 | 429402 tok/s
iter    420 | loss 3.6837 | lr 1.20e-02 | grad_norm 0.54 | 693120 tok/s
iter    430 | loss 3.6542 | lr 1.20e-02 | grad_norm 0.52 | 691228 tok/s
iter    440 | loss 3.6340 | lr 1.20e-02 | grad_norm 0.48 | 693834 tok/s
iter    450 | loss 3.6536 | lr 1.20e-02 | grad_norm 0.44 | 688859 tok/s
iter    460 | loss 3.6455 | lr 1.20e-02 | grad_norm 0.51 | 693065 tok/s
iter    470 | loss 3.6061 | lr 1.20e-02 | grad_norm 0.48 | 691798 tok/s
iter    480 | loss 3.5948 | lr 1.20e-02 | grad_norm 0.46 | 693750 tok/s
iter    490 | loss 3.5914 | lr 1.20e-02 | grad_norm 0.46 | 694379 tok/s
iter    500 | loss 3.5839 | lr 1.20e-02 | grad_norm 0.48 | 690955 tok/s
  eval | train_loss 3.5865 | val_loss 3.5860
Saved checkpoint at iter 500 -> ckpt.pt
Saved checkpoint at iter 500 -> ckpt_best.pt
  New best val_loss: 3.5860
iter    510 | loss 3.5693 | lr 1.20e-02 | grad_norm 0.43 | 427142 tok/s
iter    520 | loss 3.5719 | lr 1.20e-02 | grad_norm 0.46 | 692634 tok/s
iter    530 | loss 3.5692 | lr 1.20e-02 | grad_norm 0.42 | 689492 tok/s
iter    540 | loss 3.5383 | lr 1.20e-02 | grad_norm 0.39 | 686982 tok/s
iter    550 | loss 3.5339 | lr 1.20e-02 | grad_norm 0.42 | 690076 tok/s
iter    560 | loss 3.5204 | lr 1.20e-02 | grad_norm 0.46 | 692614 tok/s
iter    570 | loss 3.4880 | lr 1.20e-02 | grad_norm 0.39 | 694177 tok/s
iter    580 | loss 3.5089 | lr 1.20e-02 | grad_norm 0.42 | 691852 tok/s
iter    590 | loss 3.4817 | lr 1.20e-02 | grad_norm 0.38 | 694157 tok/s
iter    600 | loss 3.4876 | lr 1.20e-02 | grad_norm 0.38 | 689687 tok/s
  eval | train_loss 3.4926 | val_loss 3.4855
Saved checkpoint at iter 600 -> ckpt.pt
Saved checkpoint at iter 600 -> ckpt_best.pt
  New best val_loss: 3.4855
iter    610 | loss 3.4790 | lr 1.20e-02 | grad_norm 0.34 | 425305 tok/s
iter    620 | loss 3.4797 | lr 1.20e-02 | grad_norm 0.38 | 693723 tok/s
iter    630 | loss 3.4479 | lr 1.20e-02 | grad_norm 0.38 | 693847 tok/s
iter    640 | loss 3.4659 | lr 1.20e-02 | grad_norm 0.34 | 693813 tok/s
iter    650 | loss 3.4546 | lr 1.20e-02 | grad_norm 0.34 | 690687 tok/s
iter    660 | loss 3.4388 | lr 1.20e-02 | grad_norm 0.38 | 690326 tok/s
iter    670 | loss 3.4275 | lr 1.20e-02 | grad_norm 0.34 | 689842 tok/s
iter    680 | loss 3.4190 | lr 1.20e-02 | grad_norm 0.35 | 687299 tok/s
iter    690 | loss 3.4140 | lr 1.20e-02 | grad_norm 0.37 | 694423 tok/s
iter    700 | loss 3.4220 | lr 1.20e-02 | grad_norm 0.34 | 692950 tok/s
  eval | train_loss 3.4271 | val_loss 3.4207
Saved checkpoint at iter 700 -> ckpt.pt
Saved checkpoint at iter 700 -> ckpt_best.pt
  New best val_loss: 3.4207
iter    710 | loss 3.4258 | lr 1.20e-02 | grad_norm 0.33 | 423369 tok/s
iter    720 | loss 3.4066 | lr 1.20e-02 | grad_norm 0.33 | 691391 tok/s
iter    730 | loss 3.4192 | lr 1.20e-02 | grad_norm 0.34 | 689872 tok/s
iter    740 | loss 3.3914 | lr 1.20e-02 | grad_norm 0.31 | 691790 tok/s
iter    750 | loss 3.4145 | lr 1.20e-02 | grad_norm 0.29 | 689794 tok/s
iter    760 | loss 3.4015 | lr 1.20e-02 | grad_norm 0.32 | 688590 tok/s
iter    770 | loss 3.4054 | lr 1.20e-02 | grad_norm 0.36 | 692293 tok/s
iter    780 | loss 3.3986 | lr 1.20e-02 | grad_norm 0.34 | 691096 tok/s
iter    790 | loss 3.3720 | lr 1.20e-02 | grad_norm 0.35 | 689953 tok/s
iter    800 | loss 3.4040 | lr 1.20e-02 | grad_norm 0.32 | 693444 tok/s
  eval | train_loss 3.3782 | val_loss 3.3770
Saved checkpoint at iter 800 -> ckpt.pt
Saved checkpoint at iter 800 -> ckpt_best.pt
  New best val_loss: 3.3770
iter    810 | loss 3.3877 | lr 1.20e-02 | grad_norm 0.33 | 425735 tok/s
iter    820 | loss 3.3724 | lr 1.20e-02 | grad_norm 0.31 | 694847 tok/s
iter    830 | loss 3.3752 | lr 1.20e-02 | grad_norm 0.29 | 694688 tok/s
iter    840 | loss 3.3648 | lr 1.20e-02 | grad_norm 0.30 | 691477 tok/s
iter    850 | loss 3.3518 | lr 1.20e-02 | grad_norm 0.31 | 690923 tok/s
iter    860 | loss 3.3311 | lr 1.20e-02 | grad_norm 0.30 | 692717 tok/s
iter    870 | loss 3.3422 | lr 1.20e-02 | grad_norm 0.29 | 686827 tok/s
iter    880 | loss 3.3582 | lr 1.20e-02 | grad_norm 0.31 | 687174 tok/s
iter    890 | loss 3.3287 | lr 1.20e-02 | grad_norm 0.30 | 686020 tok/s
iter    900 | loss 3.3342 | lr 1.20e-02 | grad_norm 0.30 | 694879 tok/s
  eval | train_loss 3.3439 | val_loss 3.3347
Saved checkpoint at iter 900 -> ckpt.pt
Saved checkpoint at iter 900 -> ckpt_best.pt
  New best val_loss: 3.3347
iter    910 | loss 3.3427 | lr 1.20e-02 | grad_norm 0.30 | 427892 tok/s
iter    920 | loss 3.3494 | lr 1.20e-02 | grad_norm 0.29 | 693093 tok/s
iter    930 | loss 3.3403 | lr 1.20e-02 | grad_norm 0.30 | 684251 tok/s
iter    940 | loss 3.3558 | lr 1.20e-02 | grad_norm 0.30 | 691507 tok/s
iter    950 | loss 3.3004 | lr 1.20e-02 | grad_norm 0.29 | 691031 tok/s
iter    960 | loss 3.3151 | lr 1.20e-02 | grad_norm 0.28 | 689399 tok/s
iter    970 | loss 3.3085 | lr 1.20e-02 | grad_norm 0.31 | 692208 tok/s
iter    980 | loss 3.3098 | lr 1.20e-02 | grad_norm 0.30 | 692587 tok/s
iter    990 | loss 3.3077 | lr 1.20e-02 | grad_norm 0.29 | 691398 tok/s
iter   1000 | loss 3.2819 | lr 1.20e-02 | grad_norm 0.29 | 690641 tok/s
  eval | train_loss 3.3015 | val_loss 3.3075
Saved checkpoint at iter 1000 -> ckpt.pt
Saved checkpoint at iter 1000 -> ckpt_best.pt
  New best val_loss: 3.3075
iter   1010 | loss 3.3050 | lr 1.20e-02 | grad_norm 0.28 | 423664 tok/s
iter   1020 | loss 3.2995 | lr 1.20e-02 | grad_norm 0.28 | 692939 tok/s
iter   1030 | loss 3.2913 | lr 1.20e-02 | grad_norm 0.26 | 692958 tok/s
iter   1040 | loss 3.2825 | lr 1.20e-02 | grad_norm 0.26 | 692405 tok/s
iter   1050 | loss 3.2859 | lr 1.20e-02 | grad_norm 0.29 | 693352 tok/s
iter   1060 | loss 3.3220 | lr 1.20e-02 | grad_norm 0.27 | 695233 tok/s
iter   1070 | loss 3.2912 | lr 1.20e-02 | grad_norm 0.27 | 691752 tok/s
iter   1080 | loss 3.2843 | lr 1.20e-02 | grad_norm 0.28 | 695102 tok/s
iter   1090 | loss 3.2696 | lr 1.20e-02 | grad_norm 0.27 | 689439 tok/s
iter   1100 | loss 3.2966 | lr 1.20e-02 | grad_norm 0.27 | 691562 tok/s
  eval | train_loss 3.2849 | val_loss 3.2764
Saved checkpoint at iter 1100 -> ckpt.pt
Saved checkpoint at iter 1100 -> ckpt_best.pt
  New best val_loss: 3.2764
iter   1110 | loss 3.2929 | lr 1.20e-02 | grad_norm 0.27 | 415440 tok/s
iter   1120 | loss 3.2773 | lr 1.20e-02 | grad_norm 0.27 | 689473 tok/s
iter   1130 | loss 3.2858 | lr 1.20e-02 | grad_norm 0.26 | 687185 tok/s
iter   1140 | loss 3.2615 | lr 1.20e-02 | grad_norm 0.28 | 689981 tok/s
iter   1150 | loss 3.2600 | lr 1.20e-02 | grad_norm 0.28 | 691337 tok/s
iter   1160 | loss 3.2630 | lr 1.20e-02 | grad_norm 0.26 | 687460 tok/s
iter   1170 | loss 3.2709 | lr 1.20e-02 | grad_norm 0.26 | 687101 tok/s
iter   1180 | loss 3.2869 | lr 1.20e-02 | grad_norm 0.25 | 690190 tok/s
iter   1190 | loss 3.2465 | lr 1.20e-02 | grad_norm 0.29 | 689583 tok/s
iter   1200 | loss 3.2694 | lr 1.20e-02 | grad_norm 0.27 | 689546 tok/s
  eval | train_loss 3.2507 | val_loss 3.2510
Saved checkpoint at iter 1200 -> ckpt.pt
Saved checkpoint at iter 1200 -> ckpt_best.pt
  New best val_loss: 3.2510
iter   1210 | loss 3.2704 | lr 1.20e-02 | grad_norm 0.27 | 422621 tok/s
iter   1220 | loss 3.2408 | lr 1.20e-02 | grad_norm 0.25 | 691303 tok/s
iter   1230 | loss 3.2433 | lr 1.20e-02 | grad_norm 0.24 | 689698 tok/s
iter   1240 | loss 3.2726 | lr 1.20e-02 | grad_norm 0.28 | 689239 tok/s
iter   1250 | loss 3.2585 | lr 1.20e-02 | grad_norm 0.25 | 687592 tok/s
iter   1260 | loss 3.2476 | lr 1.20e-02 | grad_norm 0.26 | 686870 tok/s
iter   1270 | loss 3.2463 | lr 1.20e-02 | grad_norm 0.28 | 687469 tok/s
iter   1280 | loss 3.2293 | lr 1.20e-02 | grad_norm 0.27 | 695271 tok/s
iter   1290 | loss 3.2278 | lr 1.20e-02 | grad_norm 0.26 | 691767 tok/s
iter   1300 | loss 3.2333 | lr 1.20e-02 | grad_norm 0.25 | 692604 tok/s
  eval | train_loss 3.2361 | val_loss 3.2321
Saved checkpoint at iter 1300 -> ckpt.pt
Saved checkpoint at iter 1300 -> ckpt_best.pt
  New best val_loss: 3.2321
iter   1310 | loss 3.2310 | lr 1.20e-02 | grad_norm 0.25 | 423800 tok/s
iter   1320 | loss 3.2612 | lr 1.20e-02 | grad_norm 0.24 | 690485 tok/s
iter   1330 | loss 3.2586 | lr 1.19e-02 | grad_norm 0.25 | 693103 tok/s
iter   1340 | loss 3.2429 | lr 1.19e-02 | grad_norm 0.26 | 689259 tok/s
iter   1350 | loss 3.2218 | lr 1.19e-02 | grad_norm 0.24 | 689923 tok/s
iter   1360 | loss 3.2175 | lr 1.19e-02 | grad_norm 0.26 | 691256 tok/s
iter   1370 | loss 3.2373 | lr 1.19e-02 | grad_norm 0.25 | 690772 tok/s
iter   1380 | loss 3.2194 | lr 1.19e-02 | grad_norm 0.24 | 687726 tok/s
iter   1390 | loss 3.2260 | lr 1.19e-02 | grad_norm 0.24 | 689498 tok/s
iter   1400 | loss 3.2086 | lr 1.19e-02 | grad_norm 0.24 | 691848 tok/s
  eval | train_loss 3.2270 | val_loss 3.2149
Saved checkpoint at iter 1400 -> ckpt.pt
Saved checkpoint at iter 1400 -> ckpt_best.pt
  New best val_loss: 3.2149
iter   1410 | loss 3.2380 | lr 1.19e-02 | grad_norm 0.25 | 422016 tok/s
iter   1420 | loss 3.2382 | lr 1.18e-02 | grad_norm 0.24 | 698305 tok/s
iter   1430 | loss 3.2052 | lr 1.18e-02 | grad_norm 0.25 | 694828 tok/s
iter   1440 | loss 3.1961 | lr 1.18e-02 | grad_norm 0.24 | 696342 tok/s
iter   1450 | loss 3.2260 | lr 1.18e-02 | grad_norm 0.25 | 693037 tok/s
iter   1460 | loss 3.1975 | lr 1.18e-02 | grad_norm 0.25 | 689768 tok/s
iter   1470 | loss 3.1979 | lr 1.18e-02 | grad_norm 0.25 | 692925 tok/s
iter   1480 | loss 3.2166 | lr 1.18e-02 | grad_norm 0.25 | 693471 tok/s
iter   1490 | loss 3.2343 | lr 1.18e-02 | grad_norm 0.24 | 693296 tok/s
iter   1500 | loss 3.2101 | lr 1.18e-02 | grad_norm 0.25 | 694088 tok/s
  eval | train_loss 3.2024 | val_loss 3.2032
Saved checkpoint at iter 1500 -> ckpt.pt
Saved checkpoint at iter 1500 -> ckpt_best.pt
  New best val_loss: 3.2032
iter   1510 | loss 3.2309 | lr 1.18e-02 | grad_norm 0.26 | 423265 tok/s
iter   1520 | loss 3.1924 | lr 1.17e-02 | grad_norm 0.24 | 693392 tok/s
iter   1530 | loss 3.1792 | lr 1.17e-02 | grad_norm 0.25 | 690576 tok/s
iter   1540 | loss 3.2146 | lr 1.17e-02 | grad_norm 0.25 | 693209 tok/s
iter   1550 | loss 3.2010 | lr 1.17e-02 | grad_norm 0.25 | 688714 tok/s
iter   1560 | loss 3.2036 | lr 1.17e-02 | grad_norm 0.22 | 690784 tok/s
iter   1570 | loss 3.2051 | lr 1.17e-02 | grad_norm 0.25 | 691415 tok/s
iter   1580 | loss 3.2158 | lr 1.17e-02 | grad_norm 0.23 | 691463 tok/s
iter   1590 | loss 3.1911 | lr 1.17e-02 | grad_norm 0.24 | 689653 tok/s
iter   1600 | loss 3.1663 | lr 1.17e-02 | grad_norm 0.24 | 692283 tok/s
  eval | train_loss 3.1908 | val_loss 3.1925
Saved checkpoint at iter 1600 -> ckpt.pt
Saved checkpoint at iter 1600 -> ckpt_best.pt
  New best val_loss: 3.1925
iter   1610 | loss 3.1833 | lr 1.17e-02 | grad_norm 0.25 | 426256 tok/s
iter   1620 | loss 3.2182 | lr 1.16e-02 | grad_norm 0.23 | 696636 tok/s
iter   1630 | loss 3.1725 | lr 1.16e-02 | grad_norm 0.22 | 693406 tok/s
iter   1640 | loss 3.1717 | lr 1.16e-02 | grad_norm 0.23 | 693504 tok/s
iter   1650 | loss 3.1940 | lr 1.16e-02 | grad_norm 0.24 | 693805 tok/s
iter   1660 | loss 3.1809 | lr 1.16e-02 | grad_norm 0.23 | 696818 tok/s
iter   1670 | loss 3.1771 | lr 1.16e-02 | grad_norm 0.25 | 696159 tok/s
iter   1680 | loss 3.1903 | lr 1.16e-02 | grad_norm 0.24 | 694360 tok/s
iter   1690 | loss 3.1758 | lr 1.16e-02 | grad_norm 0.24 | 692987 tok/s
iter   1700 | loss 3.1501 | lr 1.16e-02 | grad_norm 0.25 | 695971 tok/s
  eval | train_loss 3.1755 | val_loss 3.1684
Saved checkpoint at iter 1700 -> ckpt.pt
Saved checkpoint at iter 1700 -> ckpt_best.pt
  New best val_loss: 3.1684
iter   1710 | loss 3.1877 | lr 1.15e-02 | grad_norm 0.24 | 425231 tok/s
iter   1720 | loss 3.1627 | lr 1.15e-02 | grad_norm 0.21 | 692555 tok/s
iter   1730 | loss 3.1815 | lr 1.15e-02 | grad_norm 0.23 | 690576 tok/s
iter   1740 | loss 3.1693 | lr 1.15e-02 | grad_norm 0.23 | 692894 tok/s
iter   1750 | loss 3.1564 | lr 1.15e-02 | grad_norm 0.23 | 693726 tok/s
iter   1760 | loss 3.1597 | lr 1.15e-02 | grad_norm 0.22 | 695415 tok/s
iter   1770 | loss 3.1700 | lr 1.15e-02 | grad_norm 0.23 | 693839 tok/s
iter   1780 | loss 3.1595 | lr 1.15e-02 | grad_norm 0.24 | 692886 tok/s
iter   1790 | loss 3.1552 | lr 1.15e-02 | grad_norm 0.22 | 691031 tok/s
iter   1800 | loss 3.1501 | lr 1.15e-02 | grad_norm 0.23 | 692005 tok/s
  eval | train_loss 3.1710 | val_loss 3.1616
Saved checkpoint at iter 1800 -> ckpt.pt
Saved checkpoint at iter 1800 -> ckpt_best.pt
  New best val_loss: 3.1616
iter   1810 | loss 3.1646 | lr 1.14e-02 | grad_norm 0.22 | 427645 tok/s
iter   1820 | loss 3.1843 | lr 1.14e-02 | grad_norm 0.23 | 692774 tok/s
iter   1830 | loss 3.1552 | lr 1.14e-02 | grad_norm 0.23 | 692051 tok/s
iter   1840 | loss 3.1596 | lr 1.14e-02 | grad_norm 0.22 | 691370 tok/s
iter   1850 | loss 3.1665 | lr 1.14e-02 | grad_norm 0.21 | 692323 tok/s
iter   1860 | loss 3.1310 | lr 1.14e-02 | grad_norm 0.21 | 691838 tok/s
iter   1870 | loss 3.1894 | lr 1.14e-02 | grad_norm 0.23 | 689281 tok/s
iter   1880 | loss 3.1763 | lr 1.14e-02 | grad_norm 0.24 | 691445 tok/s
iter   1890 | loss 3.1748 | lr 1.14e-02 | grad_norm 0.23 | 690923 tok/s
iter   1900 | loss 3.1696 | lr 1.13e-02 | grad_norm 0.22 | 689650 tok/s
  eval | train_loss 3.1485 | val_loss 3.1536
Saved checkpoint at iter 1900 -> ckpt.pt
Saved checkpoint at iter 1900 -> ckpt_best.pt
  New best val_loss: 3.1536
Saved checkpoint at iter 1900 -> ckpt_val_3.1584.pt
  save-at-val: val 3.1536 <= 3.1584, saved first-crossing checkpoint ckpt_val_3.1584.pt (iter 1900)
iter   1910 | loss 3.1530 | lr 1.13e-02 | grad_norm 0.22 | 412055 tok/s
iter   1920 | loss 3.1572 | lr 1.13e-02 | grad_norm 0.23 | 689085 tok/s
iter   1930 | loss 3.1477 | lr 1.13e-02 | grad_norm 0.22 | 689431 tok/s
iter   1940 | loss 3.1446 | lr 1.13e-02 | grad_norm 0.24 | 690921 tok/s
iter   1950 | loss 3.1371 | lr 1.13e-02 | grad_norm 0.23 | 690589 tok/s
iter   1960 | loss 3.1538 | lr 1.13e-02 | grad_norm 0.22 | 694078 tok/s
iter   1970 | loss 3.1401 | lr 1.13e-02 | grad_norm 0.22 | 690405 tok/s
iter   1980 | loss 3.1700 | lr 1.13e-02 | grad_norm 0.24 | 688502 tok/s
iter   1990 | loss 3.1366 | lr 1.13e-02 | grad_norm 0.22 | 687424 tok/s
iter   2000 | loss 3.1428 | lr 1.12e-02 | grad_norm 0.22 | 691271 tok/s
  eval | train_loss 3.1470 | val_loss 3.1510
Saved checkpoint at iter 2000 -> ckpt.pt
Saved checkpoint at iter 2000 -> ckpt_best.pt
  New best val_loss: 3.1510
iter   2010 | loss 3.1557 | lr 1.12e-02 | grad_norm 0.23 | 418799 tok/s
iter   2020 | loss 3.1347 | lr 1.12e-02 | grad_norm 0.24 | 690486 tok/s
iter   2030 | loss 3.1280 | lr 1.12e-02 | grad_norm 0.22 | 689082 tok/s
iter   2040 | loss 3.1492 | lr 1.12e-02 | grad_norm 0.22 | 692536 tok/s
iter   2050 | loss 3.1424 | lr 1.12e-02 | grad_norm 0.21 | 696445 tok/s
iter   2060 | loss 3.1404 | lr 1.12e-02 | grad_norm 0.23 | 689825 tok/s
iter   2070 | loss 3.1528 | lr 1.12e-02 | grad_norm 0.23 | 694078 tok/s
iter   2080 | loss 3.1363 | lr 1.12e-02 | grad_norm 0.22 | 690543 tok/s
iter   2090 | loss 3.1407 | lr 1.12e-02 | grad_norm 0.23 | 693766 tok/s
iter   2100 | loss 3.1344 | lr 1.11e-02 | grad_norm 0.25 | 695053 tok/s
  eval | train_loss 3.1321 | val_loss 3.1354
Saved checkpoint at iter 2100 -> ckpt.pt
Saved checkpoint at iter 2100 -> ckpt_best.pt
  New best val_loss: 3.1354
Saved checkpoint at iter 2100 -> ckpt_val_3.1484.pt
  save-at-val: val 3.1354 <= 3.1484, saved first-crossing checkpoint ckpt_val_3.1484.pt (iter 2100)
iter   2110 | loss 3.1392 | lr 1.11e-02 | grad_norm 0.24 | 406877 tok/s
iter   2120 | loss 3.1041 | lr 1.11e-02 | grad_norm 0.23 | 691570 tok/s
iter   2130 | loss 3.1361 | lr 1.11e-02 | grad_norm 0.23 | 689706 tok/s
iter   2140 | loss 3.1247 | lr 1.11e-02 | grad_norm 0.23 | 689597 tok/s
iter   2150 | loss 3.1293 | lr 1.11e-02 | grad_norm 0.24 | 687438 tok/s
iter   2160 | loss 3.1361 | lr 1.11e-02 | grad_norm 0.22 | 689505 tok/s
iter   2170 | loss 3.1389 | lr 1.11e-02 | grad_norm 0.22 | 689674 tok/s
iter   2180 | loss 3.1452 | lr 1.11e-02 | grad_norm 0.22 | 688127 tok/s
iter   2190 | loss 3.1365 | lr 1.10e-02 | grad_norm 0.23 | 691429 tok/s
iter   2200 | loss 3.1151 | lr 1.10e-02 | grad_norm 0.23 | 691086 tok/s
  eval | train_loss 3.1170 | val_loss 3.1359
Saved checkpoint at iter 2200 -> ckpt.pt
iter   2210 | loss 3.1478 | lr 1.10e-02 | grad_norm 0.21 | 431297 tok/s
iter   2220 | loss 3.1030 | lr 1.10e-02 | grad_norm 0.23 | 691036 tok/s
iter   2230 | loss 3.1271 | lr 1.10e-02 | grad_norm 0.22 | 691403 tok/s
iter   2240 | loss 3.1140 | lr 1.10e-02 | grad_norm 0.21 | 694974 tok/s
iter   2250 | loss 3.1221 | lr 1.10e-02 | grad_norm 0.22 | 689252 tok/s
iter   2260 | loss 3.1061 | lr 1.10e-02 | grad_norm 0.22 | 689377 tok/s
iter   2270 | loss 3.1108 | lr 1.10e-02 | grad_norm 0.21 | 688733 tok/s
iter   2280 | loss 3.1250 | lr 1.10e-02 | grad_norm 0.22 | 693192 tok/s
iter   2290 | loss 3.1186 | lr 1.09e-02 | grad_norm 0.22 | 692154 tok/s
iter   2300 | loss 3.1206 | lr 1.09e-02 | grad_norm 0.22 | 690479 tok/s
  eval | train_loss 3.1220 | val_loss 3.1162
Saved checkpoint at iter 2300 -> ckpt.pt
Saved checkpoint at iter 2300 -> ckpt_best.pt
  New best val_loss: 3.1162
Saved checkpoint at iter 2300 -> ckpt_val_3.1330.pt
  save-at-val: val 3.1162 <= 3.1330, saved first-crossing checkpoint ckpt_val_3.1330.pt (iter 2300)
Saved checkpoint at iter 2300 -> ckpt_val_3.1230.pt
  save-at-val: val 3.1162 <= 3.1230, saved first-crossing checkpoint ckpt_val_3.1230.pt (iter 2300)
iter   2310 | loss 3.1111 | lr 1.09e-02 | grad_norm 0.21 | 405599 tok/s
iter   2320 | loss 3.1103 | lr 1.09e-02 | grad_norm 0.21 | 691566 tok/s
iter   2330 | loss 3.1091 | lr 1.09e-02 | grad_norm 0.22 | 689826 tok/s
iter   2340 | loss 3.1157 | lr 1.09e-02 | grad_norm 0.21 | 690392 tok/s
iter   2350 | loss 3.1157 | lr 1.09e-02 | grad_norm 0.22 | 689448 tok/s
iter   2360 | loss 3.1079 | lr 1.09e-02 | grad_norm 0.23 | 692846 tok/s
iter   2370 | loss 3.1238 | lr 1.09e-02 | grad_norm 0.22 | 696981 tok/s
iter   2380 | loss 3.1223 | lr 1.09e-02 | grad_norm 0.22 | 691918 tok/s
iter   2390 | loss 3.1152 | lr 1.08e-02 | grad_norm 0.22 | 694764 tok/s
iter   2400 | loss 3.1260 | lr 1.08e-02 | grad_norm 0.22 | 696684 tok/s
  eval | train_loss 3.1113 | val_loss 3.1083
Saved checkpoint at iter 2400 -> ckpt.pt
Saved checkpoint at iter 2400 -> ckpt_best.pt
  New best val_loss: 3.1083
iter   2410 | loss 3.1143 | lr 1.08e-02 | grad_norm 0.22 | 424287 tok/s
iter   2420 | loss 3.0936 | lr 1.08e-02 | grad_norm 0.22 | 697557 tok/s
iter   2430 | loss 3.0832 | lr 1.08e-02 | grad_norm 0.20 | 696542 tok/s
iter   2440 | loss 3.1002 | lr 1.08e-02 | grad_norm 0.22 | 696641 tok/s
iter   2450 | loss 3.1221 | lr 1.08e-02 | grad_norm 0.23 | 697213 tok/s
iter   2460 | loss 3.1310 | lr 1.08e-02 | grad_norm 0.22 | 692952 tok/s
iter   2470 | loss 3.1043 | lr 1.08e-02 | grad_norm 0.21 | 692316 tok/s
iter   2480 | loss 3.0846 | lr 1.07e-02 | grad_norm 0.21 | 687970 tok/s
iter   2490 | loss 3.0939 | lr 1.07e-02 | grad_norm 0.21 | 691540 tok/s
iter   2500 | loss 3.0944 | lr 1.07e-02 | grad_norm 0.22 | 694072 tok/s
  eval | train_loss 3.0946 | val_loss 3.1069
Saved checkpoint at iter 2500 -> ckpt.pt
Saved checkpoint at iter 2500 -> ckpt_best.pt
  New best val_loss: 3.1069
  quick_eval: import failed (No module named 'evals'); skipping
iter   2510 | loss 3.1164 | lr 1.07e-02 | grad_norm 0.22 | 421216 tok/s
iter   2520 | loss 3.1001 | lr 1.07e-02 | grad_norm 0.23 | 693725 tok/s
iter   2530 | loss 3.0903 | lr 1.07e-02 | grad_norm 0.22 | 694723 tok/s
iter   2540 | loss 3.0899 | lr 1.07e-02 | grad_norm 0.21 | 696150 tok/s
iter   2550 | loss 3.0684 | lr 1.07e-02 | grad_norm 0.21 | 695833 tok/s
iter   2560 | loss 3.0968 | lr 1.07e-02 | grad_norm 0.21 | 690143 tok/s
iter   2570 | loss 3.0807 | lr 1.07e-02 | grad_norm 0.21 | 690190 tok/s
iter   2580 | loss 3.0959 | lr 1.06e-02 | grad_norm 0.22 | 692690 tok/s
iter   2590 | loss 3.0912 | lr 1.06e-02 | grad_norm 0.22 | 692767 tok/s
iter   2600 | loss 3.1187 | lr 1.06e-02 | grad_norm 0.22 | 688001 tok/s
  eval | train_loss 3.0969 | val_loss 3.0968
Saved checkpoint at iter 2600 -> ckpt.pt
Saved checkpoint at iter 2600 -> ckpt_best.pt
  New best val_loss: 3.0968
iter   2610 | loss 3.0927 | lr 1.06e-02 | grad_norm 0.22 | 423171 tok/s
iter   2620 | loss 3.0738 | lr 1.06e-02 | grad_norm 0.21 | 697362 tok/s
iter   2630 | loss 3.1009 | lr 1.06e-02 | grad_norm 0.23 | 694471 tok/s
iter   2640 | loss 3.0924 | lr 1.06e-02 | grad_norm 0.21 | 690575 tok/s
iter   2650 | loss 3.0949 | lr 1.06e-02 | grad_norm 0.21 | 687062 tok/s
iter   2660 | loss 3.0929 | lr 1.06e-02 | grad_norm 0.22 | 693980 tok/s
iter   2670 | loss 3.1120 | lr 1.06e-02 | grad_norm 0.22 | 692142 tok/s
iter   2680 | loss 3.0676 | lr 1.05e-02 | grad_norm 0.22 | 690865 tok/s
iter   2690 | loss 3.0871 | lr 1.05e-02 | grad_norm 0.23 | 690285 tok/s
iter   2700 | loss 3.0751 | lr 1.05e-02 | grad_norm 0.20 | 694445 tok/s
  eval | train_loss 3.0865 | val_loss 3.0891
Saved checkpoint at iter 2700 -> ckpt.pt
Saved checkpoint at iter 2700 -> ckpt_best.pt
  New best val_loss: 3.0891
iter   2710 | loss 3.1113 | lr 1.05e-02 | grad_norm 0.21 | 425538 tok/s
iter   2720 | loss 3.0999 | lr 1.05e-02 | grad_norm 0.21 | 692223 tok/s
iter   2730 | loss 3.1036 | lr 1.05e-02 | grad_norm 0.21 | 691192 tok/s
iter   2740 | loss 3.0670 | lr 1.05e-02 | grad_norm 0.22 | 691517 tok/s
iter   2750 | loss 3.1041 | lr 1.05e-02 | grad_norm 0.22 | 693729 tok/s
iter   2760 | loss 3.0949 | lr 1.05e-02 | grad_norm 0.22 | 695385 tok/s
iter   2770 | loss 3.0669 | lr 1.04e-02 | grad_norm 0.21 | 692577 tok/s
iter   2780 | loss 3.0588 | lr 1.04e-02 | grad_norm 0.21 | 688529 tok/s
iter   2790 | loss 3.0893 | lr 1.04e-02 | grad_norm 0.22 | 686579 tok/s
iter   2800 | loss 3.0832 | lr 1.04e-02 | grad_norm 0.22 | 693491 tok/s
  eval | train_loss 3.0920 | val_loss 3.0768
Saved checkpoint at iter 2800 -> ckpt.pt
Saved checkpoint at iter 2800 -> ckpt_best.pt
  New best val_loss: 3.0768
iter   2810 | loss 3.0617 | lr 1.04e-02 | grad_norm 0.21 | 426943 tok/s
iter   2820 | loss 3.0993 | lr 1.04e-02 | grad_norm 0.23 | 690978 tok/s
iter   2830 | loss 3.0830 | lr 1.04e-02 | grad_norm 0.23 | 687320 tok/s
iter   2840 | loss 3.0901 | lr 1.04e-02 | grad_norm 0.23 | 687569 tok/s
iter   2850 | loss 3.0871 | lr 1.04e-02 | grad_norm 0.23 | 687871 tok/s
iter   2860 | loss 3.0691 | lr 1.04e-02 | grad_norm 0.21 | 687843 tok/s
iter   2870 | loss 3.0696 | lr 1.03e-02 | grad_norm 0.21 | 688244 tok/s
iter   2880 | loss 3.0586 | lr 1.03e-02 | grad_norm 0.20 | 685665 tok/s
iter   2890 | loss 3.0709 | lr 1.03e-02 | grad_norm 0.22 | 686384 tok/s
iter   2900 | loss 3.0796 | lr 1.03e-02 | grad_norm 0.21 | 686757 tok/s
  eval | train_loss 3.0768 | val_loss 3.0877
Saved checkpoint at iter 2900 -> ckpt.pt
iter   2910 | loss 3.0898 | lr 1.03e-02 | grad_norm 0.22 | 436580 tok/s
iter   2920 | loss 3.0647 | lr 1.03e-02 | grad_norm 0.21 | 689845 tok/s
iter   2930 | loss 3.0835 | lr 1.03e-02 | grad_norm 0.21 | 690513 tok/s
iter   2940 | loss 3.0908 | lr 1.03e-02 | grad_norm 0.22 | 690526 tok/s
iter   2950 | loss 3.0899 | lr 1.03e-02 | grad_norm 0.23 | 693424 tok/s
iter   2960 | loss 3.0605 | lr 1.03e-02 | grad_norm 0.20 | 691975 tok/s
iter   2970 | loss 3.0660 | lr 1.02e-02 | grad_norm 0.21 | 692436 tok/s
iter   2980 | loss 3.0980 | lr 1.02e-02 | grad_norm 0.22 | 694459 tok/s
iter   2990 | loss 3.0616 | lr 1.02e-02 | grad_norm 0.22 | 689100 tok/s
iter   3000 | loss 3.0690 | lr 1.02e-02 | grad_norm 0.21 | 688767 tok/s
  eval | train_loss 3.0676 | val_loss 3.0782
Saved checkpoint at iter 3000 -> ckpt.pt
iter   3010 | loss 3.0624 | lr 1.02e-02 | grad_norm 0.20 | 437550 tok/s
iter   3020 | loss 3.0632 | lr 1.02e-02 | grad_norm 0.23 | 690316 tok/s
iter   3030 | loss 3.0638 | lr 1.02e-02 | grad_norm 0.23 | 692772 tok/s
iter   3040 | loss 3.0439 | lr 1.02e-02 | grad_norm 0.22 | 693264 tok/s
iter   3050 | loss 3.0735 | lr 1.02e-02 | grad_norm 0.22 | 694258 tok/s
iter   3060 | loss 3.0697 | lr 1.01e-02 | grad_norm 0.21 | 694253 tok/s
iter   3070 | loss 3.0721 | lr 1.01e-02 | grad_norm 0.21 | 690996 tok/s
iter   3080 | loss 3.0868 | lr 1.01e-02 | grad_norm 0.21 | 698397 tok/s
iter   3090 | loss 3.0652 | lr 1.01e-02 | grad_norm 0.21 | 691752 tok/s
iter   3100 | loss 3.0969 | lr 1.01e-02 | grad_norm 0.21 | 691283 tok/s
  eval | train_loss 3.0678 | val_loss 3.0638
Saved checkpoint at iter 3100 -> ckpt.pt
Saved checkpoint at iter 3100 -> ckpt_best.pt
  New best val_loss: 3.0638
iter   3110 | loss 3.0765 | lr 1.01e-02 | grad_norm 0.22 | 423717 tok/s
iter   3120 | loss 3.0683 | lr 1.01e-02 | grad_norm 0.23 | 692377 tok/s
iter   3130 | loss 3.0722 | lr 1.01e-02 | grad_norm 0.21 | 691664 tok/s
iter   3140 | loss 3.0603 | lr 1.01e-02 | grad_norm 0.21 | 691831 tok/s
iter   3150 | loss 3.0707 | lr 1.01e-02 | grad_norm 0.22 | 691137 tok/s
iter   3160 | loss 3.0760 | lr 1.00e-02 | grad_norm 0.22 | 694567 tok/s
iter   3170 | loss 3.0461 | lr 1.00e-02 | grad_norm 0.21 | 691525 tok/s
iter   3180 | loss 3.0947 | lr 1.00e-02 | grad_norm 0.21 | 691951 tok/s
iter   3190 | loss 3.0667 | lr 1.00e-02 | grad_norm 0.21 | 691270 tok/s
iter   3200 | loss 3.0752 | lr 1.00e-02 | grad_norm 0.22 | 690387 tok/s
  eval | train_loss 3.0625 | val_loss 3.0615
Saved checkpoint at iter 3200 -> ckpt.pt
Saved checkpoint at iter 3200 -> ckpt_best.pt
  New best val_loss: 3.0615
iter   3210 | loss 3.0851 | lr 9.99e-03 | grad_norm 0.21 | 422261 tok/s
iter   3220 | loss 3.0451 | lr 9.98e-03 | grad_norm 0.21 | 689961 tok/s
iter   3230 | loss 3.0542 | lr 9.97e-03 | grad_norm 0.21 | 691904 tok/s
iter   3240 | loss 3.0326 | lr 9.96e-03 | grad_norm 0.22 | 691519 tok/s
iter   3250 | loss 3.0497 | lr 9.95e-03 | grad_norm 0.21 | 689351 tok/s
iter   3260 | loss 3.0520 | lr 9.94e-03 | grad_norm 0.22 | 690240 tok/s
iter   3270 | loss 3.0611 | lr 9.93e-03 | grad_norm 0.23 | 693426 tok/s
iter   3280 | loss 3.0536 | lr 9.92e-03 | grad_norm 0.22 | 691694 tok/s
iter   3290 | loss 3.0524 | lr 9.91e-03 | grad_norm 0.23 | 690445 tok/s
iter   3300 | loss 3.0336 | lr 9.90e-03 | grad_norm 0.21 | 692185 tok/s
  eval | train_loss 3.0550 | val_loss 3.0438
Saved checkpoint at iter 3300 -> ckpt.pt
Saved checkpoint at iter 3300 -> ckpt_best.pt
  New best val_loss: 3.0438
iter   3310 | loss 3.0725 | lr 9.89e-03 | grad_norm 0.20 | 420961 tok/s
iter   3320 | loss 3.0450 | lr 9.88e-03 | grad_norm 0.22 | 688701 tok/s
iter   3330 | loss 3.0546 | lr 9.87e-03 | grad_norm 0.22 | 689718 tok/s
iter   3340 | loss 3.0477 | lr 9.86e-03 | grad_norm 0.23 | 690163 tok/s
iter   3350 | loss 3.0644 | lr 9.85e-03 | grad_norm 0.21 | 688749 tok/s
iter   3360 | loss 3.0665 | lr 9.84e-03 | grad_norm 0.21 | 686685 tok/s
iter   3370 | loss 3.0663 | lr 9.83e-03 | grad_norm 0.22 | 688746 tok/s
iter   3380 | loss 3.0507 | lr 9.82e-03 | grad_norm 0.21 | 691136 tok/s
iter   3390 | loss 3.0471 | lr 9.81e-03 | grad_norm 0.21 | 688618 tok/s
iter   3400 | loss 3.0362 | lr 9.80e-03 | grad_norm 0.22 | 691847 tok/s
  eval | train_loss 3.0396 | val_loss 3.0549
Saved checkpoint at iter 3400 -> ckpt.pt
iter   3410 | loss 3.0549 | lr 9.79e-03 | grad_norm 0.22 | 426483 tok/s
iter   3420 | loss 3.0515 | lr 9.78e-03 | grad_norm 0.21 | 690190 tok/s
iter   3430 | loss 3.0525 | lr 9.77e-03 | grad_norm 0.22 | 689519 tok/s
iter   3440 | loss 3.0706 | lr 9.76e-03 | grad_norm 0.22 | 689601 tok/s
iter   3450 | loss 3.0496 | lr 9.74e-03 | grad_norm 0.21 | 687829 tok/s
iter   3460 | loss 3.0455 | lr 9.73e-03 | grad_norm 0.21 | 690331 tok/s
iter   3470 | loss 3.0524 | lr 9.72e-03 | grad_norm 0.21 | 687895 tok/s
iter   3480 | loss 3.0286 | lr 9.71e-03 | grad_norm 0.22 | 687833 tok/s
iter   3490 | loss 3.0390 | lr 9.70e-03 | grad_norm 0.21 | 689994 tok/s
iter   3500 | loss 3.0660 | lr 9.69e-03 | grad_norm 0.22 | 689493 tok/s
  eval | train_loss 3.0459 | val_loss 3.0614
Saved checkpoint at iter 3500 -> ckpt.pt
iter   3510 | loss 3.0361 | lr 9.68e-03 | grad_norm 0.22 | 437925 tok/s
iter   3520 | loss 3.0520 | lr 9.67e-03 | grad_norm 0.22 | 696134 tok/s
iter   3530 | loss 3.0291 | lr 9.66e-03 | grad_norm 0.21 | 696544 tok/s
iter   3540 | loss 3.0530 | lr 9.65e-03 | grad_norm 0.21 | 690842 tok/s
iter   3550 | loss 3.0271 | lr 9.64e-03 | grad_norm 0.21 | 692722 tok/s
iter   3560 | loss 3.0298 | lr 9.63e-03 | grad_norm 0.22 | 687890 tok/s
iter   3570 | loss 3.0453 | lr 9.62e-03 | grad_norm 0.20 | 689201 tok/s
iter   3580 | loss 3.0731 | lr 9.61e-03 | grad_norm 0.22 | 689820 tok/s
iter   3590 | loss 3.0431 | lr 9.60e-03 | grad_norm 0.22 | 691579 tok/s
iter   3600 | loss 3.0314 | lr 9.59e-03 | grad_norm 0.20 | 690058 tok/s
  eval | train_loss 3.0402 | val_loss 3.0480
Saved checkpoint at iter 3600 -> ckpt.pt
iter   3610 | loss 3.0970 | lr 9.58e-03 | grad_norm 0.21 | 434578 tok/s
iter   3620 | loss 3.0379 | lr 9.57e-03 | grad_norm 0.21 | 691540 tok/s
iter   3630 | loss 3.0456 | lr 9.56e-03 | grad_norm 0.22 | 693276 tok/s
iter   3640 | loss 3.0234 | lr 9.55e-03 | grad_norm 0.21 | 693136 tok/s
iter   3650 | loss 3.0715 | lr 9.54e-03 | grad_norm 0.21 | 688748 tok/s
iter   3660 | loss 3.0477 | lr 9.53e-03 | grad_norm 0.20 | 690663 tok/s
iter   3670 | loss 3.0365 | lr 9.52e-03 | grad_norm 0.23 | 689831 tok/s
iter   3680 | loss 3.0575 | lr 9.51e-03 | grad_norm 0.21 | 687768 tok/s
iter   3690 | loss 3.0535 | lr 9.50e-03 | grad_norm 0.21 | 687688 tok/s
iter   3700 | loss 3.0531 | lr 9.49e-03 | grad_norm 0.22 | 690503 tok/s
  eval | train_loss 3.0305 | val_loss 3.0480
Saved checkpoint at iter 3700 -> ckpt.pt
iter   3710 | loss 3.0465 | lr 9.48e-03 | grad_norm 0.22 | 428023 tok/s
iter   3720 | loss 3.0510 | lr 9.47e-03 | grad_norm 0.23 | 692475 tok/s
iter   3730 | loss 3.0444 | lr 9.45e-03 | grad_norm 0.22 | 689398 tok/s
iter   3740 | loss 3.0420 | lr 9.44e-03 | grad_norm 0.21 | 690747 tok/s
iter   3750 | loss 3.0567 | lr 9.43e-03 | grad_norm 0.22 | 695799 tok/s
iter   3760 | loss 3.0548 | lr 9.42e-03 | grad_norm 0.21 | 694133 tok/s
iter   3770 | loss 3.0331 | lr 9.41e-03 | grad_norm 0.20 | 690381 tok/s
iter   3780 | loss 3.0312 | lr 9.40e-03 | grad_norm 0.21 | 686323 tok/s
iter   3790 | loss 3.0110 | lr 9.39e-03 | grad_norm 0.24 | 686549 tok/s
iter   3800 | loss 3.0370 | lr 9.38e-03 | grad_norm 0.22 | 689288 tok/s
  eval | train_loss 3.0428 | val_loss 3.0416
Saved checkpoint at iter 3800 -> ckpt.pt
Saved checkpoint at iter 3800 -> ckpt_best.pt
  New best val_loss: 3.0416
iter   3810 | loss 3.0332 | lr 9.37e-03 | grad_norm 0.22 | 426905 tok/s
iter   3820 | loss 3.0274 | lr 9.36e-03 | grad_norm 0.21 | 695355 tok/s
iter   3830 | loss 3.0397 | lr 9.35e-03 | grad_norm 0.21 | 692125 tok/s
iter   3840 | loss 3.0453 | lr 9.34e-03 | grad_norm 0.21 | 691456 tok/s
iter   3850 | loss 3.0337 | lr 9.33e-03 | grad_norm 0.22 | 695218 tok/s
iter   3860 | loss 3.0426 | lr 9.32e-03 | grad_norm 0.22 | 695929 tok/s
iter   3870 | loss 3.0247 | lr 9.31e-03 | grad_norm 0.21 | 691721 tok/s
iter   3880 | loss 3.0102 | lr 9.30e-03 | grad_norm 0.21 | 694717 tok/s
iter   3890 | loss 3.0136 | lr 9.29e-03 | grad_norm 0.21 | 693680 tok/s
iter   3900 | loss 3.0338 | lr 9.28e-03 | grad_norm 0.21 | 688981 tok/s
  eval | train_loss 3.0355 | val_loss 3.0324
Saved checkpoint at iter 3900 -> ckpt.pt
Saved checkpoint at iter 3900 -> ckpt_best.pt
  New best val_loss: 3.0324
iter   3910 | loss 3.0420 | lr 9.27e-03 | grad_norm 0.21 | 425133 tok/s
iter   3920 | loss 3.0277 | lr 9.26e-03 | grad_norm 0.21 | 692634 tok/s
iter   3930 | loss 3.0003 | lr 9.25e-03 | grad_norm 0.22 | 696996 tok/s
iter   3940 | loss 3.0378 | lr 9.24e-03 | grad_norm 0.21 | 691823 tok/s
iter   3950 | loss 3.0407 | lr 9.23e-03 | grad_norm 0.19 | 693441 tok/s
iter   3960 | loss 3.0215 | lr 9.22e-03 | grad_norm 0.24 | 689382 tok/s
iter   3970 | loss 3.0251 | lr 9.21e-03 | grad_norm 0.22 | 690956 tok/s
iter   3980 | loss 3.0522 | lr 9.20e-03 | grad_norm 0.23 | 691559 tok/s
iter   3990 | loss 3.0284 | lr 9.19e-03 | grad_norm 0.21 | 690153 tok/s
iter   4000 | loss 3.0127 | lr 9.18e-03 | grad_norm 0.20 | 689953 tok/s
  eval | train_loss 3.0314 | val_loss 3.0362
Saved checkpoint at iter 4000 -> ckpt.pt
iter   4010 | loss 3.0552 | lr 9.16e-03 | grad_norm 0.20 | 436062 tok/s
iter   4020 | loss 3.0200 | lr 9.15e-03 | grad_norm 0.21 | 691740 tok/s
iter   4030 | loss 3.0170 | lr 9.14e-03 | grad_norm 0.21 | 692311 tok/s
iter   4040 | loss 3.0581 | lr 9.13e-03 | grad_norm 0.22 | 690262 tok/s
iter   4050 | loss 3.0156 | lr 9.12e-03 | grad_norm 0.22 | 689323 tok/s
iter   4060 | loss 3.0245 | lr 9.11e-03 | grad_norm 0.22 | 691666 tok/s
iter   4070 | loss 3.0400 | lr 9.10e-03 | grad_norm 0.21 | 687912 tok/s
iter   4080 | loss 3.0403 | lr 9.09e-03 | grad_norm 0.22 | 686196 tok/s
iter   4090 | loss 3.0065 | lr 9.08e-03 | grad_norm 0.21 | 689787 tok/s
iter   4100 | loss 3.0281 | lr 9.07e-03 | grad_norm 0.21 | 694172 tok/s
  eval | train_loss 3.0290 | val_loss 3.0318
Saved checkpoint at iter 4100 -> ckpt.pt
Saved checkpoint at iter 4100 -> ckpt_best.pt
  New best val_loss: 3.0318
iter   4110 | loss 3.0250 | lr 9.06e-03 | grad_norm 0.20 | 424422 tok/s
iter   4120 | loss 3.0171 | lr 9.05e-03 | grad_norm 0.21 | 693448 tok/s
iter   4130 | loss 3.0261 | lr 9.04e-03 | grad_norm 0.21 | 693687 tok/s
iter   4140 | loss 3.0086 | lr 9.03e-03 | grad_norm 0.21 | 689911 tok/s
iter   4150 | loss 3.0327 | lr 9.02e-03 | grad_norm 0.22 | 692649 tok/s
iter   4160 | loss 3.0355 | lr 9.01e-03 | grad_norm 0.20 | 690695 tok/s
iter   4170 | loss 3.0114 | lr 9.00e-03 | grad_norm 0.22 | 689034 tok/s
iter   4180 | loss 3.0229 | lr 8.99e-03 | grad_norm 0.21 | 687966 tok/s
iter   4190 | loss 3.0434 | lr 8.98e-03 | grad_norm 0.21 | 688815 tok/s
iter   4200 | loss 3.0250 | lr 8.97e-03 | grad_norm 0.21 | 692074 tok/s
  eval | train_loss 3.0233 | val_loss 3.0241
Saved checkpoint at iter 4200 -> ckpt.pt
Saved checkpoint at iter 4200 -> ckpt_best.pt
  New best val_loss: 3.0241
iter   4210 | loss 3.0246 | lr 8.96e-03 | grad_norm 0.22 | 421462 tok/s
iter   4220 | loss 3.0241 | lr 8.95e-03 | grad_norm 0.22 | 693687 tok/s
iter   4230 | loss 3.0282 | lr 8.94e-03 | grad_norm 0.21 | 692794 tok/s
iter   4240 | loss 3.0296 | lr 8.93e-03 | grad_norm 0.20 | 692090 tok/s
iter   4250 | loss 3.0064 | lr 8.92e-03 | grad_norm 0.21 | 696030 tok/s
iter   4260 | loss 2.9831 | lr 8.91e-03 | grad_norm 0.21 | 695838 tok/s
iter   4270 | loss 3.0212 | lr 8.90e-03 | grad_norm 0.20 | 693593 tok/s
iter   4280 | loss 3.0041 | lr 8.89e-03 | grad_norm 0.21 | 698221 tok/s
iter   4290 | loss 3.0314 | lr 8.87e-03 | grad_norm 0.21 | 695276 tok/s
iter   4300 | loss 3.0061 | lr 8.86e-03 | grad_norm 0.21 | 689964 tok/s
  eval | train_loss 3.0100 | val_loss 3.0219
Saved checkpoint at iter 4300 -> ckpt.pt
Saved checkpoint at iter 4300 -> ckpt_best.pt
  New best val_loss: 3.0219
iter   4310 | loss 3.0077 | lr 8.85e-03 | grad_norm 0.21 | 427667 tok/s
iter   4320 | loss 2.9952 | lr 8.84e-03 | grad_norm 0.20 | 691755 tok/s
iter   4330 | loss 3.0106 | lr 8.83e-03 | grad_norm 0.22 | 689933 tok/s
iter   4340 | loss 3.0217 | lr 8.82e-03 | grad_norm 0.22 | 689424 tok/s
iter   4350 | loss 3.0110 | lr 8.81e-03 | grad_norm 0.21 | 687755 tok/s
iter   4360 | loss 3.0015 | lr 8.80e-03 | grad_norm 0.21 | 688609 tok/s
iter   4370 | loss 2.9867 | lr 8.79e-03 | grad_norm 0.20 | 688717 tok/s
iter   4380 | loss 2.9963 | lr 8.78e-03 | grad_norm 0.21 | 688564 tok/s
iter   4390 | loss 3.0104 | lr 8.77e-03 | grad_norm 0.20 | 689828 tok/s
iter   4400 | loss 3.0254 | lr 8.76e-03 | grad_norm 0.22 | 686190 tok/s
  eval | train_loss 3.0123 | val_loss 3.0128
Saved checkpoint at iter 4400 -> ckpt.pt
Saved checkpoint at iter 4400 -> ckpt_best.pt
  New best val_loss: 3.0128
iter   4410 | loss 3.0041 | lr 8.75e-03 | grad_norm 0.21 | 422742 tok/s
iter   4420 | loss 3.0106 | lr 8.74e-03 | grad_norm 0.21 | 693910 tok/s
iter   4430 | loss 2.9896 | lr 8.73e-03 | grad_norm 0.20 | 679657 tok/s
iter   4440 | loss 2.9979 | lr 8.72e-03 | grad_norm 0.22 | 690058 tok/s
iter   4450 | loss 2.9797 | lr 8.71e-03 | grad_norm 0.23 | 693897 tok/s
iter   4460 | loss 3.0131 | lr 8.70e-03 | grad_norm 0.21 | 691148 tok/s
iter   4470 | loss 2.9975 | lr 8.69e-03 | grad_norm 0.21 | 687682 tok/s
iter   4480 | loss 3.0256 | lr 8.68e-03 | grad_norm 0.21 | 685516 tok/s
iter   4490 | loss 3.0077 | lr 8.67e-03 | grad_norm 0.21 | 691212 tok/s
iter   4500 | loss 3.0094 | lr 8.66e-03 | grad_norm 0.22 | 693363 tok/s
  eval | train_loss 3.0149 | val_loss 3.0033
Saved checkpoint at iter 4500 -> ckpt.pt
Saved checkpoint at iter 4500 -> ckpt_best.pt
  New best val_loss: 3.0033
iter   4510 | loss 3.0176 | lr 8.65e-03 | grad_norm 0.22 | 427198 tok/s
iter   4520 | loss 3.0083 | lr 8.64e-03 | grad_norm 0.21 | 695302 tok/s
iter   4530 | loss 3.0073 | lr 8.63e-03 | grad_norm 0.21 | 695355 tok/s
iter   4540 | loss 3.0067 | lr 8.62e-03 | grad_norm 0.20 | 693726 tok/s
iter   4550 | loss 3.0038 | lr 8.61e-03 | grad_norm 0.20 | 691152 tok/s
iter   4560 | loss 2.9974 | lr 8.60e-03 | grad_norm 0.21 | 690397 tok/s
iter   4570 | loss 3.0049 | lr 8.58e-03 | grad_norm 0.19 | 689135 tok/s
iter   4580 | loss 2.9824 | lr 8.57e-03 | grad_norm 0.22 | 689012 tok/s
iter   4590 | loss 3.0029 | lr 8.56e-03 | grad_norm 0.23 | 687728 tok/s
iter   4600 | loss 3.0172 | lr 8.55e-03 | grad_norm 0.21 | 688219 tok/s
  eval | train_loss 3.0126 | val_loss 3.0122
Saved checkpoint at iter 4600 -> ckpt.pt
iter   4610 | loss 3.0095 | lr 8.54e-03 | grad_norm 0.21 | 431150 tok/s
iter   4620 | loss 3.0150 | lr 8.53e-03 | grad_norm 0.19 | 695507 tok/s
iter   4630 | loss 3.0235 | lr 8.52e-03 | grad_norm 0.22 | 696061 tok/s
iter   4640 | loss 2.9745 | lr 8.51e-03 | grad_norm 0.20 | 696214 tok/s
iter   4650 | loss 2.9788 | lr 8.50e-03 | grad_norm 0.21 | 694839 tok/s
iter   4660 | loss 3.0048 | lr 8.49e-03 | grad_norm 0.22 | 693364 tok/s
iter   4670 | loss 3.0186 | lr 8.48e-03 | grad_norm 0.20 | 692874 tok/s
iter   4680 | loss 2.9984 | lr 8.47e-03 | grad_norm 0.20 | 694413 tok/s
iter   4690 | loss 3.0158 | lr 8.46e-03 | grad_norm 0.21 | 693345 tok/s
iter   4700 | loss 3.0094 | lr 8.45e-03 | grad_norm 0.21 | 693185 tok/s
  eval | train_loss 2.9999 | val_loss 3.0144
Saved checkpoint at iter 4700 -> ckpt.pt
iter   4710 | loss 3.0130 | lr 8.44e-03 | grad_norm 0.22 | 436813 tok/s
iter   4720 | loss 3.0106 | lr 8.43e-03 | grad_norm 0.21 | 695496 tok/s
iter   4730 | loss 2.9958 | lr 8.42e-03 | grad_norm 0.20 | 691200 tok/s
iter   4740 | loss 2.9983 | lr 8.41e-03 | grad_norm 0.20 | 692657 tok/s
iter   4750 | loss 2.9931 | lr 8.40e-03 | grad_norm 0.20 | 689573 tok/s
iter   4760 | loss 2.9928 | lr 8.39e-03 | grad_norm 0.22 | 691598 tok/s
iter   4770 | loss 3.0230 | lr 8.38e-03 | grad_norm 0.20 | 690608 tok/s
iter   4780 | loss 2.9956 | lr 8.37e-03 | grad_norm 0.20 | 692039 tok/s
iter   4790 | loss 3.0200 | lr 8.36e-03 | grad_norm 0.22 | 695672 tok/s
iter   4800 | loss 3.0167 | lr 8.35e-03 | grad_norm 0.21 | 692201 tok/s
  eval | train_loss 2.9985 | val_loss 2.9972
Saved checkpoint at iter 4800 -> ckpt.pt
Saved checkpoint at iter 4800 -> ckpt_best.pt
  New best val_loss: 2.9972
iter   4810 | loss 3.0161 | lr 8.34e-03 | grad_norm 0.19 | 416839 tok/s
iter   4820 | loss 2.9918 | lr 8.33e-03 | grad_norm 0.22 | 692891 tok/s
iter   4830 | loss 2.9829 | lr 8.32e-03 | grad_norm 0.21 | 690639 tok/s
iter   4840 | loss 2.9731 | lr 8.31e-03 | grad_norm 0.21 | 690627 tok/s
iter   4850 | loss 3.0053 | lr 8.30e-03 | grad_norm 0.21 | 689939 tok/s
iter   4860 | loss 3.0085 | lr 8.28e-03 | grad_norm 0.21 | 689896 tok/s
iter   4870 | loss 3.0046 | lr 8.27e-03 | grad_norm 0.21 | 687399 tok/s
iter   4880 | loss 2.9757 | lr 8.26e-03 | grad_norm 0.19 | 688471 tok/s
iter   4890 | loss 2.9864 | lr 8.25e-03 | grad_norm 0.21 | 691580 tok/s
iter   4900 | loss 3.0044 | lr 8.24e-03 | grad_norm 0.21 | 688364 tok/s
  eval | train_loss 3.0044 | val_loss 2.9949
Saved checkpoint at iter 4900 -> ckpt.pt
Saved checkpoint at iter 4900 -> ckpt_best.pt
  New best val_loss: 2.9949
iter   4910 | loss 2.9890 | lr 8.23e-03 | grad_norm 0.21 | 422982 tok/s
iter   4920 | loss 3.0065 | lr 8.22e-03 | grad_norm 0.23 | 691625 tok/s
iter   4930 | loss 2.9786 | lr 8.21e-03 | grad_norm 0.21 | 691632 tok/s
iter   4940 | loss 3.0115 | lr 8.20e-03 | grad_norm 0.22 | 692758 tok/s
iter   4950 | loss 2.9966 | lr 8.19e-03 | grad_norm 0.21 | 691588 tok/s
iter   4960 | loss 3.0139 | lr 8.18e-03 | grad_norm 0.21 | 690433 tok/s
iter   4970 | loss 2.9851 | lr 8.17e-03 | grad_norm 0.20 | 689520 tok/s
iter   4980 | loss 2.9983 | lr 8.16e-03 | grad_norm 0.20 | 690223 tok/s
iter   4990 | loss 2.9847 | lr 8.15e-03 | grad_norm 0.21 | 689894 tok/s
iter   5000 | loss 2.9970 | lr 8.14e-03 | grad_norm 0.23 | 691539 tok/s
  eval | train_loss 2.9996 | val_loss 3.0014
Saved checkpoint at iter 5000 -> ckpt.pt
  quick_eval: import failed (No module named 'evals'); skipping
iter   5010 | loss 2.9996 | lr 8.13e-03 | grad_norm 0.21 | 427532 tok/s
iter   5020 | loss 2.9996 | lr 8.12e-03 | grad_norm 0.22 | 697021 tok/s
iter   5030 | loss 2.9995 | lr 8.11e-03 | grad_norm 0.21 | 697725 tok/s
iter   5040 | loss 2.9904 | lr 8.10e-03 | grad_norm 0.21 | 694141 tok/s
iter   5050 | loss 2.9934 | lr 8.09e-03 | grad_norm 0.21 | 690885 tok/s
iter   5060 | loss 2.9669 | lr 8.08e-03 | grad_norm 0.21 | 688003 tok/s
iter   5070 | loss 2.9878 | lr 8.07e-03 | grad_norm 0.20 | 688561 tok/s
iter   5080 | loss 2.9897 | lr 8.06e-03 | grad_norm 0.21 | 691593 tok/s
iter   5090 | loss 2.9934 | lr 8.05e-03 | grad_norm 0.21 | 689054 tok/s
iter   5100 | loss 2.9923 | lr 8.04e-03 | grad_norm 0.21 | 688708 tok/s
  eval | train_loss 2.9990 | val_loss 3.0036
Saved checkpoint at iter 5100 -> ckpt.pt
iter   5110 | loss 2.9791 | lr 8.03e-03 | grad_norm 0.21 | 436744 tok/s
iter   5120 | loss 2.9895 | lr 8.02e-03 | grad_norm 0.20 | 692023 tok/s
iter   5130 | loss 2.9878 | lr 8.01e-03 | grad_norm 0.21 | 686558 tok/s
iter   5140 | loss 2.9809 | lr 7.99e-03 | grad_norm 0.20 | 689967 tok/s
iter   5150 | loss 2.9805 | lr 7.98e-03 | grad_norm 0.22 | 690405 tok/s
iter   5160 | loss 2.9680 | lr 7.97e-03 | grad_norm 0.21 | 688743 tok/s
iter   5170 | loss 2.9930 | lr 7.96e-03 | grad_norm 0.21 | 689507 tok/s
iter   5180 | loss 3.0109 | lr 7.95e-03 | grad_norm 0.20 | 690834 tok/s
iter   5190 | loss 3.0096 | lr 7.94e-03 | grad_norm 0.20 | 693561 tok/s
iter   5200 | loss 3.0180 | lr 7.93e-03 | grad_norm 0.22 | 691282 tok/s
  eval | train_loss 2.9946 | val_loss 2.9921
Saved checkpoint at iter 5200 -> ckpt.pt
Saved checkpoint at iter 5200 -> ckpt_best.pt
  New best val_loss: 2.9921
iter   5210 | loss 3.0045 | lr 7.92e-03 | grad_norm 0.21 | 426444 tok/s
iter   5220 | loss 3.0185 | lr 7.91e-03 | grad_norm 0.22 | 690425 tok/s
iter   5230 | loss 3.0147 | lr 7.90e-03 | grad_norm 0.21 | 688511 tok/s
iter   5240 | loss 2.9962 | lr 7.89e-03 | grad_norm 0.22 | 687008 tok/s
iter   5250 | loss 3.0039 | lr 7.88e-03 | grad_norm 0.21 | 688890 tok/s
iter   5260 | loss 2.9848 | lr 7.87e-03 | grad_norm 0.20 | 691312 tok/s
iter   5270 | loss 2.9673 | lr 7.86e-03 | grad_norm 0.21 | 689568 tok/s
iter   5280 | loss 2.9976 | lr 7.85e-03 | grad_norm 0.21 | 691726 tok/s
iter   5290 | loss 2.9904 | lr 7.84e-03 | grad_norm 0.21 | 695039 tok/s
iter   5300 | loss 2.9849 | lr 7.83e-03 | grad_norm 0.20 | 693873 tok/s
  eval | train_loss 2.9875 | val_loss 2.9857
Saved checkpoint at iter 5300 -> ckpt.pt
Saved checkpoint at iter 5300 -> ckpt_best.pt
  New best val_loss: 2.9857
iter   5310 | loss 2.9909 | lr 7.82e-03 | grad_norm 0.19 | 426086 tok/s
iter   5320 | loss 2.9553 | lr 7.81e-03 | grad_norm 0.21 | 694238 tok/s
iter   5330 | loss 2.9959 | lr 7.80e-03 | grad_norm 0.21 | 689453 tok/s
iter   5340 | loss 3.0153 | lr 7.79e-03 | grad_norm 0.21 | 686404 tok/s
iter   5350 | loss 2.9920 | lr 7.78e-03 | grad_norm 0.20 | 690517 tok/s
iter   5360 | loss 3.0077 | lr 7.77e-03 | grad_norm 0.21 | 689117 tok/s
iter   5370 | loss 2.9710 | lr 7.76e-03 | grad_norm 0.21 | 691968 tok/s
iter   5380 | loss 2.9835 | lr 7.75e-03 | grad_norm 0.21 | 692565 tok/s
iter   5390 | loss 2.9814 | lr 7.74e-03 | grad_norm 0.20 | 688583 tok/s
iter   5400 | loss 2.9846 | lr 7.73e-03 | grad_norm 0.21 | 694439 tok/s
  eval | train_loss 2.9754 | val_loss 2.9861
Saved checkpoint at iter 5400 -> ckpt.pt
iter   5410 | loss 2.9650 | lr 7.72e-03 | grad_norm 0.20 | 430791 tok/s
iter   5420 | loss 2.9810 | lr 7.70e-03 | grad_norm 0.21 | 693256 tok/s
iter   5430 | loss 3.0142 | lr 7.69e-03 | grad_norm 0.20 | 694496 tok/s
iter   5440 | loss 2.9833 | lr 7.68e-03 | grad_norm 0.21 | 688240 tok/s
iter   5450 | loss 2.9775 | lr 7.67e-03 | grad_norm 0.22 | 690281 tok/s
iter   5460 | loss 2.9800 | lr 7.66e-03 | grad_norm 0.20 | 688238 tok/s
iter   5470 | loss 2.9721 | lr 7.65e-03 | grad_norm 0.20 | 689335 tok/s
iter   5480 | loss 2.9645 | lr 7.64e-03 | grad_norm 0.21 | 687673 tok/s
iter   5490 | loss 2.9577 | lr 7.63e-03 | grad_norm 0.20 | 687215 tok/s
iter   5500 | loss 2.9705 | lr 7.62e-03 | grad_norm 0.20 | 689343 tok/s
  eval | train_loss 2.9796 | val_loss 2.9803
Saved checkpoint at iter 5500 -> ckpt.pt
Saved checkpoint at iter 5500 -> ckpt_best.pt
  New best val_loss: 2.9803
iter   5510 | loss 2.9632 | lr 7.61e-03 | grad_norm 0.20 | 416085 tok/s
iter   5520 | loss 2.9766 | lr 7.60e-03 | grad_norm 0.20 | 690948 tok/s
iter   5530 | loss 2.9964 | lr 7.59e-03 | grad_norm 0.20 | 691091 tok/s
iter   5540 | loss 2.9921 | lr 7.58e-03 | grad_norm 0.21 | 689319 tok/s
iter   5550 | loss 2.9751 | lr 7.57e-03 | grad_norm 0.19 | 689285 tok/s
iter   5560 | loss 2.9813 | lr 7.56e-03 | grad_norm 0.20 | 689727 tok/s
iter   5570 | loss 3.0041 | lr 7.55e-03 | grad_norm 0.21 | 688515 tok/s
iter   5580 | loss 2.9760 | lr 7.54e-03 | grad_norm 0.20 | 688308 tok/s
iter   5590 | loss 2.9880 | lr 7.53e-03 | grad_norm 0.21 | 689148 tok/s
iter   5600 | loss 2.9663 | lr 7.52e-03 | grad_norm 0.20 | 688593 tok/s
  eval | train_loss 2.9834 | val_loss 2.9840
Saved checkpoint at iter 5600 -> ckpt.pt
iter   5610 | loss 2.9782 | lr 7.51e-03 | grad_norm 0.21 | 423860 tok/s
iter   5620 | loss 2.9641 | lr 7.50e-03 | grad_norm 0.20 | 694490 tok/s
iter   5630 | loss 2.9812 | lr 7.49e-03 | grad_norm 0.21 | 693059 tok/s
iter   5640 | loss 2.9831 | lr 7.48e-03 | grad_norm 0.20 | 692767 tok/s
iter   5650 | loss 2.9699 | lr 7.47e-03 | grad_norm 0.20 | 689696 tok/s
iter   5660 | loss 2.9746 | lr 7.46e-03 | grad_norm 0.20 | 690226 tok/s
iter   5670 | loss 2.9730 | lr 7.45e-03 | grad_norm 0.21 | 693058 tok/s
iter   5680 | loss 2.9743 | lr 7.44e-03 | grad_norm 0.20 | 693398 tok/s
iter   5690 | loss 2.9613 | lr 7.43e-03 | grad_norm 0.21 | 692250 tok/s
iter   5700 | loss 2.9818 | lr 7.41e-03 | grad_norm 0.20 | 691048 tok/s
  eval | train_loss 2.9784 | val_loss 2.9815
Saved checkpoint at iter 5700 -> ckpt.pt
iter   5710 | loss 2.9807 | lr 7.40e-03 | grad_norm 0.19 | 425448 tok/s
iter   5720 | loss 2.9696 | lr 7.39e-03 | grad_norm 0.20 | 693663 tok/s
iter   5730 | loss 2.9856 | lr 7.38e-03 | grad_norm 0.19 | 692893 tok/s
iter   5740 | loss 2.9809 | lr 7.37e-03 | grad_norm 0.20 | 693111 tok/s
iter   5750 | loss 2.9568 | lr 7.36e-03 | grad_norm 0.20 | 696296 tok/s
iter   5760 | loss 2.9685 | lr 7.35e-03 | grad_norm 0.20 | 693182 tok/s
iter   5770 | loss 2.9457 | lr 7.34e-03 | grad_norm 0.22 | 691391 tok/s
iter   5780 | loss 3.0052 | lr 7.33e-03 | grad_norm 0.19 | 693216 tok/s
iter   5790 | loss 2.9942 | lr 7.32e-03 | grad_norm 0.20 | 691444 tok/s
iter   5800 | loss 2.9736 | lr 7.31e-03 | grad_norm 0.20 | 695912 tok/s
  eval | train_loss 2.9879 | val_loss 2.9691
Saved checkpoint at iter 5800 -> ckpt.pt
Saved checkpoint at iter 5800 -> ckpt_best.pt
  New best val_loss: 2.9691
iter   5810 | loss 2.9745 | lr 7.30e-03 | grad_norm 0.20 | 421460 tok/s
iter   5820 | loss 2.9841 | lr 7.29e-03 | grad_norm 0.19 | 696307 tok/s
iter   5830 | loss 2.9977 | lr 7.28e-03 | grad_norm 0.20 | 696582 tok/s
iter   5840 | loss 2.9785 | lr 7.27e-03 | grad_norm 0.18 | 690818 tok/s
iter   5850 | loss 3.0062 | lr 7.26e-03 | grad_norm 0.22 | 689869 tok/s
iter   5860 | loss 2.9820 | lr 7.25e-03 | grad_norm 0.21 | 692828 tok/s
iter   5870 | loss 2.9955 | lr 7.24e-03 | grad_norm 0.20 | 692813 tok/s
iter   5880 | loss 2.9670 | lr 7.23e-03 | grad_norm 0.20 | 692242 tok/s
iter   5890 | loss 2.9496 | lr 7.22e-03 | grad_norm 0.20 | 691380 tok/s
iter   5900 | loss 2.9963 | lr 7.21e-03 | grad_norm 0.19 | 695767 tok/s
  eval | train_loss 2.9676 | val_loss 2.9735
Saved checkpoint at iter 5900 -> ckpt.pt
iter   5910 | loss 2.9499 | lr 7.20e-03 | grad_norm 0.20 | 437304 tok/s
iter   5920 | loss 2.9737 | lr 7.19e-03 | grad_norm 0.21 | 694034 tok/s
iter   5930 | loss 2.9956 | lr 7.18e-03 | grad_norm 0.20 | 689618 tok/s
iter   5940 | loss 2.9559 | lr 7.17e-03 | grad_norm 0.21 | 694212 tok/s
iter   5950 | loss 2.9932 | lr 7.16e-03 | grad_norm 0.21 | 696003 tok/s
iter   5960 | loss 2.9496 | lr 7.15e-03 | grad_norm 0.20 | 690370 tok/s
iter   5970 | loss 2.9851 | lr 7.14e-03 | grad_norm 0.20 | 688769 tok/s
iter   5980 | loss 2.9828 | lr 7.12e-03 | grad_norm 0.21 | 689916 tok/s
iter   5990 | loss 2.9738 | lr 7.11e-03 | grad_norm 0.20 | 694177 tok/s
iter   6000 | loss 2.9579 | lr 7.10e-03 | grad_norm 0.21 | 692091 tok/s
  eval | train_loss 2.9732 | val_loss 2.9703
Saved checkpoint at iter 6000 -> ckpt.pt
iter   6010 | loss 2.9833 | lr 7.09e-03 | grad_norm 0.21 | 438600 tok/s
iter   6020 | loss 2.9861 | lr 7.08e-03 | grad_norm 0.22 | 694096 tok/s
iter   6030 | loss 2.9657 | lr 7.07e-03 | grad_norm 0.19 | 690115 tok/s
iter   6040 | loss 2.9695 | lr 7.06e-03 | grad_norm 0.19 | 694618 tok/s
iter   6050 | loss 2.9761 | lr 7.05e-03 | grad_norm 0.21 | 693569 tok/s
iter   6060 | loss 2.9771 | lr 7.04e-03 | grad_norm 0.20 | 690934 tok/s
iter   6070 | loss 2.9584 | lr 7.03e-03 | grad_norm 0.21 | 691365 tok/s
iter   6080 | loss 2.9618 | lr 7.02e-03 | grad_norm 0.20 | 691404 tok/s
iter   6090 | loss 2.9879 | lr 7.01e-03 | grad_norm 0.20 | 689131 tok/s
iter   6100 | loss 2.9412 | lr 7.00e-03 | grad_norm 0.21 | 691411 tok/s
  eval | train_loss 2.9663 | val_loss 2.9724
Saved checkpoint at iter 6100 -> ckpt.pt
iter   6110 | loss 2.9781 | lr 6.99e-03 | grad_norm 0.21 | 434117 tok/s
iter   6120 | loss 2.9664 | lr 6.98e-03 | grad_norm 0.20 | 692775 tok/s
iter   6130 | loss 2.9565 | lr 6.97e-03 | grad_norm 0.20 | 696016 tok/s
iter   6140 | loss 2.9615 | lr 6.96e-03 | grad_norm 0.19 | 693273 tok/s
iter   6150 | loss 2.9718 | lr 6.95e-03 | grad_norm 0.21 | 692354 tok/s
iter   6160 | loss 2.9485 | lr 6.94e-03 | grad_norm 0.19 | 692015 tok/s
iter   6170 | loss 2.9519 | lr 6.93e-03 | grad_norm 0.20 | 690255 tok/s
iter   6180 | loss 2.9959 | lr 6.92e-03 | grad_norm 0.21 | 695535 tok/s
iter   6190 | loss 2.9670 | lr 6.91e-03 | grad_norm 0.21 | 695302 tok/s
iter   6200 | loss 2.9464 | lr 6.90e-03 | grad_norm 0.20 | 689701 tok/s
  eval | train_loss 2.9606 | val_loss 2.9693
Saved checkpoint at iter 6200 -> ckpt.pt
iter   6210 | loss 2.9426 | lr 6.89e-03 | grad_norm 0.19 | 436664 tok/s
iter   6220 | loss 2.9586 | lr 6.88e-03 | grad_norm 0.20 | 695799 tok/s
iter   6230 | loss 2.9496 | lr 6.87e-03 | grad_norm 0.21 | 692694 tok/s
iter   6240 | loss 2.9581 | lr 6.86e-03 | grad_norm 0.19 | 692511 tok/s
iter   6250 | loss 2.9661 | lr 6.85e-03 | grad_norm 0.21 | 690603 tok/s
iter   6260 | loss 2.9461 | lr 6.84e-03 | grad_norm 0.20 | 690053 tok/s
iter   6270 | loss 2.9419 | lr 6.82e-03 | grad_norm 0.20 | 690630 tok/s
iter   6280 | loss 2.9567 | lr 6.81e-03 | grad_norm 0.20 | 683964 tok/s
iter   6290 | loss 2.9712 | lr 6.80e-03 | grad_norm 0.19 | 694428 tok/s
iter   6300 | loss 2.9671 | lr 6.79e-03 | grad_norm 0.20 | 692633 tok/s
  eval | train_loss 2.9707 | val_loss 2.9576
Saved checkpoint at iter 6300 -> ckpt.pt
Saved checkpoint at iter 6300 -> ckpt_best.pt
  New best val_loss: 2.9576
iter   6310 | loss 2.9636 | lr 6.78e-03 | grad_norm 0.20 | 427459 tok/s
iter   6320 | loss 2.9516 | lr 6.77e-03 | grad_norm 0.19 | 692994 tok/s
iter   6330 | loss 2.9440 | lr 6.76e-03 | grad_norm 0.18 | 695404 tok/s
iter   6340 | loss 2.9436 | lr 6.75e-03 | grad_norm 0.20 | 690020 tok/s
iter   6350 | loss 2.9641 | lr 6.74e-03 | grad_norm 0.20 | 690189 tok/s
iter   6360 | loss 2.9689 | lr 6.73e-03 | grad_norm 0.19 | 688583 tok/s
iter   6370 | loss 2.9562 | lr 6.72e-03 | grad_norm 0.20 | 690455 tok/s
iter   6380 | loss 2.9776 | lr 6.71e-03 | grad_norm 0.19 | 688513 tok/s
iter   6390 | loss 2.9296 | lr 6.70e-03 | grad_norm 0.19 | 690504 tok/s
iter   6400 | loss 2.9403 | lr 6.69e-03 | grad_norm 0.20 | 690430 tok/s
  eval | train_loss 2.9585 | val_loss 2.9775
Saved checkpoint at iter 6400 -> ckpt.pt
iter   6410 | loss 2.9727 | lr 6.68e-03 | grad_norm 0.18 | 432319 tok/s
iter   6420 | loss 2.9597 | lr 6.67e-03 | grad_norm 0.19 | 690812 tok/s
iter   6430 | loss 2.9663 | lr 6.66e-03 | grad_norm 0.19 | 691488 tok/s
iter   6440 | loss 2.9502 | lr 6.65e-03 | grad_norm 0.20 | 691063 tok/s
iter   6450 | loss 2.9534 | lr 6.64e-03 | grad_norm 0.20 | 695377 tok/s
iter   6460 | loss 2.9569 | lr 6.63e-03 | grad_norm 0.20 | 691654 tok/s
iter   6470 | loss 2.9610 | lr 6.62e-03 | grad_norm 0.20 | 692223 tok/s
iter   6480 | loss 2.9623 | lr 6.61e-03 | grad_norm 0.20 | 692420 tok/s
iter   6490 | loss 2.9680 | lr 6.60e-03 | grad_norm 0.18 | 689905 tok/s
iter   6500 | loss 2.9678 | lr 6.59e-03 | grad_norm 0.20 | 689691 tok/s
  eval | train_loss 2.9590 | val_loss 2.9598
Saved checkpoint at iter 6500 -> ckpt.pt
iter   6510 | loss 2.9571 | lr 6.58e-03 | grad_norm 0.20 | 436862 tok/s
iter   6520 | loss 2.9273 | lr 6.57e-03 | grad_norm 0.18 | 695194 tok/s
iter   6530 | loss 2.9483 | lr 6.56e-03 | grad_norm 0.19 | 696122 tok/s
iter   6540 | loss 2.9560 | lr 6.55e-03 | grad_norm 0.19 | 690011 tok/s
iter   6550 | loss 2.9226 | lr 6.53e-03 | grad_norm 0.19 | 691810 tok/s
iter   6560 | loss 2.9495 | lr 6.52e-03 | grad_norm 0.19 | 693360 tok/s
iter   6570 | loss 2.9557 | lr 6.51e-03 | grad_norm 0.19 | 692759 tok/s
iter   6580 | loss 2.9484 | lr 6.50e-03 | grad_norm 0.20 | 690649 tok/s
iter   6590 | loss 2.9269 | lr 6.49e-03 | grad_norm 0.20 | 691030 tok/s
iter   6600 | loss 2.9119 | lr 6.48e-03 | grad_norm 0.20 | 689776 tok/s
  eval | train_loss 2.9465 | val_loss 2.9626
Saved checkpoint at iter 6600 -> ckpt.pt
iter   6610 | loss 2.9644 | lr 6.47e-03 | grad_norm 0.18 | 436087 tok/s
iter   6620 | loss 2.9556 | lr 6.46e-03 | grad_norm 0.20 | 691973 tok/s
iter   6630 | loss 2.9680 | lr 6.45e-03 | grad_norm 0.19 | 692245 tok/s
iter   6640 | loss 2.9662 | lr 6.44e-03 | grad_norm 0.20 | 690708 tok/s
iter   6650 | loss 2.9476 | lr 6.43e-03 | grad_norm 0.19 | 691189 tok/s
iter   6660 | loss 2.9644 | lr 6.42e-03 | grad_norm 0.18 | 693369 tok/s
iter   6670 | loss 2.9274 | lr 6.41e-03 | grad_norm 0.20 | 691463 tok/s
iter   6680 | loss 2.9504 | lr 6.40e-03 | grad_norm 0.20 | 688325 tok/s
iter   6690 | loss 2.9636 | lr 6.39e-03 | grad_norm 0.18 | 692680 tok/s
iter   6700 | loss 2.9504 | lr 6.38e-03 | grad_norm 0.20 | 692672 tok/s
  eval | train_loss 2.9508 | val_loss 2.9659
Saved checkpoint at iter 6700 -> ckpt.pt
iter   6710 | loss 2.9706 | lr 6.37e-03 | grad_norm 0.19 | 438250 tok/s
iter   6720 | loss 2.9474 | lr 6.36e-03 | grad_norm 0.19 | 695829 tok/s
iter   6730 | loss 2.9286 | lr 6.35e-03 | grad_norm 0.18 | 694169 tok/s
iter   6740 | loss 2.9286 | lr 6.34e-03 | grad_norm 0.19 | 689339 tok/s
iter   6750 | loss 2.9693 | lr 6.33e-03 | grad_norm 0.20 | 689985 tok/s
iter   6760 | loss 2.9446 | lr 6.32e-03 | grad_norm 0.19 | 694866 tok/s
iter   6770 | loss 2.9643 | lr 6.31e-03 | grad_norm 0.19 | 689433 tok/s
iter   6780 | loss 2.9707 | lr 6.30e-03 | grad_norm 0.19 | 693104 tok/s
iter   6790 | loss 2.9581 | lr 6.29e-03 | grad_norm 0.18 | 696021 tok/s
iter   6800 | loss 2.9922 | lr 6.28e-03 | grad_norm 0.20 | 696172 tok/s
  eval | train_loss 2.9464 | val_loss 2.9564
Saved checkpoint at iter 6800 -> ckpt.pt
Saved checkpoint at iter 6800 -> ckpt_best.pt
  New best val_loss: 2.9564
iter   6810 | loss 2.9495 | lr 6.27e-03 | grad_norm 0.19 | 423846 tok/s
iter   6820 | loss 2.9439 | lr 6.26e-03 | grad_norm 0.19 | 691776 tok/s
iter   6830 | loss 2.9433 | lr 6.24e-03 | grad_norm 0.18 | 691014 tok/s
iter   6840 | loss 2.9683 | lr 6.23e-03 | grad_norm 0.20 | 693937 tok/s
iter   6850 | loss 2.9245 | lr 6.22e-03 | grad_norm 0.19 | 695632 tok/s
iter   6860 | loss 2.9474 | lr 6.21e-03 | grad_norm 0.20 | 697696 tok/s
iter   6870 | loss 2.9257 | lr 6.20e-03 | grad_norm 0.18 | 696090 tok/s
iter   6880 | loss 2.9689 | lr 6.19e-03 | grad_norm 0.19 | 696903 tok/s
iter   6890 | loss 2.9399 | lr 6.18e-03 | grad_norm 0.18 | 692964 tok/s
iter   6900 | loss 2.9647 | lr 6.17e-03 | grad_norm 0.19 | 689639 tok/s
  eval | train_loss 2.9539 | val_loss 2.9548
Saved checkpoint at iter 6900 -> ckpt.pt
Saved checkpoint at iter 6900 -> ckpt_best.pt
  New best val_loss: 2.9548
iter   6910 | loss 2.9163 | lr 6.16e-03 | grad_norm 0.20 | 424028 tok/s
iter   6920 | loss 2.9541 | lr 6.15e-03 | grad_norm 0.20 | 693517 tok/s
iter   6930 | loss 2.9355 | lr 6.14e-03 | grad_norm 0.19 | 690521 tok/s
iter   6940 | loss 2.9322 | lr 6.13e-03 | grad_norm 0.19 | 691439 tok/s
iter   6950 | loss 2.9675 | lr 6.12e-03 | grad_norm 0.19 | 691608 tok/s
iter   6960 | loss 2.9472 | lr 6.11e-03 | grad_norm 0.19 | 693130 tok/s
iter   6970 | loss 2.9405 | lr 6.10e-03 | grad_norm 0.19 | 695247 tok/s
iter   6980 | loss 2.9284 | lr 6.09e-03 | grad_norm 0.20 | 695202 tok/s
iter   6990 | loss 2.9525 | lr 6.08e-03 | grad_norm 0.20 | 694609 tok/s
iter   7000 | loss 2.9791 | lr 6.07e-03 | grad_norm 0.18 | 696639 tok/s
  eval | train_loss 2.9570 | val_loss 2.9468
Saved checkpoint at iter 7000 -> ckpt.pt
Saved checkpoint at iter 7000 -> ckpt_best.pt
  New best val_loss: 2.9468
iter   7010 | loss 2.9401 | lr 6.06e-03 | grad_norm 0.18 | 421172 tok/s
iter   7020 | loss 2.9227 | lr 6.05e-03 | grad_norm 0.19 | 692528 tok/s
iter   7030 | loss 2.9454 | lr 6.04e-03 | grad_norm 0.18 | 696487 tok/s
iter   7040 | loss 2.9563 | lr 6.03e-03 | grad_norm 0.17 | 692731 tok/s
iter   7050 | loss 2.9229 | lr 6.02e-03 | grad_norm 0.19 | 693341 tok/s
iter   7060 | loss 2.9514 | lr 6.01e-03 | grad_norm 0.20 | 689953 tok/s
iter   7070 | loss 2.9461 | lr 6.00e-03 | grad_norm 0.18 | 686467 tok/s
iter   7080 | loss 2.9504 | lr 5.99e-03 | grad_norm 0.17 | 692139 tok/s
iter   7090 | loss 2.9404 | lr 5.98e-03 | grad_norm 0.18 | 694013 tok/s
iter   7100 | loss 2.9685 | lr 5.97e-03 | grad_norm 0.18 | 693791 tok/s
  eval | train_loss 2.9443 | val_loss 2.9449
Saved checkpoint at iter 7100 -> ckpt.pt
Saved checkpoint at iter 7100 -> ckpt_best.pt
  New best val_loss: 2.9449
iter   7110 | loss 2.9439 | lr 5.95e-03 | grad_norm 0.19 | 423202 tok/s
iter   7120 | loss 2.9112 | lr 5.94e-03 | grad_norm 0.19 | 694068 tok/s
iter   7130 | loss 2.9592 | lr 5.93e-03 | grad_norm 0.18 | 694942 tok/s
iter   7140 | loss 2.9574 | lr 5.92e-03 | grad_norm 0.20 | 694215 tok/s
iter   7150 | loss 2.9341 | lr 5.91e-03 | grad_norm 0.20 | 694466 tok/s
iter   7160 | loss 2.9206 | lr 5.90e-03 | grad_norm 0.19 | 697374 tok/s
iter   7170 | loss 2.9430 | lr 5.89e-03 | grad_norm 0.18 | 692977 tok/s
iter   7180 | loss 2.9274 | lr 5.88e-03 | grad_norm 0.18 | 693306 tok/s
iter   7190 | loss 2.9392 | lr 5.87e-03 | grad_norm 0.17 | 693964 tok/s
iter   7200 | loss 2.9520 | lr 5.86e-03 | grad_norm 0.19 | 693842 tok/s
  eval | train_loss 2.9526 | val_loss 2.9465
Saved checkpoint at iter 7200 -> ckpt.pt
iter   7210 | loss 2.9429 | lr 5.85e-03 | grad_norm 0.17 | 438301 tok/s
iter   7220 | loss 2.9491 | lr 5.84e-03 | grad_norm 0.18 | 696179 tok/s
iter   7230 | loss 2.9525 | lr 5.83e-03 | grad_norm 0.19 | 694224 tok/s
iter   7240 | loss 2.9441 | lr 5.82e-03 | grad_norm 0.17 | 696679 tok/s
iter   7250 | loss 2.9269 | lr 5.81e-03 | grad_norm 0.18 | 691887 tok/s
iter   7260 | loss 2.9571 | lr 5.80e-03 | grad_norm 0.18 | 692810 tok/s
iter   7270 | loss 2.9571 | lr 5.79e-03 | grad_norm 0.19 | 691287 tok/s
iter   7280 | loss 2.9501 | lr 5.78e-03 | grad_norm 0.17 | 692301 tok/s
iter   7290 | loss 2.9368 | lr 5.77e-03 | grad_norm 0.18 | 691692 tok/s
iter   7300 | loss 2.9402 | lr 5.76e-03 | grad_norm 0.18 | 694092 tok/s
  eval | train_loss 2.9411 | val_loss 2.9476
Saved checkpoint at iter 7300 -> ckpt.pt
iter   7310 | loss 2.9455 | lr 5.75e-03 | grad_norm 0.20 | 435442 tok/s
iter   7320 | loss 2.9285 | lr 5.74e-03 | grad_norm 0.19 | 693515 tok/s
iter   7330 | loss 2.9513 | lr 5.73e-03 | grad_norm 0.18 | 690351 tok/s
iter   7340 | loss 2.9328 | lr 5.72e-03 | grad_norm 0.18 | 687602 tok/s
iter   7350 | loss 2.9492 | lr 5.71e-03 | grad_norm 0.18 | 692649 tok/s
iter   7360 | loss 2.9385 | lr 5.70e-03 | grad_norm 0.19 | 692234 tok/s
iter   7370 | loss 2.9352 | lr 5.69e-03 | grad_norm 0.18 | 691175 tok/s
iter   7380 | loss 2.9298 | lr 5.68e-03 | grad_norm 0.18 | 693185 tok/s
iter   7390 | loss 2.9140 | lr 5.66e-03 | grad_norm 0.18 | 693317 tok/s
iter   7400 | loss 2.9271 | lr 5.65e-03 | grad_norm 0.18 | 695244 tok/s
  eval | train_loss 2.9407 | val_loss 2.9455
Saved checkpoint at iter 7400 -> ckpt.pt
iter   7410 | loss 2.9330 | lr 5.64e-03 | grad_norm 0.17 | 437828 tok/s
iter   7420 | loss 2.9444 | lr 5.63e-03 | grad_norm 0.17 | 689073 tok/s
iter   7430 | loss 2.9433 | lr 5.62e-03 | grad_norm 0.18 | 691067 tok/s
iter   7440 | loss 2.9415 | lr 5.61e-03 | grad_norm 0.18 | 693127 tok/s
iter   7450 | loss 2.9526 | lr 5.60e-03 | grad_norm 0.17 | 693905 tok/s
iter   7460 | loss 2.9522 | lr 5.59e-03 | grad_norm 0.18 | 694659 tok/s
iter   7470 | loss 2.9391 | lr 5.58e-03 | grad_norm 0.17 | 692772 tok/s
iter   7480 | loss 2.9435 | lr 5.57e-03 | grad_norm 0.18 | 694481 tok/s
iter   7490 | loss 2.9537 | lr 5.56e-03 | grad_norm 0.18 | 691908 tok/s
iter   7500 | loss 2.9263 | lr 5.55e-03 | grad_norm 0.19 | 692463 tok/s
  eval | train_loss 2.9445 | val_loss 2.9425
Saved checkpoint at iter 7500 -> ckpt.pt
Saved checkpoint at iter 7500 -> ckpt_best.pt
  New best val_loss: 2.9425
  quick_eval: import failed (No module named 'evals'); skipping
iter   7510 | loss 2.9274 | lr 5.54e-03 | grad_norm 0.18 | 419767 tok/s
iter   7520 | loss 2.9157 | lr 5.53e-03 | grad_norm 0.18 | 695825 tok/s
iter   7530 | loss 2.9618 | lr 5.52e-03 | grad_norm 0.18 | 691803 tok/s
iter   7540 | loss 2.9381 | lr 5.51e-03 | grad_norm 0.17 | 693760 tok/s
iter   7550 | loss 2.9403 | lr 5.50e-03 | grad_norm 0.17 | 696125 tok/s
iter   7560 | loss 2.9642 | lr 5.49e-03 | grad_norm 0.17 | 695938 tok/s
iter   7570 | loss 2.9435 | lr 5.48e-03 | grad_norm 0.17 | 696423 tok/s
iter   7580 | loss 2.9257 | lr 5.47e-03 | grad_norm 0.17 | 696733 tok/s
iter   7590 | loss 2.9370 | lr 5.46e-03 | grad_norm 0.18 | 681671 tok/s
iter   7600 | loss 2.9228 | lr 5.45e-03 | grad_norm 0.18 | 696129 tok/s
  eval | train_loss 2.9486 | val_loss 2.9350
Saved checkpoint at iter 7600 -> ckpt.pt
Saved checkpoint at iter 7600 -> ckpt_best.pt
  New best val_loss: 2.9350
iter   7610 | loss 2.9451 | lr 5.44e-03 | grad_norm 0.18 | 428599 tok/s
iter   7620 | loss 2.9596 | lr 5.43e-03 | grad_norm 0.18 | 695052 tok/s
iter   7630 | loss 2.9280 | lr 5.42e-03 | grad_norm 0.17 | 694189 tok/s
iter   7640 | loss 2.9381 | lr 5.41e-03 | grad_norm 0.18 | 691756 tok/s
iter   7650 | loss 2.9397 | lr 5.40e-03 | grad_norm 0.18 | 692332 tok/s
iter   7660 | loss 2.9480 | lr 5.39e-03 | grad_norm 0.17 | 690965 tok/s
iter   7670 | loss 2.9089 | lr 5.38e-03 | grad_norm 0.18 | 695215 tok/s
iter   7680 | loss 2.9444 | lr 5.36e-03 | grad_norm 0.16 | 696965 tok/s
iter   7690 | loss 2.9344 | lr 5.35e-03 | grad_norm 0.17 | 692103 tok/s
iter   7700 | loss 2.9402 | lr 5.34e-03 | grad_norm 0.18 | 689842 tok/s
  eval | train_loss 2.9310 | val_loss 2.9319
Saved checkpoint at iter 7700 -> ckpt.pt
Saved checkpoint at iter 7700 -> ckpt_best.pt
  New best val_loss: 2.9319
iter   7710 | loss 2.9424 | lr 5.33e-03 | grad_norm 0.16 | 425294 tok/s
iter   7720 | loss 2.9478 | lr 5.32e-03 | grad_norm 0.17 | 692782 tok/s
iter   7730 | loss 2.9403 | lr 5.31e-03 | grad_norm 0.18 | 694744 tok/s
iter   7740 | loss 2.9187 | lr 5.30e-03 | grad_norm 0.17 | 692736 tok/s
iter   7750 | loss 2.9256 | lr 5.29e-03 | grad_norm 0.17 | 691276 tok/s
iter   7760 | loss 2.9190 | lr 5.28e-03 | grad_norm 0.18 | 693729 tok/s
iter   7770 | loss 2.9294 | lr 5.27e-03 | grad_norm 0.17 | 693449 tok/s
iter   7780 | loss 2.9401 | lr 5.26e-03 | grad_norm 0.16 | 689570 tok/s
iter   7790 | loss 2.9145 | lr 5.25e-03 | grad_norm 0.18 | 689828 tok/s
iter   7800 | loss 2.9149 | lr 5.24e-03 | grad_norm 0.18 | 692570 tok/s
  eval | train_loss 2.9324 | val_loss 2.9324
Saved checkpoint at iter 7800 -> ckpt.pt
iter   7810 | loss 2.9075 | lr 5.23e-03 | grad_norm 0.16 | 438209 tok/s
iter   7820 | loss 2.9299 | lr 5.22e-03 | grad_norm 0.17 | 692278 tok/s
iter   7830 | loss 2.9574 | lr 5.21e-03 | grad_norm 0.16 | 695794 tok/s
iter   7840 | loss 2.9214 | lr 5.20e-03 | grad_norm 0.16 | 695553 tok/s
iter   7850 | loss 2.9216 | lr 5.19e-03 | grad_norm 0.16 | 692189 tok/s
iter   7860 | loss 2.9361 | lr 5.18e-03 | grad_norm 0.17 | 690526 tok/s
iter   7870 | loss 2.9324 | lr 5.17e-03 | grad_norm 0.17 | 690890 tok/s
iter   7880 | loss 2.9448 | lr 5.16e-03 | grad_norm 0.17 | 690977 tok/s
iter   7890 | loss 2.9250 | lr 5.15e-03 | grad_norm 0.17 | 690986 tok/s
iter   7900 | loss 2.9323 | lr 5.14e-03 | grad_norm 0.17 | 689406 tok/s
  eval | train_loss 2.9306 | val_loss 2.9257
Saved checkpoint at iter 7900 -> ckpt.pt
Saved checkpoint at iter 7900 -> ckpt_best.pt
  New best val_loss: 2.9257
iter   7910 | loss 2.9142 | lr 5.13e-03 | grad_norm 0.16 | 424391 tok/s
iter   7920 | loss 2.9357 | lr 5.12e-03 | grad_norm 0.18 | 691603 tok/s
iter   7930 | loss 2.9398 | lr 5.11e-03 | grad_norm 0.17 | 689908 tok/s
iter   7940 | loss 2.9124 | lr 5.10e-03 | grad_norm 0.17 | 690997 tok/s
iter   7950 | loss 2.9160 | lr 5.09e-03 | grad_norm 0.17 | 688282 tok/s
iter   7960 | loss 2.9284 | lr 5.07e-03 | grad_norm 0.17 | 690004 tok/s
iter   7970 | loss 2.9387 | lr 5.06e-03 | grad_norm 0.17 | 689680 tok/s
iter   7980 | loss 2.9164 | lr 5.05e-03 | grad_norm 0.17 | 694509 tok/s
iter   7990 | loss 2.9468 | lr 5.04e-03 | grad_norm 0.16 | 692092 tok/s
iter   8000 | loss 2.9245 | lr 5.03e-03 | grad_norm 0.16 | 695408 tok/s
  eval | train_loss 2.9379 | val_loss 2.9253
Saved checkpoint at iter 8000 -> ckpt.pt
Saved checkpoint at iter 8000 -> ckpt_best.pt
  New best val_loss: 2.9253
iter   8010 | loss 2.9113 | lr 5.02e-03 | grad_norm 0.17 | 424255 tok/s
iter   8020 | loss 2.9077 | lr 5.01e-03 | grad_norm 0.17 | 698204 tok/s
iter   8030 | loss 2.9277 | lr 5.00e-03 | grad_norm 0.17 | 698257 tok/s
iter   8040 | loss 2.9576 | lr 4.99e-03 | grad_norm 0.18 | 694840 tok/s
iter   8050 | loss 2.9261 | lr 4.98e-03 | grad_norm 0.16 | 691974 tok/s
iter   8060 | loss 2.9129 | lr 4.97e-03 | grad_norm 0.17 | 692959 tok/s
iter   8070 | loss 2.9451 | lr 4.96e-03 | grad_norm 0.18 | 689414 tok/s
iter   8080 | loss 2.9343 | lr 4.95e-03 | grad_norm 0.17 | 693741 tok/s
iter   8090 | loss 2.9187 | lr 4.94e-03 | grad_norm 0.17 | 694071 tok/s
iter   8100 | loss 2.9089 | lr 4.93e-03 | grad_norm 0.17 | 692926 tok/s
  eval | train_loss 2.9214 | val_loss 2.9201
Saved checkpoint at iter 8100 -> ckpt.pt
Saved checkpoint at iter 8100 -> ckpt_best.pt
  New best val_loss: 2.9201
iter   8110 | loss 2.9027 | lr 4.92e-03 | grad_norm 0.17 | 417904 tok/s
iter   8120 | loss 2.9490 | lr 4.91e-03 | grad_norm 0.18 | 695551 tok/s
iter   8130 | loss 2.9344 | lr 4.90e-03 | grad_norm 0.17 | 694121 tok/s
iter   8140 | loss 2.9338 | lr 4.89e-03 | grad_norm 0.16 | 693850 tok/s
iter   8150 | loss 2.9097 | lr 4.88e-03 | grad_norm 0.17 | 694592 tok/s
iter   8160 | loss 2.9405 | lr 4.87e-03 | grad_norm 0.16 | 692068 tok/s
iter   8170 | loss 2.9367 | lr 4.86e-03 | grad_norm 0.16 | 693026 tok/s
iter   8180 | loss 2.9140 | lr 4.85e-03 | grad_norm 0.15 | 692416 tok/s
iter   8190 | loss 2.9340 | lr 4.84e-03 | grad_norm 0.17 | 689011 tok/s
iter   8200 | loss 2.9213 | lr 4.83e-03 | grad_norm 0.16 | 690617 tok/s
  eval | train_loss 2.9253 | val_loss 2.9318
Saved checkpoint at iter 8200 -> ckpt.pt
iter   8210 | loss 2.9329 | lr 4.82e-03 | grad_norm 0.17 | 435297 tok/s
iter   8220 | loss 2.9045 | lr 4.81e-03 | grad_norm 0.15 | 694712 tok/s
iter   8230 | loss 2.9143 | lr 4.80e-03 | grad_norm 0.16 | 694676 tok/s
iter   8240 | loss 2.9374 | lr 4.78e-03 | grad_norm 0.16 | 691831 tok/s
iter   8250 | loss 2.9347 | lr 4.77e-03 | grad_norm 0.16 | 692671 tok/s
iter   8260 | loss 2.9198 | lr 4.76e-03 | grad_norm 0.14 | 695230 tok/s
iter   8270 | loss 2.9318 | lr 4.75e-03 | grad_norm 0.16 | 698382 tok/s
iter   8280 | loss 2.9068 | lr 4.74e-03 | grad_norm 0.16 | 693924 tok/s
iter   8290 | loss 2.9129 | lr 4.73e-03 | grad_norm 0.15 | 689875 tok/s
iter   8300 | loss 2.9362 | lr 4.72e-03 | grad_norm 0.17 | 692757 tok/s
  eval | train_loss 2.9248 | val_loss 2.9328
Saved checkpoint at iter 8300 -> ckpt.pt
iter   8310 | loss 2.9470 | lr 4.71e-03 | grad_norm 0.16 | 435109 tok/s
iter   8320 | loss 2.9305 | lr 4.70e-03 | grad_norm 0.15 | 691976 tok/s
iter   8330 | loss 2.9129 | lr 4.69e-03 | grad_norm 0.16 | 689945 tok/s
iter   8340 | loss 2.9205 | lr 4.68e-03 | grad_norm 0.17 | 690822 tok/s
iter   8350 | loss 2.9064 | lr 4.67e-03 | grad_norm 0.16 | 692884 tok/s
iter   8360 | loss 2.9132 | lr 4.66e-03 | grad_norm 0.17 | 696386 tok/s
iter   8370 | loss 2.9099 | lr 4.65e-03 | grad_norm 0.17 | 693664 tok/s
iter   8380 | loss 2.9013 | lr 4.64e-03 | grad_norm 0.16 | 690067 tok/s
iter   8390 | loss 2.9204 | lr 4.63e-03 | grad_norm 0.18 | 688273 tok/s
iter   8400 | loss 2.9089 | lr 4.62e-03 | grad_norm 0.15 | 688375 tok/s
  eval | train_loss 2.9248 | val_loss 2.9230
Saved checkpoint at iter 8400 -> ckpt.pt
iter   8410 | loss 2.9365 | lr 4.61e-03 | grad_norm 0.16 | 434540 tok/s
iter   8420 | loss 2.9317 | lr 4.60e-03 | grad_norm 0.16 | 692329 tok/s
iter   8430 | loss 2.9416 | lr 4.59e-03 | grad_norm 0.15 | 694604 tok/s
iter   8440 | loss 2.9206 | lr 4.58e-03 | grad_norm 0.16 | 693112 tok/s
iter   8450 | loss 2.9239 | lr 4.57e-03 | grad_norm 0.15 | 691750 tok/s
iter   8460 | loss 2.9109 | lr 4.56e-03 | grad_norm 0.16 | 695051 tok/s
iter   8470 | loss 2.9092 | lr 4.55e-03 | grad_norm 0.15 | 693593 tok/s
iter   8480 | loss 2.9305 | lr 4.54e-03 | grad_norm 0.15 | 693850 tok/s
iter   8490 | loss 2.9350 | lr 4.53e-03 | grad_norm 0.17 | 692511 tok/s
iter   8500 | loss 2.9267 | lr 4.52e-03 | grad_norm 0.16 | 693943 tok/s
  eval | train_loss 2.9258 | val_loss 2.9303
Saved checkpoint at iter 8500 -> ckpt.pt
iter   8510 | loss 2.9064 | lr 4.51e-03 | grad_norm 0.17 | 437607 tok/s
iter   8520 | loss 2.9375 | lr 4.49e-03 | grad_norm 0.16 | 693889 tok/s
iter   8530 | loss 2.9152 | lr 4.48e-03 | grad_norm 0.15 | 690418 tok/s
iter   8540 | loss 2.9008 | lr 4.47e-03 | grad_norm 0.15 | 693865 tok/s
iter   8550 | loss 2.9102 | lr 4.46e-03 | grad_norm 0.15 | 691342 tok/s
iter   8560 | loss 2.9313 | lr 4.45e-03 | grad_norm 0.17 | 693421 tok/s
iter   8570 | loss 2.9340 | lr 4.44e-03 | grad_norm 0.15 | 693986 tok/s
iter   8580 | loss 2.9292 | lr 4.43e-03 | grad_norm 0.16 | 692539 tok/s
iter   8590 | loss 2.9051 | lr 4.42e-03 | grad_norm 0.16 | 691191 tok/s
iter   8600 | loss 2.9067 | lr 4.41e-03 | grad_norm 0.15 | 691108 tok/s
  eval | train_loss 2.9155 | val_loss 2.9201
Saved checkpoint at iter 8600 -> ckpt.pt
iter   8610 | loss 2.9436 | lr 4.40e-03 | grad_norm 0.15 | 431979 tok/s
iter   8620 | loss 2.9264 | lr 4.39e-03 | grad_norm 0.15 | 693392 tok/s
iter   8630 | loss 2.9128 | lr 4.38e-03 | grad_norm 0.15 | 692329 tok/s
iter   8640 | loss 2.9076 | lr 4.37e-03 | grad_norm 0.14 | 691087 tok/s
iter   8650 | loss 2.9330 | lr 4.36e-03 | grad_norm 0.15 | 695102 tok/s
iter   8660 | loss 2.9011 | lr 4.35e-03 | grad_norm 0.15 | 692014 tok/s
iter   8670 | loss 2.9061 | lr 4.34e-03 | grad_norm 0.14 | 688759 tok/s
iter   8680 | loss 2.9050 | lr 4.33e-03 | grad_norm 0.15 | 690404 tok/s
iter   8690 | loss 2.9083 | lr 4.32e-03 | grad_norm 0.15 | 693556 tok/s
iter   8700 | loss 2.9403 | lr 4.31e-03 | grad_norm 0.15 | 692081 tok/s
  eval | train_loss 2.9221 | val_loss 2.9200
Saved checkpoint at iter 8700 -> ckpt.pt
Saved checkpoint at iter 8700 -> ckpt_best.pt
  New best val_loss: 2.9200
iter   8710 | loss 2.9176 | lr 4.30e-03 | grad_norm 0.15 | 421002 tok/s
iter   8720 | loss 2.9158 | lr 4.29e-03 | grad_norm 0.15 | 695758 tok/s
iter   8730 | loss 2.9202 | lr 4.28e-03 | grad_norm 0.14 | 692881 tok/s
iter   8740 | loss 2.9075 | lr 4.27e-03 | grad_norm 0.15 | 693310 tok/s
iter   8750 | loss 2.9046 | lr 4.26e-03 | grad_norm 0.14 | 691509 tok/s
iter   8760 | loss 2.9232 | lr 4.25e-03 | grad_norm 0.14 | 689467 tok/s
iter   8770 | loss 2.9218 | lr 4.24e-03 | grad_norm 0.14 | 691163 tok/s
iter   8780 | loss 2.9232 | lr 4.23e-03 | grad_norm 0.15 | 691322 tok/s
iter   8790 | loss 2.9215 | lr 4.22e-03 | grad_norm 0.15 | 692619 tok/s
iter   8800 | loss 2.9269 | lr 4.20e-03 | grad_norm 0.16 | 693490 tok/s
  eval | train_loss 2.9225 | val_loss 2.9257
Saved checkpoint at iter 8800 -> ckpt.pt
iter   8810 | loss 2.9286 | lr 4.19e-03 | grad_norm 0.14 | 438283 tok/s
iter   8820 | loss 2.9169 | lr 4.18e-03 | grad_norm 0.14 | 692425 tok/s
iter   8830 | loss 2.9189 | lr 4.17e-03 | grad_norm 0.14 | 690545 tok/s
iter   8840 | loss 2.9169 | lr 4.16e-03 | grad_norm 0.15 | 694244 tok/s
iter   8850 | loss 2.9101 | lr 4.15e-03 | grad_norm 0.15 | 689873 tok/s
iter   8860 | loss 2.9120 | lr 4.14e-03 | grad_norm 0.15 | 689208 tok/s
iter   8870 | loss 2.9355 | lr 4.13e-03 | grad_norm 0.15 | 688502 tok/s
iter   8880 | loss 2.9089 | lr 4.12e-03 | grad_norm 0.14 | 691625 tok/s
iter   8890 | loss 2.9004 | lr 4.11e-03 | grad_norm 0.16 | 690395 tok/s
iter   8900 | loss 2.8854 | lr 4.10e-03 | grad_norm 0.15 | 690351 tok/s
  eval | train_loss 2.9119 | val_loss 2.9021
Saved checkpoint at iter 8900 -> ckpt.pt
Saved checkpoint at iter 8900 -> ckpt_best.pt
  New best val_loss: 2.9021
iter   8910 | loss 2.9216 | lr 4.09e-03 | grad_norm 0.15 | 428843 tok/s
iter   8920 | loss 2.9288 | lr 4.08e-03 | grad_norm 0.15 | 690169 tok/s
iter   8930 | loss 2.8953 | lr 4.07e-03 | grad_norm 0.14 | 693311 tok/s
iter   8940 | loss 2.9163 | lr 4.06e-03 | grad_norm 0.14 | 691057 tok/s
iter   8950 | loss 2.8988 | lr 4.05e-03 | grad_norm 0.14 | 690469 tok/s
iter   8960 | loss 2.9184 | lr 4.04e-03 | grad_norm 0.15 | 690584 tok/s
iter   8970 | loss 2.9008 | lr 4.03e-03 | grad_norm 0.14 | 690291 tok/s
iter   8980 | loss 2.9155 | lr 4.02e-03 | grad_norm 0.15 | 691367 tok/s
iter   8990 | loss 2.9068 | lr 4.01e-03 | grad_norm 0.14 | 689073 tok/s
iter   9000 | loss 2.9271 | lr 4.00e-03 | grad_norm 0.14 | 687262 tok/s
  eval | train_loss 2.9102 | val_loss 2.9202
Saved checkpoint at iter 9000 -> ckpt.pt
iter   9010 | loss 2.9196 | lr 3.99e-03 | grad_norm 0.15 | 435859 tok/s
iter   9020 | loss 2.9140 | lr 3.98e-03 | grad_norm 0.14 | 695596 tok/s
iter   9030 | loss 2.9292 | lr 3.97e-03 | grad_norm 0.14 | 695865 tok/s
iter   9040 | loss 2.9067 | lr 3.96e-03 | grad_norm 0.14 | 692475 tok/s
iter   9050 | loss 2.9037 | lr 3.95e-03 | grad_norm 0.14 | 692382 tok/s
iter   9060 | loss 2.9117 | lr 3.94e-03 | grad_norm 0.14 | 689303 tok/s
iter   9070 | loss 2.8989 | lr 3.93e-03 | grad_norm 0.13 | 691588 tok/s
iter   9080 | loss 2.9247 | lr 3.91e-03 | grad_norm 0.14 | 691481 tok/s
iter   9090 | loss 2.9178 | lr 3.90e-03 | grad_norm 0.13 | 691892 tok/s
iter   9100 | loss 2.9198 | lr 3.89e-03 | grad_norm 0.13 | 695437 tok/s
  eval | train_loss 2.9127 | val_loss 2.9129
Saved checkpoint at iter 9100 -> ckpt.pt
iter   9110 | loss 2.9084 | lr 3.88e-03 | grad_norm 0.13 | 438033 tok/s
iter   9120 | loss 2.8953 | lr 3.87e-03 | grad_norm 0.13 | 693570 tok/s
iter   9130 | loss 2.9160 | lr 3.86e-03 | grad_norm 0.14 | 689445 tok/s
iter   9140 | loss 2.9110 | lr 3.85e-03 | grad_norm 0.13 | 694193 tok/s
iter   9150 | loss 2.9207 | lr 3.84e-03 | grad_norm 0.14 | 692308 tok/s
iter   9160 | loss 2.9205 | lr 3.83e-03 | grad_norm 0.14 | 694190 tok/s
iter   9170 | loss 2.9132 | lr 3.82e-03 | grad_norm 0.13 | 691755 tok/s
iter   9180 | loss 2.9206 | lr 3.81e-03 | grad_norm 0.14 | 690322 tok/s
iter   9190 | loss 2.8857 | lr 3.80e-03 | grad_norm 0.14 | 690357 tok/s
iter   9200 | loss 2.8796 | lr 3.79e-03 | grad_norm 0.14 | 693106 tok/s
  eval | train_loss 2.9013 | val_loss 2.9157
Saved checkpoint at iter 9200 -> ckpt.pt
iter   9210 | loss 2.8863 | lr 3.78e-03 | grad_norm 0.13 | 437144 tok/s
iter   9220 | loss 2.8914 | lr 3.77e-03 | grad_norm 0.14 | 694097 tok/s
iter   9230 | loss 2.9074 | lr 3.76e-03 | grad_norm 0.13 | 693539 tok/s
iter   9240 | loss 2.9183 | lr 3.75e-03 | grad_norm 0.15 | 693276 tok/s
iter   9250 | loss 2.9253 | lr 3.74e-03 | grad_norm 0.14 | 692981 tok/s
iter   9260 | loss 2.8910 | lr 3.73e-03 | grad_norm 0.12 | 694534 tok/s
iter   9270 | loss 2.9045 | lr 3.72e-03 | grad_norm 0.12 | 693980 tok/s
iter   9280 | loss 2.9251 | lr 3.71e-03 | grad_norm 0.14 | 690676 tok/s
iter   9290 | loss 2.9289 | lr 3.70e-03 | grad_norm 0.13 | 687769 tok/s
iter   9300 | loss 2.8736 | lr 3.69e-03 | grad_norm 0.14 | 694787 tok/s
  eval | train_loss 2.8965 | val_loss 2.9057
Saved checkpoint at iter 9300 -> ckpt.pt
iter   9310 | loss 2.8959 | lr 3.68e-03 | grad_norm 0.14 | 438953 tok/s
iter   9320 | loss 2.9157 | lr 3.67e-03 | grad_norm 0.14 | 693685 tok/s
iter   9330 | loss 2.9070 | lr 3.66e-03 | grad_norm 0.13 | 691547 tok/s
iter   9340 | loss 2.9087 | lr 3.65e-03 | grad_norm 0.14 | 692487 tok/s
iter   9350 | loss 2.9121 | lr 3.64e-03 | grad_norm 0.13 | 691931 tok/s
iter   9360 | loss 2.9028 | lr 3.63e-03 | grad_norm 0.13 | 692755 tok/s
iter   9370 | loss 2.9229 | lr 3.61e-03 | grad_norm 0.13 | 692267 tok/s
iter   9380 | loss 2.9156 | lr 3.60e-03 | grad_norm 0.13 | 693083 tok/s
iter   9390 | loss 2.9132 | lr 3.59e-03 | grad_norm 0.13 | 692904 tok/s
iter   9400 | loss 2.9220 | lr 3.58e-03 | grad_norm 0.13 | 695457 tok/s
  eval | train_loss 2.9025 | val_loss 2.9099
Saved checkpoint at iter 9400 -> ckpt.pt
iter   9410 | loss 2.9177 | lr 3.57e-03 | grad_norm 0.13 | 434481 tok/s
iter   9420 | loss 2.8965 | lr 3.56e-03 | grad_norm 0.13 | 695610 tok/s
iter   9430 | loss 2.8933 | lr 3.55e-03 | grad_norm 0.13 | 695358 tok/s
iter   9440 | loss 2.8805 | lr 3.54e-03 | grad_norm 0.13 | 693479 tok/s
iter   9450 | loss 2.8948 | lr 3.53e-03 | grad_norm 0.13 | 693154 tok/s
iter   9460 | loss 2.8972 | lr 3.52e-03 | grad_norm 0.13 | 693541 tok/s
iter   9470 | loss 2.9326 | lr 3.51e-03 | grad_norm 0.13 | 696244 tok/s
iter   9480 | loss 2.9107 | lr 3.50e-03 | grad_norm 0.13 | 695679 tok/s
iter   9490 | loss 2.9045 | lr 3.49e-03 | grad_norm 0.12 | 694130 tok/s
iter   9500 | loss 2.9244 | lr 3.48e-03 | grad_norm 0.12 | 693667 tok/s
  eval | train_loss 2.9085 | val_loss 2.9081
Saved checkpoint at iter 9500 -> ckpt.pt
iter   9510 | loss 2.9001 | lr 3.47e-03 | grad_norm 0.13 | 435409 tok/s
iter   9520 | loss 2.9313 | lr 3.46e-03 | grad_norm 0.13 | 695935 tok/s
iter   9530 | loss 2.8920 | lr 3.45e-03 | grad_norm 0.13 | 693585 tok/s
iter   9540 | loss 2.8815 | lr 3.44e-03 | grad_norm 0.13 | 690292 tok/s
iter   9550 | loss 2.8964 | lr 3.43e-03 | grad_norm 0.12 | 697154 tok/s
iter   9560 | loss 2.9177 | lr 3.42e-03 | grad_norm 0.13 | 695419 tok/s
iter   9570 | loss 2.9218 | lr 3.41e-03 | grad_norm 0.13 | 690875 tok/s
iter   9580 | loss 2.8924 | lr 3.40e-03 | grad_norm 0.12 | 696429 tok/s
iter   9590 | loss 2.8998 | lr 3.39e-03 | grad_norm 0.12 | 697928 tok/s
iter   9600 | loss 2.8877 | lr 3.38e-03 | grad_norm 0.13 | 696241 tok/s
  eval | train_loss 2.8953 | val_loss 2.9006
Saved checkpoint at iter 9600 -> ckpt.pt
Saved checkpoint at iter 9600 -> ckpt_best.pt
  New best val_loss: 2.9006
iter   9610 | loss 2.9036 | lr 3.37e-03 | grad_norm 0.13 | 421853 tok/s
iter   9620 | loss 2.8713 | lr 3.36e-03 | grad_norm 0.13 | 698453 tok/s
iter   9630 | loss 2.9027 | lr 3.35e-03 | grad_norm 0.13 | 695522 tok/s
iter   9640 | loss 2.8989 | lr 3.34e-03 | grad_norm 0.13 | 691299 tok/s
iter   9650 | loss 2.9034 | lr 3.32e-03 | grad_norm 0.13 | 690170 tok/s
iter   9660 | loss 2.9101 | lr 3.31e-03 | grad_norm 0.13 | 690856 tok/s
iter   9670 | loss 2.8899 | lr 3.30e-03 | grad_norm 0.12 | 694269 tok/s
iter   9680 | loss 2.8966 | lr 3.29e-03 | grad_norm 0.13 | 692494 tok/s
iter   9690 | loss 2.8811 | lr 3.28e-03 | grad_norm 0.12 | 694029 tok/s
iter   9700 | loss 2.9139 | lr 3.27e-03 | grad_norm 0.13 | 692717 tok/s
  eval | train_loss 2.8942 | val_loss 2.8988
Saved checkpoint at iter 9700 -> ckpt.pt
Saved checkpoint at iter 9700 -> ckpt_best.pt
  New best val_loss: 2.8988
iter   9710 | loss 2.9166 | lr 3.26e-03 | grad_norm 0.13 | 427397 tok/s
iter   9720 | loss 2.9127 | lr 3.25e-03 | grad_norm 0.12 | 693254 tok/s
iter   9730 | loss 2.8976 | lr 3.24e-03 | grad_norm 0.12 | 695042 tok/s
iter   9740 | loss 2.9185 | lr 3.23e-03 | grad_norm 0.12 | 692454 tok/s
iter   9750 | loss 2.8928 | lr 3.22e-03 | grad_norm 0.13 | 694004 tok/s
iter   9760 | loss 2.8916 | lr 3.21e-03 | grad_norm 0.12 | 694043 tok/s
iter   9770 | loss 2.8840 | lr 3.20e-03 | grad_norm 0.12 | 689601 tok/s
iter   9780 | loss 2.9285 | lr 3.19e-03 | grad_norm 0.12 | 689367 tok/s
iter   9790 | loss 2.9117 | lr 3.18e-03 | grad_norm 0.12 | 691728 tok/s
iter   9800 | loss 2.8785 | lr 3.17e-03 | grad_norm 0.12 | 693446 tok/s
  eval | train_loss 2.9014 | val_loss 2.9046
Saved checkpoint at iter 9800 -> ckpt.pt
iter   9810 | loss 2.8825 | lr 3.16e-03 | grad_norm 0.12 | 438466 tok/s
iter   9820 | loss 2.9141 | lr 3.15e-03 | grad_norm 0.12 | 695150 tok/s
iter   9830 | loss 2.9094 | lr 3.14e-03 | grad_norm 0.12 | 698416 tok/s
iter   9840 | loss 2.8896 | lr 3.13e-03 | grad_norm 0.12 | 695479 tok/s
iter   9850 | loss 2.8792 | lr 3.12e-03 | grad_norm 0.12 | 693235 tok/s
iter   9860 | loss 2.9038 | lr 3.11e-03 | grad_norm 0.13 | 690728 tok/s
iter   9870 | loss 2.8919 | lr 3.10e-03 | grad_norm 0.13 | 693788 tok/s
iter   9880 | loss 2.8976 | lr 3.09e-03 | grad_norm 0.11 | 681651 tok/s
iter   9890 | loss 2.8942 | lr 3.08e-03 | grad_norm 0.12 | 691478 tok/s
iter   9900 | loss 2.8840 | lr 3.07e-03 | grad_norm 0.12 | 693468 tok/s
  eval | train_loss 2.8980 | val_loss 2.8974
Saved checkpoint at iter 9900 -> ckpt.pt
Saved checkpoint at iter 9900 -> ckpt_best.pt
  New best val_loss: 2.8974
iter   9910 | loss 2.8845 | lr 3.06e-03 | grad_norm 0.11 | 426422 tok/s
iter   9920 | loss 2.8895 | lr 3.05e-03 | grad_norm 0.12 | 692476 tok/s
iter   9930 | loss 2.8757 | lr 3.03e-03 | grad_norm 0.12 | 689074 tok/s
iter   9940 | loss 2.8749 | lr 3.02e-03 | grad_norm 0.12 | 689914 tok/s
iter   9950 | loss 2.8939 | lr 3.01e-03 | grad_norm 0.12 | 686807 tok/s
iter   9960 | loss 2.9018 | lr 3.00e-03 | grad_norm 0.12 | 691819 tok/s
iter   9970 | loss 2.8947 | lr 2.99e-03 | grad_norm 0.13 | 691952 tok/s
iter   9980 | loss 2.8909 | lr 2.98e-03 | grad_norm 0.12 | 692336 tok/s
iter   9990 | loss 2.8911 | lr 2.97e-03 | grad_norm 0.12 | 694846 tok/s
iter  10000 | loss 2.8711 | lr 2.96e-03 | grad_norm 0.12 | 693955 tok/s
  eval | train_loss 2.8950 | val_loss 2.9010
Saved checkpoint at iter 10000 -> ckpt.pt
  quick_eval: import failed (No module named 'evals'); skipping
iter  10010 | loss 2.8928 | lr 2.95e-03 | grad_norm 0.12 | 434750 tok/s
iter  10020 | loss 2.8878 | lr 2.94e-03 | grad_norm 0.12 | 694787 tok/s
iter  10030 | loss 2.8769 | lr 2.93e-03 | grad_norm 0.12 | 695603 tok/s
iter  10040 | loss 2.8698 | lr 2.92e-03 | grad_norm 0.12 | 693129 tok/s
iter  10050 | loss 2.8848 | lr 2.91e-03 | grad_norm 0.12 | 691170 tok/s
iter  10060 | loss 2.9114 | lr 2.90e-03 | grad_norm 0.12 | 694315 tok/s
iter  10070 | loss 2.8855 | lr 2.89e-03 | grad_norm 0.12 | 690935 tok/s
iter  10080 | loss 2.8833 | lr 2.88e-03 | grad_norm 0.12 | 693017 tok/s
iter  10090 | loss 2.9209 | lr 2.87e-03 | grad_norm 0.12 | 691804 tok/s
iter  10100 | loss 2.8726 | lr 2.86e-03 | grad_norm 0.12 | 692152 tok/s
  eval | train_loss 2.8985 | val_loss 2.9042
Saved checkpoint at iter 10100 -> ckpt.pt
iter  10110 | loss 2.8879 | lr 2.85e-03 | grad_norm 0.12 | 434194 tok/s
iter  10120 | loss 2.9007 | lr 2.84e-03 | grad_norm 0.12 | 698519 tok/s
iter  10130 | loss 2.8745 | lr 2.83e-03 | grad_norm 0.12 | 696683 tok/s
iter  10140 | loss 2.8864 | lr 2.82e-03 | grad_norm 0.11 | 692994 tok/s
iter  10150 | loss 2.8705 | lr 2.81e-03 | grad_norm 0.12 | 696812 tok/s
iter  10160 | loss 2.8839 | lr 2.80e-03 | grad_norm 0.11 | 693158 tok/s
iter  10170 | loss 2.8945 | lr 2.79e-03 | grad_norm 0.12 | 692054 tok/s
iter  10180 | loss 2.9168 | lr 2.78e-03 | grad_norm 0.12 | 689269 tok/s
iter  10190 | loss 2.8880 | lr 2.77e-03 | grad_norm 0.12 | 689551 tok/s
iter  10200 | loss 2.8854 | lr 2.76e-03 | grad_norm 0.11 | 691790 tok/s
  eval | train_loss 2.8873 | val_loss 2.8983
Saved checkpoint at iter 10200 -> ckpt.pt
iter  10210 | loss 2.8872 | lr 2.74e-03 | grad_norm 0.12 | 437237 tok/s
iter  10220 | loss 2.9018 | lr 2.73e-03 | grad_norm 0.12 | 693459 tok/s
iter  10230 | loss 2.8930 | lr 2.72e-03 | grad_norm 0.11 | 692006 tok/s
iter  10240 | loss 2.8894 | lr 2.71e-03 | grad_norm 0.11 | 689760 tok/s
iter  10250 | loss 2.8873 | lr 2.70e-03 | grad_norm 0.12 | 690649 tok/s
iter  10260 | loss 2.9134 | lr 2.69e-03 | grad_norm 0.12 | 694835 tok/s
iter  10270 | loss 2.8947 | lr 2.68e-03 | grad_norm 0.12 | 694155 tok/s
iter  10280 | loss 2.8802 | lr 2.67e-03 | grad_norm 0.12 | 695821 tok/s
iter  10290 | loss 2.8654 | lr 2.66e-03 | grad_norm 0.12 | 694307 tok/s
iter  10300 | loss 2.8566 | lr 2.65e-03 | grad_norm 0.11 | 693030 tok/s
  eval | train_loss 2.8853 | val_loss 2.8862
Saved checkpoint at iter 10300 -> ckpt.pt
Saved checkpoint at iter 10300 -> ckpt_best.pt
  New best val_loss: 2.8862
iter  10310 | loss 2.8806 | lr 2.64e-03 | grad_norm 0.11 | 422752 tok/s
iter  10320 | loss 2.8822 | lr 2.63e-03 | grad_norm 0.11 | 694686 tok/s
iter  10330 | loss 2.8664 | lr 2.62e-03 | grad_norm 0.11 | 695898 tok/s
iter  10340 | loss 2.8983 | lr 2.61e-03 | grad_norm 0.11 | 688363 tok/s
iter  10350 | loss 2.9072 | lr 2.60e-03 | grad_norm 0.12 | 696623 tok/s
iter  10360 | loss 2.8701 | lr 2.59e-03 | grad_norm 0.11 | 692601 tok/s
iter  10370 | loss 2.8944 | lr 2.58e-03 | grad_norm 0.11 | 691733 tok/s
iter  10380 | loss 2.8725 | lr 2.57e-03 | grad_norm 0.12 | 694751 tok/s
iter  10390 | loss 2.8710 | lr 2.56e-03 | grad_norm 0.11 | 696144 tok/s
iter  10400 | loss 2.8875 | lr 2.55e-03 | grad_norm 0.12 | 698735 tok/s
  eval | train_loss 2.8879 | val_loss 2.9049
Saved checkpoint at iter 10400 -> ckpt.pt
iter  10410 | loss 2.8850 | lr 2.54e-03 | grad_norm 0.11 | 438951 tok/s
iter  10420 | loss 2.9117 | lr 2.53e-03 | grad_norm 0.12 | 691500 tok/s
iter  10430 | loss 2.8667 | lr 2.52e-03 | grad_norm 0.11 | 689827 tok/s
iter  10440 | loss 2.8825 | lr 2.51e-03 | grad_norm 0.11 | 690745 tok/s
iter  10450 | loss 2.8841 | lr 2.50e-03 | grad_norm 0.11 | 693295 tok/s
iter  10460 | loss 2.9099 | lr 2.49e-03 | grad_norm 0.12 | 695382 tok/s
iter  10470 | loss 2.8802 | lr 2.48e-03 | grad_norm 0.12 | 695347 tok/s
iter  10480 | loss 2.9088 | lr 2.47e-03 | grad_norm 0.11 | 690968 tok/s
iter  10490 | loss 2.8841 | lr 2.45e-03 | grad_norm 0.12 | 693985 tok/s
iter  10500 | loss 2.8720 | lr 2.44e-03 | grad_norm 0.11 | 693078 tok/s
  eval | train_loss 2.8904 | val_loss 2.8961
Saved checkpoint at iter 10500 -> ckpt.pt
iter  10510 | loss 2.8866 | lr 2.43e-03 | grad_norm 0.11 | 438133 tok/s
iter  10520 | loss 2.8794 | lr 2.42e-03 | grad_norm 0.11 | 696552 tok/s
iter  10530 | loss 2.8972 | lr 2.41e-03 | grad_norm 0.11 | 696124 tok/s
iter  10540 | loss 2.9036 | lr 2.40e-03 | grad_norm 0.11 | 691735 tok/s
iter  10550 | loss 2.9054 | lr 2.39e-03 | grad_norm 0.11 | 692412 tok/s
iter  10560 | loss 2.8649 | lr 2.38e-03 | grad_norm 0.11 | 691374 tok/s
iter  10570 | loss 2.9079 | lr 2.37e-03 | grad_norm 0.11 | 697973 tok/s
iter  10580 | loss 2.9151 | lr 2.36e-03 | grad_norm 0.11 | 695697 tok/s
iter  10590 | loss 2.8872 | lr 2.35e-03 | grad_norm 0.11 | 697770 tok/s
iter  10600 | loss 2.8799 | lr 2.34e-03 | grad_norm 0.11 | 696941 tok/s
  eval | train_loss 2.8870 | val_loss 2.8976
Saved checkpoint at iter 10600 -> ckpt.pt
iter  10610 | loss 2.8687 | lr 2.33e-03 | grad_norm 0.11 | 434854 tok/s
iter  10620 | loss 2.8551 | lr 2.32e-03 | grad_norm 0.11 | 693616 tok/s
iter  10630 | loss 2.8788 | lr 2.31e-03 | grad_norm 0.11 | 691622 tok/s
iter  10640 | loss 2.9056 | lr 2.30e-03 | grad_norm 0.12 | 691105 tok/s
iter  10650 | loss 2.8951 | lr 2.29e-03 | grad_norm 0.11 | 691915 tok/s
iter  10660 | loss 2.8889 | lr 2.28e-03 | grad_norm 0.11 | 691691 tok/s
iter  10670 | loss 2.8725 | lr 2.27e-03 | grad_norm 0.12 | 694287 tok/s
iter  10680 | loss 2.9018 | lr 2.26e-03 | grad_norm 0.11 | 691810 tok/s
iter  10690 | loss 2.9111 | lr 2.25e-03 | grad_norm 0.11 | 694110 tok/s
iter  10700 | loss 2.8901 | lr 2.24e-03 | grad_norm 0.11 | 690021 tok/s
  eval | train_loss 2.8871 | val_loss 2.8923
Saved checkpoint at iter 10700 -> ckpt.pt
iter  10710 | loss 2.8609 | lr 2.23e-03 | grad_norm 0.11 | 432563 tok/s
iter  10720 | loss 2.8842 | lr 2.22e-03 | grad_norm 0.11 | 691204 tok/s
iter  10730 | loss 2.8847 | lr 2.21e-03 | grad_norm 0.11 | 691465 tok/s
iter  10740 | loss 2.8553 | lr 2.20e-03 | grad_norm 0.11 | 689758 tok/s
iter  10750 | loss 2.8771 | lr 2.19e-03 | grad_norm 0.11 | 688354 tok/s
iter  10760 | loss 2.8522 | lr 2.18e-03 | grad_norm 0.11 | 691369 tok/s
iter  10770 | loss 2.8843 | lr 2.17e-03 | grad_norm 0.11 | 693149 tok/s
iter  10780 | loss 2.8629 | lr 2.15e-03 | grad_norm 0.11 | 694035 tok/s
iter  10790 | loss 2.8815 | lr 2.14e-03 | grad_norm 0.11 | 695278 tok/s
iter  10800 | loss 2.8714 | lr 2.13e-03 | grad_norm 0.11 | 694379 tok/s
  eval | train_loss 2.8763 | val_loss 2.8973
Saved checkpoint at iter 10800 -> ckpt.pt
iter  10810 | loss 2.8751 | lr 2.12e-03 | grad_norm 0.11 | 434521 tok/s
iter  10820 | loss 2.8776 | lr 2.11e-03 | grad_norm 0.11 | 693564 tok/s
iter  10830 | loss 2.8705 | lr 2.10e-03 | grad_norm 0.11 | 691147 tok/s
iter  10840 | loss 2.8942 | lr 2.09e-03 | grad_norm 0.11 | 695908 tok/s
iter  10850 | loss 2.8932 | lr 2.08e-03 | grad_norm 0.11 | 696167 tok/s
iter  10860 | loss 2.8815 | lr 2.07e-03 | grad_norm 0.11 | 692204 tok/s
iter  10870 | loss 2.8627 | lr 2.06e-03 | grad_norm 0.11 | 692574 tok/s
iter  10880 | loss 2.8598 | lr 2.05e-03 | grad_norm 0.11 | 690635 tok/s
iter  10890 | loss 2.8899 | lr 2.04e-03 | grad_norm 0.11 | 689191 tok/s
iter  10900 | loss 2.9006 | lr 2.03e-03 | grad_norm 0.11 | 690333 tok/s
  eval | train_loss 2.8776 | val_loss 2.8864
Saved checkpoint at iter 10900 -> ckpt.pt
iter  10910 | loss 2.8737 | lr 2.02e-03 | grad_norm 0.11 | 433775 tok/s
iter  10920 | loss 2.8837 | lr 2.01e-03 | grad_norm 0.11 | 694559 tok/s
iter  10930 | loss 2.8904 | lr 2.00e-03 | grad_norm 0.11 | 692552 tok/s
iter  10940 | loss 2.8864 | lr 1.99e-03 | grad_norm 0.11 | 692386 tok/s
iter  10950 | loss 2.8919 | lr 1.98e-03 | grad_norm 0.11 | 690912 tok/s
iter  10960 | loss 2.8830 | lr 1.97e-03 | grad_norm 0.11 | 690134 tok/s
iter  10970 | loss 2.9099 | lr 1.96e-03 | grad_norm 0.11 | 690903 tok/s
iter  10980 | loss 2.8790 | lr 1.95e-03 | grad_norm 0.11 | 692975 tok/s
iter  10990 | loss 2.8984 | lr 1.94e-03 | grad_norm 0.11 | 693987 tok/s
iter  11000 | loss 2.8914 | lr 1.93e-03 | grad_norm 0.11 | 693423 tok/s
  eval | train_loss 2.8858 | val_loss 2.8916
Saved checkpoint at iter 11000 -> ckpt.pt
iter  11010 | loss 2.8795 | lr 1.92e-03 | grad_norm 0.11 | 438711 tok/s
iter  11020 | loss 2.8732 | lr 1.91e-03 | grad_norm 0.11 | 695028 tok/s
iter  11030 | loss 2.8785 | lr 1.90e-03 | grad_norm 0.11 | 695437 tok/s
iter  11040 | loss 2.8664 | lr 1.89e-03 | grad_norm 0.11 | 695219 tok/s
iter  11050 | loss 2.8888 | lr 1.88e-03 | grad_norm 0.11 | 692752 tok/s
iter  11060 | loss 2.8671 | lr 1.86e-03 | grad_norm 0.11 | 690417 tok/s
iter  11070 | loss 2.9029 | lr 1.85e-03 | grad_norm 0.11 | 693136 tok/s
iter  11080 | loss 2.8978 | lr 1.84e-03 | grad_norm 0.11 | 691543 tok/s
iter  11090 | loss 2.8708 | lr 1.83e-03 | grad_norm 0.11 | 692648 tok/s
iter  11100 | loss 2.8796 | lr 1.82e-03 | grad_norm 0.11 | 692931 tok/s
  eval | train_loss 2.8733 | val_loss 2.8892
Saved checkpoint at iter 11100 -> ckpt.pt
iter  11110 | loss 2.8714 | lr 1.81e-03 | grad_norm 0.11 | 435481 tok/s
iter  11120 | loss 2.8688 | lr 1.80e-03 | grad_norm 0.11 | 692538 tok/s
iter  11130 | loss 2.9075 | lr 1.79e-03 | grad_norm 0.11 | 687930 tok/s
iter  11140 | loss 2.8993 | lr 1.78e-03 | grad_norm 0.11 | 685565 tok/s
iter  11150 | loss 2.8721 | lr 1.77e-03 | grad_norm 0.11 | 690075 tok/s
iter  11160 | loss 2.8653 | lr 1.76e-03 | grad_norm 0.11 | 691059 tok/s
iter  11170 | loss 2.8727 | lr 1.75e-03 | grad_norm 0.11 | 693642 tok/s
iter  11180 | loss 2.8699 | lr 1.74e-03 | grad_norm 0.11 | 694222 tok/s
iter  11190 | loss 2.8644 | lr 1.73e-03 | grad_norm 0.11 | 691051 tok/s
iter  11200 | loss 2.8839 | lr 1.72e-03 | grad_norm 0.11 | 691739 tok/s
  eval | train_loss 2.8779 | val_loss 2.8777
Saved checkpoint at iter 11200 -> ckpt.pt
Saved checkpoint at iter 11200 -> ckpt_best.pt
  New best val_loss: 2.8777
iter  11210 | loss 2.8764 | lr 1.71e-03 | grad_norm 0.11 | 427946 tok/s
iter  11220 | loss 2.8874 | lr 1.70e-03 | grad_norm 0.11 | 692539 tok/s
iter  11230 | loss 2.8598 | lr 1.69e-03 | grad_norm 0.11 | 691965 tok/s
iter  11240 | loss 2.8569 | lr 1.68e-03 | grad_norm 0.11 | 697862 tok/s
iter  11250 | loss 2.8585 | lr 1.67e-03 | grad_norm 0.10 | 694939 tok/s
iter  11260 | loss 2.8866 | lr 1.66e-03 | grad_norm 0.11 | 696622 tok/s
iter  11270 | loss 2.8692 | lr 1.65e-03 | grad_norm 0.11 | 696171 tok/s
iter  11280 | loss 2.8815 | lr 1.64e-03 | grad_norm 0.10 | 695199 tok/s
iter  11290 | loss 2.8737 | lr 1.63e-03 | grad_norm 0.10 | 692789 tok/s
iter  11300 | loss 2.8783 | lr 1.62e-03 | grad_norm 0.10 | 691621 tok/s
  eval | train_loss 2.8758 | val_loss 2.8826
Saved checkpoint at iter 11300 -> ckpt.pt
iter  11310 | loss 2.8683 | lr 1.61e-03 | grad_norm 0.11 | 438180 tok/s
iter  11320 | loss 2.8978 | lr 1.60e-03 | grad_norm 0.11 | 693804 tok/s
iter  11330 | loss 2.8863 | lr 1.59e-03 | grad_norm 0.11 | 693798 tok/s
iter  11340 | loss 2.9082 | lr 1.57e-03 | grad_norm 0.11 | 693736 tok/s
iter  11350 | loss 2.8931 | lr 1.56e-03 | grad_norm 0.11 | 691934 tok/s
iter  11360 | loss 2.8923 | lr 1.55e-03 | grad_norm 0.10 | 693447 tok/s
iter  11370 | loss 2.8562 | lr 1.54e-03 | grad_norm 0.11 | 692000 tok/s
iter  11380 | loss 2.8575 | lr 1.53e-03 | grad_norm 0.10 | 693146 tok/s
iter  11390 | loss 2.8585 | lr 1.52e-03 | grad_norm 0.10 | 692097 tok/s
iter  11400 | loss 2.8741 | lr 1.51e-03 | grad_norm 0.10 | 693000 tok/s
  eval | train_loss 2.8750 | val_loss 2.8793
Saved checkpoint at iter 11400 -> ckpt.pt
iter  11410 | loss 2.8675 | lr 1.50e-03 | grad_norm 0.10 | 437406 tok/s
iter  11420 | loss 2.8824 | lr 1.49e-03 | grad_norm 0.11 | 691361 tok/s
iter  11430 | loss 2.8603 | lr 1.48e-03 | grad_norm 0.10 | 688436 tok/s
iter  11440 | loss 2.8673 | lr 1.47e-03 | grad_norm 0.11 | 689410 tok/s
iter  11450 | loss 2.8644 | lr 1.46e-03 | grad_norm 0.10 | 689730 tok/s
iter  11460 | loss 2.8735 | lr 1.45e-03 | grad_norm 0.10 | 689362 tok/s
iter  11470 | loss 2.8566 | lr 1.44e-03 | grad_norm 0.11 | 691796 tok/s
iter  11480 | loss 2.8616 | lr 1.43e-03 | grad_norm 0.11 | 688923 tok/s
iter  11490 | loss 2.8647 | lr 1.42e-03 | grad_norm 0.10 | 691346 tok/s
iter  11500 | loss 2.8723 | lr 1.41e-03 | grad_norm 0.11 | 696823 tok/s
  eval | train_loss 2.8638 | val_loss 2.8808
Saved checkpoint at iter 11500 -> ckpt.pt
iter  11510 | loss 2.8768 | lr 1.40e-03 | grad_norm 0.11 | 437837 tok/s
iter  11520 | loss 2.8954 | lr 1.39e-03 | grad_norm 0.11 | 692769 tok/s
iter  11530 | loss 2.8738 | lr 1.38e-03 | grad_norm 0.10 | 692826 tok/s
iter  11540 | loss 2.8602 | lr 1.37e-03 | grad_norm 0.10 | 690918 tok/s
iter  11550 | loss 2.8735 | lr 1.36e-03 | grad_norm 0.10 | 688162 tok/s
iter  11560 | loss 2.8497 | lr 1.35e-03 | grad_norm 0.11 | 691159 tok/s
iter  11570 | loss 2.8762 | lr 1.34e-03 | grad_norm 0.10 | 688568 tok/s
iter  11580 | loss 2.8809 | lr 1.33e-03 | grad_norm 0.10 | 692206 tok/s
iter  11590 | loss 2.8628 | lr 1.32e-03 | grad_norm 0.10 | 696003 tok/s
iter  11600 | loss 2.8792 | lr 1.31e-03 | grad_norm 0.10 | 693335 tok/s
  eval | train_loss 2.8749 | val_loss 2.8865
Saved checkpoint at iter 11600 -> ckpt.pt
iter  11610 | loss 2.8991 | lr 1.30e-03 | grad_norm 0.10 | 438445 tok/s
iter  11620 | loss 2.8677 | lr 1.28e-03 | grad_norm 0.10 | 697229 tok/s
iter  11630 | loss 2.8517 | lr 1.27e-03 | grad_norm 0.11 | 692211 tok/s
iter  11640 | loss 2.8762 | lr 1.26e-03 | grad_norm 0.10 | 698206 tok/s
iter  11650 | loss 2.8774 | lr 1.25e-03 | grad_norm 0.10 | 696945 tok/s
iter  11660 | loss 2.8821 | lr 1.24e-03 | grad_norm 0.10 | 695895 tok/s
iter  11670 | loss 2.8385 | lr 1.23e-03 | grad_norm 0.10 | 692718 tok/s
iter  11680 | loss 2.8465 | lr 1.22e-03 | grad_norm 0.10 | 691812 tok/s
iter  11690 | loss 2.8861 | lr 1.21e-03 | grad_norm 0.11 | 691343 tok/s
iter  11700 | loss 2.8807 | lr 1.20e-03 | grad_norm 0.10 | 689185 tok/s
  eval | train_loss 2.8696 | val_loss 2.8766
Saved checkpoint at iter 11700 -> ckpt.pt
Saved checkpoint at iter 11700 -> ckpt_best.pt
  New best val_loss: 2.8766
iter  11710 | loss 2.8695 | lr 1.19e-03 | grad_norm 0.10 | 426322 tok/s
iter  11720 | loss 2.8513 | lr 1.18e-03 | grad_norm 0.10 | 698013 tok/s
iter  11730 | loss 2.8429 | lr 1.17e-03 | grad_norm 0.10 | 693263 tok/s
iter  11740 | loss 2.8711 | lr 1.16e-03 | grad_norm 0.10 | 691719 tok/s
iter  11750 | loss 2.8711 | lr 1.15e-03 | grad_norm 0.10 | 695652 tok/s
iter  11760 | loss 2.8583 | lr 1.14e-03 | grad_norm 0.10 | 693384 tok/s
iter  11770 | loss 2.8632 | lr 1.13e-03 | grad_norm 0.10 | 692064 tok/s
iter  11780 | loss 2.8677 | lr 1.12e-03 | grad_norm 0.10 | 695592 tok/s
iter  11790 | loss 2.8703 | lr 1.11e-03 | grad_norm 0.10 | 693221 tok/s
iter  11800 | loss 2.8522 | lr 1.10e-03 | grad_norm 0.10 | 690662 tok/s
  eval | train_loss 2.8615 | val_loss 2.8746
Saved checkpoint at iter 11800 -> ckpt.pt
Saved checkpoint at iter 11800 -> ckpt_best.pt
  New best val_loss: 2.8746
iter  11810 | loss 2.8656 | lr 1.09e-03 | grad_norm 0.11 | 429337 tok/s
iter  11820 | loss 2.8564 | lr 1.08e-03 | grad_norm 0.10 | 694881 tok/s
iter  11830 | loss 2.8604 | lr 1.07e-03 | grad_norm 0.10 | 693146 tok/s
iter  11840 | loss 2.8614 | lr 1.06e-03 | grad_norm 0.10 | 691967 tok/s
iter  11850 | loss 2.8641 | lr 1.05e-03 | grad_norm 0.10 | 692448 tok/s
iter  11860 | loss 2.8572 | lr 1.04e-03 | grad_norm 0.10 | 695587 tok/s
iter  11870 | loss 2.8873 | lr 1.03e-03 | grad_norm 0.10 | 693045 tok/s
iter  11880 | loss 2.8464 | lr 1.02e-03 | grad_norm 0.10 | 691055 tok/s
iter  11890 | loss 2.8903 | lr 1.01e-03 | grad_norm 0.10 | 692630 tok/s
iter  11900 | loss 2.8507 | lr 9.95e-04 | grad_norm 0.11 | 694850 tok/s
  eval | train_loss 2.8634 | val_loss 2.8834
Saved checkpoint at iter 11900 -> ckpt.pt
iter  11910 | loss 2.8769 | lr 9.85e-04 | grad_norm 0.10 | 433941 tok/s
iter  11920 | loss 2.8897 | lr 9.74e-04 | grad_norm 0.11 | 697939 tok/s
iter  11930 | loss 2.8770 | lr 9.64e-04 | grad_norm 0.10 | 690726 tok/s
iter  11940 | loss 2.8684 | lr 9.54e-04 | grad_norm 0.10 | 694466 tok/s
iter  11950 | loss 2.8710 | lr 9.43e-04 | grad_norm 0.10 | 692226 tok/s
iter  11960 | loss 2.8707 | lr 9.33e-04 | grad_norm 0.10 | 696034 tok/s
iter  11970 | loss 2.8578 | lr 9.22e-04 | grad_norm 0.10 | 693845 tok/s
iter  11980 | loss 2.8781 | lr 9.12e-04 | grad_norm 0.10 | 696249 tok/s
iter  11990 | loss 2.8501 | lr 9.02e-04 | grad_norm 0.10 | 695934 tok/s
iter  12000 | loss 2.8514 | lr 8.91e-04 | grad_norm 0.10 | 695576 tok/s
  eval | train_loss 2.8724 | val_loss 2.8661
Saved checkpoint at iter 12000 -> ckpt.pt
Saved checkpoint at iter 12000 -> ckpt_best.pt
  New best val_loss: 2.8661
iter  12010 | loss 2.8935 | lr 8.81e-04 | grad_norm 0.10 | 421764 tok/s
iter  12020 | loss 2.8635 | lr 8.71e-04 | grad_norm 0.10 | 697891 tok/s
iter  12030 | loss 2.8682 | lr 8.60e-04 | grad_norm 0.10 | 695629 tok/s
iter  12040 | loss 2.8810 | lr 8.50e-04 | grad_norm 0.10 | 691601 tok/s
iter  12050 | loss 2.8788 | lr 8.40e-04 | grad_norm 0.10 | 692831 tok/s
iter  12060 | loss 2.8760 | lr 8.29e-04 | grad_norm 0.10 | 697635 tok/s
iter  12070 | loss 2.8524 | lr 8.19e-04 | grad_norm 0.10 | 691498 tok/s
iter  12080 | loss 2.8420 | lr 8.09e-04 | grad_norm 0.10 | 691390 tok/s
iter  12090 | loss 2.8724 | lr 7.98e-04 | grad_norm 0.10 | 693327 tok/s
iter  12100 | loss 2.8856 | lr 7.88e-04 | grad_norm 0.10 | 694248 tok/s
  eval | train_loss 2.8605 | val_loss 2.8692
Saved checkpoint at iter 12100 -> ckpt.pt
iter  12110 | loss 2.8657 | lr 7.77e-04 | grad_norm 0.10 | 438857 tok/s
iter  12120 | loss 2.8781 | lr 7.67e-04 | grad_norm 0.10 | 697716 tok/s
iter  12130 | loss 2.8344 | lr 7.57e-04 | grad_norm 0.10 | 696555 tok/s
iter  12140 | loss 2.8473 | lr 7.46e-04 | grad_norm 0.10 | 693059 tok/s
iter  12150 | loss 2.8570 | lr 7.36e-04 | grad_norm 0.10 | 693123 tok/s
iter  12160 | loss 2.8694 | lr 7.26e-04 | grad_norm 0.10 | 692176 tok/s
iter  12170 | loss 2.8613 | lr 7.15e-04 | grad_norm 0.10 | 695111 tok/s
iter  12180 | loss 2.8631 | lr 7.05e-04 | grad_norm 0.10 | 693527 tok/s
iter  12190 | loss 2.8528 | lr 6.95e-04 | grad_norm 0.10 | 692814 tok/s
iter  12200 | loss 2.8422 | lr 6.84e-04 | grad_norm 0.10 | 693554 tok/s
  eval | train_loss 2.8711 | val_loss 2.8586
Saved checkpoint at iter 12200 -> ckpt.pt
Saved checkpoint at iter 12200 -> ckpt_best.pt
  New best val_loss: 2.8586
iter  12210 | loss 2.8609 | lr 6.74e-04 | grad_norm 0.10 | 424075 tok/s
iter  12220 | loss 2.8700 | lr 6.64e-04 | grad_norm 0.10 | 698122 tok/s
iter  12230 | loss 2.8553 | lr 6.53e-04 | grad_norm 0.09 | 695966 tok/s
iter  12240 | loss 2.8534 | lr 6.43e-04 | grad_norm 0.10 | 695836 tok/s
iter  12250 | loss 2.8828 | lr 6.33e-04 | grad_norm 0.10 | 694638 tok/s
iter  12260 | loss 2.8660 | lr 6.22e-04 | grad_norm 0.10 | 695869 tok/s
iter  12270 | loss 2.8894 | lr 6.12e-04 | grad_norm 0.10 | 692920 tok/s
iter  12280 | loss 2.8706 | lr 6.01e-04 | grad_norm 0.10 | 693271 tok/s
iter  12290 | loss 2.8858 | lr 5.91e-04 | grad_norm 0.10 | 692445 tok/s
iter  12300 | loss 2.8466 | lr 5.81e-04 | grad_norm 0.10 | 694910 tok/s
  eval | train_loss 2.8609 | val_loss 2.8603
Saved checkpoint at iter 12300 -> ckpt.pt
iter  12310 | loss 2.8333 | lr 5.70e-04 | grad_norm 0.10 | 437883 tok/s
iter  12320 | loss 2.8524 | lr 5.60e-04 | grad_norm 0.10 | 693058 tok/s
iter  12330 | loss 2.8797 | lr 5.50e-04 | grad_norm 0.10 | 694335 tok/s
iter  12340 | loss 2.8570 | lr 5.39e-04 | grad_norm 0.10 | 692940 tok/s
iter  12350 | loss 2.8907 | lr 5.29e-04 | grad_norm 0.10 | 694500 tok/s
iter  12360 | loss 2.8552 | lr 5.19e-04 | grad_norm 0.10 | 694674 tok/s
iter  12370 | loss 2.8847 | lr 5.08e-04 | grad_norm 0.10 | 696213 tok/s
iter  12380 | loss 2.8446 | lr 4.98e-04 | grad_norm 0.10 | 694732 tok/s
iter  12390 | loss 2.8599 | lr 4.88e-04 | grad_norm 0.10 | 694079 tok/s
iter  12400 | loss 2.8447 | lr 4.77e-04 | grad_norm 0.10 | 692350 tok/s
  eval | train_loss 2.8611 | val_loss 2.8676
Saved checkpoint at iter 12400 -> ckpt.pt
iter  12410 | loss 2.8709 | lr 4.67e-04 | grad_norm 0.10 | 429833 tok/s
iter  12420 | loss 2.8530 | lr 4.57e-04 | grad_norm 0.10 | 695276 tok/s
iter  12430 | loss 2.8843 | lr 4.46e-04 | grad_norm 0.10 | 694811 tok/s
iter  12440 | loss 2.8611 | lr 4.36e-04 | grad_norm 0.10 | 694537 tok/s
iter  12450 | loss 2.8970 | lr 4.25e-04 | grad_norm 0.10 | 694259 tok/s
iter  12460 | loss 2.8595 | lr 4.15e-04 | grad_norm 0.10 | 693405 tok/s
iter  12470 | loss 2.8546 | lr 4.05e-04 | grad_norm 0.10 | 692893 tok/s
iter  12480 | loss 2.8621 | lr 3.94e-04 | grad_norm 0.10 | 694772 tok/s
iter  12490 | loss 2.8575 | lr 3.84e-04 | grad_norm 0.10 | 693282 tok/s
iter  12500 | loss 2.8581 | lr 3.74e-04 | grad_norm 0.09 | 692389 tok/s
  eval | train_loss 2.8720 | val_loss 2.8665
Saved checkpoint at iter 12500 -> ckpt.pt
  quick_eval: import failed (No module named 'evals'); skipping
iter  12510 | loss 2.8576 | lr 3.63e-04 | grad_norm 0.10 | 429518 tok/s
iter  12520 | loss 2.8605 | lr 3.53e-04 | grad_norm 0.10 | 693588 tok/s
iter  12530 | loss 2.8716 | lr 3.43e-04 | grad_norm 0.10 | 691884 tok/s
iter  12540 | loss 2.8815 | lr 3.32e-04 | grad_norm 0.10 | 690263 tok/s
iter  12550 | loss 2.8777 | lr 3.22e-04 | grad_norm 0.10 | 694063 tok/s
iter  12560 | loss 2.8454 | lr 3.12e-04 | grad_norm 0.09 | 695611 tok/s
iter  12570 | loss 2.8834 | lr 3.01e-04 | grad_norm 0.10 | 692130 tok/s
iter  12580 | loss 2.8521 | lr 2.91e-04 | grad_norm 0.10 | 694220 tok/s
iter  12590 | loss 2.8693 | lr 2.80e-04 | grad_norm 0.10 | 696463 tok/s
iter  12600 | loss 2.8450 | lr 2.70e-04 | grad_norm 0.10 | 692260 tok/s
  eval | train_loss 2.8520 | val_loss 2.8652
Saved checkpoint at iter 12600 -> ckpt.pt
iter  12610 | loss 2.8704 | lr 2.60e-04 | grad_norm 0.10 | 438784 tok/s
iter  12620 | loss 2.8682 | lr 2.49e-04 | grad_norm 0.10 | 695263 tok/s
iter  12630 | loss 2.8734 | lr 2.39e-04 | grad_norm 0.10 | 695652 tok/s
iter  12640 | loss 2.8822 | lr 2.29e-04 | grad_norm 0.10 | 695818 tok/s
iter  12650 | loss 2.8612 | lr 2.18e-04 | grad_norm 0.10 | 693154 tok/s
iter  12660 | loss 2.8778 | lr 2.08e-04 | grad_norm 0.09 | 693661 tok/s
iter  12670 | loss 2.8533 | lr 1.98e-04 | grad_norm 0.10 | 691481 tok/s
iter  12680 | loss 2.8480 | lr 1.87e-04 | grad_norm 0.10 | 695048 tok/s
iter  12690 | loss 2.8506 | lr 1.77e-04 | grad_norm 0.09 | 691506 tok/s
iter  12700 | loss 2.8530 | lr 1.67e-04 | grad_norm 0.10 | 696089 tok/s
  eval | train_loss 2.8613 | val_loss 2.8641
Saved checkpoint at iter 12700 -> ckpt.pt
iter  12710 | loss 2.8721 | lr 1.56e-04 | grad_norm 0.09 | 431999 tok/s
  eval | train_loss 2.8748 | val_loss 2.8664
Saved checkpoint at iter 12716 -> ckpt.pt
  quick_eval: import failed (No module named 'evals'); skipping
Saved checkpoint at iter 12716 -> ckpt.pt

Training complete! Best val_loss: 2.8586
VRAM peak (rank 0): allocated 23.13 GB, reserved 25.39 GB
wandb: updating run metadata
wandb: uploading wandb-summary.json; uploading config.yaml; uploading output.log
wandb: uploading wandb-summary.json; uploading output.log
wandb: uploading output.log
wandb: uploading history steps 1271-1272, summary, console lines 1671-1677
wandb: 
wandb: Run history:
wandb:      eval/train_loss █▄▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:        eval/val_loss █▇▅▅▄▄▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▂▂▁▁▁▁▁▁▁▁▁▁▁
wandb:      train/grad_norm █▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:           train/loss █▆▆▆▅▄▄▃▄▃▃▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁
wandb:             train/lr █████████▇▇▆▆▆▅▅▅▅▅▄▄▄▄▃▃▃▃▃▃▃▂▂▂▂▂▂▁▁▁▁
wandb: train/tokens_per_sec ██▁██▁█████▁████████████████████████████
wandb: 
wandb: Run summary:
wandb:      eval/train_loss 2.87479
wandb:        eval/val_loss 2.86635
wandb:      train/grad_norm 0.09277
wandb:           train/loss 2.87208
wandb:             train/lr 0.00016
wandb: train/tokens_per_sec 431999.36436
wandb: 
wandb: 🚀 View run vanilla-small-20B at: https://wandb.ai/markhenrysoftware/sparse-nanogpt/runs/9iezcto0
wandb: ⭐️ View project at: https://wandb.ai/markhenrysoftware/sparse-nanogpt
wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20260724_122939-9iezcto0/logs