Blame for run/train.log

DOUCZ-SMOKE-OK 2026-10-23T13:55:52Z BATCH-SHA krok biegu 1051 = krok strumienia 1051: ee67ca8d5d4ca05831b84e5852c88c6f534f68c1d4c2f2d57305d1ee5aaabc5a BATCH-SHA krok biegu 1800 = krok strumienia 1800: f8efcf575e2700045a50a401ff3ea9a99636196ec1595030a93c34f71b66c801 BATCH-SHA krok biegu 1801 = krok strumienia 1801: f4e091e8441771d06191822b63d2ae18186b27d93829445d938ddb05226ff176 
DOUCZ start 2026-10-23T13:55:52Z z /root/pilot1b/start/ckpt_1050.pt sha 236108e8f204995f4adcfea7d9a99437d230c1af12708c64c68c74afbc2910fd steps 2860 switch 1801
=== doucz 2026-10-23T13:55:52Z steps 2860 ckpt_every 150
[13:55:57] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 5090 world=8
[13:55:57] dane: train=1,889,519,627B block=4096 batch=256 tok/step=1,048,576
[13:56:10] model GPT-ref: 1073.9M param (L20 d2048 h16)
[13:56:10] optimizer=muon muon_lr=0.02 (mult=33.3x)
[13:56:10] r11 pamiec: act_ckpt=True loss_chunk=1024
[13:56:10] r13: mixer_layout=AAAAAAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
[13:56:13] torch.compile enabled
[13:56:21] RESUME @ 1050
[13:56:21] START gpt-ref: steps=2860 (od 1050) lr=0.0006->6e-05
[13:56:21] sampler epoch: 461,307 okien/epoke (offset0 2245), start od kroku 1050 = epoka 0.583
[13:56:21] data-dir-2: /root/pilot1b/data_p01 1,880,520,445 tok, 459,110 okien/epoke = 1793 krokow, od kroku strumienia 1801
[13:59:53] step 1060/2860 loss 2.8356 lr 6.00e-04 gnorm 0.10 49,515 tok/s peakVRAM 27.0GB
[14:01:29] step 1070/2860 loss 2.8687 lr 6.00e-04 gnorm 0.10 109,240 tok/s peakVRAM 27.0GB
[14:03:05] step 1080/2860 loss 2.8498 lr 6.00e-04 gnorm 0.10 108,731 tok/s peakVRAM 27.0GB
[14:04:42] step 1090/2860 loss 2.8553 lr 6.00e-04 gnorm 0.11 108,248 tok/s peakVRAM 27.0GB
[14:06:19] step 1100/2860 loss 2.8493 lr 6.00e-04 gnorm 0.09 107,871 tok/s peakVRAM 27.0GB
[14:07:57] step 1110/2860 loss 2.8485 lr 6.00e-04 gnorm 0.09 107,630 tok/s peakVRAM 27.0GB
[14:09:34] step 1120/2860 loss 2.8432 lr 6.00e-04 gnorm 0.09 108,072 tok/s peakVRAM 27.0GB
[14:11:11] step 1130/2860 loss 2.8414 lr 6.00e-04 gnorm 0.09 107,987 tok/s peakVRAM 27.0GB
[14:12:48] step 1140/2860 loss 2.8192 lr 6.00e-04 gnorm 0.10 107,987 tok/s peakVRAM 27.0GB
[14:14:25] step 1150/2860 loss 2.8366 lr 6.00e-04 gnorm 0.11 107,797 tok/s peakVRAM 27.0GB
[14:16:02] step 1160/2860 loss 2.8427 lr 6.00e-04 gnorm 0.11 108,156 tok/s peakVRAM 27.0GB
[14:17:39] step 1170/2860 loss 2.8140 lr 6.00e-04 gnorm 0.10 108,311 tok/s peakVRAM 27.0GB
[14:19:16] step 1180/2860 loss 2.8157 lr 6.00e-04 gnorm 0.10 108,201 tok/s peakVRAM 27.0GB
[14:20:53] step 1190/2860 loss 2.8080 lr 6.00e-04 gnorm 0.11 107,802 tok/s peakVRAM 27.0GB
[14:22:31] step 1200/2860 loss 2.8378 lr 6.00e-04 gnorm 0.11 107,815 tok/s peakVRAM 27.0GB
[14:22:40] ckpt @ 1200
[14:24:17] step 1210/2860 loss 2.8056 lr 6.00e-04 gnorm 0.10 98,763 tok/s peakVRAM 27.0GB
[14:25:54] step 1220/2860 loss 2.8146 lr 6.00e-04 gnorm 0.10 108,307 tok/s peakVRAM 27.0GB
[14:27:31] step 1230/2860 loss 2.8082 lr 6.00e-04 gnorm 0.09 108,132 tok/s peakVRAM 27.0GB
[14:29:08] step 1240/2860 loss 2.8096 lr 6.00e-04 gnorm 0.10 107,957 tok/s peakVRAM 27.0GB
[14:30:45] step 1250/2860 loss 2.8166 lr 6.00e-04 gnorm 0.11 108,114 tok/s peakVRAM 27.0GB
[14:32:22] step 1260/2860 loss 2.8128 lr 6.00e-04 gnorm 0.10 108,061 tok/s peakVRAM 27.0GB
[14:33:59] step 1270/2860 loss 2.7992 lr 6.00e-04 gnorm 0.09 108,210 tok/s peakVRAM 27.0GB
[14:35:36] step 1280/2860 loss 2.7989 lr 6.00e-04 gnorm 0.10 108,128 tok/s peakVRAM 27.0GB
[14:37:13] step 1290/2860 loss 2.7926 lr 6.00e-04 gnorm 0.12 107,744 tok/s peakVRAM 27.0GB
[14:38:50] step 1300/2860 loss 2.7575 lr 6.00e-04 gnorm 0.10 107,733 tok/s peakVRAM 27.0GB
[14:40:28] step 1310/2860 loss 2.7676 lr 6.00e-04 gnorm 0.09 107,802 tok/s peakVRAM 27.0GB
[14:42:04] step 1320/2860 loss 2.7761 lr 6.00e-04 gnorm 0.09 108,185 tok/s peakVRAM 27.0GB
[14:43:42] step 1330/2860 loss 2.7931 lr 6.00e-04 gnorm 0.09 107,999 tok/s peakVRAM 27.0GB
[14:45:19] step 1340/2860 loss 2.7885 lr 6.00e-04 gnorm 0.09 107,917 tok/s peakVRAM 27.0GB
[14:46:56] step 1350/2860 loss 2.7733 lr 6.00e-04 gnorm 0.08 108,010 tok/s peakVRAM 27.0GB
[14:47:05] ckpt @ 1350
[14:48:42] step 1360/2860 loss 2.7871 lr 6.00e-04 gnorm 0.10 98,694 tok/s peakVRAM 27.0GB
[14:50:19] step 1370/2860 loss 2.7689 lr 6.00e-04 gnorm 0.09 108,306 tok/s peakVRAM 27.0GB
[14:51:56] step 1380/2860 loss 2.7655 lr 6.00e-04 gnorm 0.09 107,983 tok/s peakVRAM 27.0GB
[14:53:33] step 1390/2860 loss 2.7674 lr 6.00e-04 gnorm 0.10 107,907 tok/s peakVRAM 27.0GB
[14:55:10] step 1400/2860 loss 2.7652 lr 6.00e-04 gnorm 0.09 107,833 tok/s peakVRAM 27.0GB
[14:56:48] step 1410/2860 loss 2.7603 lr 6.00e-04 gnorm 0.09 107,875 tok/s peakVRAM 27.0GB
[14:58:25] step 1420/2860 loss 2.7825 lr 6.00e-04 gnorm 0.10 107,990 tok/s peakVRAM 27.0GB
[15:00:02] step 1430/2860 loss 2.7738 lr 6.00e-04 gnorm 0.08 108,140 tok/s peakVRAM 27.0GB
[15:01:39] step 1440/2860 loss 2.7555 lr 6.00e-04 gnorm 0.09 108,163 tok/s peakVRAM 27.0GB
[15:03:15] step 1450/2860 loss 2.7610 lr 6.00e-04 gnorm 0.09 108,188 tok/s peakVRAM 27.0GB
[15:04:52] step 1460/2860 loss 2.7413 lr 6.00e-04 gnorm 0.09 108,135 tok/s peakVRAM 27.0GB
[15:06:29] step 1470/2860 loss 2.7548 lr 6.00e-04 gnorm 0.09 108,067 tok/s peakVRAM 27.0GB
[15:08:06] step 1480/2860 loss 2.7579 lr 6.00e-04 gnorm 0.10 108,194 tok/s peakVRAM 27.0GB
[15:09:44] step 1490/2860 loss 2.7474 lr 6.00e-04 gnorm 0.10 107,963 tok/s peakVRAM 27.0GB
[15:11:21] step 1500/2860 loss 2.7388 lr 6.00e-04 gnorm 0.10 107,926 tok/s peakVRAM 27.0GB
[15:11:30] ckpt @ 1500
[15:13:07] step 1510/2860 loss 2.7643 lr 6.00e-04 gnorm 0.10 98,671 tok/s peakVRAM 27.0GB
[15:14:44] step 1520/2860 loss 2.7517 lr 6.00e-04 gnorm 0.09 108,297 tok/s peakVRAM 27.0GB
[15:16:21] step 1530/2860 loss 2.7416 lr 6.00e-04 gnorm 0.08 108,167 tok/s peakVRAM 27.0GB
[15:17:58] step 1540/2860 loss 2.7395 lr 6.00e-04 gnorm 0.10 108,347 tok/s peakVRAM 27.0GB
[15:19:34] step 1550/2860 loss 2.7480 lr 6.00e-04 gnorm 0.09 108,358 tok/s peakVRAM 27.0GB
[15:21:11] step 1560/2860 loss 2.7244 lr 6.00e-04 gnorm 0.08 108,232 tok/s peakVRAM 27.0GB
[15:22:48] step 1570/2860 loss 2.7390 lr 6.00e-04 gnorm 0.09 108,028 tok/s peakVRAM 27.0GB
[15:24:25] step 1580/2860 loss 2.7299 lr 6.00e-04 gnorm 0.08 108,056 tok/s peakVRAM 27.0GB
[15:26:02] step 1590/2860 loss 2.7523 lr 6.00e-04 gnorm 0.10 108,021 tok/s peakVRAM 27.0GB
[15:27:39] step 1600/2860 loss 2.7405 lr 6.00e-04 gnorm 0.08 108,154 tok/s peakVRAM 27.0GB
[15:29:16] step 1610/2860 loss 2.7168 lr 6.00e-04 gnorm 0.08 107,893 tok/s peakVRAM 27.0GB
[15:30:54] step 1620/2860 loss 2.7081 lr 6.00e-04 gnorm 0.09 107,857 tok/s peakVRAM 27.0GB
[15:32:31] step 1630/2860 loss 2.7200 lr 6.00e-04 gnorm 0.09 107,659 tok/s peakVRAM 27.0GB
[15:34:08] step 1640/2860 loss 2.7136 lr 6.00e-04 gnorm 0.16 107,823 tok/s peakVRAM 27.0GB
[15:35:46] step 1650/2860 loss 2.7406 lr 6.00e-04 gnorm 0.09 107,779 tok/s peakVRAM 27.0GB
[15:35:56] ckpt @ 1650
[15:37:33] step 1660/2860 loss 2.7050 lr 6.00e-04 gnorm 0.09 98,087 tok/s peakVRAM 27.0GB
[15:39:10] step 1670/2860 loss 2.7108 lr 6.00e-04 gnorm 0.10 108,102 tok/s peakVRAM 27.0GB
[15:40:47] step 1680/2860 loss 2.7259 lr 6.00e-04 gnorm 0.10 107,897 tok/s peakVRAM 27.0GB
[15:42:24] step 1690/2860 loss 2.7016 lr 6.00e-04 gnorm 0.10 107,780 tok/s peakVRAM 27.0GB
[15:44:01] step 1700/2860 loss 2.7218 lr 6.00e-04 gnorm 0.09 107,921 tok/s peakVRAM 27.0GB
[15:45:38] step 1710/2860 loss 2.7047 lr 6.00e-04 gnorm 0.09 107,869 tok/s peakVRAM 27.0GB
[15:47:16] step 1720/2860 loss 2.7218 lr 6.00e-04 gnorm 0.10 107,872 tok/s peakVRAM 27.0GB
[15:48:53] step 1730/2860 loss 2.7104 lr 6.00e-04 gnorm 0.10 107,897 tok/s peakVRAM 27.0GB
[15:50:30] step 1740/2860 loss 2.6869 lr 6.00e-04 gnorm 0.08 107,874 tok/s peakVRAM 27.0GB
[15:52:07] step 1750/2860 loss 2.6925 lr 6.00e-04 gnorm 0.10 107,969 tok/s peakVRAM 27.0GB
[15:53:44] step 1760/2860 loss 2.6851 lr 6.00e-04 gnorm 0.10 107,963 tok/s peakVRAM 27.0GB
[15:55:21] step 1770/2860 loss 2.6856 lr 6.00e-04 gnorm 0.08 107,892 tok/s peakVRAM 27.0GB
[15:56:59] step 1780/2860 loss 2.6989 lr 6.00e-04 gnorm 0.10 107,713 tok/s peakVRAM 27.0GB
[15:58:36] step 1790/2860 loss 2.6702 lr 6.00e-04 gnorm 0.08 107,620 tok/s peakVRAM 27.0GB
[16:00:14] step 1800/2860 loss 2.6972 lr 6.00e-04 gnorm 0.09 107,335 tok/s peakVRAM 27.0GB
[16:00:24] ckpt @ 1800
[16:02:01] step 1810/2860 loss 2.6813 lr 6.00e-04 gnorm 0.09 97,749 tok/s peakVRAM 27.0GB
[16:03:38] step 1820/2860 loss 2.6996 lr 6.00e-04 gnorm 0.10 108,119 tok/s peakVRAM 27.0GB
[16:05:15] step 1830/2860 loss 2.7000 lr 6.00e-04 gnorm 0.08 107,858 tok/s peakVRAM 27.0GB
[16:06:52] step 1840/2860 loss 2.6922 lr 6.00e-04 gnorm 0.08 108,092 tok/s peakVRAM 27.0GB
[16:08:29] step 1850/2860 loss 2.6933 lr 6.00e-04 gnorm 0.08 108,196 tok/s peakVRAM 27.0GB
[16:10:06] step 1860/2860 loss 2.7058 lr 6.00e-04 gnorm 0.08 108,083 tok/s peakVRAM 27.0GB
[16:11:43] step 1870/2860 loss 2.6963 lr 6.00e-04 gnorm 0.08 108,178 tok/s peakVRAM 27.0GB
[16:13:20] step 1880/2860 loss 2.6996 lr 6.00e-04 gnorm 0.10 108,173 tok/s peakVRAM 27.0GB
[16:14:57] step 1890/2860 loss 2.6934 lr 6.00e-04 gnorm 0.09 107,941 tok/s peakVRAM 27.0GB
[16:16:34] step 1900/2860 loss 2.6880 lr 6.00e-04 gnorm 0.09 108,129 tok/s peakVRAM 27.0GB
[16:18:11] step 1910/2860 loss 2.6927 lr 6.00e-04 gnorm 0.09 108,021 tok/s peakVRAM 27.0GB
[16:19:49] step 1920/2860 loss 2.6745 lr 6.00e-04 gnorm 0.09 107,920 tok/s peakVRAM 27.0GB
[16:21:26] step 1930/2860 loss 2.6718 lr 6.00e-04 gnorm 0.09 107,949 tok/s peakVRAM 27.0GB
[16:23:03] step 1940/2860 loss 2.6847 lr 6.00e-04 gnorm 0.09 108,210 tok/s peakVRAM 27.0GB
[16:24:40] step 1950/2860 loss 2.6858 lr 6.00e-04 gnorm 0.08 107,953 tok/s peakVRAM 27.0GB
[16:24:50] ckpt @ 1950
[16:26:27] step 1960/2860 loss 2.6984 lr 6.00e-04 gnorm 0.09 97,835 tok/s peakVRAM 27.0GB
[16:28:04] step 1970/2860 loss 2.6685 lr 6.00e-04 gnorm 0.09 107,868 tok/s peakVRAM 27.0GB
[16:29:41] step 1980/2860 loss 2.6823 lr 6.00e-04 gnorm 0.09 107,690 tok/s peakVRAM 27.0GB
[16:31:19] step 1990/2860 loss 2.6722 lr 6.00e-04 gnorm 0.08 107,691 tok/s peakVRAM 27.0GB
[16:32:56] step 2000/2860 loss 2.6615 lr 6.00e-04 gnorm 0.09 107,819 tok/s peakVRAM 27.0GB
[16:34:33] step 2010/2860 loss 2.6696 lr 6.00e-04 gnorm 0.09 108,003 tok/s peakVRAM 27.0GB
[16:36:10] step 2020/2860 loss 2.6537 lr 6.00e-04 gnorm 0.08 107,785 tok/s peakVRAM 27.0GB
[16:37:48] step 2030/2860 loss 2.6670 lr 6.00e-04 gnorm 0.09 107,549 tok/s peakVRAM 27.0GB
[16:39:25] step 2040/2860 loss 2.7043 lr 6.00e-04 gnorm 0.09 107,558 tok/s peakVRAM 27.0GB
[16:41:03] step 2050/2860 loss 2.6757 lr 6.00e-04 gnorm 0.08 107,565 tok/s peakVRAM 27.0GB
[16:42:40] step 2060/2860 loss 2.6730 lr 6.00e-04 gnorm 0.09 107,805 tok/s peakVRAM 27.0GB
[16:44:17] step 2070/2860 loss 2.6598 lr 6.00e-04 gnorm 0.08 107,969 tok/s peakVRAM 27.0GB
[16:45:54] step 2080/2860 loss 2.6709 lr 6.00e-04 gnorm 0.10 108,008 tok/s peakVRAM 27.0GB
[16:47:32] step 2090/2860 loss 2.6693 lr 6.00e-04 gnorm 0.10 107,844 tok/s peakVRAM 27.0GB
[16:49:09] step 2100/2860 loss 2.6343 lr 6.00e-04 gnorm 0.08 107,862 tok/s peakVRAM 27.0GB
[16:49:20] ckpt @ 2100
[16:50:56] step 2110/2860 loss 2.6644 lr 6.00e-04 gnorm 0.10 97,559 tok/s peakVRAM 27.0GB
[16:52:34] step 2120/2860 loss 2.6637 lr 6.00e-04 gnorm 0.07 107,791 tok/s peakVRAM 27.0GB
[16:54:11] step 2130/2860 loss 2.6517 lr 6.00e-04 gnorm 0.09 107,569 tok/s peakVRAM 27.0GB
[16:55:48] step 2140/2860 loss 2.6472 lr 6.00e-04 gnorm 0.08 107,636 tok/s peakVRAM 27.0GB
[16:57:26] step 2150/2860 loss 2.6531 lr 6.00e-04 gnorm 0.08 107,873 tok/s peakVRAM 27.0GB
[16:59:03] step 2160/2860 loss 2.6535 lr 6.00e-04 gnorm 0.08 107,675 tok/s peakVRAM 27.0GB
[17:00:41] step 2170/2860 loss 2.6566 lr 6.00e-04 gnorm 0.08 107,485 tok/s peakVRAM 27.0GB
[17:02:18] step 2180/2860 loss 2.6688 lr 6.00e-04 gnorm 0.09 107,483 tok/s peakVRAM 27.0GB
[17:03:56] step 2190/2860 loss 2.6450 lr 6.00e-04 gnorm 0.08 107,499 tok/s peakVRAM 27.0GB
[17:05:33] step 2200/2860 loss 2.6498 lr 6.00e-04 gnorm 0.09 107,383 tok/s peakVRAM 27.0GB
[17:07:11] step 2210/2860 loss 2.6386 lr 6.00e-04 gnorm 0.09 107,588 tok/s peakVRAM 27.0GB
[17:08:48] step 2220/2860 loss 2.6283 lr 6.00e-04 gnorm 0.09 107,430 tok/s peakVRAM 27.0GB
[17:10:26] step 2230/2860 loss 2.6434 lr 6.00e-04 gnorm 0.07 107,231 tok/s peakVRAM 27.0GB
[17:12:04] step 2240/2860 loss 2.6654 lr 6.00e-04 gnorm 0.08 107,516 tok/s peakVRAM 27.0GB
[17:13:41] step 2250/2860 loss 2.6318 lr 6.00e-04 gnorm 0.08 107,529 tok/s peakVRAM 27.0GB
[17:13:53] ckpt @ 2250
[17:15:30] step 2260/2860 loss 2.6379 lr 6.00e-04 gnorm 0.10 96,309 tok/s peakVRAM 27.0GB
[17:17:07] step 2270/2860 loss 2.6461 lr 6.00e-04 gnorm 0.08 107,863 tok/s peakVRAM 27.0GB
[17:18:45] step 2280/2860 loss 2.6532 lr 6.00e-04 gnorm 0.08 107,509 tok/s peakVRAM 27.0GB
[17:20:22] step 2290/2860 loss 2.6472 lr 5.77e-04 gnorm 0.08 107,551 tok/s peakVRAM 27.0GB
[17:22:00] step 2300/2860 loss 2.6419 lr 5.25e-04 gnorm 0.07 107,791 tok/s peakVRAM 27.0GB
[17:23:37] step 2310/2860 loss 2.6313 lr 4.97e-04 gnorm 0.07 107,474 tok/s peakVRAM 27.0GB
[17:25:15] step 2320/2860 loss 2.6209 lr 4.74e-04 gnorm 0.06 107,316 tok/s peakVRAM 27.0GB
[17:26:53] step 2330/2860 loss 2.6298 lr 4.55e-04 gnorm 0.07 107,306 tok/s peakVRAM 27.0GB
[17:28:30] step 2340/2860 loss 2.6196 lr 4.39e-04 gnorm 0.07 107,440 tok/s peakVRAM 27.0GB
[17:30:08] step 2350/2860 loss 2.6067 lr 4.24e-04 gnorm 0.06 107,241 tok/s peakVRAM 27.0GB
[17:31:46] step 2360/2860 loss 2.5964 lr 4.10e-04 gnorm 0.07 106,925 tok/s peakVRAM 27.0GB
[17:33:24] step 2370/2860 loss 2.5960 lr 3.97e-04 gnorm 0.06 107,135 tok/s peakVRAM 27.0GB
[17:35:02] step 2380/2860 loss 2.5998 lr 3.85e-04 gnorm 0.06 107,354 tok/s peakVRAM 27.0GB
[17:36:39] step 2390/2860 loss 2.5892 lr 3.73e-04 gnorm 0.06 107,686 tok/s peakVRAM 27.0GB
[17:38:16] step 2400/2860 loss 2.6009 lr 3.62e-04 gnorm 0.05 107,816 tok/s peakVRAM 27.0GB
[17:38:29] ckpt @ 2400
[17:40:06] step 2410/2860 loss 2.6035 lr 3.52e-04 gnorm 0.06 95,863 tok/s peakVRAM 27.0GB
[17:41:43] step 2420/2860 loss 2.5908 lr 3.42e-04 gnorm 0.06 107,941 tok/s peakVRAM 27.0GB
[17:43:20] step 2430/2860 loss 2.5813 lr 3.32e-04 gnorm 0.05 107,513 tok/s peakVRAM 27.0GB
[17:44:58] step 2440/2860 loss 2.5774 lr 3.23e-04 gnorm 0.06 107,573 tok/s peakVRAM 27.0GB
[17:46:35] step 2450/2860 loss 2.5865 lr 3.14e-04 gnorm 0.06 107,692 tok/s peakVRAM 27.0GB
[17:48:13] step 2460/2860 loss 2.5850 lr 3.05e-04 gnorm 0.05 107,579 tok/s peakVRAM 27.0GB
[17:49:50] step 2470/2860 loss 2.5728 lr 2.96e-04 gnorm 0.05 107,413 tok/s peakVRAM 27.0GB
[17:51:28] step 2480/2860 loss 2.5804 lr 2.88e-04 gnorm 0.05 107,500 tok/s peakVRAM 27.0GB
[17:53:05] step 2490/2860 loss 2.5865 lr 2.80e-04 gnorm 0.06 107,691 tok/s peakVRAM 27.0GB
[17:54:43] step 2500/2860 loss 2.5757 lr 2.72e-04 gnorm 0.05 107,346 tok/s peakVRAM 27.0GB
[17:56:20] step 2510/2860 loss 2.5683 lr 2.64e-04 gnorm 0.05 107,447 tok/s peakVRAM 27.0GB
[17:57:58] step 2520/2860 loss 2.5659 lr 2.57e-04 gnorm 0.05 107,256 tok/s peakVRAM 27.0GB
[17:59:36] step 2530/2860 loss 2.5493 lr 2.49e-04 gnorm 0.05 107,326 tok/s peakVRAM 27.0GB
[18:01:13] step 2540/2860 loss 2.5526 lr 2.42e-04 gnorm 0.05 107,522 tok/s peakVRAM 27.0GB
[18:02:51] step 2550/2860 loss 2.5513 lr 2.35e-04 gnorm 0.05 107,402 tok/s peakVRAM 27.0GB
[18:03:04] ckpt @ 2550
[18:04:41] step 2560/2860 loss 2.5376 lr 2.28e-04 gnorm 0.05 95,800 tok/s peakVRAM 27.0GB
[18:06:18] step 2570/2860 loss 2.5497 lr 2.22e-04 gnorm 0.05 107,579 tok/s peakVRAM 27.0GB
[18:07:56] step 2580/2860 loss 2.5446 lr 2.15e-04 gnorm 0.05 107,307 tok/s peakVRAM 27.0GB
[18:09:34] step 2590/2860 loss 2.5501 lr 2.08e-04 gnorm 0.05 107,222 tok/s peakVRAM 27.0GB
[18:11:11] step 2600/2860 loss 2.5475 lr 2.02e-04 gnorm 0.05 107,513 tok/s peakVRAM 27.0GB
[18:12:49] step 2610/2860 loss 2.5570 lr 1.95e-04 gnorm 0.05 107,489 tok/s peakVRAM 27.0GB
[18:14:26] step 2620/2860 loss 2.5378 lr 1.89e-04 gnorm 0.05 107,509 tok/s peakVRAM 27.0GB
[18:16:04] step 2630/2860 loss 2.5365 lr 1.83e-04 gnorm 0.05 107,670 tok/s peakVRAM 27.0GB
[18:17:41] step 2640/2860 loss 2.5356 lr 1.77e-04 gnorm 0.05 107,664 tok/s peakVRAM 27.0GB
[18:19:19] step 2650/2860 loss 2.5567 lr 1.71e-04 gnorm 0.05 107,420 tok/s peakVRAM 27.0GB
[18:20:56] step 2660/2860 loss 2.5153 lr 1.65e-04 gnorm 0.05 107,311 tok/s peakVRAM 27.0GB
[18:22:34] step 2670/2860 loss 2.5341 lr 1.59e-04 gnorm 0.05 107,136 tok/s peakVRAM 27.0GB
[18:24:12] step 2680/2860 loss 2.5276 lr 1.54e-04 gnorm 0.05 107,568 tok/s peakVRAM 27.0GB
[18:25:49] step 2690/2860 loss 2.5233 lr 1.48e-04 gnorm 0.05 107,606 tok/s peakVRAM 27.0GB
[18:27:27] step 2700/2860 loss 2.5278 lr 1.42e-04 gnorm 0.05 107,163 tok/s peakVRAM 27.0GB
[18:27:37] ckpt @ 2700
[18:29:15] step 2710/2860 loss 2.5311 lr 1.37e-04 gnorm 0.05 97,103 tok/s peakVRAM 27.0GB
[18:30:53] step 2720/2860 loss 2.5200 lr 1.31e-04 gnorm 0.05 107,242 tok/s peakVRAM 27.0GB
[18:32:30] step 2730/2860 loss 2.5417 lr 1.26e-04 gnorm 0.05 107,248 tok/s peakVRAM 27.0GB
[18:34:08] step 2740/2860 loss 2.5239 lr 1.21e-04 gnorm 0.05 107,314 tok/s peakVRAM 27.0GB
[18:35:46] step 2750/2860 loss 2.5107 lr 1.15e-04 gnorm 0.05 107,318 tok/s peakVRAM 27.0GB
[18:37:24] step 2760/2860 loss 2.4960 lr 1.10e-04 gnorm 0.05 107,012 tok/s peakVRAM 27.0GB
[18:39:02] step 2770/2860 loss 2.5277 lr 1.05e-04 gnorm 0.05 106,943 tok/s peakVRAM 27.0GB
[18:40:40] step 2780/2860 loss 2.5243 lr 9.97e-05 gnorm 0.05 107,074 tok/s peakVRAM 27.0GB
[18:42:18] step 2790/2860 loss 2.5119 lr 9.46e-05 gnorm 0.05 107,164 tok/s peakVRAM 27.0GB
[18:43:55] step 2800/2860 loss 2.5066 lr 8.96e-05 gnorm 0.05 107,364 tok/s peakVRAM 27.0GB
[18:45:33] step 2810/2860 loss 2.5089 lr 8.46e-05 gnorm 0.05 107,309 tok/s peakVRAM 27.0GB
[18:47:11] step 2820/2860 loss 2.5087 lr 7.97e-05 gnorm 0.05 107,408 tok/s peakVRAM 27.0GB
[18:48:48] step 2830/2860 loss 2.4930 lr 7.48e-05 gnorm 0.05 107,398 tok/s peakVRAM 27.0GB
[18:50:26] step 2840/2860 loss 2.5122 lr 7.00e-05 gnorm 0.05 107,539 tok/s peakVRAM 27.0GB
[18:52:03] step 2850/2860 loss 2.4829 lr 6.52e-05 gnorm 0.05 107,477 tok/s peakVRAM 27.0GB
[18:52:16] ckpt @ 2850
[18:53:53] step 2860/2860 loss 2.5067 lr 6.05e-05 gnorm 0.05 95,659 tok/s peakVRAM 27.0GB
[18:54:06] ckpt @ 2860
[18:54:06] DONE
TRAIN-DONE 2026-10-23T18:54:52Z