Hugging Face's logo

SlayerLab
/
dowody400m-checkpoints

PyTorch
polish
checkpoints
training-history
Model card Files
xet
 Community
dowody400m-checkpoints / VR0 /run /train.log
Maggio33's picture
Maggio33
Add run records (args, metrics, logs, val, environment) for all 400M runs, fala-1 finals K22/R1-R3, resume test, code
c28a890 verified
Raw History Blame Contribute Delete
13.4 kB
[04:33:11] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
[04:33:11] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
[04:33:13] model GPT-ref: 183.6M param (L16 d896 h14)
[04:33:14] optimizer=muon muon_lr=0.02 (mult=33.3x)
[04:33:14] r11 pamiec: act_ckpt=True loss_chunk=1024
[04:33:14] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
[04:33:15] torch.compile enabled
[04:33:15] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
[04:37:15] step 20/4768 loss 10.3757 lr 2.40e-05 gnorm 2.72 43,562 tok/s peakVRAM 13.2GB
[04:40:27] step 40/4768 loss 9.5564 lr 4.80e-05 gnorm 1.29 54,802 tok/s peakVRAM 13.2GB
[04:43:38] step 60/4768 loss 8.6944 lr 7.20e-05 gnorm 0.71 54,811 tok/s peakVRAM 13.2GB
[04:46:49] step 80/4768 loss 8.0439 lr 9.60e-05 gnorm 1.17 54,800 tok/s peakVRAM 13.2GB
[04:50:01] step 100/4768 loss 7.5902 lr 1.20e-04 gnorm 0.90 54,779 tok/s peakVRAM 13.2GB
[04:53:12] step 120/4768 loss 7.1203 lr 1.44e-04 gnorm 1.05 54,753 tok/s peakVRAM 13.2GB
[04:56:24] step 140/4768 loss 6.7067 lr 1.68e-04 gnorm 0.96 54,738 tok/s peakVRAM 13.2GB
[04:59:35] step 160/4768 loss 6.3500 lr 1.92e-04 gnorm 0.76 54,719 tok/s peakVRAM 13.2GB
[05:02:47] step 180/4768 loss 6.0843 lr 2.16e-04 gnorm 0.72 54,707 tok/s peakVRAM 13.2GB
[05:05:59] step 200/4768 loss 5.8631 lr 2.40e-04 gnorm 0.79 54,698 tok/s peakVRAM 13.2GB
[05:09:10] step 220/4768 loss 5.6758 lr 2.64e-04 gnorm 0.73 54,717 tok/s peakVRAM 13.2GB
[05:12:22] step 240/4768 loss 5.5004 lr 2.88e-04 gnorm 0.74 54,726 tok/s peakVRAM 13.2GB
[05:15:34] step 260/4768 loss 5.3312 lr 3.12e-04 gnorm 0.65 54,722 tok/s peakVRAM 13.2GB
[05:18:45] step 280/4768 loss 5.1733 lr 3.36e-04 gnorm 0.82 54,720 tok/s peakVRAM 13.2GB
[05:21:57] step 300/4768 loss 5.0295 lr 3.60e-04 gnorm 0.71 54,721 tok/s peakVRAM 13.2GB
[05:25:09] step 320/4768 loss 4.8940 lr 3.84e-04 gnorm 0.77 54,728 tok/s peakVRAM 13.2GB
[05:28:20] step 340/4768 loss 4.7543 lr 4.08e-04 gnorm 0.60 54,722 tok/s peakVRAM 13.2GB
[05:31:32] step 360/4768 loss 4.6273 lr 4.32e-04 gnorm 0.51 54,717 tok/s peakVRAM 13.2GB
[05:34:43] step 380/4768 loss 4.5068 lr 4.56e-04 gnorm 0.51 54,718 tok/s peakVRAM 13.2GB
[05:37:55] step 400/4768 loss 4.4057 lr 4.80e-04 gnorm 0.47 54,715 tok/s peakVRAM 13.2GB
[05:41:07] step 420/4768 loss 4.3239 lr 5.04e-04 gnorm 0.49 54,712 tok/s peakVRAM 13.2GB
[05:44:18] step 440/4768 loss 4.2493 lr 5.28e-04 gnorm 0.40 54,707 tok/s peakVRAM 13.2GB
[05:47:30] step 460/4768 loss 4.1796 lr 5.52e-04 gnorm 0.39 54,709 tok/s peakVRAM 13.2GB
[05:50:42] step 480/4768 loss 4.1352 lr 5.76e-04 gnorm 0.36 54,715 tok/s peakVRAM 13.2GB
[05:53:53] step 500/4768 loss 4.0760 lr 6.00e-04 gnorm 0.30 54,704 tok/s peakVRAM 13.2GB
[05:53:55] ckpt @ 500
[05:57:07] step 520/4768 loss 4.0400 lr 6.00e-04 gnorm 0.31 54,202 tok/s peakVRAM 13.2GB
[06:00:19] step 540/4768 loss 3.9888 lr 6.00e-04 gnorm 0.27 54,667 tok/s peakVRAM 13.2GB
[06:03:30] step 560/4768 loss 3.9374 lr 6.00e-04 gnorm 0.21 54,666 tok/s peakVRAM 13.2GB
[06:06:42] step 580/4768 loss 3.8953 lr 6.00e-04 gnorm 0.24 54,661 tok/s peakVRAM 13.2GB
[06:09:54] step 600/4768 loss 3.8538 lr 6.00e-04 gnorm 0.24 54,647 tok/s peakVRAM 13.2GB
[06:13:06] step 620/4768 loss 3.8324 lr 6.00e-04 gnorm 0.20 54,649 tok/s peakVRAM 13.2GB
[06:16:18] step 640/4768 loss 3.8060 lr 6.00e-04 gnorm 0.21 54,653 tok/s peakVRAM 13.2GB
[06:19:30] step 660/4768 loss 3.7743 lr 6.00e-04 gnorm 0.16 54,649 tok/s peakVRAM 13.2GB
[06:22:42] step 680/4768 loss 3.7522 lr 6.00e-04 gnorm 0.18 54,642 tok/s peakVRAM 13.2GB
[06:25:54] step 700/4768 loss 3.7206 lr 6.00e-04 gnorm 0.16 54,646 tok/s peakVRAM 13.2GB
[06:29:05] step 720/4768 loss 3.6912 lr 6.00e-04 gnorm 0.16 54,647 tok/s peakVRAM 13.2GB
[06:32:17] step 740/4768 loss 3.6871 lr 6.00e-04 gnorm 0.17 54,646 tok/s peakVRAM 13.2GB
[06:35:29] step 760/4768 loss 3.6618 lr 6.00e-04 gnorm 0.15 54,634 tok/s peakVRAM 13.2GB
[06:38:41] step 780/4768 loss 3.6361 lr 6.00e-04 gnorm 0.16 54,634 tok/s peakVRAM 13.2GB
[06:41:53] step 800/4768 loss 3.6228 lr 6.00e-04 gnorm 0.14 54,641 tok/s peakVRAM 13.2GB
[06:45:05] step 820/4768 loss 3.6064 lr 6.00e-04 gnorm 0.17 54,635 tok/s peakVRAM 13.2GB
[06:48:17] step 840/4768 loss 3.5923 lr 6.00e-04 gnorm 0.15 54,631 tok/s peakVRAM 13.2GB
[06:51:29] step 860/4768 loss 3.5710 lr 6.00e-04 gnorm 0.13 54,635 tok/s peakVRAM 13.2GB
[06:54:41] step 880/4768 loss 3.5654 lr 6.00e-04 gnorm 0.14 54,631 tok/s peakVRAM 13.2GB
[06:57:53] step 900/4768 loss 3.5473 lr 6.00e-04 gnorm 0.15 54,636 tok/s peakVRAM 13.2GB
[07:01:05] step 920/4768 loss 3.5404 lr 6.00e-04 gnorm 0.14 54,640 tok/s peakVRAM 13.2GB
[07:04:17] step 940/4768 loss 3.5347 lr 6.00e-04 gnorm 0.13 54,638 tok/s peakVRAM 13.2GB
[07:07:29] step 960/4768 loss 3.5092 lr 6.00e-04 gnorm 0.13 54,630 tok/s peakVRAM 13.2GB
[07:10:40] step 980/4768 loss 3.5068 lr 6.00e-04 gnorm 0.14 54,640 tok/s peakVRAM 13.2GB
[07:13:52] step 1000/4768 loss 3.4964 lr 6.00e-04 gnorm 0.12 54,636 tok/s peakVRAM 13.2GB
[07:13:55] ckpt @ 1000
[07:17:06] step 1020/4768 loss 3.4906 lr 6.00e-04 gnorm 0.14 54,035 tok/s peakVRAM 13.2GB
[07:20:18] step 1040/4768 loss 3.4636 lr 6.00e-04 gnorm 0.13 54,680 tok/s peakVRAM 13.2GB
[07:23:30] step 1060/4768 loss 3.4751 lr 6.00e-04 gnorm 0.12 54,676 tok/s peakVRAM 13.2GB
[07:26:42] step 1080/4768 loss 3.4651 lr 6.00e-04 gnorm 0.14 54,670 tok/s peakVRAM 13.2GB
[07:29:54] step 1100/4768 loss 3.4490 lr 6.00e-04 gnorm 0.13 54,677 tok/s peakVRAM 13.2GB
[07:33:05] step 1120/4768 loss 3.4309 lr 6.00e-04 gnorm 0.12 54,668 tok/s peakVRAM 13.2GB
[07:36:17] step 1140/4768 loss 3.4241 lr 6.00e-04 gnorm 0.13 54,670 tok/s peakVRAM 13.2GB
[07:39:29] step 1160/4768 loss 3.4195 lr 6.00e-04 gnorm 0.14 54,669 tok/s peakVRAM 13.2GB
[07:42:41] step 1180/4768 loss 3.4214 lr 6.00e-04 gnorm 0.13 54,674 tok/s peakVRAM 13.2GB
[07:45:53] step 1200/4768 loss 3.4033 lr 6.00e-04 gnorm 0.11 54,676 tok/s peakVRAM 13.2GB
[07:49:04] step 1220/4768 loss 3.4008 lr 6.00e-04 gnorm 0.11 54,665 tok/s peakVRAM 13.2GB
[07:52:16] step 1240/4768 loss 3.3927 lr 6.00e-04 gnorm 0.14 54,664 tok/s peakVRAM 13.2GB
[07:55:28] step 1260/4768 loss 3.3785 lr 6.00e-04 gnorm 0.13 54,668 tok/s peakVRAM 13.2GB
[07:58:40] step 1280/4768 loss 3.3814 lr 6.00e-04 gnorm 0.11 54,669 tok/s peakVRAM 13.2GB
[08:01:52] step 1300/4768 loss 3.3748 lr 6.00e-04 gnorm 0.11 54,668 tok/s peakVRAM 13.2GB
[08:05:03] step 1320/4768 loss 3.3672 lr 6.00e-04 gnorm 0.12 54,674 tok/s peakVRAM 13.2GB
[08:08:15] step 1340/4768 loss 3.3675 lr 6.00e-04 gnorm 0.13 54,666 tok/s peakVRAM 13.2GB
[08:11:27] step 1360/4768 loss 3.3585 lr 6.00e-04 gnorm 0.13 54,661 tok/s peakVRAM 13.2GB
[08:14:39] step 1380/4768 loss 3.3499 lr 6.00e-04 gnorm 0.11 54,661 tok/s peakVRAM 13.2GB
[08:17:51] step 1400/4768 loss 3.3391 lr 6.00e-04 gnorm 0.11 54,657 tok/s peakVRAM 13.2GB
[08:21:03] step 1420/4768 loss 3.3271 lr 6.00e-04 gnorm 0.13 54,658 tok/s peakVRAM 13.2GB
[08:24:14] step 1440/4768 loss 3.3344 lr 6.00e-04 gnorm 0.13 54,655 tok/s peakVRAM 13.2GB
[08:27:26] step 1460/4768 loss 3.3319 lr 6.00e-04 gnorm 0.11 54,661 tok/s peakVRAM 13.2GB
[08:30:38] step 1480/4768 loss 3.3359 lr 6.00e-04 gnorm 0.12 54,661 tok/s peakVRAM 13.2GB
[08:33:50] step 1500/4768 loss 3.3086 lr 6.00e-04 gnorm 0.12 54,664 tok/s peakVRAM 13.2GB
[08:33:52] ckpt @ 1500
[08:37:04] step 1520/4768 loss 3.3161 lr 6.00e-04 gnorm 0.12 53,964 tok/s peakVRAM 13.2GB
[08:40:16] step 1540/4768 loss 3.3000 lr 6.00e-04 gnorm 0.11 54,622 tok/s peakVRAM 13.2GB
[08:43:28] step 1560/4768 loss 3.3088 lr 6.00e-04 gnorm 0.11 54,621 tok/s peakVRAM 13.2GB
[08:46:40] step 1580/4768 loss 3.2972 lr 6.00e-04 gnorm 0.11 54,645 tok/s peakVRAM 13.2GB
[08:49:52] step 1600/4768 loss 3.2864 lr 6.00e-04 gnorm 0.12 54,681 tok/s peakVRAM 13.2GB
[08:53:04] step 1620/4768 loss 3.2873 lr 6.00e-04 gnorm 0.13 54,610 tok/s peakVRAM 13.2GB
[08:56:16] step 1640/4768 loss 3.2835 lr 6.00e-04 gnorm 0.11 54,603 tok/s peakVRAM 13.2GB
[08:59:28] step 1660/4768 loss 3.2847 lr 6.00e-04 gnorm 0.11 54,609 tok/s peakVRAM 13.2GB
[09:02:40] step 1680/4768 loss 3.2738 lr 6.00e-04 gnorm 0.10 54,612 tok/s peakVRAM 13.2GB
[09:05:52] step 1700/4768 loss 3.2668 lr 6.00e-04 gnorm 0.11 54,611 tok/s peakVRAM 13.2GB
[09:09:04] step 1720/4768 loss 3.2687 lr 6.00e-04 gnorm 0.11 54,604 tok/s peakVRAM 13.2GB
[09:12:16] step 1740/4768 loss 3.2622 lr 6.00e-04 gnorm 0.11 54,617 tok/s peakVRAM 13.2GB
[09:15:28] step 1760/4768 loss 3.2621 lr 6.00e-04 gnorm 0.12 54,605 tok/s peakVRAM 13.2GB
[09:18:40] step 1780/4768 loss 3.2487 lr 6.00e-04 gnorm 0.11 54,605 tok/s peakVRAM 13.2GB
[09:21:52] step 1800/4768 loss 3.2527 lr 6.00e-04 gnorm 0.11 54,603 tok/s peakVRAM 13.2GB
[09:25:04] step 1820/4768 loss 3.2477 lr 6.00e-04 gnorm 0.11 54,613 tok/s peakVRAM 13.2GB
[09:28:16] step 1840/4768 loss 3.2415 lr 6.00e-04 gnorm 0.10 54,610 tok/s peakVRAM 13.2GB
[09:31:28] step 1860/4768 loss 3.2405 lr 6.00e-04 gnorm 0.11 54,619 tok/s peakVRAM 13.2GB
[09:34:40] step 1880/4768 loss 3.2366 lr 6.00e-04 gnorm 0.10 54,626 tok/s peakVRAM 13.2GB
[09:37:52] step 1900/4768 loss 3.2313 lr 6.00e-04 gnorm 0.11 54,622 tok/s peakVRAM 13.2GB
[09:41:04] step 1920/4768 loss 3.2317 lr 6.00e-04 gnorm 0.11 54,616 tok/s peakVRAM 13.2GB
[09:44:16] step 1940/4768 loss 3.2239 lr 6.00e-04 gnorm 0.11 54,614 tok/s peakVRAM 13.2GB
[09:47:28] step 1960/4768 loss 3.2283 lr 6.00e-04 gnorm 0.10 54,610 tok/s peakVRAM 13.2GB
[09:50:40] step 1980/4768 loss 3.2230 lr 6.00e-04 gnorm 0.11 54,610 tok/s peakVRAM 13.2GB
[09:53:52] step 2000/4768 loss 3.2165 lr 6.00e-04 gnorm 0.12 54,610 tok/s peakVRAM 13.2GB
[09:53:54] ckpt @ 2000
[09:57:06] step 2020/4768 loss 3.2159 lr 6.00e-04 gnorm 0.11 53,972 tok/s peakVRAM 13.2GB
[10:00:18] step 2040/4768 loss 3.2089 lr 6.00e-04 gnorm 0.10 54,629 tok/s peakVRAM 13.2GB
[10:03:30] step 2060/4768 loss 3.2095 lr 6.00e-04 gnorm 0.11 54,629 tok/s peakVRAM 13.2GB
[10:06:42] step 2080/4768 loss 3.1978 lr 6.00e-04 gnorm 0.10 54,624 tok/s peakVRAM 13.2GB
[10:09:54] step 2100/4768 loss 3.2070 lr 6.00e-04 gnorm 0.10 54,619 tok/s peakVRAM 13.2GB
[10:13:06] step 2120/4768 loss 3.1962 lr 6.00e-04 gnorm 0.11 54,623 tok/s peakVRAM 13.2GB
[10:16:18] step 2140/4768 loss 3.2091 lr 6.00e-04 gnorm 0.10 54,626 tok/s peakVRAM 13.2GB
[10:19:30] step 2160/4768 loss 3.1877 lr 6.00e-04 gnorm 0.11 54,621 tok/s peakVRAM 13.2GB
[10:22:42] step 2180/4768 loss 3.1903 lr 6.00e-04 gnorm 0.10 54,629 tok/s peakVRAM 13.2GB
[10:25:54] step 2200/4768 loss 3.1908 lr 6.00e-04 gnorm 0.11 54,631 tok/s peakVRAM 13.2GB
[10:29:06] step 2220/4768 loss 3.1834 lr 6.00e-04 gnorm 0.10 54,628 tok/s peakVRAM 13.2GB
[10:32:18] step 2240/4768 loss 3.1802 lr 6.00e-04 gnorm 0.10 54,631 tok/s peakVRAM 13.2GB
[10:35:30] step 2260/4768 loss 3.1768 lr 6.00e-04 gnorm 0.11 54,632 tok/s peakVRAM 13.2GB
[10:38:42] step 2280/4768 loss 3.1731 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[10:41:54] step 2300/4768 loss 3.1769 lr 6.00e-04 gnorm 0.10 54,627 tok/s peakVRAM 13.2GB
[10:45:06] step 2320/4768 loss 3.1732 lr 6.00e-04 gnorm 0.11 54,626 tok/s peakVRAM 13.2GB
[10:48:17] step 2340/4768 loss 3.1687 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[10:51:29] step 2360/4768 loss 3.1588 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[10:54:41] step 2380/4768 loss 3.1581 lr 6.00e-04 gnorm 0.10 54,632 tok/s peakVRAM 13.2GB
[10:57:53] step 2400/4768 loss 3.1582 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[11:01:05] step 2420/4768 loss 3.1613 lr 6.00e-04 gnorm 0.09 54,632 tok/s peakVRAM 13.2GB
[11:04:17] step 2440/4768 loss 3.1547 lr 6.00e-04 gnorm 0.11 54,634 tok/s peakVRAM 13.2GB
[11:07:29] step 2460/4768 loss 3.1544 lr 6.00e-04 gnorm 0.10 54,628 tok/s peakVRAM 13.2GB
[11:10:41] step 2480/4768 loss 3.1510 lr 6.00e-04 gnorm 0.11 54,633 tok/s peakVRAM 13.2GB
[11:13:53] step 2500/4768 loss 3.1457 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[11:13:55] ckpt @ 2500
[11:17:07] step 2520/4768 loss 3.1475 lr 6.00e-04 gnorm 0.10 53,971 tok/s peakVRAM 13.2GB
[11:20:19] step 2540/4768 loss 3.1478 lr 6.00e-04 gnorm 0.10 54,626 tok/s peakVRAM 13.2GB
[11:23:31] step 2560/4768 loss 3.1449 lr 6.00e-04 gnorm 0.11 54,619 tok/s peakVRAM 13.2GB
[11:26:43] step 2580/4768 loss 3.1403 lr 6.00e-04 gnorm 0.10 54,627 tok/s peakVRAM 13.2GB
[11:29:55] step 2600/4768 loss 3.1308 lr 6.00e-04 gnorm 0.10 54,624 tok/s peakVRAM 13.2GB
[11:33:07] step 2620/4768 loss 3.1397 lr 6.00e-04 gnorm 0.10 54,625 tok/s peakVRAM 13.2GB
[11:36:19] step 2640/4768 loss 3.1406 lr 6.00e-04 gnorm 0.10 54,625 tok/s peakVRAM 13.2GB
[11:39:31] step 2660/4768 loss 3.1389 lr 6.00e-04 gnorm 0.09 54,628 tok/s peakVRAM 13.2GB
[11:42:43] step 2680/4768 loss 3.1341 lr 6.00e-04 gnorm 0.10 54,629 tok/s peakVRAM 13.2GB
[11:45:55] step 2700/4768 loss 3.1367 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[11:49:07] step 2720/4768 loss 3.1201 lr 6.00e-04 gnorm 0.11 54,629 tok/s peakVRAM 13.2GB
[11:52:19] step 2740/4768 loss 3.1216 lr 6.00e-04 gnorm 0.10 54,629 tok/s peakVRAM 13.2GB
[11:55:31] step 2760/4768 loss 3.1249 lr 6.00e-04 gnorm 0.10 54,632 tok/s peakVRAM 13.2GB
[11:58:43] step 2780/4768 loss 3.1154 lr 6.00e-04 gnorm 0.10 54,630 tok/s peakVRAM 13.2GB
[12:01:55] step 2800/4768 loss 3.1102 lr 6.00e-04 gnorm 0.10 54,629 tok/s peakVRAM 13.2GB
[12:05:07] step 2820/4768 loss 3.1080 lr 6.00e-04 gnorm 0.09 54,619 tok/s peakVRAM 13.2GB
[12:08:19] step 2840/4768 loss 3.1164 lr 6.00e-04 gnorm 0.10 54,612 tok/s peakVRAM 13.2GB
[12:11:31] step 2860/4768 loss 3.1075 lr 6.00e-04 gnorm 0.10 54,620 tok/s peakVRAM 13.2GB