Hugging Face's logo

SlayerLab
/
dowody400m-checkpoints

PyTorch
polish
checkpoints
training-history
Model card Files
xet
 Community
dowody400m-checkpoints / MU2 /run /train.log
Maggio33's picture
Maggio33
Add run records (args, metrics, logs, val, environment) for all 400M runs, fala-1 finals K22/R1-R3, resume test, code
c28a890 verified
Raw History Blame Contribute Delete
11.1 kB
[04:48:32] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
[04:48:32] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
[04:48:34] model GPT-ref: 183.6M param (L16 d896 h14)
[04:48:34] optimizer=muon muon_lr=0.04 (mult=66.7x)
[04:48:34] r11 pamiec: act_ckpt=True loss_chunk=1024
[04:48:34] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
[04:48:35] torch.compile enabled
[04:48:35] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
[04:52:32] step 20/4768 loss 10.2553 lr 2.40e-05 gnorm 2.32 44,304 tok/s peakVRAM 13.7GB
[04:55:44] step 40/4768 loss 9.2968 lr 4.80e-05 gnorm 1.00 54,450 tok/s peakVRAM 13.7GB
[04:58:57] step 60/4768 loss 8.4951 lr 7.20e-05 gnorm 1.46 54,449 tok/s peakVRAM 13.7GB
[05:02:10] step 80/4768 loss 7.8249 lr 9.60e-05 gnorm 0.69 54,439 tok/s peakVRAM 13.7GB
[05:05:22] step 100/4768 loss 7.3027 lr 1.20e-04 gnorm 0.65 54,421 tok/s peakVRAM 13.7GB
[05:08:35] step 120/4768 loss 6.7967 lr 1.44e-04 gnorm 0.71 54,420 tok/s peakVRAM 13.7GB
[05:11:48] step 140/4768 loss 6.4050 lr 1.68e-04 gnorm 0.49 54,415 tok/s peakVRAM 13.7GB
[05:15:00] step 160/4768 loss 6.1028 lr 1.92e-04 gnorm 0.54 54,413 tok/s peakVRAM 13.7GB
[05:18:13] step 180/4768 loss 5.8786 lr 2.16e-04 gnorm 0.51 54,414 tok/s peakVRAM 13.7GB
[05:21:26] step 200/4768 loss 5.6752 lr 2.40e-04 gnorm 0.56 54,412 tok/s peakVRAM 13.7GB
[05:24:38] step 220/4768 loss 5.4946 lr 2.64e-04 gnorm 0.62 54,411 tok/s peakVRAM 13.7GB
[05:27:51] step 240/4768 loss 5.3280 lr 2.88e-04 gnorm 0.57 54,408 tok/s peakVRAM 13.7GB
[05:31:04] step 260/4768 loss 5.1680 lr 3.12e-04 gnorm 0.43 54,408 tok/s peakVRAM 13.7GB
[05:34:17] step 280/4768 loss 5.0263 lr 3.36e-04 gnorm 0.37 54,403 tok/s peakVRAM 13.7GB
[05:37:29] step 300/4768 loss 4.8846 lr 3.60e-04 gnorm 0.45 54,404 tok/s peakVRAM 13.7GB
[05:40:42] step 320/4768 loss 4.7472 lr 3.84e-04 gnorm 0.32 54,402 tok/s peakVRAM 13.7GB
[05:43:55] step 340/4768 loss 4.6110 lr 4.08e-04 gnorm 0.33 54,399 tok/s peakVRAM 13.7GB
[05:47:08] step 360/4768 loss 4.5043 lr 4.32e-04 gnorm 0.26 54,397 tok/s peakVRAM 13.7GB
[05:50:20] step 380/4768 loss 4.4053 lr 4.56e-04 gnorm 0.22 54,390 tok/s peakVRAM 13.7GB
[05:53:33] step 400/4768 loss 4.3194 lr 4.80e-04 gnorm 0.25 54,389 tok/s peakVRAM 13.7GB
[05:56:46] step 420/4768 loss 4.2544 lr 5.04e-04 gnorm 0.21 54,392 tok/s peakVRAM 13.7GB
[05:59:59] step 440/4768 loss 4.1844 lr 5.28e-04 gnorm 0.21 54,392 tok/s peakVRAM 13.7GB
[06:03:12] step 460/4768 loss 4.1207 lr 5.52e-04 gnorm 0.18 54,390 tok/s peakVRAM 13.7GB
[06:06:24] step 480/4768 loss 4.0799 lr 5.76e-04 gnorm 0.22 54,389 tok/s peakVRAM 13.7GB
[06:09:37] step 500/4768 loss 4.0291 lr 6.00e-04 gnorm 0.18 54,386 tok/s peakVRAM 13.7GB
[06:09:39] ckpt @ 500
[06:12:52] step 520/4768 loss 3.9946 lr 6.00e-04 gnorm 0.16 53,918 tok/s peakVRAM 13.7GB
[06:16:04] step 540/4768 loss 3.9497 lr 6.00e-04 gnorm 0.16 54,388 tok/s peakVRAM 13.7GB
[06:19:17] step 560/4768 loss 3.8994 lr 6.00e-04 gnorm 0.20 54,386 tok/s peakVRAM 13.7GB
[06:22:30] step 580/4768 loss 3.8611 lr 6.00e-04 gnorm 0.15 54,384 tok/s peakVRAM 13.7GB
[06:25:43] step 600/4768 loss 3.8206 lr 6.00e-04 gnorm 0.15 54,386 tok/s peakVRAM 13.7GB
[06:28:56] step 620/4768 loss 3.7999 lr 6.00e-04 gnorm 0.14 54,384 tok/s peakVRAM 13.7GB
[06:32:09] step 640/4768 loss 3.7752 lr 6.00e-04 gnorm 0.12 54,382 tok/s peakVRAM 13.7GB
[06:35:21] step 660/4768 loss 3.7472 lr 6.00e-04 gnorm 0.11 54,379 tok/s peakVRAM 13.7GB
[06:38:34] step 680/4768 loss 3.7249 lr 6.00e-04 gnorm 0.13 54,376 tok/s peakVRAM 13.7GB
[06:41:47] step 700/4768 loss 3.6959 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
[06:45:00] step 720/4768 loss 3.6670 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
[06:48:13] step 740/4768 loss 3.6638 lr 6.00e-04 gnorm 0.12 54,382 tok/s peakVRAM 13.7GB
[06:51:25] step 760/4768 loss 3.6386 lr 6.00e-04 gnorm 0.12 54,383 tok/s peakVRAM 13.7GB
[06:54:38] step 780/4768 loss 3.6130 lr 6.00e-04 gnorm 0.10 54,386 tok/s peakVRAM 13.7GB
[06:57:51] step 800/4768 loss 3.6005 lr 6.00e-04 gnorm 0.12 54,385 tok/s peakVRAM 13.7GB
[07:01:04] step 820/4768 loss 3.5843 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
[07:04:17] step 840/4768 loss 3.5713 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
[07:07:29] step 860/4768 loss 3.5506 lr 6.00e-04 gnorm 0.10 54,384 tok/s peakVRAM 13.7GB
[07:10:42] step 880/4768 loss 3.5451 lr 6.00e-04 gnorm 0.11 54,385 tok/s peakVRAM 13.7GB
[07:13:55] step 900/4768 loss 3.5280 lr 6.00e-04 gnorm 0.11 54,387 tok/s peakVRAM 13.7GB
[07:17:08] step 920/4768 loss 3.5207 lr 6.00e-04 gnorm 0.10 54,382 tok/s peakVRAM 13.7GB
[07:20:21] step 940/4768 loss 3.5164 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
[07:23:34] step 960/4768 loss 3.4913 lr 6.00e-04 gnorm 0.08 54,381 tok/s peakVRAM 13.7GB
[07:26:46] step 980/4768 loss 3.4890 lr 6.00e-04 gnorm 0.08 54,380 tok/s peakVRAM 13.7GB
[07:29:59] step 1000/4768 loss 3.4788 lr 6.00e-04 gnorm 0.09 54,379 tok/s peakVRAM 13.7GB
[07:30:02] ckpt @ 1000
[07:33:14] step 1020/4768 loss 3.4739 lr 6.00e-04 gnorm 0.12 53,735 tok/s peakVRAM 13.7GB
[07:36:27] step 1040/4768 loss 3.4472 lr 6.00e-04 gnorm 0.09 54,384 tok/s peakVRAM 13.7GB
[07:39:40] step 1060/4768 loss 3.4589 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
[07:42:53] step 1080/4768 loss 3.4485 lr 6.00e-04 gnorm 0.09 54,379 tok/s peakVRAM 13.7GB
[07:46:06] step 1100/4768 loss 3.4334 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
[07:49:18] step 1120/4768 loss 3.4154 lr 6.00e-04 gnorm 0.10 54,379 tok/s peakVRAM 13.7GB
[07:52:31] step 1140/4768 loss 3.4083 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
[07:55:44] step 1160/4768 loss 3.4036 lr 6.00e-04 gnorm 0.09 54,383 tok/s peakVRAM 13.7GB
[07:58:57] step 1180/4768 loss 3.4058 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
[08:02:10] step 1200/4768 loss 3.3883 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
[08:05:23] step 1220/4768 loss 3.3856 lr 6.00e-04 gnorm 0.10 54,376 tok/s peakVRAM 13.7GB
[08:08:35] step 1240/4768 loss 3.3775 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
[08:11:48] step 1260/4768 loss 3.3644 lr 6.00e-04 gnorm 0.10 54,380 tok/s peakVRAM 13.7GB
[08:15:01] step 1280/4768 loss 3.3676 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
[08:18:14] step 1300/4768 loss 3.3609 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
[08:21:27] step 1320/4768 loss 3.3531 lr 6.00e-04 gnorm 0.10 54,380 tok/s peakVRAM 13.7GB
[08:24:39] step 1340/4768 loss 3.3537 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
[08:27:52] step 1360/4768 loss 3.3450 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
[08:31:05] step 1380/4768 loss 3.3367 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
[08:34:18] step 1400/4768 loss 3.3268 lr 6.00e-04 gnorm 0.09 54,382 tok/s peakVRAM 13.7GB
[08:37:31] step 1420/4768 loss 3.3143 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
[08:40:44] step 1440/4768 loss 3.3218 lr 6.00e-04 gnorm 0.08 54,383 tok/s peakVRAM 13.7GB
[08:43:56] step 1460/4768 loss 3.3195 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
[08:47:09] step 1480/4768 loss 3.3233 lr 6.00e-04 gnorm 0.08 54,379 tok/s peakVRAM 13.7GB
[08:50:22] step 1500/4768 loss 3.2963 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
[08:50:24] ckpt @ 1500
[08:53:37] step 1520/4768 loss 3.3041 lr 6.00e-04 gnorm 0.09 53,713 tok/s peakVRAM 13.7GB
[08:56:50] step 1540/4768 loss 3.2883 lr 6.00e-04 gnorm 0.09 54,374 tok/s peakVRAM 13.7GB
[09:00:03] step 1560/4768 loss 3.2969 lr 6.00e-04 gnorm 0.08 54,377 tok/s peakVRAM 13.7GB
[09:03:16] step 1580/4768 loss 3.2851 lr 6.00e-04 gnorm 0.09 54,376 tok/s peakVRAM 13.7GB
[09:06:29] step 1600/4768 loss 3.2750 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
[09:09:41] step 1620/4768 loss 3.2753 lr 6.00e-04 gnorm 0.09 54,376 tok/s peakVRAM 13.7GB
[09:13:59] step 1640/4768 loss 3.2726 lr 6.00e-04 gnorm 0.09 40,752 tok/s peakVRAM 13.7GB
[09:18:26] step 1660/4768 loss 3.2733 lr 6.00e-04 gnorm 0.09 39,290 tok/s peakVRAM 13.7GB
[09:22:46] step 1680/4768 loss 3.2623 lr 6.00e-04 gnorm 0.10 40,313 tok/s peakVRAM 13.7GB
[09:25:59] step 1700/4768 loss 3.2555 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
[09:29:11] step 1720/4768 loss 3.2578 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
[09:32:24] step 1740/4768 loss 3.2513 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
[09:35:37] step 1760/4768 loss 3.2508 lr 6.00e-04 gnorm 0.10 54,389 tok/s peakVRAM 13.7GB
[09:38:50] step 1780/4768 loss 3.2381 lr 6.00e-04 gnorm 0.10 54,384 tok/s peakVRAM 13.7GB
[09:42:03] step 1800/4768 loss 3.2418 lr 6.00e-04 gnorm 0.10 54,379 tok/s peakVRAM 13.7GB
[09:45:18] step 1820/4768 loss 3.2374 lr 6.00e-04 gnorm 0.08 53,646 tok/s peakVRAM 13.7GB
[09:51:02] step 1840/4768 loss 3.2313 lr 6.00e-04 gnorm 0.09 30,467 tok/s peakVRAM 13.7GB
[09:54:58] step 1860/4768 loss 3.2303 lr 6.00e-04 gnorm 0.09 44,403 tok/s peakVRAM 13.7GB
[10:00:30] step 1880/4768 loss 3.2265 lr 6.00e-04 gnorm 0.08 31,623 tok/s peakVRAM 13.7GB
[10:06:04] step 1900/4768 loss 3.2209 lr 6.00e-04 gnorm 0.09 31,365 tok/s peakVRAM 13.7GB
[10:11:58] step 1920/4768 loss 3.2205 lr 6.00e-04 gnorm 0.08 29,665 tok/s peakVRAM 13.7GB
[10:17:51] step 1940/4768 loss 3.2141 lr 6.00e-04 gnorm 0.08 29,662 tok/s peakVRAM 13.7GB
[10:23:36] step 1960/4768 loss 3.2180 lr 6.00e-04 gnorm 0.08 30,457 tok/s peakVRAM 13.7GB
[10:29:02] step 1980/4768 loss 3.2126 lr 6.00e-04 gnorm 0.09 32,076 tok/s peakVRAM 13.7GB
[10:34:30] step 2000/4768 loss 3.2058 lr 6.00e-04 gnorm 0.09 32,047 tok/s peakVRAM 13.7GB
[10:34:32] ckpt @ 2000
[10:40:00] step 2020/4768 loss 3.2055 lr 6.00e-04 gnorm 0.09 31,786 tok/s peakVRAM 13.7GB
[10:45:28] step 2040/4768 loss 3.1983 lr 6.00e-04 gnorm 0.09 31,965 tok/s peakVRAM 13.7GB
[10:50:55] step 2060/4768 loss 3.1986 lr 6.00e-04 gnorm 0.08 31,995 tok/s peakVRAM 13.7GB
[10:56:23] step 2080/4768 loss 3.1876 lr 6.00e-04 gnorm 0.08 32,010 tok/s peakVRAM 13.7GB
[11:01:51] step 2100/4768 loss 3.1961 lr 6.00e-04 gnorm 0.10 31,946 tok/s peakVRAM 13.7GB
[11:07:19] step 2120/4768 loss 3.1859 lr 6.00e-04 gnorm 0.09 31,941 tok/s peakVRAM 13.7GB
[11:12:47] step 2140/4768 loss 3.1989 lr 6.00e-04 gnorm 0.08 32,013 tok/s peakVRAM 13.7GB
[11:18:15] step 2160/4768 loss 3.1776 lr 6.00e-04 gnorm 0.08 31,944 tok/s peakVRAM 13.7GB
[11:23:44] step 2180/4768 loss 3.1793 lr 6.00e-04 gnorm 0.09 31,909 tok/s peakVRAM 13.7GB
[11:28:11] step 2200/4768 loss 3.1803 lr 6.00e-04 gnorm 0.08 39,315 tok/s peakVRAM 13.7GB
[11:33:16] step 2220/4768 loss 3.1734 lr 6.00e-04 gnorm 0.09 34,363 tok/s peakVRAM 13.7GB
[11:39:24] step 2240/4768 loss 3.1700 lr 6.00e-04 gnorm 0.08 28,500 tok/s peakVRAM 13.7GB
[11:44:46] step 2260/4768 loss 3.1662 lr 6.00e-04 gnorm 0.08 32,518 tok/s peakVRAM 13.7GB
[11:50:32] step 2280/4768 loss 3.1625 lr 6.00e-04 gnorm 0.08 30,346 tok/s peakVRAM 13.7GB
[11:56:29] step 2300/4768 loss 3.1659 lr 6.00e-04 gnorm 0.08 29,311 tok/s peakVRAM 13.7GB
[12:01:34] step 2320/4768 loss 3.1626 lr 6.00e-04 gnorm 0.08 34,480 tok/s peakVRAM 13.7GB
[12:07:48] step 2340/4768 loss 3.1578 lr 6.00e-04 gnorm 0.09 28,010 tok/s peakVRAM 13.7GB