Hugging Face's logo

SlayerLab
/
dowody400m-checkpoints

PyTorch
polish
checkpoints
training-history
Model card Files
xet
 Community
Maggio33 commited on
Commit
c28a890
·
verified ·
1 Parent(s): e6a5e28

Add run records (args, metrics, logs, val, environment) for all 400M runs, fala-1 finals K22/R1-R3, resume test, code

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. GQA/run/SHA256SUMS_kopiarka +5 -0
  2. GQA/run/args.txt +1 -0
  3. GQA/run/metrics.jsonl +146 -0
  4. GQA/run/srodowisko_GQA_mini0_resume0.txt +64 -0
  5. GQA/run/srodowisko_resume0.txt +64 -0
  6. GQA/run/train.log +159 -0
  7. GQA/run/train.out +159 -0
  8. K12/run/SHA256SUMS_kopiarka +6 -0
  9. K12/run/args.txt +1 -0
  10. K12/run/metrics.jsonl +153 -0
  11. K12/run/srodowisko_K12_mini0_resume0.txt +64 -0
  12. K12/run/srodowisko_resume0.txt +64 -0
  13. K12/run/train.log +167 -0
  14. K12/run/train.out +167 -0
  15. K22/run/args.txt +1 -0
  16. K22/run/metrics.jsonl +238 -0
  17. K22/run/srodowisko.txt +63 -0
  18. K22/run/srodowisko_K22_mini0.txt +63 -0
  19. K22/run/train.log +248 -0
  20. K22/run/train.out +248 -0
  21. K22/run/val_en.json +1 -0
  22. K22/run/val_en.out +1 -0
  23. K22/run/val_pl.json +1 -0
  24. K22/run/val_pl.out +1 -0
  25. K22/run/wynik.json +1 -0
  26. MU2/run/SHA256SUMS_kopiarka +4 -0
  27. MU2/run/args.txt +1 -0
  28. MU2/run/metrics.jsonl +117 -0
  29. MU2/run/srodowisko_MU2_mini0_resume0.txt +64 -0
  30. MU2/run/srodowisko_resume0.txt +64 -0
  31. MU2/run/train.log +129 -0
  32. MU2/run/train.out +129 -0
  33. R1/run/args.txt +1 -0
  34. R1/run/metrics.jsonl +238 -0
  35. R1/run/srodowisko.txt +63 -0
  36. R1/run/srodowisko_R1_mini0.txt +63 -0
  37. R1/run/srodowisko_R1_mini1.txt +63 -0
  38. R1/run/train.log +248 -0
  39. R1/run/train.out +248 -0
  40. R1/run/val_en.json +1 -0
  41. R1/run/val_en.out +1 -0
  42. R1/run/val_pl.json +1 -0
  43. R1/run/val_pl.out +1 -0
  44. R1/run/wynik.json +1 -0
  45. R2/run/args.txt +1 -0
  46. R2/run/metrics.jsonl +238 -0
  47. R2/run/srodowisko.txt +63 -0
  48. R2/run/srodowisko_R2_mini0.txt +63 -0
  49. R2/run/train.log +248 -0
  50. R2/run/train.out +248 -0
GQA/run/SHA256SUMS_kopiarka ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ 8296514387f2f1fbc20f5c61822007daa823f86b541dc64e830b1361358fba6a ckpt_500.pt
2
+ fb4a4a805f6e88204a3a859238567ff28f40c9f0adbf5a8fdf6c07119245dfec ckpt_1000.pt
3
+ 13ddaf2ea734dd720aaa3c55807621d3b66f44ce4d1a58d782970491fafc8699 ckpt_1500.pt
4
+ e8e632635f4bca6f8d4885502d46503c8ac16f2e83d5d1aa65129535215f828e ckpt_2000.pt
5
+ 3366e88771415a893cb14f3d209b7759bde43b7475e594bf8a81a88c68843f2f ckpt_2500.pt
GQA/run/args.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ --data-dir /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2 --out-dir /home/ubuntu/kolektyw/dowody400m/wyniki/GQA --run-id dowody_GQA --vocab 32768 --dtype uint16 --block 4096 --n-layer 18 --n-embd 896 --n-head 14 --kv-heads 2 --value-residual --batch 128 --grad-accum 8 --norm rmsnorm --pos rope --rope-theta 500000 --ffn swiglu --ffn-mult 2.667 --qk-norm --attn-bias 0 --optimizer muon --muon-lr 0.02 --lr 6e-4 --min-lr 6e-05 --wd 0.1 --schedule wsd --decay-frac 0.2 --warmup 500 --act-ckpt --loss-chunk 1024 --sampler random --seed 1 --data-seed 20261022 --compile --log-every 20 --eval-every 4769 --eval-iters 1 --steps 4768 --ckpt-every 500
GQA/run/metrics.jsonl ADDED
@@ -0,0 +1,146 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"step": 20, "loss": 10.382979106903075, "lr": 2.3999999999999997e-05, "tok_s": 40037.62144840001}
2
+ {"step": 40, "loss": 9.56647868156433, "lr": 4.7999999999999994e-05, "tok_s": 55950.47213670447}
3
+ {"step": 60, "loss": 8.688220500946045, "lr": 7.199999999999999e-05, "tok_s": 55956.14811527882}
4
+ {"step": 80, "loss": 8.038008499145509, "lr": 9.599999999999999e-05, "tok_s": 55949.9778076849}
5
+ {"step": 100, "loss": 7.590619421005249, "lr": 0.00011999999999999999, "tok_s": 55935.00502426609}
6
+ {"step": 120, "loss": 7.095160555839539, "lr": 0.00014399999999999998, "tok_s": 55916.81302638947}
7
+ {"step": 140, "loss": 6.665168881416321, "lr": 0.000168, "tok_s": 55909.52022217888}
8
+ {"step": 160, "loss": 6.325761508941651, "lr": 0.00019199999999999998, "tok_s": 55911.599787190855}
9
+ {"step": 180, "loss": 6.088504076004028, "lr": 0.00021599999999999996, "tok_s": 55907.69744272099}
10
+ {"step": 200, "loss": 5.892195367813111, "lr": 0.00023999999999999998, "tok_s": 55900.68233984899}
11
+ {"step": 220, "loss": 5.719494819641113, "lr": 0.00026399999999999997, "tok_s": 55896.15593229689}
12
+ {"step": 240, "loss": 5.564488887786865, "lr": 0.00028799999999999995, "tok_s": 55896.514616477696}
13
+ {"step": 260, "loss": 5.411128854751587, "lr": 0.000312, "tok_s": 55891.311766931656}
14
+ {"step": 280, "loss": 5.264648008346557, "lr": 0.000336, "tok_s": 55890.142672656504}
15
+ {"step": 300, "loss": 5.125132918357849, "lr": 0.00035999999999999997, "tok_s": 55889.63648279308}
16
+ {"step": 320, "loss": 4.998163890838623, "lr": 0.00038399999999999996, "tok_s": 55889.25302894563}
17
+ {"step": 340, "loss": 4.875051689147949, "lr": 0.000408, "tok_s": 55887.41942666463}
18
+ {"step": 360, "loss": 4.771002888679504, "lr": 0.00043199999999999993, "tok_s": 55884.27258425807}
19
+ {"step": 380, "loss": 4.655315661430359, "lr": 0.00045599999999999997, "tok_s": 55884.96934286311}
20
+ {"step": 400, "loss": 4.5465178966522215, "lr": 0.00047999999999999996, "tok_s": 55881.25837321708}
21
+ {"step": 420, "loss": 4.448748826980591, "lr": 0.000504, "tok_s": 55880.59188239282}
22
+ {"step": 440, "loss": 4.359668445587158, "lr": 0.0005279999999999999, "tok_s": 55881.40222428677}
23
+ {"step": 460, "loss": 4.275037050247192, "lr": 0.000552, "tok_s": 55884.09150939727}
24
+ {"step": 480, "loss": 4.2145402193069454, "lr": 0.0005759999999999999, "tok_s": 55873.46532254367}
25
+ {"step": 500, "loss": 4.149071002006531, "lr": 0.0006, "tok_s": 55877.22857218452}
26
+ {"step": 520, "loss": 4.10716700553894, "lr": 0.0006, "tok_s": 55403.22907999532}
27
+ {"step": 540, "loss": 4.052633845806122, "lr": 0.0006, "tok_s": 55892.173277238464}
28
+ {"step": 560, "loss": 3.9963469982147215, "lr": 0.0006, "tok_s": 55878.88699638042}
29
+ {"step": 580, "loss": 3.9511124730110168, "lr": 0.0006, "tok_s": 55869.96262589284}
30
+ {"step": 600, "loss": 3.9063917636871337, "lr": 0.0006, "tok_s": 55865.86760636415}
31
+ {"step": 620, "loss": 3.8814687967300414, "lr": 0.0006, "tok_s": 55861.46000312544}
32
+ {"step": 640, "loss": 3.8532938361167908, "lr": 0.0006, "tok_s": 55856.9303679642}
33
+ {"step": 660, "loss": 3.821173572540283, "lr": 0.0006, "tok_s": 55853.595086182206}
34
+ {"step": 680, "loss": 3.795495903491974, "lr": 0.0006, "tok_s": 55851.20208356764}
35
+ {"step": 700, "loss": 3.76346892118454, "lr": 0.0006, "tok_s": 55864.89045999702}
36
+ {"step": 720, "loss": 3.731121325492859, "lr": 0.0006, "tok_s": 55892.40952421342}
37
+ {"step": 740, "loss": 3.7248404145240785, "lr": 0.0006, "tok_s": 55852.72050783114}
38
+ {"step": 760, "loss": 3.6998636722564697, "lr": 0.0006, "tok_s": 55854.804361771116}
39
+ {"step": 780, "loss": 3.673145031929016, "lr": 0.0006, "tok_s": 55852.95947141054}
40
+ {"step": 800, "loss": 3.6588367223739624, "lr": 0.0006, "tok_s": 55854.218443901394}
41
+ {"step": 820, "loss": 3.6433533906936644, "lr": 0.0006, "tok_s": 55852.26386539599}
42
+ {"step": 840, "loss": 3.627117228507996, "lr": 0.0006, "tok_s": 55855.40802608909}
43
+ {"step": 860, "loss": 3.6044566869735717, "lr": 0.0006, "tok_s": 55855.47066321617}
44
+ {"step": 880, "loss": 3.598527717590332, "lr": 0.0006, "tok_s": 55857.019824091876}
45
+ {"step": 900, "loss": 3.580390286445618, "lr": 0.0006, "tok_s": 55852.60659477567}
46
+ {"step": 920, "loss": 3.5707054495811463, "lr": 0.0006, "tok_s": 55850.399922464516}
47
+ {"step": 940, "loss": 3.5649645924568176, "lr": 0.0006, "tok_s": 55852.27450469675}
48
+ {"step": 960, "loss": 3.5389197945594786, "lr": 0.0006, "tok_s": 55853.04763806798}
49
+ {"step": 980, "loss": 3.5352579712867738, "lr": 0.0006, "tok_s": 55853.98124287446}
50
+ {"step": 1000, "loss": 3.523323690891266, "lr": 0.0006, "tok_s": 55851.88170440031}
51
+ {"step": 1020, "loss": 3.5180077075958254, "lr": 0.0006, "tok_s": 55193.14358495176}
52
+ {"step": 1040, "loss": 3.4907343983650208, "lr": 0.0006, "tok_s": 55861.28688074019}
53
+ {"step": 1060, "loss": 3.501312184333801, "lr": 0.0006, "tok_s": 55862.68153976331}
54
+ {"step": 1080, "loss": 3.490531289577484, "lr": 0.0006, "tok_s": 55864.21165788537}
55
+ {"step": 1100, "loss": 3.474697434902191, "lr": 0.0006, "tok_s": 55862.198694343766}
56
+ {"step": 1120, "loss": 3.4564267992973328, "lr": 0.0006, "tok_s": 55860.18296696393}
57
+ {"step": 1140, "loss": 3.4498842358589172, "lr": 0.0006, "tok_s": 55859.96075629045}
58
+ {"step": 1160, "loss": 3.4431535243988036, "lr": 0.0006, "tok_s": 55860.42667706288}
59
+ {"step": 1180, "loss": 3.4448654890060424, "lr": 0.0006, "tok_s": 55865.909900471794}
60
+ {"step": 1200, "loss": 3.42562221288681, "lr": 0.0006, "tok_s": 55869.89520103725}
61
+ {"step": 1220, "loss": 3.423185706138611, "lr": 0.0006, "tok_s": 55867.896305433605}
62
+ {"step": 1240, "loss": 3.414616274833679, "lr": 0.0006, "tok_s": 55858.96210715924}
63
+ {"step": 1260, "loss": 3.3998213529586794, "lr": 0.0006, "tok_s": 55863.03539518924}
64
+ {"step": 1280, "loss": 3.402288496494293, "lr": 0.0006, "tok_s": 55864.79679198819}
65
+ {"step": 1300, "loss": 3.3960714101791383, "lr": 0.0006, "tok_s": 55864.42751601959}
66
+ {"step": 1320, "loss": 3.3870596051216126, "lr": 0.0006, "tok_s": 55858.42377658899}
67
+ {"step": 1340, "loss": 3.3868780732154846, "lr": 0.0006, "tok_s": 55852.23634509015}
68
+ {"step": 1360, "loss": 3.3778282761573792, "lr": 0.0006, "tok_s": 55851.77190855575}
69
+ {"step": 1380, "loss": 3.369214880466461, "lr": 0.0006, "tok_s": 55856.39157998008}
70
+ {"step": 1400, "loss": 3.3584998488426208, "lr": 0.0006, "tok_s": 55858.81000005289}
71
+ {"step": 1420, "loss": 3.3467327117919923, "lr": 0.0006, "tok_s": 55856.025110774484}
72
+ {"step": 1440, "loss": 3.35287891626358, "lr": 0.0006, "tok_s": 55860.86216782732}
73
+ {"step": 1460, "loss": 3.3505175709724426, "lr": 0.0006, "tok_s": 55860.247885166784}
74
+ {"step": 1480, "loss": 3.354390799999237, "lr": 0.0006, "tok_s": 55859.782960645454}
75
+ {"step": 1500, "loss": 3.3270774483680725, "lr": 0.0006, "tok_s": 55856.53941754428}
76
+ {"step": 1520, "loss": 3.3344152092933657, "lr": 0.0006, "tok_s": 55159.741563164695}
77
+ {"step": 1540, "loss": 3.3191637277603148, "lr": 0.0006, "tok_s": 55858.31296180528}
78
+ {"step": 1560, "loss": 3.3265525102615356, "lr": 0.0006, "tok_s": 55868.37108785107}
79
+ {"step": 1580, "loss": 3.3150747060775756, "lr": 0.0006, "tok_s": 55867.44062125878}
80
+ {"step": 1600, "loss": 3.304818665981293, "lr": 0.0006, "tok_s": 55866.26422258368}
81
+ {"step": 1620, "loss": 3.304558289051056, "lr": 0.0006, "tok_s": 55863.76383918023}
82
+ {"step": 1640, "loss": 3.30103884935379, "lr": 0.0006, "tok_s": 55858.05465581214}
83
+ {"step": 1660, "loss": 3.3019826889038084, "lr": 0.0006, "tok_s": 55855.77065656859}
84
+ {"step": 1680, "loss": 3.290108013153076, "lr": 0.0006, "tok_s": 55858.40660807259}
85
+ {"step": 1700, "loss": 3.2835023045539855, "lr": 0.0006, "tok_s": 55858.18043868478}
86
+ {"step": 1720, "loss": 3.285639023780823, "lr": 0.0006, "tok_s": 55861.454326964646}
87
+ {"step": 1740, "loss": 3.278670835494995, "lr": 0.0006, "tok_s": 55859.09633669302}
88
+ {"step": 1760, "loss": 3.2777636289596557, "lr": 0.0006, "tok_s": 55858.65094117567}
89
+ {"step": 1780, "loss": 3.265216624736786, "lr": 0.0006, "tok_s": 55855.48868117003}
90
+ {"step": 1800, "loss": 3.2684202671051024, "lr": 0.0006, "tok_s": 55849.78998337803}
91
+ {"step": 1820, "loss": 3.2644393920898436, "lr": 0.0006, "tok_s": 55852.98990057944}
92
+ {"step": 1840, "loss": 3.2582288146018983, "lr": 0.0006, "tok_s": 55857.068773244166}
93
+ {"step": 1860, "loss": 3.257483172416687, "lr": 0.0006, "tok_s": 55858.88364135634}
94
+ {"step": 1880, "loss": 3.252480161190033, "lr": 0.0006, "tok_s": 55860.95596454378}
95
+ {"step": 1900, "loss": 3.2468705415725707, "lr": 0.0006, "tok_s": 55868.200335577094}
96
+ {"step": 1920, "loss": 3.2458807229995728, "lr": 0.0006, "tok_s": 55866.75948627773}
97
+ {"step": 1940, "loss": 3.2395505785942076, "lr": 0.0006, "tok_s": 55861.67356951348}
98
+ {"step": 1960, "loss": 3.2437692284584045, "lr": 0.0006, "tok_s": 55861.859749650495}
99
+ {"step": 1980, "loss": 3.2375988006591796, "lr": 0.0006, "tok_s": 55860.48655841409}
100
+ {"step": 2000, "loss": 3.23126403093338, "lr": 0.0006, "tok_s": 55857.7651374327}
101
+ {"step": 2020, "loss": 3.2307679533958433, "lr": 0.0006, "tok_s": 55181.14931078551}
102
+ {"step": 2040, "loss": 3.2255444169044494, "lr": 0.0006, "tok_s": 55855.61686435047}
103
+ {"step": 2060, "loss": 3.225483739376068, "lr": 0.0006, "tok_s": 55853.64168857098}
104
+ {"step": 2080, "loss": 3.213956689834595, "lr": 0.0006, "tok_s": 55850.38105670706}
105
+ {"step": 2100, "loss": 3.222394919395447, "lr": 0.0006, "tok_s": 55847.23533444284}
106
+ {"step": 2120, "loss": 3.210958731174469, "lr": 0.0006, "tok_s": 55849.87991309439}
107
+ {"step": 2140, "loss": 3.2240227699279784, "lr": 0.0006, "tok_s": 55844.74197226356}
108
+ {"step": 2160, "loss": 3.2026231408119203, "lr": 0.0006, "tok_s": 55842.073205979555}
109
+ {"step": 2180, "loss": 3.204519212245941, "lr": 0.0006, "tok_s": 55843.07706555174}
110
+ {"step": 2200, "loss": 3.2058327913284304, "lr": 0.0006, "tok_s": 55847.001525742235}
111
+ {"step": 2220, "loss": 3.1977394461631774, "lr": 0.0006, "tok_s": 55847.026629767315}
112
+ {"step": 2240, "loss": 3.194538748264313, "lr": 0.0006, "tok_s": 55851.65757380692}
113
+ {"step": 2260, "loss": 3.191618466377258, "lr": 0.0006, "tok_s": 55853.358883489906}
114
+ {"step": 2280, "loss": 3.1867557644844053, "lr": 0.0006, "tok_s": 55856.43776140001}
115
+ {"step": 2300, "loss": 3.190177059173584, "lr": 0.0006, "tok_s": 55851.96979672725}
116
+ {"step": 2320, "loss": 3.1867308974266053, "lr": 0.0006, "tok_s": 55851.54799162408}
117
+ {"step": 2340, "loss": 3.182142972946167, "lr": 0.0006, "tok_s": 55849.37934584578}
118
+ {"step": 2360, "loss": 3.172446036338806, "lr": 0.0006, "tok_s": 55850.881570987054}
119
+ {"step": 2380, "loss": 3.171089839935303, "lr": 0.0006, "tok_s": 55851.00037086171}
120
+ {"step": 2400, "loss": 3.1715157270431518, "lr": 0.0006, "tok_s": 55857.74314522562}
121
+ {"step": 2420, "loss": 3.173757755756378, "lr": 0.0006, "tok_s": 55855.25501616669}
122
+ {"step": 2440, "loss": 3.168051612377167, "lr": 0.0006, "tok_s": 55852.09030350863}
123
+ {"step": 2460, "loss": 3.167601001262665, "lr": 0.0006, "tok_s": 55857.334306155106}
124
+ {"step": 2480, "loss": 3.1639285683631897, "lr": 0.0006, "tok_s": 55854.32874590251}
125
+ {"step": 2500, "loss": 3.159174644947052, "lr": 0.0006, "tok_s": 55855.53521578178}
126
+ {"step": 2520, "loss": 3.1603678584098818, "lr": 0.0006, "tok_s": 55177.35615187041}
127
+ {"step": 2540, "loss": 3.1611927270889284, "lr": 0.0006, "tok_s": 55851.00540656905}
128
+ {"step": 2560, "loss": 3.1577532649040223, "lr": 0.0006, "tok_s": 55853.75659890297}
129
+ {"step": 2580, "loss": 3.1542645573616026, "lr": 0.0006, "tok_s": 55855.57040053972}
130
+ {"step": 2600, "loss": 3.144371044635773, "lr": 0.0006, "tok_s": 55855.47924657069}
131
+ {"step": 2620, "loss": 3.15311598777771, "lr": 0.0006, "tok_s": 55850.350559456936}
132
+ {"step": 2640, "loss": 3.153462862968445, "lr": 0.0006, "tok_s": 55850.68312312965}
133
+ {"step": 2660, "loss": 3.152262246608734, "lr": 0.0006, "tok_s": 55850.7300043583}
134
+ {"step": 2680, "loss": 3.1478206515312195, "lr": 0.0006, "tok_s": 55850.78298520614}
135
+ {"step": 2700, "loss": 3.1494753122329713, "lr": 0.0006, "tok_s": 55848.569010462445}
136
+ {"step": 2720, "loss": 3.132279109954834, "lr": 0.0006, "tok_s": 55854.512323446455}
137
+ {"step": 2740, "loss": 3.1343593835830688, "lr": 0.0006, "tok_s": 55856.64653642186}
138
+ {"step": 2760, "loss": 3.1370200872421266, "lr": 0.0006, "tok_s": 55859.93840761366}
139
+ {"step": 2780, "loss": 3.12802734375, "lr": 0.0006, "tok_s": 55860.464351215545}
140
+ {"step": 2800, "loss": 3.123040521144867, "lr": 0.0006, "tok_s": 55865.237743976984}
141
+ {"step": 2820, "loss": 3.1200297117233275, "lr": 0.0006, "tok_s": 55862.59291695877}
142
+ {"step": 2840, "loss": 3.128438580036163, "lr": 0.0006, "tok_s": 55856.25268148453}
143
+ {"step": 2860, "loss": 3.120231294631958, "lr": 0.0006, "tok_s": 55853.544440752}
144
+ {"step": 2880, "loss": 3.122507381439209, "lr": 0.0006, "tok_s": 55854.86458574521}
145
+ {"step": 2900, "loss": 3.123218131065369, "lr": 0.0006, "tok_s": 55852.98663777653}
146
+ {"step": 2920, "loss": 3.1307594418525695, "lr": 0.0006, "tok_s": 55857.744989732644}
GQA/run/srodowisko_GQA_mini0_resume0.txt ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-23T02:33:05Z ARM GQA GPU 2 MINI 0 DATA_SEED 20261022 CKPT 500 RESUME 0 DEST /var/lib/docker/kolektyw/dowody400m_ckpt/wyniki_GQA
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ 0702c32c0ad602d8d12b16539e05cfbc68d1823cfd39b82b984357d78f27f664 /home/ubuntu/kolektyw/dowody400m/kod/ramie_v03.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 341aae81ce8512294dcd7de93ced26053025677b4ab449606be989f7fd898d27 /home/ubuntu/kolektyw/dowody400m/kod/kopiarka_ckpt.sh
7
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
8
+ 2.6.0+cu124 12.4
9
+ accelerate==1.15.0
10
+ annotated-doc==0.0.5
11
+ anyio==4.15.1
12
+ certifi==2026.10.19
13
+ click==8.5.0
14
+ filelock==4.2.4
15
+ fsspec==2026.9.0
16
+ h11==0.16.0
17
+ hf-xet==1.7.0
18
+ httpcore==1.0.9
19
+ httpx==0.28.1
20
+ huggingface_hub==1.33.0
21
+ idna==3.20
22
+ Jinja2==3.1.6
23
+ markdown-it-py==4.2.0
24
+ MarkupSafe==3.0.4
25
+ mdurl==0.1.2
26
+ mpmath==1.3.0
27
+ networkx==3.7
28
+ numpy==2.2.6
29
+ nvidia-cublas-cu12==12.4.5.8
30
+ nvidia-cuda-cupti-cu12==12.4.127
31
+ nvidia-cuda-nvrtc-cu12==12.4.127
32
+ nvidia-cuda-runtime-cu12==12.4.127
33
+ nvidia-cudnn-cu12==9.1.0.70
34
+ nvidia-cufft-cu12==11.2.1.3
35
+ nvidia-curand-cu12==10.3.5.147
36
+ nvidia-cusolver-cu12==11.6.1.9
37
+ nvidia-cusparse-cu12==12.3.1.170
38
+ nvidia-cusparselt-cu12==0.6.2
39
+ nvidia-nccl-cu12==2.21.5
40
+ nvidia-nvjitlink-cu12==12.4.127
41
+ nvidia-nvtx-cu12==12.4.127
42
+ packaging==26.3
43
+ pillow==12.3.0
44
+ psutil==7.2.2
45
+ Pygments==2.21.0
46
+ PyYAML==6.0.3
47
+ regex==2026.9.29
48
+ rich==15.0.0
49
+ safetensors==0.8.0
50
+ setuptools==84.2.0
51
+ shellingham==1.5.4
52
+ sympy==1.13.1
53
+ tokenizers==0.23.3
54
+ torch==2.6.0+cu124
55
+ torchvision==0.21.0+cu124
56
+ tqdm==4.70.1
57
+ transformers==5.17.0
58
+ triton==3.2.0
59
+ typer==0.27.3
60
+ typing_extensions==4.16.0
61
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
64
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
GQA/run/srodowisko_resume0.txt ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-23T02:33:05Z ARM GQA GPU 2 MINI 0 DATA_SEED 20261022 CKPT 500 RESUME 0 DEST /var/lib/docker/kolektyw/dowody400m_ckpt/wyniki_GQA
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ 0702c32c0ad602d8d12b16539e05cfbc68d1823cfd39b82b984357d78f27f664 /home/ubuntu/kolektyw/dowody400m/kod/ramie_v03.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 341aae81ce8512294dcd7de93ced26053025677b4ab449606be989f7fd898d27 /home/ubuntu/kolektyw/dowody400m/kod/kopiarka_ckpt.sh
7
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
8
+ 2.6.0+cu124 12.4
9
+ accelerate==1.15.0
10
+ annotated-doc==0.0.5
11
+ anyio==4.15.1
12
+ certifi==2026.10.19
13
+ click==8.5.0
14
+ filelock==4.2.4
15
+ fsspec==2026.9.0
16
+ h11==0.16.0
17
+ hf-xet==1.7.0
18
+ httpcore==1.0.9
19
+ httpx==0.28.1
20
+ huggingface_hub==1.33.0
21
+ idna==3.20
22
+ Jinja2==3.1.6
23
+ markdown-it-py==4.2.0
24
+ MarkupSafe==3.0.4
25
+ mdurl==0.1.2
26
+ mpmath==1.3.0
27
+ networkx==3.7
28
+ numpy==2.2.6
29
+ nvidia-cublas-cu12==12.4.5.8
30
+ nvidia-cuda-cupti-cu12==12.4.127
31
+ nvidia-cuda-nvrtc-cu12==12.4.127
32
+ nvidia-cuda-runtime-cu12==12.4.127
33
+ nvidia-cudnn-cu12==9.1.0.70
34
+ nvidia-cufft-cu12==11.2.1.3
35
+ nvidia-curand-cu12==10.3.5.147
36
+ nvidia-cusolver-cu12==11.6.1.9
37
+ nvidia-cusparse-cu12==12.3.1.170
38
+ nvidia-cusparselt-cu12==0.6.2
39
+ nvidia-nccl-cu12==2.21.5
40
+ nvidia-nvjitlink-cu12==12.4.127
41
+ nvidia-nvtx-cu12==12.4.127
42
+ packaging==26.3
43
+ pillow==12.3.0
44
+ psutil==7.2.2
45
+ Pygments==2.21.0
46
+ PyYAML==6.0.3
47
+ regex==2026.9.29
48
+ rich==15.0.0
49
+ safetensors==0.8.0
50
+ setuptools==84.2.0
51
+ shellingham==1.5.4
52
+ sympy==1.13.1
53
+ tokenizers==0.23.3
54
+ torch==2.6.0+cu124
55
+ torchvision==0.21.0+cu124
56
+ tqdm==4.70.1
57
+ transformers==5.17.0
58
+ triton==3.2.0
59
+ typer==0.27.3
60
+ typing_extensions==4.16.0
61
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
64
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
GQA/run/train.log ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [04:33:11] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [04:33:11] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [04:33:13] model GPT-ref: 178.1M param (L18 d896 h14 kv2)
4
+ [04:33:14] optimizer=muon muon_lr=0.02 (mult=33.3x)
5
+ [04:33:14] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [04:33:14] r13: mixer_layout=AAAAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [04:33:14] torch.compile enabled
8
+ [04:33:14] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [04:37:36] step 20/4768 loss 10.3830 lr 2.40e-05 gnorm 2.93 40,038 tok/s peakVRAM 13.3GB
10
+ [04:40:44] step 40/4768 loss 9.5665 lr 4.80e-05 gnorm 1.43 55,950 tok/s peakVRAM 13.3GB
11
+ [04:43:51] step 60/4768 loss 8.6882 lr 7.20e-05 gnorm 0.83 55,956 tok/s peakVRAM 13.3GB
12
+ [04:46:58] step 80/4768 loss 8.0380 lr 9.60e-05 gnorm 1.30 55,950 tok/s peakVRAM 13.3GB
13
+ [04:50:06] step 100/4768 loss 7.5906 lr 1.20e-04 gnorm 1.02 55,935 tok/s peakVRAM 13.3GB
14
+ [04:53:13] step 120/4768 loss 7.0952 lr 1.44e-04 gnorm 0.93 55,917 tok/s peakVRAM 13.3GB
15
+ [04:56:21] step 140/4768 loss 6.6652 lr 1.68e-04 gnorm 0.70 55,910 tok/s peakVRAM 13.3GB
16
+ [04:59:28] step 160/4768 loss 6.3258 lr 1.92e-04 gnorm 0.73 55,912 tok/s peakVRAM 13.3GB
17
+ [05:02:36] step 180/4768 loss 6.0885 lr 2.16e-04 gnorm 0.78 55,908 tok/s peakVRAM 13.3GB
18
+ [05:05:44] step 200/4768 loss 5.8922 lr 2.40e-04 gnorm 0.68 55,901 tok/s peakVRAM 13.3GB
19
+ [05:08:51] step 220/4768 loss 5.7195 lr 2.64e-04 gnorm 0.78 55,896 tok/s peakVRAM 13.3GB
20
+ [05:11:59] step 240/4768 loss 5.5645 lr 2.88e-04 gnorm 0.84 55,897 tok/s peakVRAM 13.3GB
21
+ [05:15:06] step 260/4768 loss 5.4111 lr 3.12e-04 gnorm 0.72 55,891 tok/s peakVRAM 13.3GB
22
+ [05:18:14] step 280/4768 loss 5.2646 lr 3.36e-04 gnorm 0.95 55,890 tok/s peakVRAM 13.3GB
23
+ [05:21:22] step 300/4768 loss 5.1251 lr 3.60e-04 gnorm 0.78 55,890 tok/s peakVRAM 13.3GB
24
+ [05:24:29] step 320/4768 loss 4.9982 lr 3.84e-04 gnorm 0.77 55,889 tok/s peakVRAM 13.3GB
25
+ [05:27:37] step 340/4768 loss 4.8751 lr 4.08e-04 gnorm 0.92 55,887 tok/s peakVRAM 13.3GB
26
+ [05:30:44] step 360/4768 loss 4.7710 lr 4.32e-04 gnorm 0.59 55,884 tok/s peakVRAM 13.3GB
27
+ [05:33:52] step 380/4768 loss 4.6553 lr 4.56e-04 gnorm 0.80 55,885 tok/s peakVRAM 13.3GB
28
+ [05:37:00] step 400/4768 loss 4.5465 lr 4.80e-04 gnorm 0.86 55,881 tok/s peakVRAM 13.3GB
29
+ [05:40:07] step 420/4768 loss 4.4487 lr 5.04e-04 gnorm 0.73 55,881 tok/s peakVRAM 13.3GB
30
+ [05:43:15] step 440/4768 loss 4.3597 lr 5.28e-04 gnorm 0.71 55,881 tok/s peakVRAM 13.3GB
31
+ [05:46:23] step 460/4768 loss 4.2750 lr 5.52e-04 gnorm 0.74 55,884 tok/s peakVRAM 13.3GB
32
+ [05:49:30] step 480/4768 loss 4.2145 lr 5.76e-04 gnorm 0.55 55,873 tok/s peakVRAM 13.3GB
33
+ [05:52:38] step 500/4768 loss 4.1491 lr 6.00e-04 gnorm 0.72 55,877 tok/s peakVRAM 13.3GB
34
+ [05:52:40] ckpt @ 500
35
+ [05:55:47] step 520/4768 loss 4.1072 lr 6.00e-04 gnorm 0.50 55,403 tok/s peakVRAM 13.3GB
36
+ [05:58:55] step 540/4768 loss 4.0526 lr 6.00e-04 gnorm 0.56 55,892 tok/s peakVRAM 13.3GB
37
+ [06:02:03] step 560/4768 loss 3.9963 lr 6.00e-04 gnorm 0.45 55,879 tok/s peakVRAM 13.3GB
38
+ [06:05:10] step 580/4768 loss 3.9511 lr 6.00e-04 gnorm 0.48 55,870 tok/s peakVRAM 13.3GB
39
+ [06:08:18] step 600/4768 loss 3.9064 lr 6.00e-04 gnorm 0.47 55,866 tok/s peakVRAM 13.3GB
40
+ [06:11:26] step 620/4768 loss 3.8815 lr 6.00e-04 gnorm 0.41 55,861 tok/s peakVRAM 13.3GB
41
+ [06:14:33] step 640/4768 loss 3.8533 lr 6.00e-04 gnorm 0.38 55,857 tok/s peakVRAM 13.3GB
42
+ [06:17:41] step 660/4768 loss 3.8212 lr 6.00e-04 gnorm 0.31 55,854 tok/s peakVRAM 13.3GB
43
+ [06:20:49] step 680/4768 loss 3.7955 lr 6.00e-04 gnorm 0.35 55,851 tok/s peakVRAM 13.3GB
44
+ [06:23:57] step 700/4768 loss 3.7635 lr 6.00e-04 gnorm 0.50 55,865 tok/s peakVRAM 13.3GB
45
+ [06:27:04] step 720/4768 loss 3.7311 lr 6.00e-04 gnorm 0.39 55,892 tok/s peakVRAM 13.3GB
46
+ [06:30:12] step 740/4768 loss 3.7248 lr 6.00e-04 gnorm 0.31 55,853 tok/s peakVRAM 13.3GB
47
+ [06:33:20] step 760/4768 loss 3.6999 lr 6.00e-04 gnorm 0.39 55,855 tok/s peakVRAM 13.3GB
48
+ [06:36:27] step 780/4768 loss 3.6731 lr 6.00e-04 gnorm 0.32 55,853 tok/s peakVRAM 13.3GB
49
+ [06:39:35] step 800/4768 loss 3.6588 lr 6.00e-04 gnorm 0.30 55,854 tok/s peakVRAM 13.3GB
50
+ [06:42:43] step 820/4768 loss 3.6434 lr 6.00e-04 gnorm 0.33 55,852 tok/s peakVRAM 13.3GB
51
+ [06:45:51] step 840/4768 loss 3.6271 lr 6.00e-04 gnorm 0.34 55,855 tok/s peakVRAM 13.3GB
52
+ [06:48:58] step 860/4768 loss 3.6045 lr 6.00e-04 gnorm 0.26 55,855 tok/s peakVRAM 13.3GB
53
+ [06:52:06] step 880/4768 loss 3.5985 lr 6.00e-04 gnorm 0.24 55,857 tok/s peakVRAM 13.3GB
54
+ [06:55:14] step 900/4768 loss 3.5804 lr 6.00e-04 gnorm 0.36 55,853 tok/s peakVRAM 13.3GB
55
+ [06:58:22] step 920/4768 loss 3.5707 lr 6.00e-04 gnorm 0.28 55,850 tok/s peakVRAM 13.3GB
56
+ [07:01:29] step 940/4768 loss 3.5650 lr 6.00e-04 gnorm 0.26 55,852 tok/s peakVRAM 13.3GB
57
+ [07:04:37] step 960/4768 loss 3.5389 lr 6.00e-04 gnorm 0.27 55,853 tok/s peakVRAM 13.3GB
58
+ [07:07:45] step 980/4768 loss 3.5353 lr 6.00e-04 gnorm 0.22 55,854 tok/s peakVRAM 13.3GB
59
+ [07:10:52] step 1000/4768 loss 3.5233 lr 6.00e-04 gnorm 0.27 55,852 tok/s peakVRAM 13.3GB
60
+ [07:10:55] ckpt @ 1000
61
+ [07:14:02] step 1020/4768 loss 3.5180 lr 6.00e-04 gnorm 0.25 55,193 tok/s peakVRAM 13.3GB
62
+ [07:17:10] step 1040/4768 loss 3.4907 lr 6.00e-04 gnorm 0.24 55,861 tok/s peakVRAM 13.3GB
63
+ [07:20:18] step 1060/4768 loss 3.5013 lr 6.00e-04 gnorm 0.23 55,863 tok/s peakVRAM 13.3GB
64
+ [07:23:26] step 1080/4768 loss 3.4905 lr 6.00e-04 gnorm 0.31 55,864 tok/s peakVRAM 13.3GB
65
+ [07:26:33] step 1100/4768 loss 3.4747 lr 6.00e-04 gnorm 0.25 55,862 tok/s peakVRAM 13.3GB
66
+ [07:29:41] step 1120/4768 loss 3.4564 lr 6.00e-04 gnorm 0.22 55,860 tok/s peakVRAM 13.3GB
67
+ [07:32:49] step 1140/4768 loss 3.4499 lr 6.00e-04 gnorm 0.26 55,860 tok/s peakVRAM 13.3GB
68
+ [07:35:56] step 1160/4768 loss 3.4432 lr 6.00e-04 gnorm 0.21 55,860 tok/s peakVRAM 13.3GB
69
+ [07:39:04] step 1180/4768 loss 3.4449 lr 6.00e-04 gnorm 0.23 55,866 tok/s peakVRAM 13.3GB
70
+ [07:42:12] step 1200/4768 loss 3.4256 lr 6.00e-04 gnorm 0.21 55,870 tok/s peakVRAM 13.3GB
71
+ [07:45:19] step 1220/4768 loss 3.4232 lr 6.00e-04 gnorm 0.24 55,868 tok/s peakVRAM 13.3GB
72
+ [07:48:27] step 1240/4768 loss 3.4146 lr 6.00e-04 gnorm 0.21 55,859 tok/s peakVRAM 13.3GB
73
+ [07:51:35] step 1260/4768 loss 3.3998 lr 6.00e-04 gnorm 0.21 55,863 tok/s peakVRAM 13.3GB
74
+ [07:54:43] step 1280/4768 loss 3.4023 lr 6.00e-04 gnorm 0.18 55,865 tok/s peakVRAM 13.3GB
75
+ [07:57:50] step 1300/4768 loss 3.3961 lr 6.00e-04 gnorm 0.25 55,864 tok/s peakVRAM 13.3GB
76
+ [08:00:58] step 1320/4768 loss 3.3871 lr 6.00e-04 gnorm 0.20 55,858 tok/s peakVRAM 13.3GB
77
+ [08:04:06] step 1340/4768 loss 3.3869 lr 6.00e-04 gnorm 0.21 55,852 tok/s peakVRAM 13.3GB
78
+ [08:07:14] step 1360/4768 loss 3.3778 lr 6.00e-04 gnorm 0.21 55,852 tok/s peakVRAM 13.3GB
79
+ [08:10:21] step 1380/4768 loss 3.3692 lr 6.00e-04 gnorm 0.20 55,856 tok/s peakVRAM 13.3GB
80
+ [08:13:29] step 1400/4768 loss 3.3585 lr 6.00e-04 gnorm 0.18 55,859 tok/s peakVRAM 13.3GB
81
+ [08:16:37] step 1420/4768 loss 3.3467 lr 6.00e-04 gnorm 0.21 55,856 tok/s peakVRAM 13.3GB
82
+ [08:19:44] step 1440/4768 loss 3.3529 lr 6.00e-04 gnorm 0.19 55,861 tok/s peakVRAM 13.3GB
83
+ [08:22:52] step 1460/4768 loss 3.3505 lr 6.00e-04 gnorm 0.18 55,860 tok/s peakVRAM 13.3GB
84
+ [08:26:00] step 1480/4768 loss 3.3544 lr 6.00e-04 gnorm 0.22 55,860 tok/s peakVRAM 13.3GB
85
+ [08:29:08] step 1500/4768 loss 3.3271 lr 6.00e-04 gnorm 0.18 55,857 tok/s peakVRAM 13.3GB
86
+ [08:29:10] ckpt @ 1500
87
+ [08:32:18] step 1520/4768 loss 3.3344 lr 6.00e-04 gnorm 0.19 55,160 tok/s peakVRAM 13.3GB
88
+ [08:35:25] step 1540/4768 loss 3.3192 lr 6.00e-04 gnorm 0.23 55,858 tok/s peakVRAM 13.3GB
89
+ [08:38:33] step 1560/4768 loss 3.3266 lr 6.00e-04 gnorm 0.19 55,868 tok/s peakVRAM 13.3GB
90
+ [08:41:41] step 1580/4768 loss 3.3151 lr 6.00e-04 gnorm 0.17 55,867 tok/s peakVRAM 13.3GB
91
+ [08:44:48] step 1600/4768 loss 3.3048 lr 6.00e-04 gnorm 0.19 55,866 tok/s peakVRAM 13.3GB
92
+ [08:47:56] step 1620/4768 loss 3.3046 lr 6.00e-04 gnorm 0.16 55,864 tok/s peakVRAM 13.3GB
93
+ [08:51:04] step 1640/4768 loss 3.3010 lr 6.00e-04 gnorm 0.17 55,858 tok/s peakVRAM 13.3GB
94
+ [08:54:12] step 1660/4768 loss 3.3020 lr 6.00e-04 gnorm 0.18 55,856 tok/s peakVRAM 13.3GB
95
+ [08:57:19] step 1680/4768 loss 3.2901 lr 6.00e-04 gnorm 0.18 55,858 tok/s peakVRAM 13.3GB
96
+ [09:00:27] step 1700/4768 loss 3.2835 lr 6.00e-04 gnorm 0.17 55,858 tok/s peakVRAM 13.3GB
97
+ [09:03:35] step 1720/4768 loss 3.2856 lr 6.00e-04 gnorm 0.18 55,861 tok/s peakVRAM 13.3GB
98
+ [09:06:42] step 1740/4768 loss 3.2787 lr 6.00e-04 gnorm 0.16 55,859 tok/s peakVRAM 13.3GB
99
+ [09:09:50] step 1760/4768 loss 3.2778 lr 6.00e-04 gnorm 0.16 55,859 tok/s peakVRAM 13.3GB
100
+ [09:12:58] step 1780/4768 loss 3.2652 lr 6.00e-04 gnorm 0.18 55,855 tok/s peakVRAM 13.3GB
101
+ [09:16:06] step 1800/4768 loss 3.2684 lr 6.00e-04 gnorm 0.15 55,850 tok/s peakVRAM 13.3GB
102
+ [09:19:13] step 1820/4768 loss 3.2644 lr 6.00e-04 gnorm 0.19 55,853 tok/s peakVRAM 13.3GB
103
+ [09:22:21] step 1840/4768 loss 3.2582 lr 6.00e-04 gnorm 0.16 55,857 tok/s peakVRAM 13.3GB
104
+ [09:25:29] step 1860/4768 loss 3.2575 lr 6.00e-04 gnorm 0.17 55,859 tok/s peakVRAM 13.3GB
105
+ [09:28:37] step 1880/4768 loss 3.2525 lr 6.00e-04 gnorm 0.15 55,861 tok/s peakVRAM 13.3GB
106
+ [09:31:44] step 1900/4768 loss 3.2469 lr 6.00e-04 gnorm 0.17 55,868 tok/s peakVRAM 13.3GB
107
+ [09:34:52] step 1920/4768 loss 3.2459 lr 6.00e-04 gnorm 0.15 55,867 tok/s peakVRAM 13.3GB
108
+ [09:38:00] step 1940/4768 loss 3.2396 lr 6.00e-04 gnorm 0.15 55,862 tok/s peakVRAM 13.3GB
109
+ [09:41:07] step 1960/4768 loss 3.2438 lr 6.00e-04 gnorm 0.18 55,862 tok/s peakVRAM 13.3GB
110
+ [09:44:15] step 1980/4768 loss 3.2376 lr 6.00e-04 gnorm 0.15 55,860 tok/s peakVRAM 13.3GB
111
+ [09:47:23] step 2000/4768 loss 3.2313 lr 6.00e-04 gnorm 0.17 55,858 tok/s peakVRAM 13.3GB
112
+ [09:47:25] ckpt @ 2000
113
+ [09:50:33] step 2020/4768 loss 3.2308 lr 6.00e-04 gnorm 0.15 55,181 tok/s peakVRAM 13.3GB
114
+ [09:53:41] step 2040/4768 loss 3.2255 lr 6.00e-04 gnorm 0.15 55,856 tok/s peakVRAM 13.3GB
115
+ [09:56:48] step 2060/4768 loss 3.2255 lr 6.00e-04 gnorm 0.16 55,854 tok/s peakVRAM 13.3GB
116
+ [09:59:56] step 2080/4768 loss 3.2140 lr 6.00e-04 gnorm 0.16 55,850 tok/s peakVRAM 13.3GB
117
+ [10:03:04] step 2100/4768 loss 3.2224 lr 6.00e-04 gnorm 0.15 55,847 tok/s peakVRAM 13.3GB
118
+ [10:06:12] step 2120/4768 loss 3.2110 lr 6.00e-04 gnorm 0.14 55,850 tok/s peakVRAM 13.3GB
119
+ [10:09:19] step 2140/4768 loss 3.2240 lr 6.00e-04 gnorm 0.15 55,845 tok/s peakVRAM 13.3GB
120
+ [10:12:27] step 2160/4768 loss 3.2026 lr 6.00e-04 gnorm 0.14 55,842 tok/s peakVRAM 13.3GB
121
+ [10:15:35] step 2180/4768 loss 3.2045 lr 6.00e-04 gnorm 0.16 55,843 tok/s peakVRAM 13.3GB
122
+ [10:18:43] step 2200/4768 loss 3.2058 lr 6.00e-04 gnorm 0.15 55,847 tok/s peakVRAM 13.3GB
123
+ [10:21:50] step 2220/4768 loss 3.1977 lr 6.00e-04 gnorm 0.14 55,847 tok/s peakVRAM 13.3GB
124
+ [10:24:58] step 2240/4768 loss 3.1945 lr 6.00e-04 gnorm 0.15 55,852 tok/s peakVRAM 13.3GB
125
+ [10:28:06] step 2260/4768 loss 3.1916 lr 6.00e-04 gnorm 0.14 55,853 tok/s peakVRAM 13.3GB
126
+ [10:31:14] step 2280/4768 loss 3.1868 lr 6.00e-04 gnorm 0.16 55,856 tok/s peakVRAM 13.3GB
127
+ [10:34:21] step 2300/4768 loss 3.1902 lr 6.00e-04 gnorm 0.14 55,852 tok/s peakVRAM 13.3GB
128
+ [10:37:29] step 2320/4768 loss 3.1867 lr 6.00e-04 gnorm 0.13 55,852 tok/s peakVRAM 13.3GB
129
+ [10:40:37] step 2340/4768 loss 3.1821 lr 6.00e-04 gnorm 0.14 55,849 tok/s peakVRAM 13.3GB
130
+ [10:43:45] step 2360/4768 loss 3.1724 lr 6.00e-04 gnorm 0.15 55,851 tok/s peakVRAM 13.3GB
131
+ [10:46:52] step 2380/4768 loss 3.1711 lr 6.00e-04 gnorm 0.12 55,851 tok/s peakVRAM 13.3GB
132
+ [10:50:00] step 2400/4768 loss 3.1715 lr 6.00e-04 gnorm 0.12 55,858 tok/s peakVRAM 13.3GB
133
+ [10:53:08] step 2420/4768 loss 3.1738 lr 6.00e-04 gnorm 0.14 55,855 tok/s peakVRAM 13.3GB
134
+ [10:56:16] step 2440/4768 loss 3.1681 lr 6.00e-04 gnorm 0.13 55,852 tok/s peakVRAM 13.3GB
135
+ [10:59:23] step 2460/4768 loss 3.1676 lr 6.00e-04 gnorm 0.17 55,857 tok/s peakVRAM 13.3GB
136
+ [11:02:31] step 2480/4768 loss 3.1639 lr 6.00e-04 gnorm 0.15 55,854 tok/s peakVRAM 13.3GB
137
+ [11:05:39] step 2500/4768 loss 3.1592 lr 6.00e-04 gnorm 0.17 55,856 tok/s peakVRAM 13.3GB
138
+ [11:05:41] ckpt @ 2500
139
+ [11:08:49] step 2520/4768 loss 3.1604 lr 6.00e-04 gnorm 0.14 55,177 tok/s peakVRAM 13.3GB
140
+ [11:11:56] step 2540/4768 loss 3.1612 lr 6.00e-04 gnorm 0.15 55,851 tok/s peakVRAM 13.3GB
141
+ [11:15:04] step 2560/4768 loss 3.1578 lr 6.00e-04 gnorm 0.13 55,854 tok/s peakVRAM 13.3GB
142
+ [11:18:12] step 2580/4768 loss 3.1543 lr 6.00e-04 gnorm 0.13 55,856 tok/s peakVRAM 13.3GB
143
+ [11:21:20] step 2600/4768 loss 3.1444 lr 6.00e-04 gnorm 0.14 55,855 tok/s peakVRAM 13.3GB
144
+ [11:24:27] step 2620/4768 loss 3.1531 lr 6.00e-04 gnorm 0.13 55,850 tok/s peakVRAM 13.3GB
145
+ [11:27:35] step 2640/4768 loss 3.1535 lr 6.00e-04 gnorm 0.13 55,851 tok/s peakVRAM 13.3GB
146
+ [11:30:43] step 2660/4768 loss 3.1523 lr 6.00e-04 gnorm 0.13 55,851 tok/s peakVRAM 13.3GB
147
+ [11:33:51] step 2680/4768 loss 3.1478 lr 6.00e-04 gnorm 0.15 55,851 tok/s peakVRAM 13.3GB
148
+ [11:36:58] step 2700/4768 loss 3.1495 lr 6.00e-04 gnorm 0.13 55,849 tok/s peakVRAM 13.3GB
149
+ [11:40:06] step 2720/4768 loss 3.1323 lr 6.00e-04 gnorm 0.13 55,855 tok/s peakVRAM 13.3GB
150
+ [11:43:14] step 2740/4768 loss 3.1344 lr 6.00e-04 gnorm 0.11 55,857 tok/s peakVRAM 13.3GB
151
+ [11:46:22] step 2760/4768 loss 3.1370 lr 6.00e-04 gnorm 0.13 55,860 tok/s peakVRAM 13.3GB
152
+ [11:49:29] step 2780/4768 loss 3.1280 lr 6.00e-04 gnorm 0.15 55,860 tok/s peakVRAM 13.3GB
153
+ [11:52:37] step 2800/4768 loss 3.1230 lr 6.00e-04 gnorm 0.12 55,865 tok/s peakVRAM 13.3GB
154
+ [11:55:45] step 2820/4768 loss 3.1200 lr 6.00e-04 gnorm 0.13 55,863 tok/s peakVRAM 13.3GB
155
+ [11:58:52] step 2840/4768 loss 3.1284 lr 6.00e-04 gnorm 0.13 55,856 tok/s peakVRAM 13.3GB
156
+ [12:02:00] step 2860/4768 loss 3.1202 lr 6.00e-04 gnorm 0.13 55,854 tok/s peakVRAM 13.3GB
157
+ [12:05:08] step 2880/4768 loss 3.1225 lr 6.00e-04 gnorm 0.14 55,855 tok/s peakVRAM 13.3GB
158
+ [12:08:16] step 2900/4768 loss 3.1232 lr 6.00e-04 gnorm 0.12 55,853 tok/s peakVRAM 13.3GB
159
+ [12:11:23] step 2920/4768 loss 3.1308 lr 6.00e-04 gnorm 0.12 55,858 tok/s peakVRAM 13.3GB
GQA/run/train.out ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [04:33:11] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [04:33:11] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [04:33:13] model GPT-ref: 178.1M param (L18 d896 h14 kv2)
4
+ [04:33:14] optimizer=muon muon_lr=0.02 (mult=33.3x)
5
+ [04:33:14] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [04:33:14] r13: mixer_layout=AAAAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [04:33:14] torch.compile enabled
8
+ [04:33:14] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [04:37:36] step 20/4768 loss 10.3830 lr 2.40e-05 gnorm 2.93 40,038 tok/s peakVRAM 13.3GB
10
+ [04:40:44] step 40/4768 loss 9.5665 lr 4.80e-05 gnorm 1.43 55,950 tok/s peakVRAM 13.3GB
11
+ [04:43:51] step 60/4768 loss 8.6882 lr 7.20e-05 gnorm 0.83 55,956 tok/s peakVRAM 13.3GB
12
+ [04:46:58] step 80/4768 loss 8.0380 lr 9.60e-05 gnorm 1.30 55,950 tok/s peakVRAM 13.3GB
13
+ [04:50:06] step 100/4768 loss 7.5906 lr 1.20e-04 gnorm 1.02 55,935 tok/s peakVRAM 13.3GB
14
+ [04:53:13] step 120/4768 loss 7.0952 lr 1.44e-04 gnorm 0.93 55,917 tok/s peakVRAM 13.3GB
15
+ [04:56:21] step 140/4768 loss 6.6652 lr 1.68e-04 gnorm 0.70 55,910 tok/s peakVRAM 13.3GB
16
+ [04:59:28] step 160/4768 loss 6.3258 lr 1.92e-04 gnorm 0.73 55,912 tok/s peakVRAM 13.3GB
17
+ [05:02:36] step 180/4768 loss 6.0885 lr 2.16e-04 gnorm 0.78 55,908 tok/s peakVRAM 13.3GB
18
+ [05:05:44] step 200/4768 loss 5.8922 lr 2.40e-04 gnorm 0.68 55,901 tok/s peakVRAM 13.3GB
19
+ [05:08:51] step 220/4768 loss 5.7195 lr 2.64e-04 gnorm 0.78 55,896 tok/s peakVRAM 13.3GB
20
+ [05:11:59] step 240/4768 loss 5.5645 lr 2.88e-04 gnorm 0.84 55,897 tok/s peakVRAM 13.3GB
21
+ [05:15:06] step 260/4768 loss 5.4111 lr 3.12e-04 gnorm 0.72 55,891 tok/s peakVRAM 13.3GB
22
+ [05:18:14] step 280/4768 loss 5.2646 lr 3.36e-04 gnorm 0.95 55,890 tok/s peakVRAM 13.3GB
23
+ [05:21:22] step 300/4768 loss 5.1251 lr 3.60e-04 gnorm 0.78 55,890 tok/s peakVRAM 13.3GB
24
+ [05:24:29] step 320/4768 loss 4.9982 lr 3.84e-04 gnorm 0.77 55,889 tok/s peakVRAM 13.3GB
25
+ [05:27:37] step 340/4768 loss 4.8751 lr 4.08e-04 gnorm 0.92 55,887 tok/s peakVRAM 13.3GB
26
+ [05:30:44] step 360/4768 loss 4.7710 lr 4.32e-04 gnorm 0.59 55,884 tok/s peakVRAM 13.3GB
27
+ [05:33:52] step 380/4768 loss 4.6553 lr 4.56e-04 gnorm 0.80 55,885 tok/s peakVRAM 13.3GB
28
+ [05:37:00] step 400/4768 loss 4.5465 lr 4.80e-04 gnorm 0.86 55,881 tok/s peakVRAM 13.3GB
29
+ [05:40:07] step 420/4768 loss 4.4487 lr 5.04e-04 gnorm 0.73 55,881 tok/s peakVRAM 13.3GB
30
+ [05:43:15] step 440/4768 loss 4.3597 lr 5.28e-04 gnorm 0.71 55,881 tok/s peakVRAM 13.3GB
31
+ [05:46:23] step 460/4768 loss 4.2750 lr 5.52e-04 gnorm 0.74 55,884 tok/s peakVRAM 13.3GB
32
+ [05:49:30] step 480/4768 loss 4.2145 lr 5.76e-04 gnorm 0.55 55,873 tok/s peakVRAM 13.3GB
33
+ [05:52:38] step 500/4768 loss 4.1491 lr 6.00e-04 gnorm 0.72 55,877 tok/s peakVRAM 13.3GB
34
+ [05:52:40] ckpt @ 500
35
+ [05:55:47] step 520/4768 loss 4.1072 lr 6.00e-04 gnorm 0.50 55,403 tok/s peakVRAM 13.3GB
36
+ [05:58:55] step 540/4768 loss 4.0526 lr 6.00e-04 gnorm 0.56 55,892 tok/s peakVRAM 13.3GB
37
+ [06:02:03] step 560/4768 loss 3.9963 lr 6.00e-04 gnorm 0.45 55,879 tok/s peakVRAM 13.3GB
38
+ [06:05:10] step 580/4768 loss 3.9511 lr 6.00e-04 gnorm 0.48 55,870 tok/s peakVRAM 13.3GB
39
+ [06:08:18] step 600/4768 loss 3.9064 lr 6.00e-04 gnorm 0.47 55,866 tok/s peakVRAM 13.3GB
40
+ [06:11:26] step 620/4768 loss 3.8815 lr 6.00e-04 gnorm 0.41 55,861 tok/s peakVRAM 13.3GB
41
+ [06:14:33] step 640/4768 loss 3.8533 lr 6.00e-04 gnorm 0.38 55,857 tok/s peakVRAM 13.3GB
42
+ [06:17:41] step 660/4768 loss 3.8212 lr 6.00e-04 gnorm 0.31 55,854 tok/s peakVRAM 13.3GB
43
+ [06:20:49] step 680/4768 loss 3.7955 lr 6.00e-04 gnorm 0.35 55,851 tok/s peakVRAM 13.3GB
44
+ [06:23:57] step 700/4768 loss 3.7635 lr 6.00e-04 gnorm 0.50 55,865 tok/s peakVRAM 13.3GB
45
+ [06:27:04] step 720/4768 loss 3.7311 lr 6.00e-04 gnorm 0.39 55,892 tok/s peakVRAM 13.3GB
46
+ [06:30:12] step 740/4768 loss 3.7248 lr 6.00e-04 gnorm 0.31 55,853 tok/s peakVRAM 13.3GB
47
+ [06:33:20] step 760/4768 loss 3.6999 lr 6.00e-04 gnorm 0.39 55,855 tok/s peakVRAM 13.3GB
48
+ [06:36:27] step 780/4768 loss 3.6731 lr 6.00e-04 gnorm 0.32 55,853 tok/s peakVRAM 13.3GB
49
+ [06:39:35] step 800/4768 loss 3.6588 lr 6.00e-04 gnorm 0.30 55,854 tok/s peakVRAM 13.3GB
50
+ [06:42:43] step 820/4768 loss 3.6434 lr 6.00e-04 gnorm 0.33 55,852 tok/s peakVRAM 13.3GB
51
+ [06:45:51] step 840/4768 loss 3.6271 lr 6.00e-04 gnorm 0.34 55,855 tok/s peakVRAM 13.3GB
52
+ [06:48:58] step 860/4768 loss 3.6045 lr 6.00e-04 gnorm 0.26 55,855 tok/s peakVRAM 13.3GB
53
+ [06:52:06] step 880/4768 loss 3.5985 lr 6.00e-04 gnorm 0.24 55,857 tok/s peakVRAM 13.3GB
54
+ [06:55:14] step 900/4768 loss 3.5804 lr 6.00e-04 gnorm 0.36 55,853 tok/s peakVRAM 13.3GB
55
+ [06:58:22] step 920/4768 loss 3.5707 lr 6.00e-04 gnorm 0.28 55,850 tok/s peakVRAM 13.3GB
56
+ [07:01:29] step 940/4768 loss 3.5650 lr 6.00e-04 gnorm 0.26 55,852 tok/s peakVRAM 13.3GB
57
+ [07:04:37] step 960/4768 loss 3.5389 lr 6.00e-04 gnorm 0.27 55,853 tok/s peakVRAM 13.3GB
58
+ [07:07:45] step 980/4768 loss 3.5353 lr 6.00e-04 gnorm 0.22 55,854 tok/s peakVRAM 13.3GB
59
+ [07:10:52] step 1000/4768 loss 3.5233 lr 6.00e-04 gnorm 0.27 55,852 tok/s peakVRAM 13.3GB
60
+ [07:10:55] ckpt @ 1000
61
+ [07:14:02] step 1020/4768 loss 3.5180 lr 6.00e-04 gnorm 0.25 55,193 tok/s peakVRAM 13.3GB
62
+ [07:17:10] step 1040/4768 loss 3.4907 lr 6.00e-04 gnorm 0.24 55,861 tok/s peakVRAM 13.3GB
63
+ [07:20:18] step 1060/4768 loss 3.5013 lr 6.00e-04 gnorm 0.23 55,863 tok/s peakVRAM 13.3GB
64
+ [07:23:26] step 1080/4768 loss 3.4905 lr 6.00e-04 gnorm 0.31 55,864 tok/s peakVRAM 13.3GB
65
+ [07:26:33] step 1100/4768 loss 3.4747 lr 6.00e-04 gnorm 0.25 55,862 tok/s peakVRAM 13.3GB
66
+ [07:29:41] step 1120/4768 loss 3.4564 lr 6.00e-04 gnorm 0.22 55,860 tok/s peakVRAM 13.3GB
67
+ [07:32:49] step 1140/4768 loss 3.4499 lr 6.00e-04 gnorm 0.26 55,860 tok/s peakVRAM 13.3GB
68
+ [07:35:56] step 1160/4768 loss 3.4432 lr 6.00e-04 gnorm 0.21 55,860 tok/s peakVRAM 13.3GB
69
+ [07:39:04] step 1180/4768 loss 3.4449 lr 6.00e-04 gnorm 0.23 55,866 tok/s peakVRAM 13.3GB
70
+ [07:42:12] step 1200/4768 loss 3.4256 lr 6.00e-04 gnorm 0.21 55,870 tok/s peakVRAM 13.3GB
71
+ [07:45:19] step 1220/4768 loss 3.4232 lr 6.00e-04 gnorm 0.24 55,868 tok/s peakVRAM 13.3GB
72
+ [07:48:27] step 1240/4768 loss 3.4146 lr 6.00e-04 gnorm 0.21 55,859 tok/s peakVRAM 13.3GB
73
+ [07:51:35] step 1260/4768 loss 3.3998 lr 6.00e-04 gnorm 0.21 55,863 tok/s peakVRAM 13.3GB
74
+ [07:54:43] step 1280/4768 loss 3.4023 lr 6.00e-04 gnorm 0.18 55,865 tok/s peakVRAM 13.3GB
75
+ [07:57:50] step 1300/4768 loss 3.3961 lr 6.00e-04 gnorm 0.25 55,864 tok/s peakVRAM 13.3GB
76
+ [08:00:58] step 1320/4768 loss 3.3871 lr 6.00e-04 gnorm 0.20 55,858 tok/s peakVRAM 13.3GB
77
+ [08:04:06] step 1340/4768 loss 3.3869 lr 6.00e-04 gnorm 0.21 55,852 tok/s peakVRAM 13.3GB
78
+ [08:07:14] step 1360/4768 loss 3.3778 lr 6.00e-04 gnorm 0.21 55,852 tok/s peakVRAM 13.3GB
79
+ [08:10:21] step 1380/4768 loss 3.3692 lr 6.00e-04 gnorm 0.20 55,856 tok/s peakVRAM 13.3GB
80
+ [08:13:29] step 1400/4768 loss 3.3585 lr 6.00e-04 gnorm 0.18 55,859 tok/s peakVRAM 13.3GB
81
+ [08:16:37] step 1420/4768 loss 3.3467 lr 6.00e-04 gnorm 0.21 55,856 tok/s peakVRAM 13.3GB
82
+ [08:19:44] step 1440/4768 loss 3.3529 lr 6.00e-04 gnorm 0.19 55,861 tok/s peakVRAM 13.3GB
83
+ [08:22:52] step 1460/4768 loss 3.3505 lr 6.00e-04 gnorm 0.18 55,860 tok/s peakVRAM 13.3GB
84
+ [08:26:00] step 1480/4768 loss 3.3544 lr 6.00e-04 gnorm 0.22 55,860 tok/s peakVRAM 13.3GB
85
+ [08:29:08] step 1500/4768 loss 3.3271 lr 6.00e-04 gnorm 0.18 55,857 tok/s peakVRAM 13.3GB
86
+ [08:29:10] ckpt @ 1500
87
+ [08:32:18] step 1520/4768 loss 3.3344 lr 6.00e-04 gnorm 0.19 55,160 tok/s peakVRAM 13.3GB
88
+ [08:35:25] step 1540/4768 loss 3.3192 lr 6.00e-04 gnorm 0.23 55,858 tok/s peakVRAM 13.3GB
89
+ [08:38:33] step 1560/4768 loss 3.3266 lr 6.00e-04 gnorm 0.19 55,868 tok/s peakVRAM 13.3GB
90
+ [08:41:41] step 1580/4768 loss 3.3151 lr 6.00e-04 gnorm 0.17 55,867 tok/s peakVRAM 13.3GB
91
+ [08:44:48] step 1600/4768 loss 3.3048 lr 6.00e-04 gnorm 0.19 55,866 tok/s peakVRAM 13.3GB
92
+ [08:47:56] step 1620/4768 loss 3.3046 lr 6.00e-04 gnorm 0.16 55,864 tok/s peakVRAM 13.3GB
93
+ [08:51:04] step 1640/4768 loss 3.3010 lr 6.00e-04 gnorm 0.17 55,858 tok/s peakVRAM 13.3GB
94
+ [08:54:12] step 1660/4768 loss 3.3020 lr 6.00e-04 gnorm 0.18 55,856 tok/s peakVRAM 13.3GB
95
+ [08:57:19] step 1680/4768 loss 3.2901 lr 6.00e-04 gnorm 0.18 55,858 tok/s peakVRAM 13.3GB
96
+ [09:00:27] step 1700/4768 loss 3.2835 lr 6.00e-04 gnorm 0.17 55,858 tok/s peakVRAM 13.3GB
97
+ [09:03:35] step 1720/4768 loss 3.2856 lr 6.00e-04 gnorm 0.18 55,861 tok/s peakVRAM 13.3GB
98
+ [09:06:42] step 1740/4768 loss 3.2787 lr 6.00e-04 gnorm 0.16 55,859 tok/s peakVRAM 13.3GB
99
+ [09:09:50] step 1760/4768 loss 3.2778 lr 6.00e-04 gnorm 0.16 55,859 tok/s peakVRAM 13.3GB
100
+ [09:12:58] step 1780/4768 loss 3.2652 lr 6.00e-04 gnorm 0.18 55,855 tok/s peakVRAM 13.3GB
101
+ [09:16:06] step 1800/4768 loss 3.2684 lr 6.00e-04 gnorm 0.15 55,850 tok/s peakVRAM 13.3GB
102
+ [09:19:13] step 1820/4768 loss 3.2644 lr 6.00e-04 gnorm 0.19 55,853 tok/s peakVRAM 13.3GB
103
+ [09:22:21] step 1840/4768 loss 3.2582 lr 6.00e-04 gnorm 0.16 55,857 tok/s peakVRAM 13.3GB
104
+ [09:25:29] step 1860/4768 loss 3.2575 lr 6.00e-04 gnorm 0.17 55,859 tok/s peakVRAM 13.3GB
105
+ [09:28:37] step 1880/4768 loss 3.2525 lr 6.00e-04 gnorm 0.15 55,861 tok/s peakVRAM 13.3GB
106
+ [09:31:44] step 1900/4768 loss 3.2469 lr 6.00e-04 gnorm 0.17 55,868 tok/s peakVRAM 13.3GB
107
+ [09:34:52] step 1920/4768 loss 3.2459 lr 6.00e-04 gnorm 0.15 55,867 tok/s peakVRAM 13.3GB
108
+ [09:38:00] step 1940/4768 loss 3.2396 lr 6.00e-04 gnorm 0.15 55,862 tok/s peakVRAM 13.3GB
109
+ [09:41:07] step 1960/4768 loss 3.2438 lr 6.00e-04 gnorm 0.18 55,862 tok/s peakVRAM 13.3GB
110
+ [09:44:15] step 1980/4768 loss 3.2376 lr 6.00e-04 gnorm 0.15 55,860 tok/s peakVRAM 13.3GB
111
+ [09:47:23] step 2000/4768 loss 3.2313 lr 6.00e-04 gnorm 0.17 55,858 tok/s peakVRAM 13.3GB
112
+ [09:47:25] ckpt @ 2000
113
+ [09:50:33] step 2020/4768 loss 3.2308 lr 6.00e-04 gnorm 0.15 55,181 tok/s peakVRAM 13.3GB
114
+ [09:53:41] step 2040/4768 loss 3.2255 lr 6.00e-04 gnorm 0.15 55,856 tok/s peakVRAM 13.3GB
115
+ [09:56:48] step 2060/4768 loss 3.2255 lr 6.00e-04 gnorm 0.16 55,854 tok/s peakVRAM 13.3GB
116
+ [09:59:56] step 2080/4768 loss 3.2140 lr 6.00e-04 gnorm 0.16 55,850 tok/s peakVRAM 13.3GB
117
+ [10:03:04] step 2100/4768 loss 3.2224 lr 6.00e-04 gnorm 0.15 55,847 tok/s peakVRAM 13.3GB
118
+ [10:06:12] step 2120/4768 loss 3.2110 lr 6.00e-04 gnorm 0.14 55,850 tok/s peakVRAM 13.3GB
119
+ [10:09:19] step 2140/4768 loss 3.2240 lr 6.00e-04 gnorm 0.15 55,845 tok/s peakVRAM 13.3GB
120
+ [10:12:27] step 2160/4768 loss 3.2026 lr 6.00e-04 gnorm 0.14 55,842 tok/s peakVRAM 13.3GB
121
+ [10:15:35] step 2180/4768 loss 3.2045 lr 6.00e-04 gnorm 0.16 55,843 tok/s peakVRAM 13.3GB
122
+ [10:18:43] step 2200/4768 loss 3.2058 lr 6.00e-04 gnorm 0.15 55,847 tok/s peakVRAM 13.3GB
123
+ [10:21:50] step 2220/4768 loss 3.1977 lr 6.00e-04 gnorm 0.14 55,847 tok/s peakVRAM 13.3GB
124
+ [10:24:58] step 2240/4768 loss 3.1945 lr 6.00e-04 gnorm 0.15 55,852 tok/s peakVRAM 13.3GB
125
+ [10:28:06] step 2260/4768 loss 3.1916 lr 6.00e-04 gnorm 0.14 55,853 tok/s peakVRAM 13.3GB
126
+ [10:31:14] step 2280/4768 loss 3.1868 lr 6.00e-04 gnorm 0.16 55,856 tok/s peakVRAM 13.3GB
127
+ [10:34:21] step 2300/4768 loss 3.1902 lr 6.00e-04 gnorm 0.14 55,852 tok/s peakVRAM 13.3GB
128
+ [10:37:29] step 2320/4768 loss 3.1867 lr 6.00e-04 gnorm 0.13 55,852 tok/s peakVRAM 13.3GB
129
+ [10:40:37] step 2340/4768 loss 3.1821 lr 6.00e-04 gnorm 0.14 55,849 tok/s peakVRAM 13.3GB
130
+ [10:43:45] step 2360/4768 loss 3.1724 lr 6.00e-04 gnorm 0.15 55,851 tok/s peakVRAM 13.3GB
131
+ [10:46:52] step 2380/4768 loss 3.1711 lr 6.00e-04 gnorm 0.12 55,851 tok/s peakVRAM 13.3GB
132
+ [10:50:00] step 2400/4768 loss 3.1715 lr 6.00e-04 gnorm 0.12 55,858 tok/s peakVRAM 13.3GB
133
+ [10:53:08] step 2420/4768 loss 3.1738 lr 6.00e-04 gnorm 0.14 55,855 tok/s peakVRAM 13.3GB
134
+ [10:56:16] step 2440/4768 loss 3.1681 lr 6.00e-04 gnorm 0.13 55,852 tok/s peakVRAM 13.3GB
135
+ [10:59:23] step 2460/4768 loss 3.1676 lr 6.00e-04 gnorm 0.17 55,857 tok/s peakVRAM 13.3GB
136
+ [11:02:31] step 2480/4768 loss 3.1639 lr 6.00e-04 gnorm 0.15 55,854 tok/s peakVRAM 13.3GB
137
+ [11:05:39] step 2500/4768 loss 3.1592 lr 6.00e-04 gnorm 0.17 55,856 tok/s peakVRAM 13.3GB
138
+ [11:05:41] ckpt @ 2500
139
+ [11:08:49] step 2520/4768 loss 3.1604 lr 6.00e-04 gnorm 0.14 55,177 tok/s peakVRAM 13.3GB
140
+ [11:11:56] step 2540/4768 loss 3.1612 lr 6.00e-04 gnorm 0.15 55,851 tok/s peakVRAM 13.3GB
141
+ [11:15:04] step 2560/4768 loss 3.1578 lr 6.00e-04 gnorm 0.13 55,854 tok/s peakVRAM 13.3GB
142
+ [11:18:12] step 2580/4768 loss 3.1543 lr 6.00e-04 gnorm 0.13 55,856 tok/s peakVRAM 13.3GB
143
+ [11:21:20] step 2600/4768 loss 3.1444 lr 6.00e-04 gnorm 0.14 55,855 tok/s peakVRAM 13.3GB
144
+ [11:24:27] step 2620/4768 loss 3.1531 lr 6.00e-04 gnorm 0.13 55,850 tok/s peakVRAM 13.3GB
145
+ [11:27:35] step 2640/4768 loss 3.1535 lr 6.00e-04 gnorm 0.13 55,851 tok/s peakVRAM 13.3GB
146
+ [11:30:43] step 2660/4768 loss 3.1523 lr 6.00e-04 gnorm 0.13 55,851 tok/s peakVRAM 13.3GB
147
+ [11:33:51] step 2680/4768 loss 3.1478 lr 6.00e-04 gnorm 0.15 55,851 tok/s peakVRAM 13.3GB
148
+ [11:36:58] step 2700/4768 loss 3.1495 lr 6.00e-04 gnorm 0.13 55,849 tok/s peakVRAM 13.3GB
149
+ [11:40:06] step 2720/4768 loss 3.1323 lr 6.00e-04 gnorm 0.13 55,855 tok/s peakVRAM 13.3GB
150
+ [11:43:14] step 2740/4768 loss 3.1344 lr 6.00e-04 gnorm 0.11 55,857 tok/s peakVRAM 13.3GB
151
+ [11:46:22] step 2760/4768 loss 3.1370 lr 6.00e-04 gnorm 0.13 55,860 tok/s peakVRAM 13.3GB
152
+ [11:49:29] step 2780/4768 loss 3.1280 lr 6.00e-04 gnorm 0.15 55,860 tok/s peakVRAM 13.3GB
153
+ [11:52:37] step 2800/4768 loss 3.1230 lr 6.00e-04 gnorm 0.12 55,865 tok/s peakVRAM 13.3GB
154
+ [11:55:45] step 2820/4768 loss 3.1200 lr 6.00e-04 gnorm 0.13 55,863 tok/s peakVRAM 13.3GB
155
+ [11:58:52] step 2840/4768 loss 3.1284 lr 6.00e-04 gnorm 0.13 55,856 tok/s peakVRAM 13.3GB
156
+ [12:02:00] step 2860/4768 loss 3.1202 lr 6.00e-04 gnorm 0.13 55,854 tok/s peakVRAM 13.3GB
157
+ [12:05:08] step 2880/4768 loss 3.1225 lr 6.00e-04 gnorm 0.14 55,855 tok/s peakVRAM 13.3GB
158
+ [12:08:16] step 2900/4768 loss 3.1232 lr 6.00e-04 gnorm 0.12 55,853 tok/s peakVRAM 13.3GB
159
+ [12:11:23] step 2920/4768 loss 3.1308 lr 6.00e-04 gnorm 0.12 55,858 tok/s peakVRAM 13.3GB
K12/run/SHA256SUMS_kopiarka ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ 846c7cc9f3c88fcda1394e757dd024eebfefc3b43cba549c4dfaa0c0efeb1594 ckpt_500.pt
2
+ 84f431e41c1a7dcd3cb3a1eda8a572094350a7dbd998669bca640e036c9fd923 ckpt_1000.pt
3
+ d3bd736d8c42dbd52568c0598ad52862dbddbf69829cbf6325faa3bf0a4083cc ckpt_1500.pt
4
+ cdc444a304c2f57727937a5ca0f58599d220eb0741a3920ddf73f3965e2117ca ckpt_2000.pt
5
+ 8f5298ada65ab57bfe1b63dcd49b956cf0f37f26e7ec5e2bab072e2e8da21e84 ckpt_2500.pt
6
+ c54e324a6d5e2b9e33bf9731ff4162df18fb774e57e7bab253576f096dd24511 ckpt_3000.pt
K12/run/args.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ --data-dir /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2 --out-dir /home/ubuntu/kolektyw/dowody400m/wyniki/K12 --run-id dowody_K12 --vocab 32768 --dtype uint16 --block 4096 --n-layer 12 --n-embd 1024 --n-head 16 --value-residual --batch 128 --grad-accum 8 --norm rmsnorm --pos rope --rope-theta 500000 --ffn swiglu --ffn-mult 2.667 --qk-norm --attn-bias 0 --optimizer muon --muon-lr 0.02 --lr 6e-4 --min-lr 6e-05 --wd 0.1 --schedule wsd --decay-frac 0.2 --warmup 500 --act-ckpt --loss-chunk 1024 --sampler random --seed 1 --data-seed 20261022 --compile --log-every 20 --eval-every 4769 --eval-iters 1 --steps 4768 --ckpt-every 500
K12/run/metrics.jsonl ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"step": 20, "loss": 10.404603719711304, "lr": 2.3999999999999997e-05, "tok_s": 43862.84977431825}
2
+ {"step": 40, "loss": 9.534286546707154, "lr": 4.7999999999999994e-05, "tok_s": 58708.75008696259}
3
+ {"step": 60, "loss": 8.592787218093871, "lr": 7.199999999999999e-05, "tok_s": 58728.55645155153}
4
+ {"step": 80, "loss": 7.972748017311096, "lr": 9.599999999999999e-05, "tok_s": 58710.56110012293}
5
+ {"step": 100, "loss": 7.537906694412231, "lr": 0.00011999999999999999, "tok_s": 58682.614600713896}
6
+ {"step": 120, "loss": 7.051306915283203, "lr": 0.00014399999999999998, "tok_s": 58663.493826476195}
7
+ {"step": 140, "loss": 6.628354525566101, "lr": 0.000168, "tok_s": 58646.106530416655}
8
+ {"step": 160, "loss": 6.276793456077575, "lr": 0.00019199999999999998, "tok_s": 58628.3912037025}
9
+ {"step": 180, "loss": 6.018369221687317, "lr": 0.00021599999999999996, "tok_s": 58626.09479047179}
10
+ {"step": 200, "loss": 5.8070940494537355, "lr": 0.00023999999999999998, "tok_s": 58611.57576779736}
11
+ {"step": 220, "loss": 5.628641867637635, "lr": 0.00026399999999999997, "tok_s": 58611.74886020378}
12
+ {"step": 240, "loss": 5.462792611122131, "lr": 0.00028799999999999995, "tok_s": 58622.5650233335}
13
+ {"step": 260, "loss": 5.3006878137588505, "lr": 0.000312, "tok_s": 58625.83103957974}
14
+ {"step": 280, "loss": 5.149684000015259, "lr": 0.000336, "tok_s": 58625.33581950116}
15
+ {"step": 300, "loss": 5.010294389724732, "lr": 0.00035999999999999997, "tok_s": 58621.081819644285}
16
+ {"step": 320, "loss": 4.881231045722961, "lr": 0.00038399999999999996, "tok_s": 58614.27217303556}
17
+ {"step": 340, "loss": 4.742927312850952, "lr": 0.000408, "tok_s": 58615.62371070732}
18
+ {"step": 360, "loss": 4.616938376426697, "lr": 0.00043199999999999993, "tok_s": 58614.1403114175}
19
+ {"step": 380, "loss": 4.496828222274781, "lr": 0.00045599999999999997, "tok_s": 58616.3796178055}
20
+ {"step": 400, "loss": 4.392631483078003, "lr": 0.00047999999999999996, "tok_s": 58608.62616789729}
21
+ {"step": 420, "loss": 4.309436774253845, "lr": 0.000504, "tok_s": 58610.82326513075}
22
+ {"step": 440, "loss": 4.235170435905457, "lr": 0.0005279999999999999, "tok_s": 58611.922500409455}
23
+ {"step": 460, "loss": 4.16366171836853, "lr": 0.000552, "tok_s": 58611.618884582764}
24
+ {"step": 480, "loss": 4.1190622568130495, "lr": 0.0005759999999999999, "tok_s": 58612.602230085446}
25
+ {"step": 500, "loss": 4.061825442314148, "lr": 0.0006, "tok_s": 58608.78963625683}
26
+ {"step": 520, "loss": 4.027750384807587, "lr": 0.0006, "tok_s": 58065.16923676389}
27
+ {"step": 540, "loss": 3.9765324115753176, "lr": 0.0006, "tok_s": 58620.23531294467}
28
+ {"step": 560, "loss": 3.9294686794281004, "lr": 0.0006, "tok_s": 58620.60480690887}
29
+ {"step": 580, "loss": 3.8874564170837402, "lr": 0.0006, "tok_s": 58616.54695746107}
30
+ {"step": 600, "loss": 3.848431408405304, "lr": 0.0006, "tok_s": 58616.59945637344}
31
+ {"step": 620, "loss": 3.8276182293891905, "lr": 0.0006, "tok_s": 58610.03383764811}
32
+ {"step": 640, "loss": 3.8016713857650757, "lr": 0.0006, "tok_s": 58612.397497159815}
33
+ {"step": 660, "loss": 3.77252653837204, "lr": 0.0006, "tok_s": 58614.99515708387}
34
+ {"step": 680, "loss": 3.7515096426010133, "lr": 0.0006, "tok_s": 58607.41678029385}
35
+ {"step": 700, "loss": 3.7216542243957518, "lr": 0.0006, "tok_s": 58603.47148561657}
36
+ {"step": 720, "loss": 3.6929709196090696, "lr": 0.0006, "tok_s": 58598.80060614448}
37
+ {"step": 740, "loss": 3.6890414118766786, "lr": 0.0006, "tok_s": 58599.98504322484}
38
+ {"step": 760, "loss": 3.664959669113159, "lr": 0.0006, "tok_s": 58595.98211951457}
39
+ {"step": 780, "loss": 3.6401920318603516, "lr": 0.0006, "tok_s": 58596.79529233795}
40
+ {"step": 800, "loss": 3.6265568852424623, "lr": 0.0006, "tok_s": 58600.643414189704}
41
+ {"step": 820, "loss": 3.611056935787201, "lr": 0.0006, "tok_s": 58598.78522517366}
42
+ {"step": 840, "loss": 3.5982728123664858, "lr": 0.0006, "tok_s": 58600.3990999932}
43
+ {"step": 860, "loss": 3.577110970020294, "lr": 0.0006, "tok_s": 58597.851139116254}
44
+ {"step": 880, "loss": 3.571992111206055, "lr": 0.0006, "tok_s": 58592.335766610355}
45
+ {"step": 900, "loss": 3.5544564604759215, "lr": 0.0006, "tok_s": 58586.29838538495}
46
+ {"step": 920, "loss": 3.547284722328186, "lr": 0.0006, "tok_s": 58593.34695760348}
47
+ {"step": 940, "loss": 3.541924238204956, "lr": 0.0006, "tok_s": 58587.187695480054}
48
+ {"step": 960, "loss": 3.517482328414917, "lr": 0.0006, "tok_s": 58588.60627705455}
49
+ {"step": 980, "loss": 3.5142526626586914, "lr": 0.0006, "tok_s": 58594.897143534705}
50
+ {"step": 1000, "loss": 3.5042595744132994, "lr": 0.0006, "tok_s": 58588.46009192064}
51
+ {"step": 1020, "loss": 3.4992708206176757, "lr": 0.0006, "tok_s": 57812.14930900983}
52
+ {"step": 1040, "loss": 3.4728254079818726, "lr": 0.0006, "tok_s": 58591.35544269182}
53
+ {"step": 1060, "loss": 3.48453768491745, "lr": 0.0006, "tok_s": 58588.209557429254}
54
+ {"step": 1080, "loss": 3.473938596248627, "lr": 0.0006, "tok_s": 58587.41902198655}
55
+ {"step": 1100, "loss": 3.4584452748298644, "lr": 0.0006, "tok_s": 58591.59749589408}
56
+ {"step": 1120, "loss": 3.440518116950989, "lr": 0.0006, "tok_s": 58594.82516712788}
57
+ {"step": 1140, "loss": 3.4338117957115175, "lr": 0.0006, "tok_s": 58588.28768337564}
58
+ {"step": 1160, "loss": 3.4283645272254946, "lr": 0.0006, "tok_s": 58584.633001453476}
59
+ {"step": 1180, "loss": 3.4317108631134032, "lr": 0.0006, "tok_s": 58589.91112681225}
60
+ {"step": 1200, "loss": 3.413036358356476, "lr": 0.0006, "tok_s": 58581.13904649471}
61
+ {"step": 1220, "loss": 3.4098459243774415, "lr": 0.0006, "tok_s": 58583.53291667873}
62
+ {"step": 1240, "loss": 3.4025972723960876, "lr": 0.0006, "tok_s": 58586.346225600704}
63
+ {"step": 1260, "loss": 3.388778018951416, "lr": 0.0006, "tok_s": 58585.91098397099}
64
+ {"step": 1280, "loss": 3.391374719142914, "lr": 0.0006, "tok_s": 58588.81107795649}
65
+ {"step": 1300, "loss": 3.3844142317771913, "lr": 0.0006, "tok_s": 58585.81171513435}
66
+ {"step": 1320, "loss": 3.376672458648682, "lr": 0.0006, "tok_s": 58590.171822747536}
67
+ {"step": 1340, "loss": 3.376765549182892, "lr": 0.0006, "tok_s": 58587.03667865518}
68
+ {"step": 1360, "loss": 3.368115746974945, "lr": 0.0006, "tok_s": 58590.08042280073}
69
+ {"step": 1380, "loss": 3.359369957447052, "lr": 0.0006, "tok_s": 58579.82163520031}
70
+ {"step": 1400, "loss": 3.3492789387702944, "lr": 0.0006, "tok_s": 58579.18729517547}
71
+ {"step": 1420, "loss": 3.337015962600708, "lr": 0.0006, "tok_s": 58575.79650089553}
72
+ {"step": 1440, "loss": 3.3437799215316772, "lr": 0.0006, "tok_s": 58579.68236010638}
73
+ {"step": 1460, "loss": 3.3416819334030152, "lr": 0.0006, "tok_s": 58573.27415937886}
74
+ {"step": 1480, "loss": 3.345796358585358, "lr": 0.0006, "tok_s": 58572.91501271516}
75
+ {"step": 1500, "loss": 3.3184831380844115, "lr": 0.0006, "tok_s": 58573.43758661024}
76
+ {"step": 1520, "loss": 3.3255487084388733, "lr": 0.0006, "tok_s": 57778.309994633084}
77
+ {"step": 1540, "loss": 3.3099217891693113, "lr": 0.0006, "tok_s": 58532.41938083946}
78
+ {"step": 1560, "loss": 3.318115973472595, "lr": 0.0006, "tok_s": 58532.337041475155}
79
+ {"step": 1580, "loss": 3.306504261493683, "lr": 0.0006, "tok_s": 58567.39411608857}
80
+ {"step": 1600, "loss": 3.296400547027588, "lr": 0.0006, "tok_s": 58572.257498665596}
81
+ {"step": 1620, "loss": 3.2961339712142945, "lr": 0.0006, "tok_s": 58576.22262042889}
82
+ {"step": 1640, "loss": 3.2931604027748107, "lr": 0.0006, "tok_s": 58574.65899026576}
83
+ {"step": 1660, "loss": 3.293904662132263, "lr": 0.0006, "tok_s": 58564.831711681014}
84
+ {"step": 1680, "loss": 3.2828875541687013, "lr": 0.0006, "tok_s": 58562.07684675201}
85
+ {"step": 1700, "loss": 3.2760611534118653, "lr": 0.0006, "tok_s": 58569.60543857275}
86
+ {"step": 1720, "loss": 3.2780789136886597, "lr": 0.0006, "tok_s": 58565.877672177565}
87
+ {"step": 1740, "loss": 3.271463096141815, "lr": 0.0006, "tok_s": 58556.22727373207}
88
+ {"step": 1760, "loss": 3.2703675389289857, "lr": 0.0006, "tok_s": 58551.24721203557}
89
+ {"step": 1780, "loss": 3.2580028653144835, "lr": 0.0006, "tok_s": 58558.9586651728}
90
+ {"step": 1800, "loss": 3.261461687088013, "lr": 0.0006, "tok_s": 58560.024688794445}
91
+ {"step": 1820, "loss": 3.257357358932495, "lr": 0.0006, "tok_s": 58560.431397326596}
92
+ {"step": 1840, "loss": 3.2504354238510134, "lr": 0.0006, "tok_s": 58555.27708088719}
93
+ {"step": 1860, "loss": 3.249496328830719, "lr": 0.0006, "tok_s": 58571.49820492442}
94
+ {"step": 1880, "loss": 3.2457431077957155, "lr": 0.0006, "tok_s": 58553.54478125217}
95
+ {"step": 1900, "loss": 3.2397770166397093, "lr": 0.0006, "tok_s": 58561.86201764043}
96
+ {"step": 1920, "loss": 3.240020728111267, "lr": 0.0006, "tok_s": 58556.22703984424}
97
+ {"step": 1940, "loss": 3.2326427340507506, "lr": 0.0006, "tok_s": 58556.623872204094}
98
+ {"step": 1960, "loss": 3.236352801322937, "lr": 0.0006, "tok_s": 58558.79929516935}
99
+ {"step": 1980, "loss": 3.2304039359092713, "lr": 0.0006, "tok_s": 58575.08602940318}
100
+ {"step": 2000, "loss": 3.224052131175995, "lr": 0.0006, "tok_s": 58571.13845602058}
101
+ {"step": 2020, "loss": 3.223966634273529, "lr": 0.0006, "tok_s": 57804.03149473105}
102
+ {"step": 2040, "loss": 3.2166189670562746, "lr": 0.0006, "tok_s": 58583.05097441445}
103
+ {"step": 2060, "loss": 3.21760436296463, "lr": 0.0006, "tok_s": 58578.82191219394}
104
+ {"step": 2080, "loss": 3.2062864303588867, "lr": 0.0006, "tok_s": 58579.01416096729}
105
+ {"step": 2100, "loss": 3.214371931552887, "lr": 0.0006, "tok_s": 58578.60610196551}
106
+ {"step": 2120, "loss": 3.2036935448646546, "lr": 0.0006, "tok_s": 58579.446569272186}
107
+ {"step": 2140, "loss": 3.217024397850037, "lr": 0.0006, "tok_s": 58588.06594983953}
108
+ {"step": 2160, "loss": 3.1952428817749023, "lr": 0.0006, "tok_s": 58584.55636797676}
109
+ {"step": 2180, "loss": 3.1975870609283445, "lr": 0.0006, "tok_s": 58580.69701578473}
110
+ {"step": 2200, "loss": 3.198507809638977, "lr": 0.0006, "tok_s": 58583.069390456294}
111
+ {"step": 2220, "loss": 3.1905167579650877, "lr": 0.0006, "tok_s": 58578.568417262235}
112
+ {"step": 2240, "loss": 3.18764910697937, "lr": 0.0006, "tok_s": 58565.70890596694}
113
+ {"step": 2260, "loss": 3.18397171497345, "lr": 0.0006, "tok_s": 58565.45965783414}
114
+ {"step": 2280, "loss": 3.1804327368736267, "lr": 0.0006, "tok_s": 58564.653437321234}
115
+ {"step": 2300, "loss": 3.1841544985771177, "lr": 0.0006, "tok_s": 58561.42214923009}
116
+ {"step": 2320, "loss": 3.1800976395606995, "lr": 0.0006, "tok_s": 58572.766487667184}
117
+ {"step": 2340, "loss": 3.1748282670974732, "lr": 0.0006, "tok_s": 58584.013462213494}
118
+ {"step": 2360, "loss": 3.16541702747345, "lr": 0.0006, "tok_s": 58581.646707292864}
119
+ {"step": 2380, "loss": 3.1643529653549196, "lr": 0.0006, "tok_s": 58584.37235480553}
120
+ {"step": 2400, "loss": 3.1648273706436156, "lr": 0.0006, "tok_s": 58586.16064033874}
121
+ {"step": 2420, "loss": 3.1675984501838683, "lr": 0.0006, "tok_s": 58585.525850858976}
122
+ {"step": 2440, "loss": 3.1628398299217224, "lr": 0.0006, "tok_s": 58580.35244713431}
123
+ {"step": 2460, "loss": 3.1619457602500916, "lr": 0.0006, "tok_s": 58582.353124208545}
124
+ {"step": 2480, "loss": 3.1585049986839295, "lr": 0.0006, "tok_s": 58588.8177902054}
125
+ {"step": 2500, "loss": 3.1526406526565554, "lr": 0.0006, "tok_s": 58584.57579946054}
126
+ {"step": 2520, "loss": 3.154686999320984, "lr": 0.0006, "tok_s": 57829.54215959935}
127
+ {"step": 2540, "loss": 3.1545638918876646, "lr": 0.0006, "tok_s": 58616.96812325964}
128
+ {"step": 2560, "loss": 3.151969051361084, "lr": 0.0006, "tok_s": 58598.46300740269}
129
+ {"step": 2580, "loss": 3.146875965595245, "lr": 0.0006, "tok_s": 58572.50220231211}
130
+ {"step": 2600, "loss": 3.1376704573631287, "lr": 0.0006, "tok_s": 58616.03166062821}
131
+ {"step": 2620, "loss": 3.1459038615226746, "lr": 0.0006, "tok_s": 58586.39359763685}
132
+ {"step": 2640, "loss": 3.1474135518074036, "lr": 0.0006, "tok_s": 58584.45702569282}
133
+ {"step": 2660, "loss": 3.1453431606292725, "lr": 0.0006, "tok_s": 58588.917693626856}
134
+ {"step": 2680, "loss": 3.140736925601959, "lr": 0.0006, "tok_s": 58578.20811337535}
135
+ {"step": 2700, "loss": 3.143047571182251, "lr": 0.0006, "tok_s": 58575.02837797594}
136
+ {"step": 2720, "loss": 3.1259115099906922, "lr": 0.0006, "tok_s": 58573.31909202556}
137
+ {"step": 2740, "loss": 3.127363657951355, "lr": 0.0006, "tok_s": 58563.766058918976}
138
+ {"step": 2760, "loss": 3.130553150177002, "lr": 0.0006, "tok_s": 58568.07078550068}
139
+ {"step": 2780, "loss": 3.1217617988586426, "lr": 0.0006, "tok_s": 58567.39442805796}
140
+ {"step": 2800, "loss": 3.1166795253753663, "lr": 0.0006, "tok_s": 58559.65580217986}
141
+ {"step": 2820, "loss": 3.1136237382888794, "lr": 0.0006, "tok_s": 58563.632006992724}
142
+ {"step": 2840, "loss": 3.122041928768158, "lr": 0.0006, "tok_s": 58584.55675816707}
143
+ {"step": 2860, "loss": 3.1131366968154905, "lr": 0.0006, "tok_s": 58601.91811302135}
144
+ {"step": 2880, "loss": 3.1153586983680723, "lr": 0.0006, "tok_s": 58558.33163810221}
145
+ {"step": 2900, "loss": 3.116631364822388, "lr": 0.0006, "tok_s": 58604.634323974504}
146
+ {"step": 2920, "loss": 3.123911237716675, "lr": 0.0006, "tok_s": 58617.125700667}
147
+ {"step": 2940, "loss": 3.1143979907035826, "lr": 0.0006, "tok_s": 58618.255485958936}
148
+ {"step": 2960, "loss": 3.116018295288086, "lr": 0.0006, "tok_s": 58609.45960832013}
149
+ {"step": 2980, "loss": 3.1031396865844725, "lr": 0.0006, "tok_s": 58601.60757331772}
150
+ {"step": 3000, "loss": 3.1038497567176817, "lr": 0.0006, "tok_s": 58569.51808067913}
151
+ {"step": 3020, "loss": 3.099448227882385, "lr": 0.0006, "tok_s": 57789.27866566382}
152
+ {"step": 3040, "loss": 3.09664865732193, "lr": 0.0006, "tok_s": 58561.89679567643}
153
+ {"step": 3060, "loss": 3.100166177749634, "lr": 0.0006, "tok_s": 58560.26196074277}
K12/run/srodowisko_K12_mini0_resume0.txt ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-23T02:33:05Z ARM K12 GPU 1 MINI 0 DATA_SEED 20261022 CKPT 500 RESUME 0 DEST /var/lib/docker/kolektyw/dowody400m_ckpt/wyniki_K12
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ 0702c32c0ad602d8d12b16539e05cfbc68d1823cfd39b82b984357d78f27f664 /home/ubuntu/kolektyw/dowody400m/kod/ramie_v03.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 341aae81ce8512294dcd7de93ced26053025677b4ab449606be989f7fd898d27 /home/ubuntu/kolektyw/dowody400m/kod/kopiarka_ckpt.sh
7
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
8
+ 2.6.0+cu124 12.4
9
+ accelerate==1.15.0
10
+ annotated-doc==0.0.5
11
+ anyio==4.15.1
12
+ certifi==2026.10.19
13
+ click==8.5.0
14
+ filelock==4.2.4
15
+ fsspec==2026.9.0
16
+ h11==0.16.0
17
+ hf-xet==1.7.0
18
+ httpcore==1.0.9
19
+ httpx==0.28.1
20
+ huggingface_hub==1.33.0
21
+ idna==3.20
22
+ Jinja2==3.1.6
23
+ markdown-it-py==4.2.0
24
+ MarkupSafe==3.0.4
25
+ mdurl==0.1.2
26
+ mpmath==1.3.0
27
+ networkx==3.7
28
+ numpy==2.2.6
29
+ nvidia-cublas-cu12==12.4.5.8
30
+ nvidia-cuda-cupti-cu12==12.4.127
31
+ nvidia-cuda-nvrtc-cu12==12.4.127
32
+ nvidia-cuda-runtime-cu12==12.4.127
33
+ nvidia-cudnn-cu12==9.1.0.70
34
+ nvidia-cufft-cu12==11.2.1.3
35
+ nvidia-curand-cu12==10.3.5.147
36
+ nvidia-cusolver-cu12==11.6.1.9
37
+ nvidia-cusparse-cu12==12.3.1.170
38
+ nvidia-cusparselt-cu12==0.6.2
39
+ nvidia-nccl-cu12==2.21.5
40
+ nvidia-nvjitlink-cu12==12.4.127
41
+ nvidia-nvtx-cu12==12.4.127
42
+ packaging==26.3
43
+ pillow==12.3.0
44
+ psutil==7.2.2
45
+ Pygments==2.21.0
46
+ PyYAML==6.0.3
47
+ regex==2026.9.29
48
+ rich==15.0.0
49
+ safetensors==0.8.0
50
+ setuptools==84.2.0
51
+ shellingham==1.5.4
52
+ sympy==1.13.1
53
+ tokenizers==0.23.3
54
+ torch==2.6.0+cu124
55
+ torchvision==0.21.0+cu124
56
+ tqdm==4.70.1
57
+ transformers==5.17.0
58
+ triton==3.2.0
59
+ typer==0.27.3
60
+ typing_extensions==4.16.0
61
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
64
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
K12/run/srodowisko_resume0.txt ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-23T02:33:05Z ARM K12 GPU 1 MINI 0 DATA_SEED 20261022 CKPT 500 RESUME 0 DEST /var/lib/docker/kolektyw/dowody400m_ckpt/wyniki_K12
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ 0702c32c0ad602d8d12b16539e05cfbc68d1823cfd39b82b984357d78f27f664 /home/ubuntu/kolektyw/dowody400m/kod/ramie_v03.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 341aae81ce8512294dcd7de93ced26053025677b4ab449606be989f7fd898d27 /home/ubuntu/kolektyw/dowody400m/kod/kopiarka_ckpt.sh
7
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
8
+ 2.6.0+cu124 12.4
9
+ accelerate==1.15.0
10
+ annotated-doc==0.0.5
11
+ anyio==4.15.1
12
+ certifi==2026.10.19
13
+ click==8.5.0
14
+ filelock==4.2.4
15
+ fsspec==2026.9.0
16
+ h11==0.16.0
17
+ hf-xet==1.7.0
18
+ httpcore==1.0.9
19
+ httpx==0.28.1
20
+ huggingface_hub==1.33.0
21
+ idna==3.20
22
+ Jinja2==3.1.6
23
+ markdown-it-py==4.2.0
24
+ MarkupSafe==3.0.4
25
+ mdurl==0.1.2
26
+ mpmath==1.3.0
27
+ networkx==3.7
28
+ numpy==2.2.6
29
+ nvidia-cublas-cu12==12.4.5.8
30
+ nvidia-cuda-cupti-cu12==12.4.127
31
+ nvidia-cuda-nvrtc-cu12==12.4.127
32
+ nvidia-cuda-runtime-cu12==12.4.127
33
+ nvidia-cudnn-cu12==9.1.0.70
34
+ nvidia-cufft-cu12==11.2.1.3
35
+ nvidia-curand-cu12==10.3.5.147
36
+ nvidia-cusolver-cu12==11.6.1.9
37
+ nvidia-cusparse-cu12==12.3.1.170
38
+ nvidia-cusparselt-cu12==0.6.2
39
+ nvidia-nccl-cu12==2.21.5
40
+ nvidia-nvjitlink-cu12==12.4.127
41
+ nvidia-nvtx-cu12==12.4.127
42
+ packaging==26.3
43
+ pillow==12.3.0
44
+ psutil==7.2.2
45
+ Pygments==2.21.0
46
+ PyYAML==6.0.3
47
+ regex==2026.9.29
48
+ rich==15.0.0
49
+ safetensors==0.8.0
50
+ setuptools==84.2.0
51
+ shellingham==1.5.4
52
+ sympy==1.13.1
53
+ tokenizers==0.23.3
54
+ torch==2.6.0+cu124
55
+ torchvision==0.21.0+cu124
56
+ tqdm==4.70.1
57
+ transformers==5.17.0
58
+ triton==3.2.0
59
+ typer==0.27.3
60
+ typing_extensions==4.16.0
61
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
64
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
K12/run/train.log ADDED
@@ -0,0 +1,167 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [04:33:11] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [04:33:11] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [04:33:13] model GPT-ref: 184.6M param (L12 d1024 h16)
4
+ [04:33:14] optimizer=muon muon_lr=0.02 (mult=33.3x)
5
+ [04:33:14] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [04:33:14] r13: mixer_layout=AAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [04:33:15] torch.compile enabled
8
+ [04:33:15] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [04:37:14] step 20/4768 loss 10.4046 lr 2.40e-05 gnorm 2.87 43,863 tok/s peakVRAM 14.2GB
10
+ [04:40:12] step 40/4768 loss 9.5343 lr 4.80e-05 gnorm 1.38 58,709 tok/s peakVRAM 14.2GB
11
+ [04:43:11] step 60/4768 loss 8.5928 lr 7.20e-05 gnorm 0.59 58,729 tok/s peakVRAM 14.2GB
12
+ [04:46:09] step 80/4768 loss 7.9727 lr 9.60e-05 gnorm 0.86 58,711 tok/s peakVRAM 14.2GB
13
+ [04:49:08] step 100/4768 loss 7.5379 lr 1.20e-04 gnorm 0.89 58,683 tok/s peakVRAM 14.2GB
14
+ [04:52:07] step 120/4768 loss 7.0513 lr 1.44e-04 gnorm 0.87 58,663 tok/s peakVRAM 14.2GB
15
+ [04:55:06] step 140/4768 loss 6.6284 lr 1.68e-04 gnorm 0.71 58,646 tok/s peakVRAM 14.2GB
16
+ [04:58:05] step 160/4768 loss 6.2768 lr 1.92e-04 gnorm 0.77 58,628 tok/s peakVRAM 14.2GB
17
+ [05:01:03] step 180/4768 loss 6.0184 lr 2.16e-04 gnorm 0.77 58,626 tok/s peakVRAM 14.2GB
18
+ [05:04:02] step 200/4768 loss 5.8071 lr 2.40e-04 gnorm 0.83 58,612 tok/s peakVRAM 14.2GB
19
+ [05:07:01] step 220/4768 loss 5.6286 lr 2.64e-04 gnorm 0.80 58,612 tok/s peakVRAM 14.2GB
20
+ [05:10:00] step 240/4768 loss 5.4628 lr 2.88e-04 gnorm 0.86 58,623 tok/s peakVRAM 14.2GB
21
+ [05:12:59] step 260/4768 loss 5.3007 lr 3.12e-04 gnorm 0.86 58,626 tok/s peakVRAM 14.2GB
22
+ [05:15:58] step 280/4768 loss 5.1497 lr 3.36e-04 gnorm 0.74 58,625 tok/s peakVRAM 14.2GB
23
+ [05:18:57] step 300/4768 loss 5.0103 lr 3.60e-04 gnorm 0.83 58,621 tok/s peakVRAM 14.2GB
24
+ [05:21:56] step 320/4768 loss 4.8812 lr 3.84e-04 gnorm 0.59 58,614 tok/s peakVRAM 14.2GB
25
+ [05:24:54] step 340/4768 loss 4.7429 lr 4.08e-04 gnorm 0.76 58,616 tok/s peakVRAM 14.2GB
26
+ [05:27:53] step 360/4768 loss 4.6169 lr 4.32e-04 gnorm 0.55 58,614 tok/s peakVRAM 14.2GB
27
+ [05:30:52] step 380/4768 loss 4.4968 lr 4.56e-04 gnorm 0.53 58,616 tok/s peakVRAM 14.2GB
28
+ [05:33:51] step 400/4768 loss 4.3926 lr 4.80e-04 gnorm 0.41 58,609 tok/s peakVRAM 14.2GB
29
+ [05:36:50] step 420/4768 loss 4.3094 lr 5.04e-04 gnorm 0.46 58,611 tok/s peakVRAM 14.2GB
30
+ [05:39:49] step 440/4768 loss 4.2352 lr 5.28e-04 gnorm 0.44 58,612 tok/s peakVRAM 14.2GB
31
+ [05:42:48] step 460/4768 loss 4.1637 lr 5.52e-04 gnorm 0.39 58,612 tok/s peakVRAM 14.2GB
32
+ [05:45:47] step 480/4768 loss 4.1191 lr 5.76e-04 gnorm 0.34 58,613 tok/s peakVRAM 14.2GB
33
+ [05:48:46] step 500/4768 loss 4.0618 lr 6.00e-04 gnorm 0.32 58,609 tok/s peakVRAM 14.2GB
34
+ [05:48:47] ckpt @ 500
35
+ [05:51:46] step 520/4768 loss 4.0278 lr 6.00e-04 gnorm 0.33 58,065 tok/s peakVRAM 14.2GB
36
+ [05:54:45] step 540/4768 loss 3.9765 lr 6.00e-04 gnorm 0.25 58,620 tok/s peakVRAM 14.2GB
37
+ [05:57:44] step 560/4768 loss 3.9295 lr 6.00e-04 gnorm 0.26 58,621 tok/s peakVRAM 14.2GB
38
+ [06:00:43] step 580/4768 loss 3.8875 lr 6.00e-04 gnorm 0.26 58,617 tok/s peakVRAM 14.2GB
39
+ [06:03:42] step 600/4768 loss 3.8484 lr 6.00e-04 gnorm 0.21 58,617 tok/s peakVRAM 14.2GB
40
+ [06:06:41] step 620/4768 loss 3.8276 lr 6.00e-04 gnorm 0.23 58,610 tok/s peakVRAM 14.2GB
41
+ [06:09:40] step 640/4768 loss 3.8017 lr 6.00e-04 gnorm 0.18 58,612 tok/s peakVRAM 14.2GB
42
+ [06:12:39] step 660/4768 loss 3.7725 lr 6.00e-04 gnorm 0.19 58,615 tok/s peakVRAM 14.2GB
43
+ [06:15:37] step 680/4768 loss 3.7515 lr 6.00e-04 gnorm 0.19 58,607 tok/s peakVRAM 14.2GB
44
+ [06:18:36] step 700/4768 loss 3.7217 lr 6.00e-04 gnorm 0.18 58,603 tok/s peakVRAM 14.2GB
45
+ [06:21:35] step 720/4768 loss 3.6930 lr 6.00e-04 gnorm 0.18 58,599 tok/s peakVRAM 14.2GB
46
+ [06:24:34] step 740/4768 loss 3.6890 lr 6.00e-04 gnorm 0.19 58,600 tok/s peakVRAM 14.2GB
47
+ [06:27:33] step 760/4768 loss 3.6650 lr 6.00e-04 gnorm 0.16 58,596 tok/s peakVRAM 14.2GB
48
+ [06:30:32] step 780/4768 loss 3.6402 lr 6.00e-04 gnorm 0.17 58,597 tok/s peakVRAM 14.2GB
49
+ [06:33:31] step 800/4768 loss 3.6266 lr 6.00e-04 gnorm 0.16 58,601 tok/s peakVRAM 14.2GB
50
+ [06:36:30] step 820/4768 loss 3.6111 lr 6.00e-04 gnorm 0.16 58,599 tok/s peakVRAM 14.2GB
51
+ [06:39:29] step 840/4768 loss 3.5983 lr 6.00e-04 gnorm 0.16 58,600 tok/s peakVRAM 14.2GB
52
+ [06:42:28] step 860/4768 loss 3.5771 lr 6.00e-04 gnorm 0.16 58,598 tok/s peakVRAM 14.2GB
53
+ [06:45:27] step 880/4768 loss 3.5720 lr 6.00e-04 gnorm 0.15 58,592 tok/s peakVRAM 14.2GB
54
+ [06:48:26] step 900/4768 loss 3.5545 lr 6.00e-04 gnorm 0.15 58,586 tok/s peakVRAM 14.2GB
55
+ [06:51:25] step 920/4768 loss 3.5473 lr 6.00e-04 gnorm 0.14 58,593 tok/s peakVRAM 14.2GB
56
+ [06:54:24] step 940/4768 loss 3.5419 lr 6.00e-04 gnorm 0.16 58,587 tok/s peakVRAM 14.2GB
57
+ [06:57:23] step 960/4768 loss 3.5175 lr 6.00e-04 gnorm 0.14 58,589 tok/s peakVRAM 14.2GB
58
+ [07:00:22] step 980/4768 loss 3.5143 lr 6.00e-04 gnorm 0.14 58,595 tok/s peakVRAM 14.2GB
59
+ [07:03:21] step 1000/4768 loss 3.5043 lr 6.00e-04 gnorm 0.15 58,588 tok/s peakVRAM 14.2GB
60
+ [07:03:23] ckpt @ 1000
61
+ [07:06:22] step 1020/4768 loss 3.4993 lr 6.00e-04 gnorm 0.14 57,812 tok/s peakVRAM 14.2GB
62
+ [07:09:21] step 1040/4768 loss 3.4728 lr 6.00e-04 gnorm 0.13 58,591 tok/s peakVRAM 14.2GB
63
+ [07:12:20] step 1060/4768 loss 3.4845 lr 6.00e-04 gnorm 0.14 58,588 tok/s peakVRAM 14.2GB
64
+ [07:15:19] step 1080/4768 loss 3.4739 lr 6.00e-04 gnorm 0.16 58,587 tok/s peakVRAM 14.2GB
65
+ [07:18:18] step 1100/4768 loss 3.4584 lr 6.00e-04 gnorm 0.15 58,592 tok/s peakVRAM 14.2GB
66
+ [07:21:17] step 1120/4768 loss 3.4405 lr 6.00e-04 gnorm 0.12 58,595 tok/s peakVRAM 14.2GB
67
+ [07:24:16] step 1140/4768 loss 3.4338 lr 6.00e-04 gnorm 0.14 58,588 tok/s peakVRAM 14.2GB
68
+ [07:27:15] step 1160/4768 loss 3.4284 lr 6.00e-04 gnorm 0.12 58,585 tok/s peakVRAM 14.2GB
69
+ [07:30:14] step 1180/4768 loss 3.4317 lr 6.00e-04 gnorm 0.14 58,590 tok/s peakVRAM 14.2GB
70
+ [07:33:13] step 1200/4768 loss 3.4130 lr 6.00e-04 gnorm 0.14 58,581 tok/s peakVRAM 14.2GB
71
+ [07:36:12] step 1220/4768 loss 3.4098 lr 6.00e-04 gnorm 0.12 58,584 tok/s peakVRAM 14.2GB
72
+ [07:39:11] step 1240/4768 loss 3.4026 lr 6.00e-04 gnorm 0.12 58,586 tok/s peakVRAM 14.2GB
73
+ [07:42:10] step 1260/4768 loss 3.3888 lr 6.00e-04 gnorm 0.12 58,586 tok/s peakVRAM 14.2GB
74
+ [07:45:09] step 1280/4768 loss 3.3914 lr 6.00e-04 gnorm 0.12 58,589 tok/s peakVRAM 14.2GB
75
+ [07:48:08] step 1300/4768 loss 3.3844 lr 6.00e-04 gnorm 0.12 58,586 tok/s peakVRAM 14.2GB
76
+ [07:51:07] step 1320/4768 loss 3.3767 lr 6.00e-04 gnorm 0.14 58,590 tok/s peakVRAM 14.2GB
77
+ [07:54:06] step 1340/4768 loss 3.3768 lr 6.00e-04 gnorm 0.12 58,587 tok/s peakVRAM 14.2GB
78
+ [07:57:05] step 1360/4768 loss 3.3681 lr 6.00e-04 gnorm 0.13 58,590 tok/s peakVRAM 14.2GB
79
+ [08:00:04] step 1380/4768 loss 3.3594 lr 6.00e-04 gnorm 0.12 58,580 tok/s peakVRAM 14.2GB
80
+ [08:03:03] step 1400/4768 loss 3.3493 lr 6.00e-04 gnorm 0.12 58,579 tok/s peakVRAM 14.2GB
81
+ [08:06:02] step 1420/4768 loss 3.3370 lr 6.00e-04 gnorm 0.13 58,576 tok/s peakVRAM 14.2GB
82
+ [08:09:01] step 1440/4768 loss 3.3438 lr 6.00e-04 gnorm 0.13 58,580 tok/s peakVRAM 14.2GB
83
+ [08:12:00] step 1460/4768 loss 3.3417 lr 6.00e-04 gnorm 0.12 58,573 tok/s peakVRAM 14.2GB
84
+ [08:14:59] step 1480/4768 loss 3.3458 lr 6.00e-04 gnorm 0.11 58,573 tok/s peakVRAM 14.2GB
85
+ [08:17:58] step 1500/4768 loss 3.3185 lr 6.00e-04 gnorm 0.13 58,573 tok/s peakVRAM 14.2GB
86
+ [08:18:00] ckpt @ 1500
87
+ [08:20:59] step 1520/4768 loss 3.3255 lr 6.00e-04 gnorm 0.11 57,778 tok/s peakVRAM 14.2GB
88
+ [08:23:58] step 1540/4768 loss 3.3099 lr 6.00e-04 gnorm 0.12 58,532 tok/s peakVRAM 14.2GB
89
+ [08:26:57] step 1560/4768 loss 3.3181 lr 6.00e-04 gnorm 0.12 58,532 tok/s peakVRAM 14.2GB
90
+ [08:29:57] step 1580/4768 loss 3.3065 lr 6.00e-04 gnorm 0.13 58,567 tok/s peakVRAM 14.2GB
91
+ [08:32:56] step 1600/4768 loss 3.2964 lr 6.00e-04 gnorm 0.12 58,572 tok/s peakVRAM 14.2GB
92
+ [08:35:55] step 1620/4768 loss 3.2961 lr 6.00e-04 gnorm 0.13 58,576 tok/s peakVRAM 14.2GB
93
+ [08:38:54] step 1640/4768 loss 3.2932 lr 6.00e-04 gnorm 0.11 58,575 tok/s peakVRAM 14.2GB
94
+ [08:41:53] step 1660/4768 loss 3.2939 lr 6.00e-04 gnorm 0.12 58,565 tok/s peakVRAM 14.2GB
95
+ [08:44:52] step 1680/4768 loss 3.2829 lr 6.00e-04 gnorm 0.11 58,562 tok/s peakVRAM 14.2GB
96
+ [08:47:51] step 1700/4768 loss 3.2761 lr 6.00e-04 gnorm 0.13 58,570 tok/s peakVRAM 14.2GB
97
+ [08:50:50] step 1720/4768 loss 3.2781 lr 6.00e-04 gnorm 0.12 58,566 tok/s peakVRAM 14.2GB
98
+ [08:53:49] step 1740/4768 loss 3.2715 lr 6.00e-04 gnorm 0.11 58,556 tok/s peakVRAM 14.2GB
99
+ [08:56:48] step 1760/4768 loss 3.2704 lr 6.00e-04 gnorm 0.13 58,551 tok/s peakVRAM 14.2GB
100
+ [08:59:47] step 1780/4768 loss 3.2580 lr 6.00e-04 gnorm 0.11 58,559 tok/s peakVRAM 14.2GB
101
+ [09:02:46] step 1800/4768 loss 3.2615 lr 6.00e-04 gnorm 0.12 58,560 tok/s peakVRAM 14.2GB
102
+ [09:05:45] step 1820/4768 loss 3.2574 lr 6.00e-04 gnorm 0.12 58,560 tok/s peakVRAM 14.2GB
103
+ [09:08:44] step 1840/4768 loss 3.2504 lr 6.00e-04 gnorm 0.11 58,555 tok/s peakVRAM 14.2GB
104
+ [09:11:43] step 1860/4768 loss 3.2495 lr 6.00e-04 gnorm 0.12 58,571 tok/s peakVRAM 14.2GB
105
+ [09:14:42] step 1880/4768 loss 3.2457 lr 6.00e-04 gnorm 0.11 58,554 tok/s peakVRAM 14.2GB
106
+ [09:17:41] step 1900/4768 loss 3.2398 lr 6.00e-04 gnorm 0.12 58,562 tok/s peakVRAM 14.2GB
107
+ [09:20:40] step 1920/4768 loss 3.2400 lr 6.00e-04 gnorm 0.11 58,556 tok/s peakVRAM 14.2GB
108
+ [09:23:39] step 1940/4768 loss 3.2326 lr 6.00e-04 gnorm 0.11 58,557 tok/s peakVRAM 14.2GB
109
+ [09:26:39] step 1960/4768 loss 3.2364 lr 6.00e-04 gnorm 0.12 58,559 tok/s peakVRAM 14.2GB
110
+ [09:29:38] step 1980/4768 loss 3.2304 lr 6.00e-04 gnorm 0.11 58,575 tok/s peakVRAM 14.2GB
111
+ [09:32:37] step 2000/4768 loss 3.2241 lr 6.00e-04 gnorm 0.11 58,571 tok/s peakVRAM 14.2GB
112
+ [09:32:39] ckpt @ 2000
113
+ [09:35:38] step 2020/4768 loss 3.2240 lr 6.00e-04 gnorm 0.11 57,804 tok/s peakVRAM 14.2GB
114
+ [09:38:37] step 2040/4768 loss 3.2166 lr 6.00e-04 gnorm 0.11 58,583 tok/s peakVRAM 14.2GB
115
+ [09:41:36] step 2060/4768 loss 3.2176 lr 6.00e-04 gnorm 0.11 58,579 tok/s peakVRAM 14.2GB
116
+ [09:44:35] step 2080/4768 loss 3.2063 lr 6.00e-04 gnorm 0.11 58,579 tok/s peakVRAM 14.2GB
117
+ [09:47:34] step 2100/4768 loss 3.2144 lr 6.00e-04 gnorm 0.10 58,579 tok/s peakVRAM 14.2GB
118
+ [09:50:33] step 2120/4768 loss 3.2037 lr 6.00e-04 gnorm 0.11 58,579 tok/s peakVRAM 14.2GB
119
+ [09:53:32] step 2140/4768 loss 3.2170 lr 6.00e-04 gnorm 0.11 58,588 tok/s peakVRAM 14.2GB
120
+ [09:56:31] step 2160/4768 loss 3.1952 lr 6.00e-04 gnorm 0.11 58,585 tok/s peakVRAM 14.2GB
121
+ [09:59:30] step 2180/4768 loss 3.1976 lr 6.00e-04 gnorm 0.10 58,581 tok/s peakVRAM 14.2GB
122
+ [10:02:29] step 2200/4768 loss 3.1985 lr 6.00e-04 gnorm 0.10 58,583 tok/s peakVRAM 14.2GB
123
+ [10:05:28] step 2220/4768 loss 3.1905 lr 6.00e-04 gnorm 0.11 58,579 tok/s peakVRAM 14.2GB
124
+ [10:08:27] step 2240/4768 loss 3.1876 lr 6.00e-04 gnorm 0.10 58,566 tok/s peakVRAM 14.2GB
125
+ [10:11:26] step 2260/4768 loss 3.1840 lr 6.00e-04 gnorm 0.11 58,565 tok/s peakVRAM 14.2GB
126
+ [10:14:25] step 2280/4768 loss 3.1804 lr 6.00e-04 gnorm 0.10 58,565 tok/s peakVRAM 14.2GB
127
+ [10:17:24] step 2300/4768 loss 3.1842 lr 6.00e-04 gnorm 0.11 58,561 tok/s peakVRAM 14.2GB
128
+ [10:20:23] step 2320/4768 loss 3.1801 lr 6.00e-04 gnorm 0.12 58,573 tok/s peakVRAM 14.2GB
129
+ [10:23:22] step 2340/4768 loss 3.1748 lr 6.00e-04 gnorm 0.11 58,584 tok/s peakVRAM 14.2GB
130
+ [10:26:21] step 2360/4768 loss 3.1654 lr 6.00e-04 gnorm 0.11 58,582 tok/s peakVRAM 14.2GB
131
+ [10:29:20] step 2380/4768 loss 3.1644 lr 6.00e-04 gnorm 0.10 58,584 tok/s peakVRAM 14.2GB
132
+ [10:32:19] step 2400/4768 loss 3.1648 lr 6.00e-04 gnorm 0.12 58,586 tok/s peakVRAM 14.2GB
133
+ [10:35:18] step 2420/4768 loss 3.1676 lr 6.00e-04 gnorm 0.11 58,586 tok/s peakVRAM 14.2GB
134
+ [10:38:17] step 2440/4768 loss 3.1628 lr 6.00e-04 gnorm 0.10 58,580 tok/s peakVRAM 14.2GB
135
+ [10:41:16] step 2460/4768 loss 3.1619 lr 6.00e-04 gnorm 0.10 58,582 tok/s peakVRAM 14.2GB
136
+ [10:44:15] step 2480/4768 loss 3.1585 lr 6.00e-04 gnorm 0.11 58,589 tok/s peakVRAM 14.2GB
137
+ [10:47:14] step 2500/4768 loss 3.1526 lr 6.00e-04 gnorm 0.09 58,585 tok/s peakVRAM 14.2GB
138
+ [10:47:16] ckpt @ 2500
139
+ [10:50:15] step 2520/4768 loss 3.1547 lr 6.00e-04 gnorm 0.12 57,830 tok/s peakVRAM 14.2GB
140
+ [10:53:14] step 2540/4768 loss 3.1546 lr 6.00e-04 gnorm 0.11 58,617 tok/s peakVRAM 14.2GB
141
+ [10:56:13] step 2560/4768 loss 3.1520 lr 6.00e-04 gnorm 0.11 58,598 tok/s peakVRAM 14.2GB
142
+ [10:59:12] step 2580/4768 loss 3.1469 lr 6.00e-04 gnorm 0.10 58,573 tok/s peakVRAM 14.2GB
143
+ [11:02:11] step 2600/4768 loss 3.1377 lr 6.00e-04 gnorm 0.11 58,616 tok/s peakVRAM 14.2GB
144
+ [11:05:10] step 2620/4768 loss 3.1459 lr 6.00e-04 gnorm 0.11 58,586 tok/s peakVRAM 14.2GB
145
+ [11:08:09] step 2640/4768 loss 3.1474 lr 6.00e-04 gnorm 0.11 58,584 tok/s peakVRAM 14.2GB
146
+ [11:11:08] step 2660/4768 loss 3.1453 lr 6.00e-04 gnorm 0.11 58,589 tok/s peakVRAM 14.2GB
147
+ [11:14:07] step 2680/4768 loss 3.1407 lr 6.00e-04 gnorm 0.10 58,578 tok/s peakVRAM 14.2GB
148
+ [11:17:06] step 2700/4768 loss 3.1430 lr 6.00e-04 gnorm 0.11 58,575 tok/s peakVRAM 14.2GB
149
+ [11:20:05] step 2720/4768 loss 3.1259 lr 6.00e-04 gnorm 0.10 58,573 tok/s peakVRAM 14.2GB
150
+ [11:23:04] step 2740/4768 loss 3.1274 lr 6.00e-04 gnorm 0.11 58,564 tok/s peakVRAM 14.2GB
151
+ [11:26:03] step 2760/4768 loss 3.1306 lr 6.00e-04 gnorm 0.11 58,568 tok/s peakVRAM 14.2GB
152
+ [11:29:02] step 2780/4768 loss 3.1218 lr 6.00e-04 gnorm 0.11 58,567 tok/s peakVRAM 14.2GB
153
+ [11:32:01] step 2800/4768 loss 3.1167 lr 6.00e-04 gnorm 0.11 58,560 tok/s peakVRAM 14.2GB
154
+ [11:35:00] step 2820/4768 loss 3.1136 lr 6.00e-04 gnorm 0.10 58,564 tok/s peakVRAM 14.2GB
155
+ [11:37:59] step 2840/4768 loss 3.1220 lr 6.00e-04 gnorm 0.10 58,585 tok/s peakVRAM 14.2GB
156
+ [11:40:58] step 2860/4768 loss 3.1131 lr 6.00e-04 gnorm 0.10 58,602 tok/s peakVRAM 14.2GB
157
+ [11:43:57] step 2880/4768 loss 3.1154 lr 6.00e-04 gnorm 0.10 58,558 tok/s peakVRAM 14.2GB
158
+ [11:46:56] step 2900/4768 loss 3.1166 lr 6.00e-04 gnorm 0.10 58,605 tok/s peakVRAM 14.2GB
159
+ [11:49:55] step 2920/4768 loss 3.1239 lr 6.00e-04 gnorm 0.10 58,617 tok/s peakVRAM 14.2GB
160
+ [11:52:54] step 2940/4768 loss 3.1144 lr 6.00e-04 gnorm 0.09 58,618 tok/s peakVRAM 14.2GB
161
+ [11:55:53] step 2960/4768 loss 3.1160 lr 6.00e-04 gnorm 0.11 58,609 tok/s peakVRAM 14.2GB
162
+ [11:58:52] step 2980/4768 loss 3.1031 lr 6.00e-04 gnorm 0.10 58,602 tok/s peakVRAM 14.2GB
163
+ [12:01:51] step 3000/4768 loss 3.1038 lr 6.00e-04 gnorm 0.09 58,570 tok/s peakVRAM 14.2GB
164
+ [12:01:53] ckpt @ 3000
165
+ [12:04:52] step 3020/4768 loss 3.0994 lr 6.00e-04 gnorm 0.09 57,789 tok/s peakVRAM 14.2GB
166
+ [12:07:51] step 3040/4768 loss 3.0966 lr 6.00e-04 gnorm 0.10 58,562 tok/s peakVRAM 14.2GB
167
+ [12:10:50] step 3060/4768 loss 3.1002 lr 6.00e-04 gnorm 0.11 58,560 tok/s peakVRAM 14.2GB
K12/run/train.out ADDED
@@ -0,0 +1,167 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [04:33:11] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [04:33:11] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [04:33:13] model GPT-ref: 184.6M param (L12 d1024 h16)
4
+ [04:33:14] optimizer=muon muon_lr=0.02 (mult=33.3x)
5
+ [04:33:14] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [04:33:14] r13: mixer_layout=AAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [04:33:15] torch.compile enabled
8
+ [04:33:15] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [04:37:14] step 20/4768 loss 10.4046 lr 2.40e-05 gnorm 2.87 43,863 tok/s peakVRAM 14.2GB
10
+ [04:40:12] step 40/4768 loss 9.5343 lr 4.80e-05 gnorm 1.38 58,709 tok/s peakVRAM 14.2GB
11
+ [04:43:11] step 60/4768 loss 8.5928 lr 7.20e-05 gnorm 0.59 58,729 tok/s peakVRAM 14.2GB
12
+ [04:46:09] step 80/4768 loss 7.9727 lr 9.60e-05 gnorm 0.86 58,711 tok/s peakVRAM 14.2GB
13
+ [04:49:08] step 100/4768 loss 7.5379 lr 1.20e-04 gnorm 0.89 58,683 tok/s peakVRAM 14.2GB
14
+ [04:52:07] step 120/4768 loss 7.0513 lr 1.44e-04 gnorm 0.87 58,663 tok/s peakVRAM 14.2GB
15
+ [04:55:06] step 140/4768 loss 6.6284 lr 1.68e-04 gnorm 0.71 58,646 tok/s peakVRAM 14.2GB
16
+ [04:58:05] step 160/4768 loss 6.2768 lr 1.92e-04 gnorm 0.77 58,628 tok/s peakVRAM 14.2GB
17
+ [05:01:03] step 180/4768 loss 6.0184 lr 2.16e-04 gnorm 0.77 58,626 tok/s peakVRAM 14.2GB
18
+ [05:04:02] step 200/4768 loss 5.8071 lr 2.40e-04 gnorm 0.83 58,612 tok/s peakVRAM 14.2GB
19
+ [05:07:01] step 220/4768 loss 5.6286 lr 2.64e-04 gnorm 0.80 58,612 tok/s peakVRAM 14.2GB
20
+ [05:10:00] step 240/4768 loss 5.4628 lr 2.88e-04 gnorm 0.86 58,623 tok/s peakVRAM 14.2GB
21
+ [05:12:59] step 260/4768 loss 5.3007 lr 3.12e-04 gnorm 0.86 58,626 tok/s peakVRAM 14.2GB
22
+ [05:15:58] step 280/4768 loss 5.1497 lr 3.36e-04 gnorm 0.74 58,625 tok/s peakVRAM 14.2GB
23
+ [05:18:57] step 300/4768 loss 5.0103 lr 3.60e-04 gnorm 0.83 58,621 tok/s peakVRAM 14.2GB
24
+ [05:21:56] step 320/4768 loss 4.8812 lr 3.84e-04 gnorm 0.59 58,614 tok/s peakVRAM 14.2GB
25
+ [05:24:54] step 340/4768 loss 4.7429 lr 4.08e-04 gnorm 0.76 58,616 tok/s peakVRAM 14.2GB
26
+ [05:27:53] step 360/4768 loss 4.6169 lr 4.32e-04 gnorm 0.55 58,614 tok/s peakVRAM 14.2GB
27
+ [05:30:52] step 380/4768 loss 4.4968 lr 4.56e-04 gnorm 0.53 58,616 tok/s peakVRAM 14.2GB
28
+ [05:33:51] step 400/4768 loss 4.3926 lr 4.80e-04 gnorm 0.41 58,609 tok/s peakVRAM 14.2GB
29
+ [05:36:50] step 420/4768 loss 4.3094 lr 5.04e-04 gnorm 0.46 58,611 tok/s peakVRAM 14.2GB
30
+ [05:39:49] step 440/4768 loss 4.2352 lr 5.28e-04 gnorm 0.44 58,612 tok/s peakVRAM 14.2GB
31
+ [05:42:48] step 460/4768 loss 4.1637 lr 5.52e-04 gnorm 0.39 58,612 tok/s peakVRAM 14.2GB
32
+ [05:45:47] step 480/4768 loss 4.1191 lr 5.76e-04 gnorm 0.34 58,613 tok/s peakVRAM 14.2GB
33
+ [05:48:46] step 500/4768 loss 4.0618 lr 6.00e-04 gnorm 0.32 58,609 tok/s peakVRAM 14.2GB
34
+ [05:48:47] ckpt @ 500
35
+ [05:51:46] step 520/4768 loss 4.0278 lr 6.00e-04 gnorm 0.33 58,065 tok/s peakVRAM 14.2GB
36
+ [05:54:45] step 540/4768 loss 3.9765 lr 6.00e-04 gnorm 0.25 58,620 tok/s peakVRAM 14.2GB
37
+ [05:57:44] step 560/4768 loss 3.9295 lr 6.00e-04 gnorm 0.26 58,621 tok/s peakVRAM 14.2GB
38
+ [06:00:43] step 580/4768 loss 3.8875 lr 6.00e-04 gnorm 0.26 58,617 tok/s peakVRAM 14.2GB
39
+ [06:03:42] step 600/4768 loss 3.8484 lr 6.00e-04 gnorm 0.21 58,617 tok/s peakVRAM 14.2GB
40
+ [06:06:41] step 620/4768 loss 3.8276 lr 6.00e-04 gnorm 0.23 58,610 tok/s peakVRAM 14.2GB
41
+ [06:09:40] step 640/4768 loss 3.8017 lr 6.00e-04 gnorm 0.18 58,612 tok/s peakVRAM 14.2GB
42
+ [06:12:39] step 660/4768 loss 3.7725 lr 6.00e-04 gnorm 0.19 58,615 tok/s peakVRAM 14.2GB
43
+ [06:15:37] step 680/4768 loss 3.7515 lr 6.00e-04 gnorm 0.19 58,607 tok/s peakVRAM 14.2GB
44
+ [06:18:36] step 700/4768 loss 3.7217 lr 6.00e-04 gnorm 0.18 58,603 tok/s peakVRAM 14.2GB
45
+ [06:21:35] step 720/4768 loss 3.6930 lr 6.00e-04 gnorm 0.18 58,599 tok/s peakVRAM 14.2GB
46
+ [06:24:34] step 740/4768 loss 3.6890 lr 6.00e-04 gnorm 0.19 58,600 tok/s peakVRAM 14.2GB
47
+ [06:27:33] step 760/4768 loss 3.6650 lr 6.00e-04 gnorm 0.16 58,596 tok/s peakVRAM 14.2GB
48
+ [06:30:32] step 780/4768 loss 3.6402 lr 6.00e-04 gnorm 0.17 58,597 tok/s peakVRAM 14.2GB
49
+ [06:33:31] step 800/4768 loss 3.6266 lr 6.00e-04 gnorm 0.16 58,601 tok/s peakVRAM 14.2GB
50
+ [06:36:30] step 820/4768 loss 3.6111 lr 6.00e-04 gnorm 0.16 58,599 tok/s peakVRAM 14.2GB
51
+ [06:39:29] step 840/4768 loss 3.5983 lr 6.00e-04 gnorm 0.16 58,600 tok/s peakVRAM 14.2GB
52
+ [06:42:28] step 860/4768 loss 3.5771 lr 6.00e-04 gnorm 0.16 58,598 tok/s peakVRAM 14.2GB
53
+ [06:45:27] step 880/4768 loss 3.5720 lr 6.00e-04 gnorm 0.15 58,592 tok/s peakVRAM 14.2GB
54
+ [06:48:26] step 900/4768 loss 3.5545 lr 6.00e-04 gnorm 0.15 58,586 tok/s peakVRAM 14.2GB
55
+ [06:51:25] step 920/4768 loss 3.5473 lr 6.00e-04 gnorm 0.14 58,593 tok/s peakVRAM 14.2GB
56
+ [06:54:24] step 940/4768 loss 3.5419 lr 6.00e-04 gnorm 0.16 58,587 tok/s peakVRAM 14.2GB
57
+ [06:57:23] step 960/4768 loss 3.5175 lr 6.00e-04 gnorm 0.14 58,589 tok/s peakVRAM 14.2GB
58
+ [07:00:22] step 980/4768 loss 3.5143 lr 6.00e-04 gnorm 0.14 58,595 tok/s peakVRAM 14.2GB
59
+ [07:03:21] step 1000/4768 loss 3.5043 lr 6.00e-04 gnorm 0.15 58,588 tok/s peakVRAM 14.2GB
60
+ [07:03:23] ckpt @ 1000
61
+ [07:06:22] step 1020/4768 loss 3.4993 lr 6.00e-04 gnorm 0.14 57,812 tok/s peakVRAM 14.2GB
62
+ [07:09:21] step 1040/4768 loss 3.4728 lr 6.00e-04 gnorm 0.13 58,591 tok/s peakVRAM 14.2GB
63
+ [07:12:20] step 1060/4768 loss 3.4845 lr 6.00e-04 gnorm 0.14 58,588 tok/s peakVRAM 14.2GB
64
+ [07:15:19] step 1080/4768 loss 3.4739 lr 6.00e-04 gnorm 0.16 58,587 tok/s peakVRAM 14.2GB
65
+ [07:18:18] step 1100/4768 loss 3.4584 lr 6.00e-04 gnorm 0.15 58,592 tok/s peakVRAM 14.2GB
66
+ [07:21:17] step 1120/4768 loss 3.4405 lr 6.00e-04 gnorm 0.12 58,595 tok/s peakVRAM 14.2GB
67
+ [07:24:16] step 1140/4768 loss 3.4338 lr 6.00e-04 gnorm 0.14 58,588 tok/s peakVRAM 14.2GB
68
+ [07:27:15] step 1160/4768 loss 3.4284 lr 6.00e-04 gnorm 0.12 58,585 tok/s peakVRAM 14.2GB
69
+ [07:30:14] step 1180/4768 loss 3.4317 lr 6.00e-04 gnorm 0.14 58,590 tok/s peakVRAM 14.2GB
70
+ [07:33:13] step 1200/4768 loss 3.4130 lr 6.00e-04 gnorm 0.14 58,581 tok/s peakVRAM 14.2GB
71
+ [07:36:12] step 1220/4768 loss 3.4098 lr 6.00e-04 gnorm 0.12 58,584 tok/s peakVRAM 14.2GB
72
+ [07:39:11] step 1240/4768 loss 3.4026 lr 6.00e-04 gnorm 0.12 58,586 tok/s peakVRAM 14.2GB
73
+ [07:42:10] step 1260/4768 loss 3.3888 lr 6.00e-04 gnorm 0.12 58,586 tok/s peakVRAM 14.2GB
74
+ [07:45:09] step 1280/4768 loss 3.3914 lr 6.00e-04 gnorm 0.12 58,589 tok/s peakVRAM 14.2GB
75
+ [07:48:08] step 1300/4768 loss 3.3844 lr 6.00e-04 gnorm 0.12 58,586 tok/s peakVRAM 14.2GB
76
+ [07:51:07] step 1320/4768 loss 3.3767 lr 6.00e-04 gnorm 0.14 58,590 tok/s peakVRAM 14.2GB
77
+ [07:54:06] step 1340/4768 loss 3.3768 lr 6.00e-04 gnorm 0.12 58,587 tok/s peakVRAM 14.2GB
78
+ [07:57:05] step 1360/4768 loss 3.3681 lr 6.00e-04 gnorm 0.13 58,590 tok/s peakVRAM 14.2GB
79
+ [08:00:04] step 1380/4768 loss 3.3594 lr 6.00e-04 gnorm 0.12 58,580 tok/s peakVRAM 14.2GB
80
+ [08:03:03] step 1400/4768 loss 3.3493 lr 6.00e-04 gnorm 0.12 58,579 tok/s peakVRAM 14.2GB
81
+ [08:06:02] step 1420/4768 loss 3.3370 lr 6.00e-04 gnorm 0.13 58,576 tok/s peakVRAM 14.2GB
82
+ [08:09:01] step 1440/4768 loss 3.3438 lr 6.00e-04 gnorm 0.13 58,580 tok/s peakVRAM 14.2GB
83
+ [08:12:00] step 1460/4768 loss 3.3417 lr 6.00e-04 gnorm 0.12 58,573 tok/s peakVRAM 14.2GB
84
+ [08:14:59] step 1480/4768 loss 3.3458 lr 6.00e-04 gnorm 0.11 58,573 tok/s peakVRAM 14.2GB
85
+ [08:17:58] step 1500/4768 loss 3.3185 lr 6.00e-04 gnorm 0.13 58,573 tok/s peakVRAM 14.2GB
86
+ [08:18:00] ckpt @ 1500
87
+ [08:20:59] step 1520/4768 loss 3.3255 lr 6.00e-04 gnorm 0.11 57,778 tok/s peakVRAM 14.2GB
88
+ [08:23:58] step 1540/4768 loss 3.3099 lr 6.00e-04 gnorm 0.12 58,532 tok/s peakVRAM 14.2GB
89
+ [08:26:57] step 1560/4768 loss 3.3181 lr 6.00e-04 gnorm 0.12 58,532 tok/s peakVRAM 14.2GB
90
+ [08:29:57] step 1580/4768 loss 3.3065 lr 6.00e-04 gnorm 0.13 58,567 tok/s peakVRAM 14.2GB
91
+ [08:32:56] step 1600/4768 loss 3.2964 lr 6.00e-04 gnorm 0.12 58,572 tok/s peakVRAM 14.2GB
92
+ [08:35:55] step 1620/4768 loss 3.2961 lr 6.00e-04 gnorm 0.13 58,576 tok/s peakVRAM 14.2GB
93
+ [08:38:54] step 1640/4768 loss 3.2932 lr 6.00e-04 gnorm 0.11 58,575 tok/s peakVRAM 14.2GB
94
+ [08:41:53] step 1660/4768 loss 3.2939 lr 6.00e-04 gnorm 0.12 58,565 tok/s peakVRAM 14.2GB
95
+ [08:44:52] step 1680/4768 loss 3.2829 lr 6.00e-04 gnorm 0.11 58,562 tok/s peakVRAM 14.2GB
96
+ [08:47:51] step 1700/4768 loss 3.2761 lr 6.00e-04 gnorm 0.13 58,570 tok/s peakVRAM 14.2GB
97
+ [08:50:50] step 1720/4768 loss 3.2781 lr 6.00e-04 gnorm 0.12 58,566 tok/s peakVRAM 14.2GB
98
+ [08:53:49] step 1740/4768 loss 3.2715 lr 6.00e-04 gnorm 0.11 58,556 tok/s peakVRAM 14.2GB
99
+ [08:56:48] step 1760/4768 loss 3.2704 lr 6.00e-04 gnorm 0.13 58,551 tok/s peakVRAM 14.2GB
100
+ [08:59:47] step 1780/4768 loss 3.2580 lr 6.00e-04 gnorm 0.11 58,559 tok/s peakVRAM 14.2GB
101
+ [09:02:46] step 1800/4768 loss 3.2615 lr 6.00e-04 gnorm 0.12 58,560 tok/s peakVRAM 14.2GB
102
+ [09:05:45] step 1820/4768 loss 3.2574 lr 6.00e-04 gnorm 0.12 58,560 tok/s peakVRAM 14.2GB
103
+ [09:08:44] step 1840/4768 loss 3.2504 lr 6.00e-04 gnorm 0.11 58,555 tok/s peakVRAM 14.2GB
104
+ [09:11:43] step 1860/4768 loss 3.2495 lr 6.00e-04 gnorm 0.12 58,571 tok/s peakVRAM 14.2GB
105
+ [09:14:42] step 1880/4768 loss 3.2457 lr 6.00e-04 gnorm 0.11 58,554 tok/s peakVRAM 14.2GB
106
+ [09:17:41] step 1900/4768 loss 3.2398 lr 6.00e-04 gnorm 0.12 58,562 tok/s peakVRAM 14.2GB
107
+ [09:20:40] step 1920/4768 loss 3.2400 lr 6.00e-04 gnorm 0.11 58,556 tok/s peakVRAM 14.2GB
108
+ [09:23:39] step 1940/4768 loss 3.2326 lr 6.00e-04 gnorm 0.11 58,557 tok/s peakVRAM 14.2GB
109
+ [09:26:39] step 1960/4768 loss 3.2364 lr 6.00e-04 gnorm 0.12 58,559 tok/s peakVRAM 14.2GB
110
+ [09:29:38] step 1980/4768 loss 3.2304 lr 6.00e-04 gnorm 0.11 58,575 tok/s peakVRAM 14.2GB
111
+ [09:32:37] step 2000/4768 loss 3.2241 lr 6.00e-04 gnorm 0.11 58,571 tok/s peakVRAM 14.2GB
112
+ [09:32:39] ckpt @ 2000
113
+ [09:35:38] step 2020/4768 loss 3.2240 lr 6.00e-04 gnorm 0.11 57,804 tok/s peakVRAM 14.2GB
114
+ [09:38:37] step 2040/4768 loss 3.2166 lr 6.00e-04 gnorm 0.11 58,583 tok/s peakVRAM 14.2GB
115
+ [09:41:36] step 2060/4768 loss 3.2176 lr 6.00e-04 gnorm 0.11 58,579 tok/s peakVRAM 14.2GB
116
+ [09:44:35] step 2080/4768 loss 3.2063 lr 6.00e-04 gnorm 0.11 58,579 tok/s peakVRAM 14.2GB
117
+ [09:47:34] step 2100/4768 loss 3.2144 lr 6.00e-04 gnorm 0.10 58,579 tok/s peakVRAM 14.2GB
118
+ [09:50:33] step 2120/4768 loss 3.2037 lr 6.00e-04 gnorm 0.11 58,579 tok/s peakVRAM 14.2GB
119
+ [09:53:32] step 2140/4768 loss 3.2170 lr 6.00e-04 gnorm 0.11 58,588 tok/s peakVRAM 14.2GB
120
+ [09:56:31] step 2160/4768 loss 3.1952 lr 6.00e-04 gnorm 0.11 58,585 tok/s peakVRAM 14.2GB
121
+ [09:59:30] step 2180/4768 loss 3.1976 lr 6.00e-04 gnorm 0.10 58,581 tok/s peakVRAM 14.2GB
122
+ [10:02:29] step 2200/4768 loss 3.1985 lr 6.00e-04 gnorm 0.10 58,583 tok/s peakVRAM 14.2GB
123
+ [10:05:28] step 2220/4768 loss 3.1905 lr 6.00e-04 gnorm 0.11 58,579 tok/s peakVRAM 14.2GB
124
+ [10:08:27] step 2240/4768 loss 3.1876 lr 6.00e-04 gnorm 0.10 58,566 tok/s peakVRAM 14.2GB
125
+ [10:11:26] step 2260/4768 loss 3.1840 lr 6.00e-04 gnorm 0.11 58,565 tok/s peakVRAM 14.2GB
126
+ [10:14:25] step 2280/4768 loss 3.1804 lr 6.00e-04 gnorm 0.10 58,565 tok/s peakVRAM 14.2GB
127
+ [10:17:24] step 2300/4768 loss 3.1842 lr 6.00e-04 gnorm 0.11 58,561 tok/s peakVRAM 14.2GB
128
+ [10:20:23] step 2320/4768 loss 3.1801 lr 6.00e-04 gnorm 0.12 58,573 tok/s peakVRAM 14.2GB
129
+ [10:23:22] step 2340/4768 loss 3.1748 lr 6.00e-04 gnorm 0.11 58,584 tok/s peakVRAM 14.2GB
130
+ [10:26:21] step 2360/4768 loss 3.1654 lr 6.00e-04 gnorm 0.11 58,582 tok/s peakVRAM 14.2GB
131
+ [10:29:20] step 2380/4768 loss 3.1644 lr 6.00e-04 gnorm 0.10 58,584 tok/s peakVRAM 14.2GB
132
+ [10:32:19] step 2400/4768 loss 3.1648 lr 6.00e-04 gnorm 0.12 58,586 tok/s peakVRAM 14.2GB
133
+ [10:35:18] step 2420/4768 loss 3.1676 lr 6.00e-04 gnorm 0.11 58,586 tok/s peakVRAM 14.2GB
134
+ [10:38:17] step 2440/4768 loss 3.1628 lr 6.00e-04 gnorm 0.10 58,580 tok/s peakVRAM 14.2GB
135
+ [10:41:16] step 2460/4768 loss 3.1619 lr 6.00e-04 gnorm 0.10 58,582 tok/s peakVRAM 14.2GB
136
+ [10:44:15] step 2480/4768 loss 3.1585 lr 6.00e-04 gnorm 0.11 58,589 tok/s peakVRAM 14.2GB
137
+ [10:47:14] step 2500/4768 loss 3.1526 lr 6.00e-04 gnorm 0.09 58,585 tok/s peakVRAM 14.2GB
138
+ [10:47:16] ckpt @ 2500
139
+ [10:50:15] step 2520/4768 loss 3.1547 lr 6.00e-04 gnorm 0.12 57,830 tok/s peakVRAM 14.2GB
140
+ [10:53:14] step 2540/4768 loss 3.1546 lr 6.00e-04 gnorm 0.11 58,617 tok/s peakVRAM 14.2GB
141
+ [10:56:13] step 2560/4768 loss 3.1520 lr 6.00e-04 gnorm 0.11 58,598 tok/s peakVRAM 14.2GB
142
+ [10:59:12] step 2580/4768 loss 3.1469 lr 6.00e-04 gnorm 0.10 58,573 tok/s peakVRAM 14.2GB
143
+ [11:02:11] step 2600/4768 loss 3.1377 lr 6.00e-04 gnorm 0.11 58,616 tok/s peakVRAM 14.2GB
144
+ [11:05:10] step 2620/4768 loss 3.1459 lr 6.00e-04 gnorm 0.11 58,586 tok/s peakVRAM 14.2GB
145
+ [11:08:09] step 2640/4768 loss 3.1474 lr 6.00e-04 gnorm 0.11 58,584 tok/s peakVRAM 14.2GB
146
+ [11:11:08] step 2660/4768 loss 3.1453 lr 6.00e-04 gnorm 0.11 58,589 tok/s peakVRAM 14.2GB
147
+ [11:14:07] step 2680/4768 loss 3.1407 lr 6.00e-04 gnorm 0.10 58,578 tok/s peakVRAM 14.2GB
148
+ [11:17:06] step 2700/4768 loss 3.1430 lr 6.00e-04 gnorm 0.11 58,575 tok/s peakVRAM 14.2GB
149
+ [11:20:05] step 2720/4768 loss 3.1259 lr 6.00e-04 gnorm 0.10 58,573 tok/s peakVRAM 14.2GB
150
+ [11:23:04] step 2740/4768 loss 3.1274 lr 6.00e-04 gnorm 0.11 58,564 tok/s peakVRAM 14.2GB
151
+ [11:26:03] step 2760/4768 loss 3.1306 lr 6.00e-04 gnorm 0.11 58,568 tok/s peakVRAM 14.2GB
152
+ [11:29:02] step 2780/4768 loss 3.1218 lr 6.00e-04 gnorm 0.11 58,567 tok/s peakVRAM 14.2GB
153
+ [11:32:01] step 2800/4768 loss 3.1167 lr 6.00e-04 gnorm 0.11 58,560 tok/s peakVRAM 14.2GB
154
+ [11:35:00] step 2820/4768 loss 3.1136 lr 6.00e-04 gnorm 0.10 58,564 tok/s peakVRAM 14.2GB
155
+ [11:37:59] step 2840/4768 loss 3.1220 lr 6.00e-04 gnorm 0.10 58,585 tok/s peakVRAM 14.2GB
156
+ [11:40:58] step 2860/4768 loss 3.1131 lr 6.00e-04 gnorm 0.10 58,602 tok/s peakVRAM 14.2GB
157
+ [11:43:57] step 2880/4768 loss 3.1154 lr 6.00e-04 gnorm 0.10 58,558 tok/s peakVRAM 14.2GB
158
+ [11:46:56] step 2900/4768 loss 3.1166 lr 6.00e-04 gnorm 0.10 58,605 tok/s peakVRAM 14.2GB
159
+ [11:49:55] step 2920/4768 loss 3.1239 lr 6.00e-04 gnorm 0.10 58,617 tok/s peakVRAM 14.2GB
160
+ [11:52:54] step 2940/4768 loss 3.1144 lr 6.00e-04 gnorm 0.09 58,618 tok/s peakVRAM 14.2GB
161
+ [11:55:53] step 2960/4768 loss 3.1160 lr 6.00e-04 gnorm 0.11 58,609 tok/s peakVRAM 14.2GB
162
+ [11:58:52] step 2980/4768 loss 3.1031 lr 6.00e-04 gnorm 0.10 58,602 tok/s peakVRAM 14.2GB
163
+ [12:01:51] step 3000/4768 loss 3.1038 lr 6.00e-04 gnorm 0.09 58,570 tok/s peakVRAM 14.2GB
164
+ [12:01:53] ckpt @ 3000
165
+ [12:04:52] step 3020/4768 loss 3.0994 lr 6.00e-04 gnorm 0.09 57,789 tok/s peakVRAM 14.2GB
166
+ [12:07:51] step 3040/4768 loss 3.0966 lr 6.00e-04 gnorm 0.10 58,562 tok/s peakVRAM 14.2GB
167
+ [12:10:50] step 3060/4768 loss 3.1002 lr 6.00e-04 gnorm 0.11 58,560 tok/s peakVRAM 14.2GB
K22/run/args.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ --data-dir /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2 --out-dir /home/ubuntu/kolektyw/dowody400m/wyniki/K22 --run-id dowody_K22 --vocab 32768 --dtype uint16 --block 4096 --n-layer 22 --n-embd 768 --n-head 12 --value-residual --batch 128 --grad-accum 8 --norm rmsnorm --pos rope --rope-theta 500000 --ffn swiglu --ffn-mult 2.667 --qk-norm --attn-bias 0 --optimizer muon --muon-lr 0.02 --lr 6e-4 --min-lr 6e-05 --wd 0.1 --schedule wsd --decay-frac 0.2 --warmup 500 --act-ckpt --loss-chunk 1024 --sampler random --seed 1 --data-seed 20261022 --compile --log-every 20 --eval-every 4769 --eval-iters 1 --steps 4768 --ckpt-every 4768
K22/run/metrics.jsonl ADDED
@@ -0,0 +1,238 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"step": 20, "loss": 10.37869439125061, "lr": 2.3999999999999997e-05, "tok_s": 36393.553282902314}
2
+ {"step": 40, "loss": 9.627096700668336, "lr": 4.7999999999999994e-05, "tok_s": 50497.672795545426}
3
+ {"step": 60, "loss": 8.837148857116699, "lr": 7.199999999999999e-05, "tok_s": 50510.12726419128}
4
+ {"step": 80, "loss": 8.161912059783935, "lr": 9.599999999999999e-05, "tok_s": 50497.238002564685}
5
+ {"step": 100, "loss": 7.683762717247009, "lr": 0.00011999999999999999, "tok_s": 50465.52822745929}
6
+ {"step": 120, "loss": 7.22345232963562, "lr": 0.00014399999999999998, "tok_s": 50440.79953964529}
7
+ {"step": 140, "loss": 6.816223430633545, "lr": 0.000168, "tok_s": 50430.647907341336}
8
+ {"step": 160, "loss": 6.451836967468262, "lr": 0.00019199999999999998, "tok_s": 50417.94875199477}
9
+ {"step": 180, "loss": 6.164482140541077, "lr": 0.00021599999999999996, "tok_s": 50412.281978101135}
10
+ {"step": 200, "loss": 5.928548049926758, "lr": 0.00023999999999999998, "tok_s": 50410.515272681805}
11
+ {"step": 220, "loss": 5.729818272590637, "lr": 0.00026399999999999997, "tok_s": 50409.25054367114}
12
+ {"step": 240, "loss": 5.550185561180115, "lr": 0.00028799999999999995, "tok_s": 50408.0463106913}
13
+ {"step": 260, "loss": 5.381895279884338, "lr": 0.000312, "tok_s": 50407.06276837466}
14
+ {"step": 280, "loss": 5.224881482124329, "lr": 0.000336, "tok_s": 50410.07464126373}
15
+ {"step": 300, "loss": 5.07756621837616, "lr": 0.00035999999999999997, "tok_s": 50409.40983744519}
16
+ {"step": 320, "loss": 4.930981469154358, "lr": 0.00038399999999999996, "tok_s": 50409.892404924765}
17
+ {"step": 340, "loss": 4.773558759689331, "lr": 0.000408, "tok_s": 50408.03908881801}
18
+ {"step": 360, "loss": 4.648498702049255, "lr": 0.00043199999999999993, "tok_s": 50400.943471663995}
19
+ {"step": 380, "loss": 4.530560612678528, "lr": 0.00045599999999999997, "tok_s": 50403.91250220873}
20
+ {"step": 400, "loss": 4.440827441215515, "lr": 0.00047999999999999996, "tok_s": 50401.6025649675}
21
+ {"step": 420, "loss": 4.364287209510803, "lr": 0.000504, "tok_s": 50404.13403394797}
22
+ {"step": 440, "loss": 4.294577264785767, "lr": 0.0005279999999999999, "tok_s": 50403.16883988858}
23
+ {"step": 460, "loss": 4.225187301635742, "lr": 0.000552, "tok_s": 50392.21966179442}
24
+ {"step": 480, "loss": 4.177277779579162, "lr": 0.0005759999999999999, "tok_s": 50388.11379174276}
25
+ {"step": 500, "loss": 4.125906372070313, "lr": 0.0006, "tok_s": 50391.59355088377}
26
+ {"step": 520, "loss": 4.089947938919067, "lr": 0.0006, "tok_s": 50395.15474966302}
27
+ {"step": 540, "loss": 4.037944316864014, "lr": 0.0006, "tok_s": 50396.28745206674}
28
+ {"step": 560, "loss": 3.9871296882629395, "lr": 0.0006, "tok_s": 50388.8140580344}
29
+ {"step": 580, "loss": 3.942476987838745, "lr": 0.0006, "tok_s": 50386.95542198973}
30
+ {"step": 600, "loss": 3.899268078804016, "lr": 0.0006, "tok_s": 50382.43288655501}
31
+ {"step": 620, "loss": 3.873553729057312, "lr": 0.0006, "tok_s": 50383.500545223214}
32
+ {"step": 640, "loss": 3.846368408203125, "lr": 0.0006, "tok_s": 50384.20264578695}
33
+ {"step": 660, "loss": 3.813744640350342, "lr": 0.0006, "tok_s": 50378.349674127}
34
+ {"step": 680, "loss": 3.788252055644989, "lr": 0.0006, "tok_s": 50378.65350300499}
35
+ {"step": 700, "loss": 3.7559040665626524, "lr": 0.0006, "tok_s": 50379.44509110364}
36
+ {"step": 720, "loss": 3.723824405670166, "lr": 0.0006, "tok_s": 50376.78597997029}
37
+ {"step": 740, "loss": 3.718950796127319, "lr": 0.0006, "tok_s": 50373.67873035097}
38
+ {"step": 760, "loss": 3.6942316770553587, "lr": 0.0006, "tok_s": 50378.68010623984}
39
+ {"step": 780, "loss": 3.667694294452667, "lr": 0.0006, "tok_s": 50381.48179766727}
40
+ {"step": 800, "loss": 3.6529730796813964, "lr": 0.0006, "tok_s": 50377.88306012204}
41
+ {"step": 820, "loss": 3.636118769645691, "lr": 0.0006, "tok_s": 50379.51341918606}
42
+ {"step": 840, "loss": 3.621321976184845, "lr": 0.0006, "tok_s": 50379.233067473695}
43
+ {"step": 860, "loss": 3.599413800239563, "lr": 0.0006, "tok_s": 50382.71471680499}
44
+ {"step": 880, "loss": 3.5940122485160826, "lr": 0.0006, "tok_s": 50370.448236529955}
45
+ {"step": 900, "loss": 3.5738147139549254, "lr": 0.0006, "tok_s": 50371.711019226}
46
+ {"step": 920, "loss": 3.566121315956116, "lr": 0.0006, "tok_s": 50372.40546419553}
47
+ {"step": 940, "loss": 3.5609045147895815, "lr": 0.0006, "tok_s": 50369.0759717683}
48
+ {"step": 960, "loss": 3.534505832195282, "lr": 0.0006, "tok_s": 50374.16540284457}
49
+ {"step": 980, "loss": 3.5308042645454405, "lr": 0.0006, "tok_s": 50370.44696737549}
50
+ {"step": 1000, "loss": 3.5204467296600344, "lr": 0.0006, "tok_s": 50368.656600237875}
51
+ {"step": 1020, "loss": 3.5146281719207764, "lr": 0.0006, "tok_s": 50360.423738338155}
52
+ {"step": 1040, "loss": 3.486905074119568, "lr": 0.0006, "tok_s": 50363.738476024184}
53
+ {"step": 1060, "loss": 3.498551595211029, "lr": 0.0006, "tok_s": 50367.67886292377}
54
+ {"step": 1080, "loss": 3.488401412963867, "lr": 0.0006, "tok_s": 50371.610001203284}
55
+ {"step": 1100, "loss": 3.471914494037628, "lr": 0.0006, "tok_s": 50366.92980914957}
56
+ {"step": 1120, "loss": 3.4547880053520204, "lr": 0.0006, "tok_s": 50363.24122025229}
57
+ {"step": 1140, "loss": 3.4481158375740053, "lr": 0.0006, "tok_s": 50367.25397723054}
58
+ {"step": 1160, "loss": 3.4420236945152283, "lr": 0.0006, "tok_s": 50367.32319515499}
59
+ {"step": 1180, "loss": 3.443655323982239, "lr": 0.0006, "tok_s": 50365.537029822786}
60
+ {"step": 1200, "loss": 3.4253238439559937, "lr": 0.0006, "tok_s": 50364.3213885592}
61
+ {"step": 1220, "loss": 3.422685420513153, "lr": 0.0006, "tok_s": 50357.29002336478}
62
+ {"step": 1240, "loss": 3.413462817668915, "lr": 0.0006, "tok_s": 50364.86180730632}
63
+ {"step": 1260, "loss": 3.3992380857467652, "lr": 0.0006, "tok_s": 50365.81434521177}
64
+ {"step": 1280, "loss": 3.4017250537872314, "lr": 0.0006, "tok_s": 50360.62176375309}
65
+ {"step": 1300, "loss": 3.395185101032257, "lr": 0.0006, "tok_s": 50359.43484555891}
66
+ {"step": 1320, "loss": 3.386663091182709, "lr": 0.0006, "tok_s": 50361.95371831432}
67
+ {"step": 1340, "loss": 3.386570703983307, "lr": 0.0006, "tok_s": 50360.018292791196}
68
+ {"step": 1360, "loss": 3.37717227935791, "lr": 0.0006, "tok_s": 50360.13858216995}
69
+ {"step": 1380, "loss": 3.3683578252792357, "lr": 0.0006, "tok_s": 50362.59056938791}
70
+ {"step": 1400, "loss": 3.3574086308479307, "lr": 0.0006, "tok_s": 50367.077530065406}
71
+ {"step": 1420, "loss": 3.3459409832954408, "lr": 0.0006, "tok_s": 50368.306052530184}
72
+ {"step": 1440, "loss": 3.351805567741394, "lr": 0.0006, "tok_s": 50366.085895769036}
73
+ {"step": 1460, "loss": 3.349338412284851, "lr": 0.0006, "tok_s": 50362.918776298226}
74
+ {"step": 1480, "loss": 3.3528262257575987, "lr": 0.0006, "tok_s": 50366.965513578914}
75
+ {"step": 1500, "loss": 3.326026129722595, "lr": 0.0006, "tok_s": 50361.33625977772}
76
+ {"step": 1520, "loss": 3.3332749724388124, "lr": 0.0006, "tok_s": 50362.476150693976}
77
+ {"step": 1540, "loss": 3.3186570405960083, "lr": 0.0006, "tok_s": 50362.37470834519}
78
+ {"step": 1560, "loss": 3.326381599903107, "lr": 0.0006, "tok_s": 50363.40616366726}
79
+ {"step": 1580, "loss": 3.3143098831176756, "lr": 0.0006, "tok_s": 50366.3740032681}
80
+ {"step": 1600, "loss": 3.303647184371948, "lr": 0.0006, "tok_s": 50366.45988820335}
81
+ {"step": 1620, "loss": 3.3041441798210145, "lr": 0.0006, "tok_s": 50364.35253297463}
82
+ {"step": 1640, "loss": 3.3000163197517396, "lr": 0.0006, "tok_s": 50371.227682520854}
83
+ {"step": 1660, "loss": 3.3013440489768984, "lr": 0.0006, "tok_s": 50375.75189930125}
84
+ {"step": 1680, "loss": 3.2892932176589964, "lr": 0.0006, "tok_s": 50375.30852879603}
85
+ {"step": 1700, "loss": 3.281483030319214, "lr": 0.0006, "tok_s": 50372.8753230105}
86
+ {"step": 1720, "loss": 3.2839611768722534, "lr": 0.0006, "tok_s": 50365.161263114795}
87
+ {"step": 1740, "loss": 3.2778034806251526, "lr": 0.0006, "tok_s": 50365.621008536684}
88
+ {"step": 1760, "loss": 3.2767387986183167, "lr": 0.0006, "tok_s": 50362.62620999453}
89
+ {"step": 1780, "loss": 3.2648613452911377, "lr": 0.0006, "tok_s": 50359.492912887785}
90
+ {"step": 1800, "loss": 3.268283021450043, "lr": 0.0006, "tok_s": 50353.9341067899}
91
+ {"step": 1820, "loss": 3.263661098480225, "lr": 0.0006, "tok_s": 50363.4779086227}
92
+ {"step": 1840, "loss": 3.2567388653755187, "lr": 0.0006, "tok_s": 50360.198957736284}
93
+ {"step": 1860, "loss": 3.2560648560523986, "lr": 0.0006, "tok_s": 50360.82371206727}
94
+ {"step": 1880, "loss": 3.250918185710907, "lr": 0.0006, "tok_s": 50367.90942092212}
95
+ {"step": 1900, "loss": 3.2456185102462767, "lr": 0.0006, "tok_s": 50366.967186324015}
96
+ {"step": 1920, "loss": 3.24539510011673, "lr": 0.0006, "tok_s": 50364.89202957227}
97
+ {"step": 1940, "loss": 3.2371082663536073, "lr": 0.0006, "tok_s": 50360.922264816654}
98
+ {"step": 1960, "loss": 3.2425138473510744, "lr": 0.0006, "tok_s": 50367.19612272798}
99
+ {"step": 1980, "loss": 3.2364680767059326, "lr": 0.0006, "tok_s": 50368.85544031163}
100
+ {"step": 2000, "loss": 3.229717528820038, "lr": 0.0006, "tok_s": 50370.214597807935}
101
+ {"step": 2020, "loss": 3.229957103729248, "lr": 0.0006, "tok_s": 50368.395578155156}
102
+ {"step": 2040, "loss": 3.222858762741089, "lr": 0.0006, "tok_s": 50365.71588867058}
103
+ {"step": 2060, "loss": 3.2233617067337037, "lr": 0.0006, "tok_s": 50366.5096081587}
104
+ {"step": 2080, "loss": 3.21190664768219, "lr": 0.0006, "tok_s": 50369.32344462723}
105
+ {"step": 2100, "loss": 3.2206087470054627, "lr": 0.0006, "tok_s": 50368.81315710099}
106
+ {"step": 2120, "loss": 3.209831988811493, "lr": 0.0006, "tok_s": 50366.00099264635}
107
+ {"step": 2140, "loss": 3.222468078136444, "lr": 0.0006, "tok_s": 50370.005997758206}
108
+ {"step": 2160, "loss": 3.20089076757431, "lr": 0.0006, "tok_s": 50368.9796945229}
109
+ {"step": 2180, "loss": 3.202888286113739, "lr": 0.0006, "tok_s": 50368.795736208114}
110
+ {"step": 2200, "loss": 3.2039575815200805, "lr": 0.0006, "tok_s": 50374.92148034183}
111
+ {"step": 2220, "loss": 3.196550261974335, "lr": 0.0006, "tok_s": 50369.89650649877}
112
+ {"step": 2240, "loss": 3.192943048477173, "lr": 0.0006, "tok_s": 50365.647771042815}
113
+ {"step": 2260, "loss": 3.1894694447517393, "lr": 0.0006, "tok_s": 50364.36118420798}
114
+ {"step": 2280, "loss": 3.185406744480133, "lr": 0.0006, "tok_s": 50366.768534199626}
115
+ {"step": 2300, "loss": 3.188775873184204, "lr": 0.0006, "tok_s": 50365.339023428074}
116
+ {"step": 2320, "loss": 3.185637879371643, "lr": 0.0006, "tok_s": 50365.96684690598}
117
+ {"step": 2340, "loss": 3.180560517311096, "lr": 0.0006, "tok_s": 50371.4503116318}
118
+ {"step": 2360, "loss": 3.1701487064361573, "lr": 0.0006, "tok_s": 50373.50783466527}
119
+ {"step": 2380, "loss": 3.1692240476608275, "lr": 0.0006, "tok_s": 50366.82823349739}
120
+ {"step": 2400, "loss": 3.169828176498413, "lr": 0.0006, "tok_s": 50362.98994314746}
121
+ {"step": 2420, "loss": 3.1725643992424013, "lr": 0.0006, "tok_s": 50363.46568197598}
122
+ {"step": 2440, "loss": 3.1676795840263368, "lr": 0.0006, "tok_s": 50364.831008316876}
123
+ {"step": 2460, "loss": 3.1653196454048156, "lr": 0.0006, "tok_s": 50360.98812064299}
124
+ {"step": 2480, "loss": 3.1631489992141724, "lr": 0.0006, "tok_s": 50367.85814076448}
125
+ {"step": 2500, "loss": 3.1574719667434694, "lr": 0.0006, "tok_s": 50370.88592556927}
126
+ {"step": 2520, "loss": 3.1593925595283507, "lr": 0.0006, "tok_s": 50366.13423069093}
127
+ {"step": 2540, "loss": 3.1598495960235597, "lr": 0.0006, "tok_s": 50369.71940625962}
128
+ {"step": 2560, "loss": 3.1563698768615724, "lr": 0.0006, "tok_s": 50377.445826522744}
129
+ {"step": 2580, "loss": 3.1529322624206544, "lr": 0.0006, "tok_s": 50373.92815243526}
130
+ {"step": 2600, "loss": 3.1432958245277405, "lr": 0.0006, "tok_s": 50370.01482399743}
131
+ {"step": 2620, "loss": 3.151778995990753, "lr": 0.0006, "tok_s": 50367.93993526121}
132
+ {"step": 2640, "loss": 3.15283340215683, "lr": 0.0006, "tok_s": 50372.823051937885}
133
+ {"step": 2660, "loss": 3.15077862739563, "lr": 0.0006, "tok_s": 50367.684861904236}
134
+ {"step": 2680, "loss": 3.1459423899650574, "lr": 0.0006, "tok_s": 50369.369593681135}
135
+ {"step": 2700, "loss": 3.1483611345291136, "lr": 0.0006, "tok_s": 50367.77998054509}
136
+ {"step": 2720, "loss": 3.1315477013587953, "lr": 0.0006, "tok_s": 50371.11322457831}
137
+ {"step": 2740, "loss": 3.1327906966209413, "lr": 0.0006, "tok_s": 50371.19560655305}
138
+ {"step": 2760, "loss": 3.1356467247009276, "lr": 0.0006, "tok_s": 50367.30923619158}
139
+ {"step": 2780, "loss": 3.1263054609298706, "lr": 0.0006, "tok_s": 50365.03783521227}
140
+ {"step": 2800, "loss": 3.1220290780067446, "lr": 0.0006, "tok_s": 50367.36686022686}
141
+ {"step": 2820, "loss": 3.1191172122955324, "lr": 0.0006, "tok_s": 50368.037073781525}
142
+ {"step": 2840, "loss": 3.1271515130996703, "lr": 0.0006, "tok_s": 50368.817252743116}
143
+ {"step": 2860, "loss": 3.118371379375458, "lr": 0.0006, "tok_s": 50365.231801855974}
144
+ {"step": 2880, "loss": 3.120266079902649, "lr": 0.0006, "tok_s": 50370.91950108056}
145
+ {"step": 2900, "loss": 3.1216858506202696, "lr": 0.0006, "tok_s": 50368.59712734414}
146
+ {"step": 2920, "loss": 3.128999102115631, "lr": 0.0006, "tok_s": 50367.76665583817}
147
+ {"step": 2940, "loss": 3.1193459153175356, "lr": 0.0006, "tok_s": 50374.91790299865}
148
+ {"step": 2960, "loss": 3.121028017997742, "lr": 0.0006, "tok_s": 50373.98919573658}
149
+ {"step": 2980, "loss": 3.1082852602005007, "lr": 0.0006, "tok_s": 50368.03234374989}
150
+ {"step": 3000, "loss": 3.1095697045326234, "lr": 0.0006, "tok_s": 50368.92350893083}
151
+ {"step": 3020, "loss": 3.104125189781189, "lr": 0.0006, "tok_s": 50373.03248304779}
152
+ {"step": 3040, "loss": 3.1018962025642396, "lr": 0.0006, "tok_s": 50368.885840426454}
153
+ {"step": 3060, "loss": 3.1055035829544066, "lr": 0.0006, "tok_s": 50370.910386074785}
154
+ {"step": 3080, "loss": 3.097902750968933, "lr": 0.0006, "tok_s": 50369.13521504951}
155
+ {"step": 3100, "loss": 3.0915641665458677, "lr": 0.0006, "tok_s": 50369.54715295462}
156
+ {"step": 3120, "loss": 3.1072227120399476, "lr": 0.0006, "tok_s": 50364.592461614484}
157
+ {"step": 3140, "loss": 3.093355858325958, "lr": 0.0006, "tok_s": 50368.89414710558}
158
+ {"step": 3160, "loss": 3.089010214805603, "lr": 0.0006, "tok_s": 50365.959406370726}
159
+ {"step": 3180, "loss": 3.095322680473328, "lr": 0.0006, "tok_s": 50363.213652891805}
160
+ {"step": 3200, "loss": 3.0964788436889648, "lr": 0.0006, "tok_s": 50369.25797080211}
161
+ {"step": 3220, "loss": 3.10577677488327, "lr": 0.0006, "tok_s": 50366.56192380935}
162
+ {"step": 3240, "loss": 3.087931716442108, "lr": 0.0006, "tok_s": 50369.38920705465}
163
+ {"step": 3260, "loss": 3.0842342734336854, "lr": 0.0006, "tok_s": 50364.850906536085}
164
+ {"step": 3280, "loss": 3.0802849888801576, "lr": 0.0006, "tok_s": 50360.093891657736}
165
+ {"step": 3300, "loss": 3.0906956791877747, "lr": 0.0006, "tok_s": 50361.87015551386}
166
+ {"step": 3320, "loss": 3.0849864363670347, "lr": 0.0006, "tok_s": 50359.74052224592}
167
+ {"step": 3340, "loss": 3.085504877567291, "lr": 0.0006, "tok_s": 50356.23731273821}
168
+ {"step": 3360, "loss": 3.075977456569672, "lr": 0.0006, "tok_s": 50359.37706668138}
169
+ {"step": 3380, "loss": 3.0861980319023132, "lr": 0.0006, "tok_s": 50372.40604112878}
170
+ {"step": 3400, "loss": 3.0712910890579224, "lr": 0.0006, "tok_s": 50370.44217920239}
171
+ {"step": 3420, "loss": 3.0787760853767394, "lr": 0.0006, "tok_s": 50371.70450006096}
172
+ {"step": 3440, "loss": 3.0708940505981444, "lr": 0.0006, "tok_s": 50365.13398203566}
173
+ {"step": 3460, "loss": 3.0802491188049315, "lr": 0.0006, "tok_s": 50362.65383438291}
174
+ {"step": 3480, "loss": 3.079214537143707, "lr": 0.0006, "tok_s": 50363.616612271275}
175
+ {"step": 3500, "loss": 3.0654635429382324, "lr": 0.0006, "tok_s": 50372.11924924044}
176
+ {"step": 3520, "loss": 3.069408428668976, "lr": 0.0006, "tok_s": 50368.90401129059}
177
+ {"step": 3540, "loss": 3.071292555332184, "lr": 0.0006, "tok_s": 50368.42695834645}
178
+ {"step": 3560, "loss": 3.0752272963523866, "lr": 0.0006, "tok_s": 50361.02819936066}
179
+ {"step": 3580, "loss": 3.068881928920746, "lr": 0.0006, "tok_s": 50364.0289213076}
180
+ {"step": 3600, "loss": 3.071395516395569, "lr": 0.0006, "tok_s": 50363.11676366342}
181
+ {"step": 3620, "loss": 3.0628724098205566, "lr": 0.0006, "tok_s": 50362.50458230678}
182
+ {"step": 3640, "loss": 3.070946991443634, "lr": 0.0006, "tok_s": 50359.04550382977}
183
+ {"step": 3660, "loss": 3.0597490549087523, "lr": 0.0006, "tok_s": 50360.805893033226}
184
+ {"step": 3680, "loss": 3.066377651691437, "lr": 0.0006, "tok_s": 50360.72025804551}
185
+ {"step": 3700, "loss": 3.06362179517746, "lr": 0.0006, "tok_s": 50359.1441649455}
186
+ {"step": 3720, "loss": 3.056973707675934, "lr": 0.0006, "tok_s": 50362.49864223086}
187
+ {"step": 3740, "loss": 3.0527987122535705, "lr": 0.0006, "tok_s": 50365.69177927745}
188
+ {"step": 3760, "loss": 3.0512983441352843, "lr": 0.0006, "tok_s": 50365.42346290032}
189
+ {"step": 3780, "loss": 3.0522939562797546, "lr": 0.0006, "tok_s": 50360.74597734455}
190
+ {"step": 3800, "loss": 3.053767907619476, "lr": 0.0006, "tok_s": 50362.55325637969}
191
+ {"step": 3820, "loss": 3.04669303894043, "lr": 0.0005609064981513237, "tok_s": 50363.96876725364}
192
+ {"step": 3840, "loss": 3.0445285201072694, "lr": 0.0005125842723878462, "tok_s": 50368.51533071292}
193
+ {"step": 3860, "loss": 3.0375494599342345, "lr": 0.0004827194944539712, "tok_s": 50367.89003943295}
194
+ {"step": 3880, "loss": 3.0345866322517394, "lr": 0.0004590463745471515, "tok_s": 50368.46491458596}
195
+ {"step": 3900, "loss": 3.0270234823226927, "lr": 0.0004388133626026285, "tok_s": 50370.28722748961}
196
+ {"step": 3920, "loss": 3.0222036361694338, "lr": 0.000420851068597545, "tok_s": 50370.17571591987}
197
+ {"step": 3940, "loss": 3.013981819152832, "lr": 0.00040453249075661876, "tok_s": 50372.37880989385}
198
+ {"step": 3960, "loss": 3.01424001455307, "lr": 0.00038947504965686095, "tok_s": 50371.7039808355}
199
+ {"step": 3980, "loss": 3.0062872052192686, "lr": 0.000375424929557729, "tok_s": 50378.14360348276}
200
+ {"step": 4000, "loss": 3.0049508571624757, "lr": 0.00036220351177672794, "tok_s": 50379.4737150075}
201
+ {"step": 4020, "loss": 3.002310800552368, "lr": 0.0003496794507866152, "tok_s": 50376.86745724493}
202
+ {"step": 4040, "loss": 2.993136239051819, "lr": 0.0003377528171635386, "tok_s": 50372.42121447796}
203
+ {"step": 4060, "loss": 2.992757511138916, "lr": 0.00032634548705926626, "tok_s": 50371.321833799}
204
+ {"step": 4080, "loss": 2.9841498017311094, "lr": 0.00031539501058484577, "tok_s": 50366.712699622156}
205
+ {"step": 4100, "loss": 2.98936128616333, "lr": 0.00030485053354754334, "tok_s": 50373.20879885303}
206
+ {"step": 4120, "loss": 2.977723276615143, "lr": 0.00029466998985632013, "tok_s": 50378.21083160903}
207
+ {"step": 4140, "loss": 2.9701634645462036, "lr": 0.0002848181118123862, "tok_s": 50375.00179759999}
208
+ {"step": 4160, "loss": 2.988078737258911, "lr": 0.000275264984658754, "tok_s": 50375.43229554511}
209
+ {"step": 4180, "loss": 2.9809946775436402, "lr": 0.0002659849737873323, "tok_s": 50370.452159371445}
210
+ {"step": 4200, "loss": 2.9741480231285093, "lr": 0.0002569559134994179, "tok_s": 50372.19724984914}
211
+ {"step": 4220, "loss": 2.9675135731697084, "lr": 0.00024815848336191376, "tok_s": 50371.85409513139}
212
+ {"step": 4240, "loss": 2.974126946926117, "lr": 0.00023957572171486765, "tok_s": 50366.25916362766}
213
+ {"step": 4260, "loss": 2.9661878824234007, "lr": 0.0002311926411735683, "tok_s": 50362.92667730802}
214
+ {"step": 4280, "loss": 2.9653461813926696, "lr": 0.0002229959211471338, "tok_s": 50365.8565657094}
215
+ {"step": 4300, "loss": 2.9630411863327026, "lr": 0.00021497365931813446, "tok_s": 50366.109428750446}
216
+ {"step": 4320, "loss": 2.9579756259918213, "lr": 0.00020711516882674415, "tok_s": 50366.67936043967}
217
+ {"step": 4340, "loss": 2.957149875164032, "lr": 0.00019941081128766396, "tok_s": 50362.23606920707}
218
+ {"step": 4360, "loss": 2.951228082180023, "lr": 0.00019185185819315875, "tok_s": 50365.82778424227}
219
+ {"step": 4380, "loss": 2.958230221271515, "lr": 0.00018443037501815989, "tok_s": 50367.44265420377}
220
+ {"step": 4400, "loss": 2.9549214482307433, "lr": 0.00017713912364145464, "tok_s": 50367.89811505163}
221
+ {"step": 4420, "loss": 2.9486687541007996, "lr": 0.00016997147966456131, "tok_s": 50481.350438563284}
222
+ {"step": 4440, "loss": 2.9544939279556273, "lr": 0.00016292136193923112, "tok_s": 50542.27322178109}
223
+ {"step": 4460, "loss": 2.9502453923225405, "lr": 0.00015598317216998812, "tok_s": 50556.77445617177}
224
+ {"step": 4480, "loss": 2.9451239466667176, "lr": 0.00014915174288526275, "tok_s": 50558.55189331388}
225
+ {"step": 4500, "loss": 2.9426443696022035, "lr": 0.00014242229240208165, "tok_s": 50564.620810065586}
226
+ {"step": 4520, "loss": 2.944092261791229, "lr": 0.00013579038566856907, "tok_s": 50564.0560981644}
227
+ {"step": 4540, "loss": 2.936077344417572, "lr": 0.00012925190007297358, "tok_s": 50562.4841099072}
228
+ {"step": 4560, "loss": 2.938389253616333, "lr": 0.0001228029954703451, "tok_s": 50564.53134127988}
229
+ {"step": 4580, "loss": 2.9421222805976868, "lr": 0.00011644008780788571, "tok_s": 50561.70216324656}
230
+ {"step": 4600, "loss": 2.940841829776764, "lr": 0.00011015982583457378, "tok_s": 50556.39164617272}
231
+ {"step": 4620, "loss": 2.9273490071296693, "lr": 0.00010395907046536478, "tok_s": 50486.87667619206}
232
+ {"step": 4640, "loss": 2.9367039680480955, "lr": 9.783487643927846e-05, "tok_s": 50538.73162112472}
233
+ {"step": 4660, "loss": 2.9295316457748415, "lr": 9.178447596720887e-05, "tok_s": 50553.55117600604}
234
+ {"step": 4680, "loss": 2.927152705192566, "lr": 8.580526411184025e-05, "tok_s": 50522.58288650106}
235
+ {"step": 4700, "loss": 2.923268437385559, "lr": 7.98947856805662e-05, "tok_s": 50501.552229447654}
236
+ {"step": 4720, "loss": 2.9241204977035524, "lr": 7.405072344432876e-05, "tok_s": 50526.73577754924}
237
+ {"step": 4740, "loss": 2.9223405003547667, "lr": 6.827088752203553e-05, "tok_s": 50521.72945441874}
238
+ {"step": 4760, "loss": 2.9310539960861206, "lr": 6.255320579263513e-05, "tok_s": 50546.140050920156}
K22/run/srodowisko.txt ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-22T11:59:50Z ARM K22 GPU 3 MINI 0 DATA_SEED 20261022
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ a607076bd6e11cad879b8213b37492a4b4f5cbf8fa96205b7ebaeace8a25213e /home/ubuntu/kolektyw/dowody400m/kod/ramie.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
7
+ 2.6.0+cu124 12.4
8
+ accelerate==1.15.0
9
+ annotated-doc==0.0.5
10
+ anyio==4.15.1
11
+ certifi==2026.10.19
12
+ click==8.5.0
13
+ filelock==4.2.4
14
+ fsspec==2026.9.0
15
+ h11==0.16.0
16
+ hf-xet==1.7.0
17
+ httpcore==1.0.9
18
+ httpx==0.28.1
19
+ huggingface_hub==1.33.0
20
+ idna==3.20
21
+ Jinja2==3.1.6
22
+ markdown-it-py==4.2.0
23
+ MarkupSafe==3.0.4
24
+ mdurl==0.1.2
25
+ mpmath==1.3.0
26
+ networkx==3.7
27
+ numpy==2.2.6
28
+ nvidia-cublas-cu12==12.4.5.8
29
+ nvidia-cuda-cupti-cu12==12.4.127
30
+ nvidia-cuda-nvrtc-cu12==12.4.127
31
+ nvidia-cuda-runtime-cu12==12.4.127
32
+ nvidia-cudnn-cu12==9.1.0.70
33
+ nvidia-cufft-cu12==11.2.1.3
34
+ nvidia-curand-cu12==10.3.5.147
35
+ nvidia-cusolver-cu12==11.6.1.9
36
+ nvidia-cusparse-cu12==12.3.1.170
37
+ nvidia-cusparselt-cu12==0.6.2
38
+ nvidia-nccl-cu12==2.21.5
39
+ nvidia-nvjitlink-cu12==12.4.127
40
+ nvidia-nvtx-cu12==12.4.127
41
+ packaging==26.3
42
+ pillow==12.3.0
43
+ psutil==7.2.2
44
+ Pygments==2.21.0
45
+ PyYAML==6.0.3
46
+ regex==2026.9.29
47
+ rich==15.0.0
48
+ safetensors==0.8.0
49
+ setuptools==84.2.0
50
+ shellingham==1.5.4
51
+ sympy==1.13.1
52
+ tokenizers==0.23.3
53
+ torch==2.6.0+cu124
54
+ torchvision==0.21.0+cu124
55
+ tqdm==4.70.1
56
+ transformers==5.17.0
57
+ triton==3.2.0
58
+ typer==0.27.3
59
+ typing_extensions==4.16.0
60
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
61
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
K22/run/srodowisko_K22_mini0.txt ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-22T11:59:50Z ARM K22 GPU 3 MINI 0 DATA_SEED 20261022
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ a607076bd6e11cad879b8213b37492a4b4f5cbf8fa96205b7ebaeace8a25213e /home/ubuntu/kolektyw/dowody400m/kod/ramie.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
7
+ 2.6.0+cu124 12.4
8
+ accelerate==1.15.0
9
+ annotated-doc==0.0.5
10
+ anyio==4.15.1
11
+ certifi==2026.10.19
12
+ click==8.5.0
13
+ filelock==4.2.4
14
+ fsspec==2026.9.0
15
+ h11==0.16.0
16
+ hf-xet==1.7.0
17
+ httpcore==1.0.9
18
+ httpx==0.28.1
19
+ huggingface_hub==1.33.0
20
+ idna==3.20
21
+ Jinja2==3.1.6
22
+ markdown-it-py==4.2.0
23
+ MarkupSafe==3.0.4
24
+ mdurl==0.1.2
25
+ mpmath==1.3.0
26
+ networkx==3.7
27
+ numpy==2.2.6
28
+ nvidia-cublas-cu12==12.4.5.8
29
+ nvidia-cuda-cupti-cu12==12.4.127
30
+ nvidia-cuda-nvrtc-cu12==12.4.127
31
+ nvidia-cuda-runtime-cu12==12.4.127
32
+ nvidia-cudnn-cu12==9.1.0.70
33
+ nvidia-cufft-cu12==11.2.1.3
34
+ nvidia-curand-cu12==10.3.5.147
35
+ nvidia-cusolver-cu12==11.6.1.9
36
+ nvidia-cusparse-cu12==12.3.1.170
37
+ nvidia-cusparselt-cu12==0.6.2
38
+ nvidia-nccl-cu12==2.21.5
39
+ nvidia-nvjitlink-cu12==12.4.127
40
+ nvidia-nvtx-cu12==12.4.127
41
+ packaging==26.3
42
+ pillow==12.3.0
43
+ psutil==7.2.2
44
+ Pygments==2.21.0
45
+ PyYAML==6.0.3
46
+ regex==2026.9.29
47
+ rich==15.0.0
48
+ safetensors==0.8.0
49
+ setuptools==84.2.0
50
+ shellingham==1.5.4
51
+ sympy==1.13.1
52
+ tokenizers==0.23.3
53
+ torch==2.6.0+cu124
54
+ torchvision==0.21.0+cu124
55
+ tqdm==4.70.1
56
+ transformers==5.17.0
57
+ triton==3.2.0
58
+ typer==0.27.3
59
+ typing_extensions==4.16.0
60
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
61
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
K22/run/train.log ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [13:59:56] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [13:59:56] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [13:59:58] model GPT-ref: 180.9M param (L22 d768 h12)
4
+ [13:59:59] optimizer=muon muon_lr=0.02 (mult=33.3x)
5
+ [13:59:59] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [13:59:59] r13: mixer_layout=AAAAAAAAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [14:00:00] torch.compile enabled
8
+ [14:00:00] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [14:04:48] step 20/4768 loss 10.3787 lr 2.40e-05 gnorm 2.38 36,394 tok/s peakVRAM 13.7GB
10
+ [14:08:15] step 40/4768 loss 9.6271 lr 4.80e-05 gnorm 1.40 50,498 tok/s peakVRAM 13.7GB
11
+ [14:11:43] step 60/4768 loss 8.8371 lr 7.20e-05 gnorm 1.11 50,510 tok/s peakVRAM 13.7GB
12
+ [14:15:11] step 80/4768 loss 8.1619 lr 9.60e-05 gnorm 1.04 50,497 tok/s peakVRAM 13.7GB
13
+ [14:18:38] step 100/4768 loss 7.6838 lr 1.20e-04 gnorm 0.69 50,466 tok/s peakVRAM 13.7GB
14
+ [14:22:06] step 120/4768 loss 7.2235 lr 1.44e-04 gnorm 0.83 50,441 tok/s peakVRAM 13.7GB
15
+ [14:25:34] step 140/4768 loss 6.8162 lr 1.68e-04 gnorm 0.88 50,431 tok/s peakVRAM 13.7GB
16
+ [14:29:02] step 160/4768 loss 6.4518 lr 1.92e-04 gnorm 0.96 50,418 tok/s peakVRAM 13.7GB
17
+ [14:32:30] step 180/4768 loss 6.1645 lr 2.16e-04 gnorm 0.75 50,412 tok/s peakVRAM 13.7GB
18
+ [14:35:58] step 200/4768 loss 5.9285 lr 2.40e-04 gnorm 0.78 50,411 tok/s peakVRAM 13.7GB
19
+ [14:39:26] step 220/4768 loss 5.7298 lr 2.64e-04 gnorm 0.55 50,409 tok/s peakVRAM 13.7GB
20
+ [14:42:54] step 240/4768 loss 5.5502 lr 2.88e-04 gnorm 0.49 50,408 tok/s peakVRAM 13.7GB
21
+ [14:46:22] step 260/4768 loss 5.3819 lr 3.12e-04 gnorm 0.59 50,407 tok/s peakVRAM 13.7GB
22
+ [14:49:50] step 280/4768 loss 5.2249 lr 3.36e-04 gnorm 0.58 50,410 tok/s peakVRAM 13.7GB
23
+ [14:53:18] step 300/4768 loss 5.0776 lr 3.60e-04 gnorm 0.52 50,409 tok/s peakVRAM 13.7GB
24
+ [14:56:46] step 320/4768 loss 4.9310 lr 3.84e-04 gnorm 0.63 50,410 tok/s peakVRAM 13.7GB
25
+ [15:00:14] step 340/4768 loss 4.7736 lr 4.08e-04 gnorm 0.59 50,408 tok/s peakVRAM 13.7GB
26
+ [15:03:42] step 360/4768 loss 4.6485 lr 4.32e-04 gnorm 0.58 50,401 tok/s peakVRAM 13.7GB
27
+ [15:07:10] step 380/4768 loss 4.5306 lr 4.56e-04 gnorm 0.55 50,404 tok/s peakVRAM 13.7GB
28
+ [15:10:38] step 400/4768 loss 4.4408 lr 4.80e-04 gnorm 0.48 50,402 tok/s peakVRAM 13.7GB
29
+ [15:14:06] step 420/4768 loss 4.3643 lr 5.04e-04 gnorm 0.43 50,404 tok/s peakVRAM 13.7GB
30
+ [15:17:34] step 440/4768 loss 4.2946 lr 5.28e-04 gnorm 0.50 50,403 tok/s peakVRAM 13.7GB
31
+ [15:21:02] step 460/4768 loss 4.2252 lr 5.52e-04 gnorm 0.44 50,392 tok/s peakVRAM 13.7GB
32
+ [15:24:31] step 480/4768 loss 4.1773 lr 5.76e-04 gnorm 0.43 50,388 tok/s peakVRAM 13.7GB
33
+ [15:27:59] step 500/4768 loss 4.1259 lr 6.00e-04 gnorm 0.40 50,392 tok/s peakVRAM 13.7GB
34
+ [15:31:27] step 520/4768 loss 4.0899 lr 6.00e-04 gnorm 0.33 50,395 tok/s peakVRAM 13.7GB
35
+ [15:34:55] step 540/4768 loss 4.0379 lr 6.00e-04 gnorm 0.25 50,396 tok/s peakVRAM 13.7GB
36
+ [15:38:23] step 560/4768 loss 3.9871 lr 6.00e-04 gnorm 0.33 50,389 tok/s peakVRAM 13.7GB
37
+ [15:41:51] step 580/4768 loss 3.9425 lr 6.00e-04 gnorm 0.29 50,387 tok/s peakVRAM 13.7GB
38
+ [15:45:19] step 600/4768 loss 3.8993 lr 6.00e-04 gnorm 0.26 50,382 tok/s peakVRAM 13.7GB
39
+ [15:48:47] step 620/4768 loss 3.8736 lr 6.00e-04 gnorm 0.25 50,384 tok/s peakVRAM 13.7GB
40
+ [15:52:15] step 640/4768 loss 3.8464 lr 6.00e-04 gnorm 0.28 50,384 tok/s peakVRAM 13.7GB
41
+ [15:55:44] step 660/4768 loss 3.8137 lr 6.00e-04 gnorm 0.29 50,378 tok/s peakVRAM 13.7GB
42
+ [15:59:12] step 680/4768 loss 3.7883 lr 6.00e-04 gnorm 0.22 50,379 tok/s peakVRAM 13.7GB
43
+ [16:02:40] step 700/4768 loss 3.7559 lr 6.00e-04 gnorm 0.25 50,379 tok/s peakVRAM 13.7GB
44
+ [16:06:08] step 720/4768 loss 3.7238 lr 6.00e-04 gnorm 0.25 50,377 tok/s peakVRAM 13.7GB
45
+ [16:09:36] step 740/4768 loss 3.7190 lr 6.00e-04 gnorm 0.28 50,374 tok/s peakVRAM 13.7GB
46
+ [16:13:04] step 760/4768 loss 3.6942 lr 6.00e-04 gnorm 0.22 50,379 tok/s peakVRAM 13.7GB
47
+ [16:16:32] step 780/4768 loss 3.6677 lr 6.00e-04 gnorm 0.22 50,381 tok/s peakVRAM 13.7GB
48
+ [16:20:01] step 800/4768 loss 3.6530 lr 6.00e-04 gnorm 0.20 50,378 tok/s peakVRAM 13.7GB
49
+ [16:23:29] step 820/4768 loss 3.6361 lr 6.00e-04 gnorm 0.21 50,380 tok/s peakVRAM 13.7GB
50
+ [16:26:57] step 840/4768 loss 3.6213 lr 6.00e-04 gnorm 0.21 50,379 tok/s peakVRAM 13.7GB
51
+ [16:30:25] step 860/4768 loss 3.5994 lr 6.00e-04 gnorm 0.18 50,383 tok/s peakVRAM 13.7GB
52
+ [16:33:53] step 880/4768 loss 3.5940 lr 6.00e-04 gnorm 0.21 50,370 tok/s peakVRAM 13.7GB
53
+ [16:37:21] step 900/4768 loss 3.5738 lr 6.00e-04 gnorm 0.18 50,372 tok/s peakVRAM 13.7GB
54
+ [16:40:49] step 920/4768 loss 3.5661 lr 6.00e-04 gnorm 0.19 50,372 tok/s peakVRAM 13.7GB
55
+ [16:44:18] step 940/4768 loss 3.5609 lr 6.00e-04 gnorm 0.18 50,369 tok/s peakVRAM 13.7GB
56
+ [16:47:46] step 960/4768 loss 3.5345 lr 6.00e-04 gnorm 0.17 50,374 tok/s peakVRAM 13.7GB
57
+ [16:51:14] step 980/4768 loss 3.5308 lr 6.00e-04 gnorm 0.17 50,370 tok/s peakVRAM 13.7GB
58
+ [16:54:42] step 1000/4768 loss 3.5204 lr 6.00e-04 gnorm 0.17 50,369 tok/s peakVRAM 13.7GB
59
+ [16:58:10] step 1020/4768 loss 3.5146 lr 6.00e-04 gnorm 0.19 50,360 tok/s peakVRAM 13.7GB
60
+ [17:01:39] step 1040/4768 loss 3.4869 lr 6.00e-04 gnorm 0.18 50,364 tok/s peakVRAM 13.7GB
61
+ [17:05:07] step 1060/4768 loss 3.4986 lr 6.00e-04 gnorm 0.18 50,368 tok/s peakVRAM 13.7GB
62
+ [17:08:35] step 1080/4768 loss 3.4884 lr 6.00e-04 gnorm 0.16 50,372 tok/s peakVRAM 13.7GB
63
+ [17:12:03] step 1100/4768 loss 3.4719 lr 6.00e-04 gnorm 0.21 50,367 tok/s peakVRAM 13.7GB
64
+ [17:15:31] step 1120/4768 loss 3.4548 lr 6.00e-04 gnorm 0.18 50,363 tok/s peakVRAM 13.7GB
65
+ [17:18:59] step 1140/4768 loss 3.4481 lr 6.00e-04 gnorm 0.16 50,367 tok/s peakVRAM 13.7GB
66
+ [17:22:28] step 1160/4768 loss 3.4420 lr 6.00e-04 gnorm 0.17 50,367 tok/s peakVRAM 13.7GB
67
+ [17:25:56] step 1180/4768 loss 3.4437 lr 6.00e-04 gnorm 0.16 50,366 tok/s peakVRAM 13.7GB
68
+ [17:29:24] step 1200/4768 loss 3.4253 lr 6.00e-04 gnorm 0.19 50,364 tok/s peakVRAM 13.7GB
69
+ [17:32:52] step 1220/4768 loss 3.4227 lr 6.00e-04 gnorm 0.14 50,357 tok/s peakVRAM 13.7GB
70
+ [17:36:20] step 1240/4768 loss 3.4135 lr 6.00e-04 gnorm 0.15 50,365 tok/s peakVRAM 13.7GB
71
+ [17:39:49] step 1260/4768 loss 3.3992 lr 6.00e-04 gnorm 0.15 50,366 tok/s peakVRAM 13.7GB
72
+ [17:43:17] step 1280/4768 loss 3.4017 lr 6.00e-04 gnorm 0.15 50,361 tok/s peakVRAM 13.7GB
73
+ [17:46:45] step 1300/4768 loss 3.3952 lr 6.00e-04 gnorm 0.16 50,359 tok/s peakVRAM 13.7GB
74
+ [17:50:13] step 1320/4768 loss 3.3867 lr 6.00e-04 gnorm 0.16 50,362 tok/s peakVRAM 13.7GB
75
+ [17:53:42] step 1340/4768 loss 3.3866 lr 6.00e-04 gnorm 0.15 50,360 tok/s peakVRAM 13.7GB
76
+ [17:57:10] step 1360/4768 loss 3.3772 lr 6.00e-04 gnorm 0.13 50,360 tok/s peakVRAM 13.7GB
77
+ [18:00:38] step 1380/4768 loss 3.3684 lr 6.00e-04 gnorm 0.13 50,363 tok/s peakVRAM 13.7GB
78
+ [18:04:06] step 1400/4768 loss 3.3574 lr 6.00e-04 gnorm 0.14 50,367 tok/s peakVRAM 13.7GB
79
+ [18:07:34] step 1420/4768 loss 3.3459 lr 6.00e-04 gnorm 0.16 50,368 tok/s peakVRAM 13.7GB
80
+ [18:11:02] step 1440/4768 loss 3.3518 lr 6.00e-04 gnorm 0.16 50,366 tok/s peakVRAM 13.7GB
81
+ [18:14:31] step 1460/4768 loss 3.3493 lr 6.00e-04 gnorm 0.13 50,363 tok/s peakVRAM 13.7GB
82
+ [18:17:59] step 1480/4768 loss 3.3528 lr 6.00e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
83
+ [18:21:27] step 1500/4768 loss 3.3260 lr 6.00e-04 gnorm 0.16 50,361 tok/s peakVRAM 13.7GB
84
+ [18:24:55] step 1520/4768 loss 3.3333 lr 6.00e-04 gnorm 0.14 50,362 tok/s peakVRAM 13.7GB
85
+ [18:28:23] step 1540/4768 loss 3.3187 lr 6.00e-04 gnorm 0.15 50,362 tok/s peakVRAM 13.7GB
86
+ [18:31:52] step 1560/4768 loss 3.3264 lr 6.00e-04 gnorm 0.16 50,363 tok/s peakVRAM 13.7GB
87
+ [18:35:20] step 1580/4768 loss 3.3143 lr 6.00e-04 gnorm 0.12 50,366 tok/s peakVRAM 13.7GB
88
+ [18:38:48] step 1600/4768 loss 3.3036 lr 6.00e-04 gnorm 0.15 50,366 tok/s peakVRAM 13.7GB
89
+ [18:42:16] step 1620/4768 loss 3.3041 lr 6.00e-04 gnorm 0.14 50,364 tok/s peakVRAM 13.7GB
90
+ [18:45:44] step 1640/4768 loss 3.3000 lr 6.00e-04 gnorm 0.14 50,371 tok/s peakVRAM 13.7GB
91
+ [18:49:13] step 1660/4768 loss 3.3013 lr 6.00e-04 gnorm 0.13 50,376 tok/s peakVRAM 13.7GB
92
+ [18:52:41] step 1680/4768 loss 3.2893 lr 6.00e-04 gnorm 0.13 50,375 tok/s peakVRAM 13.7GB
93
+ [18:56:09] step 1700/4768 loss 3.2815 lr 6.00e-04 gnorm 0.12 50,373 tok/s peakVRAM 13.7GB
94
+ [18:59:37] step 1720/4768 loss 3.2840 lr 6.00e-04 gnorm 0.15 50,365 tok/s peakVRAM 13.7GB
95
+ [19:03:05] step 1740/4768 loss 3.2778 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
96
+ [19:06:34] step 1760/4768 loss 3.2767 lr 6.00e-04 gnorm 0.13 50,363 tok/s peakVRAM 13.7GB
97
+ [19:10:02] step 1780/4768 loss 3.2649 lr 6.00e-04 gnorm 0.14 50,359 tok/s peakVRAM 13.7GB
98
+ [19:13:30] step 1800/4768 loss 3.2683 lr 6.00e-04 gnorm 0.13 50,354 tok/s peakVRAM 13.7GB
99
+ [19:16:58] step 1820/4768 loss 3.2637 lr 6.00e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
100
+ [19:20:26] step 1840/4768 loss 3.2567 lr 6.00e-04 gnorm 0.12 50,360 tok/s peakVRAM 13.7GB
101
+ [19:23:55] step 1860/4768 loss 3.2561 lr 6.00e-04 gnorm 0.13 50,361 tok/s peakVRAM 13.7GB
102
+ [19:27:23] step 1880/4768 loss 3.2509 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
103
+ [19:30:51] step 1900/4768 loss 3.2456 lr 6.00e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
104
+ [19:34:19] step 1920/4768 loss 3.2454 lr 6.00e-04 gnorm 0.11 50,365 tok/s peakVRAM 13.7GB
105
+ [19:37:47] step 1940/4768 loss 3.2371 lr 6.00e-04 gnorm 0.13 50,361 tok/s peakVRAM 13.7GB
106
+ [19:41:16] step 1960/4768 loss 3.2425 lr 6.00e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
107
+ [19:44:44] step 1980/4768 loss 3.2365 lr 6.00e-04 gnorm 0.12 50,369 tok/s peakVRAM 13.7GB
108
+ [19:48:12] step 2000/4768 loss 3.2297 lr 6.00e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
109
+ [19:51:40] step 2020/4768 loss 3.2300 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
110
+ [19:55:08] step 2040/4768 loss 3.2229 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
111
+ [19:58:36] step 2060/4768 loss 3.2234 lr 6.00e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
112
+ [20:02:05] step 2080/4768 loss 3.2119 lr 6.00e-04 gnorm 0.13 50,369 tok/s peakVRAM 13.7GB
113
+ [20:05:33] step 2100/4768 loss 3.2206 lr 6.00e-04 gnorm 0.11 50,369 tok/s peakVRAM 13.7GB
114
+ [20:09:01] step 2120/4768 loss 3.2098 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
115
+ [20:12:29] step 2140/4768 loss 3.2225 lr 6.00e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
116
+ [20:15:57] step 2160/4768 loss 3.2009 lr 6.00e-04 gnorm 0.13 50,369 tok/s peakVRAM 13.7GB
117
+ [20:19:26] step 2180/4768 loss 3.2029 lr 6.00e-04 gnorm 0.11 50,369 tok/s peakVRAM 13.7GB
118
+ [20:22:54] step 2200/4768 loss 3.2040 lr 6.00e-04 gnorm 0.12 50,375 tok/s peakVRAM 13.7GB
119
+ [20:26:22] step 2220/4768 loss 3.1966 lr 6.00e-04 gnorm 0.15 50,370 tok/s peakVRAM 13.7GB
120
+ [20:29:50] step 2240/4768 loss 3.1929 lr 6.00e-04 gnorm 0.12 50,366 tok/s peakVRAM 13.7GB
121
+ [20:33:18] step 2260/4768 loss 3.1895 lr 6.00e-04 gnorm 0.12 50,364 tok/s peakVRAM 13.7GB
122
+ [20:36:46] step 2280/4768 loss 3.1854 lr 6.00e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
123
+ [20:40:15] step 2300/4768 loss 3.1888 lr 6.00e-04 gnorm 0.12 50,365 tok/s peakVRAM 13.7GB
124
+ [20:43:43] step 2320/4768 loss 3.1856 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
125
+ [20:47:11] step 2340/4768 loss 3.1806 lr 6.00e-04 gnorm 0.14 50,371 tok/s peakVRAM 13.7GB
126
+ [20:50:39] step 2360/4768 loss 3.1701 lr 6.00e-04 gnorm 0.12 50,374 tok/s peakVRAM 13.7GB
127
+ [20:54:07] step 2380/4768 loss 3.1692 lr 6.00e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
128
+ [20:57:36] step 2400/4768 loss 3.1698 lr 6.00e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
129
+ [21:01:04] step 2420/4768 loss 3.1726 lr 6.00e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
130
+ [21:04:32] step 2440/4768 loss 3.1677 lr 6.00e-04 gnorm 0.14 50,365 tok/s peakVRAM 13.7GB
131
+ [21:08:00] step 2460/4768 loss 3.1653 lr 6.00e-04 gnorm 0.15 50,361 tok/s peakVRAM 13.7GB
132
+ [21:11:28] step 2480/4768 loss 3.1631 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
133
+ [21:14:57] step 2500/4768 loss 3.1575 lr 6.00e-04 gnorm 0.11 50,371 tok/s peakVRAM 13.7GB
134
+ [21:18:25] step 2520/4768 loss 3.1594 lr 6.00e-04 gnorm 0.12 50,366 tok/s peakVRAM 13.7GB
135
+ [21:21:53] step 2540/4768 loss 3.1598 lr 6.00e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
136
+ [21:25:21] step 2560/4768 loss 3.1564 lr 6.00e-04 gnorm 0.13 50,377 tok/s peakVRAM 13.7GB
137
+ [21:28:49] step 2580/4768 loss 3.1529 lr 6.00e-04 gnorm 0.14 50,374 tok/s peakVRAM 13.7GB
138
+ [21:32:17] step 2600/4768 loss 3.1433 lr 6.00e-04 gnorm 0.12 50,370 tok/s peakVRAM 13.7GB
139
+ [21:35:46] step 2620/4768 loss 3.1518 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
140
+ [21:39:14] step 2640/4768 loss 3.1528 lr 6.00e-04 gnorm 0.15 50,373 tok/s peakVRAM 13.7GB
141
+ [21:42:42] step 2660/4768 loss 3.1508 lr 6.00e-04 gnorm 0.14 50,368 tok/s peakVRAM 13.7GB
142
+ [21:46:10] step 2680/4768 loss 3.1459 lr 6.00e-04 gnorm 0.13 50,369 tok/s peakVRAM 13.7GB
143
+ [21:49:38] step 2700/4768 loss 3.1484 lr 6.00e-04 gnorm 0.14 50,368 tok/s peakVRAM 13.7GB
144
+ [21:53:06] step 2720/4768 loss 3.1315 lr 6.00e-04 gnorm 0.15 50,371 tok/s peakVRAM 13.7GB
145
+ [21:56:35] step 2740/4768 loss 3.1328 lr 6.00e-04 gnorm 0.14 50,371 tok/s peakVRAM 13.7GB
146
+ [22:00:03] step 2760/4768 loss 3.1356 lr 6.00e-04 gnorm 0.14 50,367 tok/s peakVRAM 13.7GB
147
+ [22:03:31] step 2780/4768 loss 3.1263 lr 6.00e-04 gnorm 0.14 50,365 tok/s peakVRAM 13.7GB
148
+ [22:06:59] step 2800/4768 loss 3.1220 lr 6.00e-04 gnorm 0.15 50,367 tok/s peakVRAM 13.7GB
149
+ [22:10:27] step 2820/4768 loss 3.1191 lr 6.00e-04 gnorm 0.14 50,368 tok/s peakVRAM 13.7GB
150
+ [22:13:56] step 2840/4768 loss 3.1272 lr 6.00e-04 gnorm 0.16 50,369 tok/s peakVRAM 13.7GB
151
+ [22:17:24] step 2860/4768 loss 3.1184 lr 6.00e-04 gnorm 0.15 50,365 tok/s peakVRAM 13.7GB
152
+ [22:20:52] step 2880/4768 loss 3.1203 lr 6.00e-04 gnorm 0.16 50,371 tok/s peakVRAM 13.7GB
153
+ [22:24:20] step 2900/4768 loss 3.1217 lr 6.00e-04 gnorm 0.14 50,369 tok/s peakVRAM 13.7GB
154
+ [22:27:48] step 2920/4768 loss 3.1290 lr 6.00e-04 gnorm 0.17 50,368 tok/s peakVRAM 13.7GB
155
+ [22:31:16] step 2940/4768 loss 3.1193 lr 6.00e-04 gnorm 0.17 50,375 tok/s peakVRAM 13.7GB
156
+ [22:34:45] step 2960/4768 loss 3.1210 lr 6.00e-04 gnorm 0.18 50,374 tok/s peakVRAM 13.7GB
157
+ [22:38:13] step 2980/4768 loss 3.1083 lr 6.00e-04 gnorm 0.17 50,368 tok/s peakVRAM 13.7GB
158
+ [22:41:41] step 3000/4768 loss 3.1096 lr 6.00e-04 gnorm 0.15 50,369 tok/s peakVRAM 13.7GB
159
+ [22:45:09] step 3020/4768 loss 3.1041 lr 6.00e-04 gnorm 0.18 50,373 tok/s peakVRAM 13.7GB
160
+ [22:48:37] step 3040/4768 loss 3.1019 lr 6.00e-04 gnorm 0.16 50,369 tok/s peakVRAM 13.7GB
161
+ [22:52:05] step 3060/4768 loss 3.1055 lr 6.00e-04 gnorm 0.18 50,371 tok/s peakVRAM 13.7GB
162
+ [22:55:34] step 3080/4768 loss 3.0979 lr 6.00e-04 gnorm 0.18 50,369 tok/s peakVRAM 13.7GB
163
+ [22:59:02] step 3100/4768 loss 3.0916 lr 6.00e-04 gnorm 0.19 50,370 tok/s peakVRAM 13.7GB
164
+ [23:02:30] step 3120/4768 loss 3.1072 lr 6.00e-04 gnorm 0.20 50,365 tok/s peakVRAM 13.7GB
165
+ [23:05:58] step 3140/4768 loss 3.0934 lr 6.00e-04 gnorm 0.18 50,369 tok/s peakVRAM 13.7GB
166
+ [23:09:26] step 3160/4768 loss 3.0890 lr 6.00e-04 gnorm 0.17 50,366 tok/s peakVRAM 13.7GB
167
+ [23:12:55] step 3180/4768 loss 3.0953 lr 6.00e-04 gnorm 0.20 50,363 tok/s peakVRAM 13.7GB
168
+ [23:16:23] step 3200/4768 loss 3.0965 lr 6.00e-04 gnorm 0.21 50,369 tok/s peakVRAM 13.7GB
169
+ [23:19:51] step 3220/4768 loss 3.1058 lr 6.00e-04 gnorm 0.24 50,367 tok/s peakVRAM 13.7GB
170
+ [23:23:19] step 3240/4768 loss 3.0879 lr 6.00e-04 gnorm 0.22 50,369 tok/s peakVRAM 13.7GB
171
+ [23:26:47] step 3260/4768 loss 3.0842 lr 6.00e-04 gnorm 0.23 50,365 tok/s peakVRAM 13.7GB
172
+ [23:30:16] step 3280/4768 loss 3.0803 lr 6.00e-04 gnorm 0.20 50,360 tok/s peakVRAM 13.7GB
173
+ [23:33:44] step 3300/4768 loss 3.0907 lr 6.00e-04 gnorm 0.23 50,362 tok/s peakVRAM 13.7GB
174
+ [23:37:12] step 3320/4768 loss 3.0850 lr 6.00e-04 gnorm 0.27 50,360 tok/s peakVRAM 13.7GB
175
+ [23:40:40] step 3340/4768 loss 3.0855 lr 6.00e-04 gnorm 0.23 50,356 tok/s peakVRAM 13.7GB
176
+ [23:44:08] step 3360/4768 loss 3.0760 lr 6.00e-04 gnorm 0.23 50,359 tok/s peakVRAM 13.7GB
177
+ [23:47:37] step 3380/4768 loss 3.0862 lr 6.00e-04 gnorm 0.25 50,372 tok/s peakVRAM 13.7GB
178
+ [23:51:05] step 3400/4768 loss 3.0713 lr 6.00e-04 gnorm 0.25 50,370 tok/s peakVRAM 13.7GB
179
+ [23:54:33] step 3420/4768 loss 3.0788 lr 6.00e-04 gnorm 0.26 50,372 tok/s peakVRAM 13.7GB
180
+ [23:58:01] step 3440/4768 loss 3.0709 lr 6.00e-04 gnorm 0.23 50,365 tok/s peakVRAM 13.7GB
181
+ [00:01:29] step 3460/4768 loss 3.0802 lr 6.00e-04 gnorm 0.26 50,363 tok/s peakVRAM 13.7GB
182
+ [00:04:58] step 3480/4768 loss 3.0792 lr 6.00e-04 gnorm 0.24 50,364 tok/s peakVRAM 13.7GB
183
+ [00:08:26] step 3500/4768 loss 3.0655 lr 6.00e-04 gnorm 0.25 50,372 tok/s peakVRAM 13.7GB
184
+ [00:11:54] step 3520/4768 loss 3.0694 lr 6.00e-04 gnorm 0.24 50,369 tok/s peakVRAM 13.7GB
185
+ [00:15:22] step 3540/4768 loss 3.0713 lr 6.00e-04 gnorm 0.27 50,368 tok/s peakVRAM 13.7GB
186
+ [00:18:50] step 3560/4768 loss 3.0752 lr 6.00e-04 gnorm 0.26 50,361 tok/s peakVRAM 13.7GB
187
+ [00:22:18] step 3580/4768 loss 3.0689 lr 6.00e-04 gnorm 0.25 50,364 tok/s peakVRAM 13.7GB
188
+ [00:25:47] step 3600/4768 loss 3.0714 lr 6.00e-04 gnorm 0.30 50,363 tok/s peakVRAM 13.7GB
189
+ [00:29:15] step 3620/4768 loss 3.0629 lr 6.00e-04 gnorm 0.26 50,363 tok/s peakVRAM 13.7GB
190
+ [00:32:43] step 3640/4768 loss 3.0709 lr 6.00e-04 gnorm 0.28 50,359 tok/s peakVRAM 13.7GB
191
+ [00:36:11] step 3660/4768 loss 3.0597 lr 6.00e-04 gnorm 0.28 50,361 tok/s peakVRAM 13.7GB
192
+ [00:39:40] step 3680/4768 loss 3.0664 lr 6.00e-04 gnorm 0.33 50,361 tok/s peakVRAM 13.7GB
193
+ [00:43:08] step 3700/4768 loss 3.0636 lr 6.00e-04 gnorm 0.29 50,359 tok/s peakVRAM 13.7GB
194
+ [00:46:36] step 3720/4768 loss 3.0570 lr 6.00e-04 gnorm 0.28 50,362 tok/s peakVRAM 13.7GB
195
+ [00:50:04] step 3740/4768 loss 3.0528 lr 6.00e-04 gnorm 0.28 50,366 tok/s peakVRAM 13.7GB
196
+ [00:53:32] step 3760/4768 loss 3.0513 lr 6.00e-04 gnorm 0.27 50,365 tok/s peakVRAM 13.7GB
197
+ [00:57:01] step 3780/4768 loss 3.0523 lr 6.00e-04 gnorm 0.28 50,361 tok/s peakVRAM 13.7GB
198
+ [01:00:29] step 3800/4768 loss 3.0538 lr 6.00e-04 gnorm 0.30 50,363 tok/s peakVRAM 13.7GB
199
+ [01:03:57] step 3820/4768 loss 3.0467 lr 5.61e-04 gnorm 0.23 50,364 tok/s peakVRAM 13.7GB
200
+ [01:07:25] step 3840/4768 loss 3.0445 lr 5.13e-04 gnorm 0.27 50,369 tok/s peakVRAM 13.7GB
201
+ [01:10:53] step 3860/4768 loss 3.0375 lr 4.83e-04 gnorm 0.23 50,368 tok/s peakVRAM 13.7GB
202
+ [01:14:22] step 3880/4768 loss 3.0346 lr 4.59e-04 gnorm 0.20 50,368 tok/s peakVRAM 13.7GB
203
+ [01:17:50] step 3900/4768 loss 3.0270 lr 4.39e-04 gnorm 0.19 50,370 tok/s peakVRAM 13.7GB
204
+ [01:21:18] step 3920/4768 loss 3.0222 lr 4.21e-04 gnorm 0.23 50,370 tok/s peakVRAM 13.7GB
205
+ [01:24:46] step 3940/4768 loss 3.0140 lr 4.05e-04 gnorm 0.20 50,372 tok/s peakVRAM 13.7GB
206
+ [01:28:14] step 3960/4768 loss 3.0142 lr 3.89e-04 gnorm 0.15 50,372 tok/s peakVRAM 13.7GB
207
+ [01:31:42] step 3980/4768 loss 3.0063 lr 3.75e-04 gnorm 0.15 50,378 tok/s peakVRAM 13.7GB
208
+ [01:35:10] step 4000/4768 loss 3.0050 lr 3.62e-04 gnorm 0.16 50,379 tok/s peakVRAM 13.7GB
209
+ [01:38:39] step 4020/4768 loss 3.0023 lr 3.50e-04 gnorm 0.16 50,377 tok/s peakVRAM 13.7GB
210
+ [01:42:07] step 4040/4768 loss 2.9931 lr 3.38e-04 gnorm 0.14 50,372 tok/s peakVRAM 13.7GB
211
+ [01:45:35] step 4060/4768 loss 2.9928 lr 3.26e-04 gnorm 0.15 50,371 tok/s peakVRAM 13.7GB
212
+ [01:49:03] step 4080/4768 loss 2.9841 lr 3.15e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
213
+ [01:52:31] step 4100/4768 loss 2.9894 lr 3.05e-04 gnorm 0.15 50,373 tok/s peakVRAM 13.7GB
214
+ [01:55:59] step 4120/4768 loss 2.9777 lr 2.95e-04 gnorm 0.14 50,378 tok/s peakVRAM 13.7GB
215
+ [01:59:28] step 4140/4768 loss 2.9702 lr 2.85e-04 gnorm 0.14 50,375 tok/s peakVRAM 13.7GB
216
+ [02:02:56] step 4160/4768 loss 2.9881 lr 2.75e-04 gnorm 0.13 50,375 tok/s peakVRAM 13.7GB
217
+ [02:06:24] step 4180/4768 loss 2.9810 lr 2.66e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
218
+ [02:09:52] step 4200/4768 loss 2.9741 lr 2.57e-04 gnorm 0.13 50,372 tok/s peakVRAM 13.7GB
219
+ [02:13:20] step 4220/4768 loss 2.9675 lr 2.48e-04 gnorm 0.13 50,372 tok/s peakVRAM 13.7GB
220
+ [02:16:48] step 4240/4768 loss 2.9741 lr 2.40e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
221
+ [02:20:17] step 4260/4768 loss 2.9662 lr 2.31e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
222
+ [02:23:45] step 4280/4768 loss 2.9653 lr 2.23e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
223
+ [02:27:13] step 4300/4768 loss 2.9630 lr 2.15e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
224
+ [02:30:41] step 4320/4768 loss 2.9580 lr 2.07e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
225
+ [02:34:09] step 4340/4768 loss 2.9571 lr 1.99e-04 gnorm 0.13 50,362 tok/s peakVRAM 13.7GB
226
+ [02:37:38] step 4360/4768 loss 2.9512 lr 1.92e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
227
+ [02:41:06] step 4380/4768 loss 2.9582 lr 1.84e-04 gnorm 0.11 50,367 tok/s peakVRAM 13.7GB
228
+ [02:44:34] step 4400/4768 loss 2.9549 lr 1.77e-04 gnorm 0.11 50,368 tok/s peakVRAM 13.7GB
229
+ [02:48:02] step 4420/4768 loss 2.9487 lr 1.70e-04 gnorm 0.12 50,481 tok/s peakVRAM 13.7GB
230
+ [02:51:29] step 4440/4768 loss 2.9545 lr 1.63e-04 gnorm 0.12 50,542 tok/s peakVRAM 13.7GB
231
+ [02:54:57] step 4460/4768 loss 2.9502 lr 1.56e-04 gnorm 0.12 50,557 tok/s peakVRAM 13.7GB
232
+ [02:58:24] step 4480/4768 loss 2.9451 lr 1.49e-04 gnorm 0.11 50,559 tok/s peakVRAM 13.7GB
233
+ [03:01:51] step 4500/4768 loss 2.9426 lr 1.42e-04 gnorm 0.11 50,565 tok/s peakVRAM 13.7GB
234
+ [03:05:19] step 4520/4768 loss 2.9441 lr 1.36e-04 gnorm 0.12 50,564 tok/s peakVRAM 13.7GB
235
+ [03:08:46] step 4540/4768 loss 2.9361 lr 1.29e-04 gnorm 0.12 50,562 tok/s peakVRAM 13.7GB
236
+ [03:12:14] step 4560/4768 loss 2.9384 lr 1.23e-04 gnorm 0.12 50,565 tok/s peakVRAM 13.7GB
237
+ [03:15:41] step 4580/4768 loss 2.9421 lr 1.16e-04 gnorm 0.11 50,562 tok/s peakVRAM 13.7GB
238
+ [03:19:08] step 4600/4768 loss 2.9408 lr 1.10e-04 gnorm 0.11 50,556 tok/s peakVRAM 13.7GB
239
+ [03:22:36] step 4620/4768 loss 2.9273 lr 1.04e-04 gnorm 0.10 50,487 tok/s peakVRAM 13.7GB
240
+ [03:26:03] step 4640/4768 loss 2.9367 lr 9.78e-05 gnorm 0.10 50,539 tok/s peakVRAM 13.7GB
241
+ [03:29:31] step 4660/4768 loss 2.9295 lr 9.18e-05 gnorm 0.11 50,554 tok/s peakVRAM 13.7GB
242
+ [03:32:58] step 4680/4768 loss 2.9272 lr 8.58e-05 gnorm 0.10 50,523 tok/s peakVRAM 13.7GB
243
+ [03:36:26] step 4700/4768 loss 2.9233 lr 7.99e-05 gnorm 0.10 50,502 tok/s peakVRAM 13.7GB
244
+ [03:39:54] step 4720/4768 loss 2.9241 lr 7.41e-05 gnorm 0.10 50,527 tok/s peakVRAM 13.7GB
245
+ [03:43:21] step 4740/4768 loss 2.9223 lr 6.83e-05 gnorm 0.10 50,522 tok/s peakVRAM 13.7GB
246
+ [03:46:49] step 4760/4768 loss 2.9311 lr 6.26e-05 gnorm 0.10 50,546 tok/s peakVRAM 13.7GB
247
+ [03:48:13] ckpt @ 4768
248
+ [03:48:13] DONE
K22/run/train.out ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [13:59:56] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [13:59:56] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [13:59:58] model GPT-ref: 180.9M param (L22 d768 h12)
4
+ [13:59:59] optimizer=muon muon_lr=0.02 (mult=33.3x)
5
+ [13:59:59] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [13:59:59] r13: mixer_layout=AAAAAAAAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [14:00:00] torch.compile enabled
8
+ [14:00:00] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [14:04:48] step 20/4768 loss 10.3787 lr 2.40e-05 gnorm 2.38 36,394 tok/s peakVRAM 13.7GB
10
+ [14:08:15] step 40/4768 loss 9.6271 lr 4.80e-05 gnorm 1.40 50,498 tok/s peakVRAM 13.7GB
11
+ [14:11:43] step 60/4768 loss 8.8371 lr 7.20e-05 gnorm 1.11 50,510 tok/s peakVRAM 13.7GB
12
+ [14:15:11] step 80/4768 loss 8.1619 lr 9.60e-05 gnorm 1.04 50,497 tok/s peakVRAM 13.7GB
13
+ [14:18:38] step 100/4768 loss 7.6838 lr 1.20e-04 gnorm 0.69 50,466 tok/s peakVRAM 13.7GB
14
+ [14:22:06] step 120/4768 loss 7.2235 lr 1.44e-04 gnorm 0.83 50,441 tok/s peakVRAM 13.7GB
15
+ [14:25:34] step 140/4768 loss 6.8162 lr 1.68e-04 gnorm 0.88 50,431 tok/s peakVRAM 13.7GB
16
+ [14:29:02] step 160/4768 loss 6.4518 lr 1.92e-04 gnorm 0.96 50,418 tok/s peakVRAM 13.7GB
17
+ [14:32:30] step 180/4768 loss 6.1645 lr 2.16e-04 gnorm 0.75 50,412 tok/s peakVRAM 13.7GB
18
+ [14:35:58] step 200/4768 loss 5.9285 lr 2.40e-04 gnorm 0.78 50,411 tok/s peakVRAM 13.7GB
19
+ [14:39:26] step 220/4768 loss 5.7298 lr 2.64e-04 gnorm 0.55 50,409 tok/s peakVRAM 13.7GB
20
+ [14:42:54] step 240/4768 loss 5.5502 lr 2.88e-04 gnorm 0.49 50,408 tok/s peakVRAM 13.7GB
21
+ [14:46:22] step 260/4768 loss 5.3819 lr 3.12e-04 gnorm 0.59 50,407 tok/s peakVRAM 13.7GB
22
+ [14:49:50] step 280/4768 loss 5.2249 lr 3.36e-04 gnorm 0.58 50,410 tok/s peakVRAM 13.7GB
23
+ [14:53:18] step 300/4768 loss 5.0776 lr 3.60e-04 gnorm 0.52 50,409 tok/s peakVRAM 13.7GB
24
+ [14:56:46] step 320/4768 loss 4.9310 lr 3.84e-04 gnorm 0.63 50,410 tok/s peakVRAM 13.7GB
25
+ [15:00:14] step 340/4768 loss 4.7736 lr 4.08e-04 gnorm 0.59 50,408 tok/s peakVRAM 13.7GB
26
+ [15:03:42] step 360/4768 loss 4.6485 lr 4.32e-04 gnorm 0.58 50,401 tok/s peakVRAM 13.7GB
27
+ [15:07:10] step 380/4768 loss 4.5306 lr 4.56e-04 gnorm 0.55 50,404 tok/s peakVRAM 13.7GB
28
+ [15:10:38] step 400/4768 loss 4.4408 lr 4.80e-04 gnorm 0.48 50,402 tok/s peakVRAM 13.7GB
29
+ [15:14:06] step 420/4768 loss 4.3643 lr 5.04e-04 gnorm 0.43 50,404 tok/s peakVRAM 13.7GB
30
+ [15:17:34] step 440/4768 loss 4.2946 lr 5.28e-04 gnorm 0.50 50,403 tok/s peakVRAM 13.7GB
31
+ [15:21:02] step 460/4768 loss 4.2252 lr 5.52e-04 gnorm 0.44 50,392 tok/s peakVRAM 13.7GB
32
+ [15:24:31] step 480/4768 loss 4.1773 lr 5.76e-04 gnorm 0.43 50,388 tok/s peakVRAM 13.7GB
33
+ [15:27:59] step 500/4768 loss 4.1259 lr 6.00e-04 gnorm 0.40 50,392 tok/s peakVRAM 13.7GB
34
+ [15:31:27] step 520/4768 loss 4.0899 lr 6.00e-04 gnorm 0.33 50,395 tok/s peakVRAM 13.7GB
35
+ [15:34:55] step 540/4768 loss 4.0379 lr 6.00e-04 gnorm 0.25 50,396 tok/s peakVRAM 13.7GB
36
+ [15:38:23] step 560/4768 loss 3.9871 lr 6.00e-04 gnorm 0.33 50,389 tok/s peakVRAM 13.7GB
37
+ [15:41:51] step 580/4768 loss 3.9425 lr 6.00e-04 gnorm 0.29 50,387 tok/s peakVRAM 13.7GB
38
+ [15:45:19] step 600/4768 loss 3.8993 lr 6.00e-04 gnorm 0.26 50,382 tok/s peakVRAM 13.7GB
39
+ [15:48:47] step 620/4768 loss 3.8736 lr 6.00e-04 gnorm 0.25 50,384 tok/s peakVRAM 13.7GB
40
+ [15:52:15] step 640/4768 loss 3.8464 lr 6.00e-04 gnorm 0.28 50,384 tok/s peakVRAM 13.7GB
41
+ [15:55:44] step 660/4768 loss 3.8137 lr 6.00e-04 gnorm 0.29 50,378 tok/s peakVRAM 13.7GB
42
+ [15:59:12] step 680/4768 loss 3.7883 lr 6.00e-04 gnorm 0.22 50,379 tok/s peakVRAM 13.7GB
43
+ [16:02:40] step 700/4768 loss 3.7559 lr 6.00e-04 gnorm 0.25 50,379 tok/s peakVRAM 13.7GB
44
+ [16:06:08] step 720/4768 loss 3.7238 lr 6.00e-04 gnorm 0.25 50,377 tok/s peakVRAM 13.7GB
45
+ [16:09:36] step 740/4768 loss 3.7190 lr 6.00e-04 gnorm 0.28 50,374 tok/s peakVRAM 13.7GB
46
+ [16:13:04] step 760/4768 loss 3.6942 lr 6.00e-04 gnorm 0.22 50,379 tok/s peakVRAM 13.7GB
47
+ [16:16:32] step 780/4768 loss 3.6677 lr 6.00e-04 gnorm 0.22 50,381 tok/s peakVRAM 13.7GB
48
+ [16:20:01] step 800/4768 loss 3.6530 lr 6.00e-04 gnorm 0.20 50,378 tok/s peakVRAM 13.7GB
49
+ [16:23:29] step 820/4768 loss 3.6361 lr 6.00e-04 gnorm 0.21 50,380 tok/s peakVRAM 13.7GB
50
+ [16:26:57] step 840/4768 loss 3.6213 lr 6.00e-04 gnorm 0.21 50,379 tok/s peakVRAM 13.7GB
51
+ [16:30:25] step 860/4768 loss 3.5994 lr 6.00e-04 gnorm 0.18 50,383 tok/s peakVRAM 13.7GB
52
+ [16:33:53] step 880/4768 loss 3.5940 lr 6.00e-04 gnorm 0.21 50,370 tok/s peakVRAM 13.7GB
53
+ [16:37:21] step 900/4768 loss 3.5738 lr 6.00e-04 gnorm 0.18 50,372 tok/s peakVRAM 13.7GB
54
+ [16:40:49] step 920/4768 loss 3.5661 lr 6.00e-04 gnorm 0.19 50,372 tok/s peakVRAM 13.7GB
55
+ [16:44:18] step 940/4768 loss 3.5609 lr 6.00e-04 gnorm 0.18 50,369 tok/s peakVRAM 13.7GB
56
+ [16:47:46] step 960/4768 loss 3.5345 lr 6.00e-04 gnorm 0.17 50,374 tok/s peakVRAM 13.7GB
57
+ [16:51:14] step 980/4768 loss 3.5308 lr 6.00e-04 gnorm 0.17 50,370 tok/s peakVRAM 13.7GB
58
+ [16:54:42] step 1000/4768 loss 3.5204 lr 6.00e-04 gnorm 0.17 50,369 tok/s peakVRAM 13.7GB
59
+ [16:58:10] step 1020/4768 loss 3.5146 lr 6.00e-04 gnorm 0.19 50,360 tok/s peakVRAM 13.7GB
60
+ [17:01:39] step 1040/4768 loss 3.4869 lr 6.00e-04 gnorm 0.18 50,364 tok/s peakVRAM 13.7GB
61
+ [17:05:07] step 1060/4768 loss 3.4986 lr 6.00e-04 gnorm 0.18 50,368 tok/s peakVRAM 13.7GB
62
+ [17:08:35] step 1080/4768 loss 3.4884 lr 6.00e-04 gnorm 0.16 50,372 tok/s peakVRAM 13.7GB
63
+ [17:12:03] step 1100/4768 loss 3.4719 lr 6.00e-04 gnorm 0.21 50,367 tok/s peakVRAM 13.7GB
64
+ [17:15:31] step 1120/4768 loss 3.4548 lr 6.00e-04 gnorm 0.18 50,363 tok/s peakVRAM 13.7GB
65
+ [17:18:59] step 1140/4768 loss 3.4481 lr 6.00e-04 gnorm 0.16 50,367 tok/s peakVRAM 13.7GB
66
+ [17:22:28] step 1160/4768 loss 3.4420 lr 6.00e-04 gnorm 0.17 50,367 tok/s peakVRAM 13.7GB
67
+ [17:25:56] step 1180/4768 loss 3.4437 lr 6.00e-04 gnorm 0.16 50,366 tok/s peakVRAM 13.7GB
68
+ [17:29:24] step 1200/4768 loss 3.4253 lr 6.00e-04 gnorm 0.19 50,364 tok/s peakVRAM 13.7GB
69
+ [17:32:52] step 1220/4768 loss 3.4227 lr 6.00e-04 gnorm 0.14 50,357 tok/s peakVRAM 13.7GB
70
+ [17:36:20] step 1240/4768 loss 3.4135 lr 6.00e-04 gnorm 0.15 50,365 tok/s peakVRAM 13.7GB
71
+ [17:39:49] step 1260/4768 loss 3.3992 lr 6.00e-04 gnorm 0.15 50,366 tok/s peakVRAM 13.7GB
72
+ [17:43:17] step 1280/4768 loss 3.4017 lr 6.00e-04 gnorm 0.15 50,361 tok/s peakVRAM 13.7GB
73
+ [17:46:45] step 1300/4768 loss 3.3952 lr 6.00e-04 gnorm 0.16 50,359 tok/s peakVRAM 13.7GB
74
+ [17:50:13] step 1320/4768 loss 3.3867 lr 6.00e-04 gnorm 0.16 50,362 tok/s peakVRAM 13.7GB
75
+ [17:53:42] step 1340/4768 loss 3.3866 lr 6.00e-04 gnorm 0.15 50,360 tok/s peakVRAM 13.7GB
76
+ [17:57:10] step 1360/4768 loss 3.3772 lr 6.00e-04 gnorm 0.13 50,360 tok/s peakVRAM 13.7GB
77
+ [18:00:38] step 1380/4768 loss 3.3684 lr 6.00e-04 gnorm 0.13 50,363 tok/s peakVRAM 13.7GB
78
+ [18:04:06] step 1400/4768 loss 3.3574 lr 6.00e-04 gnorm 0.14 50,367 tok/s peakVRAM 13.7GB
79
+ [18:07:34] step 1420/4768 loss 3.3459 lr 6.00e-04 gnorm 0.16 50,368 tok/s peakVRAM 13.7GB
80
+ [18:11:02] step 1440/4768 loss 3.3518 lr 6.00e-04 gnorm 0.16 50,366 tok/s peakVRAM 13.7GB
81
+ [18:14:31] step 1460/4768 loss 3.3493 lr 6.00e-04 gnorm 0.13 50,363 tok/s peakVRAM 13.7GB
82
+ [18:17:59] step 1480/4768 loss 3.3528 lr 6.00e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
83
+ [18:21:27] step 1500/4768 loss 3.3260 lr 6.00e-04 gnorm 0.16 50,361 tok/s peakVRAM 13.7GB
84
+ [18:24:55] step 1520/4768 loss 3.3333 lr 6.00e-04 gnorm 0.14 50,362 tok/s peakVRAM 13.7GB
85
+ [18:28:23] step 1540/4768 loss 3.3187 lr 6.00e-04 gnorm 0.15 50,362 tok/s peakVRAM 13.7GB
86
+ [18:31:52] step 1560/4768 loss 3.3264 lr 6.00e-04 gnorm 0.16 50,363 tok/s peakVRAM 13.7GB
87
+ [18:35:20] step 1580/4768 loss 3.3143 lr 6.00e-04 gnorm 0.12 50,366 tok/s peakVRAM 13.7GB
88
+ [18:38:48] step 1600/4768 loss 3.3036 lr 6.00e-04 gnorm 0.15 50,366 tok/s peakVRAM 13.7GB
89
+ [18:42:16] step 1620/4768 loss 3.3041 lr 6.00e-04 gnorm 0.14 50,364 tok/s peakVRAM 13.7GB
90
+ [18:45:44] step 1640/4768 loss 3.3000 lr 6.00e-04 gnorm 0.14 50,371 tok/s peakVRAM 13.7GB
91
+ [18:49:13] step 1660/4768 loss 3.3013 lr 6.00e-04 gnorm 0.13 50,376 tok/s peakVRAM 13.7GB
92
+ [18:52:41] step 1680/4768 loss 3.2893 lr 6.00e-04 gnorm 0.13 50,375 tok/s peakVRAM 13.7GB
93
+ [18:56:09] step 1700/4768 loss 3.2815 lr 6.00e-04 gnorm 0.12 50,373 tok/s peakVRAM 13.7GB
94
+ [18:59:37] step 1720/4768 loss 3.2840 lr 6.00e-04 gnorm 0.15 50,365 tok/s peakVRAM 13.7GB
95
+ [19:03:05] step 1740/4768 loss 3.2778 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
96
+ [19:06:34] step 1760/4768 loss 3.2767 lr 6.00e-04 gnorm 0.13 50,363 tok/s peakVRAM 13.7GB
97
+ [19:10:02] step 1780/4768 loss 3.2649 lr 6.00e-04 gnorm 0.14 50,359 tok/s peakVRAM 13.7GB
98
+ [19:13:30] step 1800/4768 loss 3.2683 lr 6.00e-04 gnorm 0.13 50,354 tok/s peakVRAM 13.7GB
99
+ [19:16:58] step 1820/4768 loss 3.2637 lr 6.00e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
100
+ [19:20:26] step 1840/4768 loss 3.2567 lr 6.00e-04 gnorm 0.12 50,360 tok/s peakVRAM 13.7GB
101
+ [19:23:55] step 1860/4768 loss 3.2561 lr 6.00e-04 gnorm 0.13 50,361 tok/s peakVRAM 13.7GB
102
+ [19:27:23] step 1880/4768 loss 3.2509 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
103
+ [19:30:51] step 1900/4768 loss 3.2456 lr 6.00e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
104
+ [19:34:19] step 1920/4768 loss 3.2454 lr 6.00e-04 gnorm 0.11 50,365 tok/s peakVRAM 13.7GB
105
+ [19:37:47] step 1940/4768 loss 3.2371 lr 6.00e-04 gnorm 0.13 50,361 tok/s peakVRAM 13.7GB
106
+ [19:41:16] step 1960/4768 loss 3.2425 lr 6.00e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
107
+ [19:44:44] step 1980/4768 loss 3.2365 lr 6.00e-04 gnorm 0.12 50,369 tok/s peakVRAM 13.7GB
108
+ [19:48:12] step 2000/4768 loss 3.2297 lr 6.00e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
109
+ [19:51:40] step 2020/4768 loss 3.2300 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
110
+ [19:55:08] step 2040/4768 loss 3.2229 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
111
+ [19:58:36] step 2060/4768 loss 3.2234 lr 6.00e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
112
+ [20:02:05] step 2080/4768 loss 3.2119 lr 6.00e-04 gnorm 0.13 50,369 tok/s peakVRAM 13.7GB
113
+ [20:05:33] step 2100/4768 loss 3.2206 lr 6.00e-04 gnorm 0.11 50,369 tok/s peakVRAM 13.7GB
114
+ [20:09:01] step 2120/4768 loss 3.2098 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
115
+ [20:12:29] step 2140/4768 loss 3.2225 lr 6.00e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
116
+ [20:15:57] step 2160/4768 loss 3.2009 lr 6.00e-04 gnorm 0.13 50,369 tok/s peakVRAM 13.7GB
117
+ [20:19:26] step 2180/4768 loss 3.2029 lr 6.00e-04 gnorm 0.11 50,369 tok/s peakVRAM 13.7GB
118
+ [20:22:54] step 2200/4768 loss 3.2040 lr 6.00e-04 gnorm 0.12 50,375 tok/s peakVRAM 13.7GB
119
+ [20:26:22] step 2220/4768 loss 3.1966 lr 6.00e-04 gnorm 0.15 50,370 tok/s peakVRAM 13.7GB
120
+ [20:29:50] step 2240/4768 loss 3.1929 lr 6.00e-04 gnorm 0.12 50,366 tok/s peakVRAM 13.7GB
121
+ [20:33:18] step 2260/4768 loss 3.1895 lr 6.00e-04 gnorm 0.12 50,364 tok/s peakVRAM 13.7GB
122
+ [20:36:46] step 2280/4768 loss 3.1854 lr 6.00e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
123
+ [20:40:15] step 2300/4768 loss 3.1888 lr 6.00e-04 gnorm 0.12 50,365 tok/s peakVRAM 13.7GB
124
+ [20:43:43] step 2320/4768 loss 3.1856 lr 6.00e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
125
+ [20:47:11] step 2340/4768 loss 3.1806 lr 6.00e-04 gnorm 0.14 50,371 tok/s peakVRAM 13.7GB
126
+ [20:50:39] step 2360/4768 loss 3.1701 lr 6.00e-04 gnorm 0.12 50,374 tok/s peakVRAM 13.7GB
127
+ [20:54:07] step 2380/4768 loss 3.1692 lr 6.00e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
128
+ [20:57:36] step 2400/4768 loss 3.1698 lr 6.00e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
129
+ [21:01:04] step 2420/4768 loss 3.1726 lr 6.00e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
130
+ [21:04:32] step 2440/4768 loss 3.1677 lr 6.00e-04 gnorm 0.14 50,365 tok/s peakVRAM 13.7GB
131
+ [21:08:00] step 2460/4768 loss 3.1653 lr 6.00e-04 gnorm 0.15 50,361 tok/s peakVRAM 13.7GB
132
+ [21:11:28] step 2480/4768 loss 3.1631 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
133
+ [21:14:57] step 2500/4768 loss 3.1575 lr 6.00e-04 gnorm 0.11 50,371 tok/s peakVRAM 13.7GB
134
+ [21:18:25] step 2520/4768 loss 3.1594 lr 6.00e-04 gnorm 0.12 50,366 tok/s peakVRAM 13.7GB
135
+ [21:21:53] step 2540/4768 loss 3.1598 lr 6.00e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
136
+ [21:25:21] step 2560/4768 loss 3.1564 lr 6.00e-04 gnorm 0.13 50,377 tok/s peakVRAM 13.7GB
137
+ [21:28:49] step 2580/4768 loss 3.1529 lr 6.00e-04 gnorm 0.14 50,374 tok/s peakVRAM 13.7GB
138
+ [21:32:17] step 2600/4768 loss 3.1433 lr 6.00e-04 gnorm 0.12 50,370 tok/s peakVRAM 13.7GB
139
+ [21:35:46] step 2620/4768 loss 3.1518 lr 6.00e-04 gnorm 0.13 50,368 tok/s peakVRAM 13.7GB
140
+ [21:39:14] step 2640/4768 loss 3.1528 lr 6.00e-04 gnorm 0.15 50,373 tok/s peakVRAM 13.7GB
141
+ [21:42:42] step 2660/4768 loss 3.1508 lr 6.00e-04 gnorm 0.14 50,368 tok/s peakVRAM 13.7GB
142
+ [21:46:10] step 2680/4768 loss 3.1459 lr 6.00e-04 gnorm 0.13 50,369 tok/s peakVRAM 13.7GB
143
+ [21:49:38] step 2700/4768 loss 3.1484 lr 6.00e-04 gnorm 0.14 50,368 tok/s peakVRAM 13.7GB
144
+ [21:53:06] step 2720/4768 loss 3.1315 lr 6.00e-04 gnorm 0.15 50,371 tok/s peakVRAM 13.7GB
145
+ [21:56:35] step 2740/4768 loss 3.1328 lr 6.00e-04 gnorm 0.14 50,371 tok/s peakVRAM 13.7GB
146
+ [22:00:03] step 2760/4768 loss 3.1356 lr 6.00e-04 gnorm 0.14 50,367 tok/s peakVRAM 13.7GB
147
+ [22:03:31] step 2780/4768 loss 3.1263 lr 6.00e-04 gnorm 0.14 50,365 tok/s peakVRAM 13.7GB
148
+ [22:06:59] step 2800/4768 loss 3.1220 lr 6.00e-04 gnorm 0.15 50,367 tok/s peakVRAM 13.7GB
149
+ [22:10:27] step 2820/4768 loss 3.1191 lr 6.00e-04 gnorm 0.14 50,368 tok/s peakVRAM 13.7GB
150
+ [22:13:56] step 2840/4768 loss 3.1272 lr 6.00e-04 gnorm 0.16 50,369 tok/s peakVRAM 13.7GB
151
+ [22:17:24] step 2860/4768 loss 3.1184 lr 6.00e-04 gnorm 0.15 50,365 tok/s peakVRAM 13.7GB
152
+ [22:20:52] step 2880/4768 loss 3.1203 lr 6.00e-04 gnorm 0.16 50,371 tok/s peakVRAM 13.7GB
153
+ [22:24:20] step 2900/4768 loss 3.1217 lr 6.00e-04 gnorm 0.14 50,369 tok/s peakVRAM 13.7GB
154
+ [22:27:48] step 2920/4768 loss 3.1290 lr 6.00e-04 gnorm 0.17 50,368 tok/s peakVRAM 13.7GB
155
+ [22:31:16] step 2940/4768 loss 3.1193 lr 6.00e-04 gnorm 0.17 50,375 tok/s peakVRAM 13.7GB
156
+ [22:34:45] step 2960/4768 loss 3.1210 lr 6.00e-04 gnorm 0.18 50,374 tok/s peakVRAM 13.7GB
157
+ [22:38:13] step 2980/4768 loss 3.1083 lr 6.00e-04 gnorm 0.17 50,368 tok/s peakVRAM 13.7GB
158
+ [22:41:41] step 3000/4768 loss 3.1096 lr 6.00e-04 gnorm 0.15 50,369 tok/s peakVRAM 13.7GB
159
+ [22:45:09] step 3020/4768 loss 3.1041 lr 6.00e-04 gnorm 0.18 50,373 tok/s peakVRAM 13.7GB
160
+ [22:48:37] step 3040/4768 loss 3.1019 lr 6.00e-04 gnorm 0.16 50,369 tok/s peakVRAM 13.7GB
161
+ [22:52:05] step 3060/4768 loss 3.1055 lr 6.00e-04 gnorm 0.18 50,371 tok/s peakVRAM 13.7GB
162
+ [22:55:34] step 3080/4768 loss 3.0979 lr 6.00e-04 gnorm 0.18 50,369 tok/s peakVRAM 13.7GB
163
+ [22:59:02] step 3100/4768 loss 3.0916 lr 6.00e-04 gnorm 0.19 50,370 tok/s peakVRAM 13.7GB
164
+ [23:02:30] step 3120/4768 loss 3.1072 lr 6.00e-04 gnorm 0.20 50,365 tok/s peakVRAM 13.7GB
165
+ [23:05:58] step 3140/4768 loss 3.0934 lr 6.00e-04 gnorm 0.18 50,369 tok/s peakVRAM 13.7GB
166
+ [23:09:26] step 3160/4768 loss 3.0890 lr 6.00e-04 gnorm 0.17 50,366 tok/s peakVRAM 13.7GB
167
+ [23:12:55] step 3180/4768 loss 3.0953 lr 6.00e-04 gnorm 0.20 50,363 tok/s peakVRAM 13.7GB
168
+ [23:16:23] step 3200/4768 loss 3.0965 lr 6.00e-04 gnorm 0.21 50,369 tok/s peakVRAM 13.7GB
169
+ [23:19:51] step 3220/4768 loss 3.1058 lr 6.00e-04 gnorm 0.24 50,367 tok/s peakVRAM 13.7GB
170
+ [23:23:19] step 3240/4768 loss 3.0879 lr 6.00e-04 gnorm 0.22 50,369 tok/s peakVRAM 13.7GB
171
+ [23:26:47] step 3260/4768 loss 3.0842 lr 6.00e-04 gnorm 0.23 50,365 tok/s peakVRAM 13.7GB
172
+ [23:30:16] step 3280/4768 loss 3.0803 lr 6.00e-04 gnorm 0.20 50,360 tok/s peakVRAM 13.7GB
173
+ [23:33:44] step 3300/4768 loss 3.0907 lr 6.00e-04 gnorm 0.23 50,362 tok/s peakVRAM 13.7GB
174
+ [23:37:12] step 3320/4768 loss 3.0850 lr 6.00e-04 gnorm 0.27 50,360 tok/s peakVRAM 13.7GB
175
+ [23:40:40] step 3340/4768 loss 3.0855 lr 6.00e-04 gnorm 0.23 50,356 tok/s peakVRAM 13.7GB
176
+ [23:44:08] step 3360/4768 loss 3.0760 lr 6.00e-04 gnorm 0.23 50,359 tok/s peakVRAM 13.7GB
177
+ [23:47:37] step 3380/4768 loss 3.0862 lr 6.00e-04 gnorm 0.25 50,372 tok/s peakVRAM 13.7GB
178
+ [23:51:05] step 3400/4768 loss 3.0713 lr 6.00e-04 gnorm 0.25 50,370 tok/s peakVRAM 13.7GB
179
+ [23:54:33] step 3420/4768 loss 3.0788 lr 6.00e-04 gnorm 0.26 50,372 tok/s peakVRAM 13.7GB
180
+ [23:58:01] step 3440/4768 loss 3.0709 lr 6.00e-04 gnorm 0.23 50,365 tok/s peakVRAM 13.7GB
181
+ [00:01:29] step 3460/4768 loss 3.0802 lr 6.00e-04 gnorm 0.26 50,363 tok/s peakVRAM 13.7GB
182
+ [00:04:58] step 3480/4768 loss 3.0792 lr 6.00e-04 gnorm 0.24 50,364 tok/s peakVRAM 13.7GB
183
+ [00:08:26] step 3500/4768 loss 3.0655 lr 6.00e-04 gnorm 0.25 50,372 tok/s peakVRAM 13.7GB
184
+ [00:11:54] step 3520/4768 loss 3.0694 lr 6.00e-04 gnorm 0.24 50,369 tok/s peakVRAM 13.7GB
185
+ [00:15:22] step 3540/4768 loss 3.0713 lr 6.00e-04 gnorm 0.27 50,368 tok/s peakVRAM 13.7GB
186
+ [00:18:50] step 3560/4768 loss 3.0752 lr 6.00e-04 gnorm 0.26 50,361 tok/s peakVRAM 13.7GB
187
+ [00:22:18] step 3580/4768 loss 3.0689 lr 6.00e-04 gnorm 0.25 50,364 tok/s peakVRAM 13.7GB
188
+ [00:25:47] step 3600/4768 loss 3.0714 lr 6.00e-04 gnorm 0.30 50,363 tok/s peakVRAM 13.7GB
189
+ [00:29:15] step 3620/4768 loss 3.0629 lr 6.00e-04 gnorm 0.26 50,363 tok/s peakVRAM 13.7GB
190
+ [00:32:43] step 3640/4768 loss 3.0709 lr 6.00e-04 gnorm 0.28 50,359 tok/s peakVRAM 13.7GB
191
+ [00:36:11] step 3660/4768 loss 3.0597 lr 6.00e-04 gnorm 0.28 50,361 tok/s peakVRAM 13.7GB
192
+ [00:39:40] step 3680/4768 loss 3.0664 lr 6.00e-04 gnorm 0.33 50,361 tok/s peakVRAM 13.7GB
193
+ [00:43:08] step 3700/4768 loss 3.0636 lr 6.00e-04 gnorm 0.29 50,359 tok/s peakVRAM 13.7GB
194
+ [00:46:36] step 3720/4768 loss 3.0570 lr 6.00e-04 gnorm 0.28 50,362 tok/s peakVRAM 13.7GB
195
+ [00:50:04] step 3740/4768 loss 3.0528 lr 6.00e-04 gnorm 0.28 50,366 tok/s peakVRAM 13.7GB
196
+ [00:53:32] step 3760/4768 loss 3.0513 lr 6.00e-04 gnorm 0.27 50,365 tok/s peakVRAM 13.7GB
197
+ [00:57:01] step 3780/4768 loss 3.0523 lr 6.00e-04 gnorm 0.28 50,361 tok/s peakVRAM 13.7GB
198
+ [01:00:29] step 3800/4768 loss 3.0538 lr 6.00e-04 gnorm 0.30 50,363 tok/s peakVRAM 13.7GB
199
+ [01:03:57] step 3820/4768 loss 3.0467 lr 5.61e-04 gnorm 0.23 50,364 tok/s peakVRAM 13.7GB
200
+ [01:07:25] step 3840/4768 loss 3.0445 lr 5.13e-04 gnorm 0.27 50,369 tok/s peakVRAM 13.7GB
201
+ [01:10:53] step 3860/4768 loss 3.0375 lr 4.83e-04 gnorm 0.23 50,368 tok/s peakVRAM 13.7GB
202
+ [01:14:22] step 3880/4768 loss 3.0346 lr 4.59e-04 gnorm 0.20 50,368 tok/s peakVRAM 13.7GB
203
+ [01:17:50] step 3900/4768 loss 3.0270 lr 4.39e-04 gnorm 0.19 50,370 tok/s peakVRAM 13.7GB
204
+ [01:21:18] step 3920/4768 loss 3.0222 lr 4.21e-04 gnorm 0.23 50,370 tok/s peakVRAM 13.7GB
205
+ [01:24:46] step 3940/4768 loss 3.0140 lr 4.05e-04 gnorm 0.20 50,372 tok/s peakVRAM 13.7GB
206
+ [01:28:14] step 3960/4768 loss 3.0142 lr 3.89e-04 gnorm 0.15 50,372 tok/s peakVRAM 13.7GB
207
+ [01:31:42] step 3980/4768 loss 3.0063 lr 3.75e-04 gnorm 0.15 50,378 tok/s peakVRAM 13.7GB
208
+ [01:35:10] step 4000/4768 loss 3.0050 lr 3.62e-04 gnorm 0.16 50,379 tok/s peakVRAM 13.7GB
209
+ [01:38:39] step 4020/4768 loss 3.0023 lr 3.50e-04 gnorm 0.16 50,377 tok/s peakVRAM 13.7GB
210
+ [01:42:07] step 4040/4768 loss 2.9931 lr 3.38e-04 gnorm 0.14 50,372 tok/s peakVRAM 13.7GB
211
+ [01:45:35] step 4060/4768 loss 2.9928 lr 3.26e-04 gnorm 0.15 50,371 tok/s peakVRAM 13.7GB
212
+ [01:49:03] step 4080/4768 loss 2.9841 lr 3.15e-04 gnorm 0.13 50,367 tok/s peakVRAM 13.7GB
213
+ [01:52:31] step 4100/4768 loss 2.9894 lr 3.05e-04 gnorm 0.15 50,373 tok/s peakVRAM 13.7GB
214
+ [01:55:59] step 4120/4768 loss 2.9777 lr 2.95e-04 gnorm 0.14 50,378 tok/s peakVRAM 13.7GB
215
+ [01:59:28] step 4140/4768 loss 2.9702 lr 2.85e-04 gnorm 0.14 50,375 tok/s peakVRAM 13.7GB
216
+ [02:02:56] step 4160/4768 loss 2.9881 lr 2.75e-04 gnorm 0.13 50,375 tok/s peakVRAM 13.7GB
217
+ [02:06:24] step 4180/4768 loss 2.9810 lr 2.66e-04 gnorm 0.13 50,370 tok/s peakVRAM 13.7GB
218
+ [02:09:52] step 4200/4768 loss 2.9741 lr 2.57e-04 gnorm 0.13 50,372 tok/s peakVRAM 13.7GB
219
+ [02:13:20] step 4220/4768 loss 2.9675 lr 2.48e-04 gnorm 0.13 50,372 tok/s peakVRAM 13.7GB
220
+ [02:16:48] step 4240/4768 loss 2.9741 lr 2.40e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
221
+ [02:20:17] step 4260/4768 loss 2.9662 lr 2.31e-04 gnorm 0.12 50,363 tok/s peakVRAM 13.7GB
222
+ [02:23:45] step 4280/4768 loss 2.9653 lr 2.23e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
223
+ [02:27:13] step 4300/4768 loss 2.9630 lr 2.15e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
224
+ [02:30:41] step 4320/4768 loss 2.9580 lr 2.07e-04 gnorm 0.12 50,367 tok/s peakVRAM 13.7GB
225
+ [02:34:09] step 4340/4768 loss 2.9571 lr 1.99e-04 gnorm 0.13 50,362 tok/s peakVRAM 13.7GB
226
+ [02:37:38] step 4360/4768 loss 2.9512 lr 1.92e-04 gnorm 0.13 50,366 tok/s peakVRAM 13.7GB
227
+ [02:41:06] step 4380/4768 loss 2.9582 lr 1.84e-04 gnorm 0.11 50,367 tok/s peakVRAM 13.7GB
228
+ [02:44:34] step 4400/4768 loss 2.9549 lr 1.77e-04 gnorm 0.11 50,368 tok/s peakVRAM 13.7GB
229
+ [02:48:02] step 4420/4768 loss 2.9487 lr 1.70e-04 gnorm 0.12 50,481 tok/s peakVRAM 13.7GB
230
+ [02:51:29] step 4440/4768 loss 2.9545 lr 1.63e-04 gnorm 0.12 50,542 tok/s peakVRAM 13.7GB
231
+ [02:54:57] step 4460/4768 loss 2.9502 lr 1.56e-04 gnorm 0.12 50,557 tok/s peakVRAM 13.7GB
232
+ [02:58:24] step 4480/4768 loss 2.9451 lr 1.49e-04 gnorm 0.11 50,559 tok/s peakVRAM 13.7GB
233
+ [03:01:51] step 4500/4768 loss 2.9426 lr 1.42e-04 gnorm 0.11 50,565 tok/s peakVRAM 13.7GB
234
+ [03:05:19] step 4520/4768 loss 2.9441 lr 1.36e-04 gnorm 0.12 50,564 tok/s peakVRAM 13.7GB
235
+ [03:08:46] step 4540/4768 loss 2.9361 lr 1.29e-04 gnorm 0.12 50,562 tok/s peakVRAM 13.7GB
236
+ [03:12:14] step 4560/4768 loss 2.9384 lr 1.23e-04 gnorm 0.12 50,565 tok/s peakVRAM 13.7GB
237
+ [03:15:41] step 4580/4768 loss 2.9421 lr 1.16e-04 gnorm 0.11 50,562 tok/s peakVRAM 13.7GB
238
+ [03:19:08] step 4600/4768 loss 2.9408 lr 1.10e-04 gnorm 0.11 50,556 tok/s peakVRAM 13.7GB
239
+ [03:22:36] step 4620/4768 loss 2.9273 lr 1.04e-04 gnorm 0.10 50,487 tok/s peakVRAM 13.7GB
240
+ [03:26:03] step 4640/4768 loss 2.9367 lr 9.78e-05 gnorm 0.10 50,539 tok/s peakVRAM 13.7GB
241
+ [03:29:31] step 4660/4768 loss 2.9295 lr 9.18e-05 gnorm 0.11 50,554 tok/s peakVRAM 13.7GB
242
+ [03:32:58] step 4680/4768 loss 2.9272 lr 8.58e-05 gnorm 0.10 50,523 tok/s peakVRAM 13.7GB
243
+ [03:36:26] step 4700/4768 loss 2.9233 lr 7.99e-05 gnorm 0.10 50,502 tok/s peakVRAM 13.7GB
244
+ [03:39:54] step 4720/4768 loss 2.9241 lr 7.41e-05 gnorm 0.10 50,527 tok/s peakVRAM 13.7GB
245
+ [03:43:21] step 4740/4768 loss 2.9223 lr 6.83e-05 gnorm 0.10 50,522 tok/s peakVRAM 13.7GB
246
+ [03:46:49] step 4760/4768 loss 2.9311 lr 6.26e-05 gnorm 0.10 50,546 tok/s peakVRAM 13.7GB
247
+ [03:48:13] ckpt @ 4768
248
+ [03:48:13] DONE
K22/run/val_en.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"schema": "val_bpb_v0", "ckpt_sha256": "0840bf3078d0f84fc20e68337cc24e8a9e4bba145f7ea0d27541792c1dd7f189", "step": 4768, "val_sha256": "046bc03b426b75f0e84db7cdf3965a9653d94e722b81b0d03fd0cb92884cf0c5", "tokenizer_sha256": "0640d3bd3674d7a6e59c540d945a2ad275a12e1bcaeb6007c37a90751f9815dd", "script_sha256": "ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1", "trainer_sha256": "db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b", "window": 4096, "n_windows": 547, "tokens": 2240512, "bytes": 9799872, "sum_nat": 6818977.61328125, "bpb": 1.0038605796701956, "ce_token": 3.0434907794652517, "P": 180916757, "dtype": "fp32, tf32 off, bez autocast", "torch": "2.6.0+cu124", "device": "NVIDIA GeForce RTX 4090"}
K22/run/val_en.out ADDED
@@ -0,0 +1 @@
 
 
1
+ VAL-BPB-OK step 4768 bpb 1.00386 ce 3.04349 bajty 9799872 P 180916757 -> /home/ubuntu/kolektyw/dowody400m/wyniki/K22/val_en.json
K22/run/val_pl.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"schema": "val_bpb_v0", "ckpt_sha256": "0840bf3078d0f84fc20e68337cc24e8a9e4bba145f7ea0d27541792c1dd7f189", "step": 4768, "val_sha256": "99e66265f47d249c72f5e76838aa23e82959b99089f720bea6e9ee2cb1f49ee1", "tokenizer_sha256": "0640d3bd3674d7a6e59c540d945a2ad275a12e1bcaeb6007c37a90751f9815dd", "script_sha256": "ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1", "trainer_sha256": "db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b", "window": 4096, "n_windows": 406, "tokens": 1662976, "bytes": 7234678, "sum_nat": 5472073.984375, "bpb": 1.0912073765598584, "ce_token": 3.290530942343726, "P": 180916757, "dtype": "fp32, tf32 off, bez autocast", "torch": "2.6.0+cu124", "device": "NVIDIA GeForce RTX 4090"}
K22/run/val_pl.out ADDED
@@ -0,0 +1 @@
 
 
1
+ VAL-BPB-OK step 4768 bpb 1.09121 ce 3.29053 bajty 7234678 P 180916757 -> /home/ubuntu/kolektyw/dowody400m/wyniki/K22/val_pl.json
K22/run/wynik.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"ramie": "K22", "ckpt_sha256": "0840bf3078d0f84fc20e68337cc24e8a9e4bba145f7ea0d27541792c1dd7f189", "bpb_pl": 1.0912073765598584, "bpb_en": 1.0038605796701956, "bpb_wazona_40_60": 1.0387992984260608, "val_pl_sha256": "99e66265f47d249c72f5e76838aa23e82959b99089f720bea6e9ee2cb1f49ee1", "val_en_sha256": "046bc03b426b75f0e84db7cdf3965a9653d94e722b81b0d03fd0cb92884cf0c5", "P": 180916757, "step": 4768}
MU2/run/SHA256SUMS_kopiarka ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ dafb39311a0abbd298efc00372e8d574ef255c2e254dd964f99f92df7ad5116e ckpt_500.pt
2
+ d1c16074f435fed71a47e0c377bc3332344ad5bc29c4dfef86dee55c66a7f396 ckpt_1000.pt
3
+ edf8bdeae17f21b10e59fe5f56e793a3bfe92a5a28da86d1d8eaf21d8fad6602 ckpt_1500.pt
4
+ 72507b5463e7716781fe783b1599cac8dd2fcb9c10823d47aada75c2bf5b7248 ckpt_2000.pt
MU2/run/args.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ --data-dir /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2 --out-dir /home/ubuntu/kolektyw/dowody400m/wyniki/MU2 --run-id dowody_MU2 --vocab 32768 --dtype uint16 --block 4096 --n-layer 16 --n-embd 896 --n-head 14 --value-residual --batch 128 --grad-accum 8 --norm rmsnorm --pos rope --rope-theta 500000 --ffn swiglu --ffn-mult 2.667 --qk-norm --attn-bias 0 --optimizer muon --muon-lr 0.04 --lr 6e-4 --min-lr 6e-05 --wd 0.1 --schedule wsd --decay-frac 0.2 --warmup 500 --act-ckpt --loss-chunk 1024 --sampler random --seed 1 --data-seed 20261022 --compile --log-every 20 --eval-every 4769 --eval-iters 1 --steps 4768 --ckpt-every 500
MU2/run/metrics.jsonl ADDED
@@ -0,0 +1,117 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"step": 20, "loss": 10.25530424118042, "lr": 2.3999999999999997e-05, "tok_s": 44303.59020061327}
2
+ {"step": 40, "loss": 9.296780109405518, "lr": 4.7999999999999994e-05, "tok_s": 54450.233286746974}
3
+ {"step": 60, "loss": 8.495148754119873, "lr": 7.199999999999999e-05, "tok_s": 54449.48407584481}
4
+ {"step": 80, "loss": 7.82485179901123, "lr": 9.599999999999999e-05, "tok_s": 54438.767439872885}
5
+ {"step": 100, "loss": 7.302724242210388, "lr": 0.00011999999999999999, "tok_s": 54420.73828892327}
6
+ {"step": 120, "loss": 6.796665596961975, "lr": 0.00014399999999999998, "tok_s": 54419.8315794307}
7
+ {"step": 140, "loss": 6.40500955581665, "lr": 0.000168, "tok_s": 54415.44303424002}
8
+ {"step": 160, "loss": 6.1027823209762575, "lr": 0.00019199999999999998, "tok_s": 54413.32530167493}
9
+ {"step": 180, "loss": 5.878649806976318, "lr": 0.00021599999999999996, "tok_s": 54414.32557584492}
10
+ {"step": 200, "loss": 5.675228881835937, "lr": 0.00023999999999999998, "tok_s": 54411.914760814034}
11
+ {"step": 220, "loss": 5.494605636596679, "lr": 0.00026399999999999997, "tok_s": 54411.29134027191}
12
+ {"step": 240, "loss": 5.32799596786499, "lr": 0.00028799999999999995, "tok_s": 54408.15432666468}
13
+ {"step": 260, "loss": 5.167963361740112, "lr": 0.000312, "tok_s": 54407.70774022487}
14
+ {"step": 280, "loss": 5.026251649856567, "lr": 0.000336, "tok_s": 54403.17558868095}
15
+ {"step": 300, "loss": 4.884607744216919, "lr": 0.00035999999999999997, "tok_s": 54403.6589795126}
16
+ {"step": 320, "loss": 4.747219347953797, "lr": 0.00038399999999999996, "tok_s": 54402.308561967555}
17
+ {"step": 340, "loss": 4.611021494865417, "lr": 0.000408, "tok_s": 54398.66464668593}
18
+ {"step": 360, "loss": 4.5043387651443485, "lr": 0.00043199999999999993, "tok_s": 54397.36379438711}
19
+ {"step": 380, "loss": 4.405326771736145, "lr": 0.00045599999999999997, "tok_s": 54389.69034144712}
20
+ {"step": 400, "loss": 4.3193655729293825, "lr": 0.00047999999999999996, "tok_s": 54388.95342288856}
21
+ {"step": 420, "loss": 4.254396152496338, "lr": 0.000504, "tok_s": 54392.29332507222}
22
+ {"step": 440, "loss": 4.184388732910156, "lr": 0.0005279999999999999, "tok_s": 54391.702709732206}
23
+ {"step": 460, "loss": 4.12065064907074, "lr": 0.000552, "tok_s": 54389.70399574904}
24
+ {"step": 480, "loss": 4.079915833473206, "lr": 0.0005759999999999999, "tok_s": 54388.5113889538}
25
+ {"step": 500, "loss": 4.029127311706543, "lr": 0.0006, "tok_s": 54385.7028387596}
26
+ {"step": 520, "loss": 3.99461567401886, "lr": 0.0006, "tok_s": 53917.76280203575}
27
+ {"step": 540, "loss": 3.9496883153915405, "lr": 0.0006, "tok_s": 54387.81579867178}
28
+ {"step": 560, "loss": 3.8994050145149233, "lr": 0.0006, "tok_s": 54386.183497997205}
29
+ {"step": 580, "loss": 3.8611037373542785, "lr": 0.0006, "tok_s": 54384.45109321471}
30
+ {"step": 600, "loss": 3.8205567002296448, "lr": 0.0006, "tok_s": 54386.24853257764}
31
+ {"step": 620, "loss": 3.79986629486084, "lr": 0.0006, "tok_s": 54384.0326021977}
32
+ {"step": 640, "loss": 3.775229072570801, "lr": 0.0006, "tok_s": 54381.57639218699}
33
+ {"step": 660, "loss": 3.747225058078766, "lr": 0.0006, "tok_s": 54379.4114047863}
34
+ {"step": 680, "loss": 3.7248820304870605, "lr": 0.0006, "tok_s": 54375.93439154698}
35
+ {"step": 700, "loss": 3.6958727717399595, "lr": 0.0006, "tok_s": 54382.27256256626}
36
+ {"step": 720, "loss": 3.6669984459877014, "lr": 0.0006, "tok_s": 54381.27561820643}
37
+ {"step": 740, "loss": 3.663788890838623, "lr": 0.0006, "tok_s": 54382.09194522191}
38
+ {"step": 760, "loss": 3.6385889649391174, "lr": 0.0006, "tok_s": 54382.89444411851}
39
+ {"step": 780, "loss": 3.6130353808403015, "lr": 0.0006, "tok_s": 54386.28518605501}
40
+ {"step": 800, "loss": 3.6004954099655153, "lr": 0.0006, "tok_s": 54384.68539181737}
41
+ {"step": 820, "loss": 3.5843114852905273, "lr": 0.0006, "tok_s": 54385.4770051577}
42
+ {"step": 840, "loss": 3.5712699770927427, "lr": 0.0006, "tok_s": 54381.674364666964}
43
+ {"step": 860, "loss": 3.5506399631500245, "lr": 0.0006, "tok_s": 54383.550703277906}
44
+ {"step": 880, "loss": 3.545134401321411, "lr": 0.0006, "tok_s": 54385.182509659986}
45
+ {"step": 900, "loss": 3.5280290246009827, "lr": 0.0006, "tok_s": 54387.28069994241}
46
+ {"step": 920, "loss": 3.5207173466682433, "lr": 0.0006, "tok_s": 54382.421710629926}
47
+ {"step": 940, "loss": 3.51641960144043, "lr": 0.0006, "tok_s": 54383.32865337053}
48
+ {"step": 960, "loss": 3.491331160068512, "lr": 0.0006, "tok_s": 54381.12076091297}
49
+ {"step": 980, "loss": 3.488969933986664, "lr": 0.0006, "tok_s": 54379.83937250632}
50
+ {"step": 1000, "loss": 3.4787667393684387, "lr": 0.0006, "tok_s": 54379.45826910843}
51
+ {"step": 1020, "loss": 3.47394437789917, "lr": 0.0006, "tok_s": 53735.04933796688}
52
+ {"step": 1040, "loss": 3.447183310985565, "lr": 0.0006, "tok_s": 54384.39453636309}
53
+ {"step": 1060, "loss": 3.458915412425995, "lr": 0.0006, "tok_s": 54382.80339353018}
54
+ {"step": 1080, "loss": 3.4484529376029966, "lr": 0.0006, "tok_s": 54378.62864135084}
55
+ {"step": 1100, "loss": 3.433352470397949, "lr": 0.0006, "tok_s": 54381.845766923994}
56
+ {"step": 1120, "loss": 3.4153954148292542, "lr": 0.0006, "tok_s": 54378.996554840145}
57
+ {"step": 1140, "loss": 3.4083414316177367, "lr": 0.0006, "tok_s": 54382.7574648779}
58
+ {"step": 1160, "loss": 3.403551959991455, "lr": 0.0006, "tok_s": 54383.10808469444}
59
+ {"step": 1180, "loss": 3.40577654838562, "lr": 0.0006, "tok_s": 54380.83626420643}
60
+ {"step": 1200, "loss": 3.3882885336875916, "lr": 0.0006, "tok_s": 54381.06421098787}
61
+ {"step": 1220, "loss": 3.385551917552948, "lr": 0.0006, "tok_s": 54376.27725923776}
62
+ {"step": 1240, "loss": 3.3774680852890016, "lr": 0.0006, "tok_s": 54378.30409899}
63
+ {"step": 1260, "loss": 3.364372217655182, "lr": 0.0006, "tok_s": 54380.226129333794}
64
+ {"step": 1280, "loss": 3.3675967693328857, "lr": 0.0006, "tok_s": 54382.87803616131}
65
+ {"step": 1300, "loss": 3.3608749508857727, "lr": 0.0006, "tok_s": 54380.05951123605}
66
+ {"step": 1320, "loss": 3.3530895113945007, "lr": 0.0006, "tok_s": 54380.28624119837}
67
+ {"step": 1340, "loss": 3.3537341475486757, "lr": 0.0006, "tok_s": 54379.937943873556}
68
+ {"step": 1360, "loss": 3.3449841737747192, "lr": 0.0006, "tok_s": 54378.28446655379}
69
+ {"step": 1380, "loss": 3.3367194056510927, "lr": 0.0006, "tok_s": 54380.54074312651}
70
+ {"step": 1400, "loss": 3.3268325209617613, "lr": 0.0006, "tok_s": 54381.70583428245}
71
+ {"step": 1420, "loss": 3.3142680764198302, "lr": 0.0006, "tok_s": 54385.200869299784}
72
+ {"step": 1440, "loss": 3.321825695037842, "lr": 0.0006, "tok_s": 54383.38130768183}
73
+ {"step": 1460, "loss": 3.3194995641708376, "lr": 0.0006, "tok_s": 54381.14503505483}
74
+ {"step": 1480, "loss": 3.323328506946564, "lr": 0.0006, "tok_s": 54379.48812246335}
75
+ {"step": 1500, "loss": 3.2963005781173704, "lr": 0.0006, "tok_s": 54380.93403229694}
76
+ {"step": 1520, "loss": 3.304063355922699, "lr": 0.0006, "tok_s": 53713.09851579601}
77
+ {"step": 1540, "loss": 3.28833954334259, "lr": 0.0006, "tok_s": 54373.54082492265}
78
+ {"step": 1560, "loss": 3.2969040393829347, "lr": 0.0006, "tok_s": 54376.58463371382}
79
+ {"step": 1580, "loss": 3.2850605249404907, "lr": 0.0006, "tok_s": 54375.742454756866}
80
+ {"step": 1600, "loss": 3.2749640941619873, "lr": 0.0006, "tok_s": 54377.58006143249}
81
+ {"step": 1620, "loss": 3.2752556204795837, "lr": 0.0006, "tok_s": 54376.30650403504}
82
+ {"step": 1640, "loss": 3.2726037979125975, "lr": 0.0006, "tok_s": 40751.557960654085}
83
+ {"step": 1660, "loss": 3.273281013965607, "lr": 0.0006, "tok_s": 39289.77490753473}
84
+ {"step": 1680, "loss": 3.26226863861084, "lr": 0.0006, "tok_s": 40313.389668948854}
85
+ {"step": 1700, "loss": 3.2555036783218383, "lr": 0.0006, "tok_s": 54379.78369934085}
86
+ {"step": 1720, "loss": 3.25777188539505, "lr": 0.0006, "tok_s": 54379.535726529546}
87
+ {"step": 1740, "loss": 3.251254212856293, "lr": 0.0006, "tok_s": 54384.92964555243}
88
+ {"step": 1760, "loss": 3.2507732033729555, "lr": 0.0006, "tok_s": 54388.55746184499}
89
+ {"step": 1780, "loss": 3.238064002990723, "lr": 0.0006, "tok_s": 54383.73502894239}
90
+ {"step": 1800, "loss": 3.2417608499526978, "lr": 0.0006, "tok_s": 54379.47595251065}
91
+ {"step": 1820, "loss": 3.2374199867248534, "lr": 0.0006, "tok_s": 53645.94184194194}
92
+ {"step": 1840, "loss": 3.2312928795814515, "lr": 0.0006, "tok_s": 30466.846568588197}
93
+ {"step": 1860, "loss": 3.230325162410736, "lr": 0.0006, "tok_s": 44403.16651628122}
94
+ {"step": 1880, "loss": 3.2265440702438353, "lr": 0.0006, "tok_s": 31623.168463345162}
95
+ {"step": 1900, "loss": 3.2208963513374327, "lr": 0.0006, "tok_s": 31365.17852899342}
96
+ {"step": 1920, "loss": 3.2205076932907106, "lr": 0.0006, "tok_s": 29664.547690026135}
97
+ {"step": 1940, "loss": 3.2140785694122314, "lr": 0.0006, "tok_s": 29662.386463764484}
98
+ {"step": 1960, "loss": 3.2180041909217834, "lr": 0.0006, "tok_s": 30456.666177317657}
99
+ {"step": 1980, "loss": 3.2126453638076784, "lr": 0.0006, "tok_s": 32075.633695387703}
100
+ {"step": 2000, "loss": 3.2058145880699156, "lr": 0.0006, "tok_s": 32046.597929714466}
101
+ {"step": 2020, "loss": 3.2054898142814636, "lr": 0.0006, "tok_s": 31786.004615394697}
102
+ {"step": 2040, "loss": 3.1983474612236025, "lr": 0.0006, "tok_s": 31965.00091165428}
103
+ {"step": 2060, "loss": 3.1985918998718263, "lr": 0.0006, "tok_s": 31994.541837655575}
104
+ {"step": 2080, "loss": 3.187583899497986, "lr": 0.0006, "tok_s": 32010.290784090703}
105
+ {"step": 2100, "loss": 3.196134352684021, "lr": 0.0006, "tok_s": 31946.15269744886}
106
+ {"step": 2120, "loss": 3.1858644843101502, "lr": 0.0006, "tok_s": 31941.206322038823}
107
+ {"step": 2140, "loss": 3.1989166855812075, "lr": 0.0006, "tok_s": 32012.582473647}
108
+ {"step": 2160, "loss": 3.1776081800460814, "lr": 0.0006, "tok_s": 31944.20210738539}
109
+ {"step": 2180, "loss": 3.179273557662964, "lr": 0.0006, "tok_s": 31909.084135585163}
110
+ {"step": 2200, "loss": 3.180311906337738, "lr": 0.0006, "tok_s": 39315.18392988774}
111
+ {"step": 2220, "loss": 3.1733631134033202, "lr": 0.0006, "tok_s": 34362.62429152448}
112
+ {"step": 2240, "loss": 3.170001435279846, "lr": 0.0006, "tok_s": 28500.10494578955}
113
+ {"step": 2260, "loss": 3.166220474243164, "lr": 0.0006, "tok_s": 32518.169542876723}
114
+ {"step": 2280, "loss": 3.1624667048454285, "lr": 0.0006, "tok_s": 30345.85548236686}
115
+ {"step": 2300, "loss": 3.165924310684204, "lr": 0.0006, "tok_s": 29311.251041075742}
116
+ {"step": 2320, "loss": 3.1626460313797, "lr": 0.0006, "tok_s": 34479.854490218255}
117
+ {"step": 2340, "loss": 3.157814919948578, "lr": 0.0006, "tok_s": 28009.653677274284}
MU2/run/srodowisko_MU2_mini0_resume0.txt ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-23T02:48:25Z ARM MU2 GPU 0 MINI 0 DATA_SEED 20261022 CKPT 500 RESUME 0 DEST /var/lib/docker/kolektyw/dowody400m_ckpt/wyniki_MU2
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ 0702c32c0ad602d8d12b16539e05cfbc68d1823cfd39b82b984357d78f27f664 /home/ubuntu/kolektyw/dowody400m/kod/ramie_v03.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 341aae81ce8512294dcd7de93ced26053025677b4ab449606be989f7fd898d27 /home/ubuntu/kolektyw/dowody400m/kod/kopiarka_ckpt.sh
7
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
8
+ 2.6.0+cu124 12.4
9
+ accelerate==1.15.0
10
+ annotated-doc==0.0.5
11
+ anyio==4.15.1
12
+ certifi==2026.10.19
13
+ click==8.5.0
14
+ filelock==4.2.4
15
+ fsspec==2026.9.0
16
+ h11==0.16.0
17
+ hf-xet==1.7.0
18
+ httpcore==1.0.9
19
+ httpx==0.28.1
20
+ huggingface_hub==1.33.0
21
+ idna==3.20
22
+ Jinja2==3.1.6
23
+ markdown-it-py==4.2.0
24
+ MarkupSafe==3.0.4
25
+ mdurl==0.1.2
26
+ mpmath==1.3.0
27
+ networkx==3.7
28
+ numpy==2.2.6
29
+ nvidia-cublas-cu12==12.4.5.8
30
+ nvidia-cuda-cupti-cu12==12.4.127
31
+ nvidia-cuda-nvrtc-cu12==12.4.127
32
+ nvidia-cuda-runtime-cu12==12.4.127
33
+ nvidia-cudnn-cu12==9.1.0.70
34
+ nvidia-cufft-cu12==11.2.1.3
35
+ nvidia-curand-cu12==10.3.5.147
36
+ nvidia-cusolver-cu12==11.6.1.9
37
+ nvidia-cusparse-cu12==12.3.1.170
38
+ nvidia-cusparselt-cu12==0.6.2
39
+ nvidia-nccl-cu12==2.21.5
40
+ nvidia-nvjitlink-cu12==12.4.127
41
+ nvidia-nvtx-cu12==12.4.127
42
+ packaging==26.3
43
+ pillow==12.3.0
44
+ psutil==7.2.2
45
+ Pygments==2.21.0
46
+ PyYAML==6.0.3
47
+ regex==2026.9.29
48
+ rich==15.0.0
49
+ safetensors==0.8.0
50
+ setuptools==84.2.0
51
+ shellingham==1.5.4
52
+ sympy==1.13.1
53
+ tokenizers==0.23.3
54
+ torch==2.6.0+cu124
55
+ torchvision==0.21.0+cu124
56
+ tqdm==4.70.1
57
+ transformers==5.17.0
58
+ triton==3.2.0
59
+ typer==0.27.3
60
+ typing_extensions==4.16.0
61
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 1, NVIDIA GeForce RTX 4090, 15082 MiB, 595.104.02
63
+ 2, NVIDIA GeForce RTX 4090, 14222 MiB, 595.104.02
64
+ 3, NVIDIA GeForce RTX 4090, 15482 MiB, 595.104.02
MU2/run/srodowisko_resume0.txt ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-23T02:48:25Z ARM MU2 GPU 0 MINI 0 DATA_SEED 20261022 CKPT 500 RESUME 0 DEST /var/lib/docker/kolektyw/dowody400m_ckpt/wyniki_MU2
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ 0702c32c0ad602d8d12b16539e05cfbc68d1823cfd39b82b984357d78f27f664 /home/ubuntu/kolektyw/dowody400m/kod/ramie_v03.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 341aae81ce8512294dcd7de93ced26053025677b4ab449606be989f7fd898d27 /home/ubuntu/kolektyw/dowody400m/kod/kopiarka_ckpt.sh
7
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
8
+ 2.6.0+cu124 12.4
9
+ accelerate==1.15.0
10
+ annotated-doc==0.0.5
11
+ anyio==4.15.1
12
+ certifi==2026.10.19
13
+ click==8.5.0
14
+ filelock==4.2.4
15
+ fsspec==2026.9.0
16
+ h11==0.16.0
17
+ hf-xet==1.7.0
18
+ httpcore==1.0.9
19
+ httpx==0.28.1
20
+ huggingface_hub==1.33.0
21
+ idna==3.20
22
+ Jinja2==3.1.6
23
+ markdown-it-py==4.2.0
24
+ MarkupSafe==3.0.4
25
+ mdurl==0.1.2
26
+ mpmath==1.3.0
27
+ networkx==3.7
28
+ numpy==2.2.6
29
+ nvidia-cublas-cu12==12.4.5.8
30
+ nvidia-cuda-cupti-cu12==12.4.127
31
+ nvidia-cuda-nvrtc-cu12==12.4.127
32
+ nvidia-cuda-runtime-cu12==12.4.127
33
+ nvidia-cudnn-cu12==9.1.0.70
34
+ nvidia-cufft-cu12==11.2.1.3
35
+ nvidia-curand-cu12==10.3.5.147
36
+ nvidia-cusolver-cu12==11.6.1.9
37
+ nvidia-cusparse-cu12==12.3.1.170
38
+ nvidia-cusparselt-cu12==0.6.2
39
+ nvidia-nccl-cu12==2.21.5
40
+ nvidia-nvjitlink-cu12==12.4.127
41
+ nvidia-nvtx-cu12==12.4.127
42
+ packaging==26.3
43
+ pillow==12.3.0
44
+ psutil==7.2.2
45
+ Pygments==2.21.0
46
+ PyYAML==6.0.3
47
+ regex==2026.9.29
48
+ rich==15.0.0
49
+ safetensors==0.8.0
50
+ setuptools==84.2.0
51
+ shellingham==1.5.4
52
+ sympy==1.13.1
53
+ tokenizers==0.23.3
54
+ torch==2.6.0+cu124
55
+ torchvision==0.21.0+cu124
56
+ tqdm==4.70.1
57
+ transformers==5.17.0
58
+ triton==3.2.0
59
+ typer==0.27.3
60
+ typing_extensions==4.16.0
61
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 1, NVIDIA GeForce RTX 4090, 15082 MiB, 595.104.02
63
+ 2, NVIDIA GeForce RTX 4090, 14222 MiB, 595.104.02
64
+ 3, NVIDIA GeForce RTX 4090, 15482 MiB, 595.104.02
MU2/run/train.log ADDED
@@ -0,0 +1,129 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [04:48:32] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [04:48:32] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [04:48:34] model GPT-ref: 183.6M param (L16 d896 h14)
4
+ [04:48:34] optimizer=muon muon_lr=0.04 (mult=66.7x)
5
+ [04:48:34] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [04:48:34] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [04:48:35] torch.compile enabled
8
+ [04:48:35] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [04:52:32] step 20/4768 loss 10.2553 lr 2.40e-05 gnorm 2.32 44,304 tok/s peakVRAM 13.7GB
10
+ [04:55:44] step 40/4768 loss 9.2968 lr 4.80e-05 gnorm 1.00 54,450 tok/s peakVRAM 13.7GB
11
+ [04:58:57] step 60/4768 loss 8.4951 lr 7.20e-05 gnorm 1.46 54,449 tok/s peakVRAM 13.7GB
12
+ [05:02:10] step 80/4768 loss 7.8249 lr 9.60e-05 gnorm 0.69 54,439 tok/s peakVRAM 13.7GB
13
+ [05:05:22] step 100/4768 loss 7.3027 lr 1.20e-04 gnorm 0.65 54,421 tok/s peakVRAM 13.7GB
14
+ [05:08:35] step 120/4768 loss 6.7967 lr 1.44e-04 gnorm 0.71 54,420 tok/s peakVRAM 13.7GB
15
+ [05:11:48] step 140/4768 loss 6.4050 lr 1.68e-04 gnorm 0.49 54,415 tok/s peakVRAM 13.7GB
16
+ [05:15:00] step 160/4768 loss 6.1028 lr 1.92e-04 gnorm 0.54 54,413 tok/s peakVRAM 13.7GB
17
+ [05:18:13] step 180/4768 loss 5.8786 lr 2.16e-04 gnorm 0.51 54,414 tok/s peakVRAM 13.7GB
18
+ [05:21:26] step 200/4768 loss 5.6752 lr 2.40e-04 gnorm 0.56 54,412 tok/s peakVRAM 13.7GB
19
+ [05:24:38] step 220/4768 loss 5.4946 lr 2.64e-04 gnorm 0.62 54,411 tok/s peakVRAM 13.7GB
20
+ [05:27:51] step 240/4768 loss 5.3280 lr 2.88e-04 gnorm 0.57 54,408 tok/s peakVRAM 13.7GB
21
+ [05:31:04] step 260/4768 loss 5.1680 lr 3.12e-04 gnorm 0.43 54,408 tok/s peakVRAM 13.7GB
22
+ [05:34:17] step 280/4768 loss 5.0263 lr 3.36e-04 gnorm 0.37 54,403 tok/s peakVRAM 13.7GB
23
+ [05:37:29] step 300/4768 loss 4.8846 lr 3.60e-04 gnorm 0.45 54,404 tok/s peakVRAM 13.7GB
24
+ [05:40:42] step 320/4768 loss 4.7472 lr 3.84e-04 gnorm 0.32 54,402 tok/s peakVRAM 13.7GB
25
+ [05:43:55] step 340/4768 loss 4.6110 lr 4.08e-04 gnorm 0.33 54,399 tok/s peakVRAM 13.7GB
26
+ [05:47:08] step 360/4768 loss 4.5043 lr 4.32e-04 gnorm 0.26 54,397 tok/s peakVRAM 13.7GB
27
+ [05:50:20] step 380/4768 loss 4.4053 lr 4.56e-04 gnorm 0.22 54,390 tok/s peakVRAM 13.7GB
28
+ [05:53:33] step 400/4768 loss 4.3194 lr 4.80e-04 gnorm 0.25 54,389 tok/s peakVRAM 13.7GB
29
+ [05:56:46] step 420/4768 loss 4.2544 lr 5.04e-04 gnorm 0.21 54,392 tok/s peakVRAM 13.7GB
30
+ [05:59:59] step 440/4768 loss 4.1844 lr 5.28e-04 gnorm 0.21 54,392 tok/s peakVRAM 13.7GB
31
+ [06:03:12] step 460/4768 loss 4.1207 lr 5.52e-04 gnorm 0.18 54,390 tok/s peakVRAM 13.7GB
32
+ [06:06:24] step 480/4768 loss 4.0799 lr 5.76e-04 gnorm 0.22 54,389 tok/s peakVRAM 13.7GB
33
+ [06:09:37] step 500/4768 loss 4.0291 lr 6.00e-04 gnorm 0.18 54,386 tok/s peakVRAM 13.7GB
34
+ [06:09:39] ckpt @ 500
35
+ [06:12:52] step 520/4768 loss 3.9946 lr 6.00e-04 gnorm 0.16 53,918 tok/s peakVRAM 13.7GB
36
+ [06:16:04] step 540/4768 loss 3.9497 lr 6.00e-04 gnorm 0.16 54,388 tok/s peakVRAM 13.7GB
37
+ [06:19:17] step 560/4768 loss 3.8994 lr 6.00e-04 gnorm 0.20 54,386 tok/s peakVRAM 13.7GB
38
+ [06:22:30] step 580/4768 loss 3.8611 lr 6.00e-04 gnorm 0.15 54,384 tok/s peakVRAM 13.7GB
39
+ [06:25:43] step 600/4768 loss 3.8206 lr 6.00e-04 gnorm 0.15 54,386 tok/s peakVRAM 13.7GB
40
+ [06:28:56] step 620/4768 loss 3.7999 lr 6.00e-04 gnorm 0.14 54,384 tok/s peakVRAM 13.7GB
41
+ [06:32:09] step 640/4768 loss 3.7752 lr 6.00e-04 gnorm 0.12 54,382 tok/s peakVRAM 13.7GB
42
+ [06:35:21] step 660/4768 loss 3.7472 lr 6.00e-04 gnorm 0.11 54,379 tok/s peakVRAM 13.7GB
43
+ [06:38:34] step 680/4768 loss 3.7249 lr 6.00e-04 gnorm 0.13 54,376 tok/s peakVRAM 13.7GB
44
+ [06:41:47] step 700/4768 loss 3.6959 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
45
+ [06:45:00] step 720/4768 loss 3.6670 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
46
+ [06:48:13] step 740/4768 loss 3.6638 lr 6.00e-04 gnorm 0.12 54,382 tok/s peakVRAM 13.7GB
47
+ [06:51:25] step 760/4768 loss 3.6386 lr 6.00e-04 gnorm 0.12 54,383 tok/s peakVRAM 13.7GB
48
+ [06:54:38] step 780/4768 loss 3.6130 lr 6.00e-04 gnorm 0.10 54,386 tok/s peakVRAM 13.7GB
49
+ [06:57:51] step 800/4768 loss 3.6005 lr 6.00e-04 gnorm 0.12 54,385 tok/s peakVRAM 13.7GB
50
+ [07:01:04] step 820/4768 loss 3.5843 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
51
+ [07:04:17] step 840/4768 loss 3.5713 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
52
+ [07:07:29] step 860/4768 loss 3.5506 lr 6.00e-04 gnorm 0.10 54,384 tok/s peakVRAM 13.7GB
53
+ [07:10:42] step 880/4768 loss 3.5451 lr 6.00e-04 gnorm 0.11 54,385 tok/s peakVRAM 13.7GB
54
+ [07:13:55] step 900/4768 loss 3.5280 lr 6.00e-04 gnorm 0.11 54,387 tok/s peakVRAM 13.7GB
55
+ [07:17:08] step 920/4768 loss 3.5207 lr 6.00e-04 gnorm 0.10 54,382 tok/s peakVRAM 13.7GB
56
+ [07:20:21] step 940/4768 loss 3.5164 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
57
+ [07:23:34] step 960/4768 loss 3.4913 lr 6.00e-04 gnorm 0.08 54,381 tok/s peakVRAM 13.7GB
58
+ [07:26:46] step 980/4768 loss 3.4890 lr 6.00e-04 gnorm 0.08 54,380 tok/s peakVRAM 13.7GB
59
+ [07:29:59] step 1000/4768 loss 3.4788 lr 6.00e-04 gnorm 0.09 54,379 tok/s peakVRAM 13.7GB
60
+ [07:30:02] ckpt @ 1000
61
+ [07:33:14] step 1020/4768 loss 3.4739 lr 6.00e-04 gnorm 0.12 53,735 tok/s peakVRAM 13.7GB
62
+ [07:36:27] step 1040/4768 loss 3.4472 lr 6.00e-04 gnorm 0.09 54,384 tok/s peakVRAM 13.7GB
63
+ [07:39:40] step 1060/4768 loss 3.4589 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
64
+ [07:42:53] step 1080/4768 loss 3.4485 lr 6.00e-04 gnorm 0.09 54,379 tok/s peakVRAM 13.7GB
65
+ [07:46:06] step 1100/4768 loss 3.4334 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
66
+ [07:49:18] step 1120/4768 loss 3.4154 lr 6.00e-04 gnorm 0.10 54,379 tok/s peakVRAM 13.7GB
67
+ [07:52:31] step 1140/4768 loss 3.4083 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
68
+ [07:55:44] step 1160/4768 loss 3.4036 lr 6.00e-04 gnorm 0.09 54,383 tok/s peakVRAM 13.7GB
69
+ [07:58:57] step 1180/4768 loss 3.4058 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
70
+ [08:02:10] step 1200/4768 loss 3.3883 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
71
+ [08:05:23] step 1220/4768 loss 3.3856 lr 6.00e-04 gnorm 0.10 54,376 tok/s peakVRAM 13.7GB
72
+ [08:08:35] step 1240/4768 loss 3.3775 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
73
+ [08:11:48] step 1260/4768 loss 3.3644 lr 6.00e-04 gnorm 0.10 54,380 tok/s peakVRAM 13.7GB
74
+ [08:15:01] step 1280/4768 loss 3.3676 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
75
+ [08:18:14] step 1300/4768 loss 3.3609 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
76
+ [08:21:27] step 1320/4768 loss 3.3531 lr 6.00e-04 gnorm 0.10 54,380 tok/s peakVRAM 13.7GB
77
+ [08:24:39] step 1340/4768 loss 3.3537 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
78
+ [08:27:52] step 1360/4768 loss 3.3450 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
79
+ [08:31:05] step 1380/4768 loss 3.3367 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
80
+ [08:34:18] step 1400/4768 loss 3.3268 lr 6.00e-04 gnorm 0.09 54,382 tok/s peakVRAM 13.7GB
81
+ [08:37:31] step 1420/4768 loss 3.3143 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
82
+ [08:40:44] step 1440/4768 loss 3.3218 lr 6.00e-04 gnorm 0.08 54,383 tok/s peakVRAM 13.7GB
83
+ [08:43:56] step 1460/4768 loss 3.3195 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
84
+ [08:47:09] step 1480/4768 loss 3.3233 lr 6.00e-04 gnorm 0.08 54,379 tok/s peakVRAM 13.7GB
85
+ [08:50:22] step 1500/4768 loss 3.2963 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
86
+ [08:50:24] ckpt @ 1500
87
+ [08:53:37] step 1520/4768 loss 3.3041 lr 6.00e-04 gnorm 0.09 53,713 tok/s peakVRAM 13.7GB
88
+ [08:56:50] step 1540/4768 loss 3.2883 lr 6.00e-04 gnorm 0.09 54,374 tok/s peakVRAM 13.7GB
89
+ [09:00:03] step 1560/4768 loss 3.2969 lr 6.00e-04 gnorm 0.08 54,377 tok/s peakVRAM 13.7GB
90
+ [09:03:16] step 1580/4768 loss 3.2851 lr 6.00e-04 gnorm 0.09 54,376 tok/s peakVRAM 13.7GB
91
+ [09:06:29] step 1600/4768 loss 3.2750 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
92
+ [09:09:41] step 1620/4768 loss 3.2753 lr 6.00e-04 gnorm 0.09 54,376 tok/s peakVRAM 13.7GB
93
+ [09:13:59] step 1640/4768 loss 3.2726 lr 6.00e-04 gnorm 0.09 40,752 tok/s peakVRAM 13.7GB
94
+ [09:18:26] step 1660/4768 loss 3.2733 lr 6.00e-04 gnorm 0.09 39,290 tok/s peakVRAM 13.7GB
95
+ [09:22:46] step 1680/4768 loss 3.2623 lr 6.00e-04 gnorm 0.10 40,313 tok/s peakVRAM 13.7GB
96
+ [09:25:59] step 1700/4768 loss 3.2555 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
97
+ [09:29:11] step 1720/4768 loss 3.2578 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
98
+ [09:32:24] step 1740/4768 loss 3.2513 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
99
+ [09:35:37] step 1760/4768 loss 3.2508 lr 6.00e-04 gnorm 0.10 54,389 tok/s peakVRAM 13.7GB
100
+ [09:38:50] step 1780/4768 loss 3.2381 lr 6.00e-04 gnorm 0.10 54,384 tok/s peakVRAM 13.7GB
101
+ [09:42:03] step 1800/4768 loss 3.2418 lr 6.00e-04 gnorm 0.10 54,379 tok/s peakVRAM 13.7GB
102
+ [09:45:18] step 1820/4768 loss 3.2374 lr 6.00e-04 gnorm 0.08 53,646 tok/s peakVRAM 13.7GB
103
+ [09:51:02] step 1840/4768 loss 3.2313 lr 6.00e-04 gnorm 0.09 30,467 tok/s peakVRAM 13.7GB
104
+ [09:54:58] step 1860/4768 loss 3.2303 lr 6.00e-04 gnorm 0.09 44,403 tok/s peakVRAM 13.7GB
105
+ [10:00:30] step 1880/4768 loss 3.2265 lr 6.00e-04 gnorm 0.08 31,623 tok/s peakVRAM 13.7GB
106
+ [10:06:04] step 1900/4768 loss 3.2209 lr 6.00e-04 gnorm 0.09 31,365 tok/s peakVRAM 13.7GB
107
+ [10:11:58] step 1920/4768 loss 3.2205 lr 6.00e-04 gnorm 0.08 29,665 tok/s peakVRAM 13.7GB
108
+ [10:17:51] step 1940/4768 loss 3.2141 lr 6.00e-04 gnorm 0.08 29,662 tok/s peakVRAM 13.7GB
109
+ [10:23:36] step 1960/4768 loss 3.2180 lr 6.00e-04 gnorm 0.08 30,457 tok/s peakVRAM 13.7GB
110
+ [10:29:02] step 1980/4768 loss 3.2126 lr 6.00e-04 gnorm 0.09 32,076 tok/s peakVRAM 13.7GB
111
+ [10:34:30] step 2000/4768 loss 3.2058 lr 6.00e-04 gnorm 0.09 32,047 tok/s peakVRAM 13.7GB
112
+ [10:34:32] ckpt @ 2000
113
+ [10:40:00] step 2020/4768 loss 3.2055 lr 6.00e-04 gnorm 0.09 31,786 tok/s peakVRAM 13.7GB
114
+ [10:45:28] step 2040/4768 loss 3.1983 lr 6.00e-04 gnorm 0.09 31,965 tok/s peakVRAM 13.7GB
115
+ [10:50:55] step 2060/4768 loss 3.1986 lr 6.00e-04 gnorm 0.08 31,995 tok/s peakVRAM 13.7GB
116
+ [10:56:23] step 2080/4768 loss 3.1876 lr 6.00e-04 gnorm 0.08 32,010 tok/s peakVRAM 13.7GB
117
+ [11:01:51] step 2100/4768 loss 3.1961 lr 6.00e-04 gnorm 0.10 31,946 tok/s peakVRAM 13.7GB
118
+ [11:07:19] step 2120/4768 loss 3.1859 lr 6.00e-04 gnorm 0.09 31,941 tok/s peakVRAM 13.7GB
119
+ [11:12:47] step 2140/4768 loss 3.1989 lr 6.00e-04 gnorm 0.08 32,013 tok/s peakVRAM 13.7GB
120
+ [11:18:15] step 2160/4768 loss 3.1776 lr 6.00e-04 gnorm 0.08 31,944 tok/s peakVRAM 13.7GB
121
+ [11:23:44] step 2180/4768 loss 3.1793 lr 6.00e-04 gnorm 0.09 31,909 tok/s peakVRAM 13.7GB
122
+ [11:28:11] step 2200/4768 loss 3.1803 lr 6.00e-04 gnorm 0.08 39,315 tok/s peakVRAM 13.7GB
123
+ [11:33:16] step 2220/4768 loss 3.1734 lr 6.00e-04 gnorm 0.09 34,363 tok/s peakVRAM 13.7GB
124
+ [11:39:24] step 2240/4768 loss 3.1700 lr 6.00e-04 gnorm 0.08 28,500 tok/s peakVRAM 13.7GB
125
+ [11:44:46] step 2260/4768 loss 3.1662 lr 6.00e-04 gnorm 0.08 32,518 tok/s peakVRAM 13.7GB
126
+ [11:50:32] step 2280/4768 loss 3.1625 lr 6.00e-04 gnorm 0.08 30,346 tok/s peakVRAM 13.7GB
127
+ [11:56:29] step 2300/4768 loss 3.1659 lr 6.00e-04 gnorm 0.08 29,311 tok/s peakVRAM 13.7GB
128
+ [12:01:34] step 2320/4768 loss 3.1626 lr 6.00e-04 gnorm 0.08 34,480 tok/s peakVRAM 13.7GB
129
+ [12:07:48] step 2340/4768 loss 3.1578 lr 6.00e-04 gnorm 0.09 28,010 tok/s peakVRAM 13.7GB
MU2/run/train.out ADDED
@@ -0,0 +1,129 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [04:48:32] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [04:48:32] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [04:48:34] model GPT-ref: 183.6M param (L16 d896 h14)
4
+ [04:48:34] optimizer=muon muon_lr=0.04 (mult=66.7x)
5
+ [04:48:34] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [04:48:34] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [04:48:35] torch.compile enabled
8
+ [04:48:35] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [04:52:32] step 20/4768 loss 10.2553 lr 2.40e-05 gnorm 2.32 44,304 tok/s peakVRAM 13.7GB
10
+ [04:55:44] step 40/4768 loss 9.2968 lr 4.80e-05 gnorm 1.00 54,450 tok/s peakVRAM 13.7GB
11
+ [04:58:57] step 60/4768 loss 8.4951 lr 7.20e-05 gnorm 1.46 54,449 tok/s peakVRAM 13.7GB
12
+ [05:02:10] step 80/4768 loss 7.8249 lr 9.60e-05 gnorm 0.69 54,439 tok/s peakVRAM 13.7GB
13
+ [05:05:22] step 100/4768 loss 7.3027 lr 1.20e-04 gnorm 0.65 54,421 tok/s peakVRAM 13.7GB
14
+ [05:08:35] step 120/4768 loss 6.7967 lr 1.44e-04 gnorm 0.71 54,420 tok/s peakVRAM 13.7GB
15
+ [05:11:48] step 140/4768 loss 6.4050 lr 1.68e-04 gnorm 0.49 54,415 tok/s peakVRAM 13.7GB
16
+ [05:15:00] step 160/4768 loss 6.1028 lr 1.92e-04 gnorm 0.54 54,413 tok/s peakVRAM 13.7GB
17
+ [05:18:13] step 180/4768 loss 5.8786 lr 2.16e-04 gnorm 0.51 54,414 tok/s peakVRAM 13.7GB
18
+ [05:21:26] step 200/4768 loss 5.6752 lr 2.40e-04 gnorm 0.56 54,412 tok/s peakVRAM 13.7GB
19
+ [05:24:38] step 220/4768 loss 5.4946 lr 2.64e-04 gnorm 0.62 54,411 tok/s peakVRAM 13.7GB
20
+ [05:27:51] step 240/4768 loss 5.3280 lr 2.88e-04 gnorm 0.57 54,408 tok/s peakVRAM 13.7GB
21
+ [05:31:04] step 260/4768 loss 5.1680 lr 3.12e-04 gnorm 0.43 54,408 tok/s peakVRAM 13.7GB
22
+ [05:34:17] step 280/4768 loss 5.0263 lr 3.36e-04 gnorm 0.37 54,403 tok/s peakVRAM 13.7GB
23
+ [05:37:29] step 300/4768 loss 4.8846 lr 3.60e-04 gnorm 0.45 54,404 tok/s peakVRAM 13.7GB
24
+ [05:40:42] step 320/4768 loss 4.7472 lr 3.84e-04 gnorm 0.32 54,402 tok/s peakVRAM 13.7GB
25
+ [05:43:55] step 340/4768 loss 4.6110 lr 4.08e-04 gnorm 0.33 54,399 tok/s peakVRAM 13.7GB
26
+ [05:47:08] step 360/4768 loss 4.5043 lr 4.32e-04 gnorm 0.26 54,397 tok/s peakVRAM 13.7GB
27
+ [05:50:20] step 380/4768 loss 4.4053 lr 4.56e-04 gnorm 0.22 54,390 tok/s peakVRAM 13.7GB
28
+ [05:53:33] step 400/4768 loss 4.3194 lr 4.80e-04 gnorm 0.25 54,389 tok/s peakVRAM 13.7GB
29
+ [05:56:46] step 420/4768 loss 4.2544 lr 5.04e-04 gnorm 0.21 54,392 tok/s peakVRAM 13.7GB
30
+ [05:59:59] step 440/4768 loss 4.1844 lr 5.28e-04 gnorm 0.21 54,392 tok/s peakVRAM 13.7GB
31
+ [06:03:12] step 460/4768 loss 4.1207 lr 5.52e-04 gnorm 0.18 54,390 tok/s peakVRAM 13.7GB
32
+ [06:06:24] step 480/4768 loss 4.0799 lr 5.76e-04 gnorm 0.22 54,389 tok/s peakVRAM 13.7GB
33
+ [06:09:37] step 500/4768 loss 4.0291 lr 6.00e-04 gnorm 0.18 54,386 tok/s peakVRAM 13.7GB
34
+ [06:09:39] ckpt @ 500
35
+ [06:12:52] step 520/4768 loss 3.9946 lr 6.00e-04 gnorm 0.16 53,918 tok/s peakVRAM 13.7GB
36
+ [06:16:04] step 540/4768 loss 3.9497 lr 6.00e-04 gnorm 0.16 54,388 tok/s peakVRAM 13.7GB
37
+ [06:19:17] step 560/4768 loss 3.8994 lr 6.00e-04 gnorm 0.20 54,386 tok/s peakVRAM 13.7GB
38
+ [06:22:30] step 580/4768 loss 3.8611 lr 6.00e-04 gnorm 0.15 54,384 tok/s peakVRAM 13.7GB
39
+ [06:25:43] step 600/4768 loss 3.8206 lr 6.00e-04 gnorm 0.15 54,386 tok/s peakVRAM 13.7GB
40
+ [06:28:56] step 620/4768 loss 3.7999 lr 6.00e-04 gnorm 0.14 54,384 tok/s peakVRAM 13.7GB
41
+ [06:32:09] step 640/4768 loss 3.7752 lr 6.00e-04 gnorm 0.12 54,382 tok/s peakVRAM 13.7GB
42
+ [06:35:21] step 660/4768 loss 3.7472 lr 6.00e-04 gnorm 0.11 54,379 tok/s peakVRAM 13.7GB
43
+ [06:38:34] step 680/4768 loss 3.7249 lr 6.00e-04 gnorm 0.13 54,376 tok/s peakVRAM 13.7GB
44
+ [06:41:47] step 700/4768 loss 3.6959 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
45
+ [06:45:00] step 720/4768 loss 3.6670 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
46
+ [06:48:13] step 740/4768 loss 3.6638 lr 6.00e-04 gnorm 0.12 54,382 tok/s peakVRAM 13.7GB
47
+ [06:51:25] step 760/4768 loss 3.6386 lr 6.00e-04 gnorm 0.12 54,383 tok/s peakVRAM 13.7GB
48
+ [06:54:38] step 780/4768 loss 3.6130 lr 6.00e-04 gnorm 0.10 54,386 tok/s peakVRAM 13.7GB
49
+ [06:57:51] step 800/4768 loss 3.6005 lr 6.00e-04 gnorm 0.12 54,385 tok/s peakVRAM 13.7GB
50
+ [07:01:04] step 820/4768 loss 3.5843 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
51
+ [07:04:17] step 840/4768 loss 3.5713 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
52
+ [07:07:29] step 860/4768 loss 3.5506 lr 6.00e-04 gnorm 0.10 54,384 tok/s peakVRAM 13.7GB
53
+ [07:10:42] step 880/4768 loss 3.5451 lr 6.00e-04 gnorm 0.11 54,385 tok/s peakVRAM 13.7GB
54
+ [07:13:55] step 900/4768 loss 3.5280 lr 6.00e-04 gnorm 0.11 54,387 tok/s peakVRAM 13.7GB
55
+ [07:17:08] step 920/4768 loss 3.5207 lr 6.00e-04 gnorm 0.10 54,382 tok/s peakVRAM 13.7GB
56
+ [07:20:21] step 940/4768 loss 3.5164 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
57
+ [07:23:34] step 960/4768 loss 3.4913 lr 6.00e-04 gnorm 0.08 54,381 tok/s peakVRAM 13.7GB
58
+ [07:26:46] step 980/4768 loss 3.4890 lr 6.00e-04 gnorm 0.08 54,380 tok/s peakVRAM 13.7GB
59
+ [07:29:59] step 1000/4768 loss 3.4788 lr 6.00e-04 gnorm 0.09 54,379 tok/s peakVRAM 13.7GB
60
+ [07:30:02] ckpt @ 1000
61
+ [07:33:14] step 1020/4768 loss 3.4739 lr 6.00e-04 gnorm 0.12 53,735 tok/s peakVRAM 13.7GB
62
+ [07:36:27] step 1040/4768 loss 3.4472 lr 6.00e-04 gnorm 0.09 54,384 tok/s peakVRAM 13.7GB
63
+ [07:39:40] step 1060/4768 loss 3.4589 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
64
+ [07:42:53] step 1080/4768 loss 3.4485 lr 6.00e-04 gnorm 0.09 54,379 tok/s peakVRAM 13.7GB
65
+ [07:46:06] step 1100/4768 loss 3.4334 lr 6.00e-04 gnorm 0.11 54,382 tok/s peakVRAM 13.7GB
66
+ [07:49:18] step 1120/4768 loss 3.4154 lr 6.00e-04 gnorm 0.10 54,379 tok/s peakVRAM 13.7GB
67
+ [07:52:31] step 1140/4768 loss 3.4083 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
68
+ [07:55:44] step 1160/4768 loss 3.4036 lr 6.00e-04 gnorm 0.09 54,383 tok/s peakVRAM 13.7GB
69
+ [07:58:57] step 1180/4768 loss 3.4058 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
70
+ [08:02:10] step 1200/4768 loss 3.3883 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
71
+ [08:05:23] step 1220/4768 loss 3.3856 lr 6.00e-04 gnorm 0.10 54,376 tok/s peakVRAM 13.7GB
72
+ [08:08:35] step 1240/4768 loss 3.3775 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
73
+ [08:11:48] step 1260/4768 loss 3.3644 lr 6.00e-04 gnorm 0.10 54,380 tok/s peakVRAM 13.7GB
74
+ [08:15:01] step 1280/4768 loss 3.3676 lr 6.00e-04 gnorm 0.10 54,383 tok/s peakVRAM 13.7GB
75
+ [08:18:14] step 1300/4768 loss 3.3609 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
76
+ [08:21:27] step 1320/4768 loss 3.3531 lr 6.00e-04 gnorm 0.10 54,380 tok/s peakVRAM 13.7GB
77
+ [08:24:39] step 1340/4768 loss 3.3537 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
78
+ [08:27:52] step 1360/4768 loss 3.3450 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
79
+ [08:31:05] step 1380/4768 loss 3.3367 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
80
+ [08:34:18] step 1400/4768 loss 3.3268 lr 6.00e-04 gnorm 0.09 54,382 tok/s peakVRAM 13.7GB
81
+ [08:37:31] step 1420/4768 loss 3.3143 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
82
+ [08:40:44] step 1440/4768 loss 3.3218 lr 6.00e-04 gnorm 0.08 54,383 tok/s peakVRAM 13.7GB
83
+ [08:43:56] step 1460/4768 loss 3.3195 lr 6.00e-04 gnorm 0.09 54,381 tok/s peakVRAM 13.7GB
84
+ [08:47:09] step 1480/4768 loss 3.3233 lr 6.00e-04 gnorm 0.08 54,379 tok/s peakVRAM 13.7GB
85
+ [08:50:22] step 1500/4768 loss 3.2963 lr 6.00e-04 gnorm 0.10 54,381 tok/s peakVRAM 13.7GB
86
+ [08:50:24] ckpt @ 1500
87
+ [08:53:37] step 1520/4768 loss 3.3041 lr 6.00e-04 gnorm 0.09 53,713 tok/s peakVRAM 13.7GB
88
+ [08:56:50] step 1540/4768 loss 3.2883 lr 6.00e-04 gnorm 0.09 54,374 tok/s peakVRAM 13.7GB
89
+ [09:00:03] step 1560/4768 loss 3.2969 lr 6.00e-04 gnorm 0.08 54,377 tok/s peakVRAM 13.7GB
90
+ [09:03:16] step 1580/4768 loss 3.2851 lr 6.00e-04 gnorm 0.09 54,376 tok/s peakVRAM 13.7GB
91
+ [09:06:29] step 1600/4768 loss 3.2750 lr 6.00e-04 gnorm 0.09 54,378 tok/s peakVRAM 13.7GB
92
+ [09:09:41] step 1620/4768 loss 3.2753 lr 6.00e-04 gnorm 0.09 54,376 tok/s peakVRAM 13.7GB
93
+ [09:13:59] step 1640/4768 loss 3.2726 lr 6.00e-04 gnorm 0.09 40,752 tok/s peakVRAM 13.7GB
94
+ [09:18:26] step 1660/4768 loss 3.2733 lr 6.00e-04 gnorm 0.09 39,290 tok/s peakVRAM 13.7GB
95
+ [09:22:46] step 1680/4768 loss 3.2623 lr 6.00e-04 gnorm 0.10 40,313 tok/s peakVRAM 13.7GB
96
+ [09:25:59] step 1700/4768 loss 3.2555 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
97
+ [09:29:11] step 1720/4768 loss 3.2578 lr 6.00e-04 gnorm 0.09 54,380 tok/s peakVRAM 13.7GB
98
+ [09:32:24] step 1740/4768 loss 3.2513 lr 6.00e-04 gnorm 0.10 54,385 tok/s peakVRAM 13.7GB
99
+ [09:35:37] step 1760/4768 loss 3.2508 lr 6.00e-04 gnorm 0.10 54,389 tok/s peakVRAM 13.7GB
100
+ [09:38:50] step 1780/4768 loss 3.2381 lr 6.00e-04 gnorm 0.10 54,384 tok/s peakVRAM 13.7GB
101
+ [09:42:03] step 1800/4768 loss 3.2418 lr 6.00e-04 gnorm 0.10 54,379 tok/s peakVRAM 13.7GB
102
+ [09:45:18] step 1820/4768 loss 3.2374 lr 6.00e-04 gnorm 0.08 53,646 tok/s peakVRAM 13.7GB
103
+ [09:51:02] step 1840/4768 loss 3.2313 lr 6.00e-04 gnorm 0.09 30,467 tok/s peakVRAM 13.7GB
104
+ [09:54:58] step 1860/4768 loss 3.2303 lr 6.00e-04 gnorm 0.09 44,403 tok/s peakVRAM 13.7GB
105
+ [10:00:30] step 1880/4768 loss 3.2265 lr 6.00e-04 gnorm 0.08 31,623 tok/s peakVRAM 13.7GB
106
+ [10:06:04] step 1900/4768 loss 3.2209 lr 6.00e-04 gnorm 0.09 31,365 tok/s peakVRAM 13.7GB
107
+ [10:11:58] step 1920/4768 loss 3.2205 lr 6.00e-04 gnorm 0.08 29,665 tok/s peakVRAM 13.7GB
108
+ [10:17:51] step 1940/4768 loss 3.2141 lr 6.00e-04 gnorm 0.08 29,662 tok/s peakVRAM 13.7GB
109
+ [10:23:36] step 1960/4768 loss 3.2180 lr 6.00e-04 gnorm 0.08 30,457 tok/s peakVRAM 13.7GB
110
+ [10:29:02] step 1980/4768 loss 3.2126 lr 6.00e-04 gnorm 0.09 32,076 tok/s peakVRAM 13.7GB
111
+ [10:34:30] step 2000/4768 loss 3.2058 lr 6.00e-04 gnorm 0.09 32,047 tok/s peakVRAM 13.7GB
112
+ [10:34:32] ckpt @ 2000
113
+ [10:40:00] step 2020/4768 loss 3.2055 lr 6.00e-04 gnorm 0.09 31,786 tok/s peakVRAM 13.7GB
114
+ [10:45:28] step 2040/4768 loss 3.1983 lr 6.00e-04 gnorm 0.09 31,965 tok/s peakVRAM 13.7GB
115
+ [10:50:55] step 2060/4768 loss 3.1986 lr 6.00e-04 gnorm 0.08 31,995 tok/s peakVRAM 13.7GB
116
+ [10:56:23] step 2080/4768 loss 3.1876 lr 6.00e-04 gnorm 0.08 32,010 tok/s peakVRAM 13.7GB
117
+ [11:01:51] step 2100/4768 loss 3.1961 lr 6.00e-04 gnorm 0.10 31,946 tok/s peakVRAM 13.7GB
118
+ [11:07:19] step 2120/4768 loss 3.1859 lr 6.00e-04 gnorm 0.09 31,941 tok/s peakVRAM 13.7GB
119
+ [11:12:47] step 2140/4768 loss 3.1989 lr 6.00e-04 gnorm 0.08 32,013 tok/s peakVRAM 13.7GB
120
+ [11:18:15] step 2160/4768 loss 3.1776 lr 6.00e-04 gnorm 0.08 31,944 tok/s peakVRAM 13.7GB
121
+ [11:23:44] step 2180/4768 loss 3.1793 lr 6.00e-04 gnorm 0.09 31,909 tok/s peakVRAM 13.7GB
122
+ [11:28:11] step 2200/4768 loss 3.1803 lr 6.00e-04 gnorm 0.08 39,315 tok/s peakVRAM 13.7GB
123
+ [11:33:16] step 2220/4768 loss 3.1734 lr 6.00e-04 gnorm 0.09 34,363 tok/s peakVRAM 13.7GB
124
+ [11:39:24] step 2240/4768 loss 3.1700 lr 6.00e-04 gnorm 0.08 28,500 tok/s peakVRAM 13.7GB
125
+ [11:44:46] step 2260/4768 loss 3.1662 lr 6.00e-04 gnorm 0.08 32,518 tok/s peakVRAM 13.7GB
126
+ [11:50:32] step 2280/4768 loss 3.1625 lr 6.00e-04 gnorm 0.08 30,346 tok/s peakVRAM 13.7GB
127
+ [11:56:29] step 2300/4768 loss 3.1659 lr 6.00e-04 gnorm 0.08 29,311 tok/s peakVRAM 13.7GB
128
+ [12:01:34] step 2320/4768 loss 3.1626 lr 6.00e-04 gnorm 0.08 34,480 tok/s peakVRAM 13.7GB
129
+ [12:07:48] step 2340/4768 loss 3.1578 lr 6.00e-04 gnorm 0.09 28,010 tok/s peakVRAM 13.7GB
R1/run/args.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ --data-dir /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2 --out-dir /home/ubuntu/kolektyw/dowody400m/wyniki/R1 --run-id dowody_R1 --vocab 32768 --dtype uint16 --block 4096 --n-layer 16 --n-embd 896 --n-head 14 --value-residual --batch 128 --grad-accum 8 --norm rmsnorm --pos rope --rope-theta 500000 --ffn swiglu --ffn-mult 2.667 --qk-norm --attn-bias 0 --optimizer muon --muon-lr 0.02 --lr 6e-4 --min-lr 6e-05 --wd 0.1 --schedule wsd --decay-frac 0.2 --warmup 500 --act-ckpt --loss-chunk 1024 --sampler random --seed 1 --data-seed 20261022 --compile --log-every 20 --eval-every 4769 --eval-iters 1 --steps 4768 --ckpt-every 4768
R1/run/metrics.jsonl ADDED
@@ -0,0 +1,238 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"step": 20, "loss": 10.375725317001343, "lr": 2.3999999999999997e-05, "tok_s": 40192.52482876788}
2
+ {"step": 40, "loss": 9.55638346672058, "lr": 4.7999999999999994e-05, "tok_s": 54573.36660410555}
3
+ {"step": 60, "loss": 8.694436502456664, "lr": 7.199999999999999e-05, "tok_s": 54576.837488206365}
4
+ {"step": 80, "loss": 8.043932247161866, "lr": 9.599999999999999e-05, "tok_s": 54574.84322672207}
5
+ {"step": 100, "loss": 7.589998555183411, "lr": 0.00011999999999999999, "tok_s": 54563.929551511224}
6
+ {"step": 120, "loss": 7.120315837860107, "lr": 0.00014399999999999998, "tok_s": 54555.47091571738}
7
+ {"step": 140, "loss": 6.705103468894959, "lr": 0.000168, "tok_s": 54545.749158098115}
8
+ {"step": 160, "loss": 6.349775433540344, "lr": 0.00019199999999999998, "tok_s": 54543.69310894704}
9
+ {"step": 180, "loss": 6.0849058628082275, "lr": 0.00021599999999999996, "tok_s": 54537.1582642568}
10
+ {"step": 200, "loss": 5.863923573493958, "lr": 0.00023999999999999998, "tok_s": 54535.132210957454}
11
+ {"step": 220, "loss": 5.675646543502808, "lr": 0.00026399999999999997, "tok_s": 54530.308564857034}
12
+ {"step": 240, "loss": 5.4996884107589725, "lr": 0.00028799999999999995, "tok_s": 54529.76869794185}
13
+ {"step": 260, "loss": 5.3284769535064695, "lr": 0.000312, "tok_s": 54527.51935625258}
14
+ {"step": 280, "loss": 5.1731541633605955, "lr": 0.000336, "tok_s": 54528.11529102413}
15
+ {"step": 300, "loss": 5.028257584571838, "lr": 0.00035999999999999997, "tok_s": 54530.33195820483}
16
+ {"step": 320, "loss": 4.893511152267456, "lr": 0.00038399999999999996, "tok_s": 54529.89418139586}
17
+ {"step": 340, "loss": 4.750537443161011, "lr": 0.000408, "tok_s": 54526.910184437875}
18
+ {"step": 360, "loss": 4.624461531639099, "lr": 0.00043199999999999993, "tok_s": 54527.126715707716}
19
+ {"step": 380, "loss": 4.506671214103699, "lr": 0.00045599999999999997, "tok_s": 54527.24461544256}
20
+ {"step": 400, "loss": 4.402042651176453, "lr": 0.00047999999999999996, "tok_s": 54527.53902899062}
21
+ {"step": 420, "loss": 4.325919914245605, "lr": 0.000504, "tok_s": 54523.748537048334}
22
+ {"step": 440, "loss": 4.247882843017578, "lr": 0.0005279999999999999, "tok_s": 54526.16122800285}
23
+ {"step": 460, "loss": 4.1796149730682375, "lr": 0.000552, "tok_s": 54526.823585931896}
24
+ {"step": 480, "loss": 4.129853081703186, "lr": 0.0005759999999999999, "tok_s": 54524.81675443662}
25
+ {"step": 500, "loss": 4.077749156951905, "lr": 0.0006, "tok_s": 54522.929438760264}
26
+ {"step": 520, "loss": 4.039146566390992, "lr": 0.0006, "tok_s": 54519.68458328312}
27
+ {"step": 540, "loss": 3.9884557843208315, "lr": 0.0006, "tok_s": 54519.765887510344}
28
+ {"step": 560, "loss": 3.936927056312561, "lr": 0.0006, "tok_s": 54520.50770685465}
29
+ {"step": 580, "loss": 3.896349775791168, "lr": 0.0006, "tok_s": 54518.879799407354}
30
+ {"step": 600, "loss": 3.8545740485191344, "lr": 0.0006, "tok_s": 54522.30002461412}
31
+ {"step": 620, "loss": 3.832847225666046, "lr": 0.0006, "tok_s": 54520.07083121125}
32
+ {"step": 640, "loss": 3.8063215017318726, "lr": 0.0006, "tok_s": 54519.17905620606}
33
+ {"step": 660, "loss": 3.7745464205741883, "lr": 0.0006, "tok_s": 54520.43511904895}
34
+ {"step": 680, "loss": 3.753047776222229, "lr": 0.0006, "tok_s": 54519.251573083806}
35
+ {"step": 700, "loss": 3.7221386671066283, "lr": 0.0006, "tok_s": 54518.82154338457}
36
+ {"step": 720, "loss": 3.6911539673805236, "lr": 0.0006, "tok_s": 54516.745966587485}
37
+ {"step": 740, "loss": 3.6872825026512146, "lr": 0.0006, "tok_s": 54513.32772194921}
38
+ {"step": 760, "loss": 3.6627037644386293, "lr": 0.0006, "tok_s": 54512.705421316496}
39
+ {"step": 780, "loss": 3.6369429230690002, "lr": 0.0006, "tok_s": 54511.63950266863}
40
+ {"step": 800, "loss": 3.623213994503021, "lr": 0.0006, "tok_s": 54508.2637155628}
41
+ {"step": 820, "loss": 3.607566463947296, "lr": 0.0006, "tok_s": 54511.56254679347}
42
+ {"step": 840, "loss": 3.5929282784461973, "lr": 0.0006, "tok_s": 54509.75944972368}
43
+ {"step": 860, "loss": 3.5715310096740724, "lr": 0.0006, "tok_s": 54509.64851666642}
44
+ {"step": 880, "loss": 3.5661336302757265, "lr": 0.0006, "tok_s": 54508.117659562035}
45
+ {"step": 900, "loss": 3.547942078113556, "lr": 0.0006, "tok_s": 54510.204267443245}
46
+ {"step": 920, "loss": 3.54037184715271, "lr": 0.0006, "tok_s": 54510.82698390962}
47
+ {"step": 940, "loss": 3.534908449649811, "lr": 0.0006, "tok_s": 54510.50660440851}
48
+ {"step": 960, "loss": 3.5100515842437745, "lr": 0.0006, "tok_s": 54510.87508845343}
49
+ {"step": 980, "loss": 3.5061208605766296, "lr": 0.0006, "tok_s": 54509.29092615169}
50
+ {"step": 1000, "loss": 3.4963834166526793, "lr": 0.0006, "tok_s": 54509.491373534765}
51
+ {"step": 1020, "loss": 3.4905299305915833, "lr": 0.0006, "tok_s": 54506.1150418635}
52
+ {"step": 1040, "loss": 3.4632843494415284, "lr": 0.0006, "tok_s": 54504.96055521561}
53
+ {"step": 1060, "loss": 3.4752270936965943, "lr": 0.0006, "tok_s": 54505.576937326645}
54
+ {"step": 1080, "loss": 3.4639549970626833, "lr": 0.0006, "tok_s": 54502.19041905023}
55
+ {"step": 1100, "loss": 3.448479986190796, "lr": 0.0006, "tok_s": 54501.500968276836}
56
+ {"step": 1120, "loss": 3.430601716041565, "lr": 0.0006, "tok_s": 54497.39305969334}
57
+ {"step": 1140, "loss": 3.423946189880371, "lr": 0.0006, "tok_s": 54497.005647397564}
58
+ {"step": 1160, "loss": 3.418174147605896, "lr": 0.0006, "tok_s": 54502.22560798331}
59
+ {"step": 1180, "loss": 3.420488750934601, "lr": 0.0006, "tok_s": 54502.96100629495}
60
+ {"step": 1200, "loss": 3.4019625663757322, "lr": 0.0006, "tok_s": 54501.15395312126}
61
+ {"step": 1220, "loss": 3.399947392940521, "lr": 0.0006, "tok_s": 54503.564914990726}
62
+ {"step": 1240, "loss": 3.390988624095917, "lr": 0.0006, "tok_s": 54502.225405359764}
63
+ {"step": 1260, "loss": 3.3768735408782957, "lr": 0.0006, "tok_s": 54503.942086096744}
64
+ {"step": 1280, "loss": 3.380297088623047, "lr": 0.0006, "tok_s": 54503.82739418731}
65
+ {"step": 1300, "loss": 3.3731743574142454, "lr": 0.0006, "tok_s": 54505.2235882436}
66
+ {"step": 1320, "loss": 3.3656145334243774, "lr": 0.0006, "tok_s": 54502.43829599525}
67
+ {"step": 1340, "loss": 3.3656646490097044, "lr": 0.0006, "tok_s": 54499.28522618628}
68
+ {"step": 1360, "loss": 3.35660183429718, "lr": 0.0006, "tok_s": 54498.06511863051}
69
+ {"step": 1380, "loss": 3.347859597206116, "lr": 0.0006, "tok_s": 54498.61644623614}
70
+ {"step": 1400, "loss": 3.337520945072174, "lr": 0.0006, "tok_s": 54504.5342633703}
71
+ {"step": 1420, "loss": 3.3259835839271545, "lr": 0.0006, "tok_s": 54506.20312827794}
72
+ {"step": 1440, "loss": 3.332594668865204, "lr": 0.0006, "tok_s": 54506.02702328437}
73
+ {"step": 1460, "loss": 3.32985098361969, "lr": 0.0006, "tok_s": 54503.571872075256}
74
+ {"step": 1480, "loss": 3.33380606174469, "lr": 0.0006, "tok_s": 54505.17745258027}
75
+ {"step": 1500, "loss": 3.3074196696281435, "lr": 0.0006, "tok_s": 54506.83271124733}
76
+ {"step": 1520, "loss": 3.314147639274597, "lr": 0.0006, "tok_s": 54505.6908935518}
77
+ {"step": 1540, "loss": 3.29811692237854, "lr": 0.0006, "tok_s": 54504.80614101511}
78
+ {"step": 1560, "loss": 3.306923580169678, "lr": 0.0006, "tok_s": 54501.88405436665}
79
+ {"step": 1580, "loss": 3.2949591875076294, "lr": 0.0006, "tok_s": 54497.369221894354}
80
+ {"step": 1600, "loss": 3.284385287761688, "lr": 0.0006, "tok_s": 54499.40327568954}
81
+ {"step": 1620, "loss": 3.2848113894462587, "lr": 0.0006, "tok_s": 54498.42040085727}
82
+ {"step": 1640, "loss": 3.2816879749298096, "lr": 0.0006, "tok_s": 54500.64674351206}
83
+ {"step": 1660, "loss": 3.282590997219086, "lr": 0.0006, "tok_s": 54495.714808041674}
84
+ {"step": 1680, "loss": 3.271337866783142, "lr": 0.0006, "tok_s": 54497.035697483174}
85
+ {"step": 1700, "loss": 3.2643879294395446, "lr": 0.0006, "tok_s": 54496.460429801904}
86
+ {"step": 1720, "loss": 3.2666760683059692, "lr": 0.0006, "tok_s": 54494.41538704}
87
+ {"step": 1740, "loss": 3.2595957279205323, "lr": 0.0006, "tok_s": 54492.01698230334}
88
+ {"step": 1760, "loss": 3.2589515805244447, "lr": 0.0006, "tok_s": 54496.70481072665}
89
+ {"step": 1780, "loss": 3.246141254901886, "lr": 0.0006, "tok_s": 54500.78195345969}
90
+ {"step": 1800, "loss": 3.250395131111145, "lr": 0.0006, "tok_s": 54497.50347016742}
91
+ {"step": 1820, "loss": 3.2456166625022886, "lr": 0.0006, "tok_s": 54502.275318338565}
92
+ {"step": 1840, "loss": 3.2389777541160583, "lr": 0.0006, "tok_s": 54501.12518172441}
93
+ {"step": 1860, "loss": 3.2379404306411743, "lr": 0.0006, "tok_s": 54499.549622665865}
94
+ {"step": 1880, "loss": 3.2338213324546814, "lr": 0.0006, "tok_s": 54502.01035508657}
95
+ {"step": 1900, "loss": 3.2283488750457763, "lr": 0.0006, "tok_s": 54500.42542478084}
96
+ {"step": 1920, "loss": 3.2282689452171325, "lr": 0.0006, "tok_s": 54502.37048420369}
97
+ {"step": 1940, "loss": 3.220784342288971, "lr": 0.0006, "tok_s": 54499.72359222135}
98
+ {"step": 1960, "loss": 3.2250001192092896, "lr": 0.0006, "tok_s": 54499.94551307226}
99
+ {"step": 1980, "loss": 3.2195660352706907, "lr": 0.0006, "tok_s": 54502.737642494045}
100
+ {"step": 2000, "loss": 3.2127142429351805, "lr": 0.0006, "tok_s": 54502.04892082413}
101
+ {"step": 2020, "loss": 3.2123374581336974, "lr": 0.0006, "tok_s": 54503.26907168031}
102
+ {"step": 2040, "loss": 3.2056957364082335, "lr": 0.0006, "tok_s": 54505.24831104701}
103
+ {"step": 2060, "loss": 3.206089961528778, "lr": 0.0006, "tok_s": 54507.531619419016}
104
+ {"step": 2080, "loss": 3.1947311878204347, "lr": 0.0006, "tok_s": 54507.16730133403}
105
+ {"step": 2100, "loss": 3.203057360649109, "lr": 0.0006, "tok_s": 54506.94586208892}
106
+ {"step": 2120, "loss": 3.192316246032715, "lr": 0.0006, "tok_s": 54510.03712214494}
107
+ {"step": 2140, "loss": 3.205461084842682, "lr": 0.0006, "tok_s": 54509.02785386576}
108
+ {"step": 2160, "loss": 3.184077429771423, "lr": 0.0006, "tok_s": 54505.62564055129}
109
+ {"step": 2180, "loss": 3.1858996033668516, "lr": 0.0006, "tok_s": 54503.09062162907}
110
+ {"step": 2200, "loss": 3.187498140335083, "lr": 0.0006, "tok_s": 54505.14435383624}
111
+ {"step": 2220, "loss": 3.179544746875763, "lr": 0.0006, "tok_s": 54505.3278159521}
112
+ {"step": 2240, "loss": 3.1762046694755552, "lr": 0.0006, "tok_s": 54505.814442362724}
113
+ {"step": 2260, "loss": 3.1724456191062926, "lr": 0.0006, "tok_s": 54505.295595159725}
114
+ {"step": 2280, "loss": 3.168820869922638, "lr": 0.0006, "tok_s": 54506.82048422928}
115
+ {"step": 2300, "loss": 3.172169578075409, "lr": 0.0006, "tok_s": 54505.36895324553}
116
+ {"step": 2320, "loss": 3.169022965431213, "lr": 0.0006, "tok_s": 54506.215760325445}
117
+ {"step": 2340, "loss": 3.1635345220565796, "lr": 0.0006, "tok_s": 54503.63495872872}
118
+ {"step": 2360, "loss": 3.154193341732025, "lr": 0.0006, "tok_s": 54503.75883594428}
119
+ {"step": 2380, "loss": 3.1532516360282896, "lr": 0.0006, "tok_s": 54501.41316721653}
120
+ {"step": 2400, "loss": 3.153403079509735, "lr": 0.0006, "tok_s": 54502.95931771989}
121
+ {"step": 2420, "loss": 3.1562620282173155, "lr": 0.0006, "tok_s": 54497.84503641175}
122
+ {"step": 2440, "loss": 3.150172305107117, "lr": 0.0006, "tok_s": 54499.854137652146}
123
+ {"step": 2460, "loss": 3.148875081539154, "lr": 0.0006, "tok_s": 54499.04176758424}
124
+ {"step": 2480, "loss": 3.146116590499878, "lr": 0.0006, "tok_s": 54499.657880764156}
125
+ {"step": 2500, "loss": 3.140621912479401, "lr": 0.0006, "tok_s": 54499.554417619314}
126
+ {"step": 2520, "loss": 3.1425031423568726, "lr": 0.0006, "tok_s": 54504.1037903509}
127
+ {"step": 2540, "loss": 3.1422348022460938, "lr": 0.0006, "tok_s": 54503.31284007723}
128
+ {"step": 2560, "loss": 3.1397387862205504, "lr": 0.0006, "tok_s": 54503.85177799809}
129
+ {"step": 2580, "loss": 3.1356419563293456, "lr": 0.0006, "tok_s": 54502.07154698277}
130
+ {"step": 2600, "loss": 3.125533878803253, "lr": 0.0006, "tok_s": 54505.3636168772}
131
+ {"step": 2620, "loss": 3.1342557787895204, "lr": 0.0006, "tok_s": 54503.0855558801}
132
+ {"step": 2640, "loss": 3.1347974300384522, "lr": 0.0006, "tok_s": 54501.72979267718}
133
+ {"step": 2660, "loss": 3.133404326438904, "lr": 0.0006, "tok_s": 54498.91284683648}
134
+ {"step": 2680, "loss": 3.1281450986862183, "lr": 0.0006, "tok_s": 54501.83981548286}
135
+ {"step": 2700, "loss": 3.1312366247177126, "lr": 0.0006, "tok_s": 54503.26940939915}
136
+ {"step": 2720, "loss": 3.1138309240341187, "lr": 0.0006, "tok_s": 54499.216814437044}
137
+ {"step": 2740, "loss": 3.1155006527900695, "lr": 0.0006, "tok_s": 54501.64347675383}
138
+ {"step": 2760, "loss": 3.118541395664215, "lr": 0.0006, "tok_s": 54500.04836987996}
139
+ {"step": 2780, "loss": 3.1097805500030518, "lr": 0.0006, "tok_s": 54502.48861461497}
140
+ {"step": 2800, "loss": 3.1045394062995912, "lr": 0.0006, "tok_s": 54499.85366490502}
141
+ {"step": 2820, "loss": 3.1016828536987306, "lr": 0.0006, "tok_s": 54501.25593648621}
142
+ {"step": 2840, "loss": 3.109878695011139, "lr": 0.0006, "tok_s": 54503.062658706505}
143
+ {"step": 2860, "loss": 3.1015298843383787, "lr": 0.0006, "tok_s": 54503.35221818577}
144
+ {"step": 2880, "loss": 3.1036748051643372, "lr": 0.0006, "tok_s": 54501.39317563004}
145
+ {"step": 2900, "loss": 3.1047995686531067, "lr": 0.0006, "tok_s": 54496.90557072221}
146
+ {"step": 2920, "loss": 3.1120428085327148, "lr": 0.0006, "tok_s": 54499.5828496846}
147
+ {"step": 2940, "loss": 3.1023404002189636, "lr": 0.0006, "tok_s": 54504.77317781704}
148
+ {"step": 2960, "loss": 3.1040467023849487, "lr": 0.0006, "tok_s": 54499.708802064546}
149
+ {"step": 2980, "loss": 3.091260254383087, "lr": 0.0006, "tok_s": 54500.4839116434}
150
+ {"step": 3000, "loss": 3.0920066237449646, "lr": 0.0006, "tok_s": 54501.659078438046}
151
+ {"step": 3020, "loss": 3.0873806476593018, "lr": 0.0006, "tok_s": 54504.41321958712}
152
+ {"step": 3040, "loss": 3.084467089176178, "lr": 0.0006, "tok_s": 54500.721169683944}
153
+ {"step": 3060, "loss": 3.0876251578330995, "lr": 0.0006, "tok_s": 54502.8793469247}
154
+ {"step": 3080, "loss": 3.0809731602668764, "lr": 0.0006, "tok_s": 54507.358410658584}
155
+ {"step": 3100, "loss": 3.0741262912750242, "lr": 0.0006, "tok_s": 54507.26741570171}
156
+ {"step": 3120, "loss": 3.090480697154999, "lr": 0.0006, "tok_s": 54506.95032057946}
157
+ {"step": 3140, "loss": 3.076385426521301, "lr": 0.0006, "tok_s": 54505.714738631345}
158
+ {"step": 3160, "loss": 3.0722206592559815, "lr": 0.0006, "tok_s": 54504.217537777884}
159
+ {"step": 3180, "loss": 3.0782618522644043, "lr": 0.0006, "tok_s": 54504.55418971427}
160
+ {"step": 3200, "loss": 3.0793707728385926, "lr": 0.0006, "tok_s": 54506.32255877922}
161
+ {"step": 3220, "loss": 3.08889821767807, "lr": 0.0006, "tok_s": 54507.27883227577}
162
+ {"step": 3240, "loss": 3.07084139585495, "lr": 0.0006, "tok_s": 54504.121487365956}
163
+ {"step": 3260, "loss": 3.0676586627960205, "lr": 0.0006, "tok_s": 54503.06833234065}
164
+ {"step": 3280, "loss": 3.0627394318580627, "lr": 0.0006, "tok_s": 54503.69372265935}
165
+ {"step": 3300, "loss": 3.0739628314971923, "lr": 0.0006, "tok_s": 54506.213868895495}
166
+ {"step": 3320, "loss": 3.0676796793937684, "lr": 0.0006, "tok_s": 54504.716032695185}
167
+ {"step": 3340, "loss": 3.0680676341056823, "lr": 0.0006, "tok_s": 54504.37701463143}
168
+ {"step": 3360, "loss": 3.0593183636665344, "lr": 0.0006, "tok_s": 54507.069957014726}
169
+ {"step": 3380, "loss": 3.0686938881874086, "lr": 0.0006, "tok_s": 54504.56479458801}
170
+ {"step": 3400, "loss": 3.0537588715553285, "lr": 0.0006, "tok_s": 54503.721483594796}
171
+ {"step": 3420, "loss": 3.061671257019043, "lr": 0.0006, "tok_s": 54503.8769048287}
172
+ {"step": 3440, "loss": 3.053264784812927, "lr": 0.0006, "tok_s": 54503.23401648718}
173
+ {"step": 3460, "loss": 3.0629284143447877, "lr": 0.0006, "tok_s": 54505.903405959434}
174
+ {"step": 3480, "loss": 3.0617974996566772, "lr": 0.0006, "tok_s": 54506.13415875287}
175
+ {"step": 3500, "loss": 3.0478788137435915, "lr": 0.0006, "tok_s": 54505.478720547515}
176
+ {"step": 3520, "loss": 3.0510458111763, "lr": 0.0006, "tok_s": 54505.30241758848}
177
+ {"step": 3540, "loss": 3.05376158952713, "lr": 0.0006, "tok_s": 54505.42245208456}
178
+ {"step": 3560, "loss": 3.058077025413513, "lr": 0.0006, "tok_s": 54503.38673317847}
179
+ {"step": 3580, "loss": 3.0513238310813904, "lr": 0.0006, "tok_s": 54502.17792395057}
180
+ {"step": 3600, "loss": 3.05366827249527, "lr": 0.0006, "tok_s": 54505.76735999967}
181
+ {"step": 3620, "loss": 3.0450515270233156, "lr": 0.0006, "tok_s": 54506.02736103739}
182
+ {"step": 3640, "loss": 3.053074300289154, "lr": 0.0006, "tok_s": 54505.65759153513}
183
+ {"step": 3660, "loss": 3.0417879939079286, "lr": 0.0006, "tok_s": 54507.56107312429}
184
+ {"step": 3680, "loss": 3.0484514594078065, "lr": 0.0006, "tok_s": 54510.41316672687}
185
+ {"step": 3700, "loss": 3.0457404017448426, "lr": 0.0006, "tok_s": 54510.74732783625}
186
+ {"step": 3720, "loss": 3.039131832122803, "lr": 0.0006, "tok_s": 54509.28234619887}
187
+ {"step": 3740, "loss": 3.0350921869277956, "lr": 0.0006, "tok_s": 54507.17371891067}
188
+ {"step": 3760, "loss": 3.0340383052825928, "lr": 0.0006, "tok_s": 54508.444969310796}
189
+ {"step": 3780, "loss": 3.034581756591797, "lr": 0.0006, "tok_s": 54507.59951160865}
190
+ {"step": 3800, "loss": 3.0352933168411256, "lr": 0.0006, "tok_s": 54509.21134212601}
191
+ {"step": 3820, "loss": 3.0289077758789062, "lr": 0.0005609064981513237, "tok_s": 54503.22158845265}
192
+ {"step": 3840, "loss": 3.0271220803260803, "lr": 0.0005125842723878462, "tok_s": 54505.45001213353}
193
+ {"step": 3860, "loss": 3.0203971028327943, "lr": 0.0004827194944539712, "tok_s": 54505.29174487891}
194
+ {"step": 3880, "loss": 3.017254042625427, "lr": 0.0004590463745471515, "tok_s": 54507.10880012612}
195
+ {"step": 3900, "loss": 3.010846734046936, "lr": 0.0004388133626026285, "tok_s": 54502.2665379703}
196
+ {"step": 3920, "loss": 3.005240833759308, "lr": 0.000420851068597545, "tok_s": 54503.841376037366}
197
+ {"step": 3940, "loss": 2.997415769100189, "lr": 0.00040453249075661876, "tok_s": 54506.9768013262}
198
+ {"step": 3960, "loss": 2.9981491565704346, "lr": 0.00038947504965686095, "tok_s": 54505.679274993694}
199
+ {"step": 3980, "loss": 2.989938259124756, "lr": 0.000375424929557729, "tok_s": 54503.48575295003}
200
+ {"step": 4000, "loss": 2.989367723464966, "lr": 0.00036220351177672794, "tok_s": 54504.00699733829}
201
+ {"step": 4020, "loss": 2.9862194895744323, "lr": 0.0003496794507866152, "tok_s": 54505.02763042305}
202
+ {"step": 4040, "loss": 2.9767078161239624, "lr": 0.0003377528171635386, "tok_s": 54503.98990830757}
203
+ {"step": 4060, "loss": 2.97679283618927, "lr": 0.00032634548705926626, "tok_s": 54504.066775257335}
204
+ {"step": 4080, "loss": 2.9677639603614807, "lr": 0.00031539501058484577, "tok_s": 54505.65819948233}
205
+ {"step": 4100, "loss": 2.973319184780121, "lr": 0.00030485053354754334, "tok_s": 54502.66665547048}
206
+ {"step": 4120, "loss": 2.9620593547821046, "lr": 0.00029466998985632013, "tok_s": 54504.37498823608}
207
+ {"step": 4140, "loss": 2.9543420553207396, "lr": 0.0002848181118123862, "tok_s": 54505.295054769405}
208
+ {"step": 4160, "loss": 2.972287619113922, "lr": 0.000275264984658754, "tok_s": 54506.23305340531}
209
+ {"step": 4180, "loss": 2.964927887916565, "lr": 0.0002659849737873323, "tok_s": 54507.81339000021}
210
+ {"step": 4200, "loss": 2.9585041284561155, "lr": 0.0002569559134994179, "tok_s": 54509.20701838254}
211
+ {"step": 4220, "loss": 2.9517848968505858, "lr": 0.00024815848336191376, "tok_s": 54509.775191174595}
212
+ {"step": 4240, "loss": 2.9586602091789245, "lr": 0.00023957572171486765, "tok_s": 54507.26390291066}
213
+ {"step": 4260, "loss": 2.9503562092781066, "lr": 0.0002311926411735683, "tok_s": 54510.10360180129}
214
+ {"step": 4280, "loss": 2.949582040309906, "lr": 0.0002229959211471338, "tok_s": 54512.47082915212}
215
+ {"step": 4300, "loss": 2.947797930240631, "lr": 0.00021497365931813446, "tok_s": 54509.605548806976}
216
+ {"step": 4320, "loss": 2.9426231741905213, "lr": 0.00020711516882674415, "tok_s": 54508.71958826259}
217
+ {"step": 4340, "loss": 2.9411731123924256, "lr": 0.00019941081128766396, "tok_s": 54506.814877366494}
218
+ {"step": 4360, "loss": 2.9349792838096618, "lr": 0.00019185185819315875, "tok_s": 54504.90469310944}
219
+ {"step": 4380, "loss": 2.942066013813019, "lr": 0.00018443037501815989, "tok_s": 54508.95245919496}
220
+ {"step": 4400, "loss": 2.9398043632507322, "lr": 0.00017713912364145464, "tok_s": 54509.180738144045}
221
+ {"step": 4420, "loss": 2.932943570613861, "lr": 0.00016997147966456131, "tok_s": 54506.15124912823}
222
+ {"step": 4440, "loss": 2.9391417503356934, "lr": 0.00016292136193923112, "tok_s": 54507.113326196624}
223
+ {"step": 4460, "loss": 2.9347523927688597, "lr": 0.00015598317216998812, "tok_s": 54504.91293394499}
224
+ {"step": 4480, "loss": 2.9299306750297545, "lr": 0.00014915174288526275, "tok_s": 54503.95809436984}
225
+ {"step": 4500, "loss": 2.927113926410675, "lr": 0.00014242229240208165, "tok_s": 54502.22655355987}
226
+ {"step": 4520, "loss": 2.929121160507202, "lr": 0.00013579038566856907, "tok_s": 54503.620436627454}
227
+ {"step": 4540, "loss": 2.920461642742157, "lr": 0.00012925190007297358, "tok_s": 54503.404632360194}
228
+ {"step": 4560, "loss": 2.9235119938850405, "lr": 0.0001228029954703451, "tok_s": 54504.438144295396}
229
+ {"step": 4580, "loss": 2.926887738704681, "lr": 0.00011644008780788571, "tok_s": 54503.777951180906}
230
+ {"step": 4600, "loss": 2.9252463817596435, "lr": 0.00011015982583457378, "tok_s": 54506.12341816673}
231
+ {"step": 4620, "loss": 2.9120384812355042, "lr": 0.00010395907046536478, "tok_s": 54506.67186897893}
232
+ {"step": 4640, "loss": 2.921280491352081, "lr": 9.783487643927846e-05, "tok_s": 54508.41929781589}
233
+ {"step": 4660, "loss": 2.9140223622322083, "lr": 9.178447596720887e-05, "tok_s": 54508.05962920361}
234
+ {"step": 4680, "loss": 2.9119210481643676, "lr": 8.580526411184025e-05, "tok_s": 54507.65997295472}
235
+ {"step": 4700, "loss": 2.9078417301177977, "lr": 7.98947856805662e-05, "tok_s": 54506.66004734608}
236
+ {"step": 4720, "loss": 2.90886070728302, "lr": 7.405072344432876e-05, "tok_s": 54505.05843238631}
237
+ {"step": 4740, "loss": 2.906625306606293, "lr": 6.827088752203553e-05, "tok_s": 54507.1871620498}
238
+ {"step": 4760, "loss": 2.9156633734703066, "lr": 6.255320579263513e-05, "tok_s": 54512.37562801159}
R1/run/srodowisko.txt ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-22T11:59:49Z ARM R1 GPU 0 MINI 0 DATA_SEED 20261022
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ a607076bd6e11cad879b8213b37492a4b4f5cbf8fa96205b7ebaeace8a25213e /home/ubuntu/kolektyw/dowody400m/kod/ramie.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
7
+ 2.6.0+cu124 12.4
8
+ accelerate==1.15.0
9
+ annotated-doc==0.0.5
10
+ anyio==4.15.1
11
+ certifi==2026.10.19
12
+ click==8.5.0
13
+ filelock==4.2.4
14
+ fsspec==2026.9.0
15
+ h11==0.16.0
16
+ hf-xet==1.7.0
17
+ httpcore==1.0.9
18
+ httpx==0.28.1
19
+ huggingface_hub==1.33.0
20
+ idna==3.20
21
+ Jinja2==3.1.6
22
+ markdown-it-py==4.2.0
23
+ MarkupSafe==3.0.4
24
+ mdurl==0.1.2
25
+ mpmath==1.3.0
26
+ networkx==3.7
27
+ numpy==2.2.6
28
+ nvidia-cublas-cu12==12.4.5.8
29
+ nvidia-cuda-cupti-cu12==12.4.127
30
+ nvidia-cuda-nvrtc-cu12==12.4.127
31
+ nvidia-cuda-runtime-cu12==12.4.127
32
+ nvidia-cudnn-cu12==9.1.0.70
33
+ nvidia-cufft-cu12==11.2.1.3
34
+ nvidia-curand-cu12==10.3.5.147
35
+ nvidia-cusolver-cu12==11.6.1.9
36
+ nvidia-cusparse-cu12==12.3.1.170
37
+ nvidia-cusparselt-cu12==0.6.2
38
+ nvidia-nccl-cu12==2.21.5
39
+ nvidia-nvjitlink-cu12==12.4.127
40
+ nvidia-nvtx-cu12==12.4.127
41
+ packaging==26.3
42
+ pillow==12.3.0
43
+ psutil==7.2.2
44
+ Pygments==2.21.0
45
+ PyYAML==6.0.3
46
+ regex==2026.9.29
47
+ rich==15.0.0
48
+ safetensors==0.8.0
49
+ setuptools==84.2.0
50
+ shellingham==1.5.4
51
+ sympy==1.13.1
52
+ tokenizers==0.23.3
53
+ torch==2.6.0+cu124
54
+ torchvision==0.21.0+cu124
55
+ tqdm==4.70.1
56
+ transformers==5.17.0
57
+ triton==3.2.0
58
+ typer==0.27.3
59
+ typing_extensions==4.16.0
60
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
61
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
R1/run/srodowisko_R1_mini0.txt ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-22T11:59:49Z ARM R1 GPU 0 MINI 0 DATA_SEED 20261022
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ a607076bd6e11cad879b8213b37492a4b4f5cbf8fa96205b7ebaeace8a25213e /home/ubuntu/kolektyw/dowody400m/kod/ramie.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
7
+ 2.6.0+cu124 12.4
8
+ accelerate==1.15.0
9
+ annotated-doc==0.0.5
10
+ anyio==4.15.1
11
+ certifi==2026.10.19
12
+ click==8.5.0
13
+ filelock==4.2.4
14
+ fsspec==2026.9.0
15
+ h11==0.16.0
16
+ hf-xet==1.7.0
17
+ httpcore==1.0.9
18
+ httpx==0.28.1
19
+ huggingface_hub==1.33.0
20
+ idna==3.20
21
+ Jinja2==3.1.6
22
+ markdown-it-py==4.2.0
23
+ MarkupSafe==3.0.4
24
+ mdurl==0.1.2
25
+ mpmath==1.3.0
26
+ networkx==3.7
27
+ numpy==2.2.6
28
+ nvidia-cublas-cu12==12.4.5.8
29
+ nvidia-cuda-cupti-cu12==12.4.127
30
+ nvidia-cuda-nvrtc-cu12==12.4.127
31
+ nvidia-cuda-runtime-cu12==12.4.127
32
+ nvidia-cudnn-cu12==9.1.0.70
33
+ nvidia-cufft-cu12==11.2.1.3
34
+ nvidia-curand-cu12==10.3.5.147
35
+ nvidia-cusolver-cu12==11.6.1.9
36
+ nvidia-cusparse-cu12==12.3.1.170
37
+ nvidia-cusparselt-cu12==0.6.2
38
+ nvidia-nccl-cu12==2.21.5
39
+ nvidia-nvjitlink-cu12==12.4.127
40
+ nvidia-nvtx-cu12==12.4.127
41
+ packaging==26.3
42
+ pillow==12.3.0
43
+ psutil==7.2.2
44
+ Pygments==2.21.0
45
+ PyYAML==6.0.3
46
+ regex==2026.9.29
47
+ rich==15.0.0
48
+ safetensors==0.8.0
49
+ setuptools==84.2.0
50
+ shellingham==1.5.4
51
+ sympy==1.13.1
52
+ tokenizers==0.23.3
53
+ torch==2.6.0+cu124
54
+ torchvision==0.21.0+cu124
55
+ tqdm==4.70.1
56
+ transformers==5.17.0
57
+ triton==3.2.0
58
+ typer==0.27.3
59
+ typing_extensions==4.16.0
60
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
61
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
R1/run/srodowisko_R1_mini1.txt ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-22T11:48:41Z ARM R1 GPU 0 MINI 1 DATA_SEED 20261022
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ a607076bd6e11cad879b8213b37492a4b4f5cbf8fa96205b7ebaeace8a25213e /home/ubuntu/kolektyw/dowody400m/kod/ramie.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
7
+ 2.6.0+cu124 12.4
8
+ accelerate==1.15.0
9
+ annotated-doc==0.0.5
10
+ anyio==4.15.1
11
+ certifi==2026.10.19
12
+ click==8.5.0
13
+ filelock==4.2.4
14
+ fsspec==2026.9.0
15
+ h11==0.16.0
16
+ hf-xet==1.7.0
17
+ httpcore==1.0.9
18
+ httpx==0.28.1
19
+ huggingface_hub==1.33.0
20
+ idna==3.20
21
+ Jinja2==3.1.6
22
+ markdown-it-py==4.2.0
23
+ MarkupSafe==3.0.4
24
+ mdurl==0.1.2
25
+ mpmath==1.3.0
26
+ networkx==3.7
27
+ numpy==2.2.6
28
+ nvidia-cublas-cu12==12.4.5.8
29
+ nvidia-cuda-cupti-cu12==12.4.127
30
+ nvidia-cuda-nvrtc-cu12==12.4.127
31
+ nvidia-cuda-runtime-cu12==12.4.127
32
+ nvidia-cudnn-cu12==9.1.0.70
33
+ nvidia-cufft-cu12==11.2.1.3
34
+ nvidia-curand-cu12==10.3.5.147
35
+ nvidia-cusolver-cu12==11.6.1.9
36
+ nvidia-cusparse-cu12==12.3.1.170
37
+ nvidia-cusparselt-cu12==0.6.2
38
+ nvidia-nccl-cu12==2.21.5
39
+ nvidia-nvjitlink-cu12==12.4.127
40
+ nvidia-nvtx-cu12==12.4.127
41
+ packaging==26.3
42
+ pillow==12.3.0
43
+ psutil==7.2.2
44
+ Pygments==2.21.0
45
+ PyYAML==6.0.3
46
+ regex==2026.9.29
47
+ rich==15.0.0
48
+ safetensors==0.8.0
49
+ setuptools==84.2.0
50
+ shellingham==1.5.4
51
+ sympy==1.13.1
52
+ tokenizers==0.23.3
53
+ torch==2.6.0+cu124
54
+ torchvision==0.21.0+cu124
55
+ tqdm==4.70.1
56
+ transformers==5.17.0
57
+ triton==3.2.0
58
+ typer==0.27.3
59
+ typing_extensions==4.16.0
60
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
61
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
R1/run/train.log ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [13:59:56] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [13:59:56] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [13:59:58] model GPT-ref: 183.6M param (L16 d896 h14)
4
+ [13:59:59] optimizer=muon muon_lr=0.02 (mult=33.3x)
5
+ [13:59:59] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [13:59:59] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [13:59:59] torch.compile enabled
8
+ [13:59:59] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [14:04:20] step 20/4768 loss 10.3757 lr 2.40e-05 gnorm 2.72 40,193 tok/s peakVRAM 13.7GB
10
+ [14:07:32] step 40/4768 loss 9.5564 lr 4.80e-05 gnorm 1.29 54,573 tok/s peakVRAM 13.7GB
11
+ [14:10:45] step 60/4768 loss 8.6944 lr 7.20e-05 gnorm 0.71 54,577 tok/s peakVRAM 13.7GB
12
+ [14:13:57] step 80/4768 loss 8.0439 lr 9.60e-05 gnorm 1.18 54,575 tok/s peakVRAM 13.7GB
13
+ [14:17:09] step 100/4768 loss 7.5900 lr 1.20e-04 gnorm 0.92 54,564 tok/s peakVRAM 13.7GB
14
+ [14:20:21] step 120/4768 loss 7.1203 lr 1.44e-04 gnorm 0.82 54,555 tok/s peakVRAM 13.7GB
15
+ [14:23:33] step 140/4768 loss 6.7051 lr 1.68e-04 gnorm 0.74 54,546 tok/s peakVRAM 13.7GB
16
+ [14:26:46] step 160/4768 loss 6.3498 lr 1.92e-04 gnorm 0.75 54,544 tok/s peakVRAM 13.7GB
17
+ [14:29:58] step 180/4768 loss 6.0849 lr 2.16e-04 gnorm 0.78 54,537 tok/s peakVRAM 13.7GB
18
+ [14:33:10] step 200/4768 loss 5.8639 lr 2.40e-04 gnorm 0.87 54,535 tok/s peakVRAM 13.7GB
19
+ [14:36:22] step 220/4768 loss 5.6756 lr 2.64e-04 gnorm 0.87 54,530 tok/s peakVRAM 13.7GB
20
+ [14:39:35] step 240/4768 loss 5.4997 lr 2.88e-04 gnorm 0.77 54,530 tok/s peakVRAM 13.7GB
21
+ [14:42:47] step 260/4768 loss 5.3285 lr 3.12e-04 gnorm 0.84 54,528 tok/s peakVRAM 13.7GB
22
+ [14:45:59] step 280/4768 loss 5.1732 lr 3.36e-04 gnorm 0.91 54,528 tok/s peakVRAM 13.7GB
23
+ [14:49:12] step 300/4768 loss 5.0283 lr 3.60e-04 gnorm 0.78 54,530 tok/s peakVRAM 13.7GB
24
+ [14:52:24] step 320/4768 loss 4.8935 lr 3.84e-04 gnorm 0.63 54,530 tok/s peakVRAM 13.7GB
25
+ [14:55:36] step 340/4768 loss 4.7505 lr 4.08e-04 gnorm 0.59 54,527 tok/s peakVRAM 13.7GB
26
+ [14:58:49] step 360/4768 loss 4.6245 lr 4.32e-04 gnorm 0.72 54,527 tok/s peakVRAM 13.7GB
27
+ [15:02:01] step 380/4768 loss 4.5067 lr 4.56e-04 gnorm 0.46 54,527 tok/s peakVRAM 13.7GB
28
+ [15:05:13] step 400/4768 loss 4.4020 lr 4.80e-04 gnorm 0.52 54,528 tok/s peakVRAM 13.7GB
29
+ [15:08:25] step 420/4768 loss 4.3259 lr 5.04e-04 gnorm 0.47 54,524 tok/s peakVRAM 13.7GB
30
+ [15:11:38] step 440/4768 loss 4.2479 lr 5.28e-04 gnorm 0.40 54,526 tok/s peakVRAM 13.7GB
31
+ [15:14:50] step 460/4768 loss 4.1796 lr 5.52e-04 gnorm 0.41 54,527 tok/s peakVRAM 13.7GB
32
+ [15:18:02] step 480/4768 loss 4.1299 lr 5.76e-04 gnorm 0.42 54,525 tok/s peakVRAM 13.7GB
33
+ [15:21:15] step 500/4768 loss 4.0777 lr 6.00e-04 gnorm 0.36 54,523 tok/s peakVRAM 13.7GB
34
+ [15:24:27] step 520/4768 loss 4.0391 lr 6.00e-04 gnorm 0.31 54,520 tok/s peakVRAM 13.7GB
35
+ [15:27:39] step 540/4768 loss 3.9885 lr 6.00e-04 gnorm 0.26 54,520 tok/s peakVRAM 13.7GB
36
+ [15:30:52] step 560/4768 loss 3.9369 lr 6.00e-04 gnorm 0.20 54,521 tok/s peakVRAM 13.7GB
37
+ [15:34:04] step 580/4768 loss 3.8963 lr 6.00e-04 gnorm 0.22 54,519 tok/s peakVRAM 13.7GB
38
+ [15:37:16] step 600/4768 loss 3.8546 lr 6.00e-04 gnorm 0.23 54,522 tok/s peakVRAM 13.7GB
39
+ [15:40:29] step 620/4768 loss 3.8328 lr 6.00e-04 gnorm 0.21 54,520 tok/s peakVRAM 13.7GB
40
+ [15:43:41] step 640/4768 loss 3.8063 lr 6.00e-04 gnorm 0.19 54,519 tok/s peakVRAM 13.7GB
41
+ [15:46:53] step 660/4768 loss 3.7745 lr 6.00e-04 gnorm 0.20 54,520 tok/s peakVRAM 13.7GB
42
+ [15:50:06] step 680/4768 loss 3.7530 lr 6.00e-04 gnorm 0.17 54,519 tok/s peakVRAM 13.7GB
43
+ [15:53:18] step 700/4768 loss 3.7221 lr 6.00e-04 gnorm 0.19 54,519 tok/s peakVRAM 13.7GB
44
+ [15:56:30] step 720/4768 loss 3.6912 lr 6.00e-04 gnorm 0.16 54,517 tok/s peakVRAM 13.7GB
45
+ [15:59:43] step 740/4768 loss 3.6873 lr 6.00e-04 gnorm 0.18 54,513 tok/s peakVRAM 13.7GB
46
+ [16:02:55] step 760/4768 loss 3.6627 lr 6.00e-04 gnorm 0.17 54,513 tok/s peakVRAM 13.7GB
47
+ [16:06:07] step 780/4768 loss 3.6369 lr 6.00e-04 gnorm 0.15 54,512 tok/s peakVRAM 13.7GB
48
+ [16:09:20] step 800/4768 loss 3.6232 lr 6.00e-04 gnorm 0.16 54,508 tok/s peakVRAM 13.7GB
49
+ [16:12:32] step 820/4768 loss 3.6076 lr 6.00e-04 gnorm 0.15 54,512 tok/s peakVRAM 13.7GB
50
+ [16:15:45] step 840/4768 loss 3.5929 lr 6.00e-04 gnorm 0.15 54,510 tok/s peakVRAM 13.7GB
51
+ [16:18:57] step 860/4768 loss 3.5715 lr 6.00e-04 gnorm 0.13 54,510 tok/s peakVRAM 13.7GB
52
+ [16:22:09] step 880/4768 loss 3.5661 lr 6.00e-04 gnorm 0.15 54,508 tok/s peakVRAM 13.7GB
53
+ [16:25:22] step 900/4768 loss 3.5479 lr 6.00e-04 gnorm 0.14 54,510 tok/s peakVRAM 13.7GB
54
+ [16:28:34] step 920/4768 loss 3.5404 lr 6.00e-04 gnorm 0.13 54,511 tok/s peakVRAM 13.7GB
55
+ [16:31:46] step 940/4768 loss 3.5349 lr 6.00e-04 gnorm 0.13 54,511 tok/s peakVRAM 13.7GB
56
+ [16:34:59] step 960/4768 loss 3.5101 lr 6.00e-04 gnorm 0.14 54,511 tok/s peakVRAM 13.7GB
57
+ [16:38:11] step 980/4768 loss 3.5061 lr 6.00e-04 gnorm 0.14 54,509 tok/s peakVRAM 13.7GB
58
+ [16:41:23] step 1000/4768 loss 3.4964 lr 6.00e-04 gnorm 0.13 54,509 tok/s peakVRAM 13.7GB
59
+ [16:44:36] step 1020/4768 loss 3.4905 lr 6.00e-04 gnorm 0.12 54,506 tok/s peakVRAM 13.7GB
60
+ [16:47:48] step 1040/4768 loss 3.4633 lr 6.00e-04 gnorm 0.12 54,505 tok/s peakVRAM 13.7GB
61
+ [16:51:01] step 1060/4768 loss 3.4752 lr 6.00e-04 gnorm 0.14 54,506 tok/s peakVRAM 13.7GB
62
+ [16:54:13] step 1080/4768 loss 3.4640 lr 6.00e-04 gnorm 0.13 54,502 tok/s peakVRAM 13.7GB
63
+ [16:57:25] step 1100/4768 loss 3.4485 lr 6.00e-04 gnorm 0.14 54,502 tok/s peakVRAM 13.7GB
64
+ [17:00:38] step 1120/4768 loss 3.4306 lr 6.00e-04 gnorm 0.11 54,497 tok/s peakVRAM 13.7GB
65
+ [17:03:50] step 1140/4768 loss 3.4239 lr 6.00e-04 gnorm 0.13 54,497 tok/s peakVRAM 13.7GB
66
+ [17:07:03] step 1160/4768 loss 3.4182 lr 6.00e-04 gnorm 0.12 54,502 tok/s peakVRAM 13.7GB
67
+ [17:10:15] step 1180/4768 loss 3.4205 lr 6.00e-04 gnorm 0.13 54,503 tok/s peakVRAM 13.7GB
68
+ [17:13:27] step 1200/4768 loss 3.4020 lr 6.00e-04 gnorm 0.12 54,501 tok/s peakVRAM 13.7GB
69
+ [17:16:40] step 1220/4768 loss 3.3999 lr 6.00e-04 gnorm 0.12 54,504 tok/s peakVRAM 13.7GB
70
+ [17:19:52] step 1240/4768 loss 3.3910 lr 6.00e-04 gnorm 0.13 54,502 tok/s peakVRAM 13.7GB
71
+ [17:23:05] step 1260/4768 loss 3.3769 lr 6.00e-04 gnorm 0.12 54,504 tok/s peakVRAM 13.7GB
72
+ [17:26:17] step 1280/4768 loss 3.3803 lr 6.00e-04 gnorm 0.11 54,504 tok/s peakVRAM 13.7GB
73
+ [17:29:29] step 1300/4768 loss 3.3732 lr 6.00e-04 gnorm 0.12 54,505 tok/s peakVRAM 13.7GB
74
+ [17:32:42] step 1320/4768 loss 3.3656 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
75
+ [17:35:54] step 1340/4768 loss 3.3657 lr 6.00e-04 gnorm 0.13 54,499 tok/s peakVRAM 13.7GB
76
+ [17:39:06] step 1360/4768 loss 3.3566 lr 6.00e-04 gnorm 0.11 54,498 tok/s peakVRAM 13.7GB
77
+ [17:42:19] step 1380/4768 loss 3.3479 lr 6.00e-04 gnorm 0.11 54,499 tok/s peakVRAM 13.7GB
78
+ [17:45:31] step 1400/4768 loss 3.3375 lr 6.00e-04 gnorm 0.12 54,505 tok/s peakVRAM 13.7GB
79
+ [17:48:44] step 1420/4768 loss 3.3260 lr 6.00e-04 gnorm 0.14 54,506 tok/s peakVRAM 13.7GB
80
+ [17:51:56] step 1440/4768 loss 3.3326 lr 6.00e-04 gnorm 0.10 54,506 tok/s peakVRAM 13.7GB
81
+ [17:55:08] step 1460/4768 loss 3.3299 lr 6.00e-04 gnorm 0.12 54,504 tok/s peakVRAM 13.7GB
82
+ [17:58:21] step 1480/4768 loss 3.3338 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
83
+ [18:01:33] step 1500/4768 loss 3.3074 lr 6.00e-04 gnorm 0.11 54,507 tok/s peakVRAM 13.7GB
84
+ [18:04:46] step 1520/4768 loss 3.3141 lr 6.00e-04 gnorm 0.11 54,506 tok/s peakVRAM 13.7GB
85
+ [18:07:58] step 1540/4768 loss 3.2981 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
86
+ [18:11:10] step 1560/4768 loss 3.3069 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
87
+ [18:14:23] step 1580/4768 loss 3.2950 lr 6.00e-04 gnorm 0.11 54,497 tok/s peakVRAM 13.7GB
88
+ [18:17:35] step 1600/4768 loss 3.2844 lr 6.00e-04 gnorm 0.11 54,499 tok/s peakVRAM 13.7GB
89
+ [18:20:48] step 1620/4768 loss 3.2848 lr 6.00e-04 gnorm 0.10 54,498 tok/s peakVRAM 13.7GB
90
+ [18:24:00] step 1640/4768 loss 3.2817 lr 6.00e-04 gnorm 0.10 54,501 tok/s peakVRAM 13.7GB
91
+ [18:27:12] step 1660/4768 loss 3.2826 lr 6.00e-04 gnorm 0.11 54,496 tok/s peakVRAM 13.7GB
92
+ [18:30:25] step 1680/4768 loss 3.2713 lr 6.00e-04 gnorm 0.12 54,497 tok/s peakVRAM 13.7GB
93
+ [18:33:37] step 1700/4768 loss 3.2644 lr 6.00e-04 gnorm 0.11 54,496 tok/s peakVRAM 13.7GB
94
+ [18:36:50] step 1720/4768 loss 3.2667 lr 6.00e-04 gnorm 0.12 54,494 tok/s peakVRAM 13.7GB
95
+ [18:40:02] step 1740/4768 loss 3.2596 lr 6.00e-04 gnorm 0.12 54,492 tok/s peakVRAM 13.7GB
96
+ [18:43:14] step 1760/4768 loss 3.2590 lr 6.00e-04 gnorm 0.10 54,497 tok/s peakVRAM 13.7GB
97
+ [18:46:27] step 1780/4768 loss 3.2461 lr 6.00e-04 gnorm 0.11 54,501 tok/s peakVRAM 13.7GB
98
+ [18:49:39] step 1800/4768 loss 3.2504 lr 6.00e-04 gnorm 0.11 54,498 tok/s peakVRAM 13.7GB
99
+ [18:52:52] step 1820/4768 loss 3.2456 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
100
+ [18:56:04] step 1840/4768 loss 3.2390 lr 6.00e-04 gnorm 0.11 54,501 tok/s peakVRAM 13.7GB
101
+ [18:59:16] step 1860/4768 loss 3.2379 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
102
+ [19:02:29] step 1880/4768 loss 3.2338 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
103
+ [19:05:41] step 1900/4768 loss 3.2283 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
104
+ [19:08:54] step 1920/4768 loss 3.2283 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
105
+ [19:12:06] step 1940/4768 loss 3.2208 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
106
+ [19:15:18] step 1960/4768 loss 3.2250 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
107
+ [19:18:31] step 1980/4768 loss 3.2196 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
108
+ [19:21:43] step 2000/4768 loss 3.2127 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
109
+ [19:24:56] step 2020/4768 loss 3.2123 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
110
+ [19:28:08] step 2040/4768 loss 3.2057 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
111
+ [19:31:20] step 2060/4768 loss 3.2061 lr 6.00e-04 gnorm 0.10 54,508 tok/s peakVRAM 13.7GB
112
+ [19:34:33] step 2080/4768 loss 3.1947 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
113
+ [19:37:45] step 2100/4768 loss 3.2031 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
114
+ [19:40:57] step 2120/4768 loss 3.1923 lr 6.00e-04 gnorm 0.11 54,510 tok/s peakVRAM 13.7GB
115
+ [19:44:10] step 2140/4768 loss 3.2055 lr 6.00e-04 gnorm 0.11 54,509 tok/s peakVRAM 13.7GB
116
+ [19:47:22] step 2160/4768 loss 3.1841 lr 6.00e-04 gnorm 0.11 54,506 tok/s peakVRAM 13.7GB
117
+ [19:50:35] step 2180/4768 loss 3.1859 lr 6.00e-04 gnorm 0.11 54,503 tok/s peakVRAM 13.7GB
118
+ [19:53:47] step 2200/4768 loss 3.1875 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
119
+ [19:56:59] step 2220/4768 loss 3.1795 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
120
+ [20:00:12] step 2240/4768 loss 3.1762 lr 6.00e-04 gnorm 0.10 54,506 tok/s peakVRAM 13.7GB
121
+ [20:03:24] step 2260/4768 loss 3.1724 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
122
+ [20:06:36] step 2280/4768 loss 3.1688 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
123
+ [20:09:49] step 2300/4768 loss 3.1722 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
124
+ [20:13:01] step 2320/4768 loss 3.1690 lr 6.00e-04 gnorm 0.11 54,506 tok/s peakVRAM 13.7GB
125
+ [20:16:14] step 2340/4768 loss 3.1635 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
126
+ [20:19:26] step 2360/4768 loss 3.1542 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
127
+ [20:22:38] step 2380/4768 loss 3.1533 lr 6.00e-04 gnorm 0.09 54,501 tok/s peakVRAM 13.7GB
128
+ [20:25:51] step 2400/4768 loss 3.1534 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
129
+ [20:29:03] step 2420/4768 loss 3.1563 lr 6.00e-04 gnorm 0.09 54,498 tok/s peakVRAM 13.7GB
130
+ [20:32:16] step 2440/4768 loss 3.1502 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
131
+ [20:35:28] step 2460/4768 loss 3.1489 lr 6.00e-04 gnorm 0.10 54,499 tok/s peakVRAM 13.7GB
132
+ [20:38:40] step 2480/4768 loss 3.1461 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
133
+ [20:41:53] step 2500/4768 loss 3.1406 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
134
+ [20:45:05] step 2520/4768 loss 3.1425 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
135
+ [20:48:18] step 2540/4768 loss 3.1422 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
136
+ [20:51:30] step 2560/4768 loss 3.1397 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
137
+ [20:54:42] step 2580/4768 loss 3.1356 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
138
+ [20:57:55] step 2600/4768 loss 3.1255 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
139
+ [21:01:07] step 2620/4768 loss 3.1343 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
140
+ [21:04:20] step 2640/4768 loss 3.1348 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
141
+ [21:07:32] step 2660/4768 loss 3.1334 lr 6.00e-04 gnorm 0.11 54,499 tok/s peakVRAM 13.7GB
142
+ [21:10:44] step 2680/4768 loss 3.1281 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
143
+ [21:13:57] step 2700/4768 loss 3.1312 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
144
+ [21:17:09] step 2720/4768 loss 3.1138 lr 6.00e-04 gnorm 0.09 54,499 tok/s peakVRAM 13.7GB
145
+ [21:20:22] step 2740/4768 loss 3.1155 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
146
+ [21:23:34] step 2760/4768 loss 3.1185 lr 6.00e-04 gnorm 0.09 54,500 tok/s peakVRAM 13.7GB
147
+ [21:26:46] step 2780/4768 loss 3.1098 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
148
+ [21:29:59] step 2800/4768 loss 3.1045 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
149
+ [21:33:11] step 2820/4768 loss 3.1017 lr 6.00e-04 gnorm 0.09 54,501 tok/s peakVRAM 13.7GB
150
+ [21:36:23] step 2840/4768 loss 3.1099 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
151
+ [21:39:36] step 2860/4768 loss 3.1015 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
152
+ [21:42:48] step 2880/4768 loss 3.1037 lr 6.00e-04 gnorm 0.10 54,501 tok/s peakVRAM 13.7GB
153
+ [21:46:01] step 2900/4768 loss 3.1048 lr 6.00e-04 gnorm 0.09 54,497 tok/s peakVRAM 13.7GB
154
+ [21:49:13] step 2920/4768 loss 3.1120 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
155
+ [21:52:25] step 2940/4768 loss 3.1023 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
156
+ [21:55:38] step 2960/4768 loss 3.1040 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
157
+ [21:58:50] step 2980/4768 loss 3.0913 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
158
+ [22:02:03] step 3000/4768 loss 3.0920 lr 6.00e-04 gnorm 0.09 54,502 tok/s peakVRAM 13.7GB
159
+ [22:05:15] step 3020/4768 loss 3.0874 lr 6.00e-04 gnorm 0.11 54,504 tok/s peakVRAM 13.7GB
160
+ [22:08:27] step 3040/4768 loss 3.0845 lr 6.00e-04 gnorm 0.09 54,501 tok/s peakVRAM 13.7GB
161
+ [22:11:40] step 3060/4768 loss 3.0876 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
162
+ [22:14:52] step 3080/4768 loss 3.0810 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
163
+ [22:18:05] step 3100/4768 loss 3.0741 lr 6.00e-04 gnorm 0.11 54,507 tok/s peakVRAM 13.7GB
164
+ [22:21:17] step 3120/4768 loss 3.0905 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
165
+ [22:24:29] step 3140/4768 loss 3.0764 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
166
+ [22:27:42] step 3160/4768 loss 3.0722 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
167
+ [22:30:54] step 3180/4768 loss 3.0783 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
168
+ [22:34:06] step 3200/4768 loss 3.0794 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
169
+ [22:37:19] step 3220/4768 loss 3.0889 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
170
+ [22:40:31] step 3240/4768 loss 3.0708 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
171
+ [22:43:44] step 3260/4768 loss 3.0677 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
172
+ [22:46:56] step 3280/4768 loss 3.0627 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
173
+ [22:50:08] step 3300/4768 loss 3.0740 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
174
+ [22:53:21] step 3320/4768 loss 3.0677 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
175
+ [22:56:33] step 3340/4768 loss 3.0681 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
176
+ [22:59:46] step 3360/4768 loss 3.0593 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
177
+ [23:02:58] step 3380/4768 loss 3.0687 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
178
+ [23:06:10] step 3400/4768 loss 3.0538 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
179
+ [23:09:23] step 3420/4768 loss 3.0617 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
180
+ [23:12:35] step 3440/4768 loss 3.0533 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
181
+ [23:15:47] step 3460/4768 loss 3.0629 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
182
+ [23:19:00] step 3480/4768 loss 3.0618 lr 6.00e-04 gnorm 0.08 54,506 tok/s peakVRAM 13.7GB
183
+ [23:22:12] step 3500/4768 loss 3.0479 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
184
+ [23:25:25] step 3520/4768 loss 3.0510 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
185
+ [23:28:37] step 3540/4768 loss 3.0538 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
186
+ [23:31:49] step 3560/4768 loss 3.0581 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
187
+ [23:35:02] step 3580/4768 loss 3.0513 lr 6.00e-04 gnorm 0.08 54,502 tok/s peakVRAM 13.7GB
188
+ [23:38:14] step 3600/4768 loss 3.0537 lr 6.00e-04 gnorm 0.08 54,506 tok/s peakVRAM 13.7GB
189
+ [23:41:27] step 3620/4768 loss 3.0451 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
190
+ [23:44:39] step 3640/4768 loss 3.0531 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
191
+ [23:47:51] step 3660/4768 loss 3.0418 lr 6.00e-04 gnorm 0.09 54,508 tok/s peakVRAM 13.7GB
192
+ [23:51:04] step 3680/4768 loss 3.0485 lr 6.00e-04 gnorm 0.09 54,510 tok/s peakVRAM 13.7GB
193
+ [23:54:16] step 3700/4768 loss 3.0457 lr 6.00e-04 gnorm 0.09 54,511 tok/s peakVRAM 13.7GB
194
+ [23:57:28] step 3720/4768 loss 3.0391 lr 6.00e-04 gnorm 0.09 54,509 tok/s peakVRAM 13.7GB
195
+ [00:00:41] step 3740/4768 loss 3.0351 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
196
+ [00:03:53] step 3760/4768 loss 3.0340 lr 6.00e-04 gnorm 0.09 54,508 tok/s peakVRAM 13.7GB
197
+ [00:07:05] step 3780/4768 loss 3.0346 lr 6.00e-04 gnorm 0.09 54,508 tok/s peakVRAM 13.7GB
198
+ [00:10:18] step 3800/4768 loss 3.0353 lr 6.00e-04 gnorm 0.08 54,509 tok/s peakVRAM 13.7GB
199
+ [00:13:30] step 3820/4768 loss 3.0289 lr 5.61e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
200
+ [00:16:43] step 3840/4768 loss 3.0271 lr 5.13e-04 gnorm 0.08 54,505 tok/s peakVRAM 13.7GB
201
+ [00:19:55] step 3860/4768 loss 3.0204 lr 4.83e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
202
+ [00:23:07] step 3880/4768 loss 3.0173 lr 4.59e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
203
+ [00:26:20] step 3900/4768 loss 3.0108 lr 4.39e-04 gnorm 0.08 54,502 tok/s peakVRAM 13.7GB
204
+ [00:29:32] step 3920/4768 loss 3.0052 lr 4.21e-04 gnorm 0.08 54,504 tok/s peakVRAM 13.7GB
205
+ [00:32:45] step 3940/4768 loss 2.9974 lr 4.05e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
206
+ [00:35:57] step 3960/4768 loss 2.9981 lr 3.89e-04 gnorm 0.08 54,506 tok/s peakVRAM 13.7GB
207
+ [00:39:09] step 3980/4768 loss 2.9899 lr 3.75e-04 gnorm 0.08 54,503 tok/s peakVRAM 13.7GB
208
+ [00:42:22] step 4000/4768 loss 2.9894 lr 3.62e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
209
+ [00:45:34] step 4020/4768 loss 2.9862 lr 3.50e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
210
+ [00:48:46] step 4040/4768 loss 2.9767 lr 3.38e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
211
+ [00:51:59] step 4060/4768 loss 2.9768 lr 3.26e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
212
+ [00:55:11] step 4080/4768 loss 2.9678 lr 3.15e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
213
+ [00:58:24] step 4100/4768 loss 2.9733 lr 3.05e-04 gnorm 0.07 54,503 tok/s peakVRAM 13.7GB
214
+ [01:01:36] step 4120/4768 loss 2.9621 lr 2.95e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
215
+ [01:04:48] step 4140/4768 loss 2.9543 lr 2.85e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
216
+ [01:08:01] step 4160/4768 loss 2.9723 lr 2.75e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
217
+ [01:11:13] step 4180/4768 loss 2.9649 lr 2.66e-04 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
218
+ [01:14:25] step 4200/4768 loss 2.9585 lr 2.57e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
219
+ [01:17:38] step 4220/4768 loss 2.9518 lr 2.48e-04 gnorm 0.07 54,510 tok/s peakVRAM 13.7GB
220
+ [01:20:50] step 4240/4768 loss 2.9587 lr 2.40e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
221
+ [01:24:03] step 4260/4768 loss 2.9504 lr 2.31e-04 gnorm 0.07 54,510 tok/s peakVRAM 13.7GB
222
+ [01:27:15] step 4280/4768 loss 2.9496 lr 2.23e-04 gnorm 0.07 54,512 tok/s peakVRAM 13.7GB
223
+ [01:30:27] step 4300/4768 loss 2.9478 lr 2.15e-04 gnorm 0.07 54,510 tok/s peakVRAM 13.7GB
224
+ [01:33:40] step 4320/4768 loss 2.9426 lr 2.07e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
225
+ [01:36:52] step 4340/4768 loss 2.9412 lr 1.99e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
226
+ [01:40:04] step 4360/4768 loss 2.9350 lr 1.92e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
227
+ [01:43:17] step 4380/4768 loss 2.9421 lr 1.84e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
228
+ [01:46:29] step 4400/4768 loss 2.9398 lr 1.77e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
229
+ [01:49:42] step 4420/4768 loss 2.9329 lr 1.70e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
230
+ [01:52:54] step 4440/4768 loss 2.9391 lr 1.63e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
231
+ [01:56:06] step 4460/4768 loss 2.9348 lr 1.56e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
232
+ [01:59:19] step 4480/4768 loss 2.9299 lr 1.49e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
233
+ [02:02:31] step 4500/4768 loss 2.9271 lr 1.42e-04 gnorm 0.07 54,502 tok/s peakVRAM 13.7GB
234
+ [02:05:43] step 4520/4768 loss 2.9291 lr 1.36e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
235
+ [02:08:56] step 4540/4768 loss 2.9205 lr 1.29e-04 gnorm 0.07 54,503 tok/s peakVRAM 13.7GB
236
+ [02:12:08] step 4560/4768 loss 2.9235 lr 1.23e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
237
+ [02:15:21] step 4580/4768 loss 2.9269 lr 1.16e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
238
+ [02:18:33] step 4600/4768 loss 2.9252 lr 1.10e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
239
+ [02:21:45] step 4620/4768 loss 2.9120 lr 1.04e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
240
+ [02:24:58] step 4640/4768 loss 2.9213 lr 9.78e-05 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
241
+ [02:28:10] step 4660/4768 loss 2.9140 lr 9.18e-05 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
242
+ [02:31:22] step 4680/4768 loss 2.9119 lr 8.58e-05 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
243
+ [02:34:35] step 4700/4768 loss 2.9078 lr 7.99e-05 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
244
+ [02:37:47] step 4720/4768 loss 2.9089 lr 7.41e-05 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
245
+ [02:41:00] step 4740/4768 loss 2.9066 lr 6.83e-05 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
246
+ [02:44:12] step 4760/4768 loss 2.9157 lr 6.26e-05 gnorm 0.07 54,512 tok/s peakVRAM 13.7GB
247
+ [02:45:31] ckpt @ 4768
248
+ [02:45:31] DONE
R1/run/train.out ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [13:59:56] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [13:59:56] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [13:59:58] model GPT-ref: 183.6M param (L16 d896 h14)
4
+ [13:59:59] optimizer=muon muon_lr=0.02 (mult=33.3x)
5
+ [13:59:59] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [13:59:59] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [13:59:59] torch.compile enabled
8
+ [13:59:59] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [14:04:20] step 20/4768 loss 10.3757 lr 2.40e-05 gnorm 2.72 40,193 tok/s peakVRAM 13.7GB
10
+ [14:07:32] step 40/4768 loss 9.5564 lr 4.80e-05 gnorm 1.29 54,573 tok/s peakVRAM 13.7GB
11
+ [14:10:45] step 60/4768 loss 8.6944 lr 7.20e-05 gnorm 0.71 54,577 tok/s peakVRAM 13.7GB
12
+ [14:13:57] step 80/4768 loss 8.0439 lr 9.60e-05 gnorm 1.18 54,575 tok/s peakVRAM 13.7GB
13
+ [14:17:09] step 100/4768 loss 7.5900 lr 1.20e-04 gnorm 0.92 54,564 tok/s peakVRAM 13.7GB
14
+ [14:20:21] step 120/4768 loss 7.1203 lr 1.44e-04 gnorm 0.82 54,555 tok/s peakVRAM 13.7GB
15
+ [14:23:33] step 140/4768 loss 6.7051 lr 1.68e-04 gnorm 0.74 54,546 tok/s peakVRAM 13.7GB
16
+ [14:26:46] step 160/4768 loss 6.3498 lr 1.92e-04 gnorm 0.75 54,544 tok/s peakVRAM 13.7GB
17
+ [14:29:58] step 180/4768 loss 6.0849 lr 2.16e-04 gnorm 0.78 54,537 tok/s peakVRAM 13.7GB
18
+ [14:33:10] step 200/4768 loss 5.8639 lr 2.40e-04 gnorm 0.87 54,535 tok/s peakVRAM 13.7GB
19
+ [14:36:22] step 220/4768 loss 5.6756 lr 2.64e-04 gnorm 0.87 54,530 tok/s peakVRAM 13.7GB
20
+ [14:39:35] step 240/4768 loss 5.4997 lr 2.88e-04 gnorm 0.77 54,530 tok/s peakVRAM 13.7GB
21
+ [14:42:47] step 260/4768 loss 5.3285 lr 3.12e-04 gnorm 0.84 54,528 tok/s peakVRAM 13.7GB
22
+ [14:45:59] step 280/4768 loss 5.1732 lr 3.36e-04 gnorm 0.91 54,528 tok/s peakVRAM 13.7GB
23
+ [14:49:12] step 300/4768 loss 5.0283 lr 3.60e-04 gnorm 0.78 54,530 tok/s peakVRAM 13.7GB
24
+ [14:52:24] step 320/4768 loss 4.8935 lr 3.84e-04 gnorm 0.63 54,530 tok/s peakVRAM 13.7GB
25
+ [14:55:36] step 340/4768 loss 4.7505 lr 4.08e-04 gnorm 0.59 54,527 tok/s peakVRAM 13.7GB
26
+ [14:58:49] step 360/4768 loss 4.6245 lr 4.32e-04 gnorm 0.72 54,527 tok/s peakVRAM 13.7GB
27
+ [15:02:01] step 380/4768 loss 4.5067 lr 4.56e-04 gnorm 0.46 54,527 tok/s peakVRAM 13.7GB
28
+ [15:05:13] step 400/4768 loss 4.4020 lr 4.80e-04 gnorm 0.52 54,528 tok/s peakVRAM 13.7GB
29
+ [15:08:25] step 420/4768 loss 4.3259 lr 5.04e-04 gnorm 0.47 54,524 tok/s peakVRAM 13.7GB
30
+ [15:11:38] step 440/4768 loss 4.2479 lr 5.28e-04 gnorm 0.40 54,526 tok/s peakVRAM 13.7GB
31
+ [15:14:50] step 460/4768 loss 4.1796 lr 5.52e-04 gnorm 0.41 54,527 tok/s peakVRAM 13.7GB
32
+ [15:18:02] step 480/4768 loss 4.1299 lr 5.76e-04 gnorm 0.42 54,525 tok/s peakVRAM 13.7GB
33
+ [15:21:15] step 500/4768 loss 4.0777 lr 6.00e-04 gnorm 0.36 54,523 tok/s peakVRAM 13.7GB
34
+ [15:24:27] step 520/4768 loss 4.0391 lr 6.00e-04 gnorm 0.31 54,520 tok/s peakVRAM 13.7GB
35
+ [15:27:39] step 540/4768 loss 3.9885 lr 6.00e-04 gnorm 0.26 54,520 tok/s peakVRAM 13.7GB
36
+ [15:30:52] step 560/4768 loss 3.9369 lr 6.00e-04 gnorm 0.20 54,521 tok/s peakVRAM 13.7GB
37
+ [15:34:04] step 580/4768 loss 3.8963 lr 6.00e-04 gnorm 0.22 54,519 tok/s peakVRAM 13.7GB
38
+ [15:37:16] step 600/4768 loss 3.8546 lr 6.00e-04 gnorm 0.23 54,522 tok/s peakVRAM 13.7GB
39
+ [15:40:29] step 620/4768 loss 3.8328 lr 6.00e-04 gnorm 0.21 54,520 tok/s peakVRAM 13.7GB
40
+ [15:43:41] step 640/4768 loss 3.8063 lr 6.00e-04 gnorm 0.19 54,519 tok/s peakVRAM 13.7GB
41
+ [15:46:53] step 660/4768 loss 3.7745 lr 6.00e-04 gnorm 0.20 54,520 tok/s peakVRAM 13.7GB
42
+ [15:50:06] step 680/4768 loss 3.7530 lr 6.00e-04 gnorm 0.17 54,519 tok/s peakVRAM 13.7GB
43
+ [15:53:18] step 700/4768 loss 3.7221 lr 6.00e-04 gnorm 0.19 54,519 tok/s peakVRAM 13.7GB
44
+ [15:56:30] step 720/4768 loss 3.6912 lr 6.00e-04 gnorm 0.16 54,517 tok/s peakVRAM 13.7GB
45
+ [15:59:43] step 740/4768 loss 3.6873 lr 6.00e-04 gnorm 0.18 54,513 tok/s peakVRAM 13.7GB
46
+ [16:02:55] step 760/4768 loss 3.6627 lr 6.00e-04 gnorm 0.17 54,513 tok/s peakVRAM 13.7GB
47
+ [16:06:07] step 780/4768 loss 3.6369 lr 6.00e-04 gnorm 0.15 54,512 tok/s peakVRAM 13.7GB
48
+ [16:09:20] step 800/4768 loss 3.6232 lr 6.00e-04 gnorm 0.16 54,508 tok/s peakVRAM 13.7GB
49
+ [16:12:32] step 820/4768 loss 3.6076 lr 6.00e-04 gnorm 0.15 54,512 tok/s peakVRAM 13.7GB
50
+ [16:15:45] step 840/4768 loss 3.5929 lr 6.00e-04 gnorm 0.15 54,510 tok/s peakVRAM 13.7GB
51
+ [16:18:57] step 860/4768 loss 3.5715 lr 6.00e-04 gnorm 0.13 54,510 tok/s peakVRAM 13.7GB
52
+ [16:22:09] step 880/4768 loss 3.5661 lr 6.00e-04 gnorm 0.15 54,508 tok/s peakVRAM 13.7GB
53
+ [16:25:22] step 900/4768 loss 3.5479 lr 6.00e-04 gnorm 0.14 54,510 tok/s peakVRAM 13.7GB
54
+ [16:28:34] step 920/4768 loss 3.5404 lr 6.00e-04 gnorm 0.13 54,511 tok/s peakVRAM 13.7GB
55
+ [16:31:46] step 940/4768 loss 3.5349 lr 6.00e-04 gnorm 0.13 54,511 tok/s peakVRAM 13.7GB
56
+ [16:34:59] step 960/4768 loss 3.5101 lr 6.00e-04 gnorm 0.14 54,511 tok/s peakVRAM 13.7GB
57
+ [16:38:11] step 980/4768 loss 3.5061 lr 6.00e-04 gnorm 0.14 54,509 tok/s peakVRAM 13.7GB
58
+ [16:41:23] step 1000/4768 loss 3.4964 lr 6.00e-04 gnorm 0.13 54,509 tok/s peakVRAM 13.7GB
59
+ [16:44:36] step 1020/4768 loss 3.4905 lr 6.00e-04 gnorm 0.12 54,506 tok/s peakVRAM 13.7GB
60
+ [16:47:48] step 1040/4768 loss 3.4633 lr 6.00e-04 gnorm 0.12 54,505 tok/s peakVRAM 13.7GB
61
+ [16:51:01] step 1060/4768 loss 3.4752 lr 6.00e-04 gnorm 0.14 54,506 tok/s peakVRAM 13.7GB
62
+ [16:54:13] step 1080/4768 loss 3.4640 lr 6.00e-04 gnorm 0.13 54,502 tok/s peakVRAM 13.7GB
63
+ [16:57:25] step 1100/4768 loss 3.4485 lr 6.00e-04 gnorm 0.14 54,502 tok/s peakVRAM 13.7GB
64
+ [17:00:38] step 1120/4768 loss 3.4306 lr 6.00e-04 gnorm 0.11 54,497 tok/s peakVRAM 13.7GB
65
+ [17:03:50] step 1140/4768 loss 3.4239 lr 6.00e-04 gnorm 0.13 54,497 tok/s peakVRAM 13.7GB
66
+ [17:07:03] step 1160/4768 loss 3.4182 lr 6.00e-04 gnorm 0.12 54,502 tok/s peakVRAM 13.7GB
67
+ [17:10:15] step 1180/4768 loss 3.4205 lr 6.00e-04 gnorm 0.13 54,503 tok/s peakVRAM 13.7GB
68
+ [17:13:27] step 1200/4768 loss 3.4020 lr 6.00e-04 gnorm 0.12 54,501 tok/s peakVRAM 13.7GB
69
+ [17:16:40] step 1220/4768 loss 3.3999 lr 6.00e-04 gnorm 0.12 54,504 tok/s peakVRAM 13.7GB
70
+ [17:19:52] step 1240/4768 loss 3.3910 lr 6.00e-04 gnorm 0.13 54,502 tok/s peakVRAM 13.7GB
71
+ [17:23:05] step 1260/4768 loss 3.3769 lr 6.00e-04 gnorm 0.12 54,504 tok/s peakVRAM 13.7GB
72
+ [17:26:17] step 1280/4768 loss 3.3803 lr 6.00e-04 gnorm 0.11 54,504 tok/s peakVRAM 13.7GB
73
+ [17:29:29] step 1300/4768 loss 3.3732 lr 6.00e-04 gnorm 0.12 54,505 tok/s peakVRAM 13.7GB
74
+ [17:32:42] step 1320/4768 loss 3.3656 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
75
+ [17:35:54] step 1340/4768 loss 3.3657 lr 6.00e-04 gnorm 0.13 54,499 tok/s peakVRAM 13.7GB
76
+ [17:39:06] step 1360/4768 loss 3.3566 lr 6.00e-04 gnorm 0.11 54,498 tok/s peakVRAM 13.7GB
77
+ [17:42:19] step 1380/4768 loss 3.3479 lr 6.00e-04 gnorm 0.11 54,499 tok/s peakVRAM 13.7GB
78
+ [17:45:31] step 1400/4768 loss 3.3375 lr 6.00e-04 gnorm 0.12 54,505 tok/s peakVRAM 13.7GB
79
+ [17:48:44] step 1420/4768 loss 3.3260 lr 6.00e-04 gnorm 0.14 54,506 tok/s peakVRAM 13.7GB
80
+ [17:51:56] step 1440/4768 loss 3.3326 lr 6.00e-04 gnorm 0.10 54,506 tok/s peakVRAM 13.7GB
81
+ [17:55:08] step 1460/4768 loss 3.3299 lr 6.00e-04 gnorm 0.12 54,504 tok/s peakVRAM 13.7GB
82
+ [17:58:21] step 1480/4768 loss 3.3338 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
83
+ [18:01:33] step 1500/4768 loss 3.3074 lr 6.00e-04 gnorm 0.11 54,507 tok/s peakVRAM 13.7GB
84
+ [18:04:46] step 1520/4768 loss 3.3141 lr 6.00e-04 gnorm 0.11 54,506 tok/s peakVRAM 13.7GB
85
+ [18:07:58] step 1540/4768 loss 3.2981 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
86
+ [18:11:10] step 1560/4768 loss 3.3069 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
87
+ [18:14:23] step 1580/4768 loss 3.2950 lr 6.00e-04 gnorm 0.11 54,497 tok/s peakVRAM 13.7GB
88
+ [18:17:35] step 1600/4768 loss 3.2844 lr 6.00e-04 gnorm 0.11 54,499 tok/s peakVRAM 13.7GB
89
+ [18:20:48] step 1620/4768 loss 3.2848 lr 6.00e-04 gnorm 0.10 54,498 tok/s peakVRAM 13.7GB
90
+ [18:24:00] step 1640/4768 loss 3.2817 lr 6.00e-04 gnorm 0.10 54,501 tok/s peakVRAM 13.7GB
91
+ [18:27:12] step 1660/4768 loss 3.2826 lr 6.00e-04 gnorm 0.11 54,496 tok/s peakVRAM 13.7GB
92
+ [18:30:25] step 1680/4768 loss 3.2713 lr 6.00e-04 gnorm 0.12 54,497 tok/s peakVRAM 13.7GB
93
+ [18:33:37] step 1700/4768 loss 3.2644 lr 6.00e-04 gnorm 0.11 54,496 tok/s peakVRAM 13.7GB
94
+ [18:36:50] step 1720/4768 loss 3.2667 lr 6.00e-04 gnorm 0.12 54,494 tok/s peakVRAM 13.7GB
95
+ [18:40:02] step 1740/4768 loss 3.2596 lr 6.00e-04 gnorm 0.12 54,492 tok/s peakVRAM 13.7GB
96
+ [18:43:14] step 1760/4768 loss 3.2590 lr 6.00e-04 gnorm 0.10 54,497 tok/s peakVRAM 13.7GB
97
+ [18:46:27] step 1780/4768 loss 3.2461 lr 6.00e-04 gnorm 0.11 54,501 tok/s peakVRAM 13.7GB
98
+ [18:49:39] step 1800/4768 loss 3.2504 lr 6.00e-04 gnorm 0.11 54,498 tok/s peakVRAM 13.7GB
99
+ [18:52:52] step 1820/4768 loss 3.2456 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
100
+ [18:56:04] step 1840/4768 loss 3.2390 lr 6.00e-04 gnorm 0.11 54,501 tok/s peakVRAM 13.7GB
101
+ [18:59:16] step 1860/4768 loss 3.2379 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
102
+ [19:02:29] step 1880/4768 loss 3.2338 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
103
+ [19:05:41] step 1900/4768 loss 3.2283 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
104
+ [19:08:54] step 1920/4768 loss 3.2283 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
105
+ [19:12:06] step 1940/4768 loss 3.2208 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
106
+ [19:15:18] step 1960/4768 loss 3.2250 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
107
+ [19:18:31] step 1980/4768 loss 3.2196 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
108
+ [19:21:43] step 2000/4768 loss 3.2127 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
109
+ [19:24:56] step 2020/4768 loss 3.2123 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
110
+ [19:28:08] step 2040/4768 loss 3.2057 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
111
+ [19:31:20] step 2060/4768 loss 3.2061 lr 6.00e-04 gnorm 0.10 54,508 tok/s peakVRAM 13.7GB
112
+ [19:34:33] step 2080/4768 loss 3.1947 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
113
+ [19:37:45] step 2100/4768 loss 3.2031 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
114
+ [19:40:57] step 2120/4768 loss 3.1923 lr 6.00e-04 gnorm 0.11 54,510 tok/s peakVRAM 13.7GB
115
+ [19:44:10] step 2140/4768 loss 3.2055 lr 6.00e-04 gnorm 0.11 54,509 tok/s peakVRAM 13.7GB
116
+ [19:47:22] step 2160/4768 loss 3.1841 lr 6.00e-04 gnorm 0.11 54,506 tok/s peakVRAM 13.7GB
117
+ [19:50:35] step 2180/4768 loss 3.1859 lr 6.00e-04 gnorm 0.11 54,503 tok/s peakVRAM 13.7GB
118
+ [19:53:47] step 2200/4768 loss 3.1875 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
119
+ [19:56:59] step 2220/4768 loss 3.1795 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
120
+ [20:00:12] step 2240/4768 loss 3.1762 lr 6.00e-04 gnorm 0.10 54,506 tok/s peakVRAM 13.7GB
121
+ [20:03:24] step 2260/4768 loss 3.1724 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
122
+ [20:06:36] step 2280/4768 loss 3.1688 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
123
+ [20:09:49] step 2300/4768 loss 3.1722 lr 6.00e-04 gnorm 0.11 54,505 tok/s peakVRAM 13.7GB
124
+ [20:13:01] step 2320/4768 loss 3.1690 lr 6.00e-04 gnorm 0.11 54,506 tok/s peakVRAM 13.7GB
125
+ [20:16:14] step 2340/4768 loss 3.1635 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
126
+ [20:19:26] step 2360/4768 loss 3.1542 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
127
+ [20:22:38] step 2380/4768 loss 3.1533 lr 6.00e-04 gnorm 0.09 54,501 tok/s peakVRAM 13.7GB
128
+ [20:25:51] step 2400/4768 loss 3.1534 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
129
+ [20:29:03] step 2420/4768 loss 3.1563 lr 6.00e-04 gnorm 0.09 54,498 tok/s peakVRAM 13.7GB
130
+ [20:32:16] step 2440/4768 loss 3.1502 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
131
+ [20:35:28] step 2460/4768 loss 3.1489 lr 6.00e-04 gnorm 0.10 54,499 tok/s peakVRAM 13.7GB
132
+ [20:38:40] step 2480/4768 loss 3.1461 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
133
+ [20:41:53] step 2500/4768 loss 3.1406 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
134
+ [20:45:05] step 2520/4768 loss 3.1425 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
135
+ [20:48:18] step 2540/4768 loss 3.1422 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
136
+ [20:51:30] step 2560/4768 loss 3.1397 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
137
+ [20:54:42] step 2580/4768 loss 3.1356 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
138
+ [20:57:55] step 2600/4768 loss 3.1255 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
139
+ [21:01:07] step 2620/4768 loss 3.1343 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
140
+ [21:04:20] step 2640/4768 loss 3.1348 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
141
+ [21:07:32] step 2660/4768 loss 3.1334 lr 6.00e-04 gnorm 0.11 54,499 tok/s peakVRAM 13.7GB
142
+ [21:10:44] step 2680/4768 loss 3.1281 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
143
+ [21:13:57] step 2700/4768 loss 3.1312 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
144
+ [21:17:09] step 2720/4768 loss 3.1138 lr 6.00e-04 gnorm 0.09 54,499 tok/s peakVRAM 13.7GB
145
+ [21:20:22] step 2740/4768 loss 3.1155 lr 6.00e-04 gnorm 0.10 54,502 tok/s peakVRAM 13.7GB
146
+ [21:23:34] step 2760/4768 loss 3.1185 lr 6.00e-04 gnorm 0.09 54,500 tok/s peakVRAM 13.7GB
147
+ [21:26:46] step 2780/4768 loss 3.1098 lr 6.00e-04 gnorm 0.11 54,502 tok/s peakVRAM 13.7GB
148
+ [21:29:59] step 2800/4768 loss 3.1045 lr 6.00e-04 gnorm 0.11 54,500 tok/s peakVRAM 13.7GB
149
+ [21:33:11] step 2820/4768 loss 3.1017 lr 6.00e-04 gnorm 0.09 54,501 tok/s peakVRAM 13.7GB
150
+ [21:36:23] step 2840/4768 loss 3.1099 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
151
+ [21:39:36] step 2860/4768 loss 3.1015 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
152
+ [21:42:48] step 2880/4768 loss 3.1037 lr 6.00e-04 gnorm 0.10 54,501 tok/s peakVRAM 13.7GB
153
+ [21:46:01] step 2900/4768 loss 3.1048 lr 6.00e-04 gnorm 0.09 54,497 tok/s peakVRAM 13.7GB
154
+ [21:49:13] step 2920/4768 loss 3.1120 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
155
+ [21:52:25] step 2940/4768 loss 3.1023 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
156
+ [21:55:38] step 2960/4768 loss 3.1040 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
157
+ [21:58:50] step 2980/4768 loss 3.0913 lr 6.00e-04 gnorm 0.10 54,500 tok/s peakVRAM 13.7GB
158
+ [22:02:03] step 3000/4768 loss 3.0920 lr 6.00e-04 gnorm 0.09 54,502 tok/s peakVRAM 13.7GB
159
+ [22:05:15] step 3020/4768 loss 3.0874 lr 6.00e-04 gnorm 0.11 54,504 tok/s peakVRAM 13.7GB
160
+ [22:08:27] step 3040/4768 loss 3.0845 lr 6.00e-04 gnorm 0.09 54,501 tok/s peakVRAM 13.7GB
161
+ [22:11:40] step 3060/4768 loss 3.0876 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
162
+ [22:14:52] step 3080/4768 loss 3.0810 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
163
+ [22:18:05] step 3100/4768 loss 3.0741 lr 6.00e-04 gnorm 0.11 54,507 tok/s peakVRAM 13.7GB
164
+ [22:21:17] step 3120/4768 loss 3.0905 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
165
+ [22:24:29] step 3140/4768 loss 3.0764 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
166
+ [22:27:42] step 3160/4768 loss 3.0722 lr 6.00e-04 gnorm 0.10 54,504 tok/s peakVRAM 13.7GB
167
+ [22:30:54] step 3180/4768 loss 3.0783 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
168
+ [22:34:06] step 3200/4768 loss 3.0794 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
169
+ [22:37:19] step 3220/4768 loss 3.0889 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
170
+ [22:40:31] step 3240/4768 loss 3.0708 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
171
+ [22:43:44] step 3260/4768 loss 3.0677 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
172
+ [22:46:56] step 3280/4768 loss 3.0627 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
173
+ [22:50:08] step 3300/4768 loss 3.0740 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
174
+ [22:53:21] step 3320/4768 loss 3.0677 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
175
+ [22:56:33] step 3340/4768 loss 3.0681 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
176
+ [22:59:46] step 3360/4768 loss 3.0593 lr 6.00e-04 gnorm 0.10 54,507 tok/s peakVRAM 13.7GB
177
+ [23:02:58] step 3380/4768 loss 3.0687 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
178
+ [23:06:10] step 3400/4768 loss 3.0538 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
179
+ [23:09:23] step 3420/4768 loss 3.0617 lr 6.00e-04 gnorm 0.09 54,504 tok/s peakVRAM 13.7GB
180
+ [23:12:35] step 3440/4768 loss 3.0533 lr 6.00e-04 gnorm 0.10 54,503 tok/s peakVRAM 13.7GB
181
+ [23:15:47] step 3460/4768 loss 3.0629 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
182
+ [23:19:00] step 3480/4768 loss 3.0618 lr 6.00e-04 gnorm 0.08 54,506 tok/s peakVRAM 13.7GB
183
+ [23:22:12] step 3500/4768 loss 3.0479 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
184
+ [23:25:25] step 3520/4768 loss 3.0510 lr 6.00e-04 gnorm 0.09 54,505 tok/s peakVRAM 13.7GB
185
+ [23:28:37] step 3540/4768 loss 3.0538 lr 6.00e-04 gnorm 0.10 54,505 tok/s peakVRAM 13.7GB
186
+ [23:31:49] step 3560/4768 loss 3.0581 lr 6.00e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
187
+ [23:35:02] step 3580/4768 loss 3.0513 lr 6.00e-04 gnorm 0.08 54,502 tok/s peakVRAM 13.7GB
188
+ [23:38:14] step 3600/4768 loss 3.0537 lr 6.00e-04 gnorm 0.08 54,506 tok/s peakVRAM 13.7GB
189
+ [23:41:27] step 3620/4768 loss 3.0451 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
190
+ [23:44:39] step 3640/4768 loss 3.0531 lr 6.00e-04 gnorm 0.09 54,506 tok/s peakVRAM 13.7GB
191
+ [23:47:51] step 3660/4768 loss 3.0418 lr 6.00e-04 gnorm 0.09 54,508 tok/s peakVRAM 13.7GB
192
+ [23:51:04] step 3680/4768 loss 3.0485 lr 6.00e-04 gnorm 0.09 54,510 tok/s peakVRAM 13.7GB
193
+ [23:54:16] step 3700/4768 loss 3.0457 lr 6.00e-04 gnorm 0.09 54,511 tok/s peakVRAM 13.7GB
194
+ [23:57:28] step 3720/4768 loss 3.0391 lr 6.00e-04 gnorm 0.09 54,509 tok/s peakVRAM 13.7GB
195
+ [00:00:41] step 3740/4768 loss 3.0351 lr 6.00e-04 gnorm 0.09 54,507 tok/s peakVRAM 13.7GB
196
+ [00:03:53] step 3760/4768 loss 3.0340 lr 6.00e-04 gnorm 0.09 54,508 tok/s peakVRAM 13.7GB
197
+ [00:07:05] step 3780/4768 loss 3.0346 lr 6.00e-04 gnorm 0.09 54,508 tok/s peakVRAM 13.7GB
198
+ [00:10:18] step 3800/4768 loss 3.0353 lr 6.00e-04 gnorm 0.08 54,509 tok/s peakVRAM 13.7GB
199
+ [00:13:30] step 3820/4768 loss 3.0289 lr 5.61e-04 gnorm 0.09 54,503 tok/s peakVRAM 13.7GB
200
+ [00:16:43] step 3840/4768 loss 3.0271 lr 5.13e-04 gnorm 0.08 54,505 tok/s peakVRAM 13.7GB
201
+ [00:19:55] step 3860/4768 loss 3.0204 lr 4.83e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
202
+ [00:23:07] step 3880/4768 loss 3.0173 lr 4.59e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
203
+ [00:26:20] step 3900/4768 loss 3.0108 lr 4.39e-04 gnorm 0.08 54,502 tok/s peakVRAM 13.7GB
204
+ [00:29:32] step 3920/4768 loss 3.0052 lr 4.21e-04 gnorm 0.08 54,504 tok/s peakVRAM 13.7GB
205
+ [00:32:45] step 3940/4768 loss 2.9974 lr 4.05e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
206
+ [00:35:57] step 3960/4768 loss 2.9981 lr 3.89e-04 gnorm 0.08 54,506 tok/s peakVRAM 13.7GB
207
+ [00:39:09] step 3980/4768 loss 2.9899 lr 3.75e-04 gnorm 0.08 54,503 tok/s peakVRAM 13.7GB
208
+ [00:42:22] step 4000/4768 loss 2.9894 lr 3.62e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
209
+ [00:45:34] step 4020/4768 loss 2.9862 lr 3.50e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
210
+ [00:48:46] step 4040/4768 loss 2.9767 lr 3.38e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
211
+ [00:51:59] step 4060/4768 loss 2.9768 lr 3.26e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
212
+ [00:55:11] step 4080/4768 loss 2.9678 lr 3.15e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
213
+ [00:58:24] step 4100/4768 loss 2.9733 lr 3.05e-04 gnorm 0.07 54,503 tok/s peakVRAM 13.7GB
214
+ [01:01:36] step 4120/4768 loss 2.9621 lr 2.95e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
215
+ [01:04:48] step 4140/4768 loss 2.9543 lr 2.85e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
216
+ [01:08:01] step 4160/4768 loss 2.9723 lr 2.75e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
217
+ [01:11:13] step 4180/4768 loss 2.9649 lr 2.66e-04 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
218
+ [01:14:25] step 4200/4768 loss 2.9585 lr 2.57e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
219
+ [01:17:38] step 4220/4768 loss 2.9518 lr 2.48e-04 gnorm 0.07 54,510 tok/s peakVRAM 13.7GB
220
+ [01:20:50] step 4240/4768 loss 2.9587 lr 2.40e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
221
+ [01:24:03] step 4260/4768 loss 2.9504 lr 2.31e-04 gnorm 0.07 54,510 tok/s peakVRAM 13.7GB
222
+ [01:27:15] step 4280/4768 loss 2.9496 lr 2.23e-04 gnorm 0.07 54,512 tok/s peakVRAM 13.7GB
223
+ [01:30:27] step 4300/4768 loss 2.9478 lr 2.15e-04 gnorm 0.07 54,510 tok/s peakVRAM 13.7GB
224
+ [01:33:40] step 4320/4768 loss 2.9426 lr 2.07e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
225
+ [01:36:52] step 4340/4768 loss 2.9412 lr 1.99e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
226
+ [01:40:04] step 4360/4768 loss 2.9350 lr 1.92e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
227
+ [01:43:17] step 4380/4768 loss 2.9421 lr 1.84e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
228
+ [01:46:29] step 4400/4768 loss 2.9398 lr 1.77e-04 gnorm 0.07 54,509 tok/s peakVRAM 13.7GB
229
+ [01:49:42] step 4420/4768 loss 2.9329 lr 1.70e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
230
+ [01:52:54] step 4440/4768 loss 2.9391 lr 1.63e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
231
+ [01:56:06] step 4460/4768 loss 2.9348 lr 1.56e-04 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
232
+ [01:59:19] step 4480/4768 loss 2.9299 lr 1.49e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
233
+ [02:02:31] step 4500/4768 loss 2.9271 lr 1.42e-04 gnorm 0.07 54,502 tok/s peakVRAM 13.7GB
234
+ [02:05:43] step 4520/4768 loss 2.9291 lr 1.36e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
235
+ [02:08:56] step 4540/4768 loss 2.9205 lr 1.29e-04 gnorm 0.07 54,503 tok/s peakVRAM 13.7GB
236
+ [02:12:08] step 4560/4768 loss 2.9235 lr 1.23e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
237
+ [02:15:21] step 4580/4768 loss 2.9269 lr 1.16e-04 gnorm 0.07 54,504 tok/s peakVRAM 13.7GB
238
+ [02:18:33] step 4600/4768 loss 2.9252 lr 1.10e-04 gnorm 0.07 54,506 tok/s peakVRAM 13.7GB
239
+ [02:21:45] step 4620/4768 loss 2.9120 lr 1.04e-04 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
240
+ [02:24:58] step 4640/4768 loss 2.9213 lr 9.78e-05 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
241
+ [02:28:10] step 4660/4768 loss 2.9140 lr 9.18e-05 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
242
+ [02:31:22] step 4680/4768 loss 2.9119 lr 8.58e-05 gnorm 0.07 54,508 tok/s peakVRAM 13.7GB
243
+ [02:34:35] step 4700/4768 loss 2.9078 lr 7.99e-05 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
244
+ [02:37:47] step 4720/4768 loss 2.9089 lr 7.41e-05 gnorm 0.07 54,505 tok/s peakVRAM 13.7GB
245
+ [02:41:00] step 4740/4768 loss 2.9066 lr 6.83e-05 gnorm 0.07 54,507 tok/s peakVRAM 13.7GB
246
+ [02:44:12] step 4760/4768 loss 2.9157 lr 6.26e-05 gnorm 0.07 54,512 tok/s peakVRAM 13.7GB
247
+ [02:45:31] ckpt @ 4768
248
+ [02:45:31] DONE
R1/run/val_en.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"schema": "val_bpb_v0", "ckpt_sha256": "c2999a7b90a8fd5a27b23e44e05c793b7359196f667d8118071fe51ae4d43df1", "step": 4768, "val_sha256": "046bc03b426b75f0e84db7cdf3965a9653d94e722b81b0d03fd0cb92884cf0c5", "tokenizer_sha256": "0640d3bd3674d7a6e59c540d945a2ad275a12e1bcaeb6007c37a90751f9815dd", "script_sha256": "ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1", "trainer_sha256": "db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b", "window": 4096, "n_windows": 547, "tokens": 2240512, "bytes": 9799872, "sum_nat": 6787999.6875, "bpb": 0.9993001425643195, "ce_token": 3.029664508603391, "P": 183561103, "dtype": "fp32, tf32 off, bez autocast", "torch": "2.6.0+cu124", "device": "NVIDIA GeForce RTX 4090"}
R1/run/val_en.out ADDED
@@ -0,0 +1 @@
 
 
1
+ VAL-BPB-OK step 4768 bpb 0.99930 ce 3.02966 bajty 9799872 P 183561103 -> /home/ubuntu/kolektyw/dowody400m/wyniki/R1/val_en.json
R1/run/val_pl.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"schema": "val_bpb_v0", "ckpt_sha256": "c2999a7b90a8fd5a27b23e44e05c793b7359196f667d8118071fe51ae4d43df1", "step": 4768, "val_sha256": "99e66265f47d249c72f5e76838aa23e82959b99089f720bea6e9ee2cb1f49ee1", "tokenizer_sha256": "0640d3bd3674d7a6e59c540d945a2ad275a12e1bcaeb6007c37a90751f9815dd", "script_sha256": "ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1", "trainer_sha256": "db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b", "window": 4096, "n_windows": 406, "tokens": 1662976, "bytes": 7234678, "sum_nat": 5448028.666015625, "bpb": 1.0864124068938046, "ce_token": 3.276071732854608, "P": 183561103, "dtype": "fp32, tf32 off, bez autocast", "torch": "2.6.0+cu124", "device": "NVIDIA GeForce RTX 4090"}
R1/run/val_pl.out ADDED
@@ -0,0 +1 @@
 
 
1
+ VAL-BPB-OK step 4768 bpb 1.08641 ce 3.27607 bajty 7234678 P 183561103 -> /home/ubuntu/kolektyw/dowody400m/wyniki/R1/val_pl.json
R1/run/wynik.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"ramie": "R1", "ckpt_sha256": "c2999a7b90a8fd5a27b23e44e05c793b7359196f667d8118071fe51ae4d43df1", "bpb_pl": 1.0864124068938046, "bpb_en": 0.9993001425643195, "bpb_wazona_40_60": 1.0341450482961134, "val_pl_sha256": "99e66265f47d249c72f5e76838aa23e82959b99089f720bea6e9ee2cb1f49ee1", "val_en_sha256": "046bc03b426b75f0e84db7cdf3965a9653d94e722b81b0d03fd0cb92884cf0c5", "P": 183561103, "step": 4768}
R2/run/args.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ --data-dir /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2 --out-dir /home/ubuntu/kolektyw/dowody400m/wyniki/R2 --run-id dowody_R2 --vocab 32768 --dtype uint16 --block 4096 --n-layer 16 --n-embd 896 --n-head 14 --value-residual --batch 128 --grad-accum 8 --norm rmsnorm --pos rope --rope-theta 500000 --ffn swiglu --ffn-mult 2.667 --qk-norm --attn-bias 0 --optimizer muon --muon-lr 0.02 --lr 6e-4 --min-lr 6e-05 --wd 0.1 --schedule wsd --decay-frac 0.2 --warmup 500 --act-ckpt --loss-chunk 1024 --sampler random --seed 2 --data-seed 20261022 --compile --log-every 20 --eval-every 4769 --eval-iters 1 --steps 4768 --ckpt-every 4768
R2/run/metrics.jsonl ADDED
@@ -0,0 +1,238 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"step": 20, "loss": 10.397884702682495, "lr": 2.3999999999999997e-05, "tok_s": 40238.725610108624}
2
+ {"step": 40, "loss": 9.583148860931397, "lr": 4.7999999999999994e-05, "tok_s": 54807.99307010977}
3
+ {"step": 60, "loss": 8.702110242843627, "lr": 7.199999999999999e-05, "tok_s": 54817.099398643084}
4
+ {"step": 80, "loss": 8.048216319084167, "lr": 9.599999999999999e-05, "tok_s": 54813.46631413231}
5
+ {"step": 100, "loss": 7.60359673500061, "lr": 0.00011999999999999999, "tok_s": 54786.64525969034}
6
+ {"step": 120, "loss": 7.134544539451599, "lr": 0.00014399999999999998, "tok_s": 54771.8693336949}
7
+ {"step": 140, "loss": 6.719778347015381, "lr": 0.000168, "tok_s": 54761.4414950081}
8
+ {"step": 160, "loss": 6.357814002037048, "lr": 0.00019199999999999998, "tok_s": 54756.25132274384}
9
+ {"step": 180, "loss": 6.088592314720154, "lr": 0.00021599999999999996, "tok_s": 54753.86717044771}
10
+ {"step": 200, "loss": 5.867264747619629, "lr": 0.00023999999999999998, "tok_s": 54750.49546345217}
11
+ {"step": 220, "loss": 5.679474687576294, "lr": 0.00026399999999999997, "tok_s": 54744.47887142565}
12
+ {"step": 240, "loss": 5.506812405586243, "lr": 0.00028799999999999995, "tok_s": 54744.254409518726}
13
+ {"step": 260, "loss": 5.342029356956482, "lr": 0.000312, "tok_s": 54745.31397583071}
14
+ {"step": 280, "loss": 5.183782267570495, "lr": 0.000336, "tok_s": 54745.73968104227}
15
+ {"step": 300, "loss": 5.037600636482239, "lr": 0.00035999999999999997, "tok_s": 54747.28016638277}
16
+ {"step": 320, "loss": 4.903152680397033, "lr": 0.00038399999999999996, "tok_s": 54747.2259872589}
17
+ {"step": 340, "loss": 4.753053069114685, "lr": 0.000408, "tok_s": 54743.62539447325}
18
+ {"step": 360, "loss": 4.627336192131042, "lr": 0.00043199999999999993, "tok_s": 54744.52377765653}
19
+ {"step": 380, "loss": 4.507805275917053, "lr": 0.00045599999999999997, "tok_s": 54744.07396910179}
20
+ {"step": 400, "loss": 4.406189775466919, "lr": 0.00047999999999999996, "tok_s": 54745.248556695624}
21
+ {"step": 420, "loss": 4.3287101030349735, "lr": 0.000504, "tok_s": 54746.63111434638}
22
+ {"step": 440, "loss": 4.250546193122863, "lr": 0.0005279999999999999, "tok_s": 54744.49393101838}
23
+ {"step": 460, "loss": 4.180587100982666, "lr": 0.000552, "tok_s": 54744.28262047694}
24
+ {"step": 480, "loss": 4.133445358276367, "lr": 0.0005759999999999999, "tok_s": 54742.05111447003}
25
+ {"step": 500, "loss": 4.077378940582276, "lr": 0.0006, "tok_s": 54735.746861491934}
26
+ {"step": 520, "loss": 4.039735519886017, "lr": 0.0006, "tok_s": 54734.59454756114}
27
+ {"step": 540, "loss": 3.9894410133361817, "lr": 0.0006, "tok_s": 54735.67090646125}
28
+ {"step": 560, "loss": 3.9379420280456543, "lr": 0.0006, "tok_s": 54734.24748686994}
29
+ {"step": 580, "loss": 3.8967807412147524, "lr": 0.0006, "tok_s": 54731.84856297202}
30
+ {"step": 600, "loss": 3.853356122970581, "lr": 0.0006, "tok_s": 54728.07101962031}
31
+ {"step": 620, "loss": 3.830194354057312, "lr": 0.0006, "tok_s": 54729.95077007126}
32
+ {"step": 640, "loss": 3.804822540283203, "lr": 0.0006, "tok_s": 54735.461230798064}
33
+ {"step": 660, "loss": 3.7740728735923765, "lr": 0.0006, "tok_s": 54729.04871074365}
34
+ {"step": 680, "loss": 3.7512762904167176, "lr": 0.0006, "tok_s": 54734.60319859043}
35
+ {"step": 700, "loss": 3.7198967695236207, "lr": 0.0006, "tok_s": 54736.26179447402}
36
+ {"step": 720, "loss": 3.689478862285614, "lr": 0.0006, "tok_s": 54732.6804213084}
37
+ {"step": 740, "loss": 3.685812699794769, "lr": 0.0006, "tok_s": 54731.935541489875}
38
+ {"step": 760, "loss": 3.6607274770736695, "lr": 0.0006, "tok_s": 54731.33303191928}
39
+ {"step": 780, "loss": 3.6346758127212526, "lr": 0.0006, "tok_s": 54731.053100378515}
40
+ {"step": 800, "loss": 3.6206260561943053, "lr": 0.0006, "tok_s": 54732.49985284107}
41
+ {"step": 820, "loss": 3.6039608001708983, "lr": 0.0006, "tok_s": 54729.91698913318}
42
+ {"step": 840, "loss": 3.5898113369941713, "lr": 0.0006, "tok_s": 54727.83477447694}
43
+ {"step": 860, "loss": 3.5693995118141175, "lr": 0.0006, "tok_s": 54730.031476715136}
44
+ {"step": 880, "loss": 3.5635278701782225, "lr": 0.0006, "tok_s": 54727.39838350762}
45
+ {"step": 900, "loss": 3.5455939054489134, "lr": 0.0006, "tok_s": 54727.977719910945}
46
+ {"step": 920, "loss": 3.5379260063171385, "lr": 0.0006, "tok_s": 54731.57727623052}
47
+ {"step": 940, "loss": 3.533212733268738, "lr": 0.0006, "tok_s": 54730.919401726045}
48
+ {"step": 960, "loss": 3.50755535364151, "lr": 0.0006, "tok_s": 54726.91051644983}
49
+ {"step": 980, "loss": 3.504886746406555, "lr": 0.0006, "tok_s": 54728.021849952805}
50
+ {"step": 1000, "loss": 3.494468700885773, "lr": 0.0006, "tok_s": 54723.011229484866}
51
+ {"step": 1020, "loss": 3.4891278624534605, "lr": 0.0006, "tok_s": 54721.26009162651}
52
+ {"step": 1040, "loss": 3.462520432472229, "lr": 0.0006, "tok_s": 54717.87878380706}
53
+ {"step": 1060, "loss": 3.473784339427948, "lr": 0.0006, "tok_s": 54717.16643817577}
54
+ {"step": 1080, "loss": 3.4629327297210692, "lr": 0.0006, "tok_s": 54721.490424638614}
55
+ {"step": 1100, "loss": 3.4472741365432737, "lr": 0.0006, "tok_s": 54723.90764245979}
56
+ {"step": 1120, "loss": 3.4296505212783814, "lr": 0.0006, "tok_s": 54723.72039083601}
57
+ {"step": 1140, "loss": 3.4227797865867613, "lr": 0.0006, "tok_s": 54728.42780881539}
58
+ {"step": 1160, "loss": 3.4173384070396424, "lr": 0.0006, "tok_s": 54723.908800019264}
59
+ {"step": 1180, "loss": 3.4195353984832764, "lr": 0.0006, "tok_s": 54724.58325697108}
60
+ {"step": 1200, "loss": 3.4017979860305787, "lr": 0.0006, "tok_s": 54724.26737344753}
61
+ {"step": 1220, "loss": 3.3993475437164307, "lr": 0.0006, "tok_s": 54726.06548645609}
62
+ {"step": 1240, "loss": 3.391629421710968, "lr": 0.0006, "tok_s": 54727.966278800595}
63
+ {"step": 1260, "loss": 3.3782228231430054, "lr": 0.0006, "tok_s": 54728.5064678723}
64
+ {"step": 1280, "loss": 3.380716252326965, "lr": 0.0006, "tok_s": 54722.33674319356}
65
+ {"step": 1300, "loss": 3.373535656929016, "lr": 0.0006, "tok_s": 54718.56738859903}
66
+ {"step": 1320, "loss": 3.365945839881897, "lr": 0.0006, "tok_s": 54718.009559518374}
67
+ {"step": 1340, "loss": 3.366354763507843, "lr": 0.0006, "tok_s": 54719.10487319138}
68
+ {"step": 1360, "loss": 3.357510209083557, "lr": 0.0006, "tok_s": 54716.39161997889}
69
+ {"step": 1380, "loss": 3.349139416217804, "lr": 0.0006, "tok_s": 54721.522220687075}
70
+ {"step": 1400, "loss": 3.338739049434662, "lr": 0.0006, "tok_s": 54723.850105006284}
71
+ {"step": 1420, "loss": 3.326588249206543, "lr": 0.0006, "tok_s": 54724.84085556827}
72
+ {"step": 1440, "loss": 3.33370680809021, "lr": 0.0006, "tok_s": 54722.42920661388}
73
+ {"step": 1460, "loss": 3.331179511547089, "lr": 0.0006, "tok_s": 54722.33115999262}
74
+ {"step": 1480, "loss": 3.33524831533432, "lr": 0.0006, "tok_s": 54718.77713908803}
75
+ {"step": 1500, "loss": 3.307989013195038, "lr": 0.0006, "tok_s": 54720.09402221159}
76
+ {"step": 1520, "loss": 3.3153640627861023, "lr": 0.0006, "tok_s": 54718.6874111585}
77
+ {"step": 1540, "loss": 3.299214768409729, "lr": 0.0006, "tok_s": 54721.35404928674}
78
+ {"step": 1560, "loss": 3.3079004168510435, "lr": 0.0006, "tok_s": 54717.641265126316}
79
+ {"step": 1580, "loss": 3.296732175350189, "lr": 0.0006, "tok_s": 54719.23122955691}
80
+ {"step": 1600, "loss": 3.2852802872657776, "lr": 0.0006, "tok_s": 54716.158606990204}
81
+ {"step": 1620, "loss": 3.2858460545539856, "lr": 0.0006, "tok_s": 54718.809680852944}
82
+ {"step": 1640, "loss": 3.282657504081726, "lr": 0.0006, "tok_s": 54718.319584186276}
83
+ {"step": 1660, "loss": 3.2839358329772947, "lr": 0.0006, "tok_s": 54722.96193272997}
84
+ {"step": 1680, "loss": 3.273321044445038, "lr": 0.0006, "tok_s": 54716.57494126232}
85
+ {"step": 1700, "loss": 3.2657527327537537, "lr": 0.0006, "tok_s": 54718.044959610204}
86
+ {"step": 1720, "loss": 3.267949402332306, "lr": 0.0006, "tok_s": 54712.358045747584}
87
+ {"step": 1740, "loss": 3.261772358417511, "lr": 0.0006, "tok_s": 54710.210878688194}
88
+ {"step": 1760, "loss": 3.260621118545532, "lr": 0.0006, "tok_s": 54712.09198879873}
89
+ {"step": 1780, "loss": 3.247983956336975, "lr": 0.0006, "tok_s": 54715.218542606846}
90
+ {"step": 1800, "loss": 3.2524597883224486, "lr": 0.0006, "tok_s": 54716.171336541316}
91
+ {"step": 1820, "loss": 3.2466320514678957, "lr": 0.0006, "tok_s": 54717.424988086896}
92
+ {"step": 1840, "loss": 3.241079556941986, "lr": 0.0006, "tok_s": 54712.10648607925}
93
+ {"step": 1860, "loss": 3.2403503537178038, "lr": 0.0006, "tok_s": 54710.4659599559}
94
+ {"step": 1880, "loss": 3.2358384132385254, "lr": 0.0006, "tok_s": 54706.745058173554}
95
+ {"step": 1900, "loss": 3.2309120416641237, "lr": 0.0006, "tok_s": 54706.46531004501}
96
+ {"step": 1920, "loss": 3.2303597569465636, "lr": 0.0006, "tok_s": 54706.425161557854}
97
+ {"step": 1940, "loss": 3.223045599460602, "lr": 0.0006, "tok_s": 54710.422334629184}
98
+ {"step": 1960, "loss": 3.2269864439964295, "lr": 0.0006, "tok_s": 54710.97000426783}
99
+ {"step": 1980, "loss": 3.222047746181488, "lr": 0.0006, "tok_s": 54713.79271514927}
100
+ {"step": 2000, "loss": 3.2153287887573243, "lr": 0.0006, "tok_s": 54721.72770439228}
101
+ {"step": 2020, "loss": 3.2151976823806763, "lr": 0.0006, "tok_s": 54719.691795279694}
102
+ {"step": 2040, "loss": 3.2076790571212768, "lr": 0.0006, "tok_s": 54721.02594781771}
103
+ {"step": 2060, "loss": 3.207830536365509, "lr": 0.0006, "tok_s": 54716.8153097971}
104
+ {"step": 2080, "loss": 3.19669691324234, "lr": 0.0006, "tok_s": 54716.61993787353}
105
+ {"step": 2100, "loss": 3.205078649520874, "lr": 0.0006, "tok_s": 54717.88620417166}
106
+ {"step": 2120, "loss": 3.1950955986976624, "lr": 0.0006, "tok_s": 54716.531510412766}
107
+ {"step": 2140, "loss": 3.20761091709137, "lr": 0.0006, "tok_s": 54717.355959514505}
108
+ {"step": 2160, "loss": 3.186537528038025, "lr": 0.0006, "tok_s": 54714.34011192798}
109
+ {"step": 2180, "loss": 3.1880959272384644, "lr": 0.0006, "tok_s": 54716.86922452477}
110
+ {"step": 2200, "loss": 3.189212453365326, "lr": 0.0006, "tok_s": 54710.697291049124}
111
+ {"step": 2220, "loss": 3.182005798816681, "lr": 0.0006, "tok_s": 54709.39535600489}
112
+ {"step": 2240, "loss": 3.17874618768692, "lr": 0.0006, "tok_s": 54712.08252813624}
113
+ {"step": 2260, "loss": 3.1753636479377745, "lr": 0.0006, "tok_s": 54714.358081866165}
114
+ {"step": 2280, "loss": 3.171137976646423, "lr": 0.0006, "tok_s": 54711.677152031705}
115
+ {"step": 2300, "loss": 3.1747127294540407, "lr": 0.0006, "tok_s": 54715.94526876368}
116
+ {"step": 2320, "loss": 3.1717222332954407, "lr": 0.0006, "tok_s": 54717.84045663478}
117
+ {"step": 2340, "loss": 3.1666620850563048, "lr": 0.0006, "tok_s": 54715.35461509705}
118
+ {"step": 2360, "loss": 3.157251071929932, "lr": 0.0006, "tok_s": 54718.45662519537}
119
+ {"step": 2380, "loss": 3.155860257148743, "lr": 0.0006, "tok_s": 54714.01481724696}
120
+ {"step": 2400, "loss": 3.1565531134605407, "lr": 0.0006, "tok_s": 54718.36880449952}
121
+ {"step": 2420, "loss": 3.1596901655197143, "lr": 0.0006, "tok_s": 54720.08728207041}
122
+ {"step": 2440, "loss": 3.154794490337372, "lr": 0.0006, "tok_s": 54721.31673852506}
123
+ {"step": 2460, "loss": 3.1532232761383057, "lr": 0.0006, "tok_s": 54719.27493701316}
124
+ {"step": 2480, "loss": 3.150437569618225, "lr": 0.0006, "tok_s": 54716.898496454254}
125
+ {"step": 2500, "loss": 3.1446684837341308, "lr": 0.0006, "tok_s": 54716.010073307494}
126
+ {"step": 2520, "loss": 3.1466462135314943, "lr": 0.0006, "tok_s": 54718.5767153474}
127
+ {"step": 2540, "loss": 3.146539342403412, "lr": 0.0006, "tok_s": 54717.183729220495}
128
+ {"step": 2560, "loss": 3.1434460043907166, "lr": 0.0006, "tok_s": 54718.556019501295}
129
+ {"step": 2580, "loss": 3.1385318756103517, "lr": 0.0006, "tok_s": 54718.684415694}
130
+ {"step": 2600, "loss": 3.1294031739234924, "lr": 0.0006, "tok_s": 54716.10578281999}
131
+ {"step": 2620, "loss": 3.137481212615967, "lr": 0.0006, "tok_s": 54711.76957943284}
132
+ {"step": 2640, "loss": 3.138892638683319, "lr": 0.0006, "tok_s": 54714.37680056432}
133
+ {"step": 2660, "loss": 3.137242865562439, "lr": 0.0006, "tok_s": 54714.25250864838}
134
+ {"step": 2680, "loss": 3.1325737953186037, "lr": 0.0006, "tok_s": 54715.855073619445}
135
+ {"step": 2700, "loss": 3.135154151916504, "lr": 0.0006, "tok_s": 54717.64630276108}
136
+ {"step": 2720, "loss": 3.117984914779663, "lr": 0.0006, "tok_s": 54717.44942723859}
137
+ {"step": 2740, "loss": 3.119256818294525, "lr": 0.0006, "tok_s": 54723.33411180745}
138
+ {"step": 2760, "loss": 3.1220457315444947, "lr": 0.0006, "tok_s": 54724.06779468343}
139
+ {"step": 2780, "loss": 3.1131870627403258, "lr": 0.0006, "tok_s": 54721.67309922795}
140
+ {"step": 2800, "loss": 3.108135437965393, "lr": 0.0006, "tok_s": 54720.03581195619}
141
+ {"step": 2820, "loss": 3.105840790271759, "lr": 0.0006, "tok_s": 54719.065659265565}
142
+ {"step": 2840, "loss": 3.1134299874305724, "lr": 0.0006, "tok_s": 54719.31775949517}
143
+ {"step": 2860, "loss": 3.1054877400398255, "lr": 0.0006, "tok_s": 54717.18046162066}
144
+ {"step": 2880, "loss": 3.1071001291275024, "lr": 0.0006, "tok_s": 54713.99916181478}
145
+ {"step": 2900, "loss": 3.1088037490844727, "lr": 0.0006, "tok_s": 54712.77173575169}
146
+ {"step": 2920, "loss": 3.115924334526062, "lr": 0.0006, "tok_s": 54717.879396497636}
147
+ {"step": 2940, "loss": 3.10650497674942, "lr": 0.0006, "tok_s": 54719.886780650384}
148
+ {"step": 2960, "loss": 3.1084991097450256, "lr": 0.0006, "tok_s": 54720.022127457996}
149
+ {"step": 2980, "loss": 3.095391809940338, "lr": 0.0006, "tok_s": 54721.01008411949}
150
+ {"step": 3000, "loss": 3.095682442188263, "lr": 0.0006, "tok_s": 54721.42118155174}
151
+ {"step": 3020, "loss": 3.0919813394546507, "lr": 0.0006, "tok_s": 54720.8383074097}
152
+ {"step": 3040, "loss": 3.0889034152030943, "lr": 0.0006, "tok_s": 54713.81034439485}
153
+ {"step": 3060, "loss": 3.092266011238098, "lr": 0.0006, "tok_s": 54717.98729833024}
154
+ {"step": 3080, "loss": 3.0846786975860594, "lr": 0.0006, "tok_s": 54720.33380892033}
155
+ {"step": 3100, "loss": 3.0786309003829957, "lr": 0.0006, "tok_s": 54724.73435667363}
156
+ {"step": 3120, "loss": 3.0947417974472047, "lr": 0.0006, "tok_s": 54720.54507078586}
157
+ {"step": 3140, "loss": 3.0798319816589355, "lr": 0.0006, "tok_s": 54724.353374571554}
158
+ {"step": 3160, "loss": 3.0761234402656554, "lr": 0.0006, "tok_s": 54722.32339798344}
159
+ {"step": 3180, "loss": 3.082308828830719, "lr": 0.0006, "tok_s": 54718.39875881372}
160
+ {"step": 3200, "loss": 3.0831666231155395, "lr": 0.0006, "tok_s": 54720.005175029415}
161
+ {"step": 3220, "loss": 3.0926347255706785, "lr": 0.0006, "tok_s": 54721.79946745477}
162
+ {"step": 3240, "loss": 3.0746402978897094, "lr": 0.0006, "tok_s": 54719.029168579524}
163
+ {"step": 3260, "loss": 3.071154201030731, "lr": 0.0006, "tok_s": 54716.623069259214}
164
+ {"step": 3280, "loss": 3.06698979139328, "lr": 0.0006, "tok_s": 54718.69204051337}
165
+ {"step": 3300, "loss": 3.0780227899551393, "lr": 0.0006, "tok_s": 54712.43155387962}
166
+ {"step": 3320, "loss": 3.071763741970062, "lr": 0.0006, "tok_s": 54712.519559708606}
167
+ {"step": 3340, "loss": 3.0722206830978394, "lr": 0.0006, "tok_s": 54715.91164125627}
168
+ {"step": 3360, "loss": 3.0630337476730345, "lr": 0.0006, "tok_s": 54717.922761410126}
169
+ {"step": 3380, "loss": 3.0727545022964478, "lr": 0.0006, "tok_s": 54714.51457057679}
170
+ {"step": 3400, "loss": 3.058015322685242, "lr": 0.0006, "tok_s": 54718.504279954526}
171
+ {"step": 3420, "loss": 3.0660104870796205, "lr": 0.0006, "tok_s": 54717.33254157685}
172
+ {"step": 3440, "loss": 3.0576372504234315, "lr": 0.0006, "tok_s": 54715.9172912192}
173
+ {"step": 3460, "loss": 3.0667299389839173, "lr": 0.0006, "tok_s": 54714.89221644115}
174
+ {"step": 3480, "loss": 3.0667102575302123, "lr": 0.0006, "tok_s": 54716.037710586104}
175
+ {"step": 3500, "loss": 3.0526495456695555, "lr": 0.0006, "tok_s": 54713.767462465934}
176
+ {"step": 3520, "loss": 3.0552218317985536, "lr": 0.0006, "tok_s": 54717.21783481713}
177
+ {"step": 3540, "loss": 3.057752025127411, "lr": 0.0006, "tok_s": 54716.22273131326}
178
+ {"step": 3560, "loss": 3.0616055846214296, "lr": 0.0006, "tok_s": 54717.16228560366}
179
+ {"step": 3580, "loss": 3.0557958483695984, "lr": 0.0006, "tok_s": 54721.54659541305}
180
+ {"step": 3600, "loss": 3.057619869709015, "lr": 0.0006, "tok_s": 54721.054815687006}
181
+ {"step": 3620, "loss": 3.0496942520141603, "lr": 0.0006, "tok_s": 54719.45923074687}
182
+ {"step": 3640, "loss": 3.0573796391487122, "lr": 0.0006, "tok_s": 54721.89206523288}
183
+ {"step": 3660, "loss": 3.046215331554413, "lr": 0.0006, "tok_s": 54720.4641197848}
184
+ {"step": 3680, "loss": 3.0530635356903075, "lr": 0.0006, "tok_s": 54724.27445508174}
185
+ {"step": 3700, "loss": 3.050299286842346, "lr": 0.0006, "tok_s": 54726.45343839958}
186
+ {"step": 3720, "loss": 3.0435306072235107, "lr": 0.0006, "tok_s": 54724.378909399136}
187
+ {"step": 3740, "loss": 3.0399710059165956, "lr": 0.0006, "tok_s": 54727.211312294094}
188
+ {"step": 3760, "loss": 3.0383835315704344, "lr": 0.0006, "tok_s": 54725.30948286721}
189
+ {"step": 3780, "loss": 3.038952612876892, "lr": 0.0006, "tok_s": 54721.125896136524}
190
+ {"step": 3800, "loss": 3.040010416507721, "lr": 0.0006, "tok_s": 54717.79967876089}
191
+ {"step": 3820, "loss": 3.0331411719322205, "lr": 0.0005609064981513237, "tok_s": 54717.83038129468}
192
+ {"step": 3840, "loss": 3.031576919555664, "lr": 0.0005125842723878462, "tok_s": 54716.754111264}
193
+ {"step": 3860, "loss": 3.0244616866111755, "lr": 0.0004827194944539712, "tok_s": 54718.43586136199}
194
+ {"step": 3880, "loss": 3.0217150330543516, "lr": 0.0004590463745471515, "tok_s": 54721.24361502659}
195
+ {"step": 3900, "loss": 3.015034055709839, "lr": 0.0004388133626026285, "tok_s": 54716.04574310089}
196
+ {"step": 3920, "loss": 3.009331476688385, "lr": 0.000420851068597545, "tok_s": 54714.75396804418}
197
+ {"step": 3940, "loss": 3.001947045326233, "lr": 0.00040453249075661876, "tok_s": 54714.83483408226}
198
+ {"step": 3960, "loss": 3.0024040579795837, "lr": 0.00038947504965686095, "tok_s": 54715.71600350052}
199
+ {"step": 3980, "loss": 2.994537925720215, "lr": 0.000375424929557729, "tok_s": 54720.73148367389}
200
+ {"step": 4000, "loss": 2.9933198213577272, "lr": 0.00036220351177672794, "tok_s": 54722.08182357052}
201
+ {"step": 4020, "loss": 2.9906443119049073, "lr": 0.0003496794507866152, "tok_s": 54713.006217316004}
202
+ {"step": 4040, "loss": 2.981214869022369, "lr": 0.0003377528171635386, "tok_s": 54714.00338197388}
203
+ {"step": 4060, "loss": 2.9809293031692503, "lr": 0.00032634548705926626, "tok_s": 54719.15021436317}
204
+ {"step": 4080, "loss": 2.972481036186218, "lr": 0.00031539501058484577, "tok_s": 54714.52505310615}
205
+ {"step": 4100, "loss": 2.977624309062958, "lr": 0.00030485053354754334, "tok_s": 54715.570126989114}
206
+ {"step": 4120, "loss": 2.96641982793808, "lr": 0.00029466998985632013, "tok_s": 54720.254764858255}
207
+ {"step": 4140, "loss": 2.9586235880851746, "lr": 0.0002848181118123862, "tok_s": 54715.854529045886}
208
+ {"step": 4160, "loss": 2.9767950534820558, "lr": 0.000275264984658754, "tok_s": 54709.52575088879}
209
+ {"step": 4180, "loss": 2.9694134473800657, "lr": 0.0002659849737873323, "tok_s": 54711.81463620257}
210
+ {"step": 4200, "loss": 2.963054656982422, "lr": 0.0002569559134994179, "tok_s": 54715.08110939802}
211
+ {"step": 4220, "loss": 2.956114983558655, "lr": 0.00024815848336191376, "tok_s": 54715.96895780181}
212
+ {"step": 4240, "loss": 2.9624460101127625, "lr": 0.00023957572171486765, "tok_s": 54717.69756415137}
213
+ {"step": 4260, "loss": 2.954618144035339, "lr": 0.0002311926411735683, "tok_s": 54720.05290056804}
214
+ {"step": 4280, "loss": 2.9539708614349367, "lr": 0.0002229959211471338, "tok_s": 54724.793530241026}
215
+ {"step": 4300, "loss": 2.9517720341682434, "lr": 0.00021497365931813446, "tok_s": 54721.169061940236}
216
+ {"step": 4320, "loss": 2.9469788312911986, "lr": 0.00020711516882674415, "tok_s": 54717.55419588281}
217
+ {"step": 4340, "loss": 2.945694601535797, "lr": 0.00019941081128766396, "tok_s": 54716.70019676318}
218
+ {"step": 4360, "loss": 2.939618408679962, "lr": 0.00019185185819315875, "tok_s": 54716.639134634876}
219
+ {"step": 4380, "loss": 2.946542811393738, "lr": 0.00018443037501815989, "tok_s": 54713.43659320793}
220
+ {"step": 4400, "loss": 2.943904232978821, "lr": 0.00017713912364145464, "tok_s": 54710.85961191617}
221
+ {"step": 4420, "loss": 2.9370927572250367, "lr": 0.00016997147966456131, "tok_s": 54713.706474949475}
222
+ {"step": 4440, "loss": 2.9435221195220946, "lr": 0.00016292136193923112, "tok_s": 54713.4350276974}
223
+ {"step": 4460, "loss": 2.9390411615371703, "lr": 0.00015598317216998812, "tok_s": 54716.44492121314}
224
+ {"step": 4480, "loss": 2.9340861797332765, "lr": 0.00014915174288526275, "tok_s": 54719.165055784084}
225
+ {"step": 4500, "loss": 2.9313199520111084, "lr": 0.00014242229240208165, "tok_s": 54719.58014227132}
226
+ {"step": 4520, "loss": 2.9331663250923157, "lr": 0.00013579038566856907, "tok_s": 54717.37767558506}
227
+ {"step": 4540, "loss": 2.9246115565299986, "lr": 0.00012925190007297358, "tok_s": 54717.161536779255}
228
+ {"step": 4560, "loss": 2.9280313849449158, "lr": 0.0001228029954703451, "tok_s": 54711.65305834222}
229
+ {"step": 4580, "loss": 2.9309247732162476, "lr": 0.00011644008780788571, "tok_s": 54709.7997455041}
230
+ {"step": 4600, "loss": 2.9298508882522585, "lr": 0.00011015982583457378, "tok_s": 54707.8044659755}
231
+ {"step": 4620, "loss": 2.916302442550659, "lr": 0.00010395907046536478, "tok_s": 54711.310780508815}
232
+ {"step": 4640, "loss": 2.9256266355514526, "lr": 9.783487643927846e-05, "tok_s": 54714.35134313796}
233
+ {"step": 4660, "loss": 2.9183580040931703, "lr": 9.178447596720887e-05, "tok_s": 54712.94945149312}
234
+ {"step": 4680, "loss": 2.9159430027008058, "lr": 8.580526411184025e-05, "tok_s": 54710.84729319185}
235
+ {"step": 4700, "loss": 2.9120609164237976, "lr": 7.98947856805662e-05, "tok_s": 54712.34722373372}
236
+ {"step": 4720, "loss": 2.9131085753440855, "lr": 7.405072344432876e-05, "tok_s": 54714.52791197849}
237
+ {"step": 4740, "loss": 2.9110165357589723, "lr": 6.827088752203553e-05, "tok_s": 54717.93188370816}
238
+ {"step": 4760, "loss": 2.920218658447266, "lr": 6.255320579263513e-05, "tok_s": 54719.83497050536}
R2/run/srodowisko.txt ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-22T11:59:50Z ARM R2 GPU 1 MINI 0 DATA_SEED 20261022
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ a607076bd6e11cad879b8213b37492a4b4f5cbf8fa96205b7ebaeace8a25213e /home/ubuntu/kolektyw/dowody400m/kod/ramie.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
7
+ 2.6.0+cu124 12.4
8
+ accelerate==1.15.0
9
+ annotated-doc==0.0.5
10
+ anyio==4.15.1
11
+ certifi==2026.10.19
12
+ click==8.5.0
13
+ filelock==4.2.4
14
+ fsspec==2026.9.0
15
+ h11==0.16.0
16
+ hf-xet==1.7.0
17
+ httpcore==1.0.9
18
+ httpx==0.28.1
19
+ huggingface_hub==1.33.0
20
+ idna==3.20
21
+ Jinja2==3.1.6
22
+ markdown-it-py==4.2.0
23
+ MarkupSafe==3.0.4
24
+ mdurl==0.1.2
25
+ mpmath==1.3.0
26
+ networkx==3.7
27
+ numpy==2.2.6
28
+ nvidia-cublas-cu12==12.4.5.8
29
+ nvidia-cuda-cupti-cu12==12.4.127
30
+ nvidia-cuda-nvrtc-cu12==12.4.127
31
+ nvidia-cuda-runtime-cu12==12.4.127
32
+ nvidia-cudnn-cu12==9.1.0.70
33
+ nvidia-cufft-cu12==11.2.1.3
34
+ nvidia-curand-cu12==10.3.5.147
35
+ nvidia-cusolver-cu12==11.6.1.9
36
+ nvidia-cusparse-cu12==12.3.1.170
37
+ nvidia-cusparselt-cu12==0.6.2
38
+ nvidia-nccl-cu12==2.21.5
39
+ nvidia-nvjitlink-cu12==12.4.127
40
+ nvidia-nvtx-cu12==12.4.127
41
+ packaging==26.3
42
+ pillow==12.3.0
43
+ psutil==7.2.2
44
+ Pygments==2.21.0
45
+ PyYAML==6.0.3
46
+ regex==2026.9.29
47
+ rich==15.0.0
48
+ safetensors==0.8.0
49
+ setuptools==84.2.0
50
+ shellingham==1.5.4
51
+ sympy==1.13.1
52
+ tokenizers==0.23.3
53
+ torch==2.6.0+cu124
54
+ torchvision==0.21.0+cu124
55
+ tqdm==4.70.1
56
+ transformers==5.17.0
57
+ triton==3.2.0
58
+ typer==0.27.3
59
+ typing_extensions==4.16.0
60
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
61
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
R2/run/srodowisko_R2_mini0.txt ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ data 2026-10-22T11:59:50Z ARM R2 GPU 1 MINI 0 DATA_SEED 20261022
2
+ CPATH=/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include/x86_64-linux-gnu/python3.12:/home/ubuntu/kolektyw/polocr/pydev/root/usr/include
3
+ a607076bd6e11cad879b8213b37492a4b4f5cbf8fa96205b7ebaeace8a25213e /home/ubuntu/kolektyw/dowody400m/kod/ramie.sh
4
+ db36c150e95c184e82b5f84c3ee783fd5fd2c62e960a438177f543dc3eba1d0b /home/ubuntu/kolektyw/dowody400m/kod/train_gpt_ref_r23.py
5
+ ce84287cbdea65e02ce8f5b4af2617aae0b8239db4984c456cee8f75a6866ab1 /home/ubuntu/kolektyw/dowody400m/kod/val_bpb.py
6
+ 61859a5b32d286a904dc5a8a1998a2a2f986f8b80366868815b32794a02dfa11 /home/ubuntu/kolektyw/dane_dowody/out_v1_fw2/SHA256SUMS
7
+ 2.6.0+cu124 12.4
8
+ accelerate==1.15.0
9
+ annotated-doc==0.0.5
10
+ anyio==4.15.1
11
+ certifi==2026.10.19
12
+ click==8.5.0
13
+ filelock==4.2.4
14
+ fsspec==2026.9.0
15
+ h11==0.16.0
16
+ hf-xet==1.7.0
17
+ httpcore==1.0.9
18
+ httpx==0.28.1
19
+ huggingface_hub==1.33.0
20
+ idna==3.20
21
+ Jinja2==3.1.6
22
+ markdown-it-py==4.2.0
23
+ MarkupSafe==3.0.4
24
+ mdurl==0.1.2
25
+ mpmath==1.3.0
26
+ networkx==3.7
27
+ numpy==2.2.6
28
+ nvidia-cublas-cu12==12.4.5.8
29
+ nvidia-cuda-cupti-cu12==12.4.127
30
+ nvidia-cuda-nvrtc-cu12==12.4.127
31
+ nvidia-cuda-runtime-cu12==12.4.127
32
+ nvidia-cudnn-cu12==9.1.0.70
33
+ nvidia-cufft-cu12==11.2.1.3
34
+ nvidia-curand-cu12==10.3.5.147
35
+ nvidia-cusolver-cu12==11.6.1.9
36
+ nvidia-cusparse-cu12==12.3.1.170
37
+ nvidia-cusparselt-cu12==0.6.2
38
+ nvidia-nccl-cu12==2.21.5
39
+ nvidia-nvjitlink-cu12==12.4.127
40
+ nvidia-nvtx-cu12==12.4.127
41
+ packaging==26.3
42
+ pillow==12.3.0
43
+ psutil==7.2.2
44
+ Pygments==2.21.0
45
+ PyYAML==6.0.3
46
+ regex==2026.9.29
47
+ rich==15.0.0
48
+ safetensors==0.8.0
49
+ setuptools==84.2.0
50
+ shellingham==1.5.4
51
+ sympy==1.13.1
52
+ tokenizers==0.23.3
53
+ torch==2.6.0+cu124
54
+ torchvision==0.21.0+cu124
55
+ tqdm==4.70.1
56
+ transformers==5.17.0
57
+ triton==3.2.0
58
+ typer==0.27.3
59
+ typing_extensions==4.16.0
60
+ 0, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
61
+ 1, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
62
+ 2, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
63
+ 3, NVIDIA GeForce RTX 4090, 1 MiB, 595.104.02
R2/run/train.log ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [13:59:56] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [13:59:56] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [13:59:58] model GPT-ref: 183.6M param (L16 d896 h14)
4
+ [13:59:59] optimizer=muon muon_lr=0.02 (mult=33.3x)
5
+ [13:59:59] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [13:59:59] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [13:59:59] torch.compile enabled
8
+ [13:59:59] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [14:04:20] step 20/4768 loss 10.3979 lr 2.40e-05 gnorm 2.60 40,239 tok/s peakVRAM 13.7GB
10
+ [14:07:31] step 40/4768 loss 9.5831 lr 4.80e-05 gnorm 1.42 54,808 tok/s peakVRAM 13.7GB
11
+ [14:10:43] step 60/4768 loss 8.7021 lr 7.20e-05 gnorm 0.76 54,817 tok/s peakVRAM 13.7GB
12
+ [14:13:54] step 80/4768 loss 8.0482 lr 9.60e-05 gnorm 1.57 54,813 tok/s peakVRAM 13.7GB
13
+ [14:17:05] step 100/4768 loss 7.6036 lr 1.20e-04 gnorm 0.99 54,787 tok/s peakVRAM 13.7GB
14
+ [14:20:17] step 120/4768 loss 7.1345 lr 1.44e-04 gnorm 0.74 54,772 tok/s peakVRAM 13.7GB
15
+ [14:23:28] step 140/4768 loss 6.7198 lr 1.68e-04 gnorm 0.81 54,761 tok/s peakVRAM 13.7GB
16
+ [14:26:40] step 160/4768 loss 6.3578 lr 1.92e-04 gnorm 0.82 54,756 tok/s peakVRAM 13.7GB
17
+ [14:29:51] step 180/4768 loss 6.0886 lr 2.16e-04 gnorm 0.72 54,754 tok/s peakVRAM 13.7GB
18
+ [14:33:03] step 200/4768 loss 5.8673 lr 2.40e-04 gnorm 0.70 54,750 tok/s peakVRAM 13.7GB
19
+ [14:36:14] step 220/4768 loss 5.6795 lr 2.64e-04 gnorm 0.63 54,744 tok/s peakVRAM 13.7GB
20
+ [14:39:26] step 240/4768 loss 5.5068 lr 2.88e-04 gnorm 0.73 54,744 tok/s peakVRAM 13.7GB
21
+ [14:42:37] step 260/4768 loss 5.3420 lr 3.12e-04 gnorm 0.89 54,745 tok/s peakVRAM 13.7GB
22
+ [14:45:49] step 280/4768 loss 5.1838 lr 3.36e-04 gnorm 0.74 54,746 tok/s peakVRAM 13.7GB
23
+ [14:49:00] step 300/4768 loss 5.0376 lr 3.60e-04 gnorm 0.77 54,747 tok/s peakVRAM 13.7GB
24
+ [14:52:12] step 320/4768 loss 4.9032 lr 3.84e-04 gnorm 0.87 54,747 tok/s peakVRAM 13.7GB
25
+ [14:55:24] step 340/4768 loss 4.7531 lr 4.08e-04 gnorm 0.75 54,744 tok/s peakVRAM 13.7GB
26
+ [14:58:35] step 360/4768 loss 4.6273 lr 4.32e-04 gnorm 0.59 54,745 tok/s peakVRAM 13.7GB
27
+ [15:01:47] step 380/4768 loss 4.5078 lr 4.56e-04 gnorm 0.56 54,744 tok/s peakVRAM 13.7GB
28
+ [15:04:58] step 400/4768 loss 4.4062 lr 4.80e-04 gnorm 0.45 54,745 tok/s peakVRAM 13.7GB
29
+ [15:08:10] step 420/4768 loss 4.3287 lr 5.04e-04 gnorm 0.46 54,747 tok/s peakVRAM 13.7GB
30
+ [15:11:21] step 440/4768 loss 4.2505 lr 5.28e-04 gnorm 0.40 54,744 tok/s peakVRAM 13.7GB
31
+ [15:14:33] step 460/4768 loss 4.1806 lr 5.52e-04 gnorm 0.39 54,744 tok/s peakVRAM 13.7GB
32
+ [15:17:44] step 480/4768 loss 4.1334 lr 5.76e-04 gnorm 0.35 54,742 tok/s peakVRAM 13.7GB
33
+ [15:20:56] step 500/4768 loss 4.0774 lr 6.00e-04 gnorm 0.31 54,736 tok/s peakVRAM 13.7GB
34
+ [15:24:07] step 520/4768 loss 4.0397 lr 6.00e-04 gnorm 0.27 54,735 tok/s peakVRAM 13.7GB
35
+ [15:27:19] step 540/4768 loss 3.9894 lr 6.00e-04 gnorm 0.22 54,736 tok/s peakVRAM 13.7GB
36
+ [15:30:31] step 560/4768 loss 3.9379 lr 6.00e-04 gnorm 0.23 54,734 tok/s peakVRAM 13.7GB
37
+ [15:33:42] step 580/4768 loss 3.8968 lr 6.00e-04 gnorm 0.23 54,732 tok/s peakVRAM 13.7GB
38
+ [15:36:54] step 600/4768 loss 3.8534 lr 6.00e-04 gnorm 0.21 54,728 tok/s peakVRAM 13.7GB
39
+ [15:40:05] step 620/4768 loss 3.8302 lr 6.00e-04 gnorm 0.21 54,730 tok/s peakVRAM 13.7GB
40
+ [15:43:17] step 640/4768 loss 3.8048 lr 6.00e-04 gnorm 0.19 54,735 tok/s peakVRAM 13.7GB
41
+ [15:46:29] step 660/4768 loss 3.7741 lr 6.00e-04 gnorm 0.17 54,729 tok/s peakVRAM 13.7GB
42
+ [15:49:40] step 680/4768 loss 3.7513 lr 6.00e-04 gnorm 0.18 54,735 tok/s peakVRAM 13.7GB
43
+ [15:52:52] step 700/4768 loss 3.7199 lr 6.00e-04 gnorm 0.18 54,736 tok/s peakVRAM 13.7GB
44
+ [15:56:03] step 720/4768 loss 3.6895 lr 6.00e-04 gnorm 0.17 54,733 tok/s peakVRAM 13.7GB
45
+ [15:59:15] step 740/4768 loss 3.6858 lr 6.00e-04 gnorm 0.19 54,732 tok/s peakVRAM 13.7GB
46
+ [16:02:26] step 760/4768 loss 3.6607 lr 6.00e-04 gnorm 0.16 54,731 tok/s peakVRAM 13.7GB
47
+ [16:05:38] step 780/4768 loss 3.6347 lr 6.00e-04 gnorm 0.16 54,731 tok/s peakVRAM 13.7GB
48
+ [16:08:50] step 800/4768 loss 3.6206 lr 6.00e-04 gnorm 0.14 54,732 tok/s peakVRAM 13.7GB
49
+ [16:12:01] step 820/4768 loss 3.6040 lr 6.00e-04 gnorm 0.14 54,730 tok/s peakVRAM 13.7GB
50
+ [16:15:13] step 840/4768 loss 3.5898 lr 6.00e-04 gnorm 0.15 54,728 tok/s peakVRAM 13.7GB
51
+ [16:18:24] step 860/4768 loss 3.5694 lr 6.00e-04 gnorm 0.16 54,730 tok/s peakVRAM 13.7GB
52
+ [16:21:36] step 880/4768 loss 3.5635 lr 6.00e-04 gnorm 0.14 54,727 tok/s peakVRAM 13.7GB
53
+ [16:24:48] step 900/4768 loss 3.5456 lr 6.00e-04 gnorm 0.14 54,728 tok/s peakVRAM 13.7GB
54
+ [16:27:59] step 920/4768 loss 3.5379 lr 6.00e-04 gnorm 0.14 54,732 tok/s peakVRAM 13.7GB
55
+ [16:31:11] step 940/4768 loss 3.5332 lr 6.00e-04 gnorm 0.14 54,731 tok/s peakVRAM 13.7GB
56
+ [16:34:22] step 960/4768 loss 3.5076 lr 6.00e-04 gnorm 0.13 54,727 tok/s peakVRAM 13.7GB
57
+ [16:37:34] step 980/4768 loss 3.5049 lr 6.00e-04 gnorm 0.12 54,728 tok/s peakVRAM 13.7GB
58
+ [16:40:46] step 1000/4768 loss 3.4945 lr 6.00e-04 gnorm 0.14 54,723 tok/s peakVRAM 13.7GB
59
+ [16:43:57] step 1020/4768 loss 3.4891 lr 6.00e-04 gnorm 0.13 54,721 tok/s peakVRAM 13.7GB
60
+ [16:47:09] step 1040/4768 loss 3.4625 lr 6.00e-04 gnorm 0.13 54,718 tok/s peakVRAM 13.7GB
61
+ [16:50:20] step 1060/4768 loss 3.4738 lr 6.00e-04 gnorm 0.14 54,717 tok/s peakVRAM 13.7GB
62
+ [16:53:32] step 1080/4768 loss 3.4629 lr 6.00e-04 gnorm 0.12 54,721 tok/s peakVRAM 13.7GB
63
+ [16:56:44] step 1100/4768 loss 3.4473 lr 6.00e-04 gnorm 0.14 54,724 tok/s peakVRAM 13.7GB
64
+ [16:59:55] step 1120/4768 loss 3.4297 lr 6.00e-04 gnorm 0.12 54,724 tok/s peakVRAM 13.7GB
65
+ [17:03:07] step 1140/4768 loss 3.4228 lr 6.00e-04 gnorm 0.12 54,728 tok/s peakVRAM 13.7GB
66
+ [17:06:19] step 1160/4768 loss 3.4173 lr 6.00e-04 gnorm 0.12 54,724 tok/s peakVRAM 13.7GB
67
+ [17:09:30] step 1180/4768 loss 3.4195 lr 6.00e-04 gnorm 0.12 54,725 tok/s peakVRAM 13.7GB
68
+ [17:12:42] step 1200/4768 loss 3.4018 lr 6.00e-04 gnorm 0.13 54,724 tok/s peakVRAM 13.7GB
69
+ [17:15:53] step 1220/4768 loss 3.3993 lr 6.00e-04 gnorm 0.13 54,726 tok/s peakVRAM 13.7GB
70
+ [17:19:05] step 1240/4768 loss 3.3916 lr 6.00e-04 gnorm 0.12 54,728 tok/s peakVRAM 13.7GB
71
+ [17:22:17] step 1260/4768 loss 3.3782 lr 6.00e-04 gnorm 0.12 54,729 tok/s peakVRAM 13.7GB
72
+ [17:25:28] step 1280/4768 loss 3.3807 lr 6.00e-04 gnorm 0.12 54,722 tok/s peakVRAM 13.7GB
73
+ [17:28:40] step 1300/4768 loss 3.3735 lr 6.00e-04 gnorm 0.11 54,719 tok/s peakVRAM 13.7GB
74
+ [17:31:51] step 1320/4768 loss 3.3659 lr 6.00e-04 gnorm 0.11 54,718 tok/s peakVRAM 13.7GB
75
+ [17:35:03] step 1340/4768 loss 3.3664 lr 6.00e-04 gnorm 0.13 54,719 tok/s peakVRAM 13.7GB
76
+ [17:38:15] step 1360/4768 loss 3.3575 lr 6.00e-04 gnorm 0.13 54,716 tok/s peakVRAM 13.7GB
77
+ [17:41:26] step 1380/4768 loss 3.3491 lr 6.00e-04 gnorm 0.11 54,722 tok/s peakVRAM 13.7GB
78
+ [17:44:38] step 1400/4768 loss 3.3387 lr 6.00e-04 gnorm 0.12 54,724 tok/s peakVRAM 13.7GB
79
+ [17:47:50] step 1420/4768 loss 3.3266 lr 6.00e-04 gnorm 0.12 54,725 tok/s peakVRAM 13.7GB
80
+ [17:51:01] step 1440/4768 loss 3.3337 lr 6.00e-04 gnorm 0.11 54,722 tok/s peakVRAM 13.7GB
81
+ [17:54:13] step 1460/4768 loss 3.3312 lr 6.00e-04 gnorm 0.11 54,722 tok/s peakVRAM 13.7GB
82
+ [17:57:24] step 1480/4768 loss 3.3352 lr 6.00e-04 gnorm 0.11 54,719 tok/s peakVRAM 13.7GB
83
+ [18:00:36] step 1500/4768 loss 3.3080 lr 6.00e-04 gnorm 0.10 54,720 tok/s peakVRAM 13.7GB
84
+ [18:03:48] step 1520/4768 loss 3.3154 lr 6.00e-04 gnorm 0.12 54,719 tok/s peakVRAM 13.7GB
85
+ [18:06:59] step 1540/4768 loss 3.2992 lr 6.00e-04 gnorm 0.11 54,721 tok/s peakVRAM 13.7GB
86
+ [18:10:11] step 1560/4768 loss 3.3079 lr 6.00e-04 gnorm 0.13 54,718 tok/s peakVRAM 13.7GB
87
+ [18:13:23] step 1580/4768 loss 3.2967 lr 6.00e-04 gnorm 0.11 54,719 tok/s peakVRAM 13.7GB
88
+ [18:16:34] step 1600/4768 loss 3.2853 lr 6.00e-04 gnorm 0.11 54,716 tok/s peakVRAM 13.7GB
89
+ [18:19:46] step 1620/4768 loss 3.2858 lr 6.00e-04 gnorm 0.12 54,719 tok/s peakVRAM 13.7GB
90
+ [18:22:57] step 1640/4768 loss 3.2827 lr 6.00e-04 gnorm 0.12 54,718 tok/s peakVRAM 13.7GB
91
+ [18:26:09] step 1660/4768 loss 3.2839 lr 6.00e-04 gnorm 0.11 54,723 tok/s peakVRAM 13.7GB
92
+ [18:29:21] step 1680/4768 loss 3.2733 lr 6.00e-04 gnorm 0.11 54,717 tok/s peakVRAM 13.7GB
93
+ [18:32:32] step 1700/4768 loss 3.2658 lr 6.00e-04 gnorm 0.12 54,718 tok/s peakVRAM 13.7GB
94
+ [18:35:44] step 1720/4768 loss 3.2679 lr 6.00e-04 gnorm 0.12 54,712 tok/s peakVRAM 13.7GB
95
+ [18:38:56] step 1740/4768 loss 3.2618 lr 6.00e-04 gnorm 0.10 54,710 tok/s peakVRAM 13.7GB
96
+ [18:42:07] step 1760/4768 loss 3.2606 lr 6.00e-04 gnorm 0.11 54,712 tok/s peakVRAM 13.7GB
97
+ [18:45:19] step 1780/4768 loss 3.2480 lr 6.00e-04 gnorm 0.11 54,715 tok/s peakVRAM 13.7GB
98
+ [18:48:31] step 1800/4768 loss 3.2525 lr 6.00e-04 gnorm 0.12 54,716 tok/s peakVRAM 13.7GB
99
+ [18:51:42] step 1820/4768 loss 3.2466 lr 6.00e-04 gnorm 0.11 54,717 tok/s peakVRAM 13.7GB
100
+ [18:54:54] step 1840/4768 loss 3.2411 lr 6.00e-04 gnorm 0.10 54,712 tok/s peakVRAM 13.7GB
101
+ [18:58:06] step 1860/4768 loss 3.2404 lr 6.00e-04 gnorm 0.12 54,710 tok/s peakVRAM 13.7GB
102
+ [19:01:17] step 1880/4768 loss 3.2358 lr 6.00e-04 gnorm 0.11 54,707 tok/s peakVRAM 13.7GB
103
+ [19:04:29] step 1900/4768 loss 3.2309 lr 6.00e-04 gnorm 0.10 54,706 tok/s peakVRAM 13.7GB
104
+ [19:07:41] step 1920/4768 loss 3.2304 lr 6.00e-04 gnorm 0.10 54,706 tok/s peakVRAM 13.7GB
105
+ [19:10:52] step 1940/4768 loss 3.2230 lr 6.00e-04 gnorm 0.12 54,710 tok/s peakVRAM 13.7GB
106
+ [19:14:04] step 1960/4768 loss 3.2270 lr 6.00e-04 gnorm 0.11 54,711 tok/s peakVRAM 13.7GB
107
+ [19:17:16] step 1980/4768 loss 3.2220 lr 6.00e-04 gnorm 0.09 54,714 tok/s peakVRAM 13.7GB
108
+ [19:20:27] step 2000/4768 loss 3.2153 lr 6.00e-04 gnorm 0.10 54,722 tok/s peakVRAM 13.7GB
109
+ [19:23:39] step 2020/4768 loss 3.2152 lr 6.00e-04 gnorm 0.10 54,720 tok/s peakVRAM 13.7GB
110
+ [19:26:50] step 2040/4768 loss 3.2077 lr 6.00e-04 gnorm 0.11 54,721 tok/s peakVRAM 13.7GB
111
+ [19:30:02] step 2060/4768 loss 3.2078 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
112
+ [19:33:14] step 2080/4768 loss 3.1967 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
113
+ [19:36:25] step 2100/4768 loss 3.2051 lr 6.00e-04 gnorm 0.11 54,718 tok/s peakVRAM 13.7GB
114
+ [19:39:37] step 2120/4768 loss 3.1951 lr 6.00e-04 gnorm 0.11 54,717 tok/s peakVRAM 13.7GB
115
+ [19:42:49] step 2140/4768 loss 3.2076 lr 6.00e-04 gnorm 0.11 54,717 tok/s peakVRAM 13.7GB
116
+ [19:46:00] step 2160/4768 loss 3.1865 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
117
+ [19:49:12] step 2180/4768 loss 3.1881 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
118
+ [19:52:24] step 2200/4768 loss 3.1892 lr 6.00e-04 gnorm 0.10 54,711 tok/s peakVRAM 13.7GB
119
+ [19:55:35] step 2220/4768 loss 3.1820 lr 6.00e-04 gnorm 0.10 54,709 tok/s peakVRAM 13.7GB
120
+ [19:58:47] step 2240/4768 loss 3.1787 lr 6.00e-04 gnorm 0.09 54,712 tok/s peakVRAM 13.7GB
121
+ [20:01:58] step 2260/4768 loss 3.1754 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
122
+ [20:05:10] step 2280/4768 loss 3.1711 lr 6.00e-04 gnorm 0.11 54,712 tok/s peakVRAM 13.7GB
123
+ [20:08:22] step 2300/4768 loss 3.1747 lr 6.00e-04 gnorm 0.09 54,716 tok/s peakVRAM 13.7GB
124
+ [20:11:33] step 2320/4768 loss 3.1717 lr 6.00e-04 gnorm 0.11 54,718 tok/s peakVRAM 13.7GB
125
+ [20:14:45] step 2340/4768 loss 3.1667 lr 6.00e-04 gnorm 0.11 54,715 tok/s peakVRAM 13.7GB
126
+ [20:17:57] step 2360/4768 loss 3.1573 lr 6.00e-04 gnorm 0.10 54,718 tok/s peakVRAM 13.7GB
127
+ [20:21:08] step 2380/4768 loss 3.1559 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
128
+ [20:24:20] step 2400/4768 loss 3.1566 lr 6.00e-04 gnorm 0.09 54,718 tok/s peakVRAM 13.7GB
129
+ [20:27:32] step 2420/4768 loss 3.1597 lr 6.00e-04 gnorm 0.10 54,720 tok/s peakVRAM 13.7GB
130
+ [20:30:43] step 2440/4768 loss 3.1548 lr 6.00e-04 gnorm 0.09 54,721 tok/s peakVRAM 13.7GB
131
+ [20:33:55] step 2460/4768 loss 3.1532 lr 6.00e-04 gnorm 0.11 54,719 tok/s peakVRAM 13.7GB
132
+ [20:37:06] step 2480/4768 loss 3.1504 lr 6.00e-04 gnorm 0.11 54,717 tok/s peakVRAM 13.7GB
133
+ [20:40:18] step 2500/4768 loss 3.1447 lr 6.00e-04 gnorm 0.10 54,716 tok/s peakVRAM 13.7GB
134
+ [20:43:30] step 2520/4768 loss 3.1466 lr 6.00e-04 gnorm 0.10 54,719 tok/s peakVRAM 13.7GB
135
+ [20:46:41] step 2540/4768 loss 3.1465 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
136
+ [20:49:53] step 2560/4768 loss 3.1434 lr 6.00e-04 gnorm 0.11 54,719 tok/s peakVRAM 13.7GB
137
+ [20:53:05] step 2580/4768 loss 3.1385 lr 6.00e-04 gnorm 0.10 54,719 tok/s peakVRAM 13.7GB
138
+ [20:56:16] step 2600/4768 loss 3.1294 lr 6.00e-04 gnorm 0.10 54,716 tok/s peakVRAM 13.7GB
139
+ [20:59:28] step 2620/4768 loss 3.1375 lr 6.00e-04 gnorm 0.09 54,712 tok/s peakVRAM 13.7GB
140
+ [21:02:40] step 2640/4768 loss 3.1389 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
141
+ [21:05:51] step 2660/4768 loss 3.1372 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
142
+ [21:09:03] step 2680/4768 loss 3.1326 lr 6.00e-04 gnorm 0.10 54,716 tok/s peakVRAM 13.7GB
143
+ [21:12:14] step 2700/4768 loss 3.1352 lr 6.00e-04 gnorm 0.10 54,718 tok/s peakVRAM 13.7GB
144
+ [21:15:26] step 2720/4768 loss 3.1180 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
145
+ [21:18:38] step 2740/4768 loss 3.1193 lr 6.00e-04 gnorm 0.10 54,723 tok/s peakVRAM 13.7GB
146
+ [21:21:49] step 2760/4768 loss 3.1220 lr 6.00e-04 gnorm 0.10 54,724 tok/s peakVRAM 13.7GB
147
+ [21:25:01] step 2780/4768 loss 3.1132 lr 6.00e-04 gnorm 0.10 54,722 tok/s peakVRAM 13.7GB
148
+ [21:28:13] step 2800/4768 loss 3.1081 lr 6.00e-04 gnorm 0.10 54,720 tok/s peakVRAM 13.7GB
149
+ [21:31:24] step 2820/4768 loss 3.1058 lr 6.00e-04 gnorm 0.10 54,719 tok/s peakVRAM 13.7GB
150
+ [21:34:36] step 2840/4768 loss 3.1134 lr 6.00e-04 gnorm 0.10 54,719 tok/s peakVRAM 13.7GB
151
+ [21:37:47] step 2860/4768 loss 3.1055 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
152
+ [21:40:59] step 2880/4768 loss 3.1071 lr 6.00e-04 gnorm 0.09 54,714 tok/s peakVRAM 13.7GB
153
+ [21:44:11] step 2900/4768 loss 3.1088 lr 6.00e-04 gnorm 0.11 54,713 tok/s peakVRAM 13.7GB
154
+ [21:47:22] step 2920/4768 loss 3.1159 lr 6.00e-04 gnorm 0.10 54,718 tok/s peakVRAM 13.7GB
155
+ [21:50:34] step 2940/4768 loss 3.1065 lr 6.00e-04 gnorm 0.08 54,720 tok/s peakVRAM 13.7GB
156
+ [21:53:46] step 2960/4768 loss 3.1085 lr 6.00e-04 gnorm 0.10 54,720 tok/s peakVRAM 13.7GB
157
+ [21:56:57] step 2980/4768 loss 3.0954 lr 6.00e-04 gnorm 0.10 54,721 tok/s peakVRAM 13.7GB
158
+ [22:00:09] step 3000/4768 loss 3.0957 lr 6.00e-04 gnorm 0.09 54,721 tok/s peakVRAM 13.7GB
159
+ [22:03:21] step 3020/4768 loss 3.0920 lr 6.00e-04 gnorm 0.09 54,721 tok/s peakVRAM 13.7GB
160
+ [22:06:32] step 3040/4768 loss 3.0889 lr 6.00e-04 gnorm 0.09 54,714 tok/s peakVRAM 13.7GB
161
+ [22:09:44] step 3060/4768 loss 3.0923 lr 6.00e-04 gnorm 0.09 54,718 tok/s peakVRAM 13.7GB
162
+ [22:12:55] step 3080/4768 loss 3.0847 lr 6.00e-04 gnorm 0.09 54,720 tok/s peakVRAM 13.7GB
163
+ [22:16:07] step 3100/4768 loss 3.0786 lr 6.00e-04 gnorm 0.10 54,725 tok/s peakVRAM 13.7GB
164
+ [22:19:19] step 3120/4768 loss 3.0947 lr 6.00e-04 gnorm 0.10 54,721 tok/s peakVRAM 13.7GB
165
+ [22:22:30] step 3140/4768 loss 3.0798 lr 6.00e-04 gnorm 0.10 54,724 tok/s peakVRAM 13.7GB
166
+ [22:25:42] step 3160/4768 loss 3.0761 lr 6.00e-04 gnorm 0.09 54,722 tok/s peakVRAM 13.7GB
167
+ [22:28:54] step 3180/4768 loss 3.0823 lr 6.00e-04 gnorm 0.09 54,718 tok/s peakVRAM 13.7GB
168
+ [22:32:05] step 3200/4768 loss 3.0832 lr 6.00e-04 gnorm 0.09 54,720 tok/s peakVRAM 13.7GB
169
+ [22:35:17] step 3220/4768 loss 3.0926 lr 6.00e-04 gnorm 0.10 54,722 tok/s peakVRAM 13.7GB
170
+ [22:38:28] step 3240/4768 loss 3.0746 lr 6.00e-04 gnorm 0.09 54,719 tok/s peakVRAM 13.7GB
171
+ [22:41:40] step 3260/4768 loss 3.0712 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
172
+ [22:44:52] step 3280/4768 loss 3.0670 lr 6.00e-04 gnorm 0.09 54,719 tok/s peakVRAM 13.7GB
173
+ [22:48:03] step 3300/4768 loss 3.0780 lr 6.00e-04 gnorm 0.09 54,712 tok/s peakVRAM 13.7GB
174
+ [22:51:15] step 3320/4768 loss 3.0718 lr 6.00e-04 gnorm 0.08 54,713 tok/s peakVRAM 13.7GB
175
+ [22:54:27] step 3340/4768 loss 3.0722 lr 6.00e-04 gnorm 0.09 54,716 tok/s peakVRAM 13.7GB
176
+ [22:57:38] step 3360/4768 loss 3.0630 lr 6.00e-04 gnorm 0.10 54,718 tok/s peakVRAM 13.7GB
177
+ [23:00:50] step 3380/4768 loss 3.0728 lr 6.00e-04 gnorm 0.10 54,715 tok/s peakVRAM 13.7GB
178
+ [23:04:02] step 3400/4768 loss 3.0580 lr 6.00e-04 gnorm 0.09 54,719 tok/s peakVRAM 13.7GB
179
+ [23:07:13] step 3420/4768 loss 3.0660 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
180
+ [23:10:25] step 3440/4768 loss 3.0576 lr 6.00e-04 gnorm 0.10 54,716 tok/s peakVRAM 13.7GB
181
+ [23:13:36] step 3460/4768 loss 3.0667 lr 6.00e-04 gnorm 0.09 54,715 tok/s peakVRAM 13.7GB
182
+ [23:16:48] step 3480/4768 loss 3.0667 lr 6.00e-04 gnorm 0.09 54,716 tok/s peakVRAM 13.7GB
183
+ [23:20:00] step 3500/4768 loss 3.0526 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
184
+ [23:23:11] step 3520/4768 loss 3.0552 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
185
+ [23:26:23] step 3540/4768 loss 3.0578 lr 6.00e-04 gnorm 0.09 54,716 tok/s peakVRAM 13.7GB
186
+ [23:29:35] step 3560/4768 loss 3.0616 lr 6.00e-04 gnorm 0.09 54,717 tok/s peakVRAM 13.7GB
187
+ [23:32:46] step 3580/4768 loss 3.0558 lr 6.00e-04 gnorm 0.08 54,722 tok/s peakVRAM 13.7GB
188
+ [23:35:58] step 3600/4768 loss 3.0576 lr 6.00e-04 gnorm 0.10 54,721 tok/s peakVRAM 13.7GB
189
+ [23:39:10] step 3620/4768 loss 3.0497 lr 6.00e-04 gnorm 0.10 54,719 tok/s peakVRAM 13.7GB
190
+ [23:42:21] step 3640/4768 loss 3.0574 lr 6.00e-04 gnorm 0.09 54,722 tok/s peakVRAM 13.7GB
191
+ [23:45:33] step 3660/4768 loss 3.0462 lr 6.00e-04 gnorm 0.09 54,720 tok/s peakVRAM 13.7GB
192
+ [23:48:44] step 3680/4768 loss 3.0531 lr 6.00e-04 gnorm 0.09 54,724 tok/s peakVRAM 13.7GB
193
+ [23:51:56] step 3700/4768 loss 3.0503 lr 6.00e-04 gnorm 0.09 54,726 tok/s peakVRAM 13.7GB
194
+ [23:55:08] step 3720/4768 loss 3.0435 lr 6.00e-04 gnorm 0.08 54,724 tok/s peakVRAM 13.7GB
195
+ [23:58:19] step 3740/4768 loss 3.0400 lr 6.00e-04 gnorm 0.08 54,727 tok/s peakVRAM 13.7GB
196
+ [00:01:31] step 3760/4768 loss 3.0384 lr 6.00e-04 gnorm 0.09 54,725 tok/s peakVRAM 13.7GB
197
+ [00:04:42] step 3780/4768 loss 3.0390 lr 6.00e-04 gnorm 0.10 54,721 tok/s peakVRAM 13.7GB
198
+ [00:07:54] step 3800/4768 loss 3.0400 lr 6.00e-04 gnorm 0.09 54,718 tok/s peakVRAM 13.7GB
199
+ [00:11:06] step 3820/4768 loss 3.0331 lr 5.61e-04 gnorm 0.09 54,718 tok/s peakVRAM 13.7GB
200
+ [00:14:17] step 3840/4768 loss 3.0316 lr 5.13e-04 gnorm 0.08 54,717 tok/s peakVRAM 13.7GB
201
+ [00:17:29] step 3860/4768 loss 3.0245 lr 4.83e-04 gnorm 0.07 54,718 tok/s peakVRAM 13.7GB
202
+ [00:20:41] step 3880/4768 loss 3.0217 lr 4.59e-04 gnorm 0.08 54,721 tok/s peakVRAM 13.7GB
203
+ [00:23:52] step 3900/4768 loss 3.0150 lr 4.39e-04 gnorm 0.08 54,716 tok/s peakVRAM 13.7GB
204
+ [00:27:04] step 3920/4768 loss 3.0093 lr 4.21e-04 gnorm 0.07 54,715 tok/s peakVRAM 13.7GB
205
+ [00:30:16] step 3940/4768 loss 3.0019 lr 4.05e-04 gnorm 0.07 54,715 tok/s peakVRAM 13.7GB
206
+ [00:33:27] step 3960/4768 loss 3.0024 lr 3.89e-04 gnorm 0.08 54,716 tok/s peakVRAM 13.7GB
207
+ [00:36:39] step 3980/4768 loss 2.9945 lr 3.75e-04 gnorm 0.07 54,721 tok/s peakVRAM 13.7GB
208
+ [00:39:50] step 4000/4768 loss 2.9933 lr 3.62e-04 gnorm 0.07 54,722 tok/s peakVRAM 13.7GB
209
+ [00:43:02] step 4020/4768 loss 2.9906 lr 3.50e-04 gnorm 0.07 54,713 tok/s peakVRAM 13.7GB
210
+ [00:46:14] step 4040/4768 loss 2.9812 lr 3.38e-04 gnorm 0.07 54,714 tok/s peakVRAM 13.7GB
211
+ [00:49:25] step 4060/4768 loss 2.9809 lr 3.26e-04 gnorm 0.07 54,719 tok/s peakVRAM 13.7GB
212
+ [00:52:37] step 4080/4768 loss 2.9725 lr 3.15e-04 gnorm 0.07 54,715 tok/s peakVRAM 13.7GB
213
+ [00:55:49] step 4100/4768 loss 2.9776 lr 3.05e-04 gnorm 0.07 54,716 tok/s peakVRAM 13.7GB
214
+ [00:59:00] step 4120/4768 loss 2.9664 lr 2.95e-04 gnorm 0.07 54,720 tok/s peakVRAM 13.7GB
215
+ [01:02:12] step 4140/4768 loss 2.9586 lr 2.85e-04 gnorm 0.07 54,716 tok/s peakVRAM 13.7GB
216
+ [01:05:24] step 4160/4768 loss 2.9768 lr 2.75e-04 gnorm 0.07 54,710 tok/s peakVRAM 13.7GB
217
+ [01:08:35] step 4180/4768 loss 2.9694 lr 2.66e-04 gnorm 0.07 54,712 tok/s peakVRAM 13.7GB
218
+ [01:11:47] step 4200/4768 loss 2.9631 lr 2.57e-04 gnorm 0.07 54,715 tok/s peakVRAM 13.7GB
219
+ [01:14:58] step 4220/4768 loss 2.9561 lr 2.48e-04 gnorm 0.07 54,716 tok/s peakVRAM 13.7GB
220
+ [01:18:10] step 4240/4768 loss 2.9624 lr 2.40e-04 gnorm 0.07 54,718 tok/s peakVRAM 13.7GB
221
+ [01:21:22] step 4260/4768 loss 2.9546 lr 2.31e-04 gnorm 0.07 54,720 tok/s peakVRAM 13.7GB
222
+ [01:24:33] step 4280/4768 loss 2.9540 lr 2.23e-04 gnorm 0.07 54,725 tok/s peakVRAM 13.7GB
223
+ [01:27:45] step 4300/4768 loss 2.9518 lr 2.15e-04 gnorm 0.07 54,721 tok/s peakVRAM 13.7GB
224
+ [01:30:57] step 4320/4768 loss 2.9470 lr 2.07e-04 gnorm 0.07 54,718 tok/s peakVRAM 13.7GB
225
+ [01:34:08] step 4340/4768 loss 2.9457 lr 1.99e-04 gnorm 0.07 54,717 tok/s peakVRAM 13.7GB
226
+ [01:37:20] step 4360/4768 loss 2.9396 lr 1.92e-04 gnorm 0.07 54,717 tok/s peakVRAM 13.7GB
227
+ [01:40:32] step 4380/4768 loss 2.9465 lr 1.84e-04 gnorm 0.07 54,713 tok/s peakVRAM 13.7GB
228
+ [01:43:43] step 4400/4768 loss 2.9439 lr 1.77e-04 gnorm 0.07 54,711 tok/s peakVRAM 13.7GB
229
+ [01:46:55] step 4420/4768 loss 2.9371 lr 1.70e-04 gnorm 0.07 54,714 tok/s peakVRAM 13.7GB
230
+ [01:50:06] step 4440/4768 loss 2.9435 lr 1.63e-04 gnorm 0.07 54,713 tok/s peakVRAM 13.7GB
231
+ [01:53:18] step 4460/4768 loss 2.9390 lr 1.56e-04 gnorm 0.07 54,716 tok/s peakVRAM 13.7GB
232
+ [01:56:30] step 4480/4768 loss 2.9341 lr 1.49e-04 gnorm 0.07 54,719 tok/s peakVRAM 13.7GB
233
+ [01:59:41] step 4500/4768 loss 2.9313 lr 1.42e-04 gnorm 0.07 54,720 tok/s peakVRAM 13.7GB
234
+ [02:02:53] step 4520/4768 loss 2.9332 lr 1.36e-04 gnorm 0.07 54,717 tok/s peakVRAM 13.7GB
235
+ [02:06:05] step 4540/4768 loss 2.9246 lr 1.29e-04 gnorm 0.07 54,717 tok/s peakVRAM 13.7GB
236
+ [02:09:16] step 4560/4768 loss 2.9280 lr 1.23e-04 gnorm 0.07 54,712 tok/s peakVRAM 13.7GB
237
+ [02:12:28] step 4580/4768 loss 2.9309 lr 1.16e-04 gnorm 0.07 54,710 tok/s peakVRAM 13.7GB
238
+ [02:15:40] step 4600/4768 loss 2.9299 lr 1.10e-04 gnorm 0.07 54,708 tok/s peakVRAM 13.7GB
239
+ [02:18:51] step 4620/4768 loss 2.9163 lr 1.04e-04 gnorm 0.07 54,711 tok/s peakVRAM 13.7GB
240
+ [02:22:03] step 4640/4768 loss 2.9256 lr 9.78e-05 gnorm 0.07 54,714 tok/s peakVRAM 13.7GB
241
+ [02:25:15] step 4660/4768 loss 2.9184 lr 9.18e-05 gnorm 0.07 54,713 tok/s peakVRAM 13.7GB
242
+ [02:28:26] step 4680/4768 loss 2.9159 lr 8.58e-05 gnorm 0.07 54,711 tok/s peakVRAM 13.7GB
243
+ [02:31:38] step 4700/4768 loss 2.9121 lr 7.99e-05 gnorm 0.07 54,712 tok/s peakVRAM 13.7GB
244
+ [02:34:50] step 4720/4768 loss 2.9131 lr 7.41e-05 gnorm 0.07 54,715 tok/s peakVRAM 13.7GB
245
+ [02:38:01] step 4740/4768 loss 2.9110 lr 6.83e-05 gnorm 0.07 54,718 tok/s peakVRAM 13.7GB
246
+ [02:41:13] step 4760/4768 loss 2.9202 lr 6.26e-05 gnorm 0.07 54,720 tok/s peakVRAM 13.7GB
247
+ [02:42:31] ckpt @ 4768
248
+ [02:42:31] DONE
R2/run/train.out ADDED
@@ -0,0 +1,248 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [13:59:56] device=cuda:0 bf16=True dev=NVIDIA GeForce RTX 4090 world=1
2
+ [13:59:56] dane: train=2,500,001,211B block=4096 batch=128 tok/step=524,288
3
+ [13:59:58] model GPT-ref: 183.6M param (L16 d896 h14)
4
+ [13:59:59] optimizer=muon muon_lr=0.02 (mult=33.3x)
5
+ [13:59:59] r11 pamiec: act_ckpt=True loss_chunk=1024
6
+ [13:59:59] r13: mixer_layout=AAAAAAAAAAAAAAAA gdn_head_dim=64 attn_bias=0 rope_style=interleaved
7
+ [13:59:59] torch.compile enabled
8
+ [13:59:59] START gpt-ref: steps=4768 (od 0) lr=0.0006->6e-05
9
+ [14:04:20] step 20/4768 loss 10.3979 lr 2.40e-05 gnorm 2.60 40,239 tok/s peakVRAM 13.7GB
10
+ [14:07:31] step 40/4768 loss 9.5831 lr 4.80e-05 gnorm 1.42 54,808 tok/s peakVRAM 13.7GB
11
+ [14:10:43] step 60/4768 loss 8.7021 lr 7.20e-05 gnorm 0.76 54,817 tok/s peakVRAM 13.7GB
12
+ [14:13:54] step 80/4768 loss 8.0482 lr 9.60e-05 gnorm 1.57 54,813 tok/s peakVRAM 13.7GB
13
+ [14:17:05] step 100/4768 loss 7.6036 lr 1.20e-04 gnorm 0.99 54,787 tok/s peakVRAM 13.7GB
14
+ [14:20:17] step 120/4768 loss 7.1345 lr 1.44e-04 gnorm 0.74 54,772 tok/s peakVRAM 13.7GB
15
+ [14:23:28] step 140/4768 loss 6.7198 lr 1.68e-04 gnorm 0.81 54,761 tok/s peakVRAM 13.7GB
16
+ [14:26:40] step 160/4768 loss 6.3578 lr 1.92e-04 gnorm 0.82 54,756 tok/s peakVRAM 13.7GB
17
+ [14:29:51] step 180/4768 loss 6.0886 lr 2.16e-04 gnorm 0.72 54,754 tok/s peakVRAM 13.7GB
18
+ [14:33:03] step 200/4768 loss 5.8673 lr 2.40e-04 gnorm 0.70 54,750 tok/s peakVRAM 13.7GB
19
+ [14:36:14] step 220/4768 loss 5.6795 lr 2.64e-04 gnorm 0.63 54,744 tok/s peakVRAM 13.7GB
20
+ [14:39:26] step 240/4768 loss 5.5068 lr 2.88e-04 gnorm 0.73 54,744 tok/s peakVRAM 13.7GB
21
+ [14:42:37] step 260/4768 loss 5.3420 lr 3.12e-04 gnorm 0.89 54,745 tok/s peakVRAM 13.7GB
22
+ [14:45:49] step 280/4768 loss 5.1838 lr 3.36e-04 gnorm 0.74 54,746 tok/s peakVRAM 13.7GB
23
+ [14:49:00] step 300/4768 loss 5.0376 lr 3.60e-04 gnorm 0.77 54,747 tok/s peakVRAM 13.7GB
24
+ [14:52:12] step 320/4768 loss 4.9032 lr 3.84e-04 gnorm 0.87 54,747 tok/s peakVRAM 13.7GB
25
+ [14:55:24] step 340/4768 loss 4.7531 lr 4.08e-04 gnorm 0.75 54,744 tok/s peakVRAM 13.7GB
26
+ [14:58:35] step 360/4768 loss 4.6273 lr 4.32e-04 gnorm 0.59 54,745 tok/s peakVRAM 13.7GB
27
+ [15:01:47] step 380/4768 loss 4.5078 lr 4.56e-04 gnorm 0.56 54,744 tok/s peakVRAM 13.7GB
28
+ [15:04:58] step 400/4768 loss 4.4062 lr 4.80e-04 gnorm 0.45 54,745 tok/s peakVRAM 13.7GB
29
+ [15:08:10] step 420/4768 loss 4.3287 lr 5.04e-04 gnorm 0.46 54,747 tok/s peakVRAM 13.7GB
30
+ [15:11:21] step 440/4768 loss 4.2505 lr 5.28e-04 gnorm 0.40 54,744 tok/s peakVRAM 13.7GB
31
+ [15:14:33] step 460/4768 loss 4.1806 lr 5.52e-04 gnorm 0.39 54,744 tok/s peakVRAM 13.7GB
32
+ [15:17:44] step 480/4768 loss 4.1334 lr 5.76e-04 gnorm 0.35 54,742 tok/s peakVRAM 13.7GB
33
+ [15:20:56] step 500/4768 loss 4.0774 lr 6.00e-04 gnorm 0.31 54,736 tok/s peakVRAM 13.7GB
34
+ [15:24:07] step 520/4768 loss 4.0397 lr 6.00e-04 gnorm 0.27 54,735 tok/s peakVRAM 13.7GB
35
+ [15:27:19] step 540/4768 loss 3.9894 lr 6.00e-04 gnorm 0.22 54,736 tok/s peakVRAM 13.7GB
36
+ [15:30:31] step 560/4768 loss 3.9379 lr 6.00e-04 gnorm 0.23 54,734 tok/s peakVRAM 13.7GB
37
+ [15:33:42] step 580/4768 loss 3.8968 lr 6.00e-04 gnorm 0.23 54,732 tok/s peakVRAM 13.7GB
38
+ [15:36:54] step 600/4768 loss 3.8534 lr 6.00e-04 gnorm 0.21 54,728 tok/s peakVRAM 13.7GB
39
+ [15:40:05] step 620/4768 loss 3.8302 lr 6.00e-04 gnorm 0.21 54,730 tok/s peakVRAM 13.7GB
40
+ [15:43:17] step 640/4768 loss 3.8048 lr 6.00e-04 gnorm 0.19 54,735 tok/s peakVRAM 13.7GB
41
+ [15:46:29] step 660/4768 loss 3.7741 lr 6.00e-04 gnorm 0.17 54,729 tok/s peakVRAM 13.7GB
42
+ [15:49:40] step 680/4768 loss 3.7513 lr 6.00e-04 gnorm 0.18 54,735 tok/s peakVRAM 13.7GB
43
+ [15:52:52] step 700/4768 loss 3.7199 lr 6.00e-04 gnorm 0.18 54,736 tok/s peakVRAM 13.7GB
44
+ [15:56:03] step 720/4768 loss 3.6895 lr 6.00e-04 gnorm 0.17 54,733 tok/s peakVRAM 13.7GB
45
+ [15:59:15] step 740/4768 loss 3.6858 lr 6.00e-04 gnorm 0.19 54,732 tok/s peakVRAM 13.7GB
46
+ [16:02:26] step 760/4768 loss 3.6607 lr 6.00e-04 gnorm 0.16 54,731 tok/s peakVRAM 13.7GB
47
+ [16:05:38] step 780/4768 loss 3.6347 lr 6.00e-04 gnorm 0.16 54,731 tok/s peakVRAM 13.7GB
48
+ [16:08:50] step 800/4768 loss 3.6206 lr 6.00e-04 gnorm 0.14 54,732 tok/s peakVRAM 13.7GB
49
+ [16:12:01] step 820/4768 loss 3.6040 lr 6.00e-04 gnorm 0.14 54,730 tok/s peakVRAM 13.7GB
50
+ [16:15:13] step 840/4768 loss 3.5898 lr 6.00e-04 gnorm 0.15 54,728 tok/s peakVRAM 13.7GB
51
+ [16:18:24] step 860/4768 loss 3.5694 lr 6.00e-04 gnorm 0.16 54,730 tok/s peakVRAM 13.7GB
52
+ [16:21:36] step 880/4768 loss 3.5635 lr 6.00e-04 gnorm 0.14 54,727 tok/s peakVRAM 13.7GB
53
+ [16:24:48] step 900/4768 loss 3.5456 lr 6.00e-04 gnorm 0.14 54,728 tok/s peakVRAM 13.7GB
54
+ [16:27:59] step 920/4768 loss 3.5379 lr 6.00e-04 gnorm 0.14 54,732 tok/s peakVRAM 13.7GB
55
+ [16:31:11] step 940/4768 loss 3.5332 lr 6.00e-04 gnorm 0.14 54,731 tok/s peakVRAM 13.7GB
56
+ [16:34:22] step 960/4768 loss 3.5076 lr 6.00e-04 gnorm 0.13 54,727 tok/s peakVRAM 13.7GB
57
+ [16:37:34] step 980/4768 loss 3.5049 lr 6.00e-04 gnorm 0.12 54,728 tok/s peakVRAM 13.7GB
58
+ [16:40:46] step 1000/4768 loss 3.4945 lr 6.00e-04 gnorm 0.14 54,723 tok/s peakVRAM 13.7GB
59
+ [16:43:57] step 1020/4768 loss 3.4891 lr 6.00e-04 gnorm 0.13 54,721 tok/s peakVRAM 13.7GB
60
+ [16:47:09] step 1040/4768 loss 3.4625 lr 6.00e-04 gnorm 0.13 54,718 tok/s peakVRAM 13.7GB
61
+ [16:50:20] step 1060/4768 loss 3.4738 lr 6.00e-04 gnorm 0.14 54,717 tok/s peakVRAM 13.7GB
62
+ [16:53:32] step 1080/4768 loss 3.4629 lr 6.00e-04 gnorm 0.12 54,721 tok/s peakVRAM 13.7GB
63
+ [16:56:44] step 1100/4768 loss 3.4473 lr 6.00e-04 gnorm 0.14 54,724 tok/s peakVRAM 13.7GB
64
+ [16:59:55] step 1120/4768 loss 3.4297 lr 6.00e-04 gnorm 0.12 54,724 tok/s peakVRAM 13.7GB
65
+ [17:03:07] step 1140/4768 loss 3.4228 lr 6.00e-04 gnorm 0.12 54,728 tok/s peakVRAM 13.7GB
66
+ [17:06:19] step 1160/4768 loss 3.4173 lr 6.00e-04 gnorm 0.12 54,724 tok/s peakVRAM 13.7GB
67
+ [17:09:30] step 1180/4768 loss 3.4195 lr 6.00e-04 gnorm 0.12 54,725 tok/s peakVRAM 13.7GB
68
+ [17:12:42] step 1200/4768 loss 3.4018 lr 6.00e-04 gnorm 0.13 54,724 tok/s peakVRAM 13.7GB
69
+ [17:15:53] step 1220/4768 loss 3.3993 lr 6.00e-04 gnorm 0.13 54,726 tok/s peakVRAM 13.7GB
70
+ [17:19:05] step 1240/4768 loss 3.3916 lr 6.00e-04 gnorm 0.12 54,728 tok/s peakVRAM 13.7GB
71
+ [17:22:17] step 1260/4768 loss 3.3782 lr 6.00e-04 gnorm 0.12 54,729 tok/s peakVRAM 13.7GB
72
+ [17:25:28] step 1280/4768 loss 3.3807 lr 6.00e-04 gnorm 0.12 54,722 tok/s peakVRAM 13.7GB
73
+ [17:28:40] step 1300/4768 loss 3.3735 lr 6.00e-04 gnorm 0.11 54,719 tok/s peakVRAM 13.7GB
74
+ [17:31:51] step 1320/4768 loss 3.3659 lr 6.00e-04 gnorm 0.11 54,718 tok/s peakVRAM 13.7GB
75
+ [17:35:03] step 1340/4768 loss 3.3664 lr 6.00e-04 gnorm 0.13 54,719 tok/s peakVRAM 13.7GB
76
+ [17:38:15] step 1360/4768 loss 3.3575 lr 6.00e-04 gnorm 0.13 54,716 tok/s peakVRAM 13.7GB
77
+ [17:41:26] step 1380/4768 loss 3.3491 lr 6.00e-04 gnorm 0.11 54,722 tok/s peakVRAM 13.7GB
78
+ [17:44:38] step 1400/4768 loss 3.3387 lr 6.00e-04 gnorm 0.12 54,724 tok/s peakVRAM 13.7GB
79
+ [17:47:50] step 1420/4768 loss 3.3266 lr 6.00e-04 gnorm 0.12 54,725 tok/s peakVRAM 13.7GB
80
+ [17:51:01] step 1440/4768 loss 3.3337 lr 6.00e-04 gnorm 0.11 54,722 tok/s peakVRAM 13.7GB
81
+ [17:54:13] step 1460/4768 loss 3.3312 lr 6.00e-04 gnorm 0.11 54,722 tok/s peakVRAM 13.7GB
82
+ [17:57:24] step 1480/4768 loss 3.3352 lr 6.00e-04 gnorm 0.11 54,719 tok/s peakVRAM 13.7GB
83
+ [18:00:36] step 1500/4768 loss 3.3080 lr 6.00e-04 gnorm 0.10 54,720 tok/s peakVRAM 13.7GB
84
+ [18:03:48] step 1520/4768 loss 3.3154 lr 6.00e-04 gnorm 0.12 54,719 tok/s peakVRAM 13.7GB
85
+ [18:06:59] step 1540/4768 loss 3.2992 lr 6.00e-04 gnorm 0.11 54,721 tok/s peakVRAM 13.7GB
86
+ [18:10:11] step 1560/4768 loss 3.3079 lr 6.00e-04 gnorm 0.13 54,718 tok/s peakVRAM 13.7GB
87
+ [18:13:23] step 1580/4768 loss 3.2967 lr 6.00e-04 gnorm 0.11 54,719 tok/s peakVRAM 13.7GB
88
+ [18:16:34] step 1600/4768 loss 3.2853 lr 6.00e-04 gnorm 0.11 54,716 tok/s peakVRAM 13.7GB
89
+ [18:19:46] step 1620/4768 loss 3.2858 lr 6.00e-04 gnorm 0.12 54,719 tok/s peakVRAM 13.7GB
90
+ [18:22:57] step 1640/4768 loss 3.2827 lr 6.00e-04 gnorm 0.12 54,718 tok/s peakVRAM 13.7GB
91
+ [18:26:09] step 1660/4768 loss 3.2839 lr 6.00e-04 gnorm 0.11 54,723 tok/s peakVRAM 13.7GB
92
+ [18:29:21] step 1680/4768 loss 3.2733 lr 6.00e-04 gnorm 0.11 54,717 tok/s peakVRAM 13.7GB
93
+ [18:32:32] step 1700/4768 loss 3.2658 lr 6.00e-04 gnorm 0.12 54,718 tok/s peakVRAM 13.7GB
94
+ [18:35:44] step 1720/4768 loss 3.2679 lr 6.00e-04 gnorm 0.12 54,712 tok/s peakVRAM 13.7GB
95
+ [18:38:56] step 1740/4768 loss 3.2618 lr 6.00e-04 gnorm 0.10 54,710 tok/s peakVRAM 13.7GB
96
+ [18:42:07] step 1760/4768 loss 3.2606 lr 6.00e-04 gnorm 0.11 54,712 tok/s peakVRAM 13.7GB
97
+ [18:45:19] step 1780/4768 loss 3.2480 lr 6.00e-04 gnorm 0.11 54,715 tok/s peakVRAM 13.7GB
98
+ [18:48:31] step 1800/4768 loss 3.2525 lr 6.00e-04 gnorm 0.12 54,716 tok/s peakVRAM 13.7GB
99
+ [18:51:42] step 1820/4768 loss 3.2466 lr 6.00e-04 gnorm 0.11 54,717 tok/s peakVRAM 13.7GB
100
+ [18:54:54] step 1840/4768 loss 3.2411 lr 6.00e-04 gnorm 0.10 54,712 tok/s peakVRAM 13.7GB
101
+ [18:58:06] step 1860/4768 loss 3.2404 lr 6.00e-04 gnorm 0.12 54,710 tok/s peakVRAM 13.7GB
102
+ [19:01:17] step 1880/4768 loss 3.2358 lr 6.00e-04 gnorm 0.11 54,707 tok/s peakVRAM 13.7GB
103
+ [19:04:29] step 1900/4768 loss 3.2309 lr 6.00e-04 gnorm 0.10 54,706 tok/s peakVRAM 13.7GB
104
+ [19:07:41] step 1920/4768 loss 3.2304 lr 6.00e-04 gnorm 0.10 54,706 tok/s peakVRAM 13.7GB
105
+ [19:10:52] step 1940/4768 loss 3.2230 lr 6.00e-04 gnorm 0.12 54,710 tok/s peakVRAM 13.7GB
106
+ [19:14:04] step 1960/4768 loss 3.2270 lr 6.00e-04 gnorm 0.11 54,711 tok/s peakVRAM 13.7GB
107
+ [19:17:16] step 1980/4768 loss 3.2220 lr 6.00e-04 gnorm 0.09 54,714 tok/s peakVRAM 13.7GB
108
+ [19:20:27] step 2000/4768 loss 3.2153 lr 6.00e-04 gnorm 0.10 54,722 tok/s peakVRAM 13.7GB
109
+ [19:23:39] step 2020/4768 loss 3.2152 lr 6.00e-04 gnorm 0.10 54,720 tok/s peakVRAM 13.7GB
110
+ [19:26:50] step 2040/4768 loss 3.2077 lr 6.00e-04 gnorm 0.11 54,721 tok/s peakVRAM 13.7GB
111
+ [19:30:02] step 2060/4768 loss 3.2078 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
112
+ [19:33:14] step 2080/4768 loss 3.1967 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
113
+ [19:36:25] step 2100/4768 loss 3.2051 lr 6.00e-04 gnorm 0.11 54,718 tok/s peakVRAM 13.7GB
114
+ [19:39:37] step 2120/4768 loss 3.1951 lr 6.00e-04 gnorm 0.11 54,717 tok/s peakVRAM 13.7GB
115
+ [19:42:49] step 2140/4768 loss 3.2076 lr 6.00e-04 gnorm 0.11 54,717 tok/s peakVRAM 13.7GB
116
+ [19:46:00] step 2160/4768 loss 3.1865 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
117
+ [19:49:12] step 2180/4768 loss 3.1881 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
118
+ [19:52:24] step 2200/4768 loss 3.1892 lr 6.00e-04 gnorm 0.10 54,711 tok/s peakVRAM 13.7GB
119
+ [19:55:35] step 2220/4768 loss 3.1820 lr 6.00e-04 gnorm 0.10 54,709 tok/s peakVRAM 13.7GB
120
+ [19:58:47] step 2240/4768 loss 3.1787 lr 6.00e-04 gnorm 0.09 54,712 tok/s peakVRAM 13.7GB
121
+ [20:01:58] step 2260/4768 loss 3.1754 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
122
+ [20:05:10] step 2280/4768 loss 3.1711 lr 6.00e-04 gnorm 0.11 54,712 tok/s peakVRAM 13.7GB
123
+ [20:08:22] step 2300/4768 loss 3.1747 lr 6.00e-04 gnorm 0.09 54,716 tok/s peakVRAM 13.7GB
124
+ [20:11:33] step 2320/4768 loss 3.1717 lr 6.00e-04 gnorm 0.11 54,718 tok/s peakVRAM 13.7GB
125
+ [20:14:45] step 2340/4768 loss 3.1667 lr 6.00e-04 gnorm 0.11 54,715 tok/s peakVRAM 13.7GB
126
+ [20:17:57] step 2360/4768 loss 3.1573 lr 6.00e-04 gnorm 0.10 54,718 tok/s peakVRAM 13.7GB
127
+ [20:21:08] step 2380/4768 loss 3.1559 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
128
+ [20:24:20] step 2400/4768 loss 3.1566 lr 6.00e-04 gnorm 0.09 54,718 tok/s peakVRAM 13.7GB
129
+ [20:27:32] step 2420/4768 loss 3.1597 lr 6.00e-04 gnorm 0.10 54,720 tok/s peakVRAM 13.7GB
130
+ [20:30:43] step 2440/4768 loss 3.1548 lr 6.00e-04 gnorm 0.09 54,721 tok/s peakVRAM 13.7GB
131
+ [20:33:55] step 2460/4768 loss 3.1532 lr 6.00e-04 gnorm 0.11 54,719 tok/s peakVRAM 13.7GB
132
+ [20:37:06] step 2480/4768 loss 3.1504 lr 6.00e-04 gnorm 0.11 54,717 tok/s peakVRAM 13.7GB
133
+ [20:40:18] step 2500/4768 loss 3.1447 lr 6.00e-04 gnorm 0.10 54,716 tok/s peakVRAM 13.7GB
134
+ [20:43:30] step 2520/4768 loss 3.1466 lr 6.00e-04 gnorm 0.10 54,719 tok/s peakVRAM 13.7GB
135
+ [20:46:41] step 2540/4768 loss 3.1465 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
136
+ [20:49:53] step 2560/4768 loss 3.1434 lr 6.00e-04 gnorm 0.11 54,719 tok/s peakVRAM 13.7GB
137
+ [20:53:05] step 2580/4768 loss 3.1385 lr 6.00e-04 gnorm 0.10 54,719 tok/s peakVRAM 13.7GB
138
+ [20:56:16] step 2600/4768 loss 3.1294 lr 6.00e-04 gnorm 0.10 54,716 tok/s peakVRAM 13.7GB
139
+ [20:59:28] step 2620/4768 loss 3.1375 lr 6.00e-04 gnorm 0.09 54,712 tok/s peakVRAM 13.7GB
140
+ [21:02:40] step 2640/4768 loss 3.1389 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
141
+ [21:05:51] step 2660/4768 loss 3.1372 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
142
+ [21:09:03] step 2680/4768 loss 3.1326 lr 6.00e-04 gnorm 0.10 54,716 tok/s peakVRAM 13.7GB
143
+ [21:12:14] step 2700/4768 loss 3.1352 lr 6.00e-04 gnorm 0.10 54,718 tok/s peakVRAM 13.7GB
144
+ [21:15:26] step 2720/4768 loss 3.1180 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
145
+ [21:18:38] step 2740/4768 loss 3.1193 lr 6.00e-04 gnorm 0.10 54,723 tok/s peakVRAM 13.7GB
146
+ [21:21:49] step 2760/4768 loss 3.1220 lr 6.00e-04 gnorm 0.10 54,724 tok/s peakVRAM 13.7GB
147
+ [21:25:01] step 2780/4768 loss 3.1132 lr 6.00e-04 gnorm 0.10 54,722 tok/s peakVRAM 13.7GB
148
+ [21:28:13] step 2800/4768 loss 3.1081 lr 6.00e-04 gnorm 0.10 54,720 tok/s peakVRAM 13.7GB
149
+ [21:31:24] step 2820/4768 loss 3.1058 lr 6.00e-04 gnorm 0.10 54,719 tok/s peakVRAM 13.7GB
150
+ [21:34:36] step 2840/4768 loss 3.1134 lr 6.00e-04 gnorm 0.10 54,719 tok/s peakVRAM 13.7GB
151
+ [21:37:47] step 2860/4768 loss 3.1055 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
152
+ [21:40:59] step 2880/4768 loss 3.1071 lr 6.00e-04 gnorm 0.09 54,714 tok/s peakVRAM 13.7GB
153
+ [21:44:11] step 2900/4768 loss 3.1088 lr 6.00e-04 gnorm 0.11 54,713 tok/s peakVRAM 13.7GB
154
+ [21:47:22] step 2920/4768 loss 3.1159 lr 6.00e-04 gnorm 0.10 54,718 tok/s peakVRAM 13.7GB
155
+ [21:50:34] step 2940/4768 loss 3.1065 lr 6.00e-04 gnorm 0.08 54,720 tok/s peakVRAM 13.7GB
156
+ [21:53:46] step 2960/4768 loss 3.1085 lr 6.00e-04 gnorm 0.10 54,720 tok/s peakVRAM 13.7GB
157
+ [21:56:57] step 2980/4768 loss 3.0954 lr 6.00e-04 gnorm 0.10 54,721 tok/s peakVRAM 13.7GB
158
+ [22:00:09] step 3000/4768 loss 3.0957 lr 6.00e-04 gnorm 0.09 54,721 tok/s peakVRAM 13.7GB
159
+ [22:03:21] step 3020/4768 loss 3.0920 lr 6.00e-04 gnorm 0.09 54,721 tok/s peakVRAM 13.7GB
160
+ [22:06:32] step 3040/4768 loss 3.0889 lr 6.00e-04 gnorm 0.09 54,714 tok/s peakVRAM 13.7GB
161
+ [22:09:44] step 3060/4768 loss 3.0923 lr 6.00e-04 gnorm 0.09 54,718 tok/s peakVRAM 13.7GB
162
+ [22:12:55] step 3080/4768 loss 3.0847 lr 6.00e-04 gnorm 0.09 54,720 tok/s peakVRAM 13.7GB
163
+ [22:16:07] step 3100/4768 loss 3.0786 lr 6.00e-04 gnorm 0.10 54,725 tok/s peakVRAM 13.7GB
164
+ [22:19:19] step 3120/4768 loss 3.0947 lr 6.00e-04 gnorm 0.10 54,721 tok/s peakVRAM 13.7GB
165
+ [22:22:30] step 3140/4768 loss 3.0798 lr 6.00e-04 gnorm 0.10 54,724 tok/s peakVRAM 13.7GB
166
+ [22:25:42] step 3160/4768 loss 3.0761 lr 6.00e-04 gnorm 0.09 54,722 tok/s peakVRAM 13.7GB
167
+ [22:28:54] step 3180/4768 loss 3.0823 lr 6.00e-04 gnorm 0.09 54,718 tok/s peakVRAM 13.7GB
168
+ [22:32:05] step 3200/4768 loss 3.0832 lr 6.00e-04 gnorm 0.09 54,720 tok/s peakVRAM 13.7GB
169
+ [22:35:17] step 3220/4768 loss 3.0926 lr 6.00e-04 gnorm 0.10 54,722 tok/s peakVRAM 13.7GB
170
+ [22:38:28] step 3240/4768 loss 3.0746 lr 6.00e-04 gnorm 0.09 54,719 tok/s peakVRAM 13.7GB
171
+ [22:41:40] step 3260/4768 loss 3.0712 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
172
+ [22:44:52] step 3280/4768 loss 3.0670 lr 6.00e-04 gnorm 0.09 54,719 tok/s peakVRAM 13.7GB
173
+ [22:48:03] step 3300/4768 loss 3.0780 lr 6.00e-04 gnorm 0.09 54,712 tok/s peakVRAM 13.7GB
174
+ [22:51:15] step 3320/4768 loss 3.0718 lr 6.00e-04 gnorm 0.08 54,713 tok/s peakVRAM 13.7GB
175
+ [22:54:27] step 3340/4768 loss 3.0722 lr 6.00e-04 gnorm 0.09 54,716 tok/s peakVRAM 13.7GB
176
+ [22:57:38] step 3360/4768 loss 3.0630 lr 6.00e-04 gnorm 0.10 54,718 tok/s peakVRAM 13.7GB
177
+ [23:00:50] step 3380/4768 loss 3.0728 lr 6.00e-04 gnorm 0.10 54,715 tok/s peakVRAM 13.7GB
178
+ [23:04:02] step 3400/4768 loss 3.0580 lr 6.00e-04 gnorm 0.09 54,719 tok/s peakVRAM 13.7GB
179
+ [23:07:13] step 3420/4768 loss 3.0660 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
180
+ [23:10:25] step 3440/4768 loss 3.0576 lr 6.00e-04 gnorm 0.10 54,716 tok/s peakVRAM 13.7GB
181
+ [23:13:36] step 3460/4768 loss 3.0667 lr 6.00e-04 gnorm 0.09 54,715 tok/s peakVRAM 13.7GB
182
+ [23:16:48] step 3480/4768 loss 3.0667 lr 6.00e-04 gnorm 0.09 54,716 tok/s peakVRAM 13.7GB
183
+ [23:20:00] step 3500/4768 loss 3.0526 lr 6.00e-04 gnorm 0.10 54,714 tok/s peakVRAM 13.7GB
184
+ [23:23:11] step 3520/4768 loss 3.0552 lr 6.00e-04 gnorm 0.10 54,717 tok/s peakVRAM 13.7GB
185
+ [23:26:23] step 3540/4768 loss 3.0578 lr 6.00e-04 gnorm 0.09 54,716 tok/s peakVRAM 13.7GB
186
+ [23:29:35] step 3560/4768 loss 3.0616 lr 6.00e-04 gnorm 0.09 54,717 tok/s peakVRAM 13.7GB
187
+ [23:32:46] step 3580/4768 loss 3.0558 lr 6.00e-04 gnorm 0.08 54,722 tok/s peakVRAM 13.7GB
188
+ [23:35:58] step 3600/4768 loss 3.0576 lr 6.00e-04 gnorm 0.10 54,721 tok/s peakVRAM 13.7GB
189
+ [23:39:10] step 3620/4768 loss 3.0497 lr 6.00e-04 gnorm 0.10 54,719 tok/s peakVRAM 13.7GB
190
+ [23:42:21] step 3640/4768 loss 3.0574 lr 6.00e-04 gnorm 0.09 54,722 tok/s peakVRAM 13.7GB
191
+ [23:45:33] step 3660/4768 loss 3.0462 lr 6.00e-04 gnorm 0.09 54,720 tok/s peakVRAM 13.7GB
192
+ [23:48:44] step 3680/4768 loss 3.0531 lr 6.00e-04 gnorm 0.09 54,724 tok/s peakVRAM 13.7GB
193
+ [23:51:56] step 3700/4768 loss 3.0503 lr 6.00e-04 gnorm 0.09 54,726 tok/s peakVRAM 13.7GB
194
+ [23:55:08] step 3720/4768 loss 3.0435 lr 6.00e-04 gnorm 0.08 54,724 tok/s peakVRAM 13.7GB
195
+ [23:58:19] step 3740/4768 loss 3.0400 lr 6.00e-04 gnorm 0.08 54,727 tok/s peakVRAM 13.7GB
196
+ [00:01:31] step 3760/4768 loss 3.0384 lr 6.00e-04 gnorm 0.09 54,725 tok/s peakVRAM 13.7GB
197
+ [00:04:42] step 3780/4768 loss 3.0390 lr 6.00e-04 gnorm 0.10 54,721 tok/s peakVRAM 13.7GB
198
+ [00:07:54] step 3800/4768 loss 3.0400 lr 6.00e-04 gnorm 0.09 54,718 tok/s peakVRAM 13.7GB
199
+ [00:11:06] step 3820/4768 loss 3.0331 lr 5.61e-04 gnorm 0.09 54,718 tok/s peakVRAM 13.7GB
200
+ [00:14:17] step 3840/4768 loss 3.0316 lr 5.13e-04 gnorm 0.08 54,717 tok/s peakVRAM 13.7GB
201
+ [00:17:29] step 3860/4768 loss 3.0245 lr 4.83e-04 gnorm 0.07 54,718 tok/s peakVRAM 13.7GB
202
+ [00:20:41] step 3880/4768 loss 3.0217 lr 4.59e-04 gnorm 0.08 54,721 tok/s peakVRAM 13.7GB
203
+ [00:23:52] step 3900/4768 loss 3.0150 lr 4.39e-04 gnorm 0.08 54,716 tok/s peakVRAM 13.7GB
204
+ [00:27:04] step 3920/4768 loss 3.0093 lr 4.21e-04 gnorm 0.07 54,715 tok/s peakVRAM 13.7GB
205
+ [00:30:16] step 3940/4768 loss 3.0019 lr 4.05e-04 gnorm 0.07 54,715 tok/s peakVRAM 13.7GB
206
+ [00:33:27] step 3960/4768 loss 3.0024 lr 3.89e-04 gnorm 0.08 54,716 tok/s peakVRAM 13.7GB
207
+ [00:36:39] step 3980/4768 loss 2.9945 lr 3.75e-04 gnorm 0.07 54,721 tok/s peakVRAM 13.7GB
208
+ [00:39:50] step 4000/4768 loss 2.9933 lr 3.62e-04 gnorm 0.07 54,722 tok/s peakVRAM 13.7GB
209
+ [00:43:02] step 4020/4768 loss 2.9906 lr 3.50e-04 gnorm 0.07 54,713 tok/s peakVRAM 13.7GB
210
+ [00:46:14] step 4040/4768 loss 2.9812 lr 3.38e-04 gnorm 0.07 54,714 tok/s peakVRAM 13.7GB
211
+ [00:49:25] step 4060/4768 loss 2.9809 lr 3.26e-04 gnorm 0.07 54,719 tok/s peakVRAM 13.7GB
212
+ [00:52:37] step 4080/4768 loss 2.9725 lr 3.15e-04 gnorm 0.07 54,715 tok/s peakVRAM 13.7GB
213
+ [00:55:49] step 4100/4768 loss 2.9776 lr 3.05e-04 gnorm 0.07 54,716 tok/s peakVRAM 13.7GB
214
+ [00:59:00] step 4120/4768 loss 2.9664 lr 2.95e-04 gnorm 0.07 54,720 tok/s peakVRAM 13.7GB
215
+ [01:02:12] step 4140/4768 loss 2.9586 lr 2.85e-04 gnorm 0.07 54,716 tok/s peakVRAM 13.7GB
216
+ [01:05:24] step 4160/4768 loss 2.9768 lr 2.75e-04 gnorm 0.07 54,710 tok/s peakVRAM 13.7GB
217
+ [01:08:35] step 4180/4768 loss 2.9694 lr 2.66e-04 gnorm 0.07 54,712 tok/s peakVRAM 13.7GB
218
+ [01:11:47] step 4200/4768 loss 2.9631 lr 2.57e-04 gnorm 0.07 54,715 tok/s peakVRAM 13.7GB
219
+ [01:14:58] step 4220/4768 loss 2.9561 lr 2.48e-04 gnorm 0.07 54,716 tok/s peakVRAM 13.7GB
220
+ [01:18:10] step 4240/4768 loss 2.9624 lr 2.40e-04 gnorm 0.07 54,718 tok/s peakVRAM 13.7GB
221
+ [01:21:22] step 4260/4768 loss 2.9546 lr 2.31e-04 gnorm 0.07 54,720 tok/s peakVRAM 13.7GB
222
+ [01:24:33] step 4280/4768 loss 2.9540 lr 2.23e-04 gnorm 0.07 54,725 tok/s peakVRAM 13.7GB
223
+ [01:27:45] step 4300/4768 loss 2.9518 lr 2.15e-04 gnorm 0.07 54,721 tok/s peakVRAM 13.7GB
224
+ [01:30:57] step 4320/4768 loss 2.9470 lr 2.07e-04 gnorm 0.07 54,718 tok/s peakVRAM 13.7GB
225
+ [01:34:08] step 4340/4768 loss 2.9457 lr 1.99e-04 gnorm 0.07 54,717 tok/s peakVRAM 13.7GB
226
+ [01:37:20] step 4360/4768 loss 2.9396 lr 1.92e-04 gnorm 0.07 54,717 tok/s peakVRAM 13.7GB
227
+ [01:40:32] step 4380/4768 loss 2.9465 lr 1.84e-04 gnorm 0.07 54,713 tok/s peakVRAM 13.7GB
228
+ [01:43:43] step 4400/4768 loss 2.9439 lr 1.77e-04 gnorm 0.07 54,711 tok/s peakVRAM 13.7GB
229
+ [01:46:55] step 4420/4768 loss 2.9371 lr 1.70e-04 gnorm 0.07 54,714 tok/s peakVRAM 13.7GB
230
+ [01:50:06] step 4440/4768 loss 2.9435 lr 1.63e-04 gnorm 0.07 54,713 tok/s peakVRAM 13.7GB
231
+ [01:53:18] step 4460/4768 loss 2.9390 lr 1.56e-04 gnorm 0.07 54,716 tok/s peakVRAM 13.7GB
232
+ [01:56:30] step 4480/4768 loss 2.9341 lr 1.49e-04 gnorm 0.07 54,719 tok/s peakVRAM 13.7GB
233
+ [01:59:41] step 4500/4768 loss 2.9313 lr 1.42e-04 gnorm 0.07 54,720 tok/s peakVRAM 13.7GB
234
+ [02:02:53] step 4520/4768 loss 2.9332 lr 1.36e-04 gnorm 0.07 54,717 tok/s peakVRAM 13.7GB
235
+ [02:06:05] step 4540/4768 loss 2.9246 lr 1.29e-04 gnorm 0.07 54,717 tok/s peakVRAM 13.7GB
236
+ [02:09:16] step 4560/4768 loss 2.9280 lr 1.23e-04 gnorm 0.07 54,712 tok/s peakVRAM 13.7GB
237
+ [02:12:28] step 4580/4768 loss 2.9309 lr 1.16e-04 gnorm 0.07 54,710 tok/s peakVRAM 13.7GB
238
+ [02:15:40] step 4600/4768 loss 2.9299 lr 1.10e-04 gnorm 0.07 54,708 tok/s peakVRAM 13.7GB
239
+ [02:18:51] step 4620/4768 loss 2.9163 lr 1.04e-04 gnorm 0.07 54,711 tok/s peakVRAM 13.7GB
240
+ [02:22:03] step 4640/4768 loss 2.9256 lr 9.78e-05 gnorm 0.07 54,714 tok/s peakVRAM 13.7GB
241
+ [02:25:15] step 4660/4768 loss 2.9184 lr 9.18e-05 gnorm 0.07 54,713 tok/s peakVRAM 13.7GB
242
+ [02:28:26] step 4680/4768 loss 2.9159 lr 8.58e-05 gnorm 0.07 54,711 tok/s peakVRAM 13.7GB
243
+ [02:31:38] step 4700/4768 loss 2.9121 lr 7.99e-05 gnorm 0.07 54,712 tok/s peakVRAM 13.7GB
244
+ [02:34:50] step 4720/4768 loss 2.9131 lr 7.41e-05 gnorm 0.07 54,715 tok/s peakVRAM 13.7GB
245
+ [02:38:01] step 4740/4768 loss 2.9110 lr 6.83e-05 gnorm 0.07 54,718 tok/s peakVRAM 13.7GB
246
+ [02:41:13] step 4760/4768 loss 2.9202 lr 6.26e-05 gnorm 0.07 54,720 tok/s peakVRAM 13.7GB
247
+ [02:42:31] ckpt @ 4768
248
+ [02:42:31] DONE