Hugging Face's logo

DelonSwartz
/
KOLM-Alpha

PyTorch
kuramoto
oscillator
language-model
synchronization-routing
tinystories
Model card Files
xet
 Community
KOLM-Alpha
File size: 10,995 Bytes
94eb60b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
transformer | 17,440,512 params | vocab 8192 | mps
step     1 | loss 9.0659 | gnorm 5.66 | 1s
step    25 | loss 8.4237 | gnorm 2.84 | 21s
step    50 | loss 8.0053 | gnorm 2.37 | 36s
step    75 | loss 7.5894 | gnorm 2.29 | 51s
step   100 | loss 7.2227 | gnorm 1.95 | 66s
step   125 | loss 6.5297 | gnorm 1.79 | 83s
step   150 | loss 6.0340 | gnorm 1.76 | 99s
step   175 | loss 5.3586 | gnorm 1.44 | 117s
step   200 | loss 5.1159 | gnorm 1.48 | 133s
step   225 | loss 5.0118 | gnorm 1.45 | 148s
step   250 | loss 4.6414 | gnorm 1.30 | 163s
  val 4.7656 | ppl 117.40 | 1M tokens
step   275 | loss 4.6810 | gnorm 1.50 | 182s
step   300 | loss 4.5977 | gnorm 1.46 | 197s
step   325 | loss 4.3595 | gnorm 1.41 | 212s
step   350 | loss 4.4616 | gnorm 1.52 | 228s
step   375 | loss 4.0337 | gnorm 1.43 | 244s
step   400 | loss 4.1460 | gnorm 1.46 | 259s
step   425 | loss 4.1583 | gnorm 1.43 | 275s
step   450 | loss 4.2407 | gnorm 1.49 | 290s
step   475 | loss 4.1234 | gnorm 1.59 | 305s
step   500 | loss 4.1939 | gnorm 1.50 | 320s
  val 4.0422 | ppl 56.95 | 1M tokens
step   525 | loss 3.9165 | gnorm 1.65 | 338s
step   550 | loss 4.0645 | gnorm 1.59 | 353s
step   575 | loss 3.9202 | gnorm 1.70 | 369s
step   600 | loss 3.7699 | gnorm 1.48 | 384s
step   625 | loss 3.6540 | gnorm 1.65 | 399s
step   650 | loss 3.8476 | gnorm 1.67 | 414s
step   675 | loss 3.7533 | gnorm 1.69 | 430s
step   700 | loss 4.0363 | gnorm 1.59 | 446s
step   725 | loss 3.8794 | gnorm 1.61 | 462s
step   750 | loss 3.7619 | gnorm 1.65 | 478s
  val 3.7400 | ppl 42.10 | 2M tokens
step   775 | loss 3.6882 | gnorm 1.73 | 498s
step   800 | loss 3.7385 | gnorm 1.74 | 515s
step   825 | loss 3.4740 | gnorm 1.78 | 531s
step   850 | loss 3.6808 | gnorm 1.80 | 547s
step   875 | loss 3.6482 | gnorm 1.77 | 562s
step   900 | loss 3.4743 | gnorm 1.75 | 577s
step   925 | loss 3.6641 | gnorm 1.71 | 593s
step   950 | loss 3.4985 | gnorm 1.95 | 608s
step   975 | loss 3.4978 | gnorm 1.86 | 623s
step  1000 | loss 3.6681 | gnorm 1.93 | 638s
  val 3.5336 | ppl 34.25 | 2M tokens
step  1025 | loss 3.3692 | gnorm 1.85 | 656s
step  1050 | loss 3.7098 | gnorm 1.80 | 674s
step  1075 | loss 3.4980 | gnorm 1.89 | 692s
step  1100 | loss 3.6340 | gnorm 1.88 | 708s
step  1125 | loss 3.4360 | gnorm 1.74 | 727s
step  1150 | loss 3.2896 | gnorm 1.88 | 744s
step  1175 | loss 3.6475 | gnorm 2.15 | 765s
step  1200 | loss 3.4324 | gnorm 2.00 | 784s
step  1225 | loss 3.3737 | gnorm 1.80 | 802s
step  1250 | loss 3.4515 | gnorm 1.92 | 818s
  val 3.3742 | ppl 29.20 | 3M tokens
step  1275 | loss 3.3330 | gnorm 1.96 | 837s
step  1300 | loss 3.2365 | gnorm 1.89 | 853s
step  1325 | loss 3.1443 | gnorm 1.79 | 869s
step  1350 | loss 3.0539 | gnorm 1.84 | 885s
step  1375 | loss 3.1282 | gnorm 1.84 | 901s
step  1400 | loss 3.3304 | gnorm 1.85 | 917s
step  1425 | loss 3.4560 | gnorm 2.02 | 934s
step  1450 | loss 3.0201 | gnorm 1.82 | 953s
step  1475 | loss 3.2094 | gnorm 1.87 | 972s
step  1500 | loss 3.1902 | gnorm 2.00 | 992s
  val 3.2581 | ppl 26.00 | 3M tokens
step  1525 | loss 3.4144 | gnorm 1.97 | 1012s
step  1550 | loss 2.9986 | gnorm 1.92 | 1027s
step  1575 | loss 3.1162 | gnorm 1.97 | 1045s
step  1600 | loss 2.9763 | gnorm 1.97 | 1061s
step  1625 | loss 3.2165 | gnorm 1.76 | 1076s
step  1650 | loss 3.2712 | gnorm 1.91 | 1090s
step  1675 | loss 3.0484 | gnorm 2.04 | 1107s
step  1700 | loss 3.1147 | gnorm 2.01 | 1122s
step  1725 | loss 3.0797 | gnorm 1.95 | 1140s
step  1750 | loss 3.1958 | gnorm 1.92 | 1155s
  val 3.1551 | ppl 23.46 | 4M tokens
step  1775 | loss 3.2327 | gnorm 1.99 | 1176s
step  1800 | loss 3.1617 | gnorm 2.09 | 1190s
step  1825 | loss 3.2977 | gnorm 1.95 | 1207s
step  1850 | loss 3.1577 | gnorm 2.26 | 1224s
step  1875 | loss 3.3915 | gnorm 2.02 | 1239s
step  1900 | loss 3.3305 | gnorm 1.93 | 1254s
step  1925 | loss 2.9534 | gnorm 2.16 | 1269s
step  1950 | loss 3.1981 | gnorm 2.04 | 1284s
step  1975 | loss 2.8632 | gnorm 1.96 | 1299s
step  2000 | loss 3.3176 | gnorm 2.07 | 1314s
  val 3.0857 | ppl 21.88 | 4M tokens
step  2025 | loss 2.8043 | gnorm 1.89 | 1332s
step  2050 | loss 2.7409 | gnorm 2.03 | 1347s
step  2075 | loss 3.0667 | gnorm 2.06 | 1363s
step  2100 | loss 3.2567 | gnorm 2.02 | 1377s
step  2125 | loss 3.1960 | gnorm 2.09 | 1392s
step  2150 | loss 2.9177 | gnorm 2.08 | 1407s
step  2175 | loss 3.0648 | gnorm 2.19 | 1422s
step  2200 | loss 3.1265 | gnorm 2.02 | 1438s
step  2225 | loss 3.0281 | gnorm 2.10 | 1453s
step  2250 | loss 3.0515 | gnorm 1.95 | 1468s
  val 3.0140 | ppl 20.37 | 5M tokens
step  2275 | loss 3.2566 | gnorm 2.13 | 1486s
step  2300 | loss 2.8999 | gnorm 2.08 | 1505s
step  2325 | loss 3.0517 | gnorm 2.23 | 1524s
step  2350 | loss 2.9837 | gnorm 2.06 | 1539s
step  2375 | loss 2.9317 | gnorm 2.04 | 1555s
step  2400 | loss 3.0361 | gnorm 2.26 | 1571s
step  2425 | loss 3.0551 | gnorm 2.24 | 1586s
step  2450 | loss 2.8550 | gnorm 2.21 | 1601s
step  2475 | loss 2.9036 | gnorm 2.09 | 1617s
step  2500 | loss 3.1739 | gnorm 2.19 | 1633s
  val 2.9613 | ppl 19.32 | 5M tokens
step  2525 | loss 2.7797 | gnorm 2.35 | 1652s
step  2550 | loss 2.8502 | gnorm 2.07 | 1668s
step  2575 | loss 3.1404 | gnorm 2.30 | 1683s
step  2600 | loss 2.8394 | gnorm 2.22 | 1700s
step  2625 | loss 2.8314 | gnorm 2.18 | 1717s
step  2650 | loss 3.0105 | gnorm 2.20 | 1733s
step  2675 | loss 2.9467 | gnorm 2.24 | 1748s
step  2700 | loss 3.0323 | gnorm 2.19 | 1764s
step  2725 | loss 2.7853 | gnorm 2.20 | 1779s
step  2750 | loss 3.0100 | gnorm 2.21 | 1794s
  val 2.9078 | ppl 18.32 | 6M tokens
step  2775 | loss 2.8640 | gnorm 2.03 | 1812s
step  2800 | loss 3.0415 | gnorm 2.24 | 1827s
step  2825 | loss 3.0081 | gnorm 2.26 | 1843s
step  2850 | loss 2.9286 | gnorm 2.36 | 1858s
step  2875 | loss 3.0645 | gnorm 2.04 | 1873s
step  2900 | loss 2.8814 | gnorm 2.10 | 1888s
step  2925 | loss 2.9102 | gnorm 2.16 | 1903s
step  2950 | loss 2.9170 | gnorm 2.10 | 1918s
step  2975 | loss 2.7310 | gnorm 2.13 | 1933s
step  3000 | loss 2.8980 | gnorm 2.19 | 1948s
  val 2.8735 | ppl 17.70 | 6M tokens
step  3025 | loss 3.1517 | gnorm 2.18 | 1966s
step  3050 | loss 2.8395 | gnorm 2.21 | 1982s
step  3075 | loss 2.6157 | gnorm 2.09 | 1997s
step  3100 | loss 2.7318 | gnorm 2.17 | 2012s
step  3125 | loss 2.7952 | gnorm 2.28 | 2027s
step  3150 | loss 2.8145 | gnorm 2.35 | 2042s
step  3175 | loss 3.0589 | gnorm 2.37 | 2057s
step  3200 | loss 3.0978 | gnorm 2.40 | 2072s
step  3225 | loss 3.0286 | gnorm 2.26 | 2087s
step  3250 | loss 2.9351 | gnorm 2.35 | 2102s
  val 2.8314 | ppl 16.97 | 7M tokens
step  3275 | loss 2.7162 | gnorm 2.53 | 2121s
step  3300 | loss 2.7243 | gnorm 2.38 | 2136s
step  3325 | loss 2.7972 | gnorm 2.46 | 2151s
step  3350 | loss 3.0039 | gnorm 2.37 | 2166s
step  3375 | loss 3.0588 | gnorm 2.30 | 2181s
step  3400 | loss 2.8216 | gnorm 2.14 | 2196s
step  3425 | loss 2.8228 | gnorm 2.26 | 2211s
step  3450 | loss 2.8083 | gnorm 2.28 | 2227s
step  3475 | loss 2.9651 | gnorm 2.26 | 2242s
step  3500 | loss 2.9724 | gnorm 2.36 | 2257s
  val 2.8072 | ppl 16.56 | 7M tokens
step  3525 | loss 2.9245 | gnorm 2.44 | 2276s
step  3550 | loss 2.8749 | gnorm 2.23 | 2291s
step  3575 | loss 2.6986 | gnorm 2.35 | 2306s
step  3600 | loss 2.5832 | gnorm 2.27 | 2321s
step  3625 | loss 2.6425 | gnorm 2.26 | 2336s
step  3650 | loss 2.7245 | gnorm 2.27 | 2351s
step  3675 | loss 2.5597 | gnorm 2.29 | 2366s
step  3700 | loss 2.9487 | gnorm 2.20 | 2381s
step  3725 | loss 2.8018 | gnorm 2.26 | 2396s
step  3750 | loss 2.8344 | gnorm 2.32 | 2411s
  val 2.7824 | ppl 16.16 | 8M tokens
step  3775 | loss 2.7982 | gnorm 2.31 | 2429s
step  3800 | loss 2.8159 | gnorm 2.21 | 2444s
step  3825 | loss 2.7541 | gnorm 2.27 | 2459s
step  3850 | loss 2.6004 | gnorm 2.32 | 2475s
step  3875 | loss 2.6492 | gnorm 2.30 | 2490s
step  3900 | loss 2.5851 | gnorm 2.19 | 2505s
step  3925 | loss 2.4567 | gnorm 2.39 | 2520s
step  3950 | loss 2.8217 | gnorm 2.25 | 2535s
step  3975 | loss 2.8198 | gnorm 2.31 | 2550s
step  4000 | loss 2.7746 | gnorm 2.45 | 2566s
  val 2.7633 | ppl 15.85 | 8M tokens
step  4025 | loss 2.9212 | gnorm 2.43 | 2584s
step  4050 | loss 2.6847 | gnorm 2.25 | 2599s
step  4075 | loss 2.8598 | gnorm 2.36 | 2615s
step  4100 | loss 2.7371 | gnorm 2.29 | 2630s
step  4125 | loss 2.7010 | gnorm 2.60 | 2645s
step  4150 | loss 2.6612 | gnorm 2.29 | 2660s
step  4175 | loss 2.7117 | gnorm 2.47 | 2676s
step  4200 | loss 2.7607 | gnorm 2.30 | 2691s
step  4225 | loss 2.8989 | gnorm 2.57 | 2706s
step  4250 | loss 2.8335 | gnorm 2.48 | 2721s
  val 2.7466 | ppl 15.59 | 9M tokens
step  4275 | loss 2.7723 | gnorm 2.65 | 2740s
step  4300 | loss 2.6089 | gnorm 2.32 | 2755s
step  4325 | loss 2.6483 | gnorm 2.29 | 2770s
step  4350 | loss 2.5808 | gnorm 2.36 | 2785s
step  4375 | loss 3.1040 | gnorm 2.95 | 2800s
step  4400 | loss 2.9378 | gnorm 2.48 | 2815s
step  4425 | loss 2.8229 | gnorm 2.37 | 2830s
step  4450 | loss 2.8420 | gnorm 2.31 | 2846s
step  4475 | loss 2.4412 | gnorm 2.31 | 2861s
step  4500 | loss 2.5466 | gnorm 2.40 | 2877s
  val 2.7340 | ppl 15.39 | 9M tokens
step  4525 | loss 2.6072 | gnorm 2.36 | 2895s
step  4550 | loss 2.6989 | gnorm 2.46 | 2911s
step  4575 | loss 2.7203 | gnorm 2.41 | 2927s
step  4600 | loss 2.7147 | gnorm 2.43 | 2944s
step  4625 | loss 2.8740 | gnorm 2.34 | 2960s
step  4650 | loss 2.7108 | gnorm 2.37 | 2977s
step  4675 | loss 3.1500 | gnorm 2.43 | 2993s
step  4700 | loss 2.7334 | gnorm 2.59 | 3008s
step  4725 | loss 2.8229 | gnorm 2.41 | 3023s
step  4750 | loss 2.6176 | gnorm 2.33 | 3038s
  val 2.7261 | ppl 15.27 | 10M tokens
step  4775 | loss 2.6204 | gnorm 2.28 | 3056s
step  4800 | loss 2.9064 | gnorm 2.38 | 3071s
step  4825 | loss 2.6115 | gnorm 2.50 | 3086s
step  4850 | loss 2.8141 | gnorm 2.48 | 3101s
step  4875 | loss 2.9632 | gnorm 2.51 | 3116s
step  4900 | loss 2.5242 | gnorm 2.53 | 3131s
step  4925 | loss 3.0410 | gnorm 2.56 | 3146s
step  4950 | loss 2.6725 | gnorm 2.47 | 3161s
step  4975 | loss 2.6360 | gnorm 2.42 | 3176s
step  5000 | loss 2.8666 | gnorm 2.64 | 3191s
  val 2.7214 | ppl 15.20 | 10M tokens
step  5025 | loss 2.7423 | gnorm 2.26 | 3209s
step  5050 | loss 2.6553 | gnorm 2.43 | 3224s
step  5075 | loss 2.6794 | gnorm 2.35 | 3239s
step  5100 | loss 2.7391 | gnorm 2.38 | 3254s
step  5125 | loss 2.9630 | gnorm 2.35 | 3270s
step  5150 | loss 2.8359 | gnorm 2.27 | 3285s
step  5175 | loss 2.7692 | gnorm 2.51 | 3301s
step  5200 | loss 2.7218 | gnorm 2.24 | 3316s
step  5225 | loss 2.6277 | gnorm 2.44 | 3332s
step  5250 | loss 3.1225 | gnorm 2.51 | 3347s
  val 2.7194 | ppl 15.17 | 11M tokens
step  5275 | loss 2.8204 | gnorm 2.35 | 3367s
step  5300 | loss 2.8327 | gnorm 2.34 | 3384s
step  5325 | loss 2.5840 | gnorm 2.23 | 3399s
step  5350 | loss 2.7542 | gnorm 2.36 | 3414s
step  5375 | loss 2.7365 | gnorm 2.32 | 3428s
step  5400 | loss 2.8519 | gnorm 2.33 | 3443s
step  5425 | loss 2.8636 | gnorm 2.54 | 3457s
step  5450 | loss 2.6134 | gnorm 2.36 | 3471s
step  5475 | loss 2.7233 | gnorm 2.65 | 3485s
step  5500 | loss 2.4590 | gnorm 2.47 | 3499s
  val 2.7188 | ppl 15.16 | 11M tokens
saved native_transformer.pt