Hugging Face's logo

DelonSwartz
/
KOLM-Alpha

PyTorch
kuramoto
oscillator
language-model
synchronization-routing
tinystories
Model card Files
xet
 Community
KOLM-Alpha
File size: 11,157 Bytes
5967f1e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
kolm | 16,916,224 params | vocab 8192 | mps
step     1 | loss 9.0707 | gnorm 14.92 | 5s
step    25 | loss 8.2714 | gnorm 3.12 | 117s
step    50 | loss 7.9729 | gnorm 2.38 | 215s
step    75 | loss 7.5448 | gnorm 2.27 | 315s
step   100 | loss 7.1723 | gnorm 1.85 | 414s
step   125 | loss 6.5443 | gnorm 1.72 | 515s
step   150 | loss 6.0590 | gnorm 1.51 | 616s
step   175 | loss 5.4919 | gnorm 2.04 | 713s
step   200 | loss 5.2394 | gnorm 1.88 | 806s
step   225 | loss 5.1267 | gnorm 1.57 | 898s
step   250 | loss 4.7845 | gnorm 1.39 | 992s
  val 4.8859 | ppl 132.41 | 1M tokens
step   275 | loss 4.8013 | gnorm 1.88 | 1113s
step   300 | loss 4.7008 | gnorm 1.89 | 1205s
step   325 | loss 4.4304 | gnorm 1.63 | 1298s
step   350 | loss 4.5788 | gnorm 1.77 | 1390s
step   375 | loss 4.1259 | gnorm 1.79 | 1483s
step   400 | loss 4.2311 | gnorm 1.64 | 1575s
step   425 | loss 4.2352 | gnorm 1.75 | 1668s
step   450 | loss 4.2986 | gnorm 1.79 | 1763s
step   475 | loss 4.1709 | gnorm 1.89 | 1855s
step   500 | loss 4.2580 | gnorm 1.78 | 1949s
  val 4.1100 | ppl 60.94 | 1M tokens
step   525 | loss 3.9729 | gnorm 1.93 | 2070s
step   550 | loss 4.1246 | gnorm 1.88 | 2178s
step   575 | loss 3.9847 | gnorm 2.04 | 2297s
step   600 | loss 3.8311 | gnorm 1.77 | 2401s
step   625 | loss 3.7055 | gnorm 1.96 | 2505s
step   650 | loss 3.8843 | gnorm 2.09 | 2605s
step   675 | loss 3.7934 | gnorm 2.10 | 2709s
step   700 | loss 4.0944 | gnorm 2.24 | 2812s
step   725 | loss 3.9397 | gnorm 2.18 | 2920s
step   750 | loss 3.8257 | gnorm 2.13 | 3016s
  val 3.7888 | ppl 44.20 | 2M tokens
step   775 | loss 3.7182 | gnorm 2.08 | 3138s
step   800 | loss 3.7808 | gnorm 2.28 | 3232s
step   825 | loss 3.5318 | gnorm 2.44 | 3326s
step   850 | loss 3.7255 | gnorm 2.02 | 3419s
step   875 | loss 3.7104 | gnorm 2.21 | 3515s
step   900 | loss 3.5115 | gnorm 2.11 | 3608s
step   925 | loss 3.7161 | gnorm 2.22 | 3701s
step   950 | loss 3.5390 | gnorm 2.31 | 3794s
step   975 | loss 3.5380 | gnorm 2.32 | 3887s
step  1000 | loss 3.6917 | gnorm 2.13 | 3980s
  val 3.5668 | ppl 35.40 | 2M tokens
step  1025 | loss 3.3968 | gnorm 2.10 | 4104s
step  1050 | loss 3.7517 | gnorm 2.33 | 4198s
step  1075 | loss 3.5378 | gnorm 2.49 | 4291s
step  1100 | loss 3.6530 | gnorm 2.24 | 4384s
step  1125 | loss 3.4710 | gnorm 2.36 | 4477s
step  1150 | loss 3.3378 | gnorm 2.55 | 4571s
step  1175 | loss 3.6750 | gnorm 2.40 | 4664s
step  1200 | loss 3.4526 | gnorm 2.60 | 4759s
step  1225 | loss 3.4160 | gnorm 2.10 | 4854s
step  1250 | loss 3.4874 | gnorm 2.24 | 4947s
  val 3.4039 | ppl 30.08 | 3M tokens
step  1275 | loss 3.3618 | gnorm 2.46 | 5069s
step  1300 | loss 3.2796 | gnorm 2.38 | 5166s
step  1325 | loss 3.1655 | gnorm 2.30 | 5259s
step  1350 | loss 3.0861 | gnorm 2.31 | 5354s
step  1375 | loss 3.1603 | gnorm 2.40 | 5449s
step  1400 | loss 3.3567 | gnorm 2.51 | 5542s
step  1425 | loss 3.4282 | gnorm 2.39 | 5635s
step  1450 | loss 3.0577 | gnorm 2.39 | 5728s
step  1475 | loss 3.2619 | gnorm 2.62 | 5821s
step  1500 | loss 3.2249 | gnorm 2.43 | 5915s
  val 3.2811 | ppl 26.61 | 3M tokens
step  1525 | loss 3.4037 | gnorm 2.31 | 6037s
step  1550 | loss 3.0126 | gnorm 2.32 | 6131s
step  1575 | loss 3.1633 | gnorm 2.59 | 6225s
step  1600 | loss 3.0170 | gnorm 2.43 | 6319s
step  1625 | loss 3.2520 | gnorm 2.37 | 6412s
step  1650 | loss 3.2816 | gnorm 2.40 | 6506s
step  1675 | loss 3.0603 | gnorm 2.63 | 6599s
step  1700 | loss 3.1222 | gnorm 2.66 | 6692s
step  1725 | loss 3.0850 | gnorm 2.44 | 6786s
step  1750 | loss 3.2087 | gnorm 2.44 | 6879s
  val 3.1717 | ppl 23.85 | 4M tokens
step  1775 | loss 3.2430 | gnorm 2.31 | 7002s
step  1800 | loss 3.1905 | gnorm 2.68 | 7096s
step  1825 | loss 3.3060 | gnorm 2.72 | 7191s
step  1850 | loss 3.1687 | gnorm 2.46 | 7284s
step  1875 | loss 3.3862 | gnorm 2.50 | 7377s
step  1900 | loss 3.3346 | gnorm 2.36 | 7470s
step  1925 | loss 2.9384 | gnorm 2.60 | 7563s
step  1950 | loss 3.2170 | gnorm 2.59 | 7657s
step  1975 | loss 2.8825 | gnorm 2.63 | 7751s
step  2000 | loss 3.3292 | gnorm 2.88 | 7844s
  val 3.1003 | ppl 22.20 | 4M tokens
step  2025 | loss 2.8098 | gnorm 2.31 | 7966s
step  2050 | loss 2.7519 | gnorm 2.46 | 8060s
step  2075 | loss 3.0944 | gnorm 2.75 | 8155s
step  2100 | loss 3.2591 | gnorm 2.77 | 8248s
step  2125 | loss 3.2165 | gnorm 2.56 | 8345s
step  2150 | loss 2.9171 | gnorm 2.70 | 8439s
step  2175 | loss 3.0677 | gnorm 2.56 | 8536s
step  2200 | loss 3.1405 | gnorm 2.64 | 8632s
step  2225 | loss 3.0420 | gnorm 2.69 | 8727s
step  2250 | loss 3.0561 | gnorm 2.64 | 8820s
  val 3.0273 | ppl 20.64 | 5M tokens
step  2275 | loss 3.2381 | gnorm 2.66 | 8944s
step  2300 | loss 2.9071 | gnorm 2.61 | 9039s
step  2325 | loss 3.0277 | gnorm 2.59 | 9134s
step  2350 | loss 2.9954 | gnorm 2.65 | 9229s
step  2375 | loss 2.9762 | gnorm 2.77 | 9325s
step  2400 | loss 3.0373 | gnorm 2.75 | 9418s
step  2425 | loss 3.0645 | gnorm 2.61 | 9512s
step  2450 | loss 2.8629 | gnorm 2.58 | 9608s
step  2475 | loss 2.8987 | gnorm 2.56 | 9723s
step  2500 | loss 3.1771 | gnorm 2.67 | 9823s
  val 2.9779 | ppl 19.65 | 5M tokens
step  2525 | loss 2.7999 | gnorm 2.63 | 9947s
step  2550 | loss 2.8363 | gnorm 2.58 | 10041s
step  2575 | loss 3.1341 | gnorm 2.98 | 10136s
step  2600 | loss 2.8335 | gnorm 2.90 | 10230s
step  2625 | loss 2.8166 | gnorm 2.70 | 10325s
step  2650 | loss 3.0086 | gnorm 2.80 | 10420s
step  2675 | loss 2.9273 | gnorm 2.79 | 10515s
step  2700 | loss 3.0408 | gnorm 2.83 | 10608s
step  2725 | loss 2.7906 | gnorm 2.91 | 10703s
step  2750 | loss 3.0174 | gnorm 2.93 | 10797s
  val 2.9121 | ppl 18.40 | 6M tokens
step  2775 | loss 2.8560 | gnorm 2.52 | 10921s
step  2800 | loss 3.0436 | gnorm 2.99 | 11015s
step  2825 | loss 3.0029 | gnorm 2.78 | 11108s
step  2850 | loss 2.9033 | gnorm 2.99 | 11203s
step  2875 | loss 3.0680 | gnorm 2.73 | 11300s
step  2900 | loss 2.8864 | gnorm 2.78 | 11393s
step  2925 | loss 2.9173 | gnorm 2.81 | 11487s
step  2950 | loss 2.9114 | gnorm 2.73 | 11582s
step  2975 | loss 2.7408 | gnorm 2.83 | 11675s
step  3000 | loss 2.9040 | gnorm 2.85 | 11770s
  val 2.8773 | ppl 17.77 | 6M tokens
step  3025 | loss 3.1609 | gnorm 3.00 | 11895s
step  3050 | loss 2.8313 | gnorm 2.93 | 11991s
step  3075 | loss 2.6176 | gnorm 3.05 | 12085s
step  3100 | loss 2.7470 | gnorm 3.09 | 12178s
step  3125 | loss 2.8237 | gnorm 3.06 | 12273s
step  3150 | loss 2.7972 | gnorm 2.97 | 12366s
step  3175 | loss 3.0566 | gnorm 3.05 | 12460s
step  3200 | loss 3.0810 | gnorm 3.08 | 12555s
step  3225 | loss 3.0267 | gnorm 2.95 | 12649s
step  3250 | loss 2.9154 | gnorm 3.09 | 12745s
  val 2.8339 | ppl 17.01 | 7M tokens
step  3275 | loss 2.7173 | gnorm 3.13 | 12867s
step  3300 | loss 2.7352 | gnorm 2.87 | 12962s
step  3325 | loss 2.7853 | gnorm 3.23 | 13056s
step  3350 | loss 3.0113 | gnorm 3.17 | 13152s
step  3375 | loss 3.0887 | gnorm 3.13 | 13249s
step  3400 | loss 2.8065 | gnorm 3.02 | 13344s
step  3425 | loss 2.8143 | gnorm 2.97 | 13439s
step  3450 | loss 2.8024 | gnorm 3.06 | 13534s
step  3475 | loss 2.9455 | gnorm 3.09 | 13628s
step  3500 | loss 2.9508 | gnorm 3.17 | 13724s
  val 2.8008 | ppl 16.46 | 7M tokens
step  3525 | loss 2.9241 | gnorm 3.22 | 13849s
step  3550 | loss 2.8766 | gnorm 3.07 | 13943s
step  3575 | loss 2.6811 | gnorm 3.14 | 14037s
step  3600 | loss 2.5598 | gnorm 2.88 | 14132s
step  3625 | loss 2.6413 | gnorm 3.08 | 14226s
step  3650 | loss 2.6940 | gnorm 3.05 | 14321s
step  3675 | loss 2.5285 | gnorm 3.18 | 14415s
step  3700 | loss 2.9373 | gnorm 3.06 | 14509s
step  3725 | loss 2.7980 | gnorm 3.18 | 14606s
step  3750 | loss 2.8113 | gnorm 3.06 | 14702s
  val 2.7743 | ppl 16.03 | 8M tokens
step  3775 | loss 2.8005 | gnorm 3.21 | 14825s
step  3800 | loss 2.8123 | gnorm 3.23 | 14919s
step  3825 | loss 2.7244 | gnorm 2.99 | 15012s
step  3850 | loss 2.6036 | gnorm 3.19 | 15107s
step  3875 | loss 2.6206 | gnorm 3.01 | 15201s
step  3900 | loss 2.5713 | gnorm 3.03 | 15315s
step  3925 | loss 2.4534 | gnorm 3.24 | 15412s
step  3950 | loss 2.8260 | gnorm 3.17 | 15507s
step  3975 | loss 2.8093 | gnorm 3.11 | 15602s
step  4000 | loss 2.7496 | gnorm 3.18 | 15697s
  val 2.7499 | ppl 15.64 | 8M tokens
step  4025 | loss 2.9053 | gnorm 3.24 | 15819s
step  4050 | loss 2.6918 | gnorm 3.02 | 15912s
step  4075 | loss 2.8410 | gnorm 3.26 | 16012s
step  4100 | loss 2.7221 | gnorm 2.95 | 16120s
step  4125 | loss 2.6604 | gnorm 3.27 | 16219s
step  4150 | loss 2.6568 | gnorm 3.22 | 16325s
step  4175 | loss 2.7244 | gnorm 3.41 | 16440s
step  4200 | loss 2.7475 | gnorm 3.21 | 16546s
step  4225 | loss 2.8774 | gnorm 3.32 | 16642s
step  4250 | loss 2.8362 | gnorm 3.35 | 16739s
  val 2.7310 | ppl 15.35 | 9M tokens
step  4275 | loss 2.7458 | gnorm 3.41 | 16865s
step  4300 | loss 2.6054 | gnorm 3.30 | 16961s
step  4325 | loss 2.6604 | gnorm 3.29 | 17057s
step  4350 | loss 2.5408 | gnorm 3.12 | 17152s
step  4375 | loss 3.0791 | gnorm 3.80 | 17246s
step  4400 | loss 2.8963 | gnorm 3.23 | 17342s
step  4425 | loss 2.7916 | gnorm 3.23 | 17448s
step  4450 | loss 2.8221 | gnorm 3.23 | 17547s
step  4475 | loss 2.4218 | gnorm 3.16 | 17641s
step  4500 | loss 2.5457 | gnorm 3.34 | 17736s
  val 2.7140 | ppl 15.09 | 9M tokens
step  4525 | loss 2.5867 | gnorm 3.32 | 17858s
step  4550 | loss 2.6694 | gnorm 3.33 | 17954s
step  4575 | loss 2.6873 | gnorm 3.40 | 18049s
step  4600 | loss 2.6940 | gnorm 3.30 | 18143s
step  4625 | loss 2.8509 | gnorm 3.33 | 18237s
step  4650 | loss 2.6937 | gnorm 3.37 | 18331s
step  4675 | loss 3.1286 | gnorm 3.44 | 18425s
step  4700 | loss 2.6899 | gnorm 3.41 | 18521s
step  4725 | loss 2.7882 | gnorm 3.41 | 18615s
step  4750 | loss 2.5785 | gnorm 3.22 | 18709s
  val 2.7055 | ppl 14.96 | 10M tokens
step  4775 | loss 2.5895 | gnorm 3.22 | 18830s
step  4800 | loss 2.8947 | gnorm 3.45 | 18922s
step  4825 | loss 2.5815 | gnorm 3.48 | 19017s
step  4850 | loss 2.7833 | gnorm 3.45 | 19115s
step  4875 | loss 2.9061 | gnorm 3.32 | 19219s
step  4900 | loss 2.4818 | gnorm 3.57 | 19314s
step  4925 | loss 3.0002 | gnorm 3.48 | 19408s
step  4950 | loss 2.6376 | gnorm 3.23 | 19520s
step  4975 | loss 2.6142 | gnorm 3.37 | 19618s
step  5000 | loss 2.8309 | gnorm 3.54 | 19717s
  val 2.6979 | ppl 14.85 | 10M tokens
step  5025 | loss 2.7186 | gnorm 3.29 | 19846s
step  5050 | loss 2.6430 | gnorm 3.65 | 24809s
step  5075 | loss 2.6572 | gnorm 3.23 | 25039s
step  5100 | loss 2.7021 | gnorm 3.35 | 25219s
step  5125 | loss 2.9417 | gnorm 3.29 | 25303s
step  5150 | loss 2.8187 | gnorm 3.22 | 25386s
step  5175 | loss 2.7659 | gnorm 3.67 | 25472s
step  5200 | loss 2.6885 | gnorm 3.25 | 25571s
step  5225 | loss 2.6109 | gnorm 3.32 | 25687s
step  5250 | loss 3.0765 | gnorm 3.43 | 25789s
  val 2.6950 | ppl 14.81 | 11M tokens
step  5275 | loss 2.7892 | gnorm 3.44 | 25918s
step  5300 | loss 2.8057 | gnorm 3.15 | 26007s
step  5325 | loss 2.5539 | gnorm 3.16 | 26097s
step  5350 | loss 2.7243 | gnorm 3.31 | 26180s
step  5375 | loss 2.7055 | gnorm 3.33 | 26295s
step  5400 | loss 2.8173 | gnorm 3.31 | 26385s
step  5425 | loss 2.8325 | gnorm 3.47 | 26482s
step  5450 | loss 2.5946 | gnorm 3.36 | 26576s
step  5475 | loss 2.6821 | gnorm 3.44 | 26670s
step  5500 | loss 2.4097 | gnorm 3.38 | 26770s
  val 2.6946 | ppl 14.80 | 11M tokens
saved native_kolm.pt