Hugging Face's logo

shikhar007
/
parameter-golf-gram-ns

Model card Files
xet
 Community
shikhar007 commited on
Commit
3a849f4
·
verified ·
1 Parent(s): 7763b43

Upload pr1493_wd_paired_s42 train log

Browse files
Files changed (1) hide show
  1. logs/pr1493_wd_paired_s42.txt +427 -0
logs/pr1493_wd_paired_s42.txt ADDED
@@ -0,0 +1,427 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ====================================================================================================
2
+ Hyperparameters:
3
+ adam_eps: 1e-08
4
+ adam_wd: 0.02
5
+ beta1: 0.9
6
+ beta2: 0.95
7
+ compressor: brotli
8
+ data_dir: ./data/
9
+ datasets_dir: ./data/datasets/fineweb10B_sp8192
10
+ distributed: True
11
+ doc_shuffle_bos_id: -1
12
+ doc_shuffle_enabled: False
13
+ ema_decay: 0.9965
14
+ embed_bits: 8
15
+ embed_clip_sigmas: 20.0
16
+ embed_lr: 0.6
17
+ embed_wd: 0.085
18
+ embedding_dim: 512
19
+ enable_looping_at: 0.35
20
+ etlb_clip: 3.0
21
+ etlb_enabled: False
22
+ etlb_lr: 0.05
23
+ etlb_steps: 5
24
+ eval_num_loops: -1
25
+ eval_seq_len: 2048
26
+ eval_stride: 64
27
+ gptq_calibration_batches: 64
28
+ gptq_reserve_seconds: 12.0
29
+ grad_accum_steps: 1
30
+ grad_clip_norm: 0.3
31
+ head_lr: 0.008
32
+ iha_enabled: False
33
+ iha_mix_v: False
34
+ is_main_process: True
35
+ iterations: 20000
36
+ ln_scale: True
37
+ local_rank: 0
38
+ logfile: logs/pr1493_wd_paired_s42.txt
39
+ logit_softcap: 30.0
40
+ loop_end: 5
41
+ loop_start: 3
42
+ matrix_bits: 6
43
+ matrix_clip_sigmas: 12.85
44
+ matrix_lr: 0.022
45
+ max_wallclock_seconds: 600.0
46
+ min_lr: 0.0
47
+ mlp_mult: 4.0
48
+ model_dim: 512
49
+ model_path: final_model.pt
50
+ mtp_steps: 0
51
+ mtp_weight: 0.0
52
+ muon_backend_steps: 5
53
+ muon_beta2: 0.95
54
+ muon_momentum: 0.99
55
+ muon_momentum_warmup_start: 0.92
56
+ muon_momentum_warmup_steps: 1500
57
+ muon_row_normalize: True
58
+ muon_wd: 0.095
59
+ num_heads: 8
60
+ num_kv_heads: 4
61
+ num_layers: 11
62
+ num_loops: 2
63
+ paired_head_muon_enabled: True
64
+ parallel_residual_start: 7
65
+ qk_gain_init: 5.25
66
+ quantized_model_path: final_model.int6.ptz
67
+ rank: 0
68
+ rope_base: 10000.0
69
+ rope_dims: 16
70
+ rope_train_seq_len: 2048
71
+ run_id: pr1493_wd_paired_s42
72
+ scalar_lr: 0.02
73
+ seed: 42
74
+ skip_gates_enabled: True
75
+ sliding_window_enabled: True
76
+ tie_embeddings: True
77
+ tied_embed_init_std: 0.005
78
+ tied_embed_lr: 0.03
79
+ tokenizer_path: ./data/tokenizers/fineweb_8192_bpe.model
80
+ train_batch_tokens: 786432
81
+ train_files: ./data/datasets/fineweb10B_sp8192/fineweb_train_*.bin
82
+ train_log_every: 500
83
+ train_seq_len: 2048
84
+ ttt_chunk_tokens: 32768
85
+ ttt_enabled: True
86
+ ttt_epochs: 5
87
+ ttt_lr: 0.007
88
+ ttt_momentum: 0.9
89
+ val_batch_tokens: 524288
90
+ val_files: ./data/datasets/fineweb10B_sp8192/fineweb_val_*.bin
91
+ val_loss_every: 4000
92
+ vocab_size: 8192
93
+ warmdown_frac: 0.72
94
+ warmup_steps: 20
95
+ wd_sched_high_factor: 1.5
96
+ wd_sched_hold_frac: 0.4
97
+ wd_sched_low_factor: 0.65
98
+ wd_sched_ramp_frac: 0.85
99
+ wd_schedule_enabled: True
100
+ world_size: 8
101
+ xsa_last_n: 11
102
+ ====================================================================================================
103
+ Running Python 3.12.3 (main, Nov 6 2025, 13:44:16) [GCC 13.3.0]
104
+ Running PyTorch 2.9.1+cu128
105
+ Wed Apr 29 09:53:57 2026
106
+ +-----------------------------------------------------------------------------------------+
107
+ | NVIDIA-SMI 580.126.09 Driver Version: 580.126.09 CUDA Version: 13.0 |
108
+ +-----------------------------------------+------------------------+----------------------+
109
+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
110
+ | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
111
+ | | | MIG M. |
112
+ |=========================================+========================+======================|
113
+ | 0 NVIDIA H100 80GB HBM3 On | 00000000:19:00.0 Off | 0 |
114
+ | N/A 38C P0 119W / 700W | 1521MiB / 81559MiB | 0% Default |
115
+ | | | Disabled |
116
+ +-----------------------------------------+------------------------+----------------------+
117
+ | 1 NVIDIA H100 80GB HBM3 On | 00000000:3B:00.0 Off | 0 |
118
+ | N/A 34C P0 118W / 700W | 1521MiB / 81559MiB | 0% Default |
119
+ | | | Disabled |
120
+ +-----------------------------------------+------------------------+----------------------+
121
+ | 2 NVIDIA H100 80GB HBM3 On | 00000000:4C:00.0 Off | 0 |
122
+ | N/A 32C P0 117W / 700W | 1521MiB / 81559MiB | 0% Default |
123
+ | | | Disabled |
124
+ +-----------------------------------------+------------------------+----------------------+
125
+ | 3 NVIDIA H100 80GB HBM3 On | 00000000:5D:00.0 Off | 0 |
126
+ | N/A 37C P0 118W / 700W | 1521MiB / 81559MiB | 4% Default |
127
+ | | | Disabled |
128
+ +-----------------------------------------+------------------------+----------------------+
129
+ | 4 NVIDIA H100 80GB HBM3 On | 00000000:9B:00.0 Off | 0 |
130
+ | N/A 39C P0 120W / 700W | 1521MiB / 81559MiB | 2% Default |
131
+ | | | Disabled |
132
+ +-----------------------------------------+------------------------+----------------------+
133
+ | 5 NVIDIA H100 80GB HBM3 On | 00000000:BB:00.0 Off | 0 |
134
+ | N/A 34C P0 121W / 700W | 1521MiB / 81559MiB | 3% Default |
135
+ | | | Disabled |
136
+ +-----------------------------------------+------------------------+----------------------+
137
+ | 6 NVIDIA H100 80GB HBM3 On | 00000000:CB:00.0 Off | 0 |
138
+ | N/A 38C P0 120W / 700W | 1521MiB / 81559MiB | 2% Default |
139
+ | | | Disabled |
140
+ +-----------------------------------------+------------------------+----------------------+
141
+ | 7 NVIDIA H100 80GB HBM3 On | 00000000:DB:00.0 Off | 0 |
142
+ | N/A 34C P0 120W / 700W | 1521MiB / 81559MiB | 0% Default |
143
+ | | | Disabled |
144
+ +-----------------------------------------+------------------------+----------------------+
145
+
146
+ +-----------------------------------------------------------------------------------------+
147
+ | Processes: |
148
+ | GPU GI CI PID Type Process name GPU Memory |
149
+ | ID ID Usage |
150
+ |=========================================================================================|
151
+ | No running processes found |
152
+ +-----------------------------------------------------------------------------------------+
153
+
154
+ ====================================================================================================
155
+ train_shards: 128
156
+ val_tokens: 40540160
157
+ model_params:35944536
158
+ muon:paired-head NS enabled for q/k matrices tagged=22
159
+ gptq:reserving 12s, effective=588000ms
160
+ warmup_step: 1/20
161
+ warmup_step: 2/20
162
+ warmup_step: 3/20
163
+ warmup_step: 4/20
164
+ warmup_step: 5/20
165
+ warmup_step: 6/20
166
+ warmup_step: 10/20
167
+ warmup_step: 20/20
168
+ loop_warmup:enabled encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10]
169
+ loop_warmup_step: 1/20
170
+ loop_warmup_step: 2/20
171
+ loop_warmup_step: 3/20
172
+ loop_warmup_step: 4/20
173
+ loop_warmup_step: 5/20
174
+ loop_warmup_step: 6/20
175
+ loop_warmup_step: 10/20
176
+ loop_warmup_step: 20/20
177
+ 0/20000 val_loss: 9.0090 val_bpb: 3.4877
178
+ 1/20000 train_loss: 9.0111 train_time: 0.0m tok/s: 8357493
179
+ 2/20000 train_loss: 12.3696 train_time: 0.0m tok/s: 8224856
180
+ 3/20000 train_loss: 11.1322 train_time: 0.0m tok/s: 8106486
181
+ 4/20000 train_loss: 9.4108 train_time: 0.0m tok/s: 8044845
182
+ 5/20000 train_loss: 8.3268 train_time: 0.0m tok/s: 8014159
183
+ 500/20000 train_loss: 3.3250 train_time: 0.8m tok/s: 7767075
184
+ 1000/20000 train_loss: 3.1904 train_time: 1.7m tok/s: 7753753
185
+ 1500/20000 train_loss: 3.1009 train_time: 2.5m tok/s: 7758001
186
+ 2000/20000 train_loss: 3.0633 train_time: 3.4m tok/s: 7762917
187
+ layer_loop:enabled step:2032 frac:0.350 encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10]
188
+ 2500/20000 train_loss: 3.0652 train_time: 4.6m tok/s: 7136927
189
+ 3000/20000 train_loss: 2.9412 train_time: 5.8m tok/s: 6743041
190
+ 3500/20000 train_loss: 2.9616 train_time: 7.1m tok/s: 6471277
191
+ 4000/20000 train_loss: 2.9060 train_time: 8.3m tok/s: 6295682
192
+ 4000/20000 val_loss: 2.8659 val_bpb: 1.1095
193
+ 4500/20000 train_loss: 2.7602 train_time: 9.6m tok/s: 6166690
194
+ 4596/20000 val_loss: 2.8082 val_bpb: 1.0871
195
+ stopping_early: wallclock_cap train_time: 588152ms step: 4596/20000
196
+ peak memory allocated: 39046 MiB reserved: 39070 MiB
197
+ ema:applying EMA weights
198
+ pre-quantization post-ema val_loss:2.80550662 val_bpb:1.08609904 eval_time:6986ms
199
+ Serialized model: 135431033 bytes
200
+ Code size: 57003 bytes
201
+ GPTQ:collecting Hessians from calibration data...
202
+ GPTQ:collected 67 Hessians in 12.8s
203
+ Quantized weights:
204
+ gptq (int6): blocks.attn.c_k.weight, blocks.attn.c_q.weight, blocks.attn.c_v.weight, blocks.attn.proj.weight, blocks.mlp.fc.weight, blocks.mlp.proj.weight
205
+ gptq (int8): tok_emb.weight
206
+ passthrough (float16): blocks.attn.q_gain, blocks.attn_scale, blocks.mlp_scale, blocks.resid_mix, skip_gates, skip_weights
207
+ Serialized model quantized+brotli: 15972921 bytes
208
+ Total submission size quantized+brotli: 16029924 bytes
209
+ quantized:eval_num_loops=2 looping_active=True encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10]
210
+ quantized val_loss:2.83861053 val_bpb:1.09891460 eval_time:24849ms
211
+ quantized_sliding_window val_loss:2.79516170 val_bpb:1.08209420 eval_time:121664ms
212
+ ttt:eval_num_loops=2 looping_active=True encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10]
213
+ ttt:start chunks=1238 ttt_lr=0.007 ttt_epochs=5
214
+ quantized_ttt val_loss:2.78909020 val_bpb:1.07974373 eval_time:514291ms
215
+ ====================================================================================================
216
+ Hyperparameters:
217
+ adam_eps: 1e-08
218
+ adam_wd: 0.02
219
+ beta1: 0.9
220
+ beta2: 0.95
221
+ compressor: brotli
222
+ data_dir: /tmp/kevclark_tok/datasets/
223
+ datasets_dir: /tmp/kevclark_tok/datasets/datasets/fineweb10B_sp8192
224
+ distributed: True
225
+ doc_shuffle_bos_id: -1
226
+ doc_shuffle_enabled: False
227
+ ema_decay: 0.9965
228
+ embed_bits: 8
229
+ embed_clip_sigmas: 20.0
230
+ embed_lr: 0.6
231
+ embed_wd: 0.085
232
+ embedding_dim: 512
233
+ enable_looping_at: 0.35
234
+ etlb_clip: 3.0
235
+ etlb_enabled: False
236
+ etlb_lr: 0.05
237
+ etlb_steps: 5
238
+ eval_num_loops: -1
239
+ eval_seq_len: 2048
240
+ eval_stride: 64
241
+ gptq_calibration_batches: 64
242
+ gptq_reserve_seconds: 12.0
243
+ grad_accum_steps: 1
244
+ grad_clip_norm: 0.3
245
+ head_lr: 0.008
246
+ iha_enabled: False
247
+ iha_mix_v: False
248
+ is_main_process: True
249
+ iterations: 20000
250
+ ln_scale: True
251
+ local_rank: 0
252
+ logfile: logs/pr1493_wd_paired_s42.txt
253
+ logit_softcap: 30.0
254
+ loop_end: 5
255
+ loop_start: 3
256
+ matrix_bits: 6
257
+ matrix_clip_sigmas: 12.85
258
+ matrix_lr: 0.022
259
+ max_wallclock_seconds: 600.0
260
+ min_lr: 0.0
261
+ mlp_mult: 4.0
262
+ model_dim: 512
263
+ model_path: final_model.pt
264
+ mtp_steps: 0
265
+ mtp_weight: 0.0
266
+ muon_backend_steps: 5
267
+ muon_beta2: 0.95
268
+ muon_momentum: 0.99
269
+ muon_momentum_warmup_start: 0.92
270
+ muon_momentum_warmup_steps: 1500
271
+ muon_row_normalize: True
272
+ muon_wd: 0.095
273
+ num_heads: 8
274
+ num_kv_heads: 4
275
+ num_layers: 11
276
+ num_loops: 2
277
+ paired_head_muon_enabled: True
278
+ parallel_residual_start: 7
279
+ qk_gain_init: 5.25
280
+ quantized_model_path: final_model.int6.ptz
281
+ rank: 0
282
+ rope_base: 10000.0
283
+ rope_dims: 16
284
+ rope_train_seq_len: 2048
285
+ run_id: pr1493_wd_paired_s42
286
+ scalar_lr: 0.02
287
+ seed: 42
288
+ skip_gates_enabled: True
289
+ sliding_window_enabled: True
290
+ tie_embeddings: True
291
+ tied_embed_init_std: 0.005
292
+ tied_embed_lr: 0.03
293
+ tokenizer_path: /tmp/kevclark_tok/datasets/tokenizers/fineweb_8192_bpe.model
294
+ train_batch_tokens: 786432
295
+ train_files: /tmp/kevclark_tok/datasets/datasets/fineweb10B_sp8192/fineweb_train_*.bin
296
+ train_log_every: 500
297
+ train_seq_len: 2048
298
+ ttt_chunk_tokens: 32768
299
+ ttt_enabled: True
300
+ ttt_epochs: 5
301
+ ttt_lr: 0.007
302
+ ttt_momentum: 0.9
303
+ val_batch_tokens: 524288
304
+ val_files: /tmp/kevclark_tok/datasets/datasets/fineweb10B_sp8192/fineweb_val_*.bin
305
+ val_loss_every: 4000
306
+ vocab_size: 8192
307
+ warmdown_frac: 0.72
308
+ warmup_steps: 20
309
+ wd_sched_high_factor: 1.5
310
+ wd_sched_hold_frac: 0.4
311
+ wd_sched_low_factor: 0.65
312
+ wd_sched_ramp_frac: 0.85
313
+ wd_schedule_enabled: True
314
+ world_size: 8
315
+ xsa_last_n: 11
316
+ ====================================================================================================
317
+ Running Python 3.12.3 (main, Nov 6 2025, 13:44:16) [GCC 13.3.0]
318
+ Running PyTorch 2.9.1+cu128
319
+ Wed Apr 29 18:42:32 2026
320
+ +-----------------------------------------------------------------------------------------+
321
+ | NVIDIA-SMI 565.57.01 Driver Version: 565.57.01 CUDA Version: 12.7 |
322
+ |-----------------------------------------+------------------------+----------------------+
323
+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
324
+ | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
325
+ | | | MIG M. |
326
+ |=========================================+========================+======================|
327
+ | 0 NVIDIA H100 80GB HBM3 On | 00000000:18:00.0 Off | 0 |
328
+ | N/A 33C P0 149W / 700W | 1518MiB / 81559MiB | 0% Default |
329
+ | | | Disabled |
330
+ +-----------------------------------------+------------------------+----------------------+
331
+ | 1 NVIDIA H100 80GB HBM3 On | 00000000:2A:00.0 Off | 0 |
332
+ | N/A 34C P0 149W / 700W | 1518MiB / 81559MiB | 0% Default |
333
+ | | | Disabled |
334
+ +-----------------------------------------+------------------------+----------------------+
335
+ | 2 NVIDIA H100 80GB HBM3 On | 00000000:3A:00.0 Off | 0 |
336
+ | N/A 35C P0 152W / 700W | 1518MiB / 81559MiB | 0% Default |
337
+ | | | Disabled |
338
+ +-----------------------------------------+------------------------+----------------------+
339
+ | 3 NVIDIA H100 80GB HBM3 On | 00000000:5D:00.0 Off | 0 |
340
+ | N/A 34C P0 150W / 700W | 1518MiB / 81559MiB | 0% Default |
341
+ | | | Disabled |
342
+ +-----------------------------------------+------------------------+----------------------+
343
+ | 4 NVIDIA H100 80GB HBM3 On | 00000000:9A:00.0 Off | 0 |
344
+ | N/A 32C P0 147W / 700W | 1518MiB / 81559MiB | 0% Default |
345
+ | | | Disabled |
346
+ +-----------------------------------------+------------------------+----------------------+
347
+ | 5 NVIDIA H100 80GB HBM3 On | 00000000:AB:00.0 Off | 0 |
348
+ | N/A 34C P0 145W / 700W | 1518MiB / 81559MiB | 0% Default |
349
+ | | | Disabled |
350
+ +-----------------------------------------+------------------------+----------------------+
351
+ | 6 NVIDIA H100 80GB HBM3 On | 00000000:BA:00.0 Off | 0 |
352
+ | N/A 33C P0 146W / 700W | 1518MiB / 81559MiB | 0% Default |
353
+ | | | Disabled |
354
+ +-----------------------------------------+------------------------+----------------------+
355
+ | 7 NVIDIA H100 80GB HBM3 On | 00000000:DB:00.0 Off | 0 |
356
+ | N/A 34C P0 150W / 700W | 1518MiB / 81559MiB | 0% Default |
357
+ | | | Disabled |
358
+ +-----------------------------------------+------------------------+----------------------+
359
+
360
+ +-----------------------------------------------------------------------------------------+
361
+ | Processes: |
362
+ | GPU GI CI PID Type Process name GPU Memory |
363
+ | ID ID Usage |
364
+ |=========================================================================================|
365
+ +-----------------------------------------------------------------------------------------+
366
+
367
+ ====================================================================================================
368
+ train_shards: 128
369
+ val_tokens: 40540160
370
+ model_params:35944536
371
+ muon:paired-head NS enabled for q/k matrices tagged=22
372
+ gptq:reserving 12s, effective=588000ms
373
+ warmup_step: 1/20
374
+ warmup_step: 2/20
375
+ warmup_step: 3/20
376
+ warmup_step: 4/20
377
+ warmup_step: 5/20
378
+ warmup_step: 6/20
379
+ warmup_step: 10/20
380
+ warmup_step: 20/20
381
+ loop_warmup:enabled encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10]
382
+ loop_warmup_step: 1/20
383
+ loop_warmup_step: 2/20
384
+ loop_warmup_step: 3/20
385
+ loop_warmup_step: 4/20
386
+ loop_warmup_step: 5/20
387
+ loop_warmup_step: 6/20
388
+ loop_warmup_step: 10/20
389
+ loop_warmup_step: 20/20
390
+ 0/20000 val_loss: 9.0090 val_bpb: 3.4877
391
+ 1/20000 train_loss: 9.0111 train_time: 0.0m tok/s: 8048285
392
+ 2/20000 train_loss: 12.3694 train_time: 0.0m tok/s: 8032509
393
+ 3/20000 train_loss: 11.1325 train_time: 0.0m tok/s: 7988924
394
+ 4/20000 train_loss: 9.4115 train_time: 0.0m tok/s: 7968219
395
+ 5/20000 train_loss: 8.3275 train_time: 0.0m tok/s: 7953115
396
+ 500/20000 train_loss: 3.3293 train_time: 0.8m tok/s: 7735285
397
+ 1000/20000 train_loss: 3.1855 train_time: 1.7m tok/s: 7740984
398
+ 1500/20000 train_loss: 3.0993 train_time: 2.5m tok/s: 7743329
399
+ 2000/20000 train_loss: 3.0658 train_time: 3.4m tok/s: 7740777
400
+ layer_loop:enabled step:2026 frac:0.350 encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10]
401
+ 2500/20000 train_loss: 3.0648 train_time: 4.6m tok/s: 7103615
402
+ 3000/20000 train_loss: 2.9371 train_time: 5.9m tok/s: 6697102
403
+ 3500/20000 train_loss: 2.9569 train_time: 7.1m tok/s: 6435673
404
+ 4000/20000 train_loss: 2.9049 train_time: 8.4m tok/s: 6260155
405
+ 4000/20000 val_loss: 2.8635 val_bpb: 1.1086
406
+ 4500/20000 train_loss: 2.7581 train_time: 9.6m tok/s: 6131209
407
+ 4573/20000 val_loss: 2.8083 val_bpb: 1.0872
408
+ stopping_early: wallclock_cap train_time: 588138ms step: 4573/20000
409
+ peak memory allocated: 39045 MiB reserved: 39124 MiB
410
+ ema:applying EMA weights
411
+ pre-quantization post-ema val_loss:2.80551469 val_bpb:1.08610217 eval_time:6952ms
412
+ Serialized model: 135431033 bytes
413
+ Code size: 57003 bytes
414
+ GPTQ:collecting Hessians from calibration data...
415
+ GPTQ:collected 67 Hessians in 12.8s
416
+ Quantized weights:
417
+ gptq (int6): blocks.attn.c_k.weight, blocks.attn.c_q.weight, blocks.attn.c_v.weight, blocks.attn.proj.weight, blocks.mlp.fc.weight, blocks.mlp.proj.weight
418
+ gptq (int8): tok_emb.weight
419
+ passthrough (float16): blocks.attn.q_gain, blocks.attn_scale, blocks.mlp_scale, blocks.resid_mix, skip_gates, skip_weights
420
+ Serialized model quantized+brotli: 15975099 bytes
421
+ Total submission size quantized+brotli: 16032102 bytes
422
+ quantized:eval_num_loops=2 looping_active=True encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10]
423
+ quantized val_loss:2.83760043 val_bpb:1.09852356 eval_time:26726ms
424
+ quantized_sliding_window val_loss:2.79426083 val_bpb:1.08174544 eval_time:126238ms
425
+ ttt:eval_num_loops=2 looping_active=True encoder:[0, 1, 2, 3, 4, 5, 3, 4] decoder:[5, 3, 4, 5, 6, 7, 8, 9, 10]
426
+ ttt:start chunks=1238 ttt_lr=0.007 ttt_epochs=5
427
+ quantized_ttt val_loss:2.78913558 val_bpb:1.07976130 eval_time:492211ms