Hugging Face's logo

logic65
/
whittle-dev

Safetensors
GGUF
whittle
research
training-checkpoints
conversational
Model card Files
xet
 Community
logic65 commited on
Commit
17c309b
·
verified ·
1 Parent(s): 5c38027

lw3 final TRAIN.log (step3711)

Browse files
Files changed (1) hide show
  1. lw3/TRAIN.log +367 -0
lw3/TRAIN.log ADDED
@@ -0,0 +1,367 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [18:35:32 + 2.6m] student streamed: 27.6 GB
2
+ [18:35:32 + 2.6m] LoRA r=8 on 130 attention/GDN projections + routed experts of layers [0, 1, 2]..39 + shared experts trained fully
3
+ [18:35:32 + 2.6m] reply-ending tokens: [248044, 248046] (hash reset on 248044)
4
+ [18:36:32 + 3.6m] table held IN RAM: 20.0 GB (flush = full rewrite to /content/work/lw3/ngram_table.npy)
5
+ [18:36:32 + 3.6m] PLE table: 8 heads x 4,880,000 rows x 256 = 9.99B params, injected before layer 2
6
+ [18:36:43 + 3.8m] resumed expert LoRA
7
+ [18:36:43 + 3.8m] resumed shared experts
8
+ [18:36:43 + 3.8m] resumed linear LoRA on 130 projections
9
+ [18:36:43 + 3.8m] resumed HC/PLE/gates/routers from /content/drive-local/lw3/ckpts/next_step3623.pt
10
+ [18:36:43 + 3.8m] lm_head + embed_tokens on cuda:0 (LMHEAD_DEV)
11
+ [18:36:43 + 3.8m] layer-wise KD: /content/work/probe/lw_train2000.npz | pairs [(35, 59), (39, 63)] | weight 0.2 | 2000 cached sequences of 512 tokens
12
+ [18:36:43 + 3.8m] HC_LAYERS=0-39: HC trains on 40 of 40 layers (grads+Muon momentum are ~8 bytes/param)
13
+ [18:36:43 + 3.8m] LoRA/AdamW: 331.5M adapter params at lr 1.0e-04
14
+ [18:36:43 + 3.8m] AUX anchors: layers [35, 39] weight 0.3 each
15
+ [18:36:43 + 3.8m] trainable: Muon 588.9M | AdamW 0.80M | table 9.99B (sparse SGD)
16
+ [18:36:43 + 3.8m] corpus source: /content/work/train/code_corpus.npy (155,781,328 tokens)
17
+ [18:36:48 + 3.8m] held-out extra: /content/work/train/slice_cs/held_code.npy (66,156 tokens, 16 windows)
18
+ [18:36:48 + 3.8m] held-out extra2: /content/work/train/slice_cs/held_sci.npy (27,167 tokens, 16 windows)
19
+ [18:36:54 + 3.9m] teacher cache: 1,840 complete replies from /content/work/tcache_new (K=128); TCACHE_FRAC=0.9 of chat rows
20
+ [18:37:13 + 4.3m] step 0 (HC identity, PLE no-op == the checkpoint): held-out CE SFT 1.2138 | OOD 2.1317
21
+ [18:37:29 + 4.5m] PLE gain @ 0: SFT +0.0047 | OOD +0.3397 (CE with memory OFF minus ON on 8+8 held rows; positive = the table helps)
22
+ [18:37:31 + 4.6m] reasoning prompts: +1315 from /content/work/onpolicy_prompts.json
23
+ [18:37:32 + 4.6m] reasoning prompts: 1315 (1315 GSM8K + 0 code), thinking=ON, tail '```<|im_end|>\n<|im_start|>assistant\n<think>\n'; on-policy runs to EOS or 4096 tok in stages [512, 2048, 4096], batch 8, KL chunk 512
24
+ [18:37:35 + 4.6m] layer-wise row 164: 1022 positions, mean KL 0.3693 = S35<-T59 0.595 | S39<-T63 0.144
25
+ /content/work/kit/code/wt/train_next36.py:1000: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
26
+ Consider using tensor.detach() first. (Triggered internally at /pytorch/torch/csrc/autograd/generated/python_variable_methods.cpp:836.)
27
+ USE_TRACE.append((float(ratio), float(use) / max(n, 1)))
28
+ [18:37:35 + 4.6m] use row (step 1): PLE/residual norm ratio 0.3289 (want >= 0.1) | term 0.0000
29
+ [18:37:36 + 4.7m] step1: grad-norm 1.643e+00 | HC probe max|delta| 5.633e-05 | table rows touched 2980
30
+ [18:37:36 + 4.7m] peak GPU 40.7 GB
31
+ [18:37:36 + 4.7m] PoSE: target window 131,072 | 1 rows offset so far | highest position index seen 109,923
32
+ [18:37:41 + 4.7m] layer-wise row 1852: 1022 positions, mean KL 0.2489 = S35<-T59 0.385 | S39<-T63 0.113
33
+ [18:37:41 + 4.7m] use row (step 3): PLE/residual norm ratio 0.2749 (want >= 0.1) | term 0.0000
34
+ [18:37:43 + 4.8m] dep row (step 3): CE_on 0.1213 | CE_off 0.3472 | gap +0.2259 (want >= 0.3) | term 0.0741
35
+ [18:37:46 + 4.8m] use row (step 5): PLE/residual norm ratio 0.2908 (want >= 0.1) | term 0.0000
36
+ [18:37:50 + 4.9m] use row (step 6): PLE/residual norm ratio 0.2400 (want >= 0.1) | term 0.0000
37
+ [18:38:42 + 5.8m] step 20 loss 0.5652 (kd 0.1033 ce 0.4619) | 3.63s/step | table rows seen 82,976
38
+ [18:39:43 + 6.8m] step 40 loss 0.5812 (kd 0.1061 ce 0.4750) | 3.34s/step | table rows seen 154,304
39
+ [18:40:39 + 7.7m] step 60 loss 0.4212 (kd 0.1327 ce 0.2885) | 3.17s/step | table rows seen 227,272
40
+ [18:41:38 + 8.7m] step 80 loss 0.4945 (kd 0.1521 ce 0.3424) | 3.12s/step | table rows seen 289,516
41
+ [18:42:57 + 10.0m] step 100 loss 0.6385 (kd 0.1087 ce 0.5298) | 3.28s/step | table rows seen 376,512
42
+ [18:43:53 + 10.9m] step 120 loss 0.4114 (kd 0.1331 ce 0.2783) | 3.20s/step | table rows seen 431,600
43
+ [18:44:58 + 12.0m] step 140 loss 0.4830 (kd 0.1542 ce 0.3288) | 3.20s/step | table rows seen 487,476
44
+ [18:46:03 + 13.1m] step 160 loss 0.5936 (kd 0.1313 ce 0.4623) | 3.21s/step | table rows seen 551,024
45
+ [18:47:02 + 14.1m] step 180 loss 0.5086 (kd 0.1515 ce 0.3570) | 3.18s/step | table rows seen 612,092
46
+ [18:48:16 + 15.3m] layer-wise row 902: 1022 positions, mean KL 0.2366 = S35<-T59 0.383 | S39<-T63 0.090
47
+ [18:48:16 + 15.3m] use row (step 200): PLE/residual norm ratio 0.3083 (want >= 0.1) | term 0.0000
48
+ [18:48:17 + 15.3m] step 200 loss 0.5689 (kd 0.1349 ce 0.4340) | 3.24s/step | table rows seen 670,116
49
+ [18:48:36 + 15.7m] held-out CE @ 200: SFT 1.2133 (start 1.2138) | OOD 2.1311 (start 2.1317)
50
+ [18:48:36 + 15.7m] PoSE @ 200: 238 rows offset | highest position index 130,411 of 131,072
51
+ [18:49:21 + 16.4m] PLE gain @ 200: SFT +0.7674 | OOD +1.0803 | IN-corpus +4.0056 | extra +3.4148 (CE 1.2524) | extra2 +0.5580 (CE 0.7409) (CE off minus on; IN = training text the table may have seen) | live rows -1
52
+ [18:49:21 + 16.4m] injection @ 200: 132 rows | PLE/residual ratio mean 0.2991 min 0.1784 max 0.4051 | mean use term 0.0000
53
+ [18:49:21 + 16.4m] dependence @ 200: 38 rows since last eval | mean gap CE_off-CE_on +1.9240 | frac meeting margin 0.97 | mean term 0.0020
54
+ [18:49:22 + 16.4m] HC stats @ 200: gate|dev| median 0.0268 (L0 0.0253, L39 0.0252) | inject w|dev| median 0.3979 | stream spread median 0.4766 (L39 0.4679)
55
+ [18:49:33 + 16.6m] new best by OOD CE: step 200
56
+ [18:50:31 + 17.6m] step 220 loss 0.5362 (kd 0.1485 ce 0.3877) | 3.55s/step | table rows seen 723,912
57
+ [18:51:29 + 18.5m] step 240 loss 0.4285 (kd 0.1246 ce 0.3040) | 3.50s/step | table rows seen 769,116
58
+ [18:52:34 + 19.6m] step 260 loss 0.6967 (kd 0.0999 ce 0.5968) | 3.48s/step | table rows seen 838,036
59
+ [18:53:39 + 20.7m] step 280 loss 0.4906 (kd 0.1358 ce 0.3547) | 3.46s/step | table rows seen 900,828
60
+ [18:54:51 + 21.9m] step 300 loss 0.3579 (kd 0.1166 ce 0.2413) | 3.47s/step | table rows seen 945,392
61
+ [18:55:52 + 22.9m] step 320 loss 0.6378 (kd 0.1382 ce 0.4997) | 3.45s/step | table rows seen 1,006,612
62
+ [18:57:08 + 24.2m] step 340 loss 0.5798 (kd 0.1012 ce 0.4787) | 3.47s/step | table rows seen 1,070,700
63
+ [18:58:10 + 25.2m] step 360 loss 0.4236 (kd 0.1095 ce 0.3141) | 3.45s/step | table rows seen 1,114,924
64
+ [18:59:08 + 26.2m] step 380 loss 0.4968 (kd 0.1356 ce 0.3612) | 3.42s/step | table rows seen 1,157,736
65
+ [19:00:18 + 27.4m] use row (step 400): PLE/residual norm ratio 0.3441 (want >= 0.1) | term 0.0000
66
+ [19:00:22 + 27.4m] step 400 loss 0.5559 (kd 0.1182 ce 0.4378) | 3.43s/step | table rows seen 1,217,472
67
+ [19:00:40 + 27.7m] held-out CE @ 400: SFT 1.2153 (start 1.2138) | OOD 2.1341 (start 2.1317)
68
+ [19:00:40 + 27.7m] PoSE @ 400: 478 rows offset | highest position index 130,674 of 131,072
69
+ [19:01:26 + 28.5m] PLE gain @ 400: SFT +0.9952 | OOD +1.6398 | IN-corpus +5.6298 | extra +4.9067 (CE 1.2504) | extra2 +0.9471 (CE 0.7660) (CE off minus on; IN = training text the table may have seen) | live rows -1
70
+ [19:01:26 + 28.5m] injection @ 400: 158 rows | PLE/residual ratio mean 0.2984 min 0.2238 max 0.4108 | mean use term 0.0000
71
+ [19:01:26 + 28.5m] dependence @ 400: 40 rows since last eval | mean gap CE_off-CE_on +2.0561 | frac meeting margin 0.97 | mean term 0.0018
72
+ [19:01:26 + 28.5m] HC stats @ 400: gate|dev| median 0.0268 (L0 0.0253, L39 0.0253) | inject w|dev| median 0.3967 | stream spread median 0.4758 (L39 0.4679)
73
+ [19:02:39 + 29.7m] step 420 loss 0.4609 (kd 0.1408 ce 0.3201) | 3.59s/step | table rows seen 1,270,128
74
+ [19:03:40 + 30.7m] step 440 loss 0.6480 (kd 0.1372 ce 0.5108) | 3.57s/step | table rows seen 1,323,520
75
+ [19:04:48 + 31.9m] step 460 loss 0.4314 (kd 0.1176 ce 0.3138) | 3.56s/step | table rows seen 1,369,344
76
+ [19:05:53 + 32.9m] step 480 loss 0.5426 (kd 0.1164 ce 0.4262) | 3.55s/step | table rows seen 1,414,312
77
+ [19:06:53 + 33.9m] use row (step 500): PLE/residual norm ratio 0.2629 (want >= 0.1) | term 0.0000
78
+ [19:06:55 + 34.0m] step 500 loss 0.4027 (kd 0.1367 ce 0.2660) | 3.53s/step | table rows seen 1,463,724
79
+ [19:08:00 + 35.1m] step 520 loss 0.5342 (kd 0.1382 ce 0.3960) | 3.52s/step | table rows seen 1,524,472
80
+ [19:09:03 + 36.1m] step 540 loss 0.6331 (kd 0.1302 ce 0.5029) | 3.51s/step | table rows seen 1,586,888
81
+ [19:10:09 + 37.2m] step 560 loss 0.5353 (kd 0.1314 ce 0.4039) | 3.50s/step | table rows seen 1,639,692
82
+ [19:11:12 + 38.3m] step 580 loss 0.6103 (kd 0.1202 ce 0.4901) | 3.49s/step | table rows seen 1,694,028
83
+ [19:12:11 + 39.2m] use row (step 600): PLE/residual norm ratio 0.3169 (want >= 0.1) | term 0.0000
84
+ [19:12:14 + 39.3m] step 600 loss 0.5087 (kd 0.1523 ce 0.3564) | 3.47s/step | table rows seen 1,743,796
85
+ [19:12:32 + 39.6m] held-out CE @ 600: SFT 1.2158 (start 1.2138) | OOD 2.1342 (start 2.1317)
86
+ [19:12:32 + 39.6m] PoSE @ 600: 711 rows offset | highest position index 130,674 of 131,072
87
+ [19:13:22 + 40.4m] PLE gain @ 600: SFT +0.6825 | OOD +1.3738 | IN-corpus +5.0787 | extra +4.2743 (CE 1.2534) | extra2 +0.8201 (CE 0.7361) (CE off minus on; IN = training text the table may have seen) | live rows -1
88
+ [19:13:22 + 40.4m] injection @ 600: 145 rows | PLE/residual ratio mean 0.2931 min 0.2047 max 0.4669 | mean use term 0.0000
89
+ [19:13:22 + 40.4m] dependence @ 600: 33 rows since last eval | mean gap CE_off-CE_on +2.3760 | frac meeting margin 1.00 | mean term 0.0000
90
+ [19:13:22 + 40.4m] HC stats @ 600: gate|dev| median 0.0268 (L0 0.0253, L39 0.0253) | inject w|dev| median 0.3927 | stream spread median 0.4744 (L39 0.4714)
91
+ [19:14:38 + 41.7m] step 620 loss 0.5032 (kd 0.1218 ce 0.3814) | 3.60s/step | table rows seen 1,794,092
92
+ [19:15:58 + 43.0m] step 640 loss 0.6964 (kd 0.0692 ce 0.6272) | 3.61s/step | table rows seen 1,851,012
93
+ [19:16:52 + 43.9m] step 660 loss 0.4806 (kd 0.1559 ce 0.3247) | 3.58s/step | table rows seen 1,891,952
94
+ [19:17:54 + 45.0m] step 680 loss 0.6918 (kd 0.1587 ce 0.5331) | 3.57s/step | table rows seen 1,951,792
95
+ [19:19:00 + 46.1m] use row (step 700): PLE/residual norm ratio 0.3071 (want >= 0.1) | term 0.0000
96
+ [19:19:01 + 46.1m] step 700 loss 0.4619 (kd 0.1301 ce 0.3318) | 3.56s/step | table rows seen 1,995,576
97
+ [19:20:02 + 47.1m] step 720 loss 0.5412 (kd 0.1056 ce 0.4356) | 3.55s/step | table rows seen 2,045,504
98
+ [19:21:15 + 48.3m] step 740 loss 0.4633 (kd 0.1348 ce 0.3286) | 3.55s/step | table rows seen 2,089,544
99
+ [19:22:25 + 49.5m] step 760 loss 0.6159 (kd 0.1478 ce 0.4682) | 3.55s/step | table rows seen 2,141,744
100
+ [19:23:45 + 50.8m] step 780 loss 0.5051 (kd 0.0834 ce 0.4217) | 3.56s/step | table rows seen 2,198,292
101
+ [19:24:49 + 51.9m] use row (step 800): PLE/residual norm ratio 0.2931 (want >= 0.1) | term 0.0000
102
+ [19:24:52 + 51.9m] step 800 loss 0.5880 (kd 0.1197 ce 0.4683) | 3.55s/step | table rows seen 2,254,108
103
+ [19:25:11 + 52.2m] held-out CE @ 800: SFT 1.2125 (start 1.2138) | OOD 2.1334 (start 2.1317)
104
+ [19:25:11 + 52.2m] PoSE @ 800: 952 rows offset | highest position index 130,955 of 131,072
105
+ [19:25:56 + 53.0m] PLE gain @ 800: SFT +1.1591 | OOD +1.8637 | IN-corpus +6.4735 | extra +5.8430 (CE 1.2522) | extra2 +1.4713 (CE 0.7465) (CE off minus on; IN = training text the table may have seen) | live rows -1
106
+ [19:25:56 + 53.0m] injection @ 800: 144 rows | PLE/residual ratio mean 0.2944 min 0.1995 max 0.4084 | mean use term 0.0000
107
+ [19:25:56 + 53.0m] dependence @ 800: 41 rows since last eval | mean gap CE_off-CE_on +3.4760 | frac meeting margin 0.98 | mean term 0.0053
108
+ [19:25:57 + 53.0m] HC stats @ 800: gate|dev| median 0.0269 (L0 0.0252, L39 0.0253) | inject w|dev| median 0.3982 | stream spread median 0.4813 (L39 0.4752)
109
+ [19:27:05 + 54.1m] step 820 loss 0.5712 (kd 0.1413 ce 0.4298) | 3.63s/step | table rows seen 2,303,492
110
+ [19:28:11 + 55.2m] step 840 loss 0.4292 (kd 0.1174 ce 0.3118) | 3.62s/step | table rows seen 2,340,176
111
+ [19:29:32 + 56.6m] step 860 loss 0.6351 (kd 0.1130 ce 0.5221) | 3.63s/step | table rows seen 2,408,968
112
+ [19:30:32 + 57.6m] step 880 loss 0.5712 (kd 0.0953 ce 0.4759) | 3.62s/step | table rows seen 2,454,844
113
+ [19:31:33 + 58.6m] use row (step 900): PLE/residual norm ratio 0.2682 (want >= 0.1) | term 0.0000
114
+ [19:31:40 + 58.7m] dep row (step 900): CE_on 0.9823 | CE_off 6.5950 | gap +5.6126 (want >= 0.3) | term 0.0000
115
+ [19:31:41 + 58.7m] step 900 loss 0.4383 (kd 0.1306 ce 0.3076) | 3.61s/step | table rows seen 2,489,404
116
+ [19:32:42 + 59.7m] step 920 loss 0.6496 (kd 0.1204 ce 0.5293) | 3.60s/step | table rows seen 2,541,848
117
+ [19:33:54 + 61.0m] step 940 loss 0.5866 (kd 0.1353 ce 0.4512) | 3.60s/step | table rows seen 2,588,432
118
+ [19:34:57 + 62.0m] step 960 loss 0.4825 (kd 0.1214 ce 0.3611) | 3.59s/step | table rows seen 2,626,148
119
+ [19:35:56 + 63.0m] step 980 loss 0.4727 (kd 0.1412 ce 0.3316) | 3.58s/step | table rows seen 2,663,784
120
+ [19:37:12 + 64.3m] step 1000 loss 0.5908 (kd 0.1128 ce 0.4780) | 3.58s/step | table rows seen 2,710,708
121
+ [19:37:31 + 64.6m] held-out CE @ 1000: SFT 1.2126 (start 1.2138) | OOD 2.1302 (start 2.1317)
122
+ [19:37:31 + 64.6m] PoSE @ 1000: 1,188 rows offset | highest position index 130,955 of 131,072
123
+ [19:38:16 + 65.3m] PLE gain @ 1000: SFT +0.5153 | OOD +1.2069 | IN-corpus +5.1250 | extra +4.0858 (CE 1.2594) | extra2 +0.8563 (CE 0.7565) (CE off minus on; IN = training text the table may have seen) | live rows -1
124
+ [19:38:16 + 65.3m] injection @ 1000: 142 rows | PLE/residual ratio mean 0.2908 min 0.2249 max 0.3984 | mean use term 0.0000
125
+ [19:38:16 + 65.3m] dependence @ 1000: 36 rows since last eval | mean gap CE_off-CE_on +2.6581 | frac meeting margin 1.00 | mean term 0.0000
126
+ [19:38:17 + 65.3m] HC stats @ 1000: gate|dev| median 0.0270 (L0 0.0252, L39 0.0254) | inject w|dev| median 0.4075 | stream spread median 0.4830 (L39 0.4745)
127
+ [19:38:20 + 65.4m] new best by OOD CE: step 1000
128
+ [19:39:16 + 66.3m] step 1020 loss 0.3953 (kd 0.1260 ce 0.2694) | 3.63s/step | table rows seen 2,737,552
129
+ [19:40:20 + 67.4m] step 1040 loss 0.6849 (kd 0.1716 ce 0.5133) | 3.63s/step | table rows seen 2,785,788
130
+ [19:41:19 + 68.4m] step 1060 loss 0.5102 (kd 0.1279 ce 0.3823) | 3.61s/step | table rows seen 2,825,248
131
+ [19:42:17 + 69.3m] step 1080 loss 0.6599 (kd 0.1189 ce 0.5410) | 3.60s/step | table rows seen 2,870,476
132
+ [19:43:20 + 70.4m] step 1100 loss 0.4893 (kd 0.1233 ce 0.3660) | 3.59s/step | table rows seen 2,916,380
133
+ [19:44:36 + 71.6m] step 1120 loss 0.5182 (kd 0.1106 ce 0.4075) | 3.60s/step | table rows seen 2,958,948
134
+ [19:45:28 + 72.5m] step 1140 loss 0.6903 (kd 0.1321 ce 0.5583) | 3.58s/step | table rows seen 2,994,984
135
+ [19:46:24 + 73.4m] step 1160 loss 0.5657 (kd 0.1503 ce 0.4154) | 3.56s/step | table rows seen 3,031,516
136
+ [19:47:25 + 74.5m] step 1180 loss 0.6283 (kd 0.1004 ce 0.5279) | 3.56s/step | table rows seen 3,083,380
137
+ [19:48:28 + 75.5m] layer-wise row 1486: 1022 positions, mean KL 0.2221 = S35<-T59 0.397 | S39<-T63 0.047
138
+ [19:48:28 + 75.5m] use row (step 1200): PLE/residual norm ratio 0.2874 (want >= 0.1) | term 0.0000
139
+ [19:48:29 + 75.5m] step 1200 loss 0.6064 (kd 0.1256 ce 0.4808) | 3.55s/step | table rows seen 3,137,992
140
+ [19:48:48 + 75.9m] held-out CE @ 1200: SFT 1.2168 (start 1.2138) | OOD 2.1322 (start 2.1317)
141
+ [19:48:48 + 75.9m] PoSE @ 1200: 1,411 rows offset | highest position index 130,955 of 131,072
142
+ [19:49:33 + 76.6m] PLE gain @ 1200: SFT +1.0990 | OOD +1.4824 | IN-corpus +5.1794 | extra +4.1689 (CE 1.2579) | extra2 +0.8264 (CE 0.7565) (CE off minus on; IN = training text the table may have seen) | live rows -1
143
+ [19:49:33 + 76.6m] injection @ 1200: 152 rows | PLE/residual ratio mean 0.2972 min 0.1990 max 0.5407 | mean use term 0.0000
144
+ [19:49:33 + 76.6m] dependence @ 1200: 23 rows since last eval | mean gap CE_off-CE_on +2.9236 | frac meeting margin 0.96 | mean term 0.0085
145
+ [19:49:34 + 76.6m] HC stats @ 1200: gate|dev| median 0.0270 (L0 0.0253, L39 0.0254) | inject w|dev| median 0.4125 | stream spread median 0.4818 (L39 0.4753)
146
+ [19:50:38 + 77.7m] step 1220 loss 0.6266 (kd 0.1834 ce 0.4432) | 3.60s/step | table rows seen 3,174,756
147
+ [19:51:52 + 78.9m] step 1240 loss 0.4542 (kd 0.0981 ce 0.3561) | 3.60s/step | table rows seen 3,214,960
148
+ [19:53:07 + 80.2m] step 1260 loss 0.6370 (kd 0.1342 ce 0.5028) | 3.60s/step | table rows seen 3,265,312
149
+ [19:54:18 + 81.4m] step 1280 loss 0.4200 (kd 0.1395 ce 0.2805) | 3.60s/step | table rows seen 3,296,720
150
+ [19:55:15 + 82.3m] use row (step 1300): PLE/residual norm ratio 0.2943 (want >= 0.1) | term 0.0000
151
+ [19:55:16 + 82.3m] step 1300 loss 0.5486 (kd 0.1483 ce 0.4004) | 3.59s/step | table rows seen 3,339,332
152
+ [19:56:36 + 83.6m] step 1320 loss 0.4928 (kd 0.1196 ce 0.3732) | 3.60s/step | table rows seen 3,377,112
153
+ [19:57:50 + 84.9m] step 1340 loss 0.6176 (kd 0.1059 ce 0.5116) | 3.60s/step | table rows seen 3,421,324
154
+ [19:58:59 + 86.0m] step 1360 loss 0.5468 (kd 0.1376 ce 0.4092) | 3.60s/step | table rows seen 3,464,436
155
+ [19:59:57 + 87.0m] step 1380 loss 0.4671 (kd 0.1155 ce 0.3516) | 3.59s/step | table rows seen 3,498,604
156
+ [20:01:05 + 88.1m] use row (step 1400): PLE/residual norm ratio 0.2856 (want >= 0.1) | term 0.0000
157
+ [20:01:08 + 88.2m] step 1400 loss 0.6458 (kd 0.1570 ce 0.4888) | 3.58s/step | table rows seen 3,547,924
158
+ [20:01:27 + 88.5m] held-out CE @ 1400: SFT 1.2199 (start 1.2138) | OOD 2.1365 (start 2.1317)
159
+ [20:01:27 + 88.5m] PoSE @ 1400: 1,655 rows offset | highest position index 130,955 of 131,072
160
+ [20:02:12 + 89.3m] PLE gain @ 1400: SFT +0.3702 | OOD +0.7709 | IN-corpus +4.0843 | extra +2.9723 (CE 1.2621) | extra2 +0.5127 (CE 0.7171) (CE off minus on; IN = training text the table may have seen) | live rows -1
161
+ [20:02:12 + 89.3m] injection @ 1400: 154 rows | PLE/residual ratio mean 0.2993 min 0.1827 max 0.4169 | mean use term 0.0000
162
+ [20:02:12 + 89.3m] dependence @ 1400: 44 rows since last eval | mean gap CE_off-CE_on +2.0227 | frac meeting margin 1.00 | mean term 0.0000
163
+ [20:02:13 + 89.3m] HC stats @ 1400: gate|dev| median 0.0270 (L0 0.0253, L39 0.0255) | inject w|dev| median 0.4156 | stream spread median 0.4847 (L39 0.4769)
164
+ [20:03:36 + 90.7m] step 1420 loss 0.5256 (kd 0.1266 ce 0.3990) | 3.64s/step | table rows seen 3,593,644
165
+ [20:04:48 + 91.9m] step 1440 loss 0.6151 (kd 0.1272 ce 0.4879) | 3.64s/step | table rows seen 3,647,804
166
+ [20:06:06 + 93.2m] step 1460 loss 0.5550 (kd 0.0855 ce 0.4695) | 3.64s/step | table rows seen 3,693,560
167
+ [20:07:09 + 94.2m] step 1480 loss 0.4166 (kd 0.1112 ce 0.3054) | 3.63s/step | table rows seen 3,715,084
168
+ [20:08:10 + 95.2m] step 1500 loss 0.4583 (kd 0.1499 ce 0.3084) | 3.63s/step | table rows seen 3,745,336
169
+ [20:09:16 + 96.3m] step 1520 loss 0.4815 (kd 0.1317 ce 0.3498) | 3.62s/step | table rows seen 3,770,380
170
+ [20:10:30 + 97.6m] step 1540 loss 0.6570 (kd 0.1213 ce 0.5357) | 3.62s/step | table rows seen 3,823,392
171
+ [20:11:35 + 98.6m] step 1560 loss 0.6040 (kd 0.1238 ce 0.4802) | 3.62s/step | table rows seen 3,874,816
172
+ [20:12:30 + 99.6m] step 1580 loss 0.5091 (kd 0.1597 ce 0.3494) | 3.61s/step | table rows seen 3,905,416
173
+ [20:13:24 +100.5m] step 1600 loss 0.5122 (kd 0.1281 ce 0.3841) | 3.60s/step | table rows seen 3,936,508
174
+ [20:13:43 +100.8m] held-out CE @ 1600: SFT 1.2153 (start 1.2138) | OOD 2.1311 (start 2.1317)
175
+ [20:13:43 +100.8m] PoSE @ 1600: 1,884 rows offset | highest position index 130,955 of 131,072
176
+ [20:14:28 +101.5m] PLE gain @ 1600: SFT +1.6227 | OOD +1.8616 | IN-corpus +5.7615 | extra +5.1352 (CE 1.2594) | extra2 +1.3110 (CE 0.7310) (CE off minus on; IN = training text the table may have seen) | live rows -1
177
+ [20:14:28 +101.5m] injection @ 1600: 146 rows | PLE/residual ratio mean 0.2954 min 0.1902 max 0.5098 | mean use term 0.0000
178
+ [20:14:28 +101.5m] dependence @ 1600: 29 rows since last eval | mean gap CE_off-CE_on +1.7480 | frac meeting margin 0.97 | mean term 0.0006
179
+ [20:14:29 +101.5m] HC stats @ 1600: gate|dev| median 0.0270 (L0 0.0253, L39 0.0255) | inject w|dev| median 0.4188 | stream spread median 0.4861 (L39 0.4750)
180
+ [20:15:32 +102.6m] step 1620 loss 0.5094 (kd 0.1474 ce 0.3619) | 3.63s/step | table rows seen 3,967,832
181
+ [20:16:28 +103.5m] step 1640 loss 0.3629 (kd 0.1580 ce 0.2049) | 3.62s/step | table rows seen 3,988,156
182
+ [20:17:50 +104.9m] step 1660 loss 0.5178 (kd 0.1138 ce 0.4041) | 3.63s/step | table rows seen 4,030,320
183
+ [20:18:53 +105.9m] step 1680 loss 0.4328 (kd 0.1458 ce 0.2870) | 3.62s/step | table rows seen 4,062,052
184
+ [20:19:47 +106.8m] use row (step 1700): PLE/residual norm ratio 0.2957 (want >= 0.1) | term 0.0000
185
+ [20:19:48 +106.9m] step 1700 loss 0.3719 (kd 0.1514 ce 0.2205) | 3.61s/step | table rows seen 4,083,076
186
+ [20:20:56 +108.0m] step 1720 loss 0.4938 (kd 0.1340 ce 0.3599) | 3.61s/step | table rows seen 4,118,324
187
+ [20:21:53 +108.9m] step 1740 loss 0.5093 (kd 0.1513 ce 0.3580) | 3.60s/step | table rows seen 4,144,888
188
+ [20:23:14 +110.3m] step 1760 loss 0.6344 (kd 0.1244 ce 0.5101) | 3.61s/step | table rows seen 4,190,172
189
+ [20:24:17 +111.3m] step 1780 loss 0.5043 (kd 0.1017 ce 0.4026) | 3.60s/step | table rows seen 4,227,396
190
+ [20:25:17 +112.3m] step 1800 loss 0.5407 (kd 0.0956 ce 0.4451) | 3.59s/step | table rows seen 4,263,064
191
+ [20:25:36 +112.7m] held-out CE @ 1800: SFT 1.2146 (start 1.2138) | OOD 2.1343 (start 2.1317)
192
+ [20:25:36 +112.7m] PoSE @ 1800: 2,128 rows offset | highest position index 130,955 of 131,072
193
+ [20:26:21 +113.4m] PLE gain @ 1800: SFT +1.8236 | OOD +2.1390 | IN-corpus +7.0003 | extra +6.5872 (CE 1.2633) | extra2 +1.4485 (CE 0.7521) (CE off minus on; IN = training text the table may have seen) | live rows -1
194
+ [20:26:21 +113.4m] injection @ 1800: 153 rows | PLE/residual ratio mean 0.3015 min 0.2205 max 0.5204 | mean use term 0.0000
195
+ [20:26:21 +113.4m] dependence @ 1800: 44 rows since last eval | mean gap CE_off-CE_on +2.3682 | frac meeting margin 0.98 | mean term 0.0001
196
+ [20:26:21 +113.4m] HC stats @ 1800: gate|dev| median 0.0270 (L0 0.0254, L39 0.0255) | inject w|dev| median 0.4165 | stream spread median 0.4834 (L39 0.4738)
197
+ [20:27:23 +114.4m] step 1820 loss 0.4858 (kd 0.1400 ce 0.3457) | 3.62s/step | table rows seen 4,294,736
198
+ [20:28:22 +115.4m] step 1840 loss 0.4255 (kd 0.1517 ce 0.2738) | 3.62s/step | table rows seen 4,324,100
199
+ [20:29:22 +116.4m] step 1860 loss 0.5383 (kd 0.1190 ce 0.4193) | 3.61s/step | table rows seen 4,363,056
200
+ [20:30:31 +117.6m] step 1880 loss 0.4598 (kd 0.0970 ce 0.3627) | 3.61s/step | table rows seen 4,399,000
201
+ [20:31:31 +118.6m] use row (step 1900): PLE/residual norm ratio 0.3230 (want >= 0.1) | term 0.0000
202
+ [20:31:32 +118.6m] step 1900 loss 0.6447 (kd 0.1058 ce 0.5389) | 3.60s/step | table rows seen 4,436,756
203
+ [20:32:35 +119.6m] step 1920 loss 0.4154 (kd 0.1371 ce 0.2783) | 3.60s/step | table rows seen 4,463,600
204
+ [20:33:42 +120.8m] step 1940 loss 0.4581 (kd 0.1267 ce 0.3314) | 3.59s/step | table rows seen 4,497,160
205
+ [20:34:50 +121.9m] step 1960 loss 0.5389 (kd 0.1186 ce 0.4203) | 3.59s/step | table rows seen 4,530,676
206
+ [20:35:53 +122.9m] step 1980 loss 0.5335 (kd 0.1124 ce 0.4211) | 3.59s/step | table rows seen 4,560,096
207
+ [20:36:50 +123.9m] layer-wise row 366: 1022 positions, mean KL 0.4187 = S35<-T59 0.636 | S39<-T63 0.202
208
+ [20:36:50 +123.9m] use row (step 2000): PLE/residual norm ratio 0.2922 (want >= 0.1) | term 0.0000
209
+ [20:36:52 +123.9m] step 2000 loss 0.4846 (kd 0.1494 ce 0.3352) | 3.58s/step | table rows seen 4,588,044
210
+ [20:37:10 +124.2m] held-out CE @ 2000: SFT 1.2206 (start 1.2138) | OOD 2.1392 (start 2.1317)
211
+ [20:37:10 +124.2m] PoSE @ 2000: 2,357 rows offset | highest position index 130,955 of 131,072
212
+ [20:37:56 +125.0m] PLE gain @ 2000: SFT +1.5347 | OOD +1.9197 | IN-corpus +7.2523 | extra +6.4835 (CE 1.2677) | extra2 +1.4159 (CE 0.7508) (CE off minus on; IN = training text the table may have seen) | live rows -1
213
+ [20:37:56 +125.0m] injection @ 2000: 154 rows | PLE/residual ratio mean 0.3044 min 0.2177 max 0.4653 | mean use term 0.0000
214
+ [20:37:56 +125.0m] dependence @ 2000: 29 rows since last eval | mean gap CE_off-CE_on +2.2464 | frac meeting margin 0.97 | mean term 0.0012
215
+ [20:37:56 +125.0m] HC stats @ 2000: gate|dev| median 0.0271 (L0 0.0253, L39 0.0255) | inject w|dev| median 0.4157 | stream spread median 0.4847 (L39 0.4739)
216
+ [20:38:59 +126.0m] step 2020 loss 0.6370 (kd 0.1100 ce 0.5270) | 3.61s/step | table rows seen 4,619,468
217
+ [20:40:11 +127.2m] step 2040 loss 0.5231 (kd 0.1257 ce 0.3974) | 3.61s/step | table rows seen 4,654,784
218
+ [20:41:25 +128.5m] step 2060 loss 0.6751 (kd 0.1038 ce 0.5713) | 3.61s/step | table rows seen 4,708,948
219
+ [20:42:29 +129.5m] step 2080 loss 0.4662 (kd 0.1307 ce 0.3356) | 3.61s/step | table rows seen 4,738,748
220
+ [20:43:47 +130.8m] use row (step 2100): PLE/residual norm ratio 0.2731 (want >= 0.1) | term 0.0000
221
+ [20:43:54 +131.0m] dep row (step 2100): CE_on 1.3710 | CE_off 6.2856 | gap +4.9146 (want >= 0.3) | term 0.0000
222
+ [20:43:54 +131.0m] step 2100 loss 0.7478 (kd 0.0777 ce 0.6700) | 3.61s/step | table rows seen 4,784,796
223
+ [20:45:02 +132.1m] step 2120 loss 0.5774 (kd 0.1490 ce 0.4283) | 3.61s/step | table rows seen 4,820,560
224
+ [20:46:16 +133.3m] step 2140 loss 0.5357 (kd 0.1231 ce 0.4125) | 3.61s/step | table rows seen 4,856,928
225
+ [20:47:09 +134.2m] step 2160 loss 0.5069 (kd 0.1435 ce 0.3634) | 3.60s/step | table rows seen 4,885,184
226
+ [20:48:18 +135.4m] step 2180 loss 0.5462 (kd 0.1573 ce 0.3889) | 3.60s/step | table rows seen 4,926,636
227
+ [20:49:33 +136.6m] step 2200 loss 0.4884 (kd 0.0978 ce 0.3906) | 3.60s/step | table rows seen 4,962,524
228
+ [20:49:52 +136.9m] held-out CE @ 2200: SFT 1.2193 (start 1.2138) | OOD 2.1401 (start 2.1317)
229
+ [20:49:52 +136.9m] PoSE @ 2200: 2,600 rows offset | highest position index 130,955 of 131,072
230
+ [20:50:38 +137.7m] PLE gain @ 2200: SFT +0.6062 | OOD +1.0096 | IN-corpus +4.6089 | extra +3.2036 (CE 1.2613) | extra2 +0.5614 (CE 0.7840) (CE off minus on; IN = training text the table may have seen) | live rows -1
231
+ [20:50:38 +137.7m] injection @ 2200: 154 rows | PLE/residual ratio mean 0.3112 min 0.1746 max 0.4870 | mean use term 0.0000
232
+ [20:50:38 +137.7m] dependence @ 2200: 43 rows since last eval | mean gap CE_off-CE_on +3.5534 | frac meeting margin 1.00 | mean term 0.0000
233
+ [20:50:38 +137.7m] HC stats @ 2200: gate|dev| median 0.0271 (L0 0.0253, L39 0.0256) | inject w|dev| median 0.4155 | stream spread median 0.4892 (L39 0.4769)
234
+ [20:51:45 +138.8m] step 2220 loss 0.3923 (kd 0.1392 ce 0.2531) | 3.63s/step | table rows seen 4,988,536
235
+ [20:52:50 +139.9m] step 2240 loss 0.4551 (kd 0.1627 ce 0.2924) | 3.63s/step | table rows seen 5,011,032
236
+ [20:53:59 +141.0m] step 2260 loss 0.6083 (kd 0.0983 ce 0.5100) | 3.62s/step | table rows seen 5,047,268
237
+ [20:55:11 +142.2m] step 2280 loss 0.5716 (kd 0.0946 ce 0.4770) | 3.62s/step | table rows seen 5,091,968
238
+ [20:56:15 +143.3m] use row (step 2300): PLE/residual norm ratio 0.3053 (want >= 0.1) | term 0.0000
239
+ [20:56:16 +143.3m] step 2300 loss 0.4645 (kd 0.1308 ce 0.3336) | 3.62s/step | table rows seen 5,125,232
240
+ [20:57:18 +144.4m] step 2320 loss 0.4670 (kd 0.1355 ce 0.3315) | 3.62s/step | table rows seen 5,155,220
241
+ [20:58:16 +145.3m] step 2340 loss 0.4817 (kd 0.1454 ce 0.3363) | 3.61s/step | table rows seen 5,182,248
242
+ [20:59:23 +146.4m] step 2360 loss 0.5113 (kd 0.1517 ce 0.3596) | 3.61s/step | table rows seen 5,213,076
243
+ [21:00:26 +147.5m] step 2380 loss 0.5359 (kd 0.1214 ce 0.4145) | 3.60s/step | table rows seen 5,240,612
244
+ [21:01:32 +148.6m] step 2400 loss 0.4265 (kd 0.1324 ce 0.2941) | 3.60s/step | table rows seen 5,264,564
245
+ [21:01:51 +148.9m] held-out CE @ 2400: SFT 1.2258 (start 1.2138) | OOD 2.1420 (start 2.1317)
246
+ [21:01:51 +148.9m] PoSE @ 2400: 2,834 rows offset | highest position index 130,955 of 131,072
247
+ [21:02:36 +149.7m] PLE gain @ 2400: SFT +0.2468 | OOD +0.6322 | IN-corpus +3.7653 | extra +2.2639 (CE 1.2596) | extra2 +0.1475 (CE 0.7674) (CE off minus on; IN = training text the table may have seen) | live rows -1
248
+ [21:02:36 +149.7m] injection @ 2400: 148 rows | PLE/residual ratio mean 0.3068 min 0.2304 max 0.4569 | mean use term 0.0000
249
+ [21:02:36 +149.7m] dependence @ 2400: 34 rows since last eval | mean gap CE_off-CE_on +1.8114 | frac meeting margin 1.00 | mean term 0.0000
250
+ [21:02:37 +149.7m] HC stats @ 2400: gate|dev| median 0.0272 (L0 0.0253, L39 0.0256) | inject w|dev| median 0.4108 | stream spread median 0.4897 (L39 0.4744)
251
+ [21:03:37 +150.7m] step 2420 loss 0.4724 (kd 0.1409 ce 0.3315) | 3.62s/step | table rows seen 5,284,268
252
+ [21:04:36 +151.7m] step 2440 loss 0.4449 (kd 0.1303 ce 0.3146) | 3.62s/step | table rows seen 5,313,076
253
+ [21:05:40 +152.7m] step 2460 loss 0.5107 (kd 0.1182 ce 0.3926) | 3.61s/step | table rows seen 5,342,620
254
+ [21:07:02 +154.1m] step 2480 loss 0.6205 (kd 0.1018 ce 0.5187) | 3.62s/step | table rows seen 5,385,820
255
+ [21:08:17 +155.3m] use row (step 2500): PLE/residual norm ratio 0.3082 (want >= 0.1) | term 0.0000
256
+ [21:08:18 +155.4m] step 2500 loss 0.5331 (kd 0.0873 ce 0.4458) | 3.62s/step | table rows seen 5,424,056
257
+ [21:09:18 +156.4m] step 2520 loss 0.4706 (kd 0.1394 ce 0.3312) | 3.61s/step | table rows seen 5,460,408
258
+ [21:10:31 +157.6m] step 2540 loss 0.5343 (kd 0.1087 ce 0.4257) | 3.61s/step | table rows seen 5,498,788
259
+ [21:11:38 +158.7m] step 2560 loss 0.6671 (kd 0.1123 ce 0.5548) | 3.61s/step | table rows seen 5,541,424
260
+ [21:12:48 +159.9m] step 2580 loss 0.5061 (kd 0.1623 ce 0.3438) | 3.61s/step | table rows seen 5,569,576
261
+ [21:14:00 +161.1m] layer-wise row 725: 1022 positions, mean KL 0.3130 = S35<-T59 0.459 | S39<-T63 0.167
262
+ [21:14:00 +161.1m] use row (step 2600): PLE/residual norm ratio 0.3188 (want >= 0.1) | term 0.0000
263
+ [21:14:02 +161.1m] step 2600 loss 0.6057 (kd 0.1519 ce 0.4538) | 3.61s/step | table rows seen 5,606,712
264
+ [21:14:20 +161.4m] held-out CE @ 2600: SFT 1.2227 (start 1.2138) | OOD 2.1379 (start 2.1317)
265
+ [21:14:20 +161.4m] PoSE @ 2600: 3,071 rows offset | highest position index 130,955 of 131,072
266
+ [21:15:06 +162.2m] PLE gain @ 2600: SFT +0.5261 | OOD +0.9650 | IN-corpus +4.1110 | extra +3.7646 (CE 1.2618) | extra2 +0.3936 (CE 0.7544) (CE off minus on; IN = training text the table may have seen) | live rows -1
267
+ [21:15:06 +162.2m] injection @ 2600: 145 rows | PLE/residual ratio mean 0.3054 min 0.2282 max 0.4501 | mean use term 0.0000
268
+ [21:15:06 +162.2m] dependence @ 2600: 37 rows since last eval | mean gap CE_off-CE_on +2.3101 | frac meeting margin 0.97 | mean term 0.0007
269
+ [21:15:06 +162.2m] HC stats @ 2600: gate|dev| median 0.0272 (L0 0.0253, L39 0.0256) | inject w|dev| median 0.4199 | stream spread median 0.4875 (L39 0.4775)
270
+ [21:16:31 +163.6m] step 2620 loss 0.5235 (kd 0.1057 ce 0.4179) | 3.64s/step | table rows seen 5,636,628
271
+ [21:17:44 +164.8m] step 2640 loss 0.3950 (kd 0.1047 ce 0.2903) | 3.64s/step | table rows seen 5,658,396
272
+ [21:18:39 +165.7m] step 2660 loss 0.4703 (kd 0.1523 ce 0.3180) | 3.64s/step | table rows seen 5,679,740
273
+ [21:19:46 +166.8m] step 2680 loss 0.4914 (kd 0.1246 ce 0.3668) | 3.63s/step | table rows seen 5,715,868
274
+ [21:20:57 +168.0m] use row (step 2700): PLE/residual norm ratio 0.2603 (want >= 0.1) | term 0.0000
275
+ [21:20:58 +168.0m] step 2700 loss 0.6716 (kd 0.1490 ce 0.5227) | 3.63s/step | table rows seen 5,754,988
276
+ [21:22:17 +169.3m] step 2720 loss 0.5058 (kd 0.0904 ce 0.4153) | 3.64s/step | table rows seen 5,796,432
277
+ [21:23:20 +170.4m] step 2740 loss 0.4298 (kd 0.1116 ce 0.3181) | 3.63s/step | table rows seen 5,830,344
278
+ [21:24:24 +171.5m] step 2760 loss 0.4845 (kd 0.1316 ce 0.3529) | 3.63s/step | table rows seen 5,857,556
279
+ [21:25:41 +172.7m] step 2780 loss 0.5654 (kd 0.1211 ce 0.4442) | 3.63s/step | table rows seen 5,893,760
280
+ [21:26:50 +173.9m] use row (step 2800): PLE/residual norm ratio 0.3013 (want >= 0.1) | term 0.0000
281
+ [21:26:53 +173.9m] step 2800 loss 0.3553 (kd 0.1262 ce 0.2291) | 3.63s/step | table rows seen 5,910,140
282
+ [21:27:12 +174.2m] held-out CE @ 2800: SFT 1.2231 (start 1.2138) | OOD 2.1442 (start 2.1317)
283
+ [21:27:12 +174.2m] PoSE @ 2800: 3,313 rows offset | highest position index 130,955 of 131,072
284
+ [21:27:57 +175.0m] PLE gain @ 2800: SFT +0.7078 | OOD +1.2372 | IN-corpus +4.5593 | extra +4.3177 (CE 1.2623) | extra2 +0.7899 (CE 0.7768) (CE off minus on; IN = training text the table may have seen) | live rows -1
285
+ [21:27:57 +175.0m] injection @ 2800: 164 rows | PLE/residual ratio mean 0.3096 min 0.2239 max 0.4460 | mean use term 0.0000
286
+ [21:27:57 +175.0m] dependence @ 2800: 42 rows since last eval | mean gap CE_off-CE_on +2.7516 | frac meeting margin 0.98 | mean term 0.0008
287
+ [21:27:57 +175.0m] HC stats @ 2800: gate|dev| median 0.0271 (L0 0.0253, L39 0.0256) | inject w|dev| median 0.4213 | stream spread median 0.4866 (L39 0.4764)
288
+ [21:29:09 +176.2m] step 2820 loss 0.6419 (kd 0.1393 ce 0.5027) | 3.65s/step | table rows seen 5,949,352
289
+ [21:30:21 +177.4m] step 2840 loss 0.5976 (kd 0.1287 ce 0.4688) | 3.65s/step | table rows seen 5,986,528
290
+ [21:31:26 +178.5m] step 2860 loss 0.6601 (kd 0.0959 ce 0.5642) | 3.65s/step | table rows seen 6,030,536
291
+ [21:32:37 +179.7m] step 2880 loss 0.4944 (kd 0.1018 ce 0.3926) | 3.65s/step | table rows seen 6,059,048
292
+ [21:33:49 +180.9m] use row (step 2900): PLE/residual norm ratio 0.2435 (want >= 0.1) | term 0.0000
293
+ [21:33:51 +180.9m] dep row (step 2900): CE_on 0.4686 | CE_off 1.7081 | gap +1.2395 (want >= 0.3) | term 0.0000
294
+ [21:33:51 +180.9m] step 2900 loss 0.4703 (kd 0.1399 ce 0.3304) | 3.65s/step | table rows seen 6,084,784
295
+ [21:35:09 +182.2m] step 2920 loss 0.7019 (kd 0.2009 ce 0.5010) | 3.65s/step | table rows seen 6,131,632
296
+ [21:36:24 +183.4m] step 2940 loss 0.5679 (kd 0.1087 ce 0.4592) | 3.65s/step | table rows seen 6,167,836
297
+ [21:37:22 +184.4m] step 2960 loss 0.4978 (kd 0.1326 ce 0.3651) | 3.65s/step | table rows seen 6,191,108
298
+ [21:38:19 +185.4m] step 2980 loss 0.4873 (kd 0.1264 ce 0.3609) | 3.64s/step | table rows seen 6,214,404
299
+ [21:39:17 +186.3m] step 3000 loss 0.3996 (kd 0.1238 ce 0.2758) | 3.64s/step | table rows seen 6,232,192
300
+ [21:39:36 +186.7m] held-out CE @ 3000: SFT 1.2243 (start 1.2138) | OOD 2.1445 (start 2.1317)
301
+ [21:39:36 +186.7m] PoSE @ 3000: 3,546 rows offset | highest position index 130,955 of 131,072
302
+ [21:40:21 +187.4m] PLE gain @ 3000: SFT +0.2893 | OOD +0.7889 | IN-corpus +3.5836 | extra +3.1686 (CE 1.2559) | extra2 +0.2485 (CE 0.7768) (CE off minus on; IN = training text the table may have seen) | live rows -1
303
+ [21:40:21 +187.4m] injection @ 3000: 154 rows | PLE/residual ratio mean 0.3072 min 0.1704 max 0.4286 | mean use term 0.0000
304
+ [21:40:21 +187.4m] dependence @ 3000: 33 rows since last eval | mean gap CE_off-CE_on +1.9253 | frac meeting margin 1.00 | mean term 0.0000
305
+ [21:40:22 +187.4m] HC stats @ 3000: gate|dev| median 0.0271 (L0 0.0254, L39 0.0257) | inject w|dev| median 0.4284 | stream spread median 0.4880 (L39 0.4786)
306
+ [21:41:51 +188.9m] step 3020 loss 0.5289 (kd 0.1617 ce 0.3672) | 3.66s/step | table rows seen 6,257,004
307
+ [21:42:52 +189.9m] step 3040 loss 0.4586 (kd 0.1666 ce 0.2920) | 3.66s/step | table rows seen 6,279,976
308
+ [21:44:01 +191.1m] step 3060 loss 0.4870 (kd 0.1394 ce 0.3476) | 3.66s/step | table rows seen 6,310,512
309
+ [21:45:15 +192.3m] step 3080 loss 0.5256 (kd 0.1060 ce 0.4196) | 3.66s/step | table rows seen 6,342,444
310
+ [21:46:14 +193.3m] step 3100 loss 0.4693 (kd 0.0958 ce 0.3735) | 3.65s/step | table rows seen 6,371,196
311
+ [21:47:10 +194.2m] step 3120 loss 0.4025 (kd 0.1341 ce 0.2683) | 3.65s/step | table rows seen 6,380,908
312
+ [21:48:17 +195.3m] step 3140 loss 0.5208 (kd 0.1103 ce 0.4105) | 3.65s/step | table rows seen 6,408,348
313
+ [21:49:26 +196.5m] step 3160 loss 0.6240 (kd 0.1104 ce 0.5136) | 3.64s/step | table rows seen 6,448,856
314
+ [21:50:28 +197.5m] step 3180 loss 0.6160 (kd 0.1735 ce 0.4424) | 3.64s/step | table rows seen 6,473,584
315
+ [21:51:36 +198.7m] use row (step 3200): PLE/residual norm ratio 0.2944 (want >= 0.1) | term 0.0000
316
+ [21:51:39 +198.7m] step 3200 loss 0.3627 (kd 0.1139 ce 0.2487) | 3.64s/step | table rows seen 6,494,304
317
+ [21:51:58 +199.0m] held-out CE @ 3200: SFT 1.2242 (start 1.2138) | OOD 2.1462 (start 2.1317)
318
+ [21:51:58 +199.0m] PoSE @ 3200: 3,784 rows offset | highest position index 130,955 of 131,072
319
+ [21:52:44 +199.8m] PLE gain @ 3200: SFT +0.5958 | OOD +1.0680 | IN-corpus +3.7487 | extra +3.6534 (CE 1.2651) | extra2 +0.5994 (CE 0.7865) (CE off minus on; IN = training text the table may have seen) | live rows -1
320
+ [21:52:44 +199.8m] injection @ 3200: 143 rows | PLE/residual ratio mean 0.3111 min 0.2277 max 0.4397 | mean use term 0.0000
321
+ [21:52:44 +199.8m] dependence @ 3200: 38 rows since last eval | mean gap CE_off-CE_on +1.9921 | frac meeting margin 0.97 | mean term 0.0056
322
+ [21:52:44 +199.8m] HC stats @ 3200: gate|dev| median 0.0271 (L0 0.0254, L39 0.0257) | inject w|dev| median 0.4236 | stream spread median 0.4885 (L39 0.4786)
323
+ [21:53:52 +200.9m] step 3220 loss 0.4263 (kd 0.1403 ce 0.2860) | 3.66s/step | table rows seen 6,517,196
324
+ [21:55:01 +202.1m] step 3240 loss 0.3882 (kd 0.1253 ce 0.2629) | 3.66s/step | table rows seen 6,540,628
325
+ [21:56:02 +203.1m] step 3260 loss 0.3776 (kd 0.1352 ce 0.2424) | 3.65s/step | table rows seen 6,555,900
326
+ [21:57:09 +204.2m] step 3280 loss 0.5856 (kd 0.1263 ce 0.4593) | 3.65s/step | table rows seen 6,585,508
327
+ [21:58:25 +205.5m] use row (step 3300): PLE/residual norm ratio 0.3783 (want >= 0.1) | term 0.0000
328
+ [21:58:27 +205.5m] dep row (step 3300): CE_on 0.1591 | CE_off 0.5846 | gap +0.4255 (want >= 0.3) | term 0.0000
329
+ [21:58:27 +205.5m] step 3300 loss 0.4600 (kd 0.1371 ce 0.3229) | 3.65s/step | table rows seen 6,604,660
330
+ [21:59:29 +206.5m] step 3320 loss 0.4916 (kd 0.1179 ce 0.3737) | 3.65s/step | table rows seen 6,628,704
331
+ [22:00:33 +207.6m] step 3340 loss 0.5504 (kd 0.1064 ce 0.4439) | 3.65s/step | table rows seen 6,657,660
332
+ [22:01:35 +208.6m] step 3360 loss 0.4846 (kd 0.1228 ce 0.3618) | 3.64s/step | table rows seen 6,684,772
333
+ [22:02:37 +209.7m] step 3380 loss 0.4445 (kd 0.1483 ce 0.2962) | 3.64s/step | table rows seen 6,704,864
334
+ [22:03:37 +210.7m] step 3400 loss 0.5240 (kd 0.1122 ce 0.4119) | 3.64s/step | table rows seen 6,726,796
335
+ [22:03:56 +211.0m] held-out CE @ 3400: SFT 1.2246 (start 1.2138) | OOD 2.1431 (start 2.1317)
336
+ [22:03:56 +211.0m] PoSE @ 3400: 4,036 rows offset | highest position index 130,955 of 131,072
337
+ [22:04:41 +211.7m] PLE gain @ 3400: SFT +1.1368 | OOD +1.5550 | IN-corpus +5.2589 | extra +5.2941 (CE 1.2673) | extra2 +1.1339 (CE 0.7528) (CE off minus on; IN = training text the table may have seen) | live rows -1
338
+ [22:04:41 +211.7m] injection @ 3400: 155 rows | PLE/residual ratio mean 0.3127 min 0.2434 max 0.5199 | mean use term 0.0000
339
+ [22:04:41 +211.7m] dependence @ 3400: 52 rows since last eval | mean gap CE_off-CE_on +1.3096 | frac meeting margin 0.98 | mean term 0.0054
340
+ [22:04:42 +211.8m] HC stats @ 3400: gate|dev| median 0.0272 (L0 0.0254, L39 0.0257) | inject w|dev| median 0.4252 | stream spread median 0.4877 (L39 0.4777)
341
+ [22:06:01 +213.1m] step 3420 loss 0.6619 (kd 0.1534 ce 0.5085) | 3.66s/step | table rows seen 6,763,220
342
+ [22:07:21 +214.4m] step 3440 loss 0.5567 (kd 0.0994 ce 0.4573) | 3.66s/step | table rows seen 6,799,528
343
+ [22:08:19 +215.4m] step 3460 loss 0.3613 (kd 0.1459 ce 0.2154) | 3.66s/step | table rows seen 6,815,680
344
+ [22:09:30 +216.6m] step 3480 loss 0.5161 (kd 0.1286 ce 0.3874) | 3.66s/step | table rows seen 6,851,532
345
+ [22:10:34 +217.6m] use row (step 3500): PLE/residual norm ratio 0.2676 (want >= 0.1) | term 0.0000
346
+ [22:10:37 +217.7m] step 3500 loss 0.5720 (kd 0.1044 ce 0.4676) | 3.65s/step | table rows seen 6,888,840
347
+ [22:11:41 +218.7m] step 3520 loss 0.4975 (kd 0.1374 ce 0.3600) | 3.65s/step | table rows seen 6,913,008
348
+ [22:12:43 +219.8m] step 3540 loss 0.3606 (kd 0.1188 ce 0.2418) | 3.65s/step | table rows seen 6,927,796
349
+ [22:13:43 +220.8m] step 3560 loss 0.4379 (kd 0.1629 ce 0.2750) | 3.64s/step | table rows seen 6,947,548
350
+ [22:14:48 +221.9m] step 3580 loss 0.4984 (kd 0.1245 ce 0.3739) | 3.64s/step | table rows seen 6,974,732
351
+ [22:16:10 +223.2m] step 3600 loss 0.4999 (kd 0.1103 ce 0.3896) | 3.64s/step | table rows seen 7,001,764
352
+ [22:16:29 +223.5m] held-out CE @ 3600: SFT 1.2286 (start 1.2138) | OOD 2.1500 (start 2.1317)
353
+ [22:16:29 +223.5m] PoSE @ 3600: 4,278 rows offset | highest position index 130,955 of 131,072
354
+ [22:17:14 +224.3m] PLE gain @ 3600: SFT +0.6549 | OOD +1.2428 | IN-corpus +4.2679 | extra +4.1953 (CE 1.2595) | extra2 +0.5401 (CE 0.7549) (CE off minus on; IN = training text the table may have seen) | live rows -1
355
+ [22:17:14 +224.3m] injection @ 3600: 149 rows | PLE/residual ratio mean 0.3157 min 0.2173 max 0.4824 | mean use term 0.0000
356
+ [22:17:14 +224.3m] dependence @ 3600: 42 rows since last eval | mean gap CE_off-CE_on +1.9727 | frac meeting margin 1.00 | mean term 0.0000
357
+ [22:17:15 +224.3m] HC stats @ 3600: gate|dev| median 0.0272 (L0 0.0254, L39 0.0257) | inject w|dev| median 0.4208 | stream spread median 0.4897 (L39 0.4785)
358
+ [22:18:23 +225.4m] step 3620 loss 0.5612 (kd 0.1538 ce 0.4074) | 3.66s/step | table rows seen 7,025,824
359
+ [22:19:22 +226.4m] step 3640 loss 0.5661 (kd 0.1216 ce 0.4446) | 3.66s/step | table rows seen 7,050,248
360
+ [22:20:19 +227.4m] step 3660 loss 0.5067 (kd 0.1519 ce 0.3548) | 3.65s/step | table rows seen 7,070,044
361
+ [22:21:19 +228.4m] step 3680 loss 0.4598 (kd 0.1473 ce 0.3124) | 3.65s/step | table rows seen 7,088,824
362
+ [22:22:16 +229.3m] use row (step 3700): PLE/residual norm ratio 0.3050 (want >= 0.1) | term 0.0000
363
+ [22:22:17 +229.3m] step 3700 loss 0.4345 (kd 0.1245 ce 0.3099) | 3.65s/step | table rows seen 7,110,956
364
+ [22:22:57 +230.0m] TIME BUDGET: stopping at step 3712
365
+ [22:23:03 +230.1m] saved the final state as next_step3711 (4 Sep: a run must never end without a checkpoint)
366
+ [22:24:01 +231.1m] table flushed to disk at run end (in-RAM table rewritten)
367
+ [22:24:01 +231.1m] DONE best step 1000 OOD 2.1302 (start 2.1317) | checkpoints in /content/work/lw3/ckpts | table /content/work/lw3/ngram_table.npy