Hugging Face's logo

logic65
/
whittle-dev

Safetensors
GGUF
whittle
research
training-checkpoints
conversational
Model card Files
xet
 Community
logic65 commited on
Commit
888f2f7
·
verified ·
1 Parent(s): cfc6cfe

colab2 trainer log

Browse files
Files changed (1) hide show
  1. colab2/eval/TRAIN.log +172 -0
colab2/eval/TRAIN.log ADDED
@@ -0,0 +1,172 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [10:27:07 + 3.1m] student streamed: 27.6 GB
2
+ [10:27:07 + 3.1m] LoRA r=8 on 130 attention/GDN projections + routed experts of layers [0, 1, 2]..39 + shared experts trained fully
3
+ [10:27:08 + 3.1m] reply-ending tokens: [248044, 248046] (hash reset on 248044)
4
+ [10:27:08 + 3.1m] PLE table: 8 heads x 976,640 rows x 256 = 2.00B params, injected before layer 2
5
+ [10:27:22 + 3.4m] resumed expert LoRA
6
+ [10:27:22 + 3.4m] resumed shared experts
7
+ [10:27:22 + 3.4m] resumed linear LoRA on 130 projections
8
+ [10:27:22 + 3.4m] resumed HC/PLE/gates/routers from /content/drive-local/arm4400/ckpts/next_step4400.pt
9
+ [10:27:22 + 3.4m] lm_head + embed_tokens on cuda:0 (LMHEAD_DEV)
10
+ [10:27:22 + 3.4m] HC_LAYERS=0-39: HC trains on 40 of 40 layers (grads+Muon momentum are ~8 bytes/param)
11
+ [10:27:22 + 3.4m] LoRA/AdamW: 331.5M adapter params at lr 1.0e-04
12
+ [10:27:22 + 3.4m] AUX anchors: off
13
+ [10:27:22 + 3.4m] trainable: Muon 588.9M | AdamW 0.80M | table 2.00B (sparse SGD)
14
+ [10:27:22 + 3.4m] corpus source: /content/work/train/code_corpus.npy (155,781,328 tokens)
15
+ [10:27:24 + 3.4m] held-out extra: /content/work/train/slice_cs/held_code.npy (66,156 tokens, 16 windows)
16
+ [10:27:24 + 3.4m] held-out extra2: /content/work/train/slice_cs/held_sci.npy (27,167 tokens, 16 windows)
17
+ [10:27:32 + 3.5m] teacher cache: 1,840 complete replies from /content/work/tcache_new (K=128); TCACHE_FRAC=0.9 of chat rows
18
+ [10:27:52 + 3.9m] step 0 (HC identity, PLE no-op == the checkpoint): held-out CE SFT 1.2332 | OOD 2.1218
19
+ [10:28:07 + 4.1m] PLE gain @ 0: SFT -0.0030 | OOD -0.0030 (CE with memory OFF minus ON on 8+8 held rows; positive = the table helps)
20
+ [10:28:08 + 4.2m] reasoning prompts: +1315 from /content/work/onpolicy_prompts.json
21
+ [10:28:09 + 4.2m] reasoning prompts: 1315 (1315 GSM8K + 0 code), thinking=ON, tail '```<|im_end|>\n<|im_start|>assistant\n<think>\n'; on-policy runs to EOS or 4096 tok in stages [512, 2048, 4096], batch 8, KL chunk 512
22
+ [10:28:11 + 4.2m] step1: grad-norm 6.370e+00 | HC probe max|delta| 7.362e-05 | table rows touched 4304
23
+ [10:28:11 + 4.2m] peak GPU 40.8 GB
24
+ [10:29:16 + 5.3m] step 20 loss 0.8339 (kd 0.1158 ce 0.7181) | 3.45s/step | table rows seen 129,500
25
+ [10:30:30 + 6.5m] step 40 loss 0.7435 (kd 0.1501 ce 0.5934) | 3.56s/step | table rows seen 225,468
26
+ [10:31:39 + 7.7m] step 60 loss 0.8047 (kd 0.1349 ce 0.6698) | 3.53s/step | table rows seen 334,228
27
+ [10:32:43 + 8.7m] step 80 loss 0.7467 (kd 0.1038 ce 0.6429) | 3.44s/step | table rows seen 408,000
28
+ [10:33:49 + 9.8m] step 100 loss 0.7250 (kd 0.1467 ce 0.5783) | 3.41s/step | table rows seen 494,708
29
+ [10:34:49 + 10.8m] step 120 loss 0.5772 (kd 0.0777 ce 0.4995) | 3.35s/step | table rows seen 565,916
30
+ [10:35:54 + 11.9m] step 140 loss 0.6729 (kd 0.0913 ce 0.5816) | 3.33s/step | table rows seen 643,388
31
+ [10:36:57 + 13.0m] step 160 loss 0.6355 (kd 0.0611 ce 0.5743) | 3.31s/step | table rows seen 712,832
32
+ [10:37:58 + 14.0m] step 180 loss 0.5405 (kd 0.1158 ce 0.4247) | 3.28s/step | table rows seen 780,956
33
+ [10:39:13 + 15.2m] step 200 loss 0.5570 (kd 0.1307 ce 0.4264) | 3.33s/step | table rows seen 854,100
34
+ [10:39:32 + 15.5m] held-out CE @ 200: SFT 1.2103 (start 1.2332) | OOD 2.1123 (start 2.1218)
35
+ [10:40:17 + 16.3m] PLE gain @ 200: SFT +0.0006 | OOD +0.0002 | IN-corpus +0.0023 | extra +0.0016 (CE 1.2426) | extra2 -0.0025 (CE 0.7707) (CE off minus on; IN = training text the table may have seen) | live rows -1
36
+ [10:40:18 + 16.3m] HC stats @ 200: gate|dev| median 0.0262 (L0 0.0250, L39 0.0252) | inject w|dev| median 0.3583 | stream spread median 0.4435 (L39 0.4643)
37
+ [10:40:33 + 16.6m] new best by OOD CE: step 200
38
+ [10:41:50 + 17.8m] step 220 loss 0.6313 (kd 0.0501 ce 0.5812) | 3.74s/step | table rows seen 931,620
39
+ [10:43:01 + 19.0m] step 240 loss 0.6724 (kd 0.1254 ce 0.5470) | 3.72s/step | table rows seen 1,007,404
40
+ [10:44:11 + 20.2m] step 260 loss 0.6858 (kd 0.0801 ce 0.6057) | 3.71s/step | table rows seen 1,093,104
41
+ [10:45:16 + 21.3m] step 280 loss 0.6755 (kd 0.0678 ce 0.6077) | 3.67s/step | table rows seen 1,159,588
42
+ [10:46:24 + 22.4m] step 300 loss 0.6186 (kd 0.1202 ce 0.4984) | 3.66s/step | table rows seen 1,222,120
43
+ [10:47:37 + 23.6m] step 320 loss 0.6943 (kd 0.0859 ce 0.6084) | 3.65s/step | table rows seen 1,292,156
44
+ [10:48:55 + 24.9m] step 340 loss 0.8735 (kd 0.1329 ce 0.7406) | 3.67s/step | table rows seen 1,367,524
45
+ [10:50:17 + 26.3m] step 360 loss 0.7344 (kd 0.1000 ce 0.6343) | 3.69s/step | table rows seen 1,430,060
46
+ [10:51:32 + 27.5m] step 380 loss 0.6982 (kd 0.1114 ce 0.5868) | 3.70s/step | table rows seen 1,493,500
47
+ [10:52:41 + 28.7m] step 400 loss 0.3930 (kd 0.0800 ce 0.3131) | 3.68s/step | table rows seen 1,533,548
48
+ [10:53:00 + 29.0m] held-out CE @ 400: SFT 1.2084 (start 1.2332) | OOD 2.1153 (start 2.1218)
49
+ [10:53:45 + 29.8m] PLE gain @ 400: SFT +0.0007 | OOD -0.0018 | IN-corpus +0.0009 | extra +0.0022 (CE 1.2447) | extra2 -0.0009 (CE 0.7286) (CE off minus on; IN = training text the table may have seen) | live rows -1
50
+ [10:53:46 + 29.8m] HC stats @ 400: gate|dev| median 0.0262 (L0 0.0250, L39 0.0253) | inject w|dev| median 0.3579 | stream spread median 0.4476 (L39 0.4615)
51
+ [10:55:19 + 31.3m] step 420 loss 0.7316 (kd 0.1574 ce 0.5743) | 3.88s/step | table rows seen 1,585,652
52
+ [10:56:31 + 32.5m] step 440 loss 0.8219 (kd 0.1314 ce 0.6904) | 3.87s/step | table rows seen 1,658,768
53
+ [10:57:52 + 33.9m] step 460 loss 0.6823 (kd 0.0829 ce 0.5994) | 3.88s/step | table rows seen 1,716,440
54
+ [10:59:10 + 35.2m] step 480 loss 0.7078 (kd 0.0976 ce 0.6102) | 3.88s/step | table rows seen 1,766,772
55
+ [11:00:22 + 36.4m] step 500 loss 0.6851 (kd 0.1192 ce 0.5660) | 3.87s/step | table rows seen 1,826,096
56
+ [11:01:29 + 37.5m] step 520 loss 0.6807 (kd 0.1354 ce 0.5453) | 3.85s/step | table rows seen 1,881,136
57
+ [11:02:38 + 38.7m] step 540 loss 0.7789 (kd 0.1056 ce 0.6733) | 3.83s/step | table rows seen 1,937,016
58
+ [11:03:52 + 39.9m] step 560 loss 0.8681 (kd 0.1475 ce 0.7206) | 3.83s/step | table rows seen 1,998,184
59
+ [11:05:02 + 41.0m] step 580 loss 0.5371 (kd 0.1174 ce 0.4197) | 3.82s/step | table rows seen 2,044,560
60
+ [11:06:17 + 42.3m] step 600 loss 0.7120 (kd 0.0914 ce 0.6206) | 3.82s/step | table rows seen 2,096,208
61
+ [11:06:36 + 42.6m] held-out CE @ 600: SFT 1.2074 (start 1.2332) | OOD 2.1141 (start 2.1218)
62
+ [11:07:22 + 43.4m] PLE gain @ 600: SFT +0.0014 | OOD -0.0007 | IN-corpus +0.0013 | extra +0.0020 (CE 1.2381) | extra2 +0.0003 (CE 0.7782) (CE off minus on; IN = training text the table may have seen) | live rows -1
63
+ [11:07:22 + 43.4m] HC stats @ 600: gate|dev| median 0.0262 (L0 0.0250, L39 0.0253) | inject w|dev| median 0.3635 | stream spread median 0.4418 (L39 0.4649)
64
+ [11:08:37 + 44.6m] step 620 loss 0.6486 (kd 0.0767 ce 0.5720) | 3.92s/step | table rows seen 2,140,464
65
+ [11:09:39 + 45.7m] step 640 loss 0.7565 (kd 0.0902 ce 0.6662) | 3.89s/step | table rows seen 2,186,692
66
+ [11:10:44 + 46.7m] step 660 loss 0.6027 (kd 0.1106 ce 0.4921) | 3.87s/step | table rows seen 2,231,324
67
+ [11:11:46 + 47.8m] step 680 loss 0.5134 (kd 0.0701 ce 0.4433) | 3.85s/step | table rows seen 2,258,908
68
+ [11:12:52 + 48.9m] step 700 loss 0.4725 (kd 0.1105 ce 0.3621) | 3.84s/step | table rows seen 2,285,804
69
+ [11:13:48 + 49.8m] step 720 loss 0.7861 (kd 0.0823 ce 0.7038) | 3.81s/step | table rows seen 2,315,900
70
+ [11:14:53 + 50.9m] step 740 loss 0.6848 (kd 0.1305 ce 0.5543) | 3.79s/step | table rows seen 2,355,668
71
+ [11:15:52 + 51.9m] step 760 loss 0.7695 (kd 0.1209 ce 0.6486) | 3.77s/step | table rows seen 2,398,132
72
+ [11:17:00 + 53.0m] step 780 loss 0.8643 (kd 0.1336 ce 0.7307) | 3.76s/step | table rows seen 2,444,376
73
+ [11:18:06 + 54.1m] step 800 loss 0.6559 (kd 0.1294 ce 0.5266) | 3.75s/step | table rows seen 2,486,344
74
+ [11:18:25 + 54.4m] held-out CE @ 800: SFT 1.2017 (start 1.2332) | OOD 2.1103 (start 2.1218)
75
+ [11:19:11 + 55.2m] PLE gain @ 800: SFT -0.0013 | OOD -0.0032 | IN-corpus +0.0003 | extra +0.0009 (CE 1.2461) | extra2 -0.0010 (CE 0.7469) (CE off minus on; IN = training text the table may have seen) | live rows -1
76
+ [11:19:11 + 55.2m] HC stats @ 800: gate|dev| median 0.0262 (L0 0.0250, L39 0.0253) | inject w|dev| median 0.3600 | stream spread median 0.4475 (L39 0.4612)
77
+ [11:19:23 + 55.4m] new best by OOD CE: step 800
78
+ [11:20:30 + 56.5m] step 820 loss 0.6804 (kd 0.0560 ce 0.6243) | 3.83s/step | table rows seen 2,527,920
79
+ [11:21:30 + 57.5m] step 840 loss 0.5647 (kd 0.1095 ce 0.4553) | 3.81s/step | table rows seen 2,554,836
80
+ [11:22:45 + 58.8m] step 860 loss 0.5461 (kd 0.0742 ce 0.4719) | 3.81s/step | table rows seen 2,583,092
81
+ [11:23:59 + 60.0m] step 880 loss 0.6886 (kd 0.0878 ce 0.6007) | 3.81s/step | table rows seen 2,626,452
82
+ [11:25:08 + 61.2m] step 900 loss 0.6783 (kd 0.1281 ce 0.5502) | 3.80s/step | table rows seen 2,665,208
83
+ [11:26:09 + 62.2m] step 920 loss 0.4526 (kd 0.0685 ce 0.3841) | 3.78s/step | table rows seen 2,694,820
84
+ [11:27:12 + 63.2m] step 940 loss 0.4655 (kd 0.0862 ce 0.3793) | 3.77s/step | table rows seen 2,720,740
85
+ [11:28:24 + 64.4m] step 960 loss 0.5784 (kd 0.1347 ce 0.4437) | 3.77s/step | table rows seen 2,747,896
86
+ [11:29:32 + 65.5m] step 980 loss 0.5764 (kd 0.0677 ce 0.5087) | 3.76s/step | table rows seen 2,785,168
87
+ [11:30:49 + 66.8m] step 1000 loss 0.6477 (kd 0.0646 ce 0.5832) | 3.76s/step | table rows seen 2,817,476
88
+ [11:31:08 + 67.2m] held-out CE @ 1000: SFT 1.2015 (start 1.2332) | OOD 2.1052 (start 2.1218)
89
+ [11:31:54 + 67.9m] PLE gain @ 1000: SFT -0.0019 | OOD +0.0004 | IN-corpus +0.0019 | extra +0.0015 (CE 1.2454) | extra2 -0.0018 (CE 0.7461) (CE off minus on; IN = training text the table may have seen) | live rows -1
90
+ [11:31:54 + 67.9m] HC stats @ 1000: gate|dev| median 0.0262 (L0 0.0250, L39 0.0253) | inject w|dev| median 0.3573 | stream spread median 0.4445 (L39 0.4613)
91
+ [11:32:09 + 68.2m] new best by OOD CE: step 1000
92
+ [11:33:14 + 69.3m] step 1020 loss 0.7309 (kd 0.0406 ce 0.6903) | 3.83s/step | table rows seen 2,853,812
93
+ [11:34:18 + 70.3m] step 1040 loss 0.6423 (kd 0.0686 ce 0.5737) | 3.82s/step | table rows seen 2,885,776
94
+ [11:35:20 + 71.3m] step 1060 loss 0.4495 (kd 0.0715 ce 0.3779) | 3.80s/step | table rows seen 2,903,460
95
+ [11:36:33 + 72.6m] step 1080 loss 0.5808 (kd 0.0770 ce 0.5038) | 3.80s/step | table rows seen 2,935,096
96
+ [11:37:42 + 73.7m] step 1100 loss 0.7386 (kd 0.1185 ce 0.6201) | 3.80s/step | table rows seen 2,969,560
97
+ [11:38:56 + 75.0m] step 1120 loss 0.6909 (kd 0.1252 ce 0.5657) | 3.79s/step | table rows seen 3,002,668
98
+ [11:40:07 + 76.1m] step 1140 loss 0.8193 (kd 0.0929 ce 0.7264) | 3.79s/step | table rows seen 3,041,248
99
+ [11:41:19 + 77.3m] step 1160 loss 0.8202 (kd 0.0786 ce 0.7416) | 3.79s/step | table rows seen 3,079,888
100
+ [11:42:26 + 78.4m] step 1180 loss 0.5295 (kd 0.0968 ce 0.4327) | 3.78s/step | table rows seen 3,110,948
101
+ [11:43:43 + 79.7m] step 1200 loss 0.6047 (kd 0.0895 ce 0.5151) | 3.78s/step | table rows seen 3,145,476
102
+ [11:44:01 + 80.0m] held-out CE @ 1200: SFT 1.2005 (start 1.2332) | OOD 2.1059 (start 2.1218)
103
+ [11:44:47 + 80.8m] PLE gain @ 1200: SFT -0.0006 | OOD -0.0013 | IN-corpus -0.0003 | extra +0.0010 (CE 1.2472) | extra2 -0.0016 (CE 0.7709) (CE off minus on; IN = training text the table may have seen) | live rows -1
104
+ [11:44:48 + 80.8m] HC stats @ 1200: gate|dev| median 0.0262 (L0 0.0250, L39 0.0252) | inject w|dev| median 0.3612 | stream spread median 0.4431 (L39 0.4623)
105
+ [11:46:15 + 82.3m] step 1220 loss 0.5524 (kd 0.1075 ce 0.4449) | 3.84s/step | table rows seen 3,175,288
106
+ [11:47:23 + 83.4m] step 1240 loss 0.5718 (kd 0.1473 ce 0.4244) | 3.84s/step | table rows seen 3,198,044
107
+ [11:48:30 + 84.5m] step 1260 loss 0.7514 (kd 0.1489 ce 0.6025) | 3.83s/step | table rows seen 3,228,712
108
+ [11:49:38 + 85.7m] step 1280 loss 0.5710 (kd 0.0672 ce 0.5038) | 3.82s/step | table rows seen 3,257,116
109
+ [11:50:50 + 86.8m] step 1300 loss 0.6990 (kd 0.0750 ce 0.6240) | 3.82s/step | table rows seen 3,285,048
110
+ [11:52:00 + 88.0m] step 1320 loss 0.6443 (kd 0.1220 ce 0.5224) | 3.81s/step | table rows seen 3,311,212
111
+ [11:53:05 + 89.1m] step 1340 loss 0.6353 (kd 0.0703 ce 0.5650) | 3.80s/step | table rows seen 3,342,864
112
+ [11:54:22 + 90.4m] step 1360 loss 0.7932 (kd 0.0629 ce 0.7303) | 3.81s/step | table rows seen 3,380,896
113
+ [11:55:38 + 91.7m] step 1380 loss 0.8595 (kd 0.0670 ce 0.7925) | 3.80s/step | table rows seen 3,416,000
114
+ [11:56:48 + 92.8m] step 1400 loss 0.7354 (kd 0.1172 ce 0.6182) | 3.80s/step | table rows seen 3,444,364
115
+ [11:57:07 + 93.1m] held-out CE @ 1400: SFT 1.1987 (start 1.2332) | OOD 2.1061 (start 2.1218)
116
+ [11:57:53 + 93.9m] PLE gain @ 1400: SFT -0.0026 | OOD -0.0019 | IN-corpus +0.0028 | extra +0.0028 (CE 1.2421) | extra2 -0.0011 (CE 0.7665) (CE off minus on; IN = training text the table may have seen) | live rows -1
117
+ [11:57:53 + 93.9m] HC stats @ 1400: gate|dev| median 0.0263 (L0 0.0251, L39 0.0252) | inject w|dev| median 0.3681 | stream spread median 0.4457 (L39 0.4620)
118
+ [11:59:14 + 95.3m] step 1420 loss 0.6459 (kd 0.1824 ce 0.4636) | 3.85s/step | table rows seen 3,464,920
119
+ [12:00:13 + 96.2m] step 1440 loss 0.4821 (kd 0.1090 ce 0.3730) | 3.84s/step | table rows seen 3,482,360
120
+ [12:01:16 + 97.3m] step 1460 loss 0.5379 (kd 0.1001 ce 0.4378) | 3.83s/step | table rows seen 3,503,412
121
+ [12:02:20 + 98.3m] step 1480 loss 0.6725 (kd 0.0634 ce 0.6091) | 3.82s/step | table rows seen 3,530,532
122
+ [12:03:23 + 99.4m] step 1500 loss 0.7896 (kd 0.0953 ce 0.6943) | 3.81s/step | table rows seen 3,558,028
123
+ [12:04:35 +100.6m] step 1520 loss 0.9037 (kd 0.1208 ce 0.7829) | 3.81s/step | table rows seen 3,590,976
124
+ [12:05:41 +101.7m] step 1540 loss 0.5257 (kd 0.0951 ce 0.4306) | 3.80s/step | table rows seen 3,610,096
125
+ [12:06:54 +102.9m] step 1560 loss 0.7672 (kd 0.0874 ce 0.6797) | 3.80s/step | table rows seen 3,640,052
126
+ [12:08:03 +104.1m] step 1580 loss 0.6485 (kd 0.0501 ce 0.5984) | 3.79s/step | table rows seen 3,664,720
127
+ [12:09:03 +105.1m] step 1600 loss 0.7833 (kd 0.0857 ce 0.6976) | 3.78s/step | table rows seen 3,685,324
128
+ [12:09:22 +105.4m] held-out CE @ 1600: SFT 1.1939 (start 1.2332) | OOD 2.1030 (start 2.1218)
129
+ [12:10:07 +106.1m] PLE gain @ 1600: SFT -0.0016 | OOD -0.0024 | IN-corpus +0.0021 | extra +0.0009 (CE 1.2492) | extra2 -0.0022 (CE 0.7920) (CE off minus on; IN = training text the table may have seen) | live rows -1
130
+ [12:10:08 +106.1m] HC stats @ 1600: gate|dev| median 0.0262 (L0 0.0251, L39 0.0252) | inject w|dev| median 0.3768 | stream spread median 0.4489 (L39 0.4614)
131
+ [12:10:23 +106.4m] new best by OOD CE: step 1600
132
+ [12:11:36 +107.6m] step 1620 loss 0.5682 (kd 0.1067 ce 0.4614) | 3.83s/step | table rows seen 3,710,248
133
+ [12:12:46 +108.8m] step 1640 loss 0.6394 (kd 0.1132 ce 0.5263) | 3.83s/step | table rows seen 3,728,380
134
+ [12:14:01 +110.0m] step 1660 loss 0.5825 (kd 0.1675 ce 0.4150) | 3.83s/step | table rows seen 3,747,492
135
+ [12:15:08 +111.2m] step 1680 loss 0.6417 (kd 0.0755 ce 0.5662) | 3.82s/step | table rows seen 3,771,352
136
+ [12:16:21 +112.4m] step 1700 loss 0.6493 (kd 0.1300 ce 0.5193) | 3.82s/step | table rows seen 3,795,868
137
+ [12:17:21 +113.4m] step 1720 loss 0.6877 (kd 0.1092 ce 0.5785) | 3.81s/step | table rows seen 3,813,320
138
+ [12:18:35 +114.6m] step 1740 loss 0.4488 (kd 0.0915 ce 0.3573) | 3.81s/step | table rows seen 3,834,924
139
+ [12:19:48 +115.8m] step 1760 loss 0.6241 (kd 0.0452 ce 0.5789) | 3.81s/step | table rows seen 3,856,020
140
+ [12:21:01 +117.0m] step 1780 loss 0.5488 (kd 0.0976 ce 0.4512) | 3.81s/step | table rows seen 3,876,564
141
+ [12:22:09 +118.2m] step 1800 loss 0.6088 (kd 0.1294 ce 0.4794) | 3.80s/step | table rows seen 3,898,708
142
+ [12:22:28 +118.5m] held-out CE @ 1800: SFT 1.1941 (start 1.2332) | OOD 2.1071 (start 2.1218)
143
+ [12:23:13 +119.2m] PLE gain @ 1800: SFT +0.0002 | OOD -0.0026 | IN-corpus +0.0014 | extra +0.0021 (CE 1.2396) | extra2 +0.0005 (CE 0.8032) (CE off minus on; IN = training text the table may have seen) | live rows -1
144
+ [12:23:14 +119.2m] HC stats @ 1800: gate|dev| median 0.0263 (L0 0.0251, L39 0.0253) | inject w|dev| median 0.3799 | stream spread median 0.4477 (L39 0.4617)
145
+ [12:24:32 +120.6m] step 1820 loss 0.6009 (kd 0.0782 ce 0.5227) | 3.84s/step | table rows seen 3,915,052
146
+ [12:25:41 +121.7m] step 1840 loss 0.6403 (kd 0.0746 ce 0.5658) | 3.83s/step | table rows seen 3,933,800
147
+ [12:27:01 +123.0m] step 1860 loss 0.6936 (kd 0.1115 ce 0.5821) | 3.84s/step | table rows seen 3,962,652
148
+ [12:28:19 +124.3m] step 1880 loss 0.4930 (kd 0.1378 ce 0.3552) | 3.84s/step | table rows seen 3,979,596
149
+ [12:29:21 +125.4m] step 1900 loss 0.7025 (kd 0.0649 ce 0.6375) | 3.83s/step | table rows seen 4,001,452
150
+ [12:30:29 +126.5m] step 1920 loss 0.5513 (kd 0.0818 ce 0.4695) | 3.82s/step | table rows seen 4,020,968
151
+ [12:31:39 +127.7m] step 1940 loss 0.5622 (kd 0.1278 ce 0.4343) | 3.82s/step | table rows seen 4,037,968
152
+ [12:32:48 +128.8m] step 1960 loss 0.4978 (kd 0.0559 ce 0.4419) | 3.82s/step | table rows seen 4,058,888
153
+ [12:33:59 +130.0m] step 1980 loss 0.4295 (kd 0.0745 ce 0.3550) | 3.81s/step | table rows seen 4,077,176
154
+ [12:35:11 +131.2m] step 2000 loss 0.6343 (kd 0.0534 ce 0.5809) | 3.81s/step | table rows seen 4,099,124
155
+ [12:35:30 +131.5m] held-out CE @ 2000: SFT 1.1978 (start 1.2332) | OOD 2.1104 (start 2.1218)
156
+ [12:36:16 +132.3m] PLE gain @ 2000: SFT +0.0004 | OOD -0.0014 | IN-corpus +0.0020 | extra +0.0015 (CE 1.2458) | extra2 -0.0016 (CE 0.7929) (CE off minus on; IN = training text the table may have seen) | live rows -1
157
+ [12:36:16 +132.3m] HC stats @ 2000: gate|dev| median 0.0263 (L0 0.0250, L39 0.0252) | inject w|dev| median 0.3713 | stream spread median 0.4479 (L39 0.4624)
158
+ [12:37:45 +133.8m] step 2020 loss 0.7549 (kd 0.0819 ce 0.6730) | 3.85s/step | table rows seen 4,117,840
159
+ [12:38:57 +135.0m] step 2040 loss 0.5973 (kd 0.0826 ce 0.5147) | 3.85s/step | table rows seen 4,141,264
160
+ [12:40:09 +136.2m] step 2060 loss 0.5086 (kd 0.1375 ce 0.3711) | 3.85s/step | table rows seen 4,153,176
161
+ [12:41:25 +137.4m] step 2080 loss 0.5420 (kd 0.1249 ce 0.4172) | 3.85s/step | table rows seen 4,174,652
162
+ [12:42:37 +138.6m] step 2100 loss 0.5472 (kd 0.1469 ce 0.4004) | 3.84s/step | table rows seen 4,190,784
163
+ [12:43:52 +139.9m] step 2120 loss 0.5826 (kd 0.1137 ce 0.4689) | 3.84s/step | table rows seen 4,216,276
164
+ [12:45:04 +141.1m] step 2140 loss 0.4896 (kd 0.0770 ce 0.4126) | 3.84s/step | table rows seen 4,234,552
165
+ [12:46:15 +142.3m] step 2160 loss 0.6703 (kd 0.1183 ce 0.5520) | 3.84s/step | table rows seen 4,254,552
166
+ [12:47:28 +143.5m] step 2180 loss 0.6259 (kd 0.0957 ce 0.5302) | 3.84s/step | table rows seen 4,271,780
167
+ [12:48:35 +144.6m] step 2200 loss 0.6595 (kd 0.1070 ce 0.5525) | 3.83s/step | table rows seen 4,291,940
168
+ [12:48:54 +144.9m] held-out CE @ 2200: SFT 1.2008 (start 1.2332) | OOD 2.1082 (start 2.1218)
169
+ [12:49:40 +145.7m] PLE gain @ 2200: SFT -0.0035 | OOD +0.0006 | IN-corpus +0.0028 | extra +0.0000 (CE 1.2435) | extra2 -0.0014 (CE 0.8040) (CE off minus on; IN = training text the table may have seen) | live rows -1
170
+ [12:49:40 +145.7m] HC stats @ 2200: gate|dev| median 0.0263 (L0 0.0250, L39 0.0253) | inject w|dev| median 0.3786 | stream spread median 0.4449 (L39 0.4601)
171
+ [12:49:55 +145.9m] TIME BUDGET: stopping at step 2201
172
+ [12:49:55 +145.9m] DONE best step 1600 OOD 2.1030 (start 2.1218) | checkpoints in /content/work/colab2/ckpts | table /content/work/colab2/ngram_table.npy