lw4 s2916: MATH-60, parity, CE curve, logs

lw4/eval/export_lw4s2916.log

@@ -0,0 +1,6 @@
+merged LoRA/shared into 330 tensors
+carried 612 tensors from /content/work/train/base-sigmoid, folded 81 norms into HC
+/content/work/kit/code/wt/export_next36.py:93: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:213.)
+  t = torch.from_numpy(np.ascontiguousarray(a))
+PLE: 8 heads x ~4,880,000 rows x 256 = 9.99B params in 5 shards, width 2048 into hidden 2048, injected at layer 2
+EXPORT_DONE /dev/shm/export_lw4 tensors=979 shards=14 gate=sigmoid ple_layer=2

lw4/eval/lw4_ce_curve.txt

@@ -0,0 +1,73 @@
+[11:33:20 +  3.0m]   PLE gain @ 0: SFT +0.3888 | OOD +0.9896  (CE with memory OFF minus ON on 8+8 held rows; positive = the table helps)
+[11:47:06 + 16.8m]   held-out CE @ 200: SFT 1.2196 (start 1.2289) | OOD 2.1410 (start 2.1543)
+[11:47:06 + 16.8m]   PoSE @ 200: 224 rows offset | highest position index 130,926 of 131,072
+[11:47:52 + 17.5m]   PLE gain @ 200: SFT +0.0580 | OOD +0.4936 | IN-corpus +2.1793 | extra +1.6923 (CE 1.2575) | extra2 +0.0204 (CE 0.7791) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[11:47:52 + 17.5m]   injection @ 200: 79 rows | PLE/residual ratio mean 0.2947 min 0.2452 max 0.3947 | mean use term 0.0000
+[11:47:52 + 17.5m]   dependence @ 200: 24 rows since last eval | mean gap CE_off-CE_on +2.4045 | frac meeting margin 1.00 | mean term 0.0000
+[12:02:59 + 32.6m]   held-out CE @ 400: SFT 1.2195 (start 1.2289) | OOD 2.1397 (start 2.1543)
+[12:02:59 + 32.6m]   PoSE @ 400: 453 rows offset | highest position index 130,926 of 131,072
+[12:03:45 + 33.4m]   PLE gain @ 400: SFT +0.4164 | OOD +0.9463 | IN-corpus +4.2369 | extra +3.9105 (CE 1.2542) | extra2 +0.3152 (CE 0.7973) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[12:03:45 + 33.4m]   injection @ 400: 104 rows | PLE/residual ratio mean 0.2955 min 0.2239 max 0.5211 | mean use term 0.0000
+[12:03:45 + 33.4m]   dependence @ 400: 29 rows since last eval | mean gap CE_off-CE_on +2.3722 | frac meeting margin 0.97 | mean term 0.0061
+[12:19:05 + 48.7m]   held-out CE @ 600: SFT 1.2190 (start 1.2289) | OOD 2.1397 (start 2.1543)
+[12:19:05 + 48.7m]   PoSE @ 600: 683 rows offset | highest position index 130,926 of 131,072
+[12:19:50 + 49.5m]   PLE gain @ 600: SFT +0.0080 | OOD +0.2472 | IN-corpus +3.1311 | extra +2.6947 (CE 1.2511) | extra2 +0.0157 (CE 0.7923) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[12:19:50 + 49.5m]   injection @ 600: 115 rows | PLE/residual ratio mean 0.2849 min 0.2054 max 0.3681 | mean use term 0.0000
+[12:19:50 + 49.5m]   dependence @ 600: 30 rows since last eval | mean gap CE_off-CE_on +3.7012 | frac meeting margin 1.00 | mean term 0.0000
+[12:33:57 + 63.6m]   held-out CE @ 800: SFT 1.2173 (start 1.2289) | OOD 2.1412 (start 2.1543)
+[12:33:57 + 63.6m]   PoSE @ 800: 900 rows offset | highest position index 130,926 of 131,072
+[12:34:43 + 64.4m]   PLE gain @ 800: SFT -0.0027 | OOD +0.1893 | IN-corpus +2.7950 | extra +2.1763 (CE 1.2491) | extra2 +0.0102 (CE 0.7720) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[12:34:43 + 64.4m]   injection @ 800: 92 rows | PLE/residual ratio mean 0.2891 min 0.1943 max 0.4813 | mean use term 0.0000
+[12:34:43 + 64.4m]   dependence @ 800: 17 rows since last eval | mean gap CE_off-CE_on +2.6320 | frac meeting margin 1.00 | mean term 0.0000
+[12:49:30 + 79.1m]   held-out CE @ 1000: SFT 1.2134 (start 1.2289) | OOD 2.1347 (start 2.1543)
+[12:49:30 + 79.1m]   PoSE @ 1000: 1,127 rows offset | highest position index 130,926 of 131,072
+[12:50:15 + 79.9m]   PLE gain @ 1000: SFT -0.0071 | OOD +0.1928 | IN-corpus +2.6352 | extra +2.0857 (CE 1.2442) | extra2 +0.0135 (CE 0.7755) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[12:50:15 + 79.9m]   injection @ 1000: 83 rows | PLE/residual ratio mean 0.2775 min 0.2219 max 0.4256 | mean use term 0.0000
+[12:50:15 + 79.9m]   dependence @ 1000: 27 rows since last eval | mean gap CE_off-CE_on +2.2593 | frac meeting margin 1.00 | mean term 0.0000
+[13:05:41 + 95.3m]   held-out CE @ 1200: SFT 1.2126 (start 1.2289) | OOD 2.1324 (start 2.1543)
+[13:05:41 + 95.3m]   PoSE @ 1200: 1,359 rows offset | highest position index 130,926 of 131,072
+[13:06:27 + 96.1m]   PLE gain @ 1200: SFT -0.0057 | OOD +0.1095 | IN-corpus +1.9552 | extra +1.3675 (CE 1.2411) | extra2 +0.0010 (CE 0.7609) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[13:06:27 + 96.1m]   injection @ 1200: 103 rows | PLE/residual ratio mean 0.2752 min 0.1922 max 0.3752 | mean use term 0.0000
+[13:06:27 + 96.1m]   dependence @ 1200: 32 rows since last eval | mean gap CE_off-CE_on +2.2328 | frac meeting margin 1.00 | mean term 0.0000
+[13:21:06 +110.8m]   held-out CE @ 1400: SFT 1.2128 (start 1.2289) | OOD 2.1350 (start 2.1543)
+[13:21:06 +110.8m]   PoSE @ 1400: 1,582 rows offset | highest position index 130,926 of 131,072
+[13:21:52 +111.5m]   PLE gain @ 1400: SFT +0.2496 | OOD +0.7552 | IN-corpus +4.5962 | extra +4.3200 (CE 1.2483) | extra2 +0.3010 (CE 0.7287) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[13:21:52 +111.5m]   injection @ 1400: 96 rows | PLE/residual ratio mean 0.2822 min 0.1875 max 0.4081 | mean use term 0.0000
+[13:21:52 +111.5m]   dependence @ 1400: 23 rows since last eval | mean gap CE_off-CE_on +1.4063 | frac meeting margin 0.96 | mean term 0.0013
+[13:36:08 +125.8m]   held-out CE @ 1600: SFT 1.2093 (start 1.2289) | OOD 2.1280 (start 2.1543)
+[13:36:08 +125.8m]   PoSE @ 1600: 1,804 rows offset | highest position index 130,926 of 131,072
+[13:36:54 +126.6m]   PLE gain @ 1600: SFT -0.0032 | OOD +0.1137 | IN-corpus +2.4953 | extra +1.9238 (CE 1.2418) | extra2 +0.0053 (CE 0.7656) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[13:36:54 +126.6m]   injection @ 1600: 92 rows | PLE/residual ratio mean 0.2741 min 0.2121 max 0.3890 | mean use term 0.0000
+[13:36:54 +126.6m]   dependence @ 1600: 22 rows since last eval | mean gap CE_off-CE_on +3.5486 | frac meeting margin 1.00 | mean term 0.0000
+[13:52:10 +141.8m]   held-out CE @ 1800: SFT 1.2126 (start 1.2289) | OOD 2.1339 (start 2.1543)
+[13:52:10 +141.8m]   PoSE @ 1800: 2,030 rows offset | highest position index 130,926 of 131,072
+[13:52:56 +142.6m]   PLE gain @ 1800: SFT -0.0035 | OOD +0.2057 | IN-corpus +2.6217 | extra +2.1397 (CE 1.2462) | extra2 +0.0102 (CE 0.7946) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[13:52:56 +142.6m]   injection @ 1800: 111 rows | PLE/residual ratio mean 0.2689 min 0.2120 max 0.3384 | mean use term 0.0000
+[13:52:56 +142.6m]   dependence @ 1800: 26 rows since last eval | mean gap CE_off-CE_on +1.8347 | frac meeting margin 1.00 | mean term 0.0000
+[14:07:59 +157.6m]   held-out CE @ 2000: SFT 1.2136 (start 1.2289) | OOD 2.1255 (start 2.1543)
+[14:07:59 +157.6m]   PoSE @ 2000: 2,255 rows offset | highest position index 130,926 of 131,072
+[14:08:45 +158.4m]   PLE gain @ 2000: SFT -0.0043 | OOD +0.0619 | IN-corpus +1.8375 | extra +1.3190 (CE 1.2488) | extra2 +0.0006 (CE 0.7583) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[14:08:45 +158.4m]   injection @ 2000: 103 rows | PLE/residual ratio mean 0.2740 min 0.1727 max 0.4025 | mean use term 0.0000
+[14:08:45 +158.4m]   dependence @ 2000: 25 rows since last eval | mean gap CE_off-CE_on +1.3632 | frac meeting margin 1.00 | mean term 0.0000
+[14:23:21 +173.0m]   held-out CE @ 2200: SFT 1.2108 (start 1.2289) | OOD 2.1312 (start 2.1543)
+[14:23:21 +173.0m]   PoSE @ 2200: 2,484 rows offset | highest position index 130,926 of 131,072
+[14:24:07 +173.8m]   PLE gain @ 2200: SFT +0.3643 | OOD +0.8212 | IN-corpus +3.4527 | extra +3.2732 (CE 1.2481) | extra2 +0.3525 (CE 0.7408) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[14:24:07 +173.8m]   injection @ 2200: 99 rows | PLE/residual ratio mean 0.2739 min 0.1941 max 0.3555 | mean use term 0.0000
+[14:24:07 +173.8m]   dependence @ 2200: 29 rows since last eval | mean gap CE_off-CE_on +1.7263 | frac meeting margin 0.97 | mean term 0.0078
+[14:39:12 +188.8m]   held-out CE @ 2400: SFT 1.2100 (start 1.2289) | OOD 2.1262 (start 2.1543)
+[14:39:12 +188.8m]   PoSE @ 2400: 2,710 rows offset | highest position index 130,926 of 131,072
+[14:39:57 +189.6m]   PLE gain @ 2400: SFT +0.0842 | OOD +0.4549 | IN-corpus +2.7320 | extra +2.2620 (CE 1.2502) | extra2 +0.0598 (CE 0.7463) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[14:39:57 +189.6m]   injection @ 2400: 99 rows | PLE/residual ratio mean 0.2701 min 0.2144 max 0.3836 | mean use term 0.0000
+[14:39:57 +189.6m]   dependence @ 2400: 26 rows since last eval | mean gap CE_off-CE_on +2.9735 | frac meeting margin 1.00 | mean term 0.0000
+[14:55:02 +204.7m]   held-out CE @ 2600: SFT 1.2142 (start 1.2289) | OOD 2.1310 (start 2.1543)
+[14:55:02 +204.7m]   PoSE @ 2600: 2,935 rows offset | highest position index 130,926 of 131,072
+[14:55:48 +205.4m]   PLE gain @ 2600: SFT +0.1178 | OOD +0.5192 | IN-corpus +2.7474 | extra +2.2613 (CE 1.2495) | extra2 +0.0682 (CE 0.7462) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[14:55:48 +205.4m]   injection @ 2600: 98 rows | PLE/residual ratio mean 0.2662 min 0.1901 max 0.3733 | mean use term 0.0000
+[14:55:48 +205.4m]   dependence @ 2600: 25 rows since last eval | mean gap CE_off-CE_on +2.0822 | frac meeting margin 1.00 | mean term 0.0000
+[15:11:01 +220.7m]   held-out CE @ 2800: SFT 1.2057 (start 1.2289) | OOD 2.1330 (start 2.1543)
+[15:11:01 +220.7m]   PoSE @ 2800: 3,165 rows offset | highest position index 130,926 of 131,072
+[15:11:47 +221.4m]   PLE gain @ 2800: SFT +0.0249 | OOD +0.3150 | IN-corpus +2.3282 | extra +1.7017 (CE 1.2511) | extra2 +0.0201 (CE 0.7495) (CE off minus on; IN = training text the table may have seen) | live rows -1
+[15:11:47 +221.4m]   injection @ 2800: 101 rows | PLE/residual ratio mean 0.2665 min 0.1969 max 0.4240 | mean use term 0.0000
+[15:11:47 +221.4m]   dependence @ 2800: 30 rows since last eval | mean gap CE_off-CE_on +1.9068 | frac meeting margin 1.00 | mean term 0.0000
+[15:20:22 +230.0m] TIME BUDGET: stopping at step 2917
+[15:21:28 +231.1m] DONE best step 2000 OOD 2.1255 (start 2.1543) | checkpoints in /content/work/lw4/ckpts | table /content/work/lw4/ngram_table.npy

lw4/eval/lw_eval_lw4s2916_held.json

@@ -0,0 +1,38 @@
+{
+ "tag": "lw4s2916_held",
+ "pairs": [
+  [
+   35,
+   59
+  ],
+  [
+   39,
+   63
+  ]
+ ],
+ "cache": "/content/work/probe/teacher_L59_L63_n400.npz",
+ "row0": 0,
+ "nseq": 8,
+ "seqlen": 512,
+ "topk": 32,
+ "positions": 480,
+ "model_dir": "/dev/shm/export_lw4",
+ "results": [
+  {
+   "student": 35,
+   "teacher": 59,
+   "agree": 0.40703125,
+   "dprob": 0.23550596833229065,
+   "teacher_mass": 0.705552339553833,
+   "student_mass": 0.7488309144973755
+  },
+  {
+   "student": 39,
+   "teacher": 63,
+   "agree": 0.8802083333333334,
+   "dprob": 0.07607011497020721,
+   "teacher_mass": 0.9730501770973206,
+   "student_mass": 0.9775782823562622
+  }
+ ]
+}

lw4/eval/lw_eval_lw4s2916_train.json

@@ -0,0 +1,38 @@
+{
+ "tag": "lw4s2916_train",
+ "pairs": [
+  [
+   35,
+   59
+  ],
+  [
+   39,
+   63
+  ]
+ ],
+ "cache": "/content/work/probe/teacher_L59_L63.npz",
+ "row0": 0,
+ "nseq": 8,
+ "seqlen": 512,
+ "topk": 32,
+ "positions": 480,
+ "model_dir": "/dev/shm/export_lw4",
+ "results": [
+  {
+   "student": 35,
+   "teacher": 59,
+   "agree": 0.44244791666666666,
+   "dprob": 0.2557154595851898,
+   "teacher_mass": 0.738440752029419,
+   "student_mass": 0.7690678238868713
+  },
+  {
+   "student": 39,
+   "teacher": 63,
+   "agree": 0.8484375,
+   "dprob": 0.0885397419333458,
+   "teacher_mass": 0.9738667607307434,
+   "student_mass": 0.979742169380188
+  }
+ ]
+}

lw4/eval/lw_lw4s2916_held.log

@@ -0,0 +1,13 @@
+[     0s] batch (8, 512) from teacher_L59_L63_n400.npz rows 0..8 | pairs [(35, 59), (39, 63)]
+[     0s] indexer budget 262,144 -> 512 for this 512-token probe (same visible tokens)
+[     0s] lw4s2916_held: 40 layers, hidden 2048, hc_count 4, attention at [3, 7, 11, 15, 19, 23, 27, 31, 35, 39]
+[     1s] streams (8, 512, 8192) | masks + rope built
+[     2s]   .. layer 0 done
+[     4s]   layer 1: n-gram memory streamed 39,040,000 rows into (39040640, 256) on cuda
+[     7s]   .. layer 8 done
+[    10s]   .. layer 16 done
+[    13s]   .. layer 24 done
+[    16s]   .. layer 32 done
+[    20s] lw4s2916_held S35 <- T59: top-1 agree 40.7% | dprob 0.236 | teacher-mass 70.6% | student-mass 74.9%  (3,840 positions)
+[    25s] lw4s2916_held S39 <- T63: top-1 agree 88.0% | dprob 0.076 | teacher-mass 97.3% | student-mass 97.8%  (3,840 positions)
+[    25s] LW_EVAL DONE lw4s2916_held: S35<-T59 agree 40.7% mass 70.6% | S39<-T63 agree 88.0% mass 97.3%

lw4/eval/lw_lw4s2916_train.log

@@ -0,0 +1,13 @@
+[     0s] batch (8, 512) from teacher_L59_L63.npz rows 0..8 | pairs [(35, 59), (39, 63)]
+[     0s] indexer budget 262,144 -> 512 for this 512-token probe (same visible tokens)
+[     0s] lw4s2916_train: 40 layers, hidden 2048, hc_count 4, attention at [3, 7, 11, 15, 19, 23, 27, 31, 35, 39]
+[     1s] streams (8, 512, 8192) | masks + rope built
+[     1s]   .. layer 0 done
+[     3s]   layer 1: n-gram memory streamed 39,040,000 rows into (39040640, 256) on cuda
+[     6s]   .. layer 8 done
+[     9s]   .. layer 16 done
+[    12s]   .. layer 24 done
+[    14s]   .. layer 32 done
+[    18s] lw4s2916_train S35 <- T59: top-1 agree 44.2% | dprob 0.256 | teacher-mass 73.8% | student-mass 76.9%  (3,840 positions)
+[    22s] lw4s2916_train S39 <- T63: top-1 agree 84.8% | dprob 0.089 | teacher-mass 97.4% | student-mass 98.0%  (3,840 positions)
+[    22s] LW_EVAL DONE lw4s2916_train: S35<-T59 agree 44.2% mass 73.8% | S39<-T63 agree 84.8% mass 97.4%

lw4/eval/math60_score.txt

@@ -0,0 +1 @@
+/content/work/lw4_gen/gen.jsonl: 42/60 correct (70%) | by level L2:18/20 L3:14/20 L4:10/20 | no_boxed 8 | cap_hit 8 | format {'ok': 52, 'cap_hit': 8} | mean 1629 tok, think ~385 tok (53 closed)