SkyIsNotGreen commited on
Commit
7674495
·
verified ·
1 Parent(s): 98fbf56

Correct the card before publication: withdrawal of the invalidated drift numbers and the rejected threshold recommendation

Browse files
Files changed (1) hide show
  1. README.md +32 -19
README.md CHANGED
@@ -109,10 +109,12 @@ protocol measures the architecture rather than the generator.
109
 
110
  **One of the failures now has a candidate mechanism.** P10 (the drift
111
  reset) measures a detector whose threshold was calibrated for a different
112
- encoder (§7): forced on, it fires on 571 of 628 turns, so "the reset has
113
- no effect" may be measuring a detector that fires on almost everything
114
- rather than a reset that does not work. The verdict stands as measured;
115
- the mechanism is now in question, which is a different claim.
 
 
116
 
117
  **And the one result that does not carry over is P3's margin.** On the
118
  27B the curated context won 115 turns outright to 26; on the 35B it is a
@@ -255,6 +257,17 @@ protocol that only reports passes is not evidence.
255
 
256
  ## 7. The encoder: what we measured on our own traffic
257
 
 
 
 
 
 
 
 
 
 
 
 
258
  The harness gives its encoder three jobs, and only one of them is a ranking:
259
 
260
  | job | question | where |
@@ -286,7 +299,7 @@ not re-derived.
286
  | dedup at the shipped threshold (0.92) | precision 1.00, **recall 0.03** - the duplicate test almost never fires |
287
  | dedup at its own best threshold (~0.56) | F1 **0.75** (precision 0.62, recall 0.95) |
288
  | drift at the shipped threshold (0.6) | precision 0.08, **recall 0.96** - the topic-change test fires on almost every turn |
289
- | drift as a *ranking* signal | **AUC 0.945** - the signal is strong; the threshold is wrong |
290
 
291
  ### 7.1 The F1 is the wrong target, and the threshold is not the lever
292
 
@@ -314,20 +327,20 @@ the price of catching only 5% of the redundancy.
314
  The topic-drift test is the opposite case: it only *penalises* the
315
  off-topic chunks, it never deletes them, so its recalibration is free.
316
 
317
- **The model is the lever, not the threshold.** A small encoder tuned on
318
- the harness's own traffic (17M parameters, one epoch, the dedup task
319
- only, labelled with the same lexical rule) merges **six times more true
320
- duplicates** at the *same* precision - and the fact recall goes *up*
321
- slightly, not down:
322
-
323
- | encoder | fact recall | the dedup's recall at 0.92 | the dedup's precision |
324
- |---|---|---|---|
325
- | the stock L3-v2 | 50.55% | 0.046 | 1.00 |
326
- | the tuned encoder | **50.93%** | **0.292** | 1.00 |
327
-
328
- The encoder can be improved because it improves the *ranking* of what is a
329
- duplicate. The threshold cannot be lowered because lowering it moves the
330
- *boundary*, and the boundary is where the distinct facts get caught.
331
 
332
  ### 7.4 The curation's cost grows with the conversation it curates
333
 
 
109
 
110
  **One of the failures now has a candidate mechanism.** P10 (the drift
111
  reset) measures a detector whose threshold was calibrated for a different
112
+ encoder (§7): forced on, it fires on most turns, so "the reset has no
113
+ effect" may be measuring a detector that fires far too often rather than
114
+ a reset that does not work. (The exact rate was measured on the
115
+ pre-correction corpus and is withdrawn with the rest of §7.) The verdict
116
+ stands as measured; the mechanism is now in question, which is a
117
+ different claim.
118
 
119
  **And the one result that does not carry over is P3's margin.** On the
120
  27B the curated context won 115 turns outright to 26; on the 35B it is a
 
257
 
258
  ## 7. The encoder: what we measured on our own traffic
259
 
260
+ > **Correction (2026-10-21).** The corpus behind this section included an
261
+ > ingested knowledge-base attachment: 17 of its 82 files were chunked
262
+ > `[block n/m]` documents (5,531 of 6,977 chunks), and one conversation
263
+ > was loaded from two state directories. The drift figures below (the
264
+ > 0.945 AUC and the "fires on 571 of 628 turns" mechanism) are
265
+ > **withdrawn**, and the tuned encoder was rejected by the replay (see
266
+ > below). The shipped encoder and the shipped 0.92 stay. The correction
267
+ > and the clean measurement are in `AB-RESULTS.md` §2C-septies; the
268
+ > retrieval results elsewhere in this card come from different runs and
269
+ > are unaffected.
270
+
271
  The harness gives its encoder three jobs, and only one of them is a ranking:
272
 
273
  | job | question | where |
 
299
  | dedup at the shipped threshold (0.92) | precision 1.00, **recall 0.03** - the duplicate test almost never fires |
300
  | dedup at its own best threshold (~0.56) | F1 **0.75** (precision 0.62, recall 0.95) |
301
  | drift at the shipped threshold (0.6) | precision 0.08, **recall 0.96** - the topic-change test fires on almost every turn |
302
+ | drift as a *ranking* signal | **withdrawn** - measured on the pre-correction corpus (79% an ingested attachment), see above |
303
 
304
  ### 7.1 The F1 is the wrong target, and the threshold is not the lever
305
 
 
327
  The topic-drift test is the opposite case: it only *penalises* the
328
  off-topic chunks, it never deletes them, so its recalibration is free.
329
 
330
+ **The model is not the lever either.** A small encoder tuned on the
331
+ harness's own traffic (17M parameters, one epoch, the dedup task only)
332
+ raises the dedup decision at 0.92 from a recall of 0.046 to 0.986 on the
333
+ held-out job test. The replay then prices that decision: at the shipped
334
+ 0.92 the tuned encoder keeps **32.66%** of the facts against the stock's
335
+ **60.51%**, for a context 2.7x smaller. The decision improved; the
336
+ curation nearly halved. The shipped encoder and the shipped 0.92 stay.
337
+
338
+ The reason is structural: a true duplicate merged is a fact deleted from
339
+ the context, so improving the dedup decision is improving the deletion.
340
+ The threshold and the model are two dials on the same trade-off curve,
341
+ and the fact-preservation constraint fixes where on it the system sits.
342
+ The clean replay and the three failed fine-tunes are in `AB-RESULTS.md`
343
+ §2C-quater through §2C-septies.
344
 
345
  ### 7.4 The curation's cost grows with the conversation it curates
346