Phase 46: overfitting audit - luck bounds, certain pool; ADOPT cert-first (k=3) ordering, all-48 1661.7 -> 1623.7, 3W/0L/45T

EXPERIMENTS.md

@@ -2368,3 +2368,130 @@ a signal J2W computes that is outside this feature space entirely, or the
 forward-reading of the split file that phase 43(a) verified and declined.
 
 Standing champion: **3c7b883 at 1661.45**. Deadline 2026-08-23.
+
+## Phase 46 - the blind champion itself, as a whole: REJECTED (lineage fully adjudicated)
+
+Question (user, 2026-08-22): the blind-repo multi-agent loop's shared champion
+(`9a969a6a7dfe.py`, blind full-40 1566.7) - reusable here? The headline number
+is not on our scale (its own 40-origin harness; ~55 of its margin was extra cut
+depth), and most pieces were already adjudicated: CV discipline ported (40),
+`vrel`/`vmin7_rank` as ADDITIONS rejected (41), in-fleet pooling a public wash
+(42), trip-merge already native as building batching + group-day search
+(phases 1/14). One piece had never been tested: the champion's exact 4-feature
+set **as a replacement ranker** - `FEATURES=[last_voltage, vmin7, n_obs, vrel]`,
+mapped here to `[v_last, v_min_7, n_30, vrel]` (`blind4` in cv.VARIANTS).
+
+### Fixed-N ablation: a real but small head-ordering gain
+
+Paired vs base, deaths caught, bootstrap 95% CI. `v_last_only` cross-check
+reproduces phase 41 (-39 [-56,-21] native N=12 vs published -39 [-57,-22]).
+
+| variant | native N=8 | native N=12 | static N=8 | static N=12 |
+|---|---|---|---|---|
+| `blind4` | +11 [-1,+23] | +9 [-8,+26] | **+27 [+7,+47]** | +14 [-2,+31] |
+| `blind3` (no vrel) | +14 | +10 | +22 | +12 |
+
+One CI of eight excludes zero (static N=8), signs consistent at N<=12 on both
+instruments, and rec@30 / miss-rank get WORSE (57 -> 65 native): the small set
+trades deep tail for head. `blind3` matching `blind4` everywhere settles the
+phase-41 question from the other side: **vrel is not load-bearing; the gain is
+the small set itself** - 4 features against effective n ~80 death-batteries,
+where 13 features overfit slightly (phase 44's C-sweep said the same thing
+through the regularisation knob).
+
+### Full harness: the ordering gain does not survive the planner
+
+| all48 | total | sched | caught/454 | early | late |
+|---|---|---|---|---|---|
+| base | 1661.7 | 13.4 | 291 | 473.1 | 987.5 |
+| blind4, alpha 0.8 | 1765.4 | 14.6 | 298 | 584.0 | 948.3 |
+| blind4, count-matched alpha 0.7342 | 1740.7 | 13.7 | 290 | 530.1 | 972.3 |
+
+Raw: +103.8/origin - the 4-feature calibration inflates `sum(p42)`, the exact
+depth confound phase 41 diagnosed, in mirror image. Count-matched: **+79.1
+[-65,+231], 22W/26L, catches identical (290 vs 291)** - the +9 fixed-N catches
+vanish under the real inclusion rule (topsum + break-even top-up + floor is not
+top-N), and the whole residual is placement: +57 early-days/origin, because
+day placement integrates the p-curve and a 4-feature curve is cruder than a
+13-feature one. Ordering is only half the score's job here; timing is the
+other half, and the small set is worse at it.
+
+**Verdict: reject; nothing in the blind lineage is left unported or untested.**
+The blind champion's margin decomposes fully into (a) its own harness's depth
+and threshold tuning - non-transferable, appetite closed; (b) mechanisms we
+already have natively; (c) a head-ordering effect real at fixed N but worth
+zero catches and negative cost through this planner. Runnable via `blind4` /
+`blind3` in `cv.VARIANTS`.
+
+Standing champion: **3c7b883 at 1661.45**. Deadline 2026-08-23.
+
+## Phase 46 - the overfitting question, audited; and the certainty-first adoption
+
+User hypothesis: the leaders are overfitting the public board. Audited with a
+4-way measurement (certain-pool size, our capture of it, the fix on both
+instruments, luck bounds). Verdict: **partially right, and the audit found a
+real local gain on the way.**
+
+### How much of the board is luck
+
+The board keeps each account's best public score, so it best-of-N selects
+every team's luckiest config. From our own per-origin variance (sd 838 over 48
+origins -> sd of the 48-scenario mean = 121), the expected luck of keeping the
+best of N equal-skill submissions is 217/260/296/338/367 points for
+N=5/10/20/50/100.
+
+- **J2W's 210-point lead over rank 2 needs only N~4.5.** The ordering WITHIN
+  the leading pack is plausibly board luck.
+- **The pack's ~500-point lead over us would need N~5300.** Not luck.
+
+### The pack's edge is real - and it is not legitimately reachable by us
+
+The certain pool (devices whose gated daily median has sustained < 2.40 V for
+3 consecutive gated readings BEFORE the origin, i.e. the label rule already
+fired but the EOL has not landed - the wobble/lag population):
+
+| rule at origin | flagged/scen | die in-window | die by +90d | never die |
+|---|---|---|---|---|
+| 2.40 sustained-3 confirmed | 1.06 | 94.4% | 100% | **0** |
+| 2.45 zone (days_below_245>=0) | 21.8 | 24.6% | - | 62% |
+
+So the free-certainty pool is ~1/scenario, of which our champion already ranks
+42/51 in the top 12. The leaders' ~2.2 extra catches/scenario cannot be built
+from any past-only certainty rule we can construct: the implied J2W marginal
+precision is 0.69-0.88 (robust across all 9 inversion assumptions), on picks
+that for us measure ~0% transfer precision. Either they hold a tail-ranking
+signal outside this feature space entirely, or they are reading outcomes (the
+phase-43a forward-read, whose ceiling - 84% recall, 1.6 misses/scen - they sit
+comfortably below at 2.45 misses; or battery-toggling against public, ~1
+battery learned per submission pair). Given phases 41-45 measured every
+legitimate axis at ~zero, the balance of evidence is that the top of the board
+is exploiting the setup, not merely lucky. Unprovable from here; recorded as
+an assessment, not a fact.
+
+### ADOPTED: certainty-first ordering (cert-first, k=3)
+
+The audit's by-product. Force confirmed-crossing devices to the top of the
+ranking (rank_score = 2 + p42; count and timing untouched):
+
+| | all-48 | panel | W/L/T | mechanism |
+|---|---|---|---|---|
+| first-reading rule (LEAKS 2 future readings) | -88.2 | -124.5 | 6/0/42 | struck |
+| **confirmed rule (past-only)** | **-37.9** | **-53.6** | **3/0/45** | adopted |
+| k=1 / k=2 confirmation | +290.7 / +199.5 | - | 9/30 / 9/31 | recoveries flood the head |
+
+The confirmation depth is load-bearing and equals the label rule's own sustain
+parameter - k=3 is the unique safe point. Fixed-N instruments agree: native +9
+catches at N=12 (6W/0L), deployment gate +6 (6W/0L, ties elsewhere) - loss-free
+on every instrument at every N tested. The wins are label-lag devices our
+model under-ranks BECAUSE they recovered (v_last back up) - exactly the wobble
+population phase 39 characterised.
+
+Deployment note: at runtime `iterate_scenarios` hands truncated data, so the
+rule is past-only by construction there; the local strict measurement is the
+honest predictor. Wired into submit.py (`BATTERYSWAP_CERT_FIRST`, default on,
+both branches; stacks on top of cliff's rank_score if that is ever enabled;
+p42_sum untouched so the count rule is byte-identical).
+
+Local champion is now **cert-first at 1623.7** (from 1661.7). Public look not
+yet spent; submission is the user's call. Expected public value if the wobble
+rate transfers: ~-35 +/- (fires 0.46 devices/scenario locally).

batteryswap_example/submit.py

@@ -152,6 +152,35 @@ CLIFF_SCALE = float(os.environ.get('BATTERYSWAP_CLIFF_SCALE', '1.0'))
 CLIFF_FRAC = 0.35
 CLIFF_V = 2.46
 CLIFF_MIN_N = 20
+# Phase 46: certainty-first ordering. The label process (reverse-engineered,
+# 81/82 exact) stamps EOL ~3d after the gated daily-median voltage first
+# sustains < 2.40 V for 3 consecutive gated readings. A device whose crossing
+# is FULLY CONFIRMED in the past yet is still in the fleet is a label-lag /
+# wobble case: locally 54 such flags, 51 die in-window, 3 die by +90d, ZERO
+# never die. Boosting them to the top of the ordering (rank only - count and
+# timing untouched) is -37.9 on the all-48 cost harness, 3W/0L/45T. The
+# confirmation depth is load-bearing: k=1 (+291) and k=2 (+200) admit
+# recoveries; k=3 - the label rule's own sustain - is the unique safe point.
+# Past-only by construction: the flag requires all 3 readings <= T.
+CERT_FIRST = os.environ.get('BATTERYSWAP_CERT_FIRST', '1') != '0'
+CERT_V = 2.40
+CERT_SUS = 3
+
+
+def cert_confirm_days(daily):
+    """Per device: the day of the CERT_SUS-th consecutive gated daily-median
+    reading below CERT_V (the confirmation day), NaT-free Series. A device is
+    certainty-flagged at origin T iff its confirmation day <= T, which uses
+    only readings at or before T."""
+    out = {}
+    for dev, g in daily.sort_values(['device_id', 'day']).groupby(
+            'device_id', sort=False):
+        b = (g['v'] < CERT_V).to_numpy()
+        for i in range(len(b) - CERT_SUS + 1):
+            if b[i:i + CERT_SUS].all():
+                out[dev] = g['day'].iloc[i + CERT_SUS - 1]
+                break
+    return pd.Series(out, dtype='datetime64[ns]')
 
 
 def build_split_table(locations, eol_times, scen_starts, daily, dall, betas,
@@ -246,6 +275,7 @@ def adaptive_split_plans(split, split_path, log=print, deadline=None):
     locations, timeseries, eol_times, scenarios = load_best_effort(split_path, log)
     scen_starts = [pd.Timestamp(s['start_time']) for s in scenarios]
     daily = daily_voltage(timeseries)
+    cert_days = cert_confirm_days(daily) if CERT_FIRST else pd.Series(dtype='datetime64[ns]')
     dall = daily_all(timeseries)
     betas = weekly_betas(dall)
 
@@ -399,6 +429,19 @@ def adaptive_split_plans(split, split_path, log=print, deadline=None):
             if n_cliff:
                 log(f'  {scenario["name"]}: cliff boost x{CLIFF_SCALE} on '
                     f'{n_cliff} knee devices')
+        if CERT_FIRST and len(cert_days):
+            ch = cert_days.reindex(pred.index)
+            cert = (ch.notna() & (ch <= T)).to_numpy()
+            if cert.any():
+                # rank only: stacks on top of whatever rank_score exists
+                # (cliff, if enabled); p42 and p42_sum are untouched, so the
+                # inclusion count and every timing curve stay byte-identical
+                base_rs = (pred['rank_score'] if 'rank_score' in pred.columns
+                           else pred['p42']).astype(float).to_numpy().copy()
+                base_rs[cert] += 2.0
+                pred['rank_score'] = base_rs
+                log(f'  {scenario["name"]}: cert-first boost on '
+                    f'{int(cert.sum())} confirmed-crossing devices')
         plan_tsroute.alpha = ALPHA_STATIC if static_model is not None else ALPHA
         plan_tsroute.ec_margin = EC_MARGIN
         plan_tsroute.ec_pmax = EC_PMAX