Phase 19-20: cold-start count floor (all-48 2146.9->1851.0) + CV harness upgrade

Gated floor k=2.5 wins warm-up 8/8, panel byte-identical. Ungated and
foreign-fleet variants refuted (panel +21..+114; fleet-holdout harness
shows the probability scale transfers across device subsets). Near-end
block s_42-47 added: top-up worth -273 there, censor pricing -64.
Three-regime scorecard codified; tie-aware selection stability;
component decomposition by default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

EXPERIMENTS.md

@@ -10,9 +10,29 @@ All numbers are mean `total_cost` per scenario from the official `evaluate_plan`
   use only data ≤ origin; the label is death within the 42-day window after it.
 - **Walk-forward**: at every scored origin the model is refit on all pairs whose
   window *closed* before that origin (fully observed labels, built-in embargo).
-- **Blocks**: warm-up s_0–s_7 (discarded: 3–20 known deaths), **dev s_8–s_23**
-  (16 origins, all tuning), **test s_24–s_41** (18 origins, counted looks only),
-  s_42–s_47 (discarded: windows extend past data end 2026-07-31).
+- **Current protocol** (since the 2026-08-18 protocol change, after phase 10):
+  champion selection on the merged **34-origin panel s_8–s_41** with
+  repeated-split stability - accept a challenger only if selected in ≥60% of
+  2000 random half-splits AND modal under blocked splits; the public
+  leaderboard is the only remaining holdout and is spent deliberately.
+- **Three-regime scorecard** (since phase 20): the panel alone samples one
+  regime (native fleet, mature calibration, expensive false swaps). Standing
+  additional blocks: **warm-up s_0–s_7** (cold-start calibration; where the
+  phase-19 floor was proven) and **near-end s_42–s_47** (censored shipped
+  truth - the same scoring the remote evaluator applies to a held-out split's
+  near-end scenarios; where the phase-11/14 mechanisms are measured).
+  Adoption of a deployment-regime mechanism requires: no panel regression
+  (tie-aware selection stability) AND a win in its target regime block AND
+  the regime being reachable by a local harness - "architecturally sound" is
+  not sufficient (see the foreign-fleet floor, built and refuted within one
+  day in phase 20). selection_stability counts decided splits only;
+  summarize() reports late/early/ops components by default.
+- **Other blocks**: warm-up s_0–s_7 (3–16 unique training deaths) is excluded
+  from champion selection but scored since phase 17 for cold-start behavior;
+  s_42–s_47 stay unscored (windows extend past data end 2026-07-31).
+- **Historical blocks (phases 1–10)**: dev s_8–s_23 (all tuning) / test
+  s_24–s_41 (counted looks only, ledger at the end); merged once 8 looks had
+  spent the test block. Per-phase "dev"/"test" columns below refer to these.
 - **Reference points** (test block): shipped example 6779 · defer-everything 2512
   · lowest-voltage-15 heuristic 2034 · oracle (true EOL) 186.
 
@@ -586,6 +606,115 @@ model-curve error rather than real savings. The knob
 This completes the phase-17 plan: all five directions adjudicated and
 rejected. Champion unchanged: panel 1679.2, all-48 Docker 2146.9.
 
+## Phase 17 - late-swap forensics: the misses are mostly NOT silent plungers (cold-start floor ADOPTED)
+
+Question: which batteries get replaced too late? Attributed the all-48
+submission's late cost through the evaluator's own transitions (its emergency
+loop books missed deaths as late_swap at 10/day, day ~43+):
+
+- 98% of late cost (73,430 of 74,850) is 263 parked deaths swapped as
+  post-window emergencies; only 1,420 is caught-but-late (deaths on days 1-7
+  swapped a few days after).
+- State at origin of the 263: **157 already below 2.45** (48,460 cost) · 71
+  in visible droop 2.45-2.55 (17,710) · only 35 healthy+fresh (7,260). The
+  phase-8-13 "silent plunger wall" is ~10% of the late cost, not the story.
+- (The repo submission.csv was the phase-5-era artifact, all-48 2202; all
+  numbers here use same-code regenerations.)
+
+Mechanism 1 - **cold-start count starvation** (s_0-s_8 = 46% of all-48 late):
+at warm-up origins the ranking is fine (AUC 0.97 at s_0; missed deaths at
+ranks 2-8) but sum(p42) undercounts prevalence ~4x (3-16 unique training
+deaths, lower-prevalence era), so n = 0.8*sum(p42) includes 1-8 batteries
+while 9-19 die - catch rate 0.00/0.00/0.06 at s_0/1/2. Invisible to every
+prior selection round: CV discards s_0-s_7 as warm-up. Cold-start late
+~700/scenario ~= the public-LB late gap to the leader (1296 vs 660) -
+phase 10's "not recoverable" verdict was derived on the test block only.
+
+Mechanism 2 - **label-lag suppression** (bulk of the remainder): long-crossed
+devices (59-275d below 2.45) score p42 0.001-0.3 - the phase-15 load-bearing
+negatives working as trained; the minority whose EOL does arrive in-window
+pays 300-500 each. Still open; LogbagLAH (KM crossing->EOL arrival curve) is
+the drafted candidate.
+
+Fix for mechanism 1 (ADOPTED): **gated inclusion floor** in plan_tsroute -
+psum_eff = max(sum(p42), k * prev_est), with prev_est = max(deaths recorded
+in the last 42d, fresh devices first crossing 2.45 within the last 42d),
+active only while unique observed training deaths < 20 (true exactly at
+s_0-s_7; the 34-origin panel is byte-identical to the champion by
+construction).
+
+| variant | warm-up s_0-s_7 mean |
+|---|---|
+| champion (no floor) | 4256.1 |
+| ungated k=1.0/1.5/2.0 | 3994/3741/3354 - leaks into the panel (+21/+95/+299; recent-deaths overshoots post-peak), rejected by selection stability (champion 89.5%/80%) |
+| gated, recorded-deaths only, k=1.5 | 3740.5 - recording lag blinds the label signal exactly at s_0-s_2 (recent42 = 1 while 9-17 die) |
+| gated + zone signal, k=1.0/1.5/2.0 | 3313/3014/2532 |
+| **gated + zone, k=2.5** | **2379.7** - wins 8/8 origins; k=3.0: 2619, k=4.0: 3191 (interior optimum) |
+
+Registered caveat: k and the gate threshold were chosen on the same 8 origins
+(no holdout exists there). Any k in [1.5, 3] recovers 29-44%, so the
+qualitative result is robust; the exact k=2.5 is not.
+
+End-to-end verification (submit.py adaptive path, floor vs same-code
+control): s_8-s_47 **byte-identical costs** (gate closed), warm-up 4764.2 ->
+2976.6 (-37.5%, 8/8 scenarios), **all-48 2146.6 -> 1848.7** (late 1465 ->
+1125, early 434 -> 515; the control reproduces the recorded Docker 2146.9).
+Champion and panel (1679.2) unchanged. Knobs: BATTERYSWAP_FLOOR (k, 0
+disables the cold-start branch), BATTERYSWAP_FLOOR_GATE. Held-out-split
+caveat: pooled training passes the gate earlier in calendar time there
+(bigger fleet, faster death accrual) and sum(p42) carries the training
+fleet's scale - the foreign-fleet floor branch in submit.py (own_frac gate)
+targets that regime; the local train split cannot exercise it.
+
+## Phase 19 - cold-start count floor (ADOPTED, gated) & ungated variants (rejected)
+
+Public resubmission of the phase-14 champion read **1717.8 (rank 3)**: late
+1296→1234, early +14 - the deployment mechanisms delivered ~-50, so public
+mostly does NOT sit in the cheap-FP regime, and the remaining -240 to the
+leader is late-swap conversion. Cold-start diagnosis (warm-up block): at
+s_0-s_7 the model RANKS deaths at the very top (AUC 0.97) but sum(p42)
+undercounts prevalence ~4x (3.8 when 17 die) - the count starves, not the
+ranking; 46% of the all-48 late cost. Fix: floor the count with observable
+signals - max(sum_p42, k x max(deaths recorded in last 42d, fresh
+2.45-crossings)) - gated to fire only while training has seen < 20 unique
+deaths. Warm-up CV: **4256 -> 2380 at k=2.5, winning 8/8 origins**; the
+34-origin panel is byte-identical (gate closed everywhere).
+All-48 Docker: **2146.9 -> 1851.0** (late 1465→1125).
+
+Ungated variants, panel-adjudicated and rejected: k=1.0 recent-only +20.9,
+k=1.0 +zone +75.1, k=1.25 +zone +113.9 (champion 96%/80% selection). Where
+sum(p42) is natively calibrated it beats the empirical death rate (deaths are
+bursty; last-42d != next-42d), and the death-zone signal counts never-dying
+limpers. Sixth refutation of explicit count/label correctors in calibrated
+regimes.
+
+## Phase 20 - CV-harness upgrade: the harness catches a blind ship (foreign floor refuted)
+
+Five CV improvements, all implemented:
+
+1. **Fleet-holdout harness** (train the pool on half the devices, plan/score
+   the full fleet): built to validate a "foreign-fleet floor" (fire the count
+   floor when training positives are mostly another fleet's - the public
+   early-origin hypothesis) that had been wired into submit.py on
+   architectural grounds. The harness REFUTED it the same day: the
+   probability scale transfers across device subsets (half-fleet training:
+   sum(p42) 15.8 vs 9.1 true deaths - it OVER-counts if anything); the floor
+   was inert on 15/17 origins and lost where it fired (+15.1, 0 wins).
+   Branch removed before it could cost a submission. Public late-heaviness is
+   therefore NOT foreign-scale starvation.
+2. **Near-end block s_42-s_47** (censored shipped truth, previously
+   discarded): native retro-adjudication of the deployment mechanisms -
+   phase-11 top-up worth **-273** (877.6 -> 604.8), phase-14 censoring-aware
+   pricing **-64** (669.2 -> 604.8). Now a standing block.
+3. **Three-regime scorecard** codified in the protocol section.
+4. **selection_stability** counts decided splits only (all-tie splits no
+   longer break alphabetically to the incumbent).
+5. **summarize()** reports late/early/ops decomposition by default.
+
+Champion: logbag / tsroute a0.8 + top-up (m0.8/pm25) + sd7 + censor-aware
+pricing + gated cold-start floor (k=2.5, gate 20). Panel 1679.2 (unchanged),
+warm-up 2380, near-end 604.8, all-48 Docker 1851.0, public 1717.8.
+
 ## Test-look ledger
 
 8 looks total (now historical - see protocol change above); look 9 = the

batteryswap_example/models_cv.py

@@ -1382,6 +1382,14 @@ def plan_tsroute(pred, included, locations, start_time, day_q,
            pay early-replacement for weeks [floor_day 0 = off]
     """
     psum = pred['p42_sum'].iloc[0] if 'p42_sum' in pred else pred['p42'].sum()
+    if plan_tsroute.floor_scale > 0:
+        # cold-start inclusion floor: sum(p42) tracks prevalence only once the
+        # training era has seen it - at warm-up origins it undercounts ~4x and
+        # the count rule parks visibly-dying batteries (46% of all-48 late
+        # cost). Deaths RECORDED in the last 42d are observable at plan time
+        # (the harness itself drops eol <= origin devices) and estimate the
+        # coming window's prevalence without trusting the probability scale.
+        psum = max(psum, plan_tsroute.floor_scale * plan_tsroute.recent_deaths)
     n = int(round(plan_tsroute.alpha * psum))
     order_col = 'rank_score' if 'rank_score' in pred.columns else 'p42'
     chosen = pred[order_col].nlargest(n).index
@@ -1450,6 +1458,8 @@ plan_tsroute.floor_day = 0
 plan_tsroute.p_conf = 0.5
 plan_tsroute.ec_margin = 0.0
 plan_tsroute.ec_pmax = 25.0
+plan_tsroute.floor_scale = 0.0   # 0 = off; k in max(sum_p42, k * recent_deaths)
+plan_tsroute.recent_deaths = 0.0  # deaths recorded in (origin-42d, origin]; set per origin
 
 def plan_tspig(pred, included, locations, start_time, day_q,
                travel_costs=None, settings=None):
@@ -1657,12 +1667,29 @@ def censored_pairs(table, origin_time):
 
 def walk_forward(table, scenario_ctx, origins, configs):
     """configs: list of dicts(model, planner, p_star, day_q, name)."""
+    # device-level EOL ledger for the cold-start floor: every death is
+    # observable from the moment it happens (eol <= origin), unlike training
+    # labels, which need the full 42d window closed
+    dev_eol = table[['device_id', 'eol']].dropna().drop_duplicates('device_id')
     fitted = {}
     records = []
     for idx in origins:
         ctx = scenario_ctx[idx]
         origin_time = ctx['start_time']
+        recent_deaths = int(((dev_eol['eol'] > origin_time - pd.Timedelta(days=WINDOW_DAYS))
+                             & (dev_eol['eol'] <= origin_time)).sum())
+        # unique dead devices visible to training: every death <= origin lands
+        # as a positive in some closed window (the 4d synthetic grid reaches
+        # back >42d), so this equals the training set's unique positives
+        train_deaths = int((dev_eol['eol'] <= origin_time).sum())
         rows = table[table['origin_idx'] == idx].set_index('device_id')
+        # label-free prevalence signal for the earliest origins, where even
+        # recent_deaths lags (recording delay): fresh devices whose corrected
+        # voltage crossed the 2.45 death threshold within the last 42d
+        zone_young = int(((rows['days_below_245'] >= 0)
+                          & (rows['days_below_245'] <= WINDOW_DAYS)
+                          & (rows['staleness'] <= 7)).sum()) \
+            if 'days_below_245' in rows.columns else 0
 
         preds = {}
         for model_name in {c['model'] for c in configs}:
@@ -1695,6 +1722,16 @@ def walk_forward(table, scenario_ctx, origins, configs):
                 PLANNERS['tsroute'].ec_margin = cfg.get('ec_margin', 0.0)
                 PLANNERS['tsroute'].ec_pmax = cfg.get('ec_pmax', 25.0)
                 PLANNERS['tsroute'].split_days = cfg.get('split_days', 0)
+                PLANNERS['tsroute'].floor_scale = cfg.get('floor', 0.0)
+                # floor_gate: apply the floor only while the training set has
+                # too few observed deaths to calibrate sum(p42) (the cold-start
+                # failure); 0 = ungated. floor_zone: strengthen the prevalence
+                # signal with the label-free death-zone count.
+                gate = cfg.get('floor_gate', 0)
+                prev = max(recent_deaths, zone_young) if cfg.get('floor_zone') \
+                    else recent_deaths
+                PLANNERS['tsroute'].recent_deaths = (
+                    float(prev) if (gate == 0 or train_deaths < gate) else 0.0)
             global PARK_MOBILITY
             PARK_MOBILITY = cfg.get('park_mobility', 0.0)
             global EMERGENCY_DAY
@@ -1739,12 +1776,24 @@ def summarize(df, label):
     for key, tag in [('blend', 'b'), ('p_be', 'be'), ('floor_day', 'f'), ('p_conf', 'c'),
                      ('alpha2', 'a2'), ('emergency_day', 'e'), ('restarts', 'r'),
                      ('iters', 'i'), ('ec_margin', 'ec'), ('ec_pmax', 'pm'),
-                     ('split_days', 'sd'), ('park_mobility', 'pk')]:
+                     ('split_days', 'sd'), ('park_mobility', 'pk'), ('floor', 'fl'),
+                     ('floor_gate', 'fg'), ('floor_zone', 'fz')]:
         if key in df.columns and df[key].nunique(dropna=False) > 1:
             df['name'] += '/' + tag + df[key].fillna(0).astype(str)
     mean = df.groupby('name')['total_cost'].agg(['mean', 'std']).sort_values('mean')
     print(f'\n=== {label} ===')
     print(mean.round(1).to_string())
+    # component decomposition by default (phase 19): late/early/ops drove
+    # every real insight since the leaderboard comparison - stop reburying it
+    comps = [c for c in ('late_swap', 'early_swap') if c in df.columns]
+    ops = [c for c in ('travel', 'overtime', 'daily_limit', 'weekly_limit',
+                       'battery_swap', 'building_change', 'room_change')
+           if c in df.columns]
+    if comps:
+        dec = df.groupby('name')[comps].mean()
+        if ops:
+            dec['ops'] = df.groupby('name')[ops].mean().sum(axis=1)
+        print(dec.round(1).reindex(mean.index).to_string())
     return mean
 
 
@@ -1771,7 +1820,7 @@ def selection_stability(df, n_random=2000, n_blocks=6, seed=0):
     for key, tag in [('blend', 'b'), ('p_be', 'be'), ('floor_day', 'f'), ('p_conf', 'c'),
                      ('alpha2', 'a2'), ('emergency_day', 'e'), ('restarts', 'r'),
                      ('iters', 'i'), ('ec_margin', 'ec'), ('ec_pmax', 'pm'),
-                     ('split_days', 'sd'), ('park_mobility', 'pk')]:
+                     ('split_days', 'sd'), ('park_mobility', 'pk'), ('floor', 'fl')]:
         if key in df.columns and df[key].nunique(dropna=False) > 1:
             df['name'] += '/' + tag + df[key].fillna(0).astype(str)
     pv = df.pivot_table(index='origin', columns='name', values='total_cost')
@@ -1782,24 +1831,35 @@ def selection_stability(df, n_random=2000, n_blocks=6, seed=0):
     rng = np.random.RandomState(seed)
 
     def run(splits, label):
+        # decided-splits-only (phase 19): a challenger byte-identical to the
+        # incumbent on most origins used to lose every all-tie split to
+        # alphabetical argmin order - selection% was mechanically capped
+        # (phase 11 had to argue around this by hand). Ties are now excluded
+        # from the denominator and reported separately.
         sel = {m: 0 for m in names}
         best = {m: 0 for m in names}
-        regrets = []
+        regrets, tied = [], 0
         for A in splits:
             B = np.setdiff1d(np.arange(n), A)
-            i = X[A].mean(0).argmin()
-            j = X[B].mean(0).argmin()
+            ma, mb = X[A].mean(0), X[B].mean(0)
+            if ma.max() - ma.min() < 1e-9:
+                tied += 1
+                continue
+            i = ma.argmin()
+            j = mb.argmin()
             sel[names[i]] += 1
             best[names[j]] += 1
-            regrets.append(X[B].mean(0)[i] - X[B].mean(0)[j])
-        k = len(splits)
-        print(f'\n--- selection stability: {label} ({k} splits) ---')
+            regrets.append(mb[i] - mb[j])
+        k = max(1, len(splits) - tied)
+        print(f'\n--- selection stability: {label} ({len(splits)} splits, '
+              f'{tied} all-tie excluded) ---')
         print(f"{'config':40s} {'selected%':>9s} {'best-holdout%':>13s} {'mean':>8s}")
         for m in sorted(names, key=lambda m: -sel[m]):
             print(f'{m:40s} {100*sel[m]/k:9.1f} {100*best[m]/k:13.1f} '
                   f'{X[:, names.index(m)].mean():8.1f}')
-        print(f'mean selection regret {np.mean(regrets):.1f} '
-              f'(90th pct {np.percentile(regrets, 90):.1f})')
+        if regrets:
+            print(f'mean selection regret {np.mean(regrets):.1f} '
+                  f'(90th pct {np.percentile(regrets, 90):.1f})')
 
     half = n // 2
     run([rng.choice(n, half, replace=False) for _ in range(n_random)],

batteryswap_example/submit.py

@@ -55,6 +55,16 @@ EC_PMAX = float(os.environ.get('BATTERYSWAP_EC_PMAX', '25'))
 # bucket (7d) instead of forcing one visit day per building. Panel 1679.2 vs
 # 1702.0, selected in 100% of random AND blocked half-splits.
 SPLIT_DAYS = int(os.environ.get('BATTERYSWAP_SPLIT_DAYS', '7'))
+# Phase 17: cold-start inclusion floor. While the training set has seen fewer
+# than FLOOR_GATE unique deaths, sum(p42) undercounts prevalence ~4x and the
+# count rule parks visibly-dying batteries (warm-up block s_0-s_7: 46% of the
+# all-48 late cost, catch rate 0.00-0.17 at s_0-s_2 with ranking AUC 0.97).
+# Floor the count estimate with max(deaths recorded in the last 42d, fresh
+# devices that crossed 2.45 within the last 42d) x FLOOR_SCALE. Inert once
+# the gate closes (s_8+ locally): the 34-origin panel is byte-identical to
+# the champion. Warm-up CV: 4256 -> 2380 at k=2.5, winning 8/8 origins.
+FLOOR_SCALE = float(os.environ.get('BATTERYSWAP_FLOOR', '2.5'))
+FLOOR_GATE = int(os.environ.get('BATTERYSWAP_FLOOR_GATE', '20'))
 
 
 def build_split_table(locations, timeseries, eol_times, scen_starts, log=print):
@@ -117,8 +127,16 @@ def adaptive_split_plans(split, split_path, log=print, deadline=None):
             # LABEL INTEGRITY: only pairs whose window closed before T
             train = pool[pool['origin'] <= T - pd.Timedelta(days=WINDOW_DAYS)]
             assert (train['origin'] + pd.Timedelta(days=WINDOW_DAYS) <= T).all()
-            fitted[T] = SklearnRUL(MODEL_KIND).fit(train)
-        model = fitted[T]
+            pos_dev = train.loc[train['dies42'] == 1, 'device_id']
+            uniq = int(pos_dev.nunique())
+            # foreign-fleet fraction: share of training positives that belong
+            # to THIS split's fleet. ~0 at a held-out split's early origins
+            # (the pooled train table dominates), -> 1 as its own history
+            # accumulates. Locally always 1.0, so the foreign branch below is
+            # panel-inert by construction.
+            own = float(pos_dev.isin(set(locations['battery'])).mean()) if uniq else 0.0
+            fitted[T] = (SklearnRUL(MODEL_KIND).fit(train), uniq, own)
+        model, train_uniq, own_frac = fitted[T]
 
         feats = features_at_origin(daily, locs, T)
         past = betas[betas['week_end'] <= T]['beta']
@@ -133,6 +151,26 @@ def adaptive_split_plans(split, split_path, log=print, deadline=None):
         plan_tsroute.ec_margin = EC_MARGIN
         plan_tsroute.ec_pmax = EC_PMAX
         plan_tsroute.split_days = SPLIT_DAYS
+        plan_tsroute.floor_scale = FLOOR_SCALE
+        plan_tsroute.recent_deaths = 0.0
+        if FLOOR_SCALE > 0 and train_uniq < FLOOR_GATE:
+            # both signals are observable at T: recorded EOLs (the harness
+            # itself drops eol <= T devices) and the split's own voltage zone
+            recent = int(((eol_times > T - pd.Timedelta(days=WINDOW_DAYS))
+                          & (eol_times <= T)).sum())
+            zone = int(((feats['days_below_245'] >= 0)
+                        & (feats['days_below_245'] <= WINDOW_DAYS)
+                        & (feats['staleness'] <= 7)).sum())
+            plan_tsroute.recent_deaths = float(max(recent, zone))
+            log(f'  {scenario["name"]}: cold-start floor active '
+                f'(train deaths {train_uniq}, recent {recent}, zone {zone})')
+        # NOTE (phase 20): a foreign-fleet floor branch (fire when training
+        # positives are mostly another fleet's, own_frac < 0.5) was built here
+        # and REMOVED the same day: the fleet-holdout harness showed the
+        # probability scale transfers across device subsets (half-fleet
+        # training: sum(p42) 15.8 vs 9.1 true deaths - if anything it
+        # OVER-counts), so the floor could only add over-inclusion (+15 in
+        # the harness, 0 wins). own_frac is still logged for diagnosis.
         plan = plan_tsroute(pred, None, locs, T, 0.05,
                             travel_costs=scenario['travel_costs'],
                             settings=scenario['settings'])