Phase 41-42: fixed-N ablation rejects the vrel port; public says pooling is a wash

Phase 41 discharges phase 39's standing directive ("do not port anything from
that lineage without re-running the ablation on recall at fixed N"). Adds
cv.ablate: holds N fixed and varies only the score, so a feature's effect on
ORDERING is separated from its effect on the topsum count. No planner and no
evaluator - recall at fixed N and the rank distribution of misses are
functions of the score alone.

Deaths caught at fixed N, paired per origin, bootstrap 95% CI:

native N=12 native N=20 static N=12 static N=20
vrel alone -6 [-12,-1] +4 [-1,+9] +6 [+0,+13] -1 [-9,+7]
vmin7_rank alone -1 [ -5,+2] +4 [+0,+9] -5 [-11,+1] +0 [-4,+4]
both -7 [-15,+0] +5 [-1,+11] +4 [ -2,+10] -1 [-9,+7]
v_last alone -39 [-57,-22] -8 [-20,+5] +7 [-10,+25] +5 [-10,+20]

REJECT the port. The sign flips with N on both instruments, every CI straddles
zero, and the ordering is byte-identical at 36-41 of 48 native origins and
228-230 of 240 static ones - these features barely enter the ranking. The one
CI excluding zero is vrel alone, native, N=12, and it is negative. The +138.7
cost penalty measured in phase 40 was therefore almost entirely depth, exactly
as the directive predicted.

The phase-39 claim reproduces and is protocol-specific: v_last alone TIES the
13-feature champion on a held-out fleet (better mean rank, 11.9 vs 13.3) and
is -8.6pp recall@12 natively. Not an over-pruned feature set - the champion's
12 extra features are worth ~8.6pp when refit on the native fleet and ~nothing
to a single static fit on a foreign one.

Phase 42: public scored the phase-40 pooling submission at 1665.89 vs the
1661.45 champion, +4.44. 9 fewer swaps, 363 fewer early-days, 2.4 fewer ops
hours, +51 late-days. A wash with the wrong sign and no recall gain. The
static branch went from 82 training deaths to 246 (164 of them in-fleet and
correctly embargoed) and the ordering did not improve, which retracts the
closing argument phase 41 made for pooling an hour earlier.

Together the two results close the training-data axis: on an unseen fleet the
ranking is essentially determined by raw voltage, and neither richer features
nor 3x more in-fleet data moves it.

Standing champion remains 3c7b883 at 1661.45.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

EXPERIMENTS.md

@@ -2024,3 +2024,114 @@ local protocol could ever catch it: the same shape as the hardcoded 42-day park
 that `repark_for_window` exists to undo. Now a module global that
 `plan_tsroute` overwrites from the settings it is handed; verified to
 propagate (30 -> 90) and to leave all-48 byte-identical at 1661.7.
+
+## Phase 41 - the phase-39 directive discharged: fleet-relative port REJECTED
+
+Phase 39 left a standing rule: *"Do not port anything from [the blind-harness
+champion] lineage without re-running the ablation on recall at fixed N."* The
+`vrel` / `vmin7_rank` port in `features.py` was exactly the case it describes,
+and it was sitting half-landed - computed by `features.py` and shipped in the
+packaged parquet, absent from `models_cv.FEATURES`.
+
+**Why the earlier cost measurement did not discharge the directive.** Under the
+topsum rule a feature moves `p42`, `p42` moves `sum(p42)`, and `sum(p42)` moves
+`N`. So a cost delta is the sum of an ordering change and a depth change and
+cannot separate them - which is how the blind lineage concluded a 4-feature set
+beat its own champion. `cv.ablate` holds N fixed and varies only the score. No
+planner and no evaluator are involved: recall at fixed N and the rank
+distribution of misses are functions of the score alone.
+
+### Deaths caught at fixed N, paired per origin, bootstrap 95% CI
+
+| variant | native N=12 | native N=20 | static N=12 | static N=20 |
+|---|---|---|---|---|
+| `vrel` alone | **-6 [-12,-1]** | +4 [-1,+9] | +6 [+0,+13] | -1 [-9,+7] |
+| `vmin7_rank` alone | -1 [-5,+2] | +4 [+0,+9] | -5 [-11,+1] | +0 [-4,+4] |
+| both | -7 [-15,+0] | +5 [-1,+11] | +4 [-2,+10] | -1 [-9,+7] |
+| `v_last` alone | **-39 [-57,-22]** | -8 [-20,+5] | +7 [-10,+25] | +5 [-10,+20] |
+
+454 deaths over 48 native origins; 976 over 240 origin-seeds on the gate.
+
+**Verdict: reject.** The sign flips with N on both instruments and every CI for
+the port straddles zero. The single CI that excludes zero is `vrel` alone,
+native, N=12 - and it is **negative**. Decisive corroboration: the ordering is
+*byte-identical* at 36-41 of 48 native origins and at 228-230 of 240 static
+ones. These features barely enter the ranking. The +138.7 cost penalty measured
+in phase 40 was therefore almost entirely a depth effect, exactly as the
+directive predicted, and the -24.6 the static gate liked was the same depth
+effect with the opposite sign.
+
+### The phase-39 claim reproduces - and it is protocol-specific, not over-pruning
+
+`v_last` alone was the specific claim: two blind sessions found raw current
+voltage beating their champion's own ranker at the operating N.
+
+- **Static gate**: `v_last` alone is statistically **tied** with the full
+  13-feature champion (+7 [-10,+25] at N=12), with a *better* mean rank
+  (11.9 vs 13.3) and a much better missed-death rank (48 vs 65).
+- **Native**: `v_last` alone is **-39 [-57,-22]** at N=12, rec@12 0.478 vs
+  0.564 - **-8.6pp**. Phase 39 measured +14.3pp on production; same sign, same
+  magnitude class, independently reproduced.
+
+So the lineage's finding was real *for its protocol* - one static fit on a
+foreign fleet - and does not survive the refit path. The cause is not an
+over-pruned feature set and not the threshold rule phase 39 suspected:
+**the champion's 12 features beyond `v_last` are worth ~8.6pp recall@12 when
+the model is refit on the native fleet and worth approximately nothing to a
+single static fit on a foreign one.**
+
+That is a statement about training data, not about features, and it read at
+the time as the first independent argument for what phase 40 shipped: in-fleet
+pooling gives the static branch 164 own deaths against the foreign 82. **The
+public leaderboard refuted that reading within the hour - see phase 42.**
+Tripling the static fit's death count bought no recall at all, so training
+volume is not the binding constraint the argument assumed.
+
+`features.py` keeps computing both columns (deleting them means regenerating
+the packaged parquet on submission day for ~1s inside a 360s build budget) but
+now carries the rejection and these numbers inline, so the port cannot be
+re-attempted by inspection.
+
+## Phase 42 - PUBLIC: in-fleet pooling is a wash (1665.89 vs 1661.45)
+
+Submission d5b03a1 (phase 40, `HELDOUT_POOL=1`). This push also carried phases
+34-37, which had never reached the remote - but they touch submit.py by one
+deleted line, so the held-out path's behaviour change is pooling.
+
+| | pooled (d5b03a1) | champion (3c7b883) | delta |
+|---|---|---|---|
+| **total** | **1665.89** | **1661.45** | **+4.44** |
+| late_swap | 1222.92 | 1212.29 | +10.63 (**+51 late-days**) |
+| early_swap | 133.69 | 137.47 | -3.78 (-363 early-days) |
+| ops | 309.28 | 311.69 | -2.41 |
+| swaps | 571 | 580 | -9 |
+
+All four inversions land on integers (571 swaps, 5870 late-days, 12834
+early-days, 54 penalised days), so public is still 48 scenarios at default
+settings - which also confirms the phase-40 `unobserved_eol_days` fix was inert
+here, as predicted.
+
+**Verdict: a wash, marginally negative.** Pooling made the model slightly less
+eager - 9 fewer swaps, 363 fewer early-days, 2.4 fewer ops hours - and paid 51
+late-days for it. Net +4.4 on a 1661 baseline is 0.27%, inside any reasonable
+noise band, but the sign is wrong and there is no recall gain hiding in it.
+
+**The finding is not "pooling is neutral", it is where the neutrality comes
+from.** The static branch went from 82 foreign deaths to 246 (82 foreign + 164
+own, in-fleet, same-fleet, correctly embargoed). That is 3x the training
+signal, on the target fleet itself, and **the ordering did not improve**. Put
+beside phase 41 - where `v_last` alone ties the full 13-feature champion on a
+held-out fleet - the two results say the same thing from opposite directions:
+
+> On a fleet the model has not seen, the ranking is essentially determined by
+> raw voltage, and neither richer features nor 3x more in-fleet training data
+> moves it. The information needed to rank better is not in what we are
+> feeding the model.
+
+This closes the training-data axis the way phases 1/6/28 closed the appetite
+axis. The remaining late cost is the voltage-silent population (recall 0.000 at
+the operating N, median rank 60 of ~450), and it is invisible to every signal
+in the current feature space - which is exactly what the phase-38/39 silent-
+death campaign concluded before it was closed.
+
+Standing champion remains **3c7b883 at 1661.45**.

batteryswap_example/cv.py

@@ -310,7 +310,148 @@ def _vrel(t):
                             if c in t.columns]
 
 
-VARIANTS = {'base': _base, 'zone': _zone, 'vcorr': _vcorr, 'vrel': _vrel}
+def _vrel_only(t):
+    return BASE_FEATURES + ['vrel']
+
+
+def _vmin7_only(t):
+    return BASE_FEATURES + ['vmin7_rank']
+
+
+def _v_last_only(t):
+    # the phase-39 claim under test: two blind sessions found RAW CURRENT
+    # VOLTAGE alone beating their champion's ranker at the operating N
+    # (.721 vs .668, .703 vs .672). If that reproduces here, our feature set
+    # is over-pruned; if it does not, that lineage's ablation was an artifact
+    # of its threshold rule and nothing from it should be ported.
+    return ['v_last']
+
+
+VARIANTS = {'base': _base, 'zone': _zone, 'vcorr': _vcorr, 'vrel': _vrel,
+            'vrel_only': _vrel_only, 'vmin7_only': _vmin7_only,
+            'v_last_only': _v_last_only}
+
+ABLATION = ('base', 'vrel_only', 'vmin7_only', 'vrel', 'v_last_only')
+
+
+# --------------------------------------------------------------------------
+# protocol 4: fixed-N ablation - ORDERING, with the count held out of it
+# --------------------------------------------------------------------------
+
+def _order_stats(p42, deaths, ns):
+    order = p42.sort_values(ascending=False)
+    ranks = {b: r for r, b in enumerate(order.index, start=1)}
+    dr = sorted(ranks[b] for b in deaths if b in ranks)
+    out = {'n_deaths': len(dr), 'mean_rank': float(np.mean(dr))}
+    # COUNTS, not ratios: pooling caught/deaths across origins is the only
+    # correct denominator (a 1-death origin must not outvote a 12-death one)
+    for n in ns:
+        out[f'c{n}'] = sum(1 for r in dr if r <= n)
+    tail = [r for r in dr if r > max(ns)]
+    out['miss_rank'] = float(np.median(tail)) if tail else np.nan
+    return out
+
+
+def ablate(variants=ABLATION, protocol='all48', origins=range(48),
+           ns=(8, 12, 16, 20, 30), seeds=(0, 1, 2, 3, 4),
+           dataset_path=Path('dataset')):
+    """Phase-39's standing directive, made runnable.
+
+    A feature ablation scored on COST cannot answer whether a feature helped:
+    under the topsum rule the feature moves p42, p42 moves sum(p42), and
+    sum(p42) moves N - so depth and order change together and the cost delta
+    is their sum. That is precisely how the blind lineage concluded a
+    4-feature set beat its own champion. This protocol holds N FIXED and
+    varies only the score, so the only thing being measured is the ordering.
+
+    No planner and no evaluator are involved: recall at fixed N and the rank
+    distribution of missed deaths are functions of the score alone.
+    """
+    t = load_table(dataset_path)
+    recs = []
+    if protocol == 'all48':
+        ctx = mc.load_context(dataset_path)
+        for idx in origins:
+            c = ctx[idx]
+            T = c['start_time']
+            tr = mc.training_pairs(t, T)
+            if tr['dies42'].sum() < 3:
+                continue
+            rows = (t[t['origin_idx'] == idx].set_index('device_id')
+                    .reindex(c['locs']['battery']).dropna(subset=['v_last']))
+            eol = c['eol'].reindex(rows.index)
+            deaths = set(eol.index[(eol > T)
+                                   & (eol <= T + pd.Timedelta(days=WINDOW_DAYS))])
+            if not deaths:
+                continue
+            for v in variants:
+                mc.FEATURES = list(VARIANTS[v](t))
+                p = mc.SklearnRUL(CHAMPION['model']).fit(tr).predict(rows)['p42']
+                recs.append(dict(_order_stats(p, deaths, ns),
+                                 origin=idx, variant=v, seed=0))
+            print(f's_{idx}: {len(deaths)} deaths', flush=True, file=sys.stderr)
+    else:
+        locations, timeseries, eol_times, scenarios = mc.load_dataset(
+            dataset_path / 'train')
+        dev_b = locations.drop_duplicates('battery').set_index('battery')['building']
+        t['bldg'] = t['device_id'].map(dev_b)
+        closed = t['origin'] <= t['origin'].max() - pd.Timedelta(days=WINDOW_DAYS)
+        allsc = list(mc.iterate_scenarios(locations, timeseries, eol_times,
+                                          scenarios))
+        for seed in seeds:
+            fleet_a, fleet_b = fleet_halves(locations, seed)
+            foreign = t[closed & t['bldg'].isin(fleet_a)]
+            models = {}
+            for v in variants:
+                mc.FEATURES = list(VARIANTS[v](t))
+                models[v] = mc.SklearnRUL(CHAMPION['model']).fit(foreign)
+            for idx in origins:
+                scenario, locs, cut, eol = allsc[idx]
+                T = pd.Timestamp(scenario['start_time'])
+                locs_b = locs[locs['building'].isin(fleet_b)]
+                rows = (t[t['origin_idx'] == idx].set_index('device_id')
+                        .reindex(locs_b['battery']).dropna(subset=['v_last']))
+                e = eol.reindex(rows.index)
+                deaths = set(e.index[(e > T)
+                                     & (e <= T + pd.Timedelta(days=WINDOW_DAYS))])
+                if not deaths:
+                    continue
+                for v in variants:
+                    mc.FEATURES = list(VARIANTS[v](t))
+                    p = models[v].predict(rows)['p42']
+                    recs.append(dict(_order_stats(p, deaths, ns),
+                                     origin=idx, variant=v, seed=seed))
+            print(f'seed {seed} done', flush=True, file=sys.stderr)
+    return pd.DataFrame(recs), ns
+
+
+def ablation_report(df, ns, label, ref='base'):
+    print(f'\n=== {label} ===')
+    g = df.groupby('variant')
+    deaths = g['n_deaths'].sum()
+    print(f"{'variant':13s} {'deaths':>6s} " +
+          ' '.join(f'rec@{n:<3d}' for n in ns) +
+          f" {'mean_rk':>7s} {'miss_rk':>7s}")
+    for v in df['variant'].unique():
+        d = df[df['variant'] == v]
+        cells = ' '.join(f'{d[f"c{n}"].sum() / deaths[v]:6.3f}' for n in ns)
+        print(f'{v:13s} {int(deaths[v]):6d} {cells} '
+              f'{d["mean_rank"].mean():7.1f} {d["miss_rank"].median():7.1f}')
+
+    # paired against the reference, per (seed, origin): the only comparison
+    # that survives the origin-to-origin variance in how many deaths there are
+    b = df[df['variant'] == ref].set_index(['seed', 'origin'])
+    print(f'\npaired vs {ref} (per origin, deaths caught at fixed N):')
+    for v in df['variant'].unique():
+        if v == ref:
+            continue
+        o = df[df['variant'] == v].set_index(['seed', 'origin'])
+        cells = []
+        for n in ns:
+            d = (o[f'c{n}'] - b[f'c{n}']).dropna()
+            w, l = int((d > 0).sum()), int((d < 0).sum())
+            cells.append(f'{d.sum():+4.0f} ({w}W/{l}L)')
+        print(f'  {v:13s} ' + '  '.join(cells))
 
 
 def main():
@@ -338,6 +479,11 @@ def main():
               'an appetite/count decision (see module docstring).')
     elif proto == 'screen':
         screen()
+    elif proto == 'ablate':
+        which = sys.argv[2] if len(sys.argv) > 2 else 'all48'
+        df, ns = ablate(protocol=which)
+        df.to_csv(f'/tmp/cv_ablate_{which}.csv', index=False)
+        ablation_report(df, ns, f'fixed-N ablation / {which}')
     else:
         raise SystemExit(f'unknown protocol {proto}')
     print(f'\n[{time.time() - t0:.0f}s]')

batteryswap_example/features.py

@@ -112,15 +112,31 @@ def features_at_origin(daily: pd.DataFrame, devices: pd.DataFrame,
 
     feats['accel'] = feats['slope_30'] - feats['slope_90']
 
-    # ---- fleet-relative encoding -------------------------------------------
+    # ---- fleet-relative encoding (REJECTED - phase 41) ----------------------
     # Ported from the blind-harness champion, where vrel and vmin7_rank were
     # the two largest single-feature wins. Both are computed at PLAN time over
     # the fleet actually in front of the planner, so they absorb a fleet-level
     # voltage shift that the absolute features (v_last, v_min_30, ...) cannot.
-    # The motivation is TRANSFER to a held-out fleet: on the native fleet there
-    # is no shift to absorb, which is why the one prior local test called them
-    # "redundant with aux features" (EXPERIMENTS.md, in-origin fleet ranks,
-    # 2055). That verdict cannot see the deployment regime.
+    # The motivation was TRANSFER to a held-out fleet.
+    #
+    # NOT IN models_cv.FEATURES, and they must not be added. Re-ablated on
+    # recall at FIXED N (the phase-39 directive), which is the only way to see
+    # a feature's effect on ordering separately from its effect on the topsum
+    # count. Deaths caught, paired per origin, bootstrap 95% CI:
+    #
+    #                  native all48            static gate (5 splits)
+    #   vrel+vmin7  N=12  -7 [-15,+0]        N=12  +4 [-2,+10]
+    #               N=20  +5 [ -1,+11]       N=20  -1 [-9, +7]
+    #
+    # The sign flips with N on BOTH instruments and every CI straddles zero.
+    # The ordering is untouched at 36-41 of 48 native origins and 228 of 240
+    # static ones - these features barely enter the ranking at all. The one CI
+    # that excludes zero is vrel alone, native, N=12: -6 [-12,-1], negative.
+    # Cost confirms it: all48 1800.4 vs 1661.7 (+138.7).
+    #
+    # Left computed rather than deleted only to avoid regenerating the
+    # packaged parquet on submission day; the two columns cost ~1s inside
+    # build_split_table's 360s budget. Delete them when there is slack.
     #
     # Deliberately NOT filled here. The classifier pipeline median-imputes and
     # then scales, which parks a stale/dark device at "unremarkable" so it is