Phase 49: four new ordering-signal families opened and closed - longitudinal cross-origin, interactions, tau re-rank, temporal ensembling; the marginal-rank defect is missing information, not noise

EXPERIMENTS.md

@@ -2498,3 +2498,83 @@ p42_sum untouched so the count rule is byte-identical).
 Local champion is now **cert-first at 1623.7** (from 1661.7). Public look not
 yet spent; submission is the user's call. Expected public value if the wobble
 rate transfers: ~-35 +/- (fires 0.46 devices/scenario locally).
+
+## Phase 49 - four new ordering-signal families opened and closed in one sweep
+
+Directive (user, 2026-08-22): test new ideas against the champion, be
+innovative, try brand-new approaches. Everything below is a fixed-N ordering
+screen (`cv.ablate` discipline, both instruments where applicable, paired per
+origin, bootstrap 95% CI), so no depth confound; costs on the 1661.7 scale
+(cv.run_all48 does not route through submit.py's cert-first). Screen code:
+`exp_x29_new_signal_sweep.py` [long|tau|tavg]. Phase 48 is reserved for the
+cert-first public adjudication (parallel session).
+
+### (A) Longitudinal cross-origin features - the first untouched signal family
+
+The feature table is a 168-origin snapshot grid (~3-4d apart) per device; every
+existing feature reads one 30/90d window at ONE origin, and no phase had ever
+read ACROSS origins. Three features, all strictly backward-looking
+(merge_asof, snapshots < T): `mom28`/`mom56` = within-fleet rank change of
+v_last vs 28/56d ago; `wolf` = count of snapshots >= 42d ago in the bottom
+decile (device provably survived - it is alive at T).
+
+The raw signal is REAL: bottom-decile chronics (wolf > 2) die at 20.7% vs
+15.5% for newcomers, and dying devices' fleet rank slides (median mom28 -0.025
+vs -0.007 for survivors). The model cannot use it:
+
+| variant | native N=12 | gate N=12 | verdict |
+|---|---|---|---|
+| mom | -5 [-13,+3] | -7 [-14,+0] | reject (gate N=16/20 significantly negative) |
+| wolf | **-24 [-42,-7]** | **-14 [-27,-1]** | reject |
+| momwolf | -27 [-47,-7] | -15 [-30,-1] | reject |
+
+Mechanism: the 30d-window slope/drop/accel features already price the
+trajectory in; the cross-origin rank version duplicates them with extra noise,
+and `wolf` duplicates the level (chronics ARE low). Effective n ~80 punishes
+every redundant column.
+
+### (B) Interaction terms - the linear champion offered its first products
+
+`z(v_last)*z(slope_30)`, `z(v_last)*z(vstd_30)`, `z(v_min_7)*z(accel)`
+(within-origin z-scores, leak-free). Flat everywhere: native +2/-2/+5/+2
+across N, gate +0/-1/-5/-1, every CI straddling zero. There is no exploitable
+interaction structure - the phase-45 ceiling holds against nonlinearity in
+feature space, not just against nonlinear learners.
+
+### (C) tau descent-band re-rank - phase 43's open candidate, closed
+
+The knn screen left one open item: logbag underweights `tau_projected` in the
+descent band where its misses live (24/36/44 of 132 panel misses in tau's
+top-3/5/10); EV-negative as extra swaps, never tested as a pure REORDER at
+fixed N. Tested three transforms (band rank-blend, global rank-blend,
+promotion-only): native **-45 of 454 at N=12 (7W/29L)**, -53 at N=8. The gate
+is mildly positive (+3..+11) - the same protocol-specificity signature as
+`v_last` in phase 41: a single static foreign fit underuses the timer, the
+weekly refit already extracts it. The gate's sign must never override a
+native loss. Candidate closed for the native path.
+
+### (D) Temporal score ensembling - the variance question, settled directly
+
+Rank-average the current origin's fit with the previous 1-2 origins' fits
+applied to the same rows (adjacent origins are near-identical draws of the
+~80-death training set). Dead flat: tavg2 -3 at N=12 (0W/3L), tavg3 -4
+(1W/5L) - refit-to-refit orderings barely differ. With phase 44's flat C
+sweep (regularisation not binding) and logbag never separating from plain
+logistic (bagging variance not binding), this is the third independent leg:
+
+> **The marginal-rank defect is missing information, not estimation noise.
+> Any future ordering candidate must bring a NEW signal, not a better
+> estimate of the current ones.**
+
+(It also moots the deployment-gate objection - there is no fit noise to
+smooth, in any branch.)
+
+### Standing
+
+Four families opened, four closed, all with mechanisms. The information
+ceiling of phase 45 now covers: re-weightings (45), replacement sets (47),
+cross-origin dynamics, interactions, timer blends, and temporal ensembles
+(49). What remains is what phase 44 said: a signal outside this feature
+space, or the declined forward-read. Local champion unchanged (cert-first
+1623.7 / base 1661.7); public champion **3c7b883 at 1661.45**. Deadline
+2026-08-23.

exp_x29_new_signal_sweep.py

@@ -0,0 +1,214 @@
+"""Phase-49 sweep: four new ordering-signal families, all REJECTED.
+
+Usage: venv/bin/python exp_x29_new_signal_sweep.py [long|tau|tavg]
+
+  long  A. LONGITUDINAL cross-origin features - the per-device history ACROSS
+           the 168-origin snapshot grid (~3-4d apart), untouched by any prior
+           phase. mom28/mom56 = fleet-rank change vs 28/56d ago; wolf = count
+           of snapshots >=42d ago in the bottom decile, survived to T.
+        B. INTERACTIONS - within-origin z-score products (the champion is
+           linear with zero interaction terms).
+  tau   C. Descent-band tau_projected re-rank at fixed N (the phase-43 knn
+           screen's "open deployment-regime candidate", never tested as a
+           pure reorder).
+  tavg  D. Temporal score ensembling - rank-average the current fit with the
+           previous 1-2 origins' fits applied to current rows (fit-noise
+           smoothing; effective n ~80 deaths).
+
+Verdicts (fixed-N deaths caught, paired per origin, bootstrap 95% CI):
+  A: wolf -24 [-42,-7] native N=12, -14 [-27,-1] gate; momwolf/allnew worse;
+     mom -5 [-13,+3] native, significantly negative gate N=16/20. The raw
+     signal EXISTS (bottom-decile chronics die 20.7% vs 15.5%; dying devices'
+     rank slides -0.025 median in 28d) but the 30d-window slope/drop features
+     already price it in - the rank version adds noise, not information.
+  B: inter flat everywhere (+2/-2 native, +0/-1 gate) - no exploitable
+     interaction structure; the phase-45 linear ceiling again.
+  C: native -45 of 454 at N=12 (7W/29L) - decisive. Gate mildly positive
+     (+3..+11), the same protocol-specificity signature as v_last in phase
+     41: the weekly refit already extracts the timer. Do not trust the gate's
+     sign against a native loss.
+  D: dead flat (tavg2 -3 at N=12, 0W/3L; tavg3 -4, 1W/5L) - adjacent 7d-apart
+     refits produce near-identical orderings. Third leg of the
+     variance-vs-information triangulation (with phase 44's flat C sweep and
+     logbag==logistic): the marginal-rank defect is MISSING INFORMATION, not
+     estimation noise. Any future ordering candidate must bring a NEW signal,
+     not a better estimate of the current ones.
+"""
+import sys
+
+import numpy as np
+import pandas as pd
+
+from batteryswap_example import cv
+from batteryswap_example import models_cv as mc
+
+NS = (8, 12, 16, 20, 30)
+
+
+# ---------------------------------------------------------------- A + B ----
+
+def _ensure_longitudinal(t):
+    if 'mom28' in t.columns:
+        return
+    snap = (t[['device_id', 'origin', 'r_vlast']]
+            .sort_values('origin').reset_index(drop=True))
+    snap['danger'] = (snap['r_vlast'] < 0.10).astype(float)
+    snap = snap.sort_values(['device_id', 'origin'])
+    snap['cum_danger'] = snap.groupby('device_id')['danger'].cumsum()
+    snap = snap.sort_values('origin')
+    base = t[['device_id', 'origin']].copy()
+    base['_row'] = np.arange(len(base))
+
+    def asof(offset_days, col, tol):
+        left = base.sort_values('origin').copy()
+        left['key'] = left['origin'] - pd.Timedelta(days=offset_days)
+        left = left.sort_values('key')
+        m = pd.merge_asof(left, snap[['device_id', 'origin', col]]
+                          .rename(columns={'origin': 'snap_origin'})
+                          .sort_values('snap_origin'),
+                          left_on='key', right_on='snap_origin',
+                          by='device_id', direction='backward',
+                          tolerance=pd.Timedelta(days=tol))
+        return m.sort_values('_row')[col].values
+
+    r28 = asof(28, 'r_vlast', 10)
+    r56 = asof(56, 'r_vlast', 10)
+    cur = t['r_vlast'].values
+    t['mom28'] = np.where(np.isnan(r28), 0.0, cur - r28)
+    t['mom56'] = np.where(np.isnan(r56), 0.0, cur - r56)
+    cd = asof(42, 'cum_danger', 400)
+    t['wolf'] = np.where(np.isnan(cd), 0.0, cd)
+
+
+def _ensure_inter(t):
+    if 'ix_v_slope' in t.columns:
+        return
+
+    def z(col):
+        g = t.groupby('origin_idx')[col]
+        return (t[col] - g.transform('mean')) / g.transform('std').replace(0, 1)
+
+    zv, zs = z('v_last'), z('slope_30')
+    zm7, za = z('v_min_7'), z('accel')
+    zvs = z('vstd_30')
+    t['ix_v_slope'] = (zv * zs).clip(-30, 30).fillna(0)
+    t['ix_v_vstd'] = (zv * zvs).clip(-30, 30).fillna(0)
+    t['ix_m7_accel'] = (zm7 * za).clip(-30, 30).fillna(0)
+
+
+LONG_VARIANTS = {
+    'mom': lambda t: cv.BASE_FEATURES + ['mom28', 'mom56'],
+    'wolf': lambda t: cv.BASE_FEATURES + ['wolf'],
+    'momwolf': lambda t: cv.BASE_FEATURES + ['mom28', 'mom56', 'wolf'],
+    'inter': lambda t: cv.BASE_FEATURES + ['ix_v_slope', 'ix_v_vstd',
+                                           'ix_m7_accel'],
+    'allnew': lambda t: cv.BASE_FEATURES + ['mom28', 'mom56', 'wolf',
+                                            'ix_v_slope', 'ix_v_vstd',
+                                            'ix_m7_accel'],
+}
+
+
+def run_long():
+    cv.VARIANTS.update(LONG_VARIANTS)
+    # ablate() slices training_pairs BEFORE the variant loop, so the columns
+    # must exist at load time (same latent limitation bites the zone variant)
+    orig = cv.load_table
+
+    def load_plus(dataset_path=cv.Path('dataset')):
+        t = orig(dataset_path)
+        _ensure_longitudinal(t)
+        _ensure_inter(t)
+        return t
+
+    cv.load_table = load_plus
+    var = ('base',) + tuple(LONG_VARIANTS)
+    df, ns = cv.ablate(variants=var, protocol='all48')
+    df.to_csv('/tmp/x29_long_all48.csv', index=False)
+    cv.ablation_report(df, ns, 'native all48 / longitudinal + interactions')
+    df2, ns2 = cv.ablate(variants=var, protocol='static')
+    df2.to_csv('/tmp/x29_long_static.csv', index=False)
+    cv.ablation_report(df2, ns2, 'static gate / longitudinal + interactions')
+
+
+# -------------------------------------------------------------------- C ----
+
+def _tau_transforms(rows, p42):
+    s = pd.DataFrame({'p42': p42})
+    s['rp'] = s['p42'].rank(pct=True)
+    tau = rows['tau_projected'].reindex(s.index)
+    vc = rows['v_corr_last'].reindex(s.index)
+    rt = (-tau).rank(pct=True)
+    band = (vc < 2.55) & tau.notna()
+    out = {'base': s['p42']}
+    b = s['rp'].copy()
+    b[band] = 0.5 * s['rp'][band] + 0.5 * rt[band]
+    out['blend_b'] = b
+    a = s['rp'].copy()
+    m = tau.notna()
+    a[m] = 0.5 * s['rp'][m] + 0.5 * rt[m]
+    out['blend_all'] = a
+    p = s['rp'].copy()
+    p[band] = np.maximum(s['rp'][band], rt[band])
+    out['promote'] = p
+    return out
+
+
+def _native_scores(score_fn):
+    """Shared walk: fit champion per origin, emit _order_stats per variant."""
+    t = cv.load_table()
+    ctx = mc.load_context(cv.Path('dataset'))
+    mc.FEATURES = list(cv.BASE_FEATURES)
+    recs, models = [], {}
+    for idx in range(48):
+        c = ctx[idx]
+        T = c['start_time']
+        tr = mc.training_pairs(t, T)
+        if tr['dies42'].sum() < 3:
+            continue
+        models[idx] = mc.SklearnRUL(cv.CHAMPION['model']).fit(tr)
+        rows = (t[t['origin_idx'] == idx].set_index('device_id')
+                .reindex(c['locs']['battery']).dropna(subset=['v_last']))
+        eol = c['eol'].reindex(rows.index)
+        deaths = set(eol.index[(eol > T)
+                               & (eol <= T + pd.Timedelta(days=42))])
+        if not deaths:
+            continue
+        for name, score in score_fn(models, idx, rows).items():
+            recs.append(dict(cv._order_stats(score, deaths, NS),
+                             origin=idx, variant=name, seed=0))
+        print(f's_{idx} done', file=sys.stderr, flush=True)
+    return pd.DataFrame(recs)
+
+
+def run_tau():
+    def fn(models, idx, rows):
+        p = models[idx].predict(rows)['p42']
+        return _tau_transforms(rows, p)
+
+    df = _native_scores(fn)
+    df.to_csv('/tmp/x29_tau_all48.csv', index=False)
+    cv.ablation_report(df, NS, 'native all48 / tau descent-band re-rank')
+
+
+# -------------------------------------------------------------------- D ----
+
+def run_tavg():
+    def fn(models, idx, rows):
+        p_now = models[idx].predict(rows)['p42']
+        r_now = p_now.rank(pct=True)
+        scores = {'base': p_now}
+        prev = [models[j].predict(rows)['p42'].rank(pct=True)
+                for j in (idx - 1, idx - 2) if j in models]
+        if len(prev) >= 1:
+            scores['tavg2'] = 0.6 * r_now + 0.4 * prev[0]
+        if len(prev) >= 2:
+            scores['tavg3'] = 0.5 * r_now + 0.3 * prev[0] + 0.2 * prev[1]
+        return scores
+
+    df = _native_scores(fn)
+    df.to_csv('/tmp/x29_tavg_all48.csv', index=False)
+    cv.ablation_report(df, NS, 'native all48 / temporal score ensembling')
+
+
+if __name__ == '__main__':
+    {'long': run_long, 'tau': run_tau, 'tavg': run_tavg}[sys.argv[1]]()