| # Experiment history |
|
|
| Chronological record of everything tried, with the CV protocol and scores. |
| All numbers are mean `total_cost` per scenario from the official `evaluate_plan` |
| (lower is better). |
|
|
| ## CV protocol |
|
|
| - **Unit**: (device, origin) pairs; an *origin* is a planning time-zero. Features |
| use only data ≤ origin; the label is death within the 42-day window after it. |
| - **Walk-forward**: at every scored origin the model is refit on all pairs whose |
| window *closed* before that origin (fully observed labels, built-in embargo). |
| - **Current protocol** (since the 2026-08-18 protocol change, after phase 10): |
| champion selection on the merged **34-origin panel s_8–s_41** with |
| repeated-split stability - accept a challenger only if selected in ≥60% of |
| 2000 random half-splits AND modal under blocked splits; the public |
| leaderboard is the only remaining holdout and is spent deliberately. |
| - **Three-regime scorecard** (since phase 20): the panel alone samples one |
| regime (native fleet, mature calibration, expensive false swaps). Standing |
| additional blocks: **warm-up s_0–s_7** (cold-start calibration; where the |
| phase-19 floor was proven) and **near-end s_42–s_47** (censored shipped |
| truth - the same scoring the remote evaluator applies to a held-out split's |
| near-end scenarios; where the phase-11/14 mechanisms are measured). |
| Adoption of a deployment-regime mechanism requires: no panel regression |
| (tie-aware selection stability) AND a win in its target regime block AND |
| the regime being reachable by a local harness - "architecturally sound" is |
| not sufficient (see the foreign-fleet floor, built and refuted within one |
| day in phase 20). selection_stability counts decided splits only; |
| summarize() reports late/early/ops components by default. |
| - **Other blocks**: warm-up s_0–s_7 (3–16 unique training deaths) is excluded |
| from champion selection but scored since phase 17 for cold-start behavior; |
| s_42–s_47 stay unscored (windows extend past data end 2026-07-31). |
| - **Historical blocks (phases 1–10)**: dev s_8–s_23 (all tuning) / test |
| s_24–s_41 (counted looks only, ledger at the end); merged once 8 looks had |
| spent the test block. Per-phase "dev"/"test" columns below refer to these. |
| - **Reference points** (test block): shipped example 6779 · defer-everything 2512 |
| · lowest-voltage-15 heuristic 2034 · oracle (true EOL) 186. |
| - **Protocols live in `batteryswap_example/cv.py`** (phase 40), not in `/tmp`: |
| `all48` (full-fleet walk-forward, three-regime scorecard), `static` (the |
| deployment gate - one foreign-fleet fit, no refits, ORDERING questions only), |
| `screen` (offline model screen grouped by BATTERY, never by scenario or row). |
| All three report ranking precision at the operating N beside the cost: cost |
| alone cannot distinguish a change that reordered the list from one that only |
| cut it deeper, and AUC 0.947 has already coexisted with recall 0.000. |
| |
| ## Phase 0 - understanding the problem |
| |
| | finding | consequence | |
| |---|---| |
| | Plans past day 42 are dropped free (`evaluate.py`) | the decision is *swap in-window vs park*; ~9 of ~450 batteries/scenario matter | |
| | Penalty asymmetry 0.5/day early vs 10/day late | cost-optimal swap day ≈ 5th percentile of EOL distribution | |
| | Example model predicts constant 120d for every battery | its 43 in-window swaps are arbitrary; 8673 all-48 | |
| | Defer-everything scores 3325 all-48 (−62%) | the real bar for any model | |
| | Last in-band voltage: AUC 0.95, 100% coverage | age (install+2y) is useless: AUC 0.63, P@5=0.00 (survivorship: median alive age > median lifetime) | |
| |
| ## Phase 1 - first model lineage (v1, submitted 2026-08-17) |
| |
| | step | dev | test | notes | |
| |---|---|---|---| |
| | voltage-k heuristic, k=15 | — | 2034 | beats defer 14/18; k tuned on cost, not precision | |
| | 72-config sweep: {rank-v, logistic, gbm, aft} × {onepd, sched, batch} × p\* × day-q | best 2580 (aft/batch/p0.35) | — | batch > sched > onepd for every model | |
| | logistic/batch, p\*=0.5 | 2473 | **1735** (look 1) | threshold minimum confirmed (0.35: 2678, 0.65: 2601, 0.8: 2989) | |
| | **diagnosis**: p42 scale drifts as training grows; at s_38–41 the 0.5 threshold includes 0 batteries while 5–7 die | | | fixed thresholds are structurally brittle | |
| | **topsum**: include top-N by p42, N = 0.8·Σp42 | 2502 | **1368** (look 2) | beats fixed on 15/18; **v1, public LB 1964.29 (rank 3)** | |
|
|
| Rejected in phase 1: AFT expected-cost planner (3182 dev - uncalibrated |
| probabilities), Platt calibration (+40–50 dev), GBM (behind logistic at every |
| data size). |
|
|
| ## Phase 2 - features, expected cost, routing |
|
|
| | step | dev | test | notes | |
| |---|---|---|---| |
| | recall diagnostic | | | deaths ranked fine (median rank 13) but excluded; worst misses had NaN slopes (sparse traces) | |
| | +5 features (slope_lastk, drop_recent, below_frac, accel, n_30) | 2390 | — | window-free slope closes sparse-trace gap | |
| | ecday: include/park + swap day by expected cost under predicted day-distribution | 2327 | — | low-p batteries automatically scheduled later | |
| | **ecroute**: + local search over building-group days vs exact mobility simulator | **2197** | 1478 (look 3) | simulator reproduces evaluator to 0.000 after replicating its return-leg double-count quirk | |
| | tsroute hybrid (topsum inclusion + ecroute scheduling) | 2229 | 1502 (look 4) | tied on dev | |
| | physics: master discharge curve | | | validated: EOL = 2.40 V crossing (±0.034 across all 82 deaths), temp coeff +4.9 mV/°C; `tau_hat` AUC 0.927 alone but **redundant** with voltage features → rejected (2274 dev) | |
|
|
| ## Phase 3 - training-data augmentation |
|
|
| | step | dev | test | notes | |
| |---|---|---|---| |
| | sudden-death check | | | only 2/79 deaths lack voltage droop → late_swap is recoverable | |
| | **synthetic origins** every 4 days back to Feb-2025 | tsroute **2126**, ecroute 2161 | 1476 (look 5) | positives at s_24: 253 → 688; recovers 3 deaths no scenario window covered | |
| | day-distribution blend toward uniform | 2174–2214 | — | rejected by dev | |
|
|
| ## Phase 4 - error screen and targeted fixes |
|
|
| Error buckets on test (per origin): miss→emergency **718** · false-positive |
| early **340** · timing of caught deaths 41 (solved). |
|
|
| | step | dev | test | notes | |
| |---|---|---|---| |
| | **tau_projected** = C⁻¹(v_corr_last) − staleness | 2147 (neutral) | **1439** (look 6) | targets the 8 "dark" devices (staleness ~62d) = 25 of 60 misses; AUC 0.676 where model was blind | |
| | break-even top-up (include p42 ≥ 0.2) | 2487–2742 | — | rejected by dev | |
| | placement floor for uncertain inclusions | 2263 | — | rejected by dev | |
| |
| **Recurring lesson**: every absolute-probability knob motivated by test-block |
| failures (blend, top-up, floor) was refuted on dev - the p42 scale shifts with |
| death prevalence. Only *scale-free* mechanisms survive: count-based inclusion, |
| ranks, physics projections. |
| |
| ## Phase 5 - spending the 30-minute runtime budget (current, fc9c286) |
| |
| | step | dev | test | notes | |
| |---|---|---|---| |
| | within-day visit ordering fix (building/room sort) + merge-move search with restarts | 2120 | — | interleaved rows paid repeated building changes the search never saw | |
| | **logbag**: 15-fold bagged logistic | **2074** | **1398** (look 7) | variance reduction near the inclusion cutoff | |
| | aggressive challenger α=1.15 | — | 1516 (look 7) | leader-like profile (late 722/early 550); kept as `BATTERYSWAP_ALPHA` knob, not default | |
| | **adaptive runtime refit** (`submit.py`) | | | refits at every scenario origin on the target split's own self-masked history; honest all-48 train metric 2202; 280 s/split native; deadline guards → static-pickle fallback | |
|
|
| **Public leaderboard (2026-08-18, adaptive submission): 1767.87, rank 2** - v1 was |
| 1964.29. Mobility+limits 315.9 now beats the leader astar's 334.6; the whole |
| remaining gap to astar (1652.6) is timing balance (our late/early 1296/156 vs |
| their 660/658). An α=1.15 change motivated by that profile was committed, then |
| reverted before submission: the local α sweep rejected it (see phase 6) - |
| selection stays on CV, not public feedback. |
|
|
| ## Phase 6 - voltage-silent deaths (exploration, all rejected by CV) |
|
|
| Silent deaths are **fast plungers**, not a separate mechanism: all 16 deaths that |
| looked healthy 30d out (v_corr > 2.55) were at ~2.47 by 7d out and ~2.41 the day |
| before - the same 2.40 V crossing, compressed into ~3 weeks. |
| |
| | candidate | evidence in isolation | dev CV | verdict | |
| |---|---|---|---| |
| | room/building failure clustering | AUC 0.52-0.54 (chance) | not attempted | dead on arrival | |
| | lifetime mean temperature | AUC 0.64; corr(lifetime, temp) 0.21 | not attempted | too weak alone | |
| | install-cohort clock (age - batch median lifetime) | **AUC 0.906** on silent pairs; batch explains ~90% of lifetime variance | 2074 -> 2279 (raw), 2433 (overdue-neutral encoding) | **rejected**: 14%/54% coverage - the no-information majority degrades the fit in every encoding | |
| | alpha re-sweep 0.7-1.3 (public-LB-motivated 1.15) | public late/early ratio 8 vs leaders' ~1 | minimum stays at **0.8** (2074; 1.15: 2219, losing 12/16 origins) | 1.15 reverted; late saturates ~1100 for alpha >= 1.15 - remaining misses are not an inclusion problem | |
| |
| ## Phase 7 - full coordinate sweep (nothing adopted; champion is a local optimum) |
| |
| One-knob-at-a-time sweep around the champion, ~25 knob values across both axes. |
| Acceptance rule fixed in advance: adopt only if dev improves by >= 60 AND wins |
| >= 10/16 origins head-to-head. Full cartesian grids were ruled out: dev's |
| resolution is ~150 points (16 correlated origins), so a large grid guarantees |
| spurious winners. |
| |
| | axis | best variant | Δ vs champion 2073.9 | verdict | |
| |---|---|---|---| |
| | bag count | 30 bags | −9.4 | closest of all; 6x short of the bar | |
| | search budget | restarts 4 / iters 120 | −0.4 | saturated | |
| | day-model GBR | dtrees=50 | −0.6 | insensitive | |
| | regularization | C=0.3 | +64.8 | C=1.0 confirmed (0.1: +148.6, 3.0: +76.7) | |
| | training set | dthaz (discrete-time hazard, open windows) | +15.4 | statistically tied while using a different likelihood and more data - the natural committee partner if one is ever wanted | |
| | recency weighting | half-life 240d | +30.7 | no drift-adaptation gain (120d: +99.7) | |
| | blends | logbag+gbm | +190.6 | GBM drags; blend_dh +1367.9 | |
| | routing-aware inclusion | tspig membership repair | +46.5 | right shape, does not pay at these travel costs | |
| | piggyback inclusion | tspig2 α₂=1.3 / 1.6 | +293.1 / +497.7 | over-includes | |
| | determined labels | logbag_open | +3080 (std 11k) | blows up at early origins - tiny determined sets | |
| |
| Disclosures: `emergency_day` is structurally inert for tsroute (it prices parking, |
| which the count rule never consults); the q6 day-quantile grid is inert because |
| cost_curves reads only the four standard quantile columns (CDF-resolution axis |
| untested); `logbag_aux*` variants await their feature columns in the table. |
| |
| **Conclusion**: every axis is flat or worse around the champion - remaining cost |
| is structural (fast-plunge deaths), not hyperparameters. |
| |
| ### Classifier-family addendum (RandomForest and friends) |
| |
| | model | dev | Δ | note | |
| |---|---|---|---| |
| | RandomForest (500, leaf 5) | 2273.1 | +199 | ranks well (AUC 0.94 at s_23 vs logbag 0.85) but probabilities too compressed for topsum's count | |
| | RF deep (leaf 1, 1000) | 2325.9 | +252 | | |
| | RF class-balanced | 2890.2 | +816 | worst config ever tested - inflated Σp42 over-includes massively | |
| | ExtraTrees | 2245.3 | +171 | | |
| | MLP (32,16, early stop) | 2768.7 | +695 | | |
| | rank_rf hybrid (RF ranking, logbag scale - Σp42 preserved) | 2166.2 | +92 | RF's ranking edge does not transfer across origins even with the scale problem removed | |
| |
| Complete p42-classifier ledger: logistic-family (champion), HGB/GBM x4, |
| Weibull AFT, discrete hazard (tied), RF x3, ExtraTrees, MLP, and five blends - |
| linear + bagging wins throughout. With ~1000 positives, one dominant monotone |
| signal (voltage level), and a planner that needs a stable probability scale, |
| that is the expected outcome. |
| |
| ## Phase 8 - the silent-death campaign (closed: unactionable within this data) |
| |
| Target: the ~27 test-block misses with p42 < 0.10 - fast plungers whose voltage |
| collapse starts AFTER the planning origin. |
| |
| Precursor screen (9 candidates, silent subpopulation = 11,599 healthy-voltage |
| pairs, 24 deaths): only volatility carries signal - rng_30 AUC 0.833, vstd_30 |
| 0.810; diurnal-amplitude drift 0.60, temp-sensitivity drift (beta) 0.57, |
| reading-rate change 0.58, low-side outliers and range-ratio ~ chance. |
| |
| | attempt | dev | test | verdict | |
| |---|---|---|---| |
| | rng_30 + vstd_30 as global features | +17 | 1438.7 vs 1397.5 (worse) | rejected - signal diluted across the healthy majority | |
| | SilentFix override (blind-zone logistic re-ranks silent batteries only; count untouched, scale-free) | w=0.15 inert, w=0.3 +6, w=0.6 +38 | w=0.3: byte-identical plans (1397.5) | rejected - inert at safe weight, harmful at active weight | |
| | cohort clock (phases 6) | +205..+359 | - | rejected twice | |
| | inclusion expansion (alpha, p_be, floors) | all rejected | late saturates ~1100 regardless | not an inclusion problem | |
| |
| **Why this is a wall, quantified**: blind-zone death prevalence is 24/11,599 = |
| 0.2%. Even at AUC 0.83 the precursor's precision at any usable operating point |
| is a few percent, so acting on it loses in expectation: adding a silent battery |
| costs ~50-90 (early) against ~0.02-0.05 x 215 (expected emergency saved); |
| swapping one in displaces a ~30%-p42 candidate. Every mechanism - global |
| feature, ordering override, count expansion - hits the same arithmetic. The |
| remaining silent misses (~290/origin on test) are the price of predicting a |
| 42-day window from a signal that only turns on ~3 weeks before death. This |
| ceiling moves only with new information: more observed deaths maturing the |
| cohort clock (batch explains ~90% of lifetime variance - the physics is there, |
| the coverage is not), or upstream data (duty cycles, battery lot IDs). |
| |
| ## Trajectory |
| |
| | champion | dev | test | combined (34) | all-48 metric | public LB | |
| |---|---|---|---|---|---| |
| | shipped example | — | 6779 | — | 8673 | — | |
| | defer-everything | — | 2512 | — | 3325 | — | |
| | voltage-15 | — | 2034 | — | — | — | |
| | logistic + fixed p0.5 | 2473 | 1735 | 2082 | — | — | |
| | **v1** topsum | 2502 | 1368 | 1901 | 2150 | 1964.29 (rank 3) | |
| | + features + ecroute | 2197 | 1478 | 1816 | — | — | |
| | + synthetic origins (tsroute) | 2126 | 1476 | 1782 | — | — | |
| | + tau_projected | 2147 | 1439 | 1772 | 2081 | — | |
| | **current**: + bagging + search + adaptive runtime | **2074** | **1398** | **1716** | 2202* | **1767.87 (rank 2)** | |
|
|
| \* honest walk-forward (adaptive path), not comparable to earlier in-sample-ish |
| pickle numbers. |
|
|
| Current champion in full: **logbag (15-bag logistic, C=1.0, 13 features) + |
| tsroute (topsum inclusion α=0.8, expected-cost day placement, merge-move day |
| search against the exact mobility simulator) + adaptive per-origin runtime |
| refits**, confirmed a local optimum by the phase-7 sweep. |
|
|
| Oracle bound: 186 (test). Remaining gap is dominated by deaths with p42 < 0.10 - |
| batteries whose voltage has not moved by the origin (27 of 60 test misses). |
|
|
| ## Phase 8 - restart-from-baseline sweep of unexplored directions (2026-08-18) |
|
|
| Systematic pass over mechanisms *not* tried in phases 0-5. Champion reproduced |
| first (logbag/tsroute α=0.8: dev 2073.9). All tuning on dev only. |
|
|
| | idea | dev | notes | |
| |---|---|---| |
| | EV-based piggyback: post-routing add/drop by expected value vs exact mobility delta (`tspig`) | 2120 | EV decisions inherit the p42-scale brittleness - again | |
| | rank-gated piggyback: extend inclusion to next-ranked batteries in already-visited buildings (`tspig2`, α2=1.0/1.3/1.6) | 2162/2367/2572 | monotonically worse: α=0.8 is the true inclusion optimum; even zero-travel extras lose | |
| | recency-weighted training (half-life 120/240 d) | 2174/2105 | rejected | |
| | discrete-time hazard on censored open windows (`dthaz`) | 2089 | close but no win; open windows add little given synthetic-origin augmentation | |
| | determined-label training (open-window deaths as positives, `logbag_open`) | 5154 | prevalence bias wrecks sum(p42) → runaway inclusion | |
| | logbag+gbm blend / logbag+dthaz blend | 2265 / 2078 | rejected | |
| | logistic C sweep 0.1/0.3/3.0 | 2223/2139/2151 | C=1.0 was already optimal | |
| | in-origin fleet ranks of v_last, tau_projected (leak-free) | 2055 | helps bare logbag, redundant with aux features | |
| | v_proj = v_last + slope_lastk·staleness | 2122 | redundant with tau_projected | |
| | **aux usage-dynamics features** from raw hourly data (`aux_features.py`: daily discharge depth rng_90 AUC 0.78, vstd_30 0.77, rng_ratio, gap_max_60) | **2035** (α 0.85: 2030) | only dev-positive find; paired stats soft (7/16 wins, t≈-0.7) | |
| | UCB inclusion (rank by bagged mean + k·std) | 2038/2053 | neutral | |
| | log1p-compressed heavy-tail features | 2098 | rejected | |
| | more bagging (30/50) / more search restarts (8) | 2046-2058 / 2031 | flat | |
| | cohort_* columns (batch-age clock) | — | skipped: AUC ≤ 0.55 standalone | |
|
|
| **Test look 8**: logbag_aux/tsroute/α0.8 → 1510 (vs incumbent 1398); combined |
| 1757 vs 1716. Dev gain does not transfer - the incumbent **stays champion**. |
| Conclusion: the (device,origin) table is mined out around the current operating |
| point; inclusion count, probability model class, training-set construction, and |
| routing search are all at local optima that reinforce each other. The remaining |
| gap to oracle is the no-signal-yet deaths, and neither usage stress, hazard |
| pooling, nor uncertainty boosting reaches them on test. |
| |
| ## Phase 9 - literature-driven approaches (2026-08-18, all rejected by CV) |
| |
| Two approaches from the prognostics/decision-learning literature, implemented |
| and walk-forward tested. In-run reference: logbag/tsroute α0.8 = 2091 (FEATURES |
| had gained vstd_30/rng_30 from the phase-7 sweep by this point). |
| |
| **Wiener first-hitting-time RUL** (`fht_features.py` + `WienerFHT`): temp- |
| corrected voltage as Brownian motion with per-device drift (shrunk toward the |
| population), EOL = first passage of the validated 2.40 V threshold, closed-form |
| barrier-crossing CDF, dark devices conditioned on surviving their silence. |
| Ranks well (AUC 0.91-0.96 spot-checked) but probability mass is overdispersed |
| (sum p42 ≈ 35-54 vs 8-15 true deaths), and the planner runs on probability mass: |
|
|
| | config | dev | |
| |---|---| |
| | wiener standalone, α 0.8 / 0.3 | 2868 / 2884 | |
| | 0.5·logbag + 0.5·wiener, α 0.8 / 0.6 | 2309 / 2329 | |
| | wiener overrides logbag where staleness > 14 d | 2503 | |
| | FHT projections as logistic features (fht_days, wnr_m, wnr_s2 / fht_days) | 2405 / 2222 | |
|
|
| Redundant with tau_projected where data is dense, noisy where it is sparse. |
| |
| **Conformal risk-controlled inclusion** (`ConformalCut` + `tscut`): per-refit |
| cutoff = risk_q quantile of the model's scores on dying calibration pairs from |
| the last 120 d of closed windows - scale-free by construction, the literature's |
| answer to exactly our probability-drift failure mode. |
|
|
| | risk_q (allowed miss rate) | dev | |
| |---|---| |
| | 0.15 in-sample / 0.15 split-calibration | 3266 / 5907 | |
| | 0.25 / 0.40 / 0.50 in-sample | 2480 / 2286 / 2296 | |
| |
| Even at the cost-matched operating point (~0.4-0.5) it loses ~200 to topsum: |
| the cutoff is a quantile of ~10-30 recent death scores and inherits their |
| sampling noise, while topsum's count integrates over the whole fleet's p42 |
| mass. Verdict: count-based inclusion beats threshold-based inclusion even when |
| the threshold is conformally recalibrated at every refit. |
| |
| Not attempted from the same review, with reasons: decision-focused learning / |
| SPO (16 dev origins cannot support cost-gradient tuning without overfitting the |
| block), deep sequence RUL (82 deaths), RL scheduling (sample complexity), |
| MNAR state-space voltage models (gap features were already dev-neutral; |
| highest-effort/lowest-evidence of the batch). |
| |
| ## Phase 10 - late_swap gap: cohort-gated rescue (rejected; bucket is empty) |
|
|
| Motivation: public-LB late/early = 1296/156 vs leader 660/658; misses→emergency |
| is ~718/origin of test cost. Hypothesis: the phase-6 cohort clock (AUC 0.906 on |
| silent pairs) was unfairly rejected because dev-era coverage was 35%, vs 72% |
| test / 84% deployment-era - so deploy it *gated to covered devices only* and |
| validate on the late-dev block s_18-23 where coverage is representative. |
| |
| Quantifying first where the cohort signal lives (covered pairs, s_18-41): |
|
|
| | subgroup | n | deaths | cohort_overdue_m AUC | |
| |---|---|---|---| |
| | all covered | 6134 | 131 | 0.584 | |
| | dark >14d / >30d (the actual miss bucket) | 1282/1144 | 26/23 | **0.426 / 0.410 - worse than chance** | |
| | looks healthy (v_corr > 2.55) | 4734 | **4** | 0.902 | |
| |
| The 0.906 was real but rests on 4 positives: fast plungers that were still |
| healthy at origin. Even perfect recall there is ~4 catches per 24 origins, and |
| reaching them means including high-overdue healthy devices at hopeless |
| precision (~60:1) x ~100 early-cost each. |
| |
| CV confirmation (`CohortGate`: logbag + per-device lift to a cohort-hazard |
| logistic on covered devices; 'silent' mode lifts only p42 < 0.2), late-dev |
| s_18-23: logbag 1943 · cg_all 2421 · cg_silent 2421 (identical plans - all |
| lifts land in the p<0.2 bucket) · cg_silent α0.9 2588. **Rejected.** |
| |
| Conclusion on the late_swap gap: the remaining misses' plunge onset is *after* |
| the planning origin. No origin-time feature in this dataset sees them: voltage |
| is healthy, usage stress is flat (phase 8), FHT drift is flat (phase 9), and |
| the batch clock is uninformative exactly where the misses live. The late/early |
| imbalance vs the leader is not recoverable by rebalancing (phase 6 α-sweep) or |
| by any tested signal; parity would require either mid-window replanning (not |
| allowed by the submission format) or a signal source we do not have. |
|
|
| ## Protocol change (2026-08-18): dev and test merged into one 34-origin block |
|
|
| After 8 counted looks the test block no longer provides an unbiased holdout, |
| and dev s_8-23 is structurally unrepresentative of deployment (smaller refits, |
| 35% cohort coverage vs 84% at deployment-era origins). Before merging, every |
| near-miss candidate without a test number was scored on s_24-41 (look 9): |
|
|
| | model | dev16 | test18 | combined34 | |
| |---|---|---|---| |
| | **logbag (incumbent)** | 2073.9 | 1398.0 | **1716.1** | |
| | logbag_rank | 2055.3 | 1446.0 | 1732.7 | |
| | logbag_aux | 2035.5 | 1510.2 | 1757.4 | |
| | dthaz | 2089.3 | 1502.6 | 1778.7 | |
| | aux_proj | 2036.5 | 1596.7 | 1803.7 | |
| |
| The incumbent wins the combined block against every dev-winner - merging the |
| blocks retroactively changes no decision ever made, which is the strongest |
| available evidence the dev-only selection was sound. Going forward: tune on all |
| 34 origins (s_8-41) with a pre-registered acceptance rule (mean improvement |
| AND per-origin win count AND no era-block regression); the public leaderboard |
| is the only remaining holdout and is spent deliberately. |
|
|
| **Repeated-split selection (adopted with the merge)**: training stays |
| walk-forward (time-locked), but champion selection is resampled - choose the |
| best config on a random half of the 34 origins, measure it on the other half, |
| repeat (2000 iid splits + all 20 blocked 3-of-6-chunk splits, which respect |
| window-overlap correlation). On the 5-finalist panel: logbag is the modal |
| champion (63% iid / 50% blocked selection frequency, and just as often the |
| true best on the held-out half); logbag_rank is the only live challenger |
| (27%/35%); aux/dthaz variants essentially never win. Mean selection regret |
| ~38-64 cost units. The historical dev->test split turns out to have been an |
| unlucky draw: on the current feature set, dev-16 alone would select |
| logbag_rank while the test half's best is logbag. Acceptance rule for future |
| candidates: selected in >=60% of random splits AND modal under blocked splits. |
|
|
| ## Phase 11 - price-adaptive inclusion (ADOPTED: break-even top-up, fresh-only, m0.8/pm25) |
|
|
| Trigger: new public leader (sackriti 1477.48, late/early 804/343 + 179 in |
| worker-limit penalties). Their +187 early for -492 late vs us is a 2.6:1 |
| winning trade; our own dev frontier re-measured that same trade LOSING at |
| 0.5:1 (α0.8→1.2: -213 late, +406 early). Not a modelling gap - a *price* gap. |
|
|
| **The price structure**: a false swap costs 0.5/day against the imputed EOL |
| (device end_time + 30d), so it depends on distance to the split's data end: |
| dev origins 81-133 per false swap (day-42 placement), test 18-77 falling |
| monotonically, s_41 17.7 - and ~0.5 for scenarios adjacent to a split's data |
| end, which the runtime splits contain. α=0.8 is the CV-optimal response to |
| dev's prices, not a universal constant. Crucially the runtime planner and the |
| evaluator read the same /tmp/data metadata, so a price-based rule cannot |
| disagree with the scorer about prices. |
|
|
| **Counterfactual harness** (`/tmp/counterfactual.py`): same dev origins, same |
| real deaths, same evaluator - end_time/EOL observability truncated at |
| origin+50d to reproduce cheap pricing (~19/false swap) from local data only. |
| There, break-even inclusion (ecroute m1.0) wins 1678 vs champion 1754; α |
| widening still fails (+10 at best) - a fixed count cannot express "include |
| everything break-even at today's prices". |
| |
| **Iterations under the merged-block protocol** (panel = all 34 origins + |
| repeated-split selection): |
| |
| | candidate | panel mean | random-split sel% | verdict | |
| |---|---|---|---| |
| | champion (fixed α0.8) | 1715.8 | - | baseline | |
| | ungated break-even top-up m1.0 | +24 vs champ on real dev, worst +461 | - | rejected: EV errors are punished where wrong costs ~100 | |
| | + price gate pm25 (v1 e_far clip) | 1760.5 | 5.5% | rejected: clip made long-stale devices (imputed EOL in the past) look FREE; evaluator charges those swaps LATE 10/day (s_34 +411, s_36 +850, s_37 +668) | |
| | + two-sided unclipped pricing | 1706.6 | 42.5% | closer; price-0 "stale, imputed EOL in-window" fiction still costs mobility (s_23/s_41 flips) | |
| | + fresh-only (e_far >= 42), m1.0 | 1703.8 | 25.6% | s_39 -78 / s_40 -406 / s_41 +76 | |
| | **+ fresh-only, m0.8** | **1702.0** | **63.0%** | **ADOPTED**: s_40 -438.7, s_41 -102.2, s_39 +72.5; 31/34 exact ties | |
|
|
| Blocked splits: 50/50 with champion, but every champion "win" is an |
| alphabetical tie-break on byte-identical costs (splits excluding the last |
| chunk are exact ties); the hybrid wins every decided split. Mean selection |
| regret drops 36 -> 8. The cost_curves fix (two-sided, unclipped survivor |
| pricing) is byte-inert for the old champion on all 34 origins and is adopted |
| as a pure correction. |
| |
| New champion: **logbag / tsroute α=0.8 + break-even top-up (ec_margin=0.8, |
| ec_pmax=25, fresh devices only) + fixed cost curves**. The top-up is |
| structurally empty at dev-like prices (all 16 dev origins byte-identical) and |
| widens exactly where a wrong inclusion is provably cheap - which is where the |
| runtime splits live. Expected effect is largest on public/private scenarios |
| near their data end; the panel contains only ~3 such origins, so the local |
| delta (-14) understates the deployable one. |
| |
| Follow-up sweeps, both rejected by the panel: pm40 (1705.1, s_38's -721 does |
| not survive the fresh-only restriction) and pm40/m1.0 (1715.1). pm25/m0.8 is |
| the local optimum of the family. Docker all-48 verification of the adopted |
| champion: **2175.4** (was 2202); late 1460 / early 451. |
|
|
| ## Phase 12 - spatial peer features (rejected: panel +106, 0% selection) |
|
|
| User idea: cross-room/building features, age-scaled ("what would my peers |
| have measured at my age?"), to catch fast plungers relatively. Attractive on |
| paper because it fixes phase-6's fatal flaw: batch death-clocks needed peers' |
| DEATHS (14-54% coverage), while peer VOLTAGE at matched age has ~100% |
| coverage. Recon: rooms too thin (median 4 peers); buildings workable (median |
| 9, max 117); install-week batches age-match by construction (median 8). |
| Building death rates genuinely cluster (4% to 100% across 24 buildings). |
|
|
| Screen on the blind zone (11,599 healthy-voltage pairs, 24 silent deaths): |
|
|
| | feature | blind AUC | all-pairs AUC | |
| |---|---|---| |
| | deficit vs building peers (age-regressed) | 0.761 | 0.869 | |
| | deficit vs install-batch peers | 0.663 | 0.808 | |
| | building death-rate prior (smoothed, leak-free) | 0.646 | 0.721 | |
| | rng_30 (phase-8 incumbent) | 0.832 | 0.764 | |
| | v_corr_last (already in FEATURES) | 0.867 | 0.945 | |
| |
| Additive test: blind logistic rng_30 alone 0.843; +peer features 0.825-0.837 |
| (worse). Panel adjudication anyway (logbag + deficit_bldg + bldg_rate on the |
| phase-11 planner, 34 origins): **1807.6 vs 1702.0**, champion selected in |
| 100% of random and blocked splits. Per-origin AUC is unchanged (s_41: 0.967 |
| vs 0.968) - the features do not add ranking signal, only calibration noise, |
| which the count rule converts into worse inclusion counts. |
| |
| Why it cannot work here: the master curve already performs fleet-level |
| age-matching (tau_projected IS "voltage vs expected-at-this-point"); local |
| peer medians re-estimate the same quantity from 5-20 samples instead of the |
| fleet. The building death-rate signal is real but weak exactly where the |
| misses live, and largely confounded with install batch (~90% of lifetime |
| variance, phase 6). |
|
|
| ## Phase 13 - plunger mechanism hypotheses: gaps, heat, room cascade (all rejected) |
|
|
| Three user hypotheses for what precedes a fast plunge, each built leak-free |
| per (device, origin) over all 168 origins and screened on the blind zone |
| (11,599 healthy-voltage pairs / 24 silent deaths), then panel-adjudicated: |
|
|
| | hypothesis | features | blind AUC | panel (champ 1702.0) | sel% | |
| |---|---|---|---|---| |
| | telemetry gaps precede death | gap_max_90 / gap_frac_90 / gap_trend | 0.42 / 0.34 / 0.59 - INVERTED: plungers report MORE cleanly than survivors | 1875.0 | 0.0% | |
| | hot-running or temp spikes | temp_mean_life / temp_rise_30 / tmax30_rel / temp_hot_frac_90 | 0.68 / 0.42 / 0.39 / 0.49 - only lifetime heat mildly ages (phase-6 confirmed); recent temp is INVERTED | 1988.0 | 0.1% | |
| | room cascade (peer fails -> room heats -> I fail) | room_death_90 / bldg_death_90 / room_peer_temp_rise_30 | 0.54 / 0.52 / 0.37 - peers' temp FALLS before a death, recent peer deaths ~chance | 1871.8 | 0.1% | |
| |
| Champion selected in 99.8% of random and 95% of blocked splits. Even the gap |
| family's small additive screen nudge (rng_30 0.843 -> 0.867) becomes +173 in |
| CV - three noisy features degrade p42 calibration, and the count rule turns |
| that into worse inclusion counts. |
|
|
| Mechanism picture after phases 6-13: the fast plunger runs slightly COOLER |
| than its own baseline, spikes less, reports cleanly, and shows no |
| environmental or neighbourhood stress - but its voltage wobbles (rng_30 |
| 0.833, the one real precursor, itself unactionable at 0.2% prevalence). This |
| is the signature of an internal cell failure, not an external stressor: |
| nothing in temperature, telemetry health, or spatial context flags it before |
| the plunge starts. |
| |
| ## Phase 14 - planner structure audit (ADOPTED: building day-splitting + censoring-aware pricing) |
| |
| With WHO-prediction mined out (phases 6-13) and ops already beating the |
| leader's, the audit targeted WHEN precision and pricing fidelity. Four |
| candidates, all CV-adjudicated: |
| |
| | candidate | verdict | evidence | |
| |---|---|---| |
| | **split_days=7**: sub-group buildings by best-day bucket so the day search may split a building across days (merge moves can rejoin at zero travel cost - same-day groups share the mobility simulation) | **ADOPTED** | panel **1679.2 vs 1702.0**, selected in **100% of random AND 100% of blocked** half-splits; sd14 also wins (1694.4) but sd7 dominates | |
| | **censoring-aware cost curves**: death mass beyond the split's data end cannot be charged late/emergency (never in shipped eol_times) - price it via the end_time+30 imputation branch and drop it from park | **ADOPTED** | byte-inert on all covered windows (panel identical to the decimal); counterfactual TRUNC=30 (last 12 window days unobservable): **1302.2 vs 1334.4** (-32), 11/16 wins - early drops 572->480 as the planner stops paying bookkeeping on unscoreable deaths | |
| | hazard-based WHEN (logbag WHO + DiscreteHazard conditional bucket-CDF day quantiles) | rejected | panel 1728.4 (+26); s_8's -26 did not generalize - the 4-knot GBR quantiles are the better day estimator | |
| | park_mobility (emergency also pays travel/overtime) | rejected | +0.8 to +2.9 on the counterfactual - noise; emergency-cost underestimation was not the binding error | |
| |
| One-visit-day-per-building had been implicit in plan_tsroute since phase 5 |
| (`offs = bldg.map(day_of)`): a building holding both a dies-day-5 and a |
| dies-day-40 inclusion was forced onto one compromise day, paying both sides. |
|
|
| New champion: **logbag / tsroute alpha=0.8 + break-even top-up (m0.8/pm25, |
| fresh-only) + split_days=7 + censoring-aware two-sided cost curves**. |
| Panel 1679.2 (from 1715.8 pre-phase-11). |
| |
| ## Phase 15 - undocumented swaps & unlabeled deaths (all three fixes rejected; the "noise" is the label process) |
| |
| User finding (voltage_all_devices notebook): timeseries extend beyond battery |
| lifetimes - unrecorded preventive swaps. Parallel forensics (3 agents) |
| established the full scale: **31 devices with undocumented swaps** (32 events, |
| clustering on building maintenance days - 2025-07-14 x8 and 2025-07-23 x7 in |
| b_afdd8b1a066d, 2026-05-08 x6), detectable by a rule with a wide dead zone |
| (consecutive-day corrected-voltage jump from <= 2.85 up to >= 3.00: identical |
| 33-event set for any jump threshold 0.10-0.25 V; real swaps jump >= 0.28, |
| nearest noise landing 2.989). Beyond the 2 suspected unlabeled deaths: **26 |
| confirmed ghosts** (crossed 2.45, never recovered, stopped reporting, no EOL; |
| dropout days cluster on batch-removal dates: 2025-12-07 x13, 2026-02-19 x10) |
| plus 32 deaths-in-progress still limping below 2.45 at data end. |
|
|
| Three CV-adjudicated treatments (panel, sd7 champion 1679.2): |
|
|
| | treatment | panel | why it failed | |
| |---|---|---| |
| | A: truncate history at detected swap (age_days restarts, windows never span the swap) | 1684.5 (+5.3; ties except s_9 +101, s_14 +122, s_32 -43) | the age error only touches age_days (weak, linear); cohort features were already rejected; truncation shortens post-swap history and loses more than it fixes | |
| | B: exclude ghosts from plans (p42=0 if below 2.45 > 42d, still < 2.55, no EOL) | **2226.0 (+547)** | WRONG PREMISE: EOL is the ~2.40 crossing and batteries spend >42d in the 2.45->2.40 twilight - the rule zeroed genuinely dying devices | |
| | C: quarantine "looks dead, labeled survives" training rows (3,027 rows) | **2519.1 (+840)**, per-origin AUC collapses to ~0.51 | those negatives are LOAD-BEARING: they teach the model that deep-low voltage does not reliably produce a recorded death - which is exactly what the metric scores | |
| |
| Conclusion: what looks like label noise IS the label-generating process, and |
| the incumbent model has already learned it (via staleness and the low-voltage |
| negatives). The evaluator scores recorded deaths, not physical deaths; ghosts |
| are unscoreable, and their "survives" labels are the correct training signal |
| for that objective. detect_swaps() and the GHOST_ZERO/QUARANTINE knobs remain |
| in the code, inert by default, with the production table and runtime paths |
| unchanged. Champion unchanged: panel 1679.2, all-48 Docker 2146.9. |
| |
| ## Phase 16 - dataset semantics (two community observations verified; no model change needed) |
| |
| **Categorical metadata leak**: the train parquets' device_id dtype carries |
| **2,451 categories** while train uses 461 - the held-out fleets total ~1,990 |
| devices (~4.3x train). Implications: (a) per-split adaptive refits are the |
| right architecture (each split has its own large fleet); (b) worker limits |
| (fixed 100/day) bind much harder at 4x fleet size - the public leader's |
| daily_limit penalties (108/scenario) are rational there; our optimize_days |
| prices limits exactly, but local CV never exercises limit-saturated regimes. |
|
|
| **end_time semantics**: locations.end_time is the ADMIN snapshot end, not |
| per-device last data - 445/461 devices share 2026-08-01. All 82 dead slots |
| keep reporting past EOL (mean 107d, max 305d): replacement batteries under |
| the same device_id, invisible to scoring. Verified harmless everywhere: |
| master_curve filters tau >= 0 (replacement data never enters), dead devices |
| are excluded from planning, and the 15 admin-removed unscoreable devices |
| (early end_time, no EOL; e.g. 11 removed 2025-12-26) are scored p42 ~ 0.01 |
| by the model - the "stale + frozen voltage -> survives" pattern learned from |
| exactly these rows (phase 15's load-bearing negatives) keeps them out of |
| every plan. The e_far imputation pricing in cost_curves makes them |
| prohibitively expensive to the break-even top-up as well (two-sided fix, |
| phase 11). Example decoded: d_57e99ff00802 - EOL 2026-02-02 (scored death), |
| replacement battery reports until 2026-05-13, admin end_time 2026-08-01. |
| |
| ## Phase 17 - label-process modeling & count correction (all five directions closed) |
| |
| A structured attack on the 1679 -> 1400 gap along the label-process and |
| count-error channels (user-provided plan). Every direction adjudicated: |
| |
| | candidate | panel | verdict | |
| |---|---|---| |
| | label-arrival hazard (two-regime: logbag for not-crossed, KM P(EOL recorded in window \| crossed a days ago) for the death zone) | **2837.7 (+1158, 1 win/33)** | refuted: the classifier's p42 for crossed devices is ALREADY the correctly calibrated mixture - dies42 labels contain the arrival process; overriding ~0.15 with KM arrival ~0.4-0.9 inflates sum(p42), explodes N | |
| | + domain-split training (classifier never trained/queried in death zone) | **2938.1 (+1259)** | refuted: most positives ARE crossed near death - splitting starves the classifier of deaths | |
| | cost-weighted training (miss ~200 vs era FP price) | **2693.7 (+1015)** | refuted; worst exactly at early origins where FP weights are largest | |
| | oracle-N diagnostic (per-origin N sweep, real evaluator) | mean regret **222.6**/origin, median 185 | the pool is real but the oracle direction is era-inconsistent (winner's curse around a shallow optimum) | |
| | backtest-alpha corrector (pick alpha from last-K closed origins' replayed curves - simulated offline from the saved sweeps) | K=4: +113, K=6: +89, K=8: +36 vs fixed 0.8 | refuted: converges to picking 0.8 (18/26 at K=8); per-origin optimal alpha is unpredictable noise | |
| |
| The label-process channel has now been attacked five ways across phases 15 |
| and 17 (binary zeroing, hard quarantine, domain split, KM arrival hazard, |
| cost weighting) - each loses by hundreds. Conclusion: the bagged logistic's |
| implicit calibration of the label-recording process is better than every |
| explicit model of it we can impose, and the fixed CV-chosen count (alpha=0.8) |
| is better than every adaptive corrector tested. |
| |
| Per the plan's own decision rule ("if the first two deliver under ~120 |
| combined, 1400 on the panel likely isn't reachable"): they delivered negative |
| amounts; **1400 is not reachable on this panel without new information |
| channels**. The honest target remains deployment-era origins (ghost-dense, |
| cheap-FP), which the shipped mechanisms - adaptive per-origin refits, |
| break-even top-up, censoring-aware pricing, building day-splitting - already |
| serve, and whose panel deltas systematically understate their deployment |
| value. Champion unchanged: panel 1679.2, all-48 Docker 2146.9. |
| |
| ## Phase 18 - limit-aware constructive initializer (rejected; the limit tail is a correct trade) |
| |
| Last item of the phase-17 plan: greedy urgency-ordered construction (each |
| group placed on the day minimizing exact partial timing+mobility cost) as an |
| additional best-of-starts candidate in optimize_days, targeting the |
| overtime/limit bucket (~172/origin, spikes 500-850 at s_21/23/33-38). |
| |
| Panel: **1692.3 vs 1679.2** (+13.1), champion selected 88.5% random / 75% |
| blocked. Three findings close the direction: (1) 28/34 byte-ties - the |
| argmin-start climb already finds these optima; (2) the limit spikes are |
| UNTOUCHED (s_21 781, s_23 832 identical) - with exact limit pricing in the |
| objective, the optimizer genuinely prefers paying penalties because deaths |
| cluster in time and spreading costs more in timing than it saves in limits; |
| (3) where the constructive start did win the search objective (s_34/s_35), |
| realized cost got WORSE (+215/+226) - at the margin, deeper search harvests |
| model-curve error rather than real savings. The knob |
| (optimize_days.constructive) stays available, default off. |
| |
| This completes the phase-17 plan: all five directions adjudicated and |
| rejected. Champion unchanged: panel 1679.2, all-48 Docker 2146.9. |
| |
| ## Phase 17 - late-swap forensics: the misses are mostly NOT silent plungers (cold-start floor ADOPTED) |
| |
| Question: which batteries get replaced too late? Attributed the all-48 |
| submission's late cost through the evaluator's own transitions (its emergency |
| loop books missed deaths as late_swap at 10/day, day ~43+): |
| |
| - 98% of late cost (73,430 of 74,850) is 263 parked deaths swapped as |
| post-window emergencies; only 1,420 is caught-but-late (deaths on days 1-7 |
| swapped a few days after). |
| - State at origin of the 263: **157 already below 2.45** (48,460 cost) · 71 |
| in visible droop 2.45-2.55 (17,710) · only 35 healthy+fresh (7,260). The |
| phase-8-13 "silent plunger wall" is ~10% of the late cost, not the story. |
| - (The repo submission.csv was the phase-5-era artifact, all-48 2202; all |
| numbers here use same-code regenerations.) |
| |
| Mechanism 1 - **cold-start count starvation** (s_0-s_8 = 46% of all-48 late): |
| at warm-up origins the ranking is fine (AUC 0.97 at s_0; missed deaths at |
| ranks 2-8) but sum(p42) undercounts prevalence ~4x (3-16 unique training |
| deaths, lower-prevalence era), so n = 0.8*sum(p42) includes 1-8 batteries |
| while 9-19 die - catch rate 0.00/0.00/0.06 at s_0/1/2. Invisible to every |
| prior selection round: CV discards s_0-s_7 as warm-up. Cold-start late |
| ~700/scenario ~= the public-LB late gap to the leader (1296 vs 660) - |
| phase 10's "not recoverable" verdict was derived on the test block only. |
| |
| Mechanism 2 - **label-lag suppression** (bulk of the remainder): long-crossed |
| devices (59-275d below 2.45) score p42 0.001-0.3 - the phase-15 load-bearing |
| negatives working as trained; the minority whose EOL does arrive in-window |
| pays 300-500 each. Still open; LogbagLAH (KM crossing->EOL arrival curve) is |
| the drafted candidate. |
| |
| Fix for mechanism 1 (ADOPTED): **gated inclusion floor** in plan_tsroute - |
| psum_eff = max(sum(p42), k * prev_est), with prev_est = max(deaths recorded |
| in the last 42d, fresh devices first crossing 2.45 within the last 42d), |
| active only while unique observed training deaths < 20 (true exactly at |
| s_0-s_7; the 34-origin panel is byte-identical to the champion by |
| construction). |
|
|
| | variant | warm-up s_0-s_7 mean | |
| |---|---| |
| | champion (no floor) | 4256.1 | |
| | ungated k=1.0/1.5/2.0 | 3994/3741/3354 - leaks into the panel (+21/+95/+299; recent-deaths overshoots post-peak), rejected by selection stability (champion 89.5%/80%) | |
| | gated, recorded-deaths only, k=1.5 | 3740.5 - recording lag blinds the label signal exactly at s_0-s_2 (recent42 = 1 while 9-17 die) | |
| | gated + zone signal, k=1.0/1.5/2.0 | 3313/3014/2532 | |
| | **gated + zone, k=2.5** | **2379.7** - wins 8/8 origins; k=3.0: 2619, k=4.0: 3191 (interior optimum) | |
|
|
| Registered caveat: k and the gate threshold were chosen on the same 8 origins |
| (no holdout exists there). Any k in [1.5, 3] recovers 29-44%, so the |
| qualitative result is robust; the exact k=2.5 is not. |
|
|
| End-to-end verification (submit.py adaptive path, floor vs same-code |
| control): s_8-s_47 **byte-identical costs** (gate closed), warm-up 4764.2 -> |
| 2976.6 (-37.5%, 8/8 scenarios), **all-48 2146.6 -> 1848.7** (late 1465 -> |
| 1125, early 434 -> 515; the control reproduces the recorded Docker 2146.9). |
| Champion and panel (1679.2) unchanged. Knobs: BATTERYSWAP_FLOOR (k, 0 |
| disables the cold-start branch), BATTERYSWAP_FLOOR_GATE. Held-out-split |
| caveat: pooled training passes the gate earlier in calendar time there |
| (bigger fleet, faster death accrual) and sum(p42) carries the training |
| fleet's scale - the foreign-fleet floor branch in submit.py (own_frac gate) |
| targets that regime; the local train split cannot exercise it. |
|
|
| ## Phase 19 - cold-start count floor (ADOPTED, gated) & ungated variants (rejected) |
|
|
| Public resubmission of the phase-14 champion read **1717.8 (rank 3)**: late |
| 1296→1234, early +14 - the deployment mechanisms delivered ~-50, so public |
| mostly does NOT sit in the cheap-FP regime, and the remaining -240 to the |
| leader is late-swap conversion. Cold-start diagnosis (warm-up block): at |
| s_0-s_7 the model RANKS deaths at the very top (AUC 0.97) but sum(p42) |
| undercounts prevalence ~4x (3.8 when 17 die) - the count starves, not the |
| ranking; 46% of the all-48 late cost. Fix: floor the count with observable |
| signals - max(sum_p42, k x max(deaths recorded in last 42d, fresh |
| 2.45-crossings)) - gated to fire only while training has seen < 20 unique |
| deaths. Warm-up CV: **4256 -> 2380 at k=2.5, winning 8/8 origins**; the |
| 34-origin panel is byte-identical (gate closed everywhere). |
| All-48 Docker: **2146.9 -> 1851.0** (late 1465→1125). |
| |
| Ungated variants, panel-adjudicated and rejected: k=1.0 recent-only +20.9, |
| k=1.0 +zone +75.1, k=1.25 +zone +113.9 (champion 96%/80% selection). Where |
| sum(p42) is natively calibrated it beats the empirical death rate (deaths are |
| bursty; last-42d != next-42d), and the death-zone signal counts never-dying |
| limpers. Sixth refutation of explicit count/label correctors in calibrated |
| regimes. |
| |
| ## Phase 20 - CV-harness upgrade: the harness catches a blind ship (foreign floor refuted) |
| |
| Five CV improvements, all implemented: |
| |
| 1. **Fleet-holdout harness** (train the pool on half the devices, plan/score |
| the full fleet): built to validate a "foreign-fleet floor" (fire the count |
| floor when training positives are mostly another fleet's - the public |
| early-origin hypothesis) that had been wired into submit.py on |
| architectural grounds. The harness REFUTED it the same day: the |
| probability scale transfers across device subsets (half-fleet training: |
| sum(p42) 15.8 vs 9.1 true deaths - it OVER-counts if anything); the floor |
| was inert on 15/17 origins and lost where it fired (+15.1, 0 wins). |
| Branch removed before it could cost a submission. Public late-heaviness is |
| therefore NOT foreign-scale starvation. |
| 2. **Near-end block s_42-s_47** (censored shipped truth, previously |
| discarded): native retro-adjudication of the deployment mechanisms - |
| phase-11 top-up worth **-273** (877.6 -> 604.8), phase-14 censoring-aware |
| pricing **-64** (669.2 -> 604.8). Now a standing block. |
| 3. **Three-regime scorecard** codified in the protocol section. |
| 4. **selection_stability** counts decided splits only (all-tie splits no |
| longer break alphabetically to the incumbent). |
| 5. **summarize()** reports late/early/ops decomposition by default. |
| |
| Champion: logbag / tsroute a0.8 + top-up (m0.8/pm25) + sd7 + censor-aware |
| pricing + gated cold-start floor (k=2.5, gate 20). Panel 1679.2 (unchanged), |
| warm-up 2380, near-end 604.8, all-48 Docker 1851.0, public 1717.8. |
| |
| ## Phase 21 - forced-emergency timing: exact pricing tested, rejected |
| |
| Evaluator fact (established by reading evaluate_plan's emergency loop): every |
| recorded in-window EOL the plan never swaps gets a FORCED emergency visit - |
| late penalty + full unoptimized ops - and the queue does not start at day 42. |
| It starts at the **end of the calendar week containing end_time** |
| (`end_day = end_time + (6 - weekday)`), then processes one battery per day in |
| alphabetical order. All 48 train scenarios start on Mondays, so the first |
| emergency is always **day 48**, and the k-th miss lands at day 48+(k-1). Our |
| cost model prices parking against `EMERGENCY_DAY = 45` - a systematic ~30 |
| underprice per expected death, plus an unmodeled queue-position term. |
| |
| This supersedes the phase-10 disclosure that `emergency_day` is "structurally |
| inert": the phase-11 top-up and optimize_days' add/drop passes both consult |
| `park`, so the knob is live now. |
| |
| | candidate | verdict | evidence | |
| |---|---|---| |
| | emergency_day=48 (exact) | rejected | panel **1682.8 vs 1679.2** (+3.6); 31/34 plans byte-identical - the repricing only flips decisions at near-data-end origins (s_39 -74, s_40 +67, s_41 +130); near-end block s_42-47 **631.9 vs 604.8** (+27), 0 wins / 4 losses / 2 ties | |
| | emergency_day=50 / 52 (exact + queue pad) | rejected | +9.4 / +13.1, monotonically worse | |
| |
| Why the exact price loses: the deaths we park are mostly voltage-silent |
| (p42 ~ 0.05), so +30 x p42 adds ~1.5 to their park price - never enough to |
| include them. What the pricier park DOES flip are marginal p42 ~ 0.3-0.5 |
| devices that mostly survive: late improves -3.5 while early pays +6.6 |
| (panel decomposition). The champion's underpriced park is a de-facto |
| regularizer against over-inclusion where FP prices collapse. Same conclusion |
| as the phase-14 park_mobility rejection from the other side: emergency-cost |
| underestimation is not the binding error - p42 on silent deaths is. |
| |
| ## Phase 22 - optimizer-constraint challenges: sd<7 and NN visit order (both rejected) |
| |
| The evaluator's "constraints" are soft prices (weekly limit flat 100/blown |
| week, daily flat 100, overtime 2x/h past 8h; ONE worker hard-wired in the |
| state machine - no worker field exists in the plan schema) and mobility_cost |
| replicates its cost machine exactly, so the ~152/origin the champion pays in |
| penalties is a purchased trade (phase 18 already adversarially confirmed). |
| The two remaining SELF-imposed constraints were challenged: |
| |
| | candidate | verdict | evidence | |
| |---|---|---| |
| | split_days=3 / 1 (finer group granularity) | rejected | panel **1685.0 / 1691.1 vs 1679.2**; ops rises 216.6 -> 223.7 -> 228.2 - finer groups fragment the local search's argmin start and it converges to worse optima with more travel; more freedom != better local search | |
| | nn=True (visit a day's buildings in NN route order from base instead of name order, priced in the sim AND emitted identically) | rejected | panel **1679.4 vs 1679.2** - tie; 8 wins / 13 losses / 13 ties; per-origin moves up to +-40 are search-trajectory perturbations, not route savings (with a symmetric travel matrix 2-building days are order-invariant; 3+ building days are rare and route deltas are <1 each) | |
| |
| Machinery kept inert: optimize_days.nn_order (default False, cfg key 'nn'), |
| assemble(sort_key=). Ops pot accounting: 217/origin total = 152 penalties |
| (correctly purchased) + 54 travel/changes + 11 swap/room time; the recoverable |
| slice was bounded by ~5-10/origin and the tests found ~0. The ops channel is |
| CLOSED - remaining cost concentrates where it always was: late_swap (p42 on |
| silent deaths). |
| |
| Provenance note: the first nn panel run was invalidated by a parallel-session |
| file race (a peer harness's frozen-file guard checked models_cv.py out to HEAD |
| mid-experiment; nn configs silently ran as no-ops - detected because knob |
| configs came back byte-identical to baseline). Long background runs now print |
| the imported module's md5 and assert the knob under test exists. |
| |
| ## Phase 23 - worker-limit saturation campaign (all rejected; non-adaptation is optimal) |
| |
| The one regime local CV never samples: held-out fleets are ~4.3x train |
| (phase 16), where fixed worker limits bind and the public leader pays |
| ~108/scenario in daily-limit penalties. Harness (`research/run_sat.py`, |
| frozen): divide each scenario's worker limits (weekly/daily/overtime_start) |
| by 4 on real panel origins - planner and evaluator share the transformed |
| settings (the phase-11 property). Champion baseline on the block: **2115.1** |
| (vs 1679.2 native) = late 1022 / early 446 / OT 112 / daily 262 / weekly 212 / |
| travel 41 - so ~585/scenario of limit-driven penalties, with late/early |
| unchanged from native: the champion does not adapt its plan to congestion. |
|
|
| Autoresearch protocol (mutable surface = opt-knob JSON only): s0 = 6-origin |
| saturated screen; s1 = all 34 saturated origins + native no-regression leg. |
| 10 s0 experiments + 1 s1, all rejected: |
|
|
| | knob family | best variant | s0 delta | verdict | |
| |---|---|---|---| |
| | constructive start (phase 18) | on | +16.2 | worse basin under saturation too | |
| | search budget | r6/i120 | -13.4 | one -80 escape in six origins; 5 ties | |
| | split_days | 12 / 4 | -4.1 / +19.4 | both directions wash; limits untouched | |
| | day_q | 0.2 | 0.0 | identical plans - dead knob under saturation | |
| | top-up gate | pm12 | +116.3 | starving the top-up raises late AND limits | |
| | alpha | 0.75 / 0.7 | **-59.8** (4/6) / -10.5 | promoted at 0.75+sd12 combo -> s1 | |
|
|
| s1 (alpha 0.75 + sd12): sat **+20.0** (18/34 wins but 4 blow-ups sum +1835: |
| s_17 +628, s_8 +459, s_31 +391, s_23 +357 vs 18 wins summing -1317), native |
| **+42.0**. Decomposition: late +43.5 / early -27.0 / daily +5.9 / weekly -2.9 - |
| inclusion cuts buy ZERO congestion relief; they are a mortality bet with |
| fat-tailed downside, made worse by saturation (emergencies consume the same |
| scarce worker-hours). Screen lesson: 6-origin s0 is high-variance for |
| inclusion knobs (-59.8 flipped to +20.0 at s1); the s1 mean gate caught it. |
|
|
| **Conclusion**: the ~585/scenario limit spend is the shadow price of |
| scheduling MASS, not of bad choices - the day-assignment search is converged |
| (extra search, day quantiles, both sd directions, constructive start: all |
| flat or worse), and paying the flat fines beats the timing cost of leaving |
| peak mortality weeks. The champion's non-adaptation to congestion is optimal |
| within its plan space; the leader's limit penalties on the LB are likewise a |
| purchased trade, not an exploitable gap. The saturated-regime channel is |
| CLOSED; ec_margin was left untested as the same condemned inclusion family. |
| |
| ## Phase 23 - unmasked foreign pooling in deployment (ADOPTED; user change, holdout-validated) |
| |
| User change to submit.py: the packaged train table is pooled UNMASKED on |
| held-out splits (all 66,890 closed-window pairs, 82 dead devices, explicit |
| window_closed filter), while the target split's own pairs - and any train-table |
| rows for devices in the target fleet - stay under the T-42d mask. The old |
| uniform mask was CV hygiene misapplied to deployment: it starved a held-out |
| split's early origins to 3-20 unique training deaths (the cold-start |
| mechanism). Byte-inert when the target split IS train (own-fleet guard), so |
| panel / warm-up / all-48 are unchanged by construction. |
|
|
| Validation (pooling-holdout harness /tmp/pool_holdout.py: fleet A = packaged |
| table, fleet B = target, B-only scenarios scored): |
| |
| | regime | warm-up s_0-7 | spots s_16/24/32/40 | |
| |---|---|---| |
| | old masked pool, no floor | 1478.9 (late 1274) | 1141.9 | |
| | old + gated k=2.5 floor (shipped before) | **1091.0** | 1141.9 (gate closed) | |
| | **new unmasked pool (ships now)** | 1108.6 | **1088.4** | |
| |
| - Count calibration transfers: held-out s_0 sum(p42) 0.5 -> 3.4 vs 4 true |
| deaths. The starvation the phase-19 floor patched is gone at its source. |
| - Warm-up vs the floor: statistical tie (5W/3L; s_4 -827 for the floor - a |
| local prevalence burst that own-fleet recent-death signals catch and |
| foreign calibration lags). Profiles differ: floor buys insurance early |
| (480 early), new stays lean (177 early / 786 late). |
| - The undersold gain is MID/LATE held-out origins: -53.5 (3W/1L, s_16 -438) - |
| the unmasked pool improves every held-out origin's training set, and a real |
| split is mostly mid-history origins. |
| - Deployment consequence: on held-out splits the floor gate NEVER opens now |
| (uniq >= 82 from origin 0); the floor remains live only for genuinely |
| history-poor local regimes (train warm-up block, all-48 metric). |
|
|
| ## Phase 24 - pretrained time-series foundation models (rejected at tier-0 screen) |
|
|
| Question: can a pretrained model improve the metric? Rules check |
| (nora.ai docs 04): pretrained models ARE allowed if open-source, documented, |
| reproducible, and fully self-contained at evaluation (no network; 30-min |
| CPU-only, 32 GB). The runtime image ships torch 2.13 but NOT |
| transformers/chronos/timesfm - deployment would mean vendoring inference code |
| + committing weights. Applicable class: zero-shot TS foundation models |
| (Chronos-Bolt, TimesFM, TTM, MOMENT - all Apache-2.0/MIT). No public |
| battery-specific pretrained RUL model fits this data (BatteryML et al. target |
| Li-ion charge/discharge cycling curves, not discharge-to-death telemetry). |
|
|
| Test: amazon/chronos-bolt-small zero-shot on per-device daily temp-corrected |
| voltage (365-d context, 64-d quantile forecast, leak-free at each origin; |
| 0.2-0.5 s per ~400-device origin on MPS; scripts /tmp/fm_screen.py, |
| /tmp/fm_build_all.py, /tmp/fm_tier0.py; features for all 71,658 (device, |
| origin) pairs built in 26 s). Derived: P(cross 2.40 V in window) per day |
| (quantile-interpolated), q10/median margins at window end, days-to-crossing. |
|
|
| Signal screen (6 tier-0 origins): fm_margin_end AUC 0.937 all-pairs vs |
| v_corr_last 0.948; crossed bucket 0.621 vs slope_lastk 0.672; blind/dark |
| "wins" (0.815/0.986) rest on 3-4 deaths. Raw crossing-prob mass overdispersed |
| 4-9x (sum 52-62 vs 6-15 true deaths) - the exact WienerFHT failure (phase 9). |
| |
| Tier-0 (exact replica of run_experiment.tier0, lab.py untouched; baseline |
| reproduced bit-exact): +fm_margin_end auc_crossed 0.757, +fm_pmax42 0.801, |
| +trio 0.752, +fm_pmax42 dark-gated 0.781 vs baseline 0.802 (gate >=0.822 or |
| dying_p >=0.544): **all four REJECT** - neutral at best, and the margin/trio |
| variants actively degrade the crossed bucket. |
|
|
| Why, mechanically: a zero-shot forecaster of the voltage series is a smoothed |
| level/trend extrapolator - exactly the signal v_corr_last / tau_projected / |
| slope_lastk already saturate (phase-9 redundancy, third confirmation). The |
| misses that matter have flat voltage at origin (phases 8-13): no forecaster |
| OF THAT SERIES can see them, pretrained or not. Other FM families (TimesFM, |
| TTM, MOMENT embeddings) read the same series and inherit the same wall; |
| embeddings-as-features on 82 deaths is the already-rejected deep-RUL |
| sample-complexity problem. Direction CLOSED absent a new signal source. |
|
|
| ## Phase 24b - static fit for held-out splits (ADOPTED; deployment simplification) |
|
|
| Follow-up question to phase 23: with the full foreign pool available at every |
| origin, do the per-origin refits (foreign pool + target's own accumulating |
| masked history) still earn their runtime? Holdout harness |
| (/tmp/static_holdout.py, fleet A=packaged / B=target, 16 origins): |
| |
| | regime | warm-up s_0-7 | mid/late s_12-40 | all 16 | |
| |---|---|---|---| |
| | rolling refit (phase-23 shipped) | 1108.6 | 1065.4 | 1087.0 | |
| | **static: ONE fit on A_full** | 1072.8 | 1042.6 | **1057.7** | |
| |
| 6W/7L/3T - a statistical tie with a static lean (mean gap driven by s_36 |
| -372 vs s_40 +142). The sum(p42) columns are near-identical everywhere: with |
| 82 foreign deaths in the pool, the target's own few masked deaths perturb the |
| fit without informing it. Phase 20's "the probability scale transfers" holds |
| all the way down: even fleet-specific own-history adds no measurable |
| calibration. |
| |
| Adopted in submit.py: held-out splits (own-fleet fraction < 0.5 vs the |
| packaged table) fit ONE model per split on the foreign pool and skip the |
| split-table build entirely - equal-or-better score, minutes instead of ~17 |
| per split, near-zero time-budget fallback risk. Train-fleet splits keep the |
| rolling self-masked path (their labels are the answer key; also how the |
| all-48 metric is produced). Verified: synthetic held-out split (B-only data, |
| A-only table) takes the static branch and plans sanely (4/5/7 swaps vs 4/6/7 |
| true deaths at s_0-2); all-48 Docker must reproduce 1851.0186805555559 |
| byte-exact (train path untouched). |
|
|
| Deliberately NOT adopted: training on the target split's own post-T labels |
| (the runtime ships each split's full eol_times.csv). That is answer-key |
| training - near-oracle scores, unmistakable on a leaderboard, and the |
| organizers run and read this code. The label-integrity assertion stays. |
| |
| ## Phase 25 - HVAC environment class as a feature (rejected at tier 0; gradient is real but absorbed by voltage state) |
| |
| EDA first (research/hvac_room_profiles.py, hvac_signatures.png): device |
| temperature signatures cleanly classify the equipment environment - 8 classes |
| from level, weekday-vs-weekend diurnal schedule, and seasonal swing (scheduled |
| AHU in unheated fan rooms sits at 1.5 C nights/weekends and steps to 8 C on |
| weekday operating hours; hot plants sit >30 C year-round; the -18 C minima are |
| Nordic AHU intakes in winter, not freezers). Raw death rates are U-shaped in |
| the class: cold classes 25-40%, hot plants 20-30%, mild indoor ~10%; median |
| dead lifetime 750 d (cold) vs 1000 d (warm). Motivation vs the three prior |
| temperature rejections (phases 6/8/13): those were monotone or recent-window |
| features; two separate exposure arms can express the U, and a static class is |
| low-noise where the phase-12 building death-rate prior was sampling-starved. |
|
|
| Leak-free encoding (research/hvac_origin_features.py, expanding life-so-far |
| per (device, origin_idx)): cold_frac (days with min < 5 C), hot_frac (days |
| with max > 32 C), sched_amp (weekday-weekend daily-range gap). Candidate |
| hvac_env_logbag = exact logbag replica + the 3 columns in the WHO classifier. |
|
|
| Tier 0: **REJECT** - auc_crossed 0.776 vs baseline 0.802 (gate +0.02), |
| dying_crossed_mean_p 0.450 vs 0.453; only count_err improved (5.23 vs 5.70). |
| |
| Post-mortem screen (66,890 closed pairs): cold_frac standalone AUC 0.623 |
| all-pairs (same league as phase-6 lifetime temp) but **0.400 in the crossed |
| bucket - inverted**, sched_amp 0.419; additive in-sample gains +0.001/+0.009. |
| Within v_corr_last deciles, cold_frac AUC is 0.23-0.56 (noise-to-inverted): |
| the lifetime mortality gradient is fully absorbed by voltage state at origin - |
| a cold-stressed battery's depressed state of charge IS the voltage the model |
| already reads, and among crossed devices the environment prior points the |
| wrong way (consistent with phase 13: plungers run cooler than baseline). |
| Class one-hots would inherit the same wall (functions of the same signatures). |
| Fourth and final temperature rejection; the environment channel is CLOSED for |
| WHO-ranking. The EDA artifacts stay in research/ (the class map itself is |
| sound and may serve ops/reporting, e.g. cold-room devices as a maintenance |
| batch - just not the model). |
|
|
| ## Phase 26 - deployment alpha under the static foreign fit (ADOPTED: ALPHA_STATIC=1.5) |
| |
| Public 2026-08-19 (phase-23 submission 3c7b883): **1661.45, rank 3** (from |
| 1717.8; the holdout harness predicted the 30-80/origin band correctly). |
| Remaining gap to sackriti (1477.48) = 184, entirely the late/early exchange: |
| ours 1212/137 vs theirs 804/343 - they buy -408 late for +206 early. Ops now |
| slightly better than theirs. |
| |
| Mechanism found: on a held-out fleet the static foreign fit's sum(p42) |
| UNDERCOUNTS (5.5 vs 8 true at s_16, 6.2 vs 9 at s_6), so N = alpha*sum(p42) |
| needs a larger alpha than the panel optimum. The panel's 0.8 is the optimum |
| for a CALIBRATED count on the native fleet - a regime public never sees. |
| Holdout sweep, 4 runs (3 seeds + reversed fleet direction), 16 origins each: |
| |
| | alpha | 0.8 | 1.0 | 1.15 | 1.3 | 1.5 | 1.7 | 2.0 | |
| |---|---|---|---|---|---|---|---| |
| | mean total | 1156.3 | 1045.1 | 1006.8 | 956.5/967.2* | 916.4 | 908.3 | 898.8 | |
| |
| (*separate run batches; 1.3 anchors both.) alpha=1.3 beat 0.8 in ALL FOUR |
| runs; the curve flattens from 1.5 (1.5->2.0 gains only -18, within noise). |
| **ALPHA_STATIC = 1.5 adopted** (env BATTERYSWAP_ALPHA_STATIC): captures ~93% |
| of the gain from 0.8 while staying off the plateau's unbounded edge - |
| over-inclusion exposure grows there if the real public fleet is better |
| calibrated than the half-fleet proxies. At 2.0 the late/early profile |
| (446/331) matches the leader's shape almost exactly. |
| |
| Applies ONLY to the static held-out path; the panel champion (rolling, |
| native fleet) keeps alpha=0.8, where the same sweep was decisively rejected |
| (phases 1/6). The train path never reads ALPHA_STATIC - all-48 unaffected. |
|
|
| ## Phase 27 - held-out-parity audit & dress rehearsal (2 fixes shipped) |
|
|
| Question: does our setup correctly reflect public/private mechanics? Audit of |
| what can vary per scenario + a Docker dress rehearsal of the EXACT remote |
| invocation (main() over fake public+private splits with renamed device IDs so |
| the own-fleet guard routes to the static path). |
|
|
| Findings: |
| 1. **Per-scenario settings DO vary in train** - 27 of 48 scenarios differ, |
| but only in base_location/base_room (16 distinct technician bases). The |
| mobility sim reads the live settings object, so this was already handled. |
| All penalties/limits/window are constant 0.5/10.0/42 across train. |
| 2. **Hardening shipped: penalty wiring.** cost_curves priced with hardcoded |
| EARLY/LATE while the evaluator charges per-scenario settings values - now |
| wired through per scenario (byte-inert at default values; checksum |
| confirms). planning_window_days != 42 now logs loudly (cannot be rewired: |
| the 42d grid is baked into features/quantiles). |
| 3. **Hardening shipped: failure isolation.** The rehearsal exposed that a |
| data-shaped exception (e.g. a check_dataset assertion) killed the adaptive |
| path AND the fallback identically - the exception escaped main(), and NO |
| submission.csv was written: one bad split destroyed all splits' plans. |
| Now three layers: load_best_effort (consistency check demoted to a logged |
| warning), individually guarded pickle fallback, and a raw-file last resort |
| (defer-everything plans; finite score, cannot poison other splits), itself |
| guarded so a split can at worst lose only itself. |
| 4. Rehearsal after fixes: public + private both take the static path (82 |
| deaths, own-fleet 0.00), the injected inconsistency logs as a warning, |
| complete 2-split submission.csv in 35s. All-48 checksum byte-exact |
| (1851.0186805555559). |
| 5. Not verifiable locally, accepted as residual risk: held-out fleet size |
| (~4x per the metadata leak; count rule is a ratio, runtime proven by two |
| real submissions), scenario dates/weekdays, and the held-out label-lag |
| process (the ALPHA_STATIC=1.5 bet rides on the undercount direction). |
| |
| ## Phase 28 - public-repo port: Attention-Heads solution mined (features/floors/queue rejected; cliff-gated static appetite CONFIRMED by the barrier-event resampler - adoption candidate) |
| |
| Source: team Attention-Heads (public rank 5, 1798) left their full solution |
| public at huggingface.co/Attention-Heads/BatterySwapAI2026-attention-heads, |
| including a SOLUTION.md with measured ablations and a component breakdown |
| against us. (Other public repos are untouched templates; sackriti's is |
| private again.) Everything transferable was ported and adjudicated. Runner: |
| `exp_x28.py` (new file; frozen modules imported, never edited, md5-asserted |
| per run); outputs /tmp/x28/. Parity: champion byte-equal on s_8/s_9 vs |
| research/baselines/tier1_champion.csv. Their independently-derived core |
| matches ours (recorded-EOL-only force visits, exact evaluator replica as the |
| objective, censored-anchor survivor pricing, return-leg double count) - the |
| deltas tested here were their hourly "cliff" machinery, per-battery appetite |
| scaling, miss-queue park pricing, and peer-offset features. |
| |
| Recon (facts stage): the data is exactly hourly (tele_n7 median 168). Their |
| headline mechanism - hourly mins/hours-below-threshold see a knee days before |
| the OFFICIAL smoothed series (temp-gated daily median + 7d rolling median), |
| ~30% of their false negatives - does NOT transfer as detection: our features |
| never used the 7d roll, so we never had that lag. Cliff mask (>=35% of the |
| last 7d's readings below 2.45 V, >=20 readings, corrected v < 2.46) catches |
| 54% of dying panel rows but only **2/61 of the champion-blind pre-crossing |
| deaths**; its catches are already-crossed knee devices - i.e. an *inclusion* |
| signal, not a detection signal (consistent with phase 30's label finding). |
|
|
| | candidate | tier-0 | panel (champ 1697.4) | static holdout, α=1.5 (base 922.9, 5 fleet splits x 16 origins, paired) | |
| |---|---|---|---| |
| | hrly / hrly2 / peer features | REJECT (auc_crossed 0.802/0.795/0.790 vs 0.802) | not run | - | |
| | cliff_floor25 / 50 (floor p42 on cliff rows) | floor25 promoted (auc_crossed 0.835) | **+68 / +149 regression** - floors distort the calibrated count channel the native path lives on | −16 / −35.5, worst +818 | |
| | cliff_floor50p (park-mode: floor rank/count only, timing raw) | - | not run | −11.3 - their scale_mode='park' lesson does NOT transfer | |
| | cliff_rank22 (rank-only reorder) | - | +13 tie | - | |
| | **cliff_count22** (rank_score & p42_sum = clip(2.2·p42,1) on cliff rows; raw p42 keeps timing honest) | n/a (p42 untouched) | +27 (late 904 / early 591 vs 1022/442) | **−54.5** (t −1.9, 4/5 splits, worst +319; late 452 / early 308 vs 580/226) | |
| | cliff_count_lg (+ lingerer 0.7 downweight) | - | 1698.9 tie (modal 40% blocked, 31% random < 60% bar) | −25.7 - lingerer downweight subtracts value | |
| | logbag+tsq (miss-queue park penalty in the ec top-up) | - | +1.7 inert | +0.8, 0/48 wins - the queue term never clears the gated break-even channel; REJECT | |
| |
| Controls: plain alpha raise gets the same mean (α=2.0: −55.3) but noisier |
| (sd 298 vs 260), worse tail (worst +421), and the curve TURNS at α=2.5 |
| (+32.5) - phase 26's over-inclusion fear is real; the cliff gate is |
| self-limiting (only knee devices scale, appetite cannot run away on |
| survivors). α-pairing sweep: 1.5 stays optimal under the boost (1.2: −31, |
| 1.35: −23, 1.7: −44 noisier). |
| |
| **Verdict (initial): cliff_count22 @ ALPHA_STATIC=1.5 looked like a CANDIDATE |
| for the static held-out path only** (same gating pattern as ALPHA_STATIC; the |
| native rolling path decisively prefers the raw count - panel +27), pending a |
| label-noise robustness check because the paired t ≈ −1.9 sat inside the range |
| where phase-29's label-arrival luck can decide the sign. |
| |
| **Label-noise check (final): REFUTED.** Fixed holdout plans for logbag@1.5, |
| logbag@2.0 and cliff_count22@1.5 re-scored under 120 common-random-number |
| label draws (phase-29 lognormal+cure lag model refit on the full table: |
| mu 4.08 / sigma 1.33 / cure 0.205; crossing dates real; per-cell synth eol |
| covers ALL fleet-B batteries - verified, so any crossed device can die in any |
| draw; /tmp/x28/holdout_noise.py, results holdout_noise.csv). Paired delta vs |
| logbag@1.5: cliff_count22 **+60.5 ± 51.2, better in 8% of draws**; plain |
| α=2.0 +79.6 ± 50.9, better in 6%. The real-label −54.5 edge flips sign: it |
| was one favourable realisation of which knee-pool members got labelled |
| in-window, not mechanism. Caveat the check inherits: the fitted unconditional |
| lag median ~59 d is longer than the observed-event median 33 d (cure/censored |
| mass drags the fit), so the simulator over-weights long-lag worlds. |
|
|
| **Sensitivity bracket (short-lag world)**: same fixed plans and draws, lag |
| refit on observed events only (median 37 d, sigma 1.06, pi fixed 0.205; |
| /tmp/x28/holdout_noise_short.py): cliff_count22 flips to **−34.8 ± 41.1, |
| better in 81% of draws** (plain α=2.0: −18.2, 68%). The candidate's sign is therefore |
| decided by the label-process model, so a third, mechanism-true test was |
| built. |
| |
| **Barrier-event resampler (decisive): the edge is REAL.** Phase 30 |
| established the recorded EOL IS the 2.40 V corrected crossing (+ recording |
| lag, measured here independently: median 5 d, p90 21 d, n=60) - so the right |
| randomness for a parked knee device is a barrier event (descent continues |
| through 2.40 vs stalls above it), not a lag draw; both lognormal simulators |
| conflate descent time with recording delay. Resampler |
| (/tmp/x28/barrier_resampler.py): reality kept everywhere EXCEPT the contested |
| pool (in the 2.45 band at origin, not yet through 2.40, staleness<=7; |
| pool sizes 2-25 per cell), whose outcomes are redrawn from empirical |
| level-bucketed continuation rates (P(cross 2.40 within 42d): 0.47 / 0.69 / |
| 0.79 for [2.44,2.45)/[2.42,2.44)/[2.40,2.42), pooled 7d state grid) plus |
| empirical delay and recording-lag quantiles, common random numbers across |
| plans. Result over 120 draws: **cliff_count22@1.5 −66.8 ± 24.1 vs |
| logbag@1.5, better in 120/120 draws (p95 −28.3)**; plain α=2.0 −51.2 ± 15.6 |
| (also 100%, but ~16 worse than the cliff gate in every regime that rewards |
| appetite). |
| |
| **Verdict (final): cliff_count22 @ ALPHA_STATIC=1.5 is a CANDIDATE FOR |
| ADOPTION on the static held-out path** - real-label holdout −54.5 (4/5 fleet |
| splits), barrier-event resampler −66.8 (sign held in every draw), and the one |
| adverse result (censoring-corrected lag world, +60.5) is attributable to that |
| model's known mechanism mismatch. The native rolling path keeps the raw |
| count (panel +27; same gating pattern as ALPHA_STATIC itself). Wiring cost: |
| submit.py computes 3 hourly columns (frac245_7, tele_n7, corrected-voltage |
| gate) from the split's own raw timeseries and wraps the static model's |
| predict; all-48 must stay byte-exact (train path untouched by construction). |
| Adoption is a human decision - not shipped in this phase. Side finding: |
| under the censoring-corrected draws, appetite raises beyond ALPHA_STATIC=1.5 |
| price negatively across the board, retroactively supporting phase 26 |
| stopping at 1.5. All other phase-28 candidates REJECTED. |
| |
| **Independence scrutiny (closed).** Continuation events are NOT independent |
| across devices: per-device first-band-entry states (n=127, 54% continue) |
| overdisperse strongly across calendar quarters (dispersion 28.5 vs binomial |
| 6.0, bootstrap p<0.001 - hot/cold continuation regimes exist; buildings only |
| mildly, p=0.10; /tmp/x28/dispersion_check.py). A correlated variant of the |
| barrier resampler (shared per-quarter logit shock, sigma=0.88 calibrated to |
| the observed ~4.7x variance inflation, maximally correlated within draws; |
| /tmp/x28/barrier_corr.py) leaves the verdict unchanged: cliff_count22 |
| **−66.1 ± 26.1, better in 120/120 draws (p95 −20.2)** - cold-quarter tails |
| are bounded because a wrongly-included knee device sits near its imputed EOL |
| (cheap false swap), while hot quarters pay the 20:1 late rate. The quarterly |
| continuation regime itself is new label-process structure future phases |
| should model - see phase 31 ideas 3-5 for two candidate mechanisms recorded |
| there: seasonal temperature (warmer devices live longer, r=+0.20, so winter |
| = hot-continuation regime) and install-batch waves (85% of lifetime variance |
| is batch; batches descend together, confounding quarter with batch). The |
| correlated-shock verdict above is agnostic to the mechanism: the shock was |
| calibrated to the observed dispersion, and batch waves cluster in calendar |
| time, so per-quarter sharing proxies either cause. |
| Not ported (already ours or already refuted): unconditional recorded-EOL |
| target, censored-anchor pricing, exact-cost-model optimization, building |
| group moves (phase-7/22 search saturation), isotonic rejection, CDF lag |
| (they over-swap early; we under-swap). |
|
|
| **Public outcome (2026-08-20, submission 6ea03da): REFUTED - and it refutes |
| phase 26 with it.** 2188.78 vs 1661.45 (3c7b883, still the live phase-23 |
| rolling config at alpha=0.8). This was the first public exposure of the |
| ENTIRE static stack at once (24b static fit + 26 ALPHA_STATIC=1.5 + 27 |
| wiring + 28 cliff boost). Component signature: battery_swap 3.02 -> 5.08 |
| (+68% swaps), early 137 -> 582 (+444), ops +70, late 1212 -> 1225 (+13 - |
| ZERO recall gain). That is pure over-inclusion: on the real held-out fleet |
| the probability scale transfers (phase 20's finding: half-fleet sum(p42) |
| 15.8 vs 9.1 true, it OVER-counts), and the half-fleet holdout construction |
| that phase 26 and phase 28 both relied on manufactures the undercount their |
| gains priced in. Late cost on public is saturated by voltage-silent deaths |
| (the phases-1/6 result, again). Defaults reverted (ALPHA_STATIC=0.8, |
| CLIFF_SCALE=1.0; static-fit machinery and phase-27 hardening retained). |
| Standing lesson for every future phase: **the half-fleet holdout is refuted |
| as an appetite adjudicator** - it disagreed with phase 20 and public sided |
| with phase 20; appetite changes are public-look-only decisions now. WHY it |
| mispredicts (parallel session, deployment-faithful sweep same day): the |
| half-fleet scenarios carry ~6 deaths against public's much larger fleet, so |
| added inclusions look cheaper than they are; the measured marginal exchange |
| rate is 0.46 late-saved per early-spent at alpha 1.5->2.0 locally (0.21 at |
| 2.0->2.5) and ~0.03 on public - all far below the 1.0 break-even. The |
| half-fleet holdout remains valid for structural/mechanical questions |
| (static-vs-rolling, failure isolation), not for anything count-related. The |
| cliff boost's solo effect at alpha=0.8 is unproven (it was only ever |
| measured on top of the refuted 1.5 baseline) and stays off by default. |
|
|
| ## Phase 29 - fresh-eyes slate: generative label model, arrival rank/features, planner add/drop, policy control (all rejected; label-noise floor quantified) |
|
|
| Trigger: user-requested outside review ("maybe we went in the wrong direction |
| from the start"), ideated deliberately BLIND to this file (code + raw data |
| only), then every proposal tested. Runner: `exp_x27.py` (new file; frozen |
| modules imported, never edited; md5s logged per run - models_cv.py |
| 7ead54af... matched the pinned hash in all four runs). Parity: champion |
| reproduced byte-equal on s_8/s_9 vs research/baselines/tier1_champion.csv. |
| All runs at the tier-1 budget (r0/i20); champion reference **1697.4** on the |
| 34-origin panel. Outputs in /tmp/x27/. (Numbered 29: phase 27 = parity |
| audit, 28 = public-repo port, both from parallel sessions.) |
|
|
| Raw-data facts established during the blind review: |
|
|
| - All 82 recorded deaths crossed 2.45 V before EOL, and 42d before their |
| crossing ALL sat in the 2.47-2.77 descent band (median 2.575); the |
| >2.75 -> 2.45 descent takes median 229d (p25 108, min 4). Detection is |
| saturated; the residual is timing (independently re-derives phase 15/17). |
| - Crossing -> recorded-EOL lag: median 33d, p90 ~140d - wider than the |
| window; the label process is the dominant randomness. |
| - All 82 deaths transmitted up to their EOL day (median pre-EOL data silence |
| 0d, p90 also 0): labels REQUIRE telemetry, devices dark at their would-be |
| death ghost out. 33 undocumented swaps detectable (median pre-swap v_corr |
| 2.667: proactive replacements; 30/33 unlabeled). |
| |
| | candidate | mechanism | panel | sel% rand/blk | verdict | |
| |---|---|---|---|---| |
| | lagrank_b03 | lognormal+cure arrival prob as RANK blend (w=0.3), count untouched | 1716.6 (+19; 10W/7L/17T) | 14.8/5 | rejected | |
| | lagrank_ovr | same, rank override for crossed devices | 2698.2 (+1001; 1W/33L) | 0/0 | rejected - closes the phase-17 loophole: LAH fails through ORDERING too, not only count explosion | |
| | twostage | generative EOL date: Wiener 2.45-crossing CDF (x) lognormal+cure lag | 2924.6 | 0/0 | rejected | |
| | ts_anchor | logbag p42 multiset (exact count), twostage order | 2659.6 | 0/0 | rejected - the ordering itself is destructive, not just the count | |
| | logbag_db245 | days_below_245 as plain logistic feature | 1747.2 (12W/20L) | 9.3/15 | rejected | |
| | logbag_arr | stacked arrival probability as feature (soft LAH; count cannot explode) | 1755.5 (15W/15L) | 7.2/20 | rejected | |
| | logbag_dark | staleness>14 cliff flag | 1733.0 (13W/9L; one +1245 blow-up s_36) | 18.9/5 | rejected | |
| | adrop planner | add/drop moves in the day search, EXACT marginal-mobility pricing | 1769.4 (+72) | - | rejected: buys late -46 for early +130 - phase-21's losing trade reached from the planner side | |
| | policy rule | 3-param direct rule (tau_projected<=theta, dark cliff, day frac), odd-tuned / even-holdout | 2094.8 full (holdout 1997.6) | - | rejected: the ML stack is worth ~400/scenario; the grid independently selects the 14d dark cliff | |
| |
| Champion selected 49.9% random / 55% blocked against the whole slate - |
| stands. Static-holdout regime not run: nothing passed the panel gate |
| (necessary condition per the three-regime protocol). |
| |
| Mechanism findings: |
| |
| 1. **Why generative label models fail: cure-tail unidentifiability.** The |
| walk-forward lognormal+cure MLE collapses pi -> 0.00 at every panel-era |
| refit (63-69 events): sigma inflates to absorb the censored ghosts, every |
| crossed device "eventually gets a label", and sum(p42) overcounts 2-3x |
| (s_24: twostage 26.9 vs 9 true deaths; logbag 15.2). On the FULL table |
| the cure mass IS identified (pi=0.205, mu=4.08, sigma=1.33) - the |
| ghost-vs-tail decomposition becomes identifiable only with data no |
| walk-forward fit can have. This is the structural reason every explicit |
| label-process model (phases 15, 17, 29) loses to the discriminative |
| classifier: dies42 already integrates the arrival process over exactly |
| the horizon that matters, using each of the 82 events once at full |
| efficiency. |
| 2. **Label-luck noise floor (new standing capability: `exp_x27.py noise`).** |
| FIXED champion plans re-scored under 150 resampled label processes (real |
| crossing dates; lognormal+cure lags, per-scenario truncation so devices |
| unlabeled at an origin stay unlabeled): panel-mean sd ~710 across label |
| worlds; the PAIRED champion-vs-challenger delta (common draws) has sd 49 |
| (p5 -60 / p95 +103) for a pair whose real-panel gap is 19. Caveat: the |
| fit makes synthetic worlds harsher than reality (mean 3895 vs 1697), so |
| this overstates - level-scaled it is still ~+-22. Reading: panel deltas |
| under ~50 are substantially label-arrival luck; this is the mechanism |
| behind the 61-76 selection regret the stability runs measure. Corollary: |
| lagrank_b03 BEATS logbag by 25+-49 in the synthetic worlds (where ghost |
| labels do arrive) - its mechanism is real; the real world's label |
| generosity decides the sign, and it decided against. |
| |
| Verdict: champion unchanged (panel 1697.4 tier-1 / 1679.2 full budget). The |
| wrong-direction-from-the-start hypothesis is adjudicated NO from four |
| independent angles (generative date model, arrival-informed ranking and |
| features, EV-driven planner inclusion, model-free policy rule): at 82 |
| events, bagged logistic + count-based topsum remains the sweet spot. The |
| durable addition is the label-noise simulator - a future candidate whose |
| panel delta is under ~50 should be treated as undecidable on this panel |
| regardless of selection percentage. |
| |
| ## Phase 30 - label event identified: recorded EOL IS the 2.40 V crossing (knowledge adopted; both exploitation channels closed by forensics, no panel runs spent) |
| |
| Follow-up to phase 29 ("try harder"), forensics-first: identify the label |
| mechanism, then test exploitability directly on the champion's actual panel |
| misses (the saved phase-29 plans), BEFORE building any model. |
| |
| 1. **Mechanism.** 81/82 recorded deaths reached v_corr < 2.40 before their |
| EOL; lag from first-2.40-crossing to recorded EOL: **median 5d, p25 1d** |
| (vs 33d from the 2.45 crossing - phases 15/17 measured the wrong clock: |
| most of the "administrative lag" is descent time 2.45 -> 2.40). v_corr at |
| EOL median 2.396. Ghosts: of 34 devices crossed-2.45 >= 42d with no EOL, |
| 20 NEVER went below 2.40 (min v_corr median 2.405) and the 14 that dipped |
| sit at 2.400 exactly. To first order the label process is a barrier event |
| at ~2.40 with days-scale recording, gated on the device transmitting. |
| 2. **Channel A - knee-projection inclusion: closed.** Champion panel misses: |
| 132 (3.9/origin; v_corr median 2.468: 60 crossed / 48 droop / 24 |
| healthy). Under per-device sub-threshold extrapolation (below-2.55 slope |
| over the last 90d, projected days to 2.398 minus staleness) the misses |
| rank **median 35** among non-swapped transmitting devices (best 6, ZERO |
| in top-5, 7 in top-10); their own projections say median 53d - the final |
| descent accelerates past any linear fit. Catching the median miss means |
| ~35 false swaps x ~100-150 early cost against ~300-450 saved. The knee |
| pool (30-90 devices/origin) is near-exchangeable at the 42d horizon: |
| WHICH member punches through is not predictable from trajectory state. |
| 3. **Channel B - hover-ghost demotion: closed (and instructive).** The rule |
| "v_corr in [2.395, 2.47), stale <= 14d, knee slope < 0.5 mV/d" looks like |
| the ghost signature at device level, but at pair level that cell dies at |
| **34%** in-window - and the champion's inclusions from it die at **47%** |
| vs 17% for its exclusions. logbag's p42 already discriminates within the |
| knee pool BETTER than mechanistic extrapolation (drop_recent/below_frac |
| carry it). Phase-15's GHOST_ZERO loss now has a mechanism: "hover" is a |
| device-level survivorship artifact, not a pair-level state. |
|
|
| Conclusion: the panel's residual late (~1022/origin) is the exchangeable |
| knee-pool hazard, not a selection error. Combined with the phase-29 noise |
| floor (paired sd ~49), the per-device channel on the native panel is closed |
| with a mechanism, not just a scoreboard. The 2.40-barrier fact stands as the |
| correct mental model of the metric's label side - and of ALPHA_STATIC's bet: |
| held-out knee pools are larger and FP is cheap near data end, so buying the |
| pool wholesale is right exactly there. Champion unchanged. Artifacts: |
| /tmp/x27/miss_forensics.csv; zero panel runs consumed by this phase. |
|
|
| ## Phase 31 - user ideas screened: donor-curve k-NN and best-case-deficit (both rejected at tier 0; tau_projected residual re-ranking discovered as a deployment candidate) |
| |
| Two user-proposed similarity channels, adjudicated by the phase-30 forensic |
| protocol (rank-of-misses on the champion's saved panel plans + cell AUCs) - |
| zero panel runs spent. Screen code: exp_knn_screen.py, /tmp/x27/knn_screen.log. |
|
|
| **Idea 1 - donor-curve k-NN EOL** (align recent v_corr history against each |
| INDIVIDUAL dead device's trajectory, k=5 weighted by fit error, donors |
| restricted to deaths <= origin): the only channel all day that beats the |
| champion's own residual ordering - misses rank median 16.5 (knee channel: |
| 35), 27/102 in the residual top-5 vs logbag-next's 21. But it is DOMINATED |
| by plain tau_projected re-ranking (35 in top-5), loses global AUC to it |
| (0.883 vs 0.926, 1/34 origins) and times deaths worse (median |err| 14.7d vs |
| 11.5d). Verdict: rejected - the median master curve + per-device alignment |
| already saturates the donor-similarity signal; individual donors add k-NN |
| variance, not information (descent shapes are stereotyped; speed varies but |
| speed is not identifiable early - phase 30). |
|
|
| **Idea 2 - deficit vs the longest-lived donor** (age-aligned v_corr gap to a |
| best-case reference curve from the top-3 longest-lived donors dead by |
| origin; target: EARLY degradation sign on the healthy cohort): AUC on |
| healthy pairs (v_corr > 2.55) **0.565** vs v_drop 0.795 and tau_projected |
| 0.725 - worse than the existing own-baseline features exactly where it was |
| supposed to help; all-pairs 0.812 (dominated); residual top-5 23 vs tau 36. |
| Verdict: rejected. Mechanism: cross-device age alignment inherits per-device |
| voltage offsets (own-baseline v_drop cancels them) and the ~190d-sd lifetime |
| spread means same-age devices sit at wildly different life fractions on a |
| flat plateau - the deficit is offset noise until the knee, where level |
| features already fire. |
| |
| **Discovery (the useful residue): tau_projected as residual re-ranker.** |
| Ordering the champion's NON-swapped descent-band devices by tau_projected |
| catches 24/36/44 of its 132 panel misses in top-3/5/10 - better than knn |
| (16/27/41) and than logbag's own next-by-p42 (15/21/35). The linear logistic |
| underweights its best timer exactly in the descent band where its misses |
| live. At panel FP prices (~120/false swap) even 36% top-5 precision is |
| EV-negative (+5 adds ~600 vs ~360 saved), so NOT a panel candidate. It IS |
| the natural ordering for the static path's ALPHA_STATIC marginal inclusions |
| (held-out FP ~20-40 -> strongly +EV if precision transfers): adjudicate in |
| the fleet-holdout harness with the phase-28/29 label-noise check before any |
| adoption. Logged as the open deployment-regime candidate. |
| |
| **Idea 3 - V-T coupling (rolling correlation/slope of voltage vs |
| temperature, seasonality removed): rejected.** Physics hypothesis: rising |
| internal resistance couples voltage to temperature before the level moves. |
| Four leak-free variants screened (exp_vt_screen.py): within-day slope |
| bvt_60 (seasonality-proof by construction), its own-baseline change |
| bvt_delta, within-day correlation cvt_60, and the literal detrended 90d |
| rolling correlation rcorr_90. The physics is REAL but lagging: bvt_60 |
| carries AUC 0.779 on all pairs - and is dominated by v_drop (0.888) and |
| tau_projected (0.909) in every cell, including the healthy early-sign |
| target (best variant bvt_delta 0.675 vs v_drop 0.795, beats it on 3/12 |
| origins). Residual-miss test: 2-5 in top-10 vs tau_projected's 44. Twist: |
| the correlation form is ANTI-predictive on healthy pairs (cvt_60 AUC 0.288, |
| inverted 0.712) - on the healthy cohort the informative direction is V-T |
| DE-coupling (erratic voltage), which is the already-adjudicated |
| vstd_30/rng_30 plunge-precursor channel (phase 6/SilentFix, rejected). |
| Same verdict shape as phase 25: the gradient exists, the voltage state has |
| already priced it in. |
|
|
| **Idea 4 - temperature std as plunge precursor: rejected (no signal).** |
| User recollection check: the recorded precursor was VOLTAGE std |
| (vstd_30/rng_30, phase 6/SilentFix, tested and rejected); temperature std |
| itself had never been tested. Three leak-free variants screened |
| (/tmp/x27/tstd_screen.log): within-day temp std (30d median), its |
| own-baseline delta, and detrended 30d rolling std of daily medians. All at |
| or near chance (best cell: detrended rolling std on healthy pairs, AUC |
| 0.628 vs v_drop 0.795); residual-miss test 3-8 in top-10 vs tau_projected's |
| 44. Curiosity only: within-day temp std is INVERTED on healthy pairs (AUC |
| 0.217, i.e., 0.783 flipped - flat temperature readings precede death), |
| still below v_drop and consistent with a sensor-quality artifact of the |
| sagging supply voltage rather than an independent channel. |
|
|
| **Idea 5 - infer per-device load, improve the theoretical (coulomb) model: |
| rejected at the physics level.** Load IS partially inferable (reads/day |
| 17-24 p10-p90, thermal exposure, within-day sag all vary), but lifetime |
| variance decomposition on the 82 deaths (full-life averages; mechanistic |
| study, not an origin feature) shows there is nothing for it to explain: |
| install-date batch alone R^2 = 0.853 (30 batches; confirms the recorded |
| ~90% claim), all four load proxies together R^2 = 0.102 alone and +0.04 |
| incremental over batch-month; within-batch residual sd 90d with best proxy |
| correlation r = +0.23 (t_mean) - perfect load inference would shrink the |
| 90d within-batch spread by ~2d. Death timing in this fleet is set at |
| manufacture (lot capacity), not by usage; the load-adjusted clock collapses |
| to the batch clock, already rejected (phases 6/10). Sign note: warmer |
| devices live LONGER (r = +0.20), opposite the self-discharge story - |
| consistent with cold raising internal resistance and pulling the 2.40 |
| crossing earlier. |
| |
| ## Phase 32 - "is p42 dangerous?" - window-portability bug FIXED, miss-cost ranking rejected |
| |
| Two user hypotheses about p42, both tested; one found a real catastrophic-tail |
| bug, the other closed a channel. |
| |
| **(a) Window portability.** Worry: public/private could use a window other |
| than 42d. The evaluator honours `planning_window_days` per scenario, so this |
| is directly measurable: score unchanged plans against 28/56/84d evaluators. |
| |
| | evaluator window | before fix | after fix | |
| |---|---|---| |
| | 28d | 728.5 | 728.5 (inert) | |
| | 42d | 1086.6 | 1086.6 (byte-identical) | |
| | 56d | **23581.9** | **1688.0** | |
| | 84d | (same failure mode) | 3283.1 | |
| |
| The model side degrades gracefully; the danger was in the OUTPUT convention. |
| plan_tsroute parks non-included batteries at day 43 and relies on the |
| evaluator cutting rows past the window - with W > 42 the evaluator EXECUTES |
| all ~200 parked rows as real swaps and charges early penalties on the whole |
| fleet. **Fixed** (`repark_for_window`, applied in all three plan paths): |
| parked rows move to `max(W, 42)+1` read from the live settings. Inert at |
| W <= 42 by construction (all-48 checksum byte-exact). Residual rise at 56/84d |
| is legitimate, not a bug: a longer window simply contains deaths a 42d-ahead |
| planner never scheduled for. |
|
|
| **(b) Miss-cost inclusion ranking (rejected).** Hypothesis: late_swap is high |
| because inclusion ranks by raw p42, which treats a day-5 death (~430 if |
| missed) like a day-40 death (~80). Tested `rank_score = p42 x E[late penalty |
| if parked]` on the static held-out path (same alpha=1.5 count, same planner, |
| only the boundary ORDER changes), 3 fleet splits x 16 origins: |
|
|
| | run | p42 | miss-cost | delta | |
| |---|---|---|---| |
| | s0 | 925.9 | 894.9 | -31.0 | |
| | s0 reversed | 778.6 | 815.1 | +36.4 | |
| | s1 | 914.0 | 898.7 | -15.3 | |
|
|
| 13W/12L/23T - no unanimity, rejected under the phase-26 standard. The |
| mechanism does what it claims where it wins (late 462->439, 524->504) but the |
| gains are small and reverse on one split: **p42 and death-day are strongly |
| correlated** (a confident p42 usually means already-deep decline, i.e. dying |
| soon), so the boundary is already nearly cost-ordered and the differences |
| trade on day-quantile noise. Conclusion for the record: the residual |
| late_swap is NOT mis-ordering among visible deaths - it is the voltage-silent |
| deaths (channel closed, phases 6-13) plus label-lag noise. |
| |
| ## Phase 33 - late-swap root cause: the count is optimal, ranking is the wall |
| |
| Public 6ea03da = 2188.78 (vs 3c7b883 1661.45) triggered a full attack on |
| late_swap. Six mechanisms tested, all rejected; the diagnosis is now closed |
| and quantitative. |
|
|
| ### Forensics (panel s_8-41, alpha=0.8, full evaluator) |
| - late is **97.9% MISSES** (31070) vs 2.1% caught-but-swapped-late (660 over |
| 34 origins). We already swap a median **12.2 days BEFORE** EOL, so the |
| "shift swap days earlier" 20:1 lever DOES NOT EXIST. |
| - recall 176/308 = 57%. Caught deaths: max rank 20, median rank 5. |
| Missed deaths: **median rank 35** of ~450 - not marginal, not invisible. |
| - miss signature = **LINGERERS**: median days_below_245 **63** vs 29.5 for |
| caught; tau_projected 17.7 vs 6.4. They entered the 2.45 V zone long ago |
| and decline slowly. |
| - marginal economics: a marginal swap costs ~54 early and catches 8% x 251 = |
| **EV 21**. Break-even density is 54/251 = **21.5%**. Ranks 11-12 run 32% |
| (keep), 13-15 run 11.7%, 13-30 run 8% (reject). |
|
|
| ### The alpha question, settled on the SHIPPING path |
| Definitive curve (rolling native fit, panel, full cost): |
|
|
| | alpha | 0.6 | **0.8** | 1.0 | 1.2 | 1.5 | 2.0 | |
| |---|---|---|---|---|---|---| |
| | swaps | 13.6 | **15.2** | 17.4 | 19.4 | 22.8 | 28.5 | |
| | late | 1270 | **1019** | 944 | 866 | 817 | 691 | |
| | early | 330 | **444** | 604 | 758 | 1017 | 1409 | |
| | total | 1844 | **1679** | 1771 | 1862 | 2084 | 2369 | |
|
|
| Clean U with the minimum exactly at 0.8 (1.0 is better on 7 origins, worse on |
| 24). More swaps DO cut misses - late falls 1019->691 - but early rises 3x |
| faster. At 28.5 swaps our late (691) still never reaches the leader's 499 |
| while our early (1409) is 2.3x theirs: **there is no alpha that reaches the |
| leader profile**. Their 19 swaps are a CONSEQUENCE of better ranking |
| (marginal hit rate ~40% vs our 8%), not a cause. Matches the public result |
| (+8 swaps -> late -13, early +444) and the 0.46 -> 0.21 -> 0.03 marginal |
| rates from phase 28's addendum. |
|
|
| ### Ranking attacks - all rejected, all with the same signature |
| Metric note (IMPORTANT): score WHO candidates on **top-12 recall**, never |
| AUC. Every candidate below RAISED AUC while LOWERING the recall that |
| converts to cost. |
|
|
| | candidate | AUC | top-12 recall (base 175/308) | verdict | |
| |---|---|---|---| |
| | zone features (days_below_245, in-zone, zone rate) | +0.0061 | 167 | rejected | |
| | v_corr_last + thresh_dist (the label IS the 2.40 crossing) | +0.0088 | 173 | rejected | |
| | both | +0.0111 | 169 | rejected | |
| | RF ranking (rank_score) + logbag count | +0.0140 | 161 | rejected | |
| | rank-avg 50/50, 75/25 logbag | +0.014 | 172 / 176 | noise (4W/3W) | |
| | cohort model (train only on at-risk) | +0.0163 | 170 | rejected | |
| | cohort + zone | +0.0185 | 169 | rejected | |
| | sub-daily/intra-day (daily min, dip depth, dip accel, min-slope, dispersion) | +0.0009 | band hits 73->72 | rejected | |
| | miss-cost ranking (p42 x E[late if parked]) | - | 13W/12L/23T | rejected | |
| | prevalence recalibration (floor count by recently RECORDED deaths) | - | corr -0.03..-0.18 vs sum(p42) +0.39..+0.52 | rejected | |
|
|
| Cohort diagnostic (why the AUC/top-k divergence): the at-risk cohort is |
| 28.9% of devices and holds **99% of deaths** (7.6% death rate inside vs 0.03% |
| outside). A global model spends most capacity on the healthy-vs-sick boundary |
| that AUC rewards and we never act on; but restricting to the cohort did NOT |
| help either (170 vs 175), so the dilution hypothesis is refuted too. |
|
|
| Sanity check - ML ranking vs raw physics (top-12 of 308): logbag **175**, |
| own-trajectory time-to-2.40 119, thresh_dist 108, v_corr_last 108, |
| master-curve tau 51. Our ranking is far better than physics; it is simply not |
| good enough at ranks 13-40 to make extra swaps pay. |
| |
| **Conclusion**: late_swap is bounded by WHO-ranking quality among lingerers, |
| not by appetite, timing, features-on-hand, or model family. The count channel |
| is closed on the panel AND on the public leaderboard. |
|
|
| ## Test-look ledger |
|
|
| 8 looks total (now historical - see protocol change above); look 9 = the |
| near-miss backfill table. Original ledger: 8 looks (threshold stack, topsum, ecroute+ablations, tsroute, aug-tsroute, |
| +tau_projected, logbag+α-challenger, aux-features). Dev/test disagreed |
| persistently by ~0.7σ (dev prefers richer stacks, test the simpler v1); |
| selection followed dev + the combined mean, with the public leaderboard as |
| final arbiter. |
| |
| ## Phase 34 - cross-fleet CV produces its SECOND confident false positive |
| |
| Four candidates confirmed on the real leak-free full-fleet all-48 walk-forward |
| (`/tmp/all48_rank.py`, `/tmp/all48.py shape`). Base reproduced **1661.7 exactly**, |
| so the harness is sound. |
|
|
| | candidate | all-48 | delta | W/L/T | warm-up | panel | near-end | |
| |---|---|---|---|---|---|---| |
| | base (champion) | **1661.7** | - | - | 2379.7 | 1679.2 | 604.8 | |
| | rankavg (.75 logbag + .25 rf rank) | 1717.0 | **+55.3** | 12/19/17 | 2610.2 | 1703.1 | 604.8 | |
| | cohort_hybrid (at-risk cohort refit, gated) | 1713.1 | **+51.5** | 13/20/15 | 2573.3 | 1706.4 | 604.8 | |
| | both | 1679.2 | +17.5 | 16/23/9 | 2519.2 | 1670.7 | 607.4 | |
| | shape features (sh_*) | 1791.4 | +129.7 | - | 2500.6 | 1830.0 | 627.1 | |
| |
| `rankavg` had won cross-fleet CV at **-18.4, 27/32 folds, fold-level p=0.0002**, |
| robust to outlier removal and surviving Bonferroni across 5 candidates. It is |
| +55.3 **worse** on real data. This is not a warm-up-weighting artifact: it loses |
| on the **panel alone** (+23.9), where models are well trained. |
| |
| **Second validity boundary of cross-fleet CV** (the first was half-fleet appetite |
| bias, which refuted `ec_pmax=inf`): every fold trains on the FULL history, so the |
| harness *structurally cannot* reproduce cold-start scarcity. RF ranks well when |
| well-trained and is noise at 3-20 deaths; blending it helps in a world where |
| every model is well-trained, and hurts in ours (warm-up +230.5 of the +55.3). |
|
|
| **Verdict: cross-fleet CV is retired as a selection instrument.** Both effects it |
| called significant failed to transfer. It over-rejects nothing and over-accepts |
| anything that (a) shifts count/appetite or (b) benefits from a large train set - |
| which is nearly every candidate worth testing. Screen on full-fleet all-48 only. |
|
|
| ### Also closed this phase |
|
|
| - **Warm-up physics prior** (`/tmp/warmup_prior.py`): at the cold origins s_0-7 |
| logbag is *already* the best ranking source - top-16 recall **75** vs |
| blend25phys 72, blend50 70, phys_ttc 66, master-curve tau 50. At s_8-11 logbag |
| dominates outright (39 vs 31/30/29). No physics prior to graft on. Line closed. |
| - **Optimizer budget** (`/tmp/a48opt.py`): 2 restarts x 60 iters is at the |
| optimum. 6x120 = +2.1 (13W/12L/23T), 12x200 = +6.7 (19W/20L/9T). More search |
| buys nothing; the day-placement objective is not search-limited. |
| - **Feature LOO at bags=120**: best removal is `-age_days` at top12 177 vs 175 |
| (+2/308, well inside noise); `-v_last` collapses it to 160, confirming v_last |
| as the load-bearing feature. No removal adopted. |
| |
| ### Note on regime weighting (not acted on) |
| |
| Warm-up is 23.9% of the all-48 checksum, but `submit.py` fits held-out splits |
| with a **single static foreign model** (line ~218) - those splits never see |
| cold-start. The checksum therefore over-weights a regime the scored public and |
| private splits do not have. Left as-is: the leak-free alternatives all |
| reintroduce the half-fleet appetite bias, and every phase-34 candidate loses on |
| the panel too, so nothing hinges on the weighting. |
| |
| ## Phase 35 - full re-screen on the surviving instrument (72 configs) |
| |
| Everything ever judged on a retired instrument (cross-fleet CV, the half-fleet |
| static holdout, panel-only, top-12 recall) re-run on the leak-free full-fleet |
| all-48 walk-forward. Base reproduced 1661.7 in every group. `/tmp/screen48.py`. |
| |
| **Nothing beat the champion.** Summary of deltas (negative = better): |
| |
| | group | result | |
| |---|---| |
| | cliff (8 flavours) | count 1.5/2.2/3.0 = +19.0/+21.0/+3.0; rank-only +22.9; linger +9.5; p42-floor 0.25/0.50 = +40.3/+104.0; park-floor +102.5 | |
| | appetite alpha | 0.9 +67.8, 1.0 +113.8, 1.2 +228.2, **1.5 +479.1** (W3 L40) | |
| | ec_margin | 0 +44.2, 0.4 +12.5, 0.5 -10.9, 0.6 -3.2, 0.7 -5.9, 0.75 -4.2, 0.9 +7.1, 1.0 +19.2, 1.2 +47.8 | |
| | ec_pmax | 10 +41.9, 60 +16.0, inf +28.8 | |
| | route | sd0 +17.9, sd3 +16.3, sd14 +16.6, nn +0.6, emerg48 +6.0, emerg52 +21.3, **ecroute 0.8/1.0 = +441.1/+436.3** | |
| | floor | scale 1.5/2.0/3.0/3.5/4.5 = +105.8/+25.3/+39.8/+83.3/+172.5; gate 0/10/30/40 = +530.4/+158.2/+127.3/+150.8; nozone +113.3 | |
| | daycurve | day_q and p_conf **inert (T48, delta exactly 0)**; pbe0.3 +125.8, pbe0.5 +2.4, fday3 +7.7, fday7 +12.7 | |
| | model | **rf +151.3**, C0.3 +48.5, C3 +60.4, bags200 +11.9 | |
| | features | zone +104.9, vcorr +32.8, ts +116.3, shape +129.8 | |
| |
| ### Cross-fleet CV: four false positives and one sign inversion |
| |
| | mechanism | cross-fleet said | all-48 says | |
| |---|---|---| |
| | ec_pmax=inf | -35.3, p=0.031 | +28.8 | |
| | rankavg | -18.4, p=0.0002, 27/32 folds | +55.3 | |
| | ecroute1.0 | -27.7 (862.1 vs 889.8) | **+441.1** | |
| | rf | -26.4 (863.4 vs 889.8) at 8.2 swaps vs 7.3 | **+151.3** | |
| | alpha | monotone DOWN to 1.2 (805.4) | monotone UP (+479 at 1.5) | |
|
|
| Every one is an appetite artifact of scoring half a fleet. Retired for good. |
|
|
| ### gbm is not merely bad, it is undeployable |
|
|
| `HistGradientBoostingClassifier` raises `ValueError: window shape cannot be |
| larger than input array shape` binning the walk-forward training sets - at |
| s_0-7 AND at s_8-47. It only ever ran under cross-fleet CV because every fold |
| there trains on the full history. Shipping it would have been a hard crash. |
|
|
| ### The break-even top-up is a switch, not a dial |
|
|
| `ec_margin`/`ec_pmax` move only s_39-46; s_0-38 and s_47 are byte-identical. |
| Cause: the gate is `fp_price = EARLY*(e_far-42) <= ec_pmax` with |
| `e_far = (device end_time - T) + 30`, and `end_time` is one global constant |
| (the dataset's last timestamp, 2026-08-01). Eligibility is therefore a step |
| function of calendar distance to the data end: |
| |
| | scenarios | eligible | |
| |---|---| |
| | s_0-38 | 0-2.6% | |
| | s_39-46 | **96.1%** | |
| | s_47 | 0% (e_far < 42, gate fails) | |
| |
| s_39 sits on the edge at fp_price 24.7 vs the 25.0 cap. Consequences: |
| |
| 1. The apparent `ec_margin=0.5/0.7` wins (-10.9/-5.9) rest on 8 scenarios, one |
| of which (s_40) swings +389/+463. The curve is non-monotone across |
| 0.5->0.6->0.7. Noise, not an optimum. **Not adopted.** |
| 2. `park_mobility` 1.0/2.0/3.0 are byte-identical and move exactly ONE |
| scenario (s_39, -247). Single-scenario luck. **Not adopted.** |
| 3. **Deployment has the SAME geometry - our harness is faithful here.** |
| `iterate_scenarios` (batteryswap_public/utils.py:163) truncates the |
| *timeseries* per scenario (`flat.loc[:scenario_start]`) but passes the |
| locations table through untruncated, so `end_time` at runtime is the |
| split's global data end, exactly as locally. And `load_context` |
| (models_cv.py:1814) calls `iterate_scenarios` itself, so the all-48 |
| harness reproduces the runtime geometry exactly. The switch therefore |
| fires on the same ~8-of-48 tail fraction on a held-out split, assuming a |
| similar scenario spread; it is NOT secretly active everywhere at runtime. |
| (An earlier draft of this section claimed the opposite, reading the |
| plan_tsroute comment's "widens automatically ... at runtime" as meaning |
| every held-out scenario. It does not - only those within ~12-61 days of |
| the split's data end.) Residual unknown: the held-out splits' scenario |
| spread relative to their data end is not observable from here. |
| |
| ### Latent hole: `unobserved_eol_days` is hardcoded |
| |
| The evaluator imputes EOL = `location end_time + settings.unobserved_eol_days` |
| (evaluate.py:291). models_cv hardcodes `+ 30.0` twice: cost_curves:1169 (the |
| survivor branch of EVERY day-cost curve) and plan_tsroute:1451 (the top-up |
| gate). submit.py wires EARLY/LATE per scenario and warns on |
| planning_window_days, but not this. |
| |
| Exposure measured (`/tmp/unobs.py`, planner-assumes-30 vs planner-wired, the |
| evaluator using U in both arms; U=30 control reproduced 1661.7 with the arms |
| byte-identical): |
| |
| | U | unwired | wired | gap | worst scenario gap | |
| |---|---|---|---|---| |
| | 10 | 1623.9 | 1619.4 | +4.5 | +790 | |
| | 20 | 1642.7 | 1621.4 | +21.3 | +747 | |
| | 30 | 1661.7 | 1661.7 | **0.0** | 0 | |
| | 45 | 1699.6 | 1698.8 | +0.9 | +170 | |
| | 60 | 1737.6 | **1756.8** | **-19.2** | +289 | |
| |
| Wiring is a provable no-op at the default and helps in 3 of 4 off-default |
| cases, but is 19.2 WORSE at U=60 - because `ec_pmax=25` was itself tuned |
| assuming 30, so correcting the price without retuning the cap just moves the |
| switch. Not a free correctness win. |
|
|
| **Risk assessment**: all 48 local scenarios carry an explicit settings block, |
| and the ONLY fields the organizers vary are `base_location` (16 values) and |
| `base_room` (27). Every economic setting - window, penalties, |
| unobserved_eol_days, limits, overtime - is constant. So the hole is real but |
| unlikely to fire. Left unwired pending a decision; models_cv.py is hash-pinned |
| (exp_x28.py PINS) and the fix is not a measured gain. |
|
|
| ## Phase 36 - the fit-protocol axis: a real warm-up mechanism that deployment never runs |
|
|
| Features, model families, hyperparameters and planner knobs are all swept out. |
| The untouched axis was **how the training set is built and weighted**. |
| Motivation: at s_0 the training set is ~10k rows holding 3-20 positives, each |
| device contributes ~168 autocorrelated grid rows (so long-lived devices |
| dominate by negative count), and `tau_projected` is entirely NaN there - the |
| imputer silently drops it. `/tmp/screenfit.py`. |
|
|
| Ungated arms, full-fleet all-48 (base 1661.7): |
|
|
| | arm | warm-up | panel | all-48 | |
| |---|---|---|---| |
| | base | 2379.7 | 1679.2 | 1661.7 | |
| | quar (QUARANTINE on) | **2202.4 (-177.3)** | 2172.7 | +395.0 | |
| | devw (device-equal weight) | **2211.0 (-168.7)** | 2053.0 | +254.7 | |
| | bal (class_weight balanced) | 2467.2 | 4709.4 | +2442.9 | |
| | recw180 / recw365 | 2465.6 / 2505.1 | 1748.4 / 1747.0 | +51.5 / +64.8 | |
| | thin2 / thin4 | 2401.4 / 2533.6 | 1697.4 / 1906.5 | +20.3 / +191.1 | |
| | logcal | 2756.1 | 1772.6 | +139.6 | |
| | ddepth3 | 2499.7 | 1680.9 | +16.7 | |
| |
| Clean regime split: `quar` and `devw` help warm-up and wreck the panel. Mechanism |
| is coherent - dropping the 26 "looks dead, labeled survives" ghosts and |
| de-weighting long-lived devices removes label noise/imbalance that only bites |
| when positives are scarce; with hundreds of positives those same rows are |
| informative hard negatives. |
| |
| **Gate them the way the floor is gated** (`train_deaths < 20`). Verified the |
| gate splits exactly at s_0-7: gated arms leave panel (1679.2) and near-end |
| (604.8) byte-identical. |
| |
| | gated arm | warm-up | all-48 | delta | W/L/T | |
| |---|---|---|---|---| |
| | devw_g12 / g20 / g30 | 2318.1 / 2211.0 / 2211.0 | 1651.4 / 1633.5 / 1653.2 | -10.3 / **-28.1** / -8.4 | 6/2/40 at g20 | |
| | quar_g12 / g20 / g30 | 2249.8 / 2202.4 / 2202.4 | 1640.0 / 1632.1 / 1663.9 | -21.6 / **-29.6** / +2.3 | 7/1/40 at g20 | |
| | **dq_g20** (both stacked) | **2124.7** | **1619.1** | **-42.5** | 7/1/40 | |
| | dq_g30 | 2124.7 | 1659.4 | -2.3 | 7/5/36 | |
|
|
| Gate 30 leaks into the panel (1736.0) and destroys the gain; 20 is the optimum, |
| matching FLOOR_GATE. Seed stability for the `devw` resample (`/tmp/seedcheck.py`, |
| 6 seeds): warm-up 2081.2 +/- 36.0, all-48 delta **-49.8 +/- 6.0**, range |
| [-58.8, -42.5]. **Every seed improves**, and the reported seed 0 is the worst |
| of the six. The mechanism is real and not a resampling artifact. |
| |
| ### ...and it is inert in deployment |
| |
| `submit.py` scores `public,private` only; both are held-out, so both take the |
| static branch where `train_uniq = static_uniq` = unique dead devices in the |
| full foreign train pool. **That number is 82.** Every gate tested (12/20/30/40) |
| is far below it, so the branch never executes on a scored split. |
| |
| This generalises beyond the candidate: **the whole warm-up regime - 23.9% of |
| the all-48 checksum - is structurally absent from the scored splits**, and the |
| existing cold-start floor has the same property. It retro-explains the record: |
| the floor moved all-48 2146.9 -> 1851.0 (a huge local gain) while public did |
| not move until phase 23's unmasked pooling landed independently. |
| |
| **Consequence for measurement**: s_8-47 is the deployment-faithful window - the |
| gate is shut there locally exactly as it is in deployment. Base on that metric |
| is **1518.0**. dq_g20 is byte-identical on it. |
| |
| Not adopted: inert where it is scored, and it would reset the checksum baseline |
| and break comparability with every prior phase. |
| |
| ### Free re-analysis: all 86 screened configs re-ranked on s_8-47 |
|
|
| If warm-up had been dragging a good config down, re-scoring on the deployment |
| window would surface it. It does not. The only negatives are the same |
| tail artifacts already rejected in phase 35 - ec0.5 -13.0 (still hostage to |
| s_40 +389), ec0.7 -7.1, parkmob -6.2 (one scenario), ddepth3 -4.0 (21W/18L, |
| noise). No hidden winner. |
| |
| ## Phase 37 - can the phase-36 mechanism be made always-on? No. |
| |
| dq_g20 is worth -49.8 on all-48 but never fires in deployment |
| (static_uniq=82 vs gate 20). Two hypotheses for why the UNGATED version cost |
| +374/+493 on the panel, both tested. `/tmp/always.py`. |
| |
| ### H1: devw's harm was a double-bootstrap artifact - REFUTED |
| |
| The phase-36 `devw` resampled with replacement and then handed the result to |
| BaggingClassifier, which bootstraps again. Re-implemented with true |
| `sample_weight` inside a manual 15-bag loop, with a uniform-weight control |
| (`mbag`) to absorb the implementation difference (that control is itself +90.5, |
| so comparisons are made against IT, not against base): |
|
|
| | arm | vs base | **vs mbag control** | |
| |---|---|---| |
| | mbag (uniform) | +90.5 | - | |
| | mbag_devw (1/n) | +479.7 | **+389** | |
| | mbag_devsq (1/sqrt n) | +288.9 | +198 | |
| | mbag_dev25 (1/n^0.25) | +91.0 | +0.5 | |
| |
| Clean dose-response: the more device weighting, the worse. With the double |
| bootstrap removed the harm is unchanged, so the implementation was not the |
| cause. Weak weighting is simply inert. |
| |
| ### H2: quar's threshold (2.48) is over-broad - REFUTED |
| |
| The filter is meant to remove the ~26 "ghost" trajectories (below the 2.45 |
| death threshold, no EOL recorded). At s_30 it actually drops: |
|
|
| | v_thr | rows | devices | |
| |---|---|---| |
| | 2.40 | 169 (0.3%) | **24** | |
| | 2.45 | 642 (1.3%) | 53 | |
| | 2.48 | 1066 (2.2%) | **77** | |
| |
| So 2.48 removes 3x the genuine ghost count - mostly real slow-decliners that |
| survived. Tightening it toward the ghosts should have kept the cold-start gain |
| and dropped the panel harm. It does not. On s_8-47 (base 1518.0): |
|
|
| | v_thr | 2.40 | 2.44 | 2.45 | 2.46 | 2.47 | 2.48 | k=0.5 | k=0.8 | |
| |---|---|---|---|---|---|---|---|---| |
| | delta | **+36.7** | +163.0 | +181.3 | +198.9 | +458.8 | +509.5 | +138.6 | +78.3 | |
| |
| Monotone, every value positive, and the limit of the trend is "do not filter" |
| (base = 0). Softening (keeping 50%/80% of the poison rows instead of deleting |
| them) walks the same curve back toward zero without crossing it. And at v_thr |
| 2.40 the warm-up block is 2379.7 - **identical to base** - i.e. the tight |
| filter that spares the panel also delivers no cold-start gain. |
|
|
| ### Conclusion |
|
|
| The benefit and the harm come from the SAME rows. Devices that look dead and |
| survive are label noise when the model has 3-20 positives and informative hard |
| negatives when it has 800+; the effect does not attenuate with data, it |
| **changes sign**. No threshold, no soft-weighting and no sample-weight scheme |
| separates the two. dq_g20 is intrinsically a cold-start mechanism and cannot be |
| made always-on - which, given static_uniq=82, means it cannot be made to matter |
| in deployment at all. |
|
|
| ## Phase 38 - the label process reverse-engineered (2026-08-21) |
|
|
| User question ("the voltage continues past the EOL line - if the battery fails |
| we should not have voltage anymore") exposed a documented error and, chasing |
| it, the exact rule that generates `eol_times.csv`. |
|
|
| ### Correction: post-EOL telemetry is the SAME battery, not a replacement |
|
|
| Phase 16 recorded: "All 82 dead slots keep reporting past EOL (mean 107d, max |
| 305d): **replacement batteries** under the same device_id", marked |
| human-verified. The durations are right (measured: median 85 d, mean 105, max |
| 305). **The interpretation is wrong.** Median voltage tracked against |
| days-since-EOL over all 82: |
| |
| | window | -90..-61 | -60..-31 | -30..-8 | -7..0 | +1..14 | +15..30 | +31..60 | +91..300 | |
| |---|---|---|---|---|---|---|---|---| |
| | median V | 2.562 | 2.510 | 2.454 | **2.398** | 2.379 | 2.363 | 2.352 | 2.295 | |
| |
| Smooth through the line, no discontinuity, and only **3 of 78** devices ever |
| reach fresh-battery level (>2.9 V) afterwards. Phase 16's own named example |
| `d_57e99ff00802` reports to 2026-05-13 as claimed and never exceeds 2.559 V. |
|
|
| Genuine swaps DO exist and are unmistakable - 34 censored devices jump to |
| >=2.95 V and hold (caveat 1 verified, ~30 claimed). They just never occur at |
| recorded EOLs. **Consequence**: a feature window spanning a recorded EOL does |
| not mix two batteries; segmenting history at recorded EOLs is unnecessary. The |
| only genuine two-battery contamination is those 34 censored resets. |
|
|
| ### The rule: gated 10-30 C, threshold 2.40 V, sustained crossing |
|
|
| `smooth_series`'s 10-30 C band is the ORGANISERS' default |
| (batteryswap_public/utils.py:42) but is never called by the evaluator - we |
| reimplemented it and inherited the constants. Never swept until now. Which |
| (band, threshold) best reproduces the recorded EOL dates: |
| |
| | band | median lag at 2.40 | coverage | |
| |---|---|---| |
| | 0-50 | 6.0 d | 100% | |
| | 5-35 | 6.0 d | 100% | |
| | 8-32 | 4.0 d | 100% | |
| | **10-30** | **3.0 d** | **100%** | |
| | 12-28 | 3.0 d | 98% | |
| | 15-25 | 5.5 d | 88% | |
| |
| 2.42 is far worse everywhere (10-23 d); 2.38 loses coverage. **(10-30, 2.40) |
| is the unique pair with 100% coverage at minimum lag.** The gate and the |
| threshold are one definition, not two choices that happen to agree - a voltage |
| reading is meaningless without a temperature. Independently, 10-30 is also the |
| tightest band retaining all 82 deaths (12-28 drops to 77, 15-25 to 70) and |
| peaks voltage AUC at 0.9604. |
| |
| **EOL marks the SUSTAINED crossing.** Median lag 3 d (IQR 3-6), but 12 of 81 |
| devices lag >21 d, max 163. Those are not measurement gaps: during the lag they |
| have a median of 23 gated readings and **83% go back above 2.40** (mean 57% of |
| readings above). They wobble. The lag is NOT predictable from origin-time |
| features: RF leave-out Spearman **+0.006** (p=0.95), R2 -0.193, and flagging |
| the tail hits base rate exactly. Volatility runs the wrong way (vstd_30 rho |
| -0.194). Characterisation, not a lever. |
|
|
| **No calendar structure**: EOL day-of-week is flat (weekend share 26% vs 29% |
| chance), so no human/administrative recording process; month is confounded with |
| fleet aging by window exposure; 68 distinct dates for 82 deaths is random |
| collision. Calendar features rejected as a memorisation trap - fleet-history |
| facts cannot transfer to splits at other dates. |
|
|
| **Consequence for relabelling (user-asked): do not.** The evaluator scores |
| against `eol_times.csv` verbatim, the label is tight (100% within 7 d of a |
| sub-2.40 gated reading, median 0), and the nearest relabelling - adding the ~26 |
| ghost deaths - was measured harmful at six thresholds in phase 37. Residual |
| error is entirely trajectory-forecast error with no label-process component, |
| which retro-explains phase 17's label-arrival-hazard rejection: arrival is |
| near-deterministic, so there was no hazard to model. |
|
|
| Artifacts: `notebooks/all_batteries.ipynb` (all 461 devices, categorised |
| DEAD-SILENT 18 / DEAD-VISIBLE 64 / RESET 34 / GHOST 17 / ADMIN-GONE 67 / |
| HEALTHY 261), `notebooks/silent_deaths.ipynb`. |
|
|
| ## Phase 39 - four hypotheses closed, four measurement laws (2026-08-21) |
|
|
| Five parallel sessions. Every line closed negative; the durable output is the |
| laws. |
|
|
| | hypothesis | verdict | |
| |---|---| |
| | install-batch death clock | CLOSED on 3 implementations. The famous "AUC 0.91 on silent pairs" rests on **4 positives** (phase 10 already knew: all-covered 0.584, dark bucket 0.426-0.410 - worse than chance). Coverage was never the blocker: 83-95% monthly at late origins. Gated rank-only form measured **+97.9 harm**. | |
| | `temp_mean` (30 d, pre-filter) | Mechanism replicates (deaths 18.67 C vs survivors 20.18) but **absorbed**: -1.1pp recall@12 on production. Fifth temperature rejection; the beta correction already occupies the channel. Gains +6.4pp on a champion with no temperature handling. | |
| | battery-type clustering | Types are real (13x risk spread surviving level-removal) but miss rate is FLAT across types. Cannot buy recall. | |
| | dv42 voltage-drop regression | CLOSED on 4 implementations. **Out-of-time R2 is negative everywhere** (-0.117 / -0.029 / -0.172): the drop cannot be forecast. The score was a differently-parameterised linear classifier anchored on voltage - ties the incumbent at N=12. Censoring-aware variant exactly inert (3% of rows); quantile variant worst on a full panel. Phase 10's wall from the other side. | |
|
|
| ### The four measurement laws |
|
|
| All the same shape: **pick the denominator that matches the question.** |
|
|
| 1. **Report recall at the operating N, not AUC.** With ~400 negatives/origin, |
| AUC 0.93-0.95 says nothing about a rule taking 12. Silent deaths measured |
| AUC 0.947 ("p42's strongest region") and recall@12 **0.000 of 24**, median |
| rank 60. Same fact, one currency. |
| 2. **A candidate must beat the incumbent ranker ON ORDERING before any |
| architecture can make blending it safe.** "Gated + rank-only" excludes |
| calibration and count damage but NOT ordering damage: reordering a good list |
| with a chance-level signal costs in proportion to the weight (+97.9 at |
| w=0.05). No safe non-zero weight exists for a ~0.5-AUC feature. |
| 3. **Condition on the outcome, not row shares.** Any partition (building, |
| cohort, type, room) looks significant against row shares because missed |
| deaths concentrate where deaths are. chi2 13.00 -> 2.67 on conditioning. |
| 4. **Always report the test panel.** Requiring a recent in-band reading on test |
| rows deleted 18% of deaths and swung v_now recall@12 from **0.800 to 0.382** |
| - larger than every modelling effect measured that day, and it reversed the |
| ranking of every arm. A recency filter deletes the silent population by |
| construction. |
| |
| ### Caution on the blind-harness champion lineage |
| |
| Two sessions independently found raw current voltage BEATS their champion's own |
| ranker at the operating N (.721 vs .668; .703 vs .672). On production the |
| reverse holds (.572 vs .429, +14.3pp). That lineage was ablated on COST under a |
| threshold rule where every feature change moves depth as well as order, so a |
| 4-feature set losing to `last_voltage` on ordering is more likely over-pruned |
| than optimal. **Do not port anything from it without re-running the ablation on |
| recall at fixed N.** |
|
|
| ## Phase 40 - the research loop's CV discipline, ported into this repo |
|
|
| Every prior phase re-derived its harness in `/tmp` and lost it. The three |
| measurement disciplines the blind loop paid leaderboard points to learn are now |
| repo-resident in `batteryswap_example/cv.py`, with the champion config itself |
| promoted out of `/tmp/all48.py` into `cv.CHAMPION`. |
|
|
| venv/bin/python -m batteryswap_example.cv all48 [base|vcorr|zone|vrel] |
| venv/bin/python -m batteryswap_example.cv static [variant] [n_seeds] |
| venv/bin/python -m batteryswap_example.cv screen |
| |
| **1. Ranking precision at the operating N, reported beside the cost.** |
| `models_cv.rank_stats` now runs inside `walk_forward`, so `dev`/`panel`/`test` |
| and the new protocols all emit rec12, rec20, operating-point recall/precision, |
| the count actually bought, and the median rank of missed deaths. |
| `summarize()` prints them, pooling caught/deaths over origins rather than |
| averaging per-origin ratios (a 1-death origin must not outvote a 12-death one). |
|
|
| Base reproduces **1661.7 / 2379.7 / 1679.2 / 604.8** exactly, so the additions |
| are diagnostic only. What they show that eight phases of cost-only reporting |
| could not: |
|
|
| | block | n | total | rec12 | rec20 | recall | prec | sched | median miss rank | |
| |---|---|---|---|---|---|---|---|---| |
| | warmup s_0-7 | 8 | 2379.7 | 0.526 | 0.735 | 0.763 | 0.570 | 19.8 | 30.5 | |
| | panel s_8-41 | 34 | 1679.2 | 0.572 | 0.675 | 0.571 | 0.458 | 11.3 | **57.0** | |
| | nearend s_42-47 | 6 | 604.8 | 0.837 | 0.935 | 0.893 | 0.250 | 16.7 | 50.8 | |
| | ALL-48 | 48 | 1661.7 | 0.597 | 0.718 | 0.641 | 0.453 | 13.4 | 49.2 | |
| |
| **The panel's missed deaths sit at median rank 57**, 45 ranks past the cut. That |
| is the closed appetite axis stated as a measurement rather than as an argument: |
| no value of N reaches rank 57, which is why +395 swaps on the real fleet caught |
| zero. It also localises the warm-up floor - rec12 0.526 there but recall 0.763, |
| i.e. the floor buys catches with depth (19.8 swaps), not with order. |
| |
| **2. `screen`: leave-BATTERY-out vs leave-SCENARIO-out, side by side.** |
| 66503 closed pairs, 461 batteries, 1130 positive rows from **80 batteries**. |
| The effective n is 80, not 1130. |
| |
| | model | battery AUC | battery rec12 | scenario AUC | scenario rec12 | |
| |---|---|---|---|---| |
| | logistic | 0.9669 | 0.541 | 0.9716 | 0.639 | |
| | logbag | 0.9669 | 0.535 | 0.9718 | 0.641 | |
| | rf | 0.9745 | 0.564 | **0.9946** | **0.832** | |
| | gbm | 0.9749 | 0.540 | 0.9921 | 0.786 | |
| |
| Scenario grouping inflates rf's margin over logistic from +0.008 to +0.023 AUC |
| and from +0.023 to +0.194 rec12 - an 8x exaggeration of the tree advantage, |
| because holding out a scenario leaves the same batteries in train and test. |
| Levels are not comparable across columns (a battery fold ranks ~92 devices, a |
| scenario fold ~430); the margin within a column is. Selection reads the battery |
| column. This is the offline-screen counterpart of what all-48 already knew: the |
| harness rejected `rankavg` (+55.3) after cross-fleet CV had confirmed it at |
| -18.4, p=0.0002. |
| |
| **3. `static`: the deployment gate, made measurable.** |
| Fit ONE model on half the buildings' fully-closed pairs, plan every origin of |
| the other half with it - no refits, no accumulating own history, cold-start |
| floor off (82 foreign deaths keeps gate 20 permanently shut in deployment). |
| That is submit.py's held-out branch. 11 s a seed, so variants are screened |
| across 5 building splits, not one. |
| |
| Its levels are NOT comparable to all48 (half the fleet, half the deaths, and |
| the count rule scales with the fleet) and its appetite readings are known-bad - |
| this construction manufactured the sum(p42) undercount that phase 20 refuted |
| and phase 26 believed (public 1661 -> 2189). **Ordering only, variant vs |
| variant.** Both facts are enforced in the printout. |
| |
| ### What the two instruments said about three feature variants |
| |
| | variant | static gate (5 seeds, paired) | mechanism on the gate | all48 | verdict | |
| |---|---|---|---|---| |
| | `vrel` (+vrel, vmin7_rank) | **-24.6** +/- 9.3, W143/L66 | caught 460 -> 463, **sched 1176 -> 1060**, rec12/rec20 flat | **+138.7** | reject | |
| | `vcorr` (+v_corr_last, thresh_dist) | -24.8 +/- 13.5, W115/L98 | caught 460 -> **478**, sched 1176 -> 1191, rec20 +0.007 | **+32.7** | reject | |
| | `zone` (+z_days, z_in, z_rate) | +12.3, prec 0.391 -> 0.349 | sched 4.9 -> 5.6 | +104.9 (phase 34) | reject | |
|
|
| Champion unchanged. Two things are worth keeping from the exercise. |
|
|
| `vcorr`'s +32.7 reproduces the phase-34 measurement of +32.8 to within 0.1 - a |
| third independent confirmation that the all-48 harness is deterministic and |
| sound. |
|
|
| `vrel` is the discipline working. It won the gate by the widest margin of the |
| three (W143/L66, 4 of 5 seeds) and it won it **entirely through the count**: |
| 116 fewer swaps for 3 more deaths, with rec12 and rec20 flat to three decimals. |
| The gate's own docstring says a count reading from this protocol is invalid, |
| and the native path then charged +138.7 for it. Had the gate been read the way |
| phase 26 read its ancestor, this would have shipped. |
|
|
| ### Latent deployment bug fixed: `unobserved_eol_days` |
|
|
| `cost_curves` and `plan_tsroute` hardcoded the evaluator's imputed-EOL offset |
| as `+ 30.0` while the evaluator reads `settings.unobserved_eol_days` |
| per-scenario - and submit.py imports `plan_tsroute` from this module, so the |
| hardcode shipped. All 48 train scenarios use 30.0, which is precisely why no |
| local protocol could ever catch it: the same shape as the hardcoded 42-day park |
| that `repark_for_window` exists to undo. Now a module global that |
| `plan_tsroute` overwrites from the settings it is handed; verified to |
| propagate (30 -> 90) and to leave all-48 byte-identical at 1661.7. |
|
|
| ## Phase 41 - the phase-39 directive discharged: fleet-relative port REJECTED |
|
|
| Phase 39 left a standing rule: *"Do not port anything from [the blind-harness |
| champion] lineage without re-running the ablation on recall at fixed N."* The |
| `vrel` / `vmin7_rank` port in `features.py` was exactly the case it describes, |
| and it was sitting half-landed - computed by `features.py` and shipped in the |
| packaged parquet, absent from `models_cv.FEATURES`. |
|
|
| **Why the earlier cost measurement did not discharge the directive.** Under the |
| topsum rule a feature moves `p42`, `p42` moves `sum(p42)`, and `sum(p42)` moves |
| `N`. So a cost delta is the sum of an ordering change and a depth change and |
| cannot separate them - which is how the blind lineage concluded a 4-feature set |
| beat its own champion. `cv.ablate` holds N fixed and varies only the score. No |
| planner and no evaluator are involved: recall at fixed N and the rank |
| distribution of misses are functions of the score alone. |
|
|
| ### Deaths caught at fixed N, paired per origin, bootstrap 95% CI |
|
|
| | variant | native N=12 | native N=20 | static N=12 | static N=20 | |
| |---|---|---|---|---| |
| | `vrel` alone | **-6 [-12,-1]** | +4 [-1,+9] | +6 [+0,+13] | -1 [-9,+7] | |
| | `vmin7_rank` alone | -1 [-5,+2] | +4 [+0,+9] | -5 [-11,+1] | +0 [-4,+4] | |
| | both | -7 [-15,+0] | +5 [-1,+11] | +4 [-2,+10] | -1 [-9,+7] | |
| | `v_last` alone | **-39 [-57,-22]** | -8 [-20,+5] | +7 [-10,+25] | +5 [-10,+20] | |
|
|
| 454 deaths over 48 native origins; 976 over 240 origin-seeds on the gate. |
|
|
| **Verdict: reject.** The sign flips with N on both instruments and every CI for |
| the port straddles zero. The single CI that excludes zero is `vrel` alone, |
| native, N=12 - and it is **negative**. Decisive corroboration: the ordering is |
| *byte-identical* at 36-41 of 48 native origins and at 228-230 of 240 static |
| ones. These features barely enter the ranking. The +138.7 cost penalty measured |
| in phase 40 was therefore almost entirely a depth effect, exactly as the |
| directive predicted, and the -24.6 the static gate liked was the same depth |
| effect with the opposite sign. |
|
|
| ### The phase-39 claim reproduces - and it is protocol-specific, not over-pruning |
|
|
| `v_last` alone was the specific claim: two blind sessions found raw current |
| voltage beating their champion's own ranker at the operating N. |
|
|
| - **Static gate**: `v_last` alone is statistically **tied** with the full |
| 13-feature champion (+7 [-10,+25] at N=12), with a *better* mean rank |
| (11.9 vs 13.3) and a much better missed-death rank (48 vs 65). |
| - **Native**: `v_last` alone is **-39 [-57,-22]** at N=12, rec@12 0.478 vs |
| 0.564 - **-8.6pp**. Phase 39 measured +14.3pp on production; same sign, same |
| magnitude class, independently reproduced. |
|
|
| So the lineage's finding was real *for its protocol* - one static fit on a |
| foreign fleet - and does not survive the refit path. The cause is not an |
| over-pruned feature set and not the threshold rule phase 39 suspected: |
| **the champion's 12 features beyond `v_last` are worth ~8.6pp recall@12 when |
| the model is refit on the native fleet and worth approximately nothing to a |
| single static fit on a foreign one.** |
| |
| That is a statement about training data, not about features, and it read at |
| the time as the first independent argument for what phase 40 shipped: in-fleet |
| pooling gives the static branch 164 own deaths against the foreign 82. **The |
| public leaderboard refuted that reading within the hour - see phase 42.** |
| Tripling the static fit's death count bought no recall at all, so training |
| volume is not the binding constraint the argument assumed. |
| |
| `features.py` keeps computing both columns (deleting them means regenerating |
| the packaged parquet on submission day for ~1s inside a 360s build budget) but |
| now carries the rejection and these numbers inline, so the port cannot be |
| re-attempted by inspection. |
| |
| ## Phase 42 - PUBLIC: in-fleet pooling is a wash (1665.89 vs 1661.45) |
| |
| Submission d5b03a1 (phase 40, `HELDOUT_POOL=1`). This push also carried phases |
| 34-37, which had never reached the remote - but they touch submit.py by one |
| deleted line, so the held-out path's behaviour change is pooling. |
| |
| | | pooled (d5b03a1) | champion (3c7b883) | delta | |
| |---|---|---|---| |
| | **total** | **1665.89** | **1661.45** | **+4.44** | |
| | late_swap | 1222.92 | 1212.29 | +10.63 (**+51 late-days**) | |
| | early_swap | 133.69 | 137.47 | -3.78 (-363 early-days) | |
| | ops | 309.28 | 311.69 | -2.41 | |
| | swaps | 571 | 580 | -9 | |
|
|
| All four inversions land on integers (571 swaps, 5870 late-days, 12834 |
| early-days, 54 penalised days), so public is still 48 scenarios at default |
| settings - which also confirms the phase-40 `unobserved_eol_days` fix was inert |
| here, as predicted. |
|
|
| **Verdict: a wash, marginally negative.** Pooling made the model slightly less |
| eager - 9 fewer swaps, 363 fewer early-days, 2.4 fewer ops hours - and paid 51 |
| late-days for it. Net +4.4 on a 1661 baseline is 0.27%, inside any reasonable |
| noise band, but the sign is wrong and there is no recall gain hiding in it. |
|
|
| **The finding is not "pooling is neutral", it is where the neutrality comes |
| from.** The static branch went from 82 foreign deaths to 246 (82 foreign + 164 |
| own, in-fleet, same-fleet, correctly embargoed). That is 3x the training |
| signal, on the target fleet itself, and **the ordering did not improve**. Put |
| beside phase 41 - where `v_last` alone ties the full 13-feature champion on a |
| held-out fleet - the two results say the same thing from opposite directions: |
|
|
| > On a fleet the model has not seen, the ranking is essentially determined by |
| > raw voltage, and neither richer features nor 3x more in-fleet training data |
| > moves it. The information needed to rank better is not in what we are |
| > feeding the model. |
|
|
| This closes the training-data axis the way phases 1/6/28 closed the appetite |
| axis. The remaining late cost is the voltage-silent population (recall 0.000 at |
| the operating N, median rank 60 of ~450), and it is invisible to every signal |
| in the current feature space - which is exactly what the phase-38/39 silent- |
| death campaign concluded before it was closed. |
|
|
| Standing champion remains **3c7b883 at 1661.45**. |
|
|
| ## Phase 43 - innovation sweep: three new axes opened and closed, one leak found |
|
|
| Directive: challenge assumptions, look for exploits in the setup, do not stop |
| early. Five things were tested; four are new. |
|
|
| ### (a) The setup leaks the answer (VERIFIED, NOT USED - see the caveat) |
|
|
| Scenarios are spaced **exactly 7 days** apart and the split ships **one** |
| timeseries file. For 42 of 48 scenarios the entire 42-day death window is |
| already present in that file; mean window coverage is **93.9%**. Combined with |
| the reverse-engineered label rule (gated 10-30 C daily median voltage sustained |
| below 2.40 V), reading forward reconstructs the labels directly: |
|
|
| | | recall | precision | |
| |---|---|---| |
| | future-voltage reconstruction (10-line rule, all 48) | **0.841** | **0.874** | |
| | production model at its operating point | 0.641 | 0.453 | |
|
|
| 382 of 454 in-window deaths identified with 55 false positives, day error |
| median -3 d (the confirmation lag). A tuned version reaches the 81/82-exact |
| level `check_labels.py` already demonstrates, i.e. approximately the oracle |
| (186 vs our 1661.7). |
|
|
| **This circumvents the sanctioned accessor.** `iterate_scenarios` explicitly |
| truncates (`cut = flat.loc[:scenario_start]`), so past-only is the intended |
| contract and reading the raw file past the origin breaks it. Recorded because |
| it bounds what the leaderboard can mean, **not** adopted. Our own path is |
| clean: `features_at_origin` filters `day <= origin` (features.py:73) and every |
| physics/beta path does the same. |
|
|
| ### (b) Late cost decomposed: 85% ranking, 15% timing |
|
|
| Of 454 deaths: **163 never swapped** (ranking failure -> emergency queue), |
| **95 swapped AFTER EOL** (timing failure, 150.8/scenario, mean 7.6 d late), |
| 196 swapped before EOL at a median 13 d early. So 84.7% of late cost is |
| inclusion, 15.3% is placement. First time these have been separated. |
|
|
| ### (c) The emergency queue is convex - and unexploitable |
|
|
| From `evaluate_plan`: misses are swapped one per day in **alphabetical** |
| battery order from `end_day`, each with a dedicated round trip and its own |
| `end_previous_day`. A miss at queue position k therefore pays an extra 10k, |
| and including a battery shifts every alphabetically-later miss one day earlier. |
| That externality is real and unpriced - the true marginal cost of a miss is |
| ~330, not the ~220 previously used, which moves break-even from 25% to ~16%. |
|
|
| Tested as a rank bonus. First form (additive) scored **+1007**: an additive |
| bonus is just alphabetical sorting, because the shift only occurs if the |
| battery actually dies. Corrected to the p42-conditional form |
| `p42 * (1 + qw * 10 * E[misses after b] / 54)`: |
|
|
| | qw | 0.25 | 1.0 | 3.0 | |
| |---|---|---|---| |
| | all-48 delta | +60.0 | +160.1 | +275.1 | |
|
|
| Monotone loss, rejected. The mechanism is a ~9% pricing correction but it |
| reorders by **alphabet**, a variable with zero predictive content - measurement |
| law #2 again: ordering damage beats pricing gain at the boundary. |
|
|
| ### (d) Lead-time bias on day placement - real trade, break-even |
|
|
| (b) says the day model is systematically optimistic, and late/early trades at |
| 20:1. Two implementations: |
|
|
| | k days early | 0 | 2 | 4 | 7 | 10 | |
| |---|---|---|---|---|---| |
| | shift the OUTPUT | 1661.7 | +36.2 | +33.6 | +53.0 | +87.9 | |
| | shift the BELIEF (quantiles, re-solve) | 1661.7 | **-12.0** | -0.3 | -9.7 | -11.3 | |
|
|
| Shifting the output works on late (987 -> 878) but compresses all work into |
| earlier weeks and costs **+157 ops** in limit penalties. Shifting the predicted |
| death-day quantiles instead lets `optimize_days` re-solve with routing and the |
| weekly limit intact: ops stays at 206-227 and late falls to 922-962. Net best |
| is **-12.0 at k=2, on 23W/25L** - a coin flip, non-monotone across k. Rejected, |
| but it establishes that **day placement is already near-optimal**: the 95 |
| late-swapped catches cannot be recovered by earlier scheduling because the |
| early penalty on the 196 correctly-timed ones plus the ops cost consumes the |
| whole gain. |
|
|
| ### Standing after this sweep |
|
|
| Every axis now has a measurement against it: appetite (public), features |
| (phase 41 fixed-N), training volume (phase 42 public), routing-aware inclusion |
| (phases 7/22, "even zero-travel extras lose"), expected-cost ranking (phase |
| 27b), queue pricing (c), day placement (d), silent deaths (phases 6-13, 38-39). |
|
|
| **The honest residual**: the leader is at 1477.48, so ~184 points are |
| demonstrably extractable by someone. We cannot account for them from any |
| mechanism measurable here, and the only path this sweep demonstrated is the |
| one in (a), which we are not taking. |
|
|
| ## Phase 44 - why our late_swap is the field's highest (full leaderboard inversion) |
| |
| The public board exposes all 14 teams' components. Inverting each team's |
| `late_swap` through the emergency-queue cost model (each miss = one dedicated |
| day at `(24 + k) * 10`, solved for k self-consistently) gives, per scenario: |
|
|
| | team | swaps | misses | caught | precision | |
| |---|---|---|---|---| |
| | J2W | 14.9 | 2.45 | 7.65 | 0.51 | |
| | YassY | 17.4 | 2.37 | 7.73 | 0.44 | |
| | CarlAlbert | 15.5 | 2.80 | 7.30 | 0.47 | |
| | sackriti | 14.5 | 3.21 | 6.89 | 0.48 | |
| | **us** | **12.1** | **4.69** | **5.41** | **0.45** | |
| | kalsach | 28.2 | 2.14 | 7.96 | 0.28 | |
|
|
| Deaths solve to ~10.1/scenario (local all-48 is 9.5 - good agreement). |
|
|
| **Our per-swap precision, 0.45, is mid-pack in a field of 0.28-0.51.** The |
| ranking is not broadly worse. We simply buy the fewest swaps in the |
| competition - 580, against 695 for the next-lowest and 1355 for the highest - |
| so we catch 5.41 where everyone else catches 7.3-8.0, and each of the ~2 extra |
| misses costs ~285. |
|
|
| ### Appetite is genuinely closed - now with numbers, not an argument |
|
|
| From phase 28's own measurement: +8.2 swaps/scenario moved `early_swap` +444, |
| so a **marginal false swap costs 54**. Our ops (309 at 4.69 misses) against |
| J2W's (252 at 2.45) prices the emergency day at **25 per miss**, so a miss |
| costs 260 + 25 = **285**. Break-even is therefore **15.9%**, and our local |
| precision in ranks 13-20 is `(0.705-0.564)*9.5/8` = **16.7%**. The marginal |
| pick sits exactly on break-even. Raising alpha cannot pay. |
|
|
| The cliff-contamination escape is also closed: **cliff rows die at 39.4%, 17x |
| the base rate**, at 12.0 rows/scenario. They are good picks, so phase 28's |
| "zero extra catches" was not an artifact of the boost's reordering. |
|
|
| ### The real defect, and why no local instrument can see it |
|
|
| Our ranks 13-20 are 16.7% precise locally and **~0% on public** (phase 28: 395 |
| extra swaps bought `late_swap +13`). J2W's 13th-15th picks run at **80%**. The |
| head transfers to a foreign fleet; the tail is noise. |
|
|
| Where the misses live (native, 454 deaths): |
|
|
| | rank band | n | frac stale > 7d | median n_30 | |
| |---|---|---|---| |
| | 1-12 (head) | 256 | 0.04 | 30 | |
| | 13-30 (tail) | 104 | 0.03 | 30 | |
| | 31+ | 94 | **0.30** | 30 | |
| | fleet baseline | - | 0.16 | - | |
| |
| Rank 31+ is the closed silent-death channel (staleness at 2x the base rate). |
| The tail is **fully-observed devices** - 3% stale, 30 readings in 30 days - |
| that we simply order below rank 12 and J2W orders above it. |
| |
| Two things were tested against it and both failed: |
| |
| - **Fleet calibration shift.** The half-fleet gate splits ONE fleet, so it has |
| no calibration offset by construction. Injected one (+/-0.03, +/-0.06 V on |
| the target fleet only, with tau moved along the curve at 572 d/V). Recall |
| moved by at most 0.005 at every N. The reason is elementary and closes the |
| hypothesis: **a uniform offset shifts every device's linear predictor |
| equally and cannot reorder a linear model at all.** It also explains why |
| `vrel` does nothing - it protects against the one perturbation that was |
| already harmless. |
| - **Regularisation.** Effective n is ~80 death-batteries, not 66503 rows, so |
| `logbag`'s `C=1.0` is tuned for a sample ~800x too large. Swept C over 100x: |
| |
| | | C=1.0 | C=0.3 | C=0.1 | C=0.03 | C=0.01 | |
| |---|---|---|---|---|---| |
| | gate, tail gain 13-30 | 0.154 | 0.155 | 0.157 | 0.158 | 0.153 | |
| | native, tail gain 13-30 | 0.229 | 0.231 | **0.247** | 0.244 | 0.240 | |
| | native rec@12 | **0.564** | 0.562 | 0.551 | 0.557 | 0.555 | |
| |
| Flat on the gate. Natively it trades head for tail (C=0.3: +5 at N=16 and +6 |
| at N=20, both 5W/0L, but -1 at N=12). At our operating N of ~12 it is |
| negative. Rejected. |
| |
| **The blocker is instrumental.** The gate reports a perfectly healthy tail - |
| rec@12 0.766, tail gain 0.154 - while public reports ~0% marginal precision |
| there. No local protocol reproduces the defect, so no local protocol can |
| validate a fix for it. That is the honest reason this stops here rather than |
| a claim that nothing is left: J2W demonstrably finds 2.2 deaths per scenario, |
| in well-observed data, that our ordering puts below the cut. |
| |
| Standing champion remains **3c7b883 at 1661.45**. |
| |
| ## Phase 45 - the objective axis: pairwise ranking loss REJECTED (last open axis) |
| |
| Rationale: log-loss on a 2.3% positive rate spends its capacity on the 97% |
| that obviously survive; the decision is a top-3% ordering, and the entire |
| 638-point gap to the leader sits at ranks 12-20. A pairwise loss (RankNet- |
| style, linear scorer, within-origin death-vs-survivor difference vectors, no |
| intercept) optimises exactly that boundary. Two samplers: uniform survivors, |
| and hard negatives (survivors the incumbent ranks highest). Scored on recall |
| at fixed N, both instruments; score drops into rank_score so p42/count/timing |
| would stay byte-identical if shipped. |
|
|
| | variant | native rec@12 | native tail 13-30 | gate rec@12 | gate tail | |
| |---|---|---|---|---| |
| | logbag (base) | **0.564** | 0.229 | **0.766** | 0.154 | |
| | pair-uniform | 0.509 | 0.264 | 0.743 | 0.154 | |
| | pair-hard | 0.518 | **0.271** | 0.746 | 0.149 | |
|
|
| Paired at N=12: native -21 (4W/15L), gate -20 (19W/36L). Loses everywhere at |
| the operating point. |
|
|
| The apparent tail win (0.271 vs 0.229 native) motivated a head/tail hybrid: |
| logbag's top-K byte-identical, pairwise re-orders only below it. Result: the |
| hybrid loses at every K on both instruments (gate K12: -18 at N=16, 8W/24L; |
| native K12: -12 at N=16, 4W/12L). That exposes the tail win as an artifact: |
| **through rank 30 the two objectives catch the same deaths (native rec@30 |
| 0.793 vs 0.789, gate 0.920 vs 0.894)** - pairwise does not discover new deaths |
| in the tail, it demotes base's head deaths INTO the tail. When the hybrid |
| keeps base's head and asks pairwise to improve the tail with its own picks, |
| there is nothing there. |
|
|
| **Conclusion: the pairwise scorer extracts no information the log-loss fit |
| does not already have.** Same features, same data, same deaths found, lower |
| positions. Combined with phases 41 (features), 42 (training volume), 44 |
| (regularisation, calibration shift), the information ceiling of this feature |
| space is the champion's current ordering. Every axis is now measured: |
|
|
| | axis | closed by | |
| |---|---| |
| | appetite / count | public (phase 28), break-even arithmetic (44) | |
| | features | fixed-N ablation (41) | |
| | training volume | public pooling (42) | |
| | regularisation | C sweep, flat/negative at N=12 (44) | |
| | calibration robustness | offset cannot reorder a linear model (44) | |
| | objective | this phase | |
| | timing / day placement | lead-time bias, coin flip (43) | |
| | queue pricing | alphabetical ordering damage (43) | |
| | routing-aware inclusion | phases 7/22 | |
| | silent deaths | phases 6-13, 38-39 | |
|
|
| What would beat 1661 from here is information we do not have locally: either |
| a signal J2W computes that is outside this feature space entirely, or the |
| forward-reading of the split file that phase 43(a) verified and declined. |
|
|
| Standing champion: **3c7b883 at 1661.45**. Deadline 2026-08-23. |
|
|
| ## Phase 47 - the blind champion itself, as a whole: REJECTED (lineage fully adjudicated) |
|
|
| (Numbered out of order: written before the phase-46 commit landed from the |
| parallel session; the work predates it, the number does not.) |
|
|
| Question (user, 2026-08-22): the blind-repo multi-agent loop's shared champion |
| (`9a969a6a7dfe.py`, blind full-40 1566.7) - reusable here? The headline number |
| is not on our scale (its own 40-origin harness; ~55 of its margin was extra cut |
| depth), and most pieces were already adjudicated: CV discipline ported (40), |
| `vrel`/`vmin7_rank` as ADDITIONS rejected (41), in-fleet pooling a public wash |
| (42), trip-merge already native as building batching + group-day search |
| (phases 1/14). One piece had never been tested: the champion's exact 4-feature |
| set **as a replacement ranker** - `FEATURES=[last_voltage, vmin7, n_obs, vrel]`, |
| mapped here to `[v_last, v_min_7, n_30, vrel]` (`blind4` in cv.VARIANTS). |
|
|
| ### Fixed-N ablation: a real but small head-ordering gain |
|
|
| Paired vs base, deaths caught, bootstrap 95% CI. `v_last_only` cross-check |
| reproduces phase 41 (-39 [-56,-21] native N=12 vs published -39 [-57,-22]). |
|
|
| | variant | native N=8 | native N=12 | static N=8 | static N=12 | |
| |---|---|---|---|---| |
| | `blind4` | +11 [-1,+23] | +9 [-8,+26] | **+27 [+7,+47]** | +14 [-2,+31] | |
| | `blind3` (no vrel) | +14 | +10 | +22 | +12 | |
|
|
| One CI of eight excludes zero (static N=8), signs consistent at N<=12 on both |
| instruments, and rec@30 / miss-rank get WORSE (57 -> 65 native): the small set |
| trades deep tail for head. `blind3` matching `blind4` everywhere settles the |
| phase-41 question from the other side: **vrel is not load-bearing; the gain is |
| the small set itself** - 4 features against effective n ~80 death-batteries, |
| where 13 features overfit slightly (phase 44's C-sweep said the same thing |
| through the regularisation knob). |
|
|
| ### Full harness: the ordering gain does not survive the planner |
|
|
| | all48 | total | sched | caught/454 | early | late | |
| |---|---|---|---|---|---| |
| | base | 1661.7 | 13.4 | 291 | 473.1 | 987.5 | |
| | blind4, alpha 0.8 | 1765.4 | 14.6 | 298 | 584.0 | 948.3 | |
| | blind4, count-matched alpha 0.7342 | 1740.7 | 13.7 | 290 | 530.1 | 972.3 | |
|
|
| Raw: +103.8/origin - the 4-feature calibration inflates `sum(p42)`, the exact |
| depth confound phase 41 diagnosed, in mirror image. Count-matched: **+79.1 |
| [-65,+231], 22W/26L, catches identical (290 vs 291)** - the +9 fixed-N catches |
| vanish under the real inclusion rule (topsum + break-even top-up + floor is not |
| top-N), and the whole residual is placement: +57 early-days/origin, because |
| day placement integrates the p-curve and a 4-feature curve is cruder than a |
| 13-feature one. Ordering is only half the score's job here; timing is the |
| other half, and the small set is worse at it. |
|
|
| **Verdict: reject; nothing in the blind lineage is left unported or untested.** |
| The blind champion's margin decomposes fully into (a) its own harness's depth |
| and threshold tuning - non-transferable, appetite closed; (b) mechanisms we |
| already have natively; (c) a head-ordering effect real at fixed N but worth |
| zero catches and negative cost through this planner. Runnable via `blind4` / |
| `blind3` in `cv.VARIANTS`. |
|
|
| Standing champion: **3c7b883 at 1661.45**. Deadline 2026-08-23. |
|
|
| ## Phase 46 - the overfitting question, audited; and the certainty-first adoption |
|
|
| User hypothesis: the leaders are overfitting the public board. Audited with a |
| 4-way measurement (certain-pool size, our capture of it, the fix on both |
| instruments, luck bounds). Verdict: **partially right, and the audit found a |
| real local gain on the way.** |
|
|
| ### How much of the board is luck |
|
|
| The board keeps each account's best public score, so it best-of-N selects |
| every team's luckiest config. From our own per-origin variance (sd 838 over 48 |
| origins -> sd of the 48-scenario mean = 121), the expected luck of keeping the |
| best of N equal-skill submissions is 217/260/296/338/367 points for |
| N=5/10/20/50/100. |
|
|
| - **J2W's 210-point lead over rank 2 needs only N~4.5.** The ordering WITHIN |
| the leading pack is plausibly board luck. |
| - **The pack's ~500-point lead over us would need N~5300.** Not luck. |
|
|
| ### The pack's edge is real - and it is not legitimately reachable by us |
|
|
| The certain pool (devices whose gated daily median has sustained < 2.40 V for |
| 3 consecutive gated readings BEFORE the origin, i.e. the label rule already |
| fired but the EOL has not landed - the wobble/lag population): |
|
|
| | rule at origin | flagged/scen | die in-window | die by +90d | never die | |
| |---|---|---|---|---| |
| | 2.40 sustained-3 confirmed | 1.06 | 94.4% | 100% | **0** | |
| | 2.45 zone (days_below_245>=0) | 21.8 | 24.6% | - | 62% | |
|
|
| So the free-certainty pool is ~1/scenario, of which our champion already ranks |
| 42/51 in the top 12. The leaders' ~2.2 extra catches/scenario cannot be built |
| from any past-only certainty rule we can construct: the implied J2W marginal |
| precision is 0.69-0.88 (robust across all 9 inversion assumptions), on picks |
| that for us measure ~0% transfer precision. Either they hold a tail-ranking |
| signal outside this feature space entirely, or they are reading outcomes (the |
| phase-43a forward-read, whose ceiling - 84% recall, 1.6 misses/scen - they sit |
| comfortably below at 2.45 misses; or battery-toggling against public, ~1 |
| battery learned per submission pair). Given phases 41-45 measured every |
| legitimate axis at ~zero, the balance of evidence is that the top of the board |
| is exploiting the setup, not merely lucky. Unprovable from here; recorded as |
| an assessment, not a fact. |
|
|
| ### ADOPTED: certainty-first ordering (cert-first, k=3) |
|
|
| The audit's by-product. Force confirmed-crossing devices to the top of the |
| ranking (rank_score = 2 + p42; count and timing untouched): |
| |
| | | all-48 | panel | W/L/T | mechanism | |
| |---|---|---|---|---| |
| | first-reading rule (LEAKS 2 future readings) | -88.2 | -124.5 | 6/0/42 | struck | |
| | **confirmed rule (past-only)** | **-37.9** | **-53.6** | **3/0/45** | adopted | |
| | k=1 / k=2 confirmation | +290.7 / +199.5 | - | 9/30 / 9/31 | recoveries flood the head | |
| |
| The confirmation depth is load-bearing and equals the label rule's own sustain |
| parameter - k=3 is the unique safe point. Fixed-N instruments agree: native +9 |
| catches at N=12 (6W/0L), deployment gate +6 (6W/0L, ties elsewhere) - loss-free |
| on every instrument at every N tested. The wins are label-lag devices our |
| model under-ranks BECAUSE they recovered (v_last back up) - exactly the wobble |
| population phase 39 characterised. |
|
|
| Deployment note: at runtime `iterate_scenarios` hands truncated data, so the |
| rule is past-only by construction there; the local strict measurement is the |
| honest predictor. Wired into submit.py (`BATTERYSWAP_CERT_FIRST`, default on, |
| both branches; stacks on top of cliff's rank_score if that is ever enabled; |
| p42_sum untouched so the count rule is byte-identical). |
|
|
| Local champion is now **cert-first at 1623.7** (from 1661.7). Public look not |
| yet spent; submission is the user's call. Expected public value if the wobble |
| rate transfers: ~-35 +/- (fires 0.46 devices/scenario locally). |
|
|
| ## Phase 49 - four new ordering-signal families opened and closed in one sweep |
|
|
| Directive (user, 2026-08-22): test new ideas against the champion, be |
| innovative, try brand-new approaches. Everything below is a fixed-N ordering |
| screen (`cv.ablate` discipline, both instruments where applicable, paired per |
| origin, bootstrap 95% CI), so no depth confound; costs on the 1661.7 scale |
| (cv.run_all48 does not route through submit.py's cert-first). Screen code: |
| `exp_x29_new_signal_sweep.py` [long|tau|tavg]. Phase 48 is reserved for the |
| cert-first public adjudication (parallel session). |
| |
| ### (A) Longitudinal cross-origin features - the first untouched signal family |
| |
| The feature table is a 168-origin snapshot grid (~3-4d apart) per device; every |
| existing feature reads one 30/90d window at ONE origin, and no phase had ever |
| read ACROSS origins. Three features, all strictly backward-looking |
| (merge_asof, snapshots < T): `mom28`/`mom56` = within-fleet rank change of |
| v_last vs 28/56d ago; `wolf` = count of snapshots >= 42d ago in the bottom |
| decile (device provably survived - it is alive at T). |
| |
| The raw signal is REAL: bottom-decile chronics (wolf > 2) die at 20.7% vs |
| 15.5% for newcomers, and dying devices' fleet rank slides (median mom28 -0.025 |
| vs -0.007 for survivors). The model cannot use it: |
| |
| | variant | native N=12 | gate N=12 | verdict | |
| |---|---|---|---| |
| | mom | -5 [-13,+3] | -7 [-14,+0] | reject (gate N=16/20 significantly negative) | |
| | wolf | **-24 [-42,-7]** | **-14 [-27,-1]** | reject | |
| | momwolf | -27 [-47,-7] | -15 [-30,-1] | reject | |
| |
| Mechanism: the 30d-window slope/drop/accel features already price the |
| trajectory in; the cross-origin rank version duplicates them with extra noise, |
| and `wolf` duplicates the level (chronics ARE low). Effective n ~80 punishes |
| every redundant column. |
| |
| ### (B) Interaction terms - the linear champion offered its first products |
| |
| `z(v_last)*z(slope_30)`, `z(v_last)*z(vstd_30)`, `z(v_min_7)*z(accel)` |
| (within-origin z-scores, leak-free). Flat everywhere: native +2/-2/+5/+2 |
| across N, gate +0/-1/-5/-1, every CI straddling zero. There is no exploitable |
| interaction structure - the phase-45 ceiling holds against nonlinearity in |
| feature space, not just against nonlinear learners. |
| |
| ### (C) tau descent-band re-rank - phase 43's open candidate, closed |
| |
| The knn screen left one open item: logbag underweights `tau_projected` in the |
| descent band where its misses live (24/36/44 of 132 panel misses in tau's |
| top-3/5/10); EV-negative as extra swaps, never tested as a pure REORDER at |
| fixed N. Tested three transforms (band rank-blend, global rank-blend, |
| promotion-only): native **-45 of 454 at N=12 (7W/29L)**, -53 at N=8. The gate |
| is mildly positive (+3..+11) - the same protocol-specificity signature as |
| `v_last` in phase 41: a single static foreign fit underuses the timer, the |
| weekly refit already extracts it. The gate's sign must never override a |
| native loss. Candidate closed for the native path. |
|
|
| ### (D) Temporal score ensembling - the variance question, settled directly |
|
|
| Rank-average the current origin's fit with the previous 1-2 origins' fits |
| applied to the same rows (adjacent origins are near-identical draws of the |
| ~80-death training set). Dead flat: tavg2 -3 at N=12 (0W/3L), tavg3 -4 |
| (1W/5L) - refit-to-refit orderings barely differ. With phase 44's flat C |
| sweep (regularisation not binding) and logbag never separating from plain |
| logistic (bagging variance not binding), this is the third independent leg: |
|
|
| > **The marginal-rank defect is missing information, not estimation noise. |
| > Any future ordering candidate must bring a NEW signal, not a better |
| > estimate of the current ones.** |
|
|
| (It also moots the deployment-gate objection - there is no fit noise to |
| smooth, in any branch.) |
|
|
| ### Standing |
|
|
| Four families opened, four closed, all with mechanisms. The information |
| ceiling of phase 45 now covers: re-weightings (45), replacement sets (47), |
| cross-origin dynamics, interactions, timer blends, and temporal ensembles |
| (49). What remains is what phase 44 said: a signal outside this feature |
| space, or the declined forward-read. Local champion unchanged (cert-first |
| 1623.7 / base 1661.7); public champion **3c7b883 at 1661.45**. Deadline |
| 2026-08-23. |
|
|
| ### Phase 46 addendum: the rule is one point wide in BOTH dimensions |
|
|
| Threshold micro-sweep at k=3 (cost harness): 2.40 -> **-37.9** (3W/0L); |
| 2.41 -> +93.6 (8W/32L); 2.42 -> +383.9; 2.43 -> +314.1. With the earlier |
| k-sweep (k=1 +291, k=2 +200), cert-first works ONLY at the label process's |
| exact parameters (2.40 V, 3 consecutive gated readings). The mechanism is |
| label-rule replay on past data, not prediction; any deviation reverts to |
| guessing at sub-break-even precision. Direction closed. |
|
|
| ## Phase 48 - PUBLIC REFUTES cert-first (1704.64); the third and sharpest CV failure |
|
|
| Submission 9438aac (cert-first isolated, pooling default off): **1704.64 vs |
| champion 1661.45, +43.19.** Inversion (all integers - settings unchanged): |
|
|
| | | swaps | late-days | early-days | total | |
| |---|---|---|---|---| |
| | champion 3c7b883 | 580 | 5819 | 13197 | 1661.45 | |
| | cert-first 9438aac | 598 | 5894 | 15203 | 1704.64 | |
| | delta | **+18** | +75 | **+2006** | +43.19 | |
|
|
| **+18 swaps at ~111 early-days each.** The diagnosis writes itself: on train, |
| every device that sustains <2.40 V dies within 90 days (54/54, phase 46), so |
| the boost was loss-free locally. The public fleet contains devices that |
| sustain-cross and NEVER receive an EOL label - the ghost population phase 15 |
| found 26 of in train (at 2.45-2.48; train has none at 2.40, public evidently |
| does). A ghost is force-swapped in every scenario it survives, and with no |
| EOL on file the evaluator prices each swap against end_time+30: ~111 |
| early-days apiece. The count rule could not protect us - a boosted device |
| displaces a marginal pick, and when the displaced pick sits in the break-even |
| top-up set it re-enters, so the union grows. |
| |
| The premise that failed is not the label rule (81/82 exact) but its |
| CONVERSE on an unseen fleet: "crossed implies labeled-dead-soon" is a |
| train-fleet fact, not a label-process fact. |
| |
| ### What this makes three of |
| |
| Locally-validated interventions refuted by public: ALPHA_STATIC=1.5 (holdout |
| -240 -> public +527), cliff boost (adjudicated -55 -> refuted stacked), and |
| now cert-first (**-37.9 at 3W/0L/45T -> public +43.2**). Cert-first is the |
| sharpest of the three: mechanistically derived, past-only, loss-free on every |
| local instrument, refuted anyway - by a fleet-composition fact no local |
| protocol can exhibit because train contains zero counterexamples. When the |
| local evidence for a class of intervention is this good and it still fails, |
| the class is uninsurable, not under-validated. |
|
|
| Action: CERT_FIRST default off (code retained behind the env var); champion |
| remains 3c7b883 at 1661.45 on the board (best-per-account row unaffected). |
| A recency-guarded variant (trust the crossing only while fresh) is measured |
| locally in this phase's sweep, but shipping it would repeat the same bet with |
| a smaller stake and no way to price it - the ghost rate at 2.40 on the scored |
| fleets is unobservable from here. |
| |
| ### Phase 48 addendum: the guard is measured, and the late component kills it |
| |
| Recency-guarded sweep (flag only crossings confirmed within the last X days): |
| g14 -18.7 (2W/0L), g28/g42/gInf all **-37.9 identical** - locally every catch |
| happens within 28d of confirmation, so a 28d guard is free AND caps a public |
| ghost at ~4 scenarios. But the public inversion settles whether to ship it: |
| **late-days went UP (+75)**. If cert-first had caught even one real wobbler |
| death per few scenarios on public, late would have dropped by ~240 days per |
| catch. It caught ZERO. The local -37.9 was wobbler catches that the public |
| fleet does not offer below the cut; a guard cannot create catches that are |
| not there. Cert-first family closed entirely - the label-lag population |
| either does not exist on the scored fleets or is already inside the head. |
| |
| ## Phase 52 - building-grouped CV adopted; the contamination gradient quantified |
| |
| Trigger: a sweep of every public competitor artifact (HF repos, hidden HF |
| branches, GitHub repos, and deleted-HF-repo cards recovered via the |
| 686f6c61/radar-huggingface-dataset archive) established that the scored |
| public/private fleets hold FOREIGN BUILDINGS: the Attention-Heads card states |
| it outright, and the FAQ transcription in ziadnasif77-debug's repo puts the |
| full pool at ~200 buildings vs train's 24. Both serious hidden teams |
| (YassY-The-AlchemYst's RobustHazardPlanner lineage, ziad) validate and |
| calibrate building-grouped. Organizer-shaped conclusion, user directive: |
| the trusted offline fold must be leave-BUILDING-out. |
| |
| Change (cv.py): `screen` gains a third grouping column, whole buildings |
| assigned to 5 folds greedily balanced by unique observed deaths (deaths are |
| lumpy - the top building holds 15/82, nine hold exactly one; random building |
| assignment starves folds). `static`/`ablate` already split by building |
| (fleet_halves), so this closes the last battery-grouped selection surface. |
| Selection guidance in the module now reads: building > battery > scenario. |
|
|
| Result, 3 fold-assignment seeds - rec12 margin over logistic (AUC margins |
| agree in sign; levels are NOT comparable across columns, margins are the |
| decision quantity): |
|
|
| | model | scenario | battery (s0/s1/s2) | building (s0/s1/s2) | |
| |---|---|---|---| |
| | logbag | +.003 | -.006 / -.004 / +.001 | -.002 / +.000 / +.000 | |
| | rf | **+.194** | +.023 / +.031 / -.008 | +.005 / -.005 / -.014 | |
| | gbm | **+.148** | -.000 / -.016 / -.026 | -.022 / -.039 / -.030 | |
|
|
| The contamination gradient is monotone: rf's +.194 phantom margin on |
| scenario folds collapses to ~+.02 on battery folds and to zero-or-negative |
| on building folds; gbm goes outright negative. The residual rf margin in the |
| battery column (sign-unstable across seeds) is sibling leakage - a held-out |
| device's building/batch mates stay in train. logbag == logistic (margin ~0) |
| in every column and every seed: the champion's model choice never rested on |
| fold luck, and the deployment-shaped fold CONFIRMS it rather than reversing |
| anything. No selection changes; the protocol does. |
|
|
| Standing rule upgrade (supersedes the blind-loop "group by battery" rule): |
| **group by BUILDING when the deployment is cross-building.** Battery |
| grouping is necessary but not sufficient; it still leaks the environment/ |
| install-batch channel that phase 10/13 closed on the feature side. |
|
|
| ### Phase 52 addendum - the building split extended to the COST harness (`all48oof`) |
|
|
| Directive (user, 2026-08-22): "building split not only for the AUC but for |
| the scenario split as well." New protocol `cv all48oof`: the walk-forward |
| cost harness with BUILDING-CLEAN predictions - each device's p42 at every |
| origin comes from a champion-config logbag fitted on pairs from every |
| building except its own (5 greedy death-balanced folds, OOF-stitched), then |
| the FULL fleet is planned as usual (planning must stay full-fleet: the count |
| rule scales with fleet size - the phase-26 static-gate lesson). Paired |
| `self` arm through identical machinery, cold-start floor OFF in both, so |
| the per-origin delta isolates the self-building fit effect. Read the DELTA, |
| not the level. Validity check: the self arm reproduces the canonical panel |
| 1679.2 and near-end 604.8 exactly (floor only fires in warm-up). |
|
|
| | block | self | oof | delta/origin | oof-better W / self-better L | |
| |---|---|---|---|---| |
| | warmup s_0-7 | 4256.1 | 4180.0 | -76.1 | 6W / 2L | |
| | panel s_8-41 | **1679.2** | **1813.6** | **+134.4** | 8W / 26L | |
| | nearend s_42-47 | 604.8 | 757.3 | +152.5 | 1W / 5L | |
| | ALL-48 | 1974.4 | 2075.9 | +101.6 | 15W / 33L | |
| |
| Ranking moves with the cost: panel rec12 0.572 -> 0.538, recall 0.571 -> |
| 0.542, ALL-48 rec12 0.597 -> 0.560. Warm-up's negative delta is noise |
| (floor off, 3-20 train deaths). Reading: **self-building fitting flatters |
| the walk-forward by ~+134/origin on the panel**; the transfer-shaped cost |
| estimate for the champion on foreign buildings is the oof column. This is |
| the cost-denominated size of the contamination the phase-52 screen showed |
| as AUC margins - it prices the OPTIMISM of the harness, not a defect of the |
| champion (both arms use the same model family; there is no building-free |
| challenger that closes the gap, per the phase-52 screen). No selection |
| change. Results /tmp/cv_all48oof.csv, seed 0. |
|
|
| ## Phase 50 - the forecast axis: fleet-learned conditional dynamics (construction and screens) |
|
|
| Directive (user, 2026-08-22): innovate on the FORECAST part; reinforced with |
| "benefit from more training data" and "a forecast as a model ensemble". The |
| phase-49 wall requires a NEW signal; this candidate brings one by |
| construction - auxiliary supervision on ~70k UNLABELED device-week |
| transitions from the 168-origin grid, so the effective-n~80 wall does not |
| apply. Machinery: `exp_x30_dynamics_forecast.py`; session-55 built an |
| independent implementation in parallel (HGBR rate model, refit cadence 8, |
| MC 120x6, generation-time ghost guard; artifacts /tmp/dyn/). |
|
|
| **Construction.** One-step (7d) voltage forecaster - HGBR conditional drift |
| on state + HGBR sigma + standardized empirical residual pool (plunge tails |
| survive sampling) - rolled 6 weekly steps, 200 paths; p_dyn = P(cross 2.40 V |
| within 42d). Dynamics AND residuals refit per origin on transitions ending |
| <= T (corpus 22k -> 64k across the walk). Differs from rejected WienerFHT |
| (phase 9): fleet-pooled nonlinear conditional drift, not per-device linear |
| drift; fixed-N delivery, so calibration never enters. GHOST GUARD from the |
| phase-48 lesson: below-2.40 devices keep base rank/probability - NOT a local |
| no-op (342 below-threshold rows at scenario origins, 236 never-labeled). |
| |
| **Screens - the strongest fixed-N profile of the campaign** (paired, |
| bootstrap 95% CI, ghost-guarded): native +11 at N=16 (13W/6L), +13 [+5,+22] |
| at N=20 (12W/1L); gate +24 [+11,+38] at N=8, +23 [+11,+36] at N=12 (28W/7L). |
| Weight sweep w=0.1..0.5 positive in 39/40 cells - plateau, not knife edge. |
| Tail diagnostic: 30 of 104 champion misses at ranks 13-30 promoted into |
| p_dyn's top-12. Fuel: descent-band AUC 0.8005 vs v_last 0.8024 (891 |
| positives), Spearman(p_dyn, v_last) -0.683. Runtime: 4.0s per split at 2x |
| fleet vs the 1800s budget. Checks passed: no leakage (per-origin refits), |
| weight robustness, runtime. What follows is why none of it survived. |
| |
| ## Phase 51 - the dynamics family: CLOSED (eight deliveries, two implementations, and the building-split autopsy) |
| |
| **Cost harness, all arms, 1661.7 scale** (count pinned where stated): |
| |
| | delivery | total | caught | mechanism of failure | |
| |---|---|---|---| |
| | rank_score blend37 | 1744.5 (+83) | - | inclusion reordered; placement/pricing believe base p42 | |
| | global p42 quantile-blend, pinned | 1717.1 (+55) | 287 | every device's day placement perturbed | |
| | band-limited (ranks 9-30), pinned | 1690.8 (+29) | 290 | catch parity; +13 late of timing noise | |
| | B: p_dyn as 14th feature, retrain (55) | 1710.1 (+48) | 295 | sched 13.4->13.9 - appetite in disguise | |
| | B pinned, 55's table | 1707.4 (+46) | 288 | count pinned -> catches DROP below base | |
| | B on c6 table | 1702.5 (+41) | 288 | same | |
| | B on c6 table, pinned | 1706.8 (+45) | 286 | same | |
| | rank-mean ensemble | moot | - | both parents dead on both tables | |
| |
| Post-hoc blends fail as a class (beliefs inconsistent with count/timing |
| machinery); retrains fail as a class (the only "conversion" was calibration |
| inflation that pinning exposed as appetite). |
| |
| **Cross-implementation check.** Two independent implementations agree at |
| Spearman only 0.369 on above-threshold nonzero rows; on max-disagreement |
| rows the ground-truth referee splits (0.680 vs 0.628 - both mediocre). The |
| MC tail probability is implementation-fragile; only the coarse ordering is |
| robust. |
| |
| **The building-split autopsy (user question: "are you using the new |
| building-split CV here?").** The gate's logbag was always building-split, |
| but the DYNAMICS corpus pooled both fleets' transitions - deployment-legal |
| (label-free; the scored split ships its own telemetry), yet untested under |
| phase 52's trusted fold. Refit on FOREIGN-building transitions only: |
| |
| | gate blend37 | N=8 | N=12 | N=16 | N=20 | |
| |---|---|---|---|---| |
| | pooled dynamics | +24 [+11,+38] | +23 [+11,+36] | +7 | +5 | |
| | foreign-only dynamics | -14 [-30,+2] | +1 [-13,+15] | -12 [-25,+1] | -10 [-22,+1] | |
| |
| The entire screen gain was own-fleet telemetry memorization: each device |
| contributes ~160 near-duplicate transitions, HGBR memorizes device-specific |
| drift signatures, and p_dyn partially encodes "this device has been |
| declining" - which the level features already price. This is the phase-40 |
| CV-contamination lesson recurring one level down, in a LABEL-FREE model: |
| grouping discipline applies to self-supervised corpora too, not just |
| label fits. It also explains the implementation fragility (different |
| configurations memorize different signatures - fine values diverge, real |
| descent agrees) and why the harness never converted what the screens |
| promised. |
|
|
| **Verdict: family closed for this competition. No submission.** The signal |
| looked real (band AUC 0.80, -0.68 to v_last) but the transferable component |
| is ~zero under the building fold; the apparent independence was memorized |
| device identity. Corollaries, binding on the next candidate: (a) the |
| phase-52 building fold applies to auxiliary/self-supervised training |
| corpora, not only label models; (b) a screen gain that the cost harness |
| cannot convert is now a two-time pattern (47, 51) - suspect the screen's |
| population before suspecting the delivery. |
| |
| Standing: local champion base 1661.7 (cert-first off), public champion |
| **3c7b883 at 1661.45**. Deadline 2026-08-23. |
| |
| ## Phase 53 - the user's re-adjudication directive: promising ideas vs the building-split CV |
| |
| Directive: retry the most promising rejected ideas against the new |
| building-split instruments. Inventory first: everything from phases 41-51 had |
| ALREADY faced a building-split instrument (the gate splits by building since |
| phase 40), so the genuinely untested cells were the phase-15 ghost mechanisms |
| (rejected pre-gate, on the native panel only) and the feature variants through |
| the Phase-52 building screen column. Four parallel measurements, base |
| reproduced at 1661.7 in every arm. |
| |
| | candidate | cost harness (ALL-48) | building gate | verdict | |
| |---|---|---|---| |
| | GHOST_ZERO | **+472.7** (10W/37L; late +573, early -109) | rec@20 0.866 -> **0.681**, -180 caught (4W/136L) | REJECT | |
| | QUARANTINE | **+395.0 +/- 68.2** (13W/35L; late -189, early +539) | flat: -10/-6/-10/+4/-4, all noise | REJECT | |
| | both | **+682.9** (9W/39L) | rec@20 -> 0.674, -187 caught | REJECT | |
| | vcorr / zone / vrel | (native refs +32.7/+104.9/+138.7) | screen margins **negative at rec@12 in nearly every fold draw** | CONFIRMED REJECTED | |
|
|
| Two mechanisms worth the record: |
|
|
| - **GHOST_ZERO's mask is not ghost-specific.** days_below_245 is sticky, so |
| the zeroed pool GROWS across the walk (2-6 devices early, 33-42 by s_24+, |
| mean 22.4/scenario) and ~12% of it (128 devices over 48 scenarios) are real |
| in-window deaths - each becomes a guaranteed miss. The foreign-fleet hope is |
| quantified away: a target fleet would need a ghost-to-death ratio in the |
| below-2.45 pool far beyond anything the building split simulates before the |
| ~111-early-days-per-avoided-force-swap saving covers the misses. |
| - **QUARANTINE's "label noise" is calibration ballast.** Dropping the |
| looks-dead-labeled-alive negatives barely changes ordering (p42 Spearman |
| 0.9905 at s_20) but nearly DOUBLES the p42 level (0.0187 -> 0.0345): those |
| mislabeled rows are the only counter-evidence that plunge signatures can |
| survive, and removing them inflates every probability the count rule and |
| the ec-margin logic consume. Late improves (-189), early explodes (+539). |
| |
| The building screen column itself AGREED with the native cost harness on all |
| three feature variants - tiny uniformly-positive AUC margins with negative |
| operating-point recall, the familiar mid-list-reshuffle signature. The |
| transfer-shaped fold offers no grounds to revisit anything. |
| |
| **The re-adjudication is complete and the record is now closed on every axis |
| under the deployment-shaped instruments.** Champion 3c7b883 / current head |
| (static @ 0.8, cert and pooling off) stands. Endgame per the user's mechanics |
| (private = user-selected past versions): score the current head publicly; if |
| ~1661, select the head for private (robustness-hardened); else select 3c7b883. |
| |
| ## Phase 54 - full re-adjudication complete + the CV settles (user-driven) |
| |
| ### (a) The exhaustive re-try of every rejected idea (user directive) |
| |
| 10-agent sweep over the entire EXPERIMENTS.md candidate history on the |
| then-current instrument pair, base reproduced at 1661.65 in every arm. |
| All planners (14 arms, +343..+954, all REJECT), all model families (4 slices, |
| ~50 arms, zero ALIVE - logbag==logistic now confirmed on four instruments), |
| all feature re-adds (calendar/vproj/ranks/fht_days/temp30/dbz/stale2 - |
| REJECT; temp_mean upgraded from 'absorbed' to actively harmful +54.5), |
| timing target days-to-crossing (+35.2 REJECT, with the label-process |
| refinement that the administrative EOL is stamped ~AT the confirmation day, |
| not +3d after), and the 87-config timing-knob grid whose single nominal |
| survivor cfg59 (ec_margin 0.6 / emergency_day 48, -6.61 +/- 3.10, 6W/0L/42T) |
| was killed on the building-OOF surface (+14.97 +/- 22.05) - textbook |
| best-of-87 selection noise with the cert-first W/T profile. The peer session's |
| parallel building-OOF candidate sweep confirmed 23 of 24 rejections and |
| parked its one coin-flip survivor (rankavg25fht: wf-OOF panel -48.6 at |
| 17W/16L, self-fit +17.8 with catches -2, static multi-seed never run). |
| |
| ### (b) THE SCORE - the CV settles (user decisions, in order) |
| |
| 1. Building split is the score, always (not the self-fit 1661.7). |
| 2. Foreign buildings' FULL history (before and after the origin) is legal |
| training data - deployment itself hands the model future-of-origin |
| foreign-building knowledge, and it removes the cold-start distortion. |
| 3. Per-building planning (cv lobo) was specified, built, and superseded the |
| same day: it breaks the count rule by construction (129 zero-swap-death |
| building-scenarios, recall 0.367 vs 0.542; per-device thresholds fix the |
| recall - 0.766 at p*=0.10 - confirming the break was count granularity, |
| not prediction). Kept as a diagnostic. |
| 4. Final protocol: EXACT leave-one-building-out predictions (24 models, each |
| trained on the other 23 buildings' full closed history), ONE full-fleet |
| plan per scenario, single value = ALL-48 mean. |
| |
| **Champion on the final CV: `cv all48oof` SCORE = 1877.3** (72 s per run; |
| 5-fold approximation was 1913.3; panel 1857.7). Reference chain for level |
| sanity: self-fit 1661.7 < exact-LOBO 1877.3 < 5-fold-OOF 1913.3 - more |
| foreign training data helps, self-building identity flatters by ~200. |
| |
| Everything is judged against 1877.3 paired per scenario from here, sched-flat |
| rule in force. Public best remains 3c7b883 at 1661.45; private selection |
| recommendation unchanged (score the hardened head publicly, then select). |
| |
| ## Phase 55 - THE SCORE final: temporal-purge CV, sole validator (user directive) |
| |
| The CV settles for good, three user decisions in one afternoon session arc: |
| |
| 1. **Protocol** (implemented by session 59): for each scenario, ONE model |
| fitted on ALL buildings' pairs whose 42d label windows lie fully outside |
| the scenario window (`origin+42 <= T or origin >= T+42` - label-overlap |
| purge), whole-fleet prediction, ONE full-fleet plan, cost per scenario. |
| SCORE = ALL-48 mean, floor off. Rationale (user): admits every forecasting |
| method (no fold plumbing), one-sentence protocol, no leak within the |
| scenario, and the temporal shape matches deployment (training data spans |
| before and after most scored origins - phase 43a measured 93.9% window |
| coverage). The building axis is deliberately not validated (user judgment: |
| same battery type across buildings). |
| 2. **Sole validator** (user, verbatim intent): "we only use purge score going |
| forward, nothing else to validate a new model or optimization method." |
| Documented as standing rule. LOBO's chimera-plan critique (24 stitched |
| calibrations, no per-building cost attribution, seams inflate the transfer |
| penalty) was the user's decisive argument against building-holdout cost. |
| 3. **Consolidation**: cv.py now contains ONLY the purge harness + |
| oofcand/oofreport. all48/static/screen/lobo/ablate/wf/self removed; |
| recover from git 2000281. exp_x29/exp_x30 no longer run as-is. |
| |
| **Champion on THE SCORE: 1972.0** (warmup 3560.3 / panel 1839.6 / nearend |
| 604.8). Level chain, never cross-compare: self-fit 1661.7 (floor on) < |
| exact-LOBO 1877.3 < 5-fold-OOF 1913.3 < purge 1972.0. |
| |
| Paired purge-vs-LOBO diagnosis (59's run; LOBO reproduced 1877.3 exactly |
| before removal): the purge is HARSHER overall (+94.7) - entirely warm-up |
| (3560.3 vs 2875.3: purging the calendar-local +/-42d window starves early |
| scenarios more than building-holdout does) - and modestly softer on the panel |
| (-18.1/origin; sched 8.9 vs 9.2 flat, recall 0.468 vs 0.461, rec12 0.576 vs |
| 0.509). Reading: the same-device-fate channel exists but is ~-18/origin for |
| the LINEAR champion - an order of magnitude below the +134/+216 self-fit |
| flattery - because the label-overlap purge removes the direct channel and |
| logbag cannot memorize identity. The known, accepted residual risk: flexible |
| challengers (rf/gbm/dynamics) are flattered by same-battery training; the |
| receipts are phases 51-54 and they are not to be re-argued unprompted. |
| |
| The leaderboard closures (appetite, cert-first, pooling) rest on public |
| measurements and survive any CV change. Public best 3c7b883 at 1661.45; |
| private selection recommendation unchanged. |
| |
| **Phase 55 addendum (2026-08-22, user): harness de-tuned from the champion's |
| output shape.** "cv.py is tuned towards our current model (rec12 rec20 etc), |
| remove those to make sure we can use the cv on any method we try." Removed |
| the `rank_stats` call and every column built on it (rec12/rec20/recall/prec/ |
| miss_rank/n_sched/caught/n_deaths) - all assumed the method emits a |
| per-device p42 ranking. The harness now requires only that a method produce |
| a PLAN; the one diagnostic kept beside the cost blocks is `swaps` (in-window |
| unique batteries read from the plan itself), preserving the phase-39/51 |
| appetite-in-disguise guard method-agnostically. Cost path untouched: SCORE |
| reproduced 1972.0 exactly post-change (106s). `models_cv.rank_stats` remains |
| for ad-hoc ordering diagnostics on methods that do rank. CSV SCHEMA CHANGE: |
| /tmp/cv_all48oof.csv and cv_oofcand_* now carry n_swaps instead of the rank |
| columns; the pre-change stale candidate CSVs were deleted (they were |
| denominated in dead instruments regardless - any candidate must re-run fresh |
| through `cv oofcand <label>`). |
| |
| ## Phase 56 - full re-adjudication under the purge + radical-pipeline campaign; NEW CHAMPION logbag_rank+gbm30 (1630.3) |
|
|
| User directive: re-run ALL improvement ideas on the purge instrument, including |
| previously public-refuted axes ("remove the closed line... we can re-try all of |
| that"), spin off agents, be radical ("change the whole pipeline method"). |
| Execution: 11 parallel lanes, ~250 candidates, every full-48 run paired per |
| scenario against the champion baseline. Harness verified DETERMINISTIC |
| (champion repeat = bit-identical on all 48), so paired SE is scenario-sampling |
| noise only; survivor bar = delta < 0 beyond ~2*SE at ~flat swaps. |
| Live per-lane records: agent_reports/*.md (kept in-repo). |
|
|
| ### NEW CHAMPION (adopted): logbag_rank + gbm value-preserving rank blend, w=0.30 |
| |
| cv.run_all48_oof(kind='logbag_rank', blend=('gbm', 0.30)) |
| THE SCORE 1630.3 vs 1972.0 (-341.7 +/- 103.8, t=-3.3, 37W/11L, |
| swaps 9.23 vs 9.21 flat; wins all three blocks: wu 2276/panel 1677/ne 504; |
| gain entirely late_swap: -305 of -342) |
| |
| Components, independently discovered and individually significant: |
| - logbag_rank (champion + within-origin fleet-rank features r_vlast/r_tau): |
| -225.6 +/- 74.0, 34W/11L. Linear, rank-based, no same-device channel. This |
| kind was REJECTED on the old self-fit instrument (phase 8); purge reverses. |
| - gbm order increment on that base: -116.1 +/- 53.6, 31W/13L. Broad weight |
| plateau (.25/.30/.35 all <= -335). CAVEAT (md lane): a monotonic-constrained |
| HGB partner DESTROYS the increment (+142..+269) - the gbm contribution is |
| specifically non-monotone response, i.e. real interactions or residual |
| same-battery flattery. Accepted risk per standing directive; public arbiter. |
| Local optimum confirmed by 5 lanes: hyperparams (C=3 coin-flip -15.0+/-13.2, |
| W/L against - kept C=1), features (12 full runs, all subsumed by r_vlast/r_tau; |
| r_tau load-bearing), training weights (proximity/dedup/hard-example all |
| neutral-to-worse on the frontier), neural rankers (RankNet/ListNet lose |
| head-to-head +99..+120), physics blends (sim-rank 1.9sigma on old champ, ghost- |
| promoting). vrel subsumed by ranks (worse in both combos). |
|
|
| DEPLOYMENT: submit.py FrontierRUL (BATTERYSWAP_FRONTIER=0 reverts to logbag). |
| Fidelity test: bit-exact vs the CV candidate on s_10/s_30 (max p42 diff 0.0, |
| ordering identical). Fit ~3.3s incl. gbm; budgets and degradation paths |
| unchanged. cv.py CHAMPION now carries the blend (candidates with explicit |
| kind/blend do NOT inherit it). |
| |
| ### INSTRUMENT CALIBRATION - the campaign's most important negative result |
| |
| The purge CANNOT adjudicate two classes, proven by public anti-correlation: |
| 1. COUNT-COUPLED changes: purge alpha curve is monotone DOWN to 1.5 (-202) |
| - the exact config the public refuted at +527 (phase 28). Same shape via |
| p_be top-ups, EC-inclusion counts, dedup class-rebalance (tr: the -117 |
| dedup win is 80% a smuggled +1.7-swap appetite shift; its count-matched |
| twin is -24 n.s.), gamma flattening, cohort gates. |
| 2. LABEL-PROCESS exploits: wd re-derived cert-first on the frontier at |
| -26.2 (3W/0L/45T) - the EXACT local signature (-37.9, 3W/0L/45T) that |
| preceded the public +43.2 (phase 48, tested ISOLATED on the clean stack; |
| the lane's exculpation claim was checked against the record and is FALSE). |
| Ghosts (crossed-never-labeled devices) exist publicly, are zero in train, |
| and are invisible to ANY local instrument. |
| SHIPPING BAR going forward: ordering-only, label-process-free, swaps flat. |
|
|
| ### Report-only results (real on the purge; fail the shipping bar) |
|
|
| - pl_stack_hgatez (frontier + arrival-triggered p_be=0.2 top-up): 1559.4 |
| overall best raw number, but the gate's marginal on the frontier is |
| -70.8 +/- 42.4 at +3.2 swaps with panel n.s. - count-coupled. The frontier |
| self-repairs warmup (its wu swaps 9.9 vs 5.0), removing the gate's niche. |
| - pl_stack_shiftp4 (visit day 4*p42 days earlier): 1614.0; swap-neutral and |
| reproducibly negative on two bases (champ -13.2+/-9.4, frontier marginal |
| -16.3+/-10.9) but sub-2sigma each. Optional public A/B candidate. |
| - tr_hard_prox60 (hard-band x3 + proximity hl60 weights): -107.5 +/- 41.2 on |
| the PLAIN base only (2.6sigma, swaps flat) - subsumed by the frontier. |
| - sv_dhgate (censored-pool discrete-time hazard when train deaths < 20): |
| -51.4 +/- 22.1 - real, but the gate NEVER fires in deployment (full-history |
| fit; cf. phase 36's inert warm-up mechanism, same effect size). |
| - tj_blend25e3 (MC voltage-trajectory sim rank blend 25%): -86.1 +/- 45.5 |
| (1.9sigma) on the old champion base; promotes crossed-unlabeled devices |
| (cert class) - report-only. Standalone physics pipeline: honest negative |
| (labels carry unrecoverable info; count-matched it catches fewer deaths). |
| |
| ### Closed by this campaign (with numbers, on the purge) |
| |
| Planner: day placement CONVERGED (e2 surrogate local search -1.7+/-1.6 for 4x |
| effort; ed/sd/blend day variants reject); piggyback/second-chance swaps dead |
| (+8.8/+52); EC-economics count dead (inferior appetite); CVaR/portfolio dead |
| (deaths do not cluster by building). Models: survival family dead (aft +337, |
| cox +546, wiener +1100); conformal = byte-identical to logbag; building fixed |
| effects dead (+423/+48); MoE/staleness gates inert; calibration null-to- |
| harmful (Platt +21, isotonic -16.8 n.s.); MLP noise. Features: raw levels |
| fail where ranks transfer (whit +43 raw vs -49 ranked); aux families dead |
| (+95..+124); v_last_only +453 (blind-repo phase-39 claim dead on this |
| instrument); top-12 slot squatters are label-process ghosts that still die at |
| 19% - no flag can demote them profitably. Training: recency dead, proximity |
| real-but-subsumed; label windows d35/d21 count-starve (+72/+105); building- |
| balanced +40; thinning +19..+32; past-only pool +19 (future-side purge pairs |
| stabilize fits). Candidate CSVs: /tmp/cv_oofcand_*.csv (~180 full-48 files). |
| |
| ### Phase 56 addendum - PUBLIC REFUTES the unanchored frontier (698401a: 1877.08); mechanism found and fixed same-day |
| |
| | | swaps | late-days | early-days | total | |
| |---|---|---|---|---| |
| | champion 3c7b883 | 580 | 5819 | 13197 | 1661.45 | |
| | frontier 698401a | 517 | 7076 | 8011 | 1877.08 | |
| | delta | **-63** | **+1257** | -5186 | +215.63 | |
| |
| The failure is a COUNT collapse, not (necessarily) an ordering failure: the |
| value-preserving blend preserves the PRIMARY's (logbag_rank's) p42 values, |
| and under deployment's PAST-ONLY fits that model's calibration collapses as |
| the pool grows - measured sum(p42) ratio vs plain logbag: 1.02 (s_5), 1.01 |
| (s_15), 0.95 (s_25), 0.80 (s_35), 0.78 (s_45). Public scenarios live in the |
| late-origin regime -> -11% swaps -> ~+5 extra misses/48. The purge CV could |
| not see it: its fits train on BOTH sides of the window, where the two models |
| calibrate identically (swaps 9.23 vs 9.21). Fourth public refutation of a |
| locally-validated intervention (alpha, cliff, cert, unanchored frontier) - |
| but the first with the failing channel measured and separable. |
| |
| FIX (same commit): FrontierRUL now VALUE-ANCHORS to plain logbag |
| (BATTERYSWAP_FRONTIER_ANCHOR=1 default): the donor logbag supplies p42 |
| values - count/pricing/timing byte-equivalent to the old champion's measured |
| 580-swap calibration - and the rank+gbm frontier supplies ONLY the order. |
| Ordering-only in deployment by construction, not just in CV. Verified at |
| s_35/s_45 (worst collapse origins): anchored sum(p42) == plain logbag |
| exactly; order == frontier blend exactly (top-13/top-25 sets identical; |
| tie-aware rank comparison zero mismatches). What the anchored head tests |
| publicly, cleanly for the first time: is the frontier ORDER better on the |
| scored fleet? Downside is bounded to ordering (count machinery is the |
| measured champion's). |
| |
| ### Phase 56 addendum 2 - full decomposition: the frontier's purge gain was a hidden per-scenario appetite dial; head reverted to legacy default |
| |
| Anchor decomposition on the purge (all paired, 48 scenarios): |
| | arm | values (count source) | order | SCORE | vs old champ | |
| |---|---|---|---|---| |
| | unanchored frontier | logbag_rank | frontier | 1630.3 | -341.7 (publicly +215.6) | |
| | scale-anchored | lbr shape, donor sums | frontier | 1925.1 | -46.9 +/- 42.6 (33W/14L, n.s.) | |
| | value-anchored | plain logbag | frontier | 1934.8 | -37.2 +/- 44.1 (n.s.) | |
|
|
| The smoking gun: mean swaps are FLAT (9.21 vs 9.23) but per-scenario counts |
| swing |d| = 2.56 on average (23/48 scenarios differ by >= 2), and |
| corr(count delta, cost gain) = 0.759. The 15 scenarios where logbag_rank's |
| sum(p42) swelled the budget carry +956 gain each (~= the entire -342); the |
| 25 where it shrank carry -2. MECHANISM: in both-sides purge fits, future-side |
| pairs from a deadly era raise the fitted base rate exactly in deadly |
| scenarios - era-level hazard leaking through CALIBRATION. The label-window |
| purge cannot block it; deployment's past-only fits cannot replicate it |
| (measured collapse 0.78-0.80x at late origins); the public board refuted it |
| (-63 swaps, +215.6). PURGE LIMITATION #3, and it retroactively weakens every |
| "swaps flat at the mean" clearance in this campaign: the shipping bar is now |
| COUNT-MATCHED (value-anchored) evaluation, where the frontier's true |
| ordering value is -37..-47 +/- 43 - real-looking W/L (33/14) but sub-bar. |
| |
| ACTIONS: BATTERYSWAP_FRONTIER default OFF (head = legacy logbag champion, |
| the measured-1661.45 behavior, plus all hardening); frontier dials retained |
| for explicit A/B (ANCHOR=scale|value|off, W2). No model-side campaign |
| candidate survives count-matched adjudication. Instruments going forward: |
| deployment-mirror walk-forward (past-only fits) as the public predictor; |
| building-GroupKFold with test-group-planned-as-fleet (user proposal, |
| 2026-08-22) as the held-out/private-fleet predictor. |
|
|
| ## Phase 57 - CV replaced (user directive v3): building-GroupKFold, test group planned as a standalone fleet |
|
|
| The temporal purge is RETIRED as THE SCORE (its three deployment blind spots |
| are the phase-56 record). New sole harness in cv.py: buildings -> 5 folds, |
| stratified deterministically on failure count (snake assignment; fold deaths |
| 22/18/14/15/13 over 5/5/5/5/4 buildings) and travel centrality as the |
| location proxy. Per fold: ONE fit on the TRAIN buildings' closed pairs (full |
| history - deployment-legal for foreign fleets), then for each of the 48 |
| scenario times the TEST buildings are planned as a COMPLETE standalone fleet |
| and costed. SCORE = mean over folds of fold-mean cost. This is a measured |
| simulation of submit.py's held-out (static foreign fit) path - the path the |
| private split's unseen fleets will take - with no per-building attribution |
| problem (the fleet IS the test group) and no partial-fleet count bias |
| (phase 26). Fits never see the test buildings; nothing sees the future of a |
| label it trains on beyond what deployment itself would know. |
| CLI: `cv bgkf` / `cv cand <label>` / `cv report`. Purge harness recoverable |
| at f5f3fda; walk-forward family at 2000281. Candidate rule strengthened by |
| phase 56: wins must hold at ~flat swaps PER CELL, not just at the mean. |
|
|
| Phase 57 baseline (champion = deployed plain logbag): **SCORE 352.0** |
| (folds 489.0/322.6/365.9/289.1/293.7; blocks wu 523.2 / panel 354.7 / |
| nearend 108.8; swaps 1.9/scenario on ~90-device test fleets - the count rule |
| scales with fleet size as designed). NEW DENOMINATION: per-test-fleet cost, |
| never cross-compare with the full-fleet instruments (1972.0 purge / 1661.7 |
| self-fit). Harness runtime 22s - cheap enough to run per candidate without |
| triage subsets. |
|
|
| Phase 57 alignment check vs recorded public scores (user request; pooling |
| arm killed mid-run): champion 352.0 reference; cert-first bgkf -7.3+/-4.3 vs |
| public +43 (flip); frontier -5.3+/-7.4 vs public +216 (flip, but far less |
| fooled than the purge's -342, and it reproduces the public swap-count-drop |
| direction); alpha 1.5 -28.8+/-13.6 at swaps 3.15 vs public +527 (flip - |
| still rewards appetite; note 65W/118L, the count-asymmetry signature). |
| Verdict: better than the purge, still sign-flips all three public killers - |
| fleet-composition facts (uncatchable public tail, train-absent ghosts) are |
| invisible to ANY local split. The structural shipping bar (flat swaps per |
| cell, no label-process exploits) remains the only local defense. One |
| coherent cross-fleet signal: vrel -13.4+/-7.9, 104W/61L, swaps flat. |
|
|
| ## Phase 58 - definitive local-vs-public coherence check on the v5 harness (deployment-faithful GroupKFold) |
|
|
| All five publicly-measured configs reproduced AS SHIPPED (the alpha point as |
| the true ALPHA_STATIC=1.5 + CLIFF_SCALE=2.2 stack driving the count through |
| p42_sum; cert stacked on rank_score; pooling with time-masked own pairs; |
| deployment-exact planner state incl. the gated floor; v5 baseline = 352.0, |
| byte-identical to v3, floor confirmed inert): |
|
|
| | config | local delta +/- SE | swaps | public delta | decision at the 2-sigma bar | |
| |---|---|---|---|---| |
| | alpha15+cliff22 | -15.4 +/- 16.8 | 4.18 | +527.3 | NOT adopted (0.9 sigma) - correct | |
| | cert-first | -7.3 +/- 4.3 | 1.90 | +43.2 | NOT adopted (1.7 sigma) - correct | |
| | pooling | +10.2 +/- 3.6 | 1.92 | +4.4 | rejected - SIGN AGREES | |
| | frontier | -5.3 +/- 7.4 | 1.79 | +215.6 | NOT adopted (0.7 sigma) - correct | |
| | (alpha15 solo, no public point) | -28.8 +/- 13.6 | 3.15 | - | 2.1 sigma BUT count-coupled (swaps +1.25) -> quarantine rule catches it | |
|
|
| VERDICT: the v5 CV cannot reproduce public MAGNITUDES (ghosts and the |
| uncatchable tail are absent from every local fleet - unfixable), but as a |
| DECISION INSTRUMENT the full protocol - v5 harness + 2-sigma adoption bar + |
| count-flat rule + no-label-process rule - makes the correct call on all five |
| public measurements: zero false adoptions, one correct rejection with sign |
| agreement. Notably the faithful reproduction itself mattered: alpha-solo |
| reads a confident-looking -28.8 (2.1 sigma) while the actually-shipped |
| alpha+cliff stack reads a statistical zero - earlier alignment checks that |
| approximated configs overstated the misalignment. Residual sharpest risk: |
| cert-class (1.7 sigma local win, public +43) - the label-process quarantine |
| is what protects against it, not the harness. |
|
|
| THE CV going forward: v5 building-GroupKFold (deployment-faithful held-out |
| simulation, one instrument for public AND private - both are ~2x held-out |
| fleets), used for ordering/model questions under the quarantine rules. |
|
|
| Phase 58 addendum (user): SCORE aggregation switched to the familiar |
| full-fleet magnitude - per scenario the 5 fold costs are SUMMED (every |
| building is out-of-fold exactly once per scenario -> the sum is the |
| all-24-building out-of-fold cost), SCORE = mean of the 48 totals. |
| **Champion baseline: SCORE = 1760.2** (warmup 2616.1 / panel 1773.4 / |
| nearend 544.0; swaps 9.5/scenario). Identical information to the fold-mean |
| form (352.0 x 5), same paired decisions; stale pre-v5 candidate CSVs purged |
| from /tmp (only the phase-58 faithful runs remain valid). |
|
|
| ## Phase 59 - campaign 2 (bgkf instrument, 7 lanes) closes; frontier ft_vmin7r_wnr20 ADOPTED and deployed (3f9bf12) |
|
|
| Deadline-day morning. Campaign 2 re-tested every prior method plus radical |
| non-p42 pipelines on THE SCORE (bgkf v5 summed, baseline 1760.2), with a |
| coordinator-maintained FRONTIER board for cross-lane composition |
| (agent_reports_bgkf/). Overnight outage (repo re-cloned, /tmp+venv+dataset |
| lost) fully recovered: dataset copied from rul_mgt_zero, venv symlinked to |
| the blind repo (versions identical), baseline and every headline number |
| reproduced BIT-EXACT (the harness is deterministic end to end). |
|
|
| **ADOPTED - ft_vmin7r_wnr20 (commit 3f9bf12, BATTERYSWAP_FT2 default ON):** |
| logbag on FEATURES+vmin7_rank, wiener .20 value-preserving order blend. |
| 1688.4 vs 1760.2 = -71.8 +/- 23.5 (3.05 sigma, Wilcoxon .037, swaps 9.58 |
| flat, count-clean by every control, components additive -50.3 wiener / |
| -23.2 vmin7r, reproduced exactly by ft, md and rk lanes independently). |
| Deployment review that preceded adoption: |
| - Reference-shift worst case (the vrel killer): vmin7_rank recomputed |
| WITHIN each ~5-building test fleet -> still -60.8 +/- 26.3 (2.31 sigma); |
| consistent small degradation vs table-native (+11.0 +/- 17.9, 9W/35L). |
| Deployment ranks over a full ~922-device fleet (features_at_origin |
| own-fleet semantics), closer to the shared-reference case. |
| - Ft2RUL parity: submit.py implementation run through the CV harness |
| reproduces the frontier CELL-EXACT (max diff 0.0, swaps identical). |
| - Deployment data path: wnr_m/s2/n merged into the shipped train table |
| (97% coverage; WienerFHT shrinks through NaNs); split-path wiener params |
| computed past-only from the scored split's own telemetry (1.3s once + |
| 0.1s/scenario). Soft degradation: missing wnr -> pure vm7 ordering |
| (values identical by construction); any fit error -> legacy champion. |
| BATTERYSWAP_FT2=0 = the measured-1661.45 config. |
| |
| **NOT adopted (the two bigger nominal numbers):** |
| - rk_4w (vm7 values; order .35 logbag + .20 wiener + .20 RankNet + .25 |
| aux_ho) = -111.5 +/- 27.7, 4.02 sigma, Wilcoxon .001 - the campaign's |
| strongest fully-anchored number, seed-block replicated. Head-to-head vs |
| the frontier: -39.7 +/- 20.3 but 15W/15L/18T, Wilcoxon .131 - mean-only |
| support; plus a torch dependency (MLP 32-16 x 7 seeds) the Docker is not |
| known to carry. SHELVED (agent_reports_bgkf/artifacts/). |
| - wd raw vm7+aux_ho+wiener.20 = -126.4 +/- 43.8: rk's 4-way control run |
| settled the adoption form - aux_ho as an ORDER source adds +6.1 n.s. on |
| the frontier, so the raw excess is VALUE-CHANNEL (the calibration channel |
| publicly refuted in phase 56). Report-only. aux_ho's anchored ordering |
| core (-59.7 on champion base) is real but subsumed. |
| |
| Lane verdicts (all closed, findings files final): pl - ranking is the wall |
| (~99% of late cost = exclusion misses; routing axis +/-1). tr - keep the |
| training pool exactly as-is; value-anchored twin flipped every count-smuggle. |
| bl - wiener .20 the one piece; bagging/local-blend/calibration refuted. |
| md - logbag(C=1,b=15) locally optimal; monotone HGB real (-44.7 anchored) |
| but subsumed (+1.0 as third source); MORE BAGS HURT. ft - vmin7_rank the |
| piece; entire vrel family quarantined (anchored twin +21.4, within-fleet |
| recompute +89). rk - see above. wd/radical - physics loses to the 82 |
| labels at matched count (+604); EC planner = count knob; direct death-day |
| rejected both ways. |
| |
| Private-selection matrix as of this phase: candidate head = 3f9bf12 (FT2 |
| frontier); fallback = 3c7b883 (measured 1661.45); never 698401a (1877.08). |
| The user manually triggers public scoring; a public look at 3f9bf12 prices |
| the residual ghost/composition risk the CV cannot see (phase 58). |
| |
| ## Phase 60 - campaign 3: exhaustive post-frontier exploration; frontier confirmed as the pipeline ceiling |
| |
| User directive: improve the champion CV further, radical options included, |
| review all assumptions (podium context: team-repo scale ~1400). ~45 |
| candidates over 7 waves + 2 device-level autopsies + full mining of the |
| team repo (rul-mgt-zero, 1493.65 public vs our 1661.45). Full detail: |
| agent_reports_bgkf/c3_findings.md. |
| |
| KEY FINDINGS: |
| 1. AUTOPSY overturns the silent-death assumption: frontier recall is 48%; |
| 237 missed deaths at median rank 5 (gap quartiles 2/3/8 below a ~rank-2 |
| cutoff) carry 70% of remaining cost; only 45/237 are voltage-silent. |
| False includes are plateau devices at 2.41-2.44V for 40+ days (36% die, |
| median 83d — just outside the window). The prize is top-band ordering. |
| 2. No candidate reaches it: every feature/order/model/count axis measured; |
| best survivor is a -6-point crossing-label micro-piece (xlb, Wilcoxon |
| .0475, shelved). ft2 (1688.4, 3f9bf12) is the ceiling of this pipeline |
| under the count-clean bar. |
| 3. Appetite re-audited and re-closed: alpha 1.0 reads -64.5 mean but |
| 17W/24L, Wilcoxon .375 — warmup-carried count redistribution; gap-band |
| precision ~11% < ~16% break-even. |
| 4. Team repo decoded: their edge = multi-level self-supervised targets + |
| poskeep_target sampling (-155.6, not portable — we already train on all |
| rows) + CP-SAT planner over all nine cost components. Their CV is |
| in-fleet (1412.6 not comparable to our held-out 1688.4). Direct feature |
| transplant fails in our model form (+15..+167). Cross-pollination |
| candidates TO them: WienerFHT order blend, vmin7_rank, building-held-out |
| transfer check; their gen-10 state-dependent dV/dT (resistance channel) |
| is the strongest open lead anywhere. |
| |
| STANDING RECOMMENDATION unchanged: public-trigger 3f9bf12/head; private = |
| 3f9bf12 unless publicly refuted, else 3c7b883; never 698401a. |
| |
| ## Phase 61 - team solution evaluated ON our bgkf; operating-point finding; alpha-1.0 A/B prepared |
| |
| User directive: run the team repo's stack under our CV and adopt structural |
| changes if they improve it. Full port (panel rebuilt on our dataset with |
| 81/81 exact EOL reproduction; model retrained building-OOF, h42 AUC 0.961; |
| CP-SAT planner) plus four mix-and-match decompositions. NO ADOPTION: |
| their full stack +228 vs ft2 (p .017, over-buys on ~90-device fold |
| fleets); their ordering loses decisively anchored (+412, monotone |
| dose-response at every weight); our ft2 model driving their CP-SAT lands |
| at -96.3 mean but 16.4 swaps/scen, 25W/23L, Wilcoxon .72 = count-carried, |
| mean-only. Instrument caveat recorded: their machinery targets 460-920 |
| device fleets; our folds are ~90 - public is the only cross-pipeline |
| arbiter (them 1493.65 vs us 1661.45). |
| |
| OPERATING-POINT FINDING (leaderboard decomposition): the entire podium |
| swaps 15-17.6/scenario with ~60-70 late-days/scen vs our 12.1 swaps and |
| 121 late-days; the CP-SAT mix independently lands at the podium's |
| operating point on our predictions. The alpha-1.5 public failure (6ea03da, |
| zero catches) was on the OLD ordering; whether ft2's ordering converts |
| extra appetite publicly is the one open question a public trigger can |
| answer. PREPARED: commit 296f8f2 = alpha 1.0 on ft2 (single variable, |
| selectable for public scoring); head 7d7632c keeps the safe 0.8. |
| |
| ## Phase 62 - synthetic scaling study (user directive): CV tested on fleet-size scaling; fixed-reference vmin7 dial shipped |
| |
| Synthetic fleets at 1x/2x/4x (461/922/1844 devices) from time-shifted |
| building clones with OFFICIAL-mechanism-derived EOLs (0 mismatches on the |
| checkable set at every scale); deployment-faithful static-path runs; |
| anchor k=1 champ = 1762.5 ~= bgkf 1760.2 (same recall .471). Verdicts: |
| count rule scales cleanly (swaps/death 0.98-1.07); live-rank vmin7_rank |
| decays and inverts at 4x while a train-ECDF fixed-reference form is |
| scale-neutral (shipped as BATTERYSWAP_FT2_FIXEDREF, default 0 - the Docker |
| rehearsal measured 922-device splits = the safe 2x regime); alpha 1.0 best |
| at EVERY scale with recall +7pts held (third independent support for the |
| 296f8f2 public A/B); runtime 3.4s/scenario at 4x - no budget risk. Full |
| tables: agent_reports_bgkf/c3_findings.md. |
| |
| ## Phase 63 - PUBLIC CONVERSION: head 4c93883 measured 1575.31 (deadline day); wave T all dead |
| |
| The user triggered public scoring on head 4c93883 (CV 1545.4; values |
| lb(vm7)@alpha1.0, order .50 logbag + .30 wiener + .20 gapHGB, |
| value-preserving): **public 1575.31**, -86.1 vs the 3c7b883 champion's |
| 1661.45. Components invert to exact integers (clean run, default |
| settings, 48 scenarios): 644 swaps (13.4/scen, was 580 at alpha .8), |
| 5009 late-days (104.4/scen, was ~121), 19716 early-days. CV->public |
| transfer +29.9 (~2%): bgkf v5 priced the whole stack (ft2 ordering + |
| alpha 1.0 + gap-HGB + weight tweaks) correctly - the first |
| locally-validated intervention stack to survive public after three |
| pre-bgkf refutations. PRIVATE SELECTION: 4c93883 (hedge slot, if any: |
| 3c7b883). |
| |
| Wave T (post-ship, 13 arms, paired vs 1545.4): ALL DEAD - the champion |
| is a local optimum on every axis touched. Wiener THRESH 2.40 beats |
| 2.38/2.42 (+39/+45 - the labeled barrier is also the ordering-optimal |
| barrier); K_SHRINK 6 vs 3/12 flat/+9; gap-weight .15 shelf +30/+29; |
| gapHGB seed-bag +34 at 2.0sig (seed-0 is favorable; do NOT "stabilize" |
| it away) and lr .03 +12; rank-shape transforms pow2/sqrt/logit |
| +4/+42/+58 (linear pct-rank blending confirmed); GAP4 inside the logbag |
| order-primary +123 at 2.3sig (gap features only work as a separate |
| voice); RankNet 4th source at .10 +22. Driver /tmp/c9/run.py (volatile). |
| |
| ## Phase 64 - T2 five-lane subagent campaign on the 1545.4 champion: one adoption (lam2sp3 planner objective), four axes closed with structure |
| |
| Five parallel lanes, each paired vs the champion on bgkf (reports: |
| agent_reports_bgkf/t2_{timing,planner,order,counts,hazard}.md): |
| |
| - TIMING: closed. Early cost decomposed - 74% imputed-EOL false positives |
| (never die in-window), 19% real deaths beyond the window, 7% genuine |
| early timing; planner packing is 0.6%. Every date shift trades early |
| for late at slope >=0.86; best arms 1.0sig AND count-coupled. EARLY is |
| a selection problem, not a dates problem. |
| - PLANNER: **ADOPTED lam2sp3** - timing curves x2 inside optimize_days' |
| objective (new TIMING_WEIGHT dial) + SPLIT_DAYS 7->3. -4.11 +/- 1.41 |
| (2.9sig), 25W/4L, Wilcoxon 8.8e-6, count-flat with battery sets |
| cell-identical; dose-response plateau on both dials; additivity checks; |
| survives ~35-arm multiplicity (sign p 1e-4). Mechanism: refuses cheap |
| trip-merges that drag swaps off timing-optimal days (late -3.5, early |
| -1.2, mobility +0.8). Also: plan-phase scheduling is SATURATED past its |
| own surrogate optimum (exact enumeration is +5.1 WORSE - shallow search |
| is an accidental regularizer; closes the CP-SAT question for good); |
| of 232/scen "ops" cost only 78 is schedulable, 154 rides on missed |
| swaps (recall in disguise). |
| - ORDER: closed. Boundary cascade fails structurally (cutoffs are n~2-3, |
| the band IS the confident top; +147 at 2.6sig); per-building rank |
| normalization catastrophic (+549/+939, 7-8sig) - global pooled ranking |
| is load-bearing; prob-space blending loses to rank-space (+76). The |
| rank-gap-1-10 frontier is unreachable by aggregation on this table. |
| - COUNTS (all report-only): flat alpha 1.1-1.5 dead (podium-posture 1.4 |
| is a 1.0sig coin-flip, median-worse); calibrated counts REFUTED with a |
| count-matched control (+67.5 at 3.4sig - allocation failure); risk-N |
| refuted. Headline: warmup-gated alpha 1.4 (s_0-7 only) -> 1488.0, |
| -57.4 +/- 22.0 (2.6sig nominal, post-hoc block, 8-scenario support, |
| count-coupled) - candidate for a public trigger, user's call. |
| - HAZARD: closed for the deadline. dth3 crash fixed; discrete-time |
| hazard HGB is a standalone statistical peer of the 3-way blend |
| (h_alone 1537.0, 25W/22L) with its edge at COLD-START origins |
| (censored-pool training; warmup -178/cell) - converges with the counts |
| lane on warmup as the only exploitable block; unresolvable at 2sig on |
| 8 cells of +/-800 scatter. AFT-lite dead on arrival. |
| |
| SHIP (lam2sp3): submit.py TIMING_WEIGHT (default 2.0, env |
| BATTERYSWAP_TIMING_WEIGHT) + SPLIT_DAYS default 7->3; models_cv.py |
| plan_tsroute.timing_weight attr (default 1.0 = old path bit-exact, |
| verified: ft2a10_sp7 control cell-diff 0.0 vs pre-T2 reference); |
| cv.py run_bgkf split_days/timing_weight hooks, ft2a10/head track the |
| deployed dials. Verified: cand ft2a10 = 1541.3, cell-exact (0.0) vs the |
| lane's lam2sp3 measurement; paired -4.11 +/- 1.41 (2.9sig) 25W/4L |
| p8.8e-6, swaps flat 2.27. New champion CV 1541.3. |
|
|
| ## Phase 65 - CV frontier 1398.0 (user CV-first directive): warmup-gated alpha 1.4 x gated hazard order, both at deaths_seen<35 |
| |
| User directive: CV is the reported/optimized metric; count-coupled |
| candidates are KEPT (not auto-shipped) as champions-in-waiting. Frontier |
| config st_hz100t35_g16: at origins where the planned fleet's recorded |
| deaths <= T number fewer than 35 (~s_0-15 on bgkf), ORDER = discrete-time |
| hazard HGB (censored-pool trained, c14/c17 machinery) replacing the 3-way |
| blend, and planner alpha = 1.4; everywhere else byte-identical to the |
| 1541.3 head. CV 1398.0, -143.3 +/- 37.4 (3.8sig), 16W/0L, Wilcoxon <.001, |
| swaps 2.53/cell (vs 2.27; count-coupled by design). |
|
|
| Decomposition of the -143.3: gated alpha 1.4 width-16 alone -125.4 |
| (4.0sig, clean bowl: width 6/8/10/12/14/16/20/24/32/48 -> |
| 1501/1483/1462/1436/1429/1416/1414/1438/1475/1502; level bowl 1.3/1.4/ |
| 1.45/1.5/1.6 -> 1476/1436-1403/1416/1427-1449/1490 with 1.4 optimal at |
| every width); hazard order adds -13 at thr20 and -18 at thr35 (16W/0L |
| final). EC break-even expansion (teammate mechanism through our pricing) |
| REFUTED: margin 1.0/1.2/1.5 monotone worse (+17/+29/+46), so the count |
| posture belongs to gated alpha, not EC top-ups. Teammate public analysis |
| (1486.57 vs our 1575.31): +191 swaps -> late -353, early +316, ops -53; |
| pure operating-point difference, no luck involved either way. |
|
|
| Caveats attached to 1398.0: gate widths selected on the shared 48 |
| scenarios (multiplicity); deployment rule re-expressed as deaths_seen<35 |
| (plan-time observable, FLOOR_GATE family); the composite harness carries |
| exact-rank-tie noise vs a deployed-class implementation (16/240 cells, |
| documented in t3_coldstart.md) - a real ship needs the gate inside the |
| deployed class with non-gated origins byte-identical. Pending: t3_gm |
| label-statistic lane; synthetic-scale validation of the count-coupled |
| family (user-commissioned; key question = does the gate close early on |
| large fleets, bounding the downside). Drivers /tmp/c19, /tmp/c20 |
| (volatile); arm CSVs /tmp/cv_bgkfcand_c19_*, c20_*. |
| |
| ## Phase 66 - SHIPPED the warmup gate (user call): head CV 1418.3 |
| |
| Deployed form of the phase-65 frontier inside Ft2RUL (user approved the |
| ship for a public trigger): rate-form gate WARMUP_RATE=0.0759 deaths per |
| training-fleet device (= 35/461; a fixed count fires wider on the CV's |
| 4/5-size fold ledgers than on deployment's full ledger - measured 1429.7 |
| fixed vs 1418.3 rate), TRAIN-ledger observable seen = deaths recorded <= |
| plan origin; while gated: ORDER = censored-pool discrete-time hazard HGB |
| (_DiscHaz, person-period buckets, depth3/lr.06/250it), appetite x1.4 via |
| a consume-once plan_tsroute.alpha_boost. eol column derived from |
| origin+days_to_eol when absent (the cached parquet ships no eol; exact, |
| 0 diff on 8206 dying rows - this bug made the first rehearsal silently |
| inert). VERIFIED: cand ft2a10 = 1418.3, -123.0 +/- 43.1 (2.9sig, 14W/5L, |
| Wilcoxon .0017, 19 changed origins, swaps 2.54); WARMUP_RATE=0 control |
| bit-exact vs the 1541.3 head (0.0 max cell diff); rehearsal: gate fired |
| on exactly s_0-14 (+2..+5 swaps each), s_15+ byte-identical, 0 warnings, |
| 41s. Delta vs the phase-65 composite 1398.0 (~+20) = the composite's |
| harness-only advantages (full-table hazard pool, literal-16 gate, tie |
| noise); the deployed number is the honest one. COUNT-COUPLED - public |
| trigger arbitrates; previous head remains 60d65da (1541.3/public |
| lineage 1575.31) via WARMUP_RATE=0. |
| |
| **Phase 66 postscript — PUBLIC REFUTATION.** e4745d2 publicly scored |
| **1594.89** (+19.6 vs the 1575.31 head). Mechanism executed exactly as |
| designed: +38 swaps in the cold scenarios, late-days -263 (-54.8) — but |
| early +59.8 and ops +14.4 made the trade ~1.4:1 against, vs ~6:1 |
| favorable on bgkf warmup. Fourth count-coupled public refutation |
| (alpha_static, cliff, cert-first, warmup-gate); alpha 1.0 remains the |
| lone converter. bgkf's warmup block over-prices cold-scenario catches |
| relative to the public fleet. PRIVATE SELECTION: 60d65da (count-flat |
| lam2sp3, expected ~1571) or measured 4c93883 (1575.31); not e4745d2. |
| |
| ## Phase 67 - T4 count-flat campaign (6 lanes, ~50 arms): ZERO adoptions - 60d65da confirmed as the count-flat ceiling of this feature space |
| |
| User-directed sub-1400 count-flat push. Every lane paired vs 60d65da |
| (1541.3) at 2sigma, parity-verified, swaps flat. All six REJECTED with |
| mechanisms (reports agent_reports_bgkf/t4_*.md): |
| - STACKING: learned metas over-weight the best-AUC source (gapHGB); plan |
| cost is decided at the cutoff, which the cost-tuned .50/.30/.20 already |
| encodes. Loss monotone in meta dose (+16.5 at 25% ... +139 HGB-meta). |
| - EXTERNAL data/models: NASA transfer collapses to a monotone transform |
| of the current voltage gap (Spearman .999) - lab aging corpora have |
| near-uniform fade slopes; foundation models unloadable in the runtime |
| stack (no transformers/safetensors). Measured +32.5 / wash. |
| - SURVIVAL/LTR: Cox, cost-weighted listwise, martingale-GBM all lose the |
| gap slot by >=2.6sig WITH identical features - the dies42 classification |
| objective itself is the value. Cost-aligned listwise is the worst |
| (gradient too diffuse at ~1.2k positives/157 groups). |
| - LITERATURE features (knee/Bacon-Watts, sample entropy, monotonicity, |
| relaxation, variance): all lose or wash inside the gapHGB; best |
| standalone-AUC features lose MOST (dilute the depth-2 splits on the |
| validated quartet). Anchored logbag control failed on cue (+61.6). |
| - TRANSPLANT (rank-4 MnesisLab, full source read): geo-aware ordering |
| tilt +135..+452 (their compactness gate passes on all our fold-fleets, |
| so gating does not rescue it); seasonal temp layer = plan no-op (gated |
| devices already top-ranked); left-truncated FP-AFT faithful (params |
| match theirs) but moves ~1 unit; their EB wiener +128. THEIR 1407.63 IS |
| OPERATING POINT + their untransplantable base pipeline, not a readable |
| layer: 743 swaps converting at ~16.8 late-days/swap vs our 6.0 marginal. |
| - SEQUENCE (GRU-D on 60d daily telemetry, 5-seed, fold-safe): encoder |
| genuine (OOF AUC 0.94-0.96) but fully redundant with the snapshot |
| stack; monotone-against dose response (+21.6 at w=.05). |
|
|
| CONCLUSION: 60d65da (CV 1541.3) is the measured count-flat ceiling of |
| this feature space across our own features, the literature's, and the |
| readable competitors' - the ordering axis is closed from every direction |
| tried. PRIVATE SELECTION stands: 60d65da first, 4c93883 (measured |
| 1575.31) as the no-inference alternative. |
|
|
| ## Phase 68 - T5 global-optimum probe (6 structural lanes): ZERO adoptions - 60d65da is the honest optimum of this framing, search closed |
|
|
| User-directed final campaign ("it's a local optimum but probably not the |
| global one") - four commissioned angles plus the two rank-3-derived |
| prototypes, every arm paired vs 60d65da (1541.3), parity-verified, |
| count-flat unless noted (reports agent_reports_bgkf/t5_*.md): |
| - BASE REBUILD: competitor-style wide-space stacked base (78 leak-verified |
| cols, 4-learner stack) IS a better dies42 classifier than logbag |
| (per-fold OOF AUC 0.965 vs 0.938, 4/5 folds, biggest where labels are |
| scarce) and it does NOT price: |delta| <= 9.2 (<=0.3sig) at every mixing |
| weight. AUC gains live in separating obvious cases; cost lives in the |
| fine ordering near the inclusion cutoff, which logbag already has. |
| Base-model capacity is not the binding constraint. |
| - JOINT SEARCH: 8-dim differential evolution, 90 evals, tune/validate |
| origin split. Best tune delta -104; 100% of it sign-reverses on the |
| held-out origins (top-3 all flip). The CV's resolution is exhausted at |
| this scale - any future multi-dial "win" of this size is presumed noise |
| unless it passes the same split. |
| - SELECTION ARCHITECTURE: expected-gain (cost-based) selection at pinned |
| count: best arm -5.0+/-28.8 (0.2sig); the geography term is a measured |
| LOSS +27..+31/scen (2sig, robust across three geo-on/off contrasts). |
| The rank-4 team's -14.6 selection edge exists only count-floating. |
| - DATA WALL: learning curve 40->100% is shelf-not-slope (local slope ~0, |
| p=.68); a hypothetical DOUBLED corpus prices at ~-44 [-100..0]; label |
| audit recovers zero positives. The surface rewards building/failure-mode |
| COVERAGE, not volume - the optimum is not gated on labels nobody has. |
| - SEASONAL MARGIN (rank-3 recipe): the fleet seasonal factor is REAL here |
| (p2p 0.0585 V in demeaned fleet-median voltage, matches their 0.048) |
| but injecting it into the wiener barrier distance loses at every |
| strength on both climatology maps (best arm +1.0, and the mild setting |
| is significantly harmful, wilcoxon p=.031; one Dec origin +546 from a |
| 2 mV shift). Their -104 lived in calibrated-probability expected-cost |
| selection; in a rank-only frozen-count slot the factor only reorders, |
| and it reorders worst where deaths cluster. Wrong lever, axis closed. |
| - DRAWDOWN REFRAME (rank-3 core method, faithful rebuild): 1.72M |
| (h,m)-conditioned censored drawdown rows from 71,878 battery-days, their |
| exact HGB (450/.07/63/200), 107 causal features, fold-safe. Monotone- |
| against dose response: alone +255.6 (3.9sig), as 4th source @.10 +14.1 |
| (0.7sig loss, best arm); survival-curve read strictly worse. OOF AUC |
| 0.958 vs raw v_min_7's 0.961 - it learns first-passage of the LAGGING |
| smoothed statistic while the champion sits on leading raw-voltage |
| indicators. Their 1350.7 is their whole stack (expected-cost scheduler, |
| seasonality, timing), not an order signal that transplants. |
| |
| CONCLUSION: with T2-T5 plus peer transplants (~150 arms) the ordering, |
| base-capacity, selection, data, seasonality and reframing axes are ALL |
| closed with mechanisms. 60d65da (CV 1541.3, count-flat) stands as the |
| genuine optimum of this framing at this instrument's resolution. |
| |
| FINAL SELECTION (deadline 2026-08-23 23:59): private pick 60d65da first; |
| 4c93883 (publicly measured 1575.31) as the no-inference alternative. |
| e4745d2 publicly refuted (1594.89). Never 698401a. Count-coupled CV |
| frontier (gated hazard+alpha 1398.0) remains report-only: the family is |
| 0-for-4 on public conversion. |
| |
| ## Phase 69 - PUBLIC read on the restored head (71bfff2 = 60d65da config): 1583.11, lam2sp3 does NOT transfer - private pick flips to 4c93883 |
| |
| User-triggered public scoring of the gate-off head: total 1583.109 vs |
| 4c93883's measured 1575.307 (+7.80). Component inversion (all integers, |
| clean run): swaps 644 = IDENTICAL to 4c93883 (count-flat exactly as |
| designed); late-days 5005 (-4); early-days 19638 (-78) -> timing |
| components -1.64; ops/packing residual +9.45 (daily_limit 112.5, |
| overtime 89.2, weekly_limit 64.6, travel 49.1, bldg/room 17.2). |
| MECHANISM: timing_weight=2.0 + split_days=3 re-packs swap days - it |
| bought the intended timing gain but pays +9.4 in schedule/ops constraints |
| on the public fleets. The bgkf priced the same trade at -4.1+/-1.4 |
| (2.9sig, 25W/4L) on train buildings: the ops cost of tighter packing is |
| fleet-composition-sensitive (building sizes/geography), which |
| building-OOF CV on train buildings cannot see. First COUNT-FLAT |
| CV->public inversion (prior four were count-coupled); the CV remains |
| accurate on ordering/inclusion but does not price planner-packing |
| trades across fleets. |
| FINAL SELECTION (updated): private pick 4c93883 (measured 1575.31, |
| best public we own) FIRST; 60d65da/71bfff2 (measured 1583.11, CV |
| favorite at 1541.3) as the second slot / CV-side hedge if two picks |
| exist. Never e4745d2 (1594.89) or 698401a (1877.08). |
| |
| ## Phase 70 - Fleet-composition probe: the CV's lam2sp3 win is a LOW-PRESSURE artifact - pick locks to 4c93883 |
| |
| Post-refutation diagnostic (report agent_reports_bgkf/t5_fleetcomp.md, |
| raw /tmp/c35): A = tw2.0/sd3 (60d65da) vs B = tw1.0/sd7 (4c93883), |
| identical model, paired, count-flat in all 1536 stage-2 cells. |
| - Fold fleets: A wins 5/5 but pays an ops tax in 4/5 - the CV win is |
| entirely late/early timing on small sub-fleets. |
| - 32 random subset fleets (3-12 bldgs): 20W/8L/4T, median -0.14; the |
| dials only ACTIVATE under pressure - P(cell differs) 5.5%->85% from |
| <50 to 300-400 devices, corr(delta, n_dev) -0.71. |
| - PRESSURE fleets (18 bldgs, 250-384 devices): 3/6 flip to A-loses; the |
| two big losses replicate the public signature almost exactly (+7.90 |
| with dOps +10.82; +9.31 with dOps +9.35; public +7.8 with ops +9.45, |
| timing slightly better in all three). |
| - CAPSTONE: the full 24-building universe planned as ONE fleet flips A |
| to +0.97 (24W/20L, dLate -4.4, dOps +6.47). The CV's 2.9sig win exists |
| only because bgkf test fleets are five small sub-fleets where |
| daily/weekly limits never bind. |
| VERDICT: the -4.1 CV delta is not transferable to large fleets; at |
| public-like pressure A-vs-B is a coin flip locally and A measured worse |
| publicly. INSTRUMENT FIX (for future work): adjudicate planner/packing |
| dials on pressure-matched fleets (whole-universe-as-one-fleet + stress |
| fleets at 250-400 devices), not the pooled small-sub-fleet mean. |
| FINAL SELECTION locked: 4c93883 (measured 1575.31). 60d65da/71bfff2 |
| only as a second slot. Never e4745d2 / 698401a. |
| |
| ## Phase 71 - Pressure-matched CV instrument: CALIBRATION FAILED - no instrument change, no final round; selection stays 4c93883 |
| |
| User-approved attempt to upgrade the CV to pressure-matched evaluation |
| (whole-24-building-universe as one fleet, OOF-stitched per-device |
| predictions from the 5 fold fits) and re-run a packing-dial grid on it. |
| Calibration gate: retrodict the public pair (B=tw1/sd7 1575.31 < |
| A=tw2/sd3 1583.11, dOps +9.45). Result: PRESSURE(B)=1619.8, |
| PRESSURE(A)=1615.9, A-B = -3.88+/-5.17 (22W/19L/7T), dOps -1.71 - |
| WRONG SIGN on both total and ops; grid auto-stopped per protocol. |
| Diagnosis (t6_pressure.md): at binding pressure the A-B delta is a |
| daily/weekly-limit COLLISION LOTTERY - +/-100/200 quanta ~13x the +7.8 |
| target signal, landing on different origins under OOF-stitched vs |
| leaky-fit predictions (per-origin deltas uncorrelated, r=-0.08) - with |
| only 48 origins the one-fleet mean cannot be calibrated; any grid |
| ranking would fit the lottery. Corollary: the public +7.8 itself is |
| likely 1-2 collision events, i.e. the packing channel may be |
| UNRESOLVABLE by any instrument at this scale; the measured public |
| number is then the only defensible ranking, and it favors B. |
| Redesign direction for future work (not attempted tonight): |
| sign-frequency over a many-fleet stress ensemble rather than a |
| one-fleet mean; stress validator ready at /tmp/c36/stress_winner.py. |
| FINAL: cv.py UNCHANGED (an uncalibrated instrument would mislead); |
| selection stays 4c93883 first, 60d65da/71bfff2 second slot only. |
| |
| ## Phase 72 - Final count-flat closure (c28/c37/c38) + alpha 1.4 public A/B ship (user call) |
| |
| Deadline corrected to 02:00 CEST (midnight UTC) - time for one more |
| public A/B. Count-flat space closed with three sweeps (all paired vs |
| 1541.3, parity exact in each driver): |
| - c28 hyperparams (logbag C/bags, gapHGB l2/leaf): 8 arms, ALL lose or |
| wash (+5.6..+34.2). Champion hyperparameters locally optimal. |
| - c37 blend-weight jitter (+/-.05 around .50/.30/.20, 6 directions): |
| ALL lose +23..+50 (several ~2sig AGAINST). Sharp optimum. |
| - c38 averaging ensembles (leaders' style, order-only: device-bootstrap |
| x3, drop-fifth x5, time-snapshot x3, feature-set x2): best is a 0.3sig |
| wash (devboot -6.2+/-24.9), timebag a 2.2sig LOSS. Averaging destroys |
| the favorable seed-0 configuration (same lesson as wave T seed-bag). |
| ALPHA 1.4 SHIP (62e9d91, count-coupled, explicit user call): fresh |
| confirm on head config bgkf 1498.3 (-43.0+/-43.0, 1.0sig, 19W/24L, |
| swaps 2.27->2.97/fleet = podium operating point 15-18/scen; mean carried |
| by death-heavy origins - the public read arbitrates). Static path |
| governs held-out fleets (submit.py:836 ALPHA_STATIC), both knobs set. |
| Rehearsal: 520->688 swaps (+32.3%), 43/48 scenarios; ship-code counts |
| byte-identical to env-override run. Old-ordering alpha 1.5 failure |
| (2189) predates ft2 - this is the operating-point hypothesis' first |
| test on the current ordering. DECISION RULE: public < 1575.31 -> select |
| 62e9d91; else select 4c93883. Revert = both alpha defaults 1.0. |
|
|
| ## Phase 73 - PUBLIC CONVERTS alpha 1.4: 1480.85 (-94.5) - both pre-registered gates pass; SELECTION = 62e9d91 |
|
|
| User-triggered public read of 62e9d91: total 1480.85 vs 1575.31 |
| (4c93883). Pre-registered decision rule (phase 72): margin >= 15 AND |
| late-swap fingerprint (late down >= 100). Both pass decisively: |
| - margin -94.46 (6x threshold); components invert to integers (clean). |
| - swaps 747 (+103, 15.6/scen - INTO the podium band 15-17.6); |
| - late-days 4006 (-1003; component -208.96 = 2x the required |
| fingerprint) - the marginal-rank batteries DO contain real deaths on |
| held-out fleets: ~0.8 extra catches/scenario at a strongly |
| profitable exchange (early +115.35, ops -1.40 flat). |
| FIRST large count-coupled public conversion (family was 0-for-4; the |
| old "ranks 13-20 convert at ~0%" verdict was an OLD-ordering fact - |
| the ft2 ordering's marginal band converts). The public-overfit |
| objection was pre-answered: single pre-registered hypothesis from |
| external evidence (podium operating points + bgkf -43), component |
| fingerprint predicted before the read, margin rule against collision |
| noise. By sibling-split exchangeability the effect transfers to |
| private. SELECTION: 62e9d91 (public 1480.85) PRIMARY; 4c93883 |
| (1575.31) fallback slot. Dose-response CV (alpha 1.6/1.8/2.0) running |
| to decide whether ONE more A/B is justified - stopping rule: only if |
| the CV curve is still falling at 1.6+; iterating alpha values on |
| public beyond one more point = the probing pattern we reject. |
|
|
| Phase 73 postscript - dose-response complete: alpha 1.4 -> 1498.3 |
| (-43.0), 1.6 -> 1562.1 (+20.8), 1.8 -> 1573.7 (+32.4), 2.0 -> 1589.0 |
| (+47.7). Clear minimum at 1.4; monotone rise beyond. NO second public |
| A/B per the stopping rule. FINAL: select 62e9d91 (1480.85); fallback |
| 4c93883 (1575.31). |
|
|
| ## Phase 74 - Re-tune at alpha 1.4 (user-directed): dials orthogonal EXCEPT the wiener weight - basin floor .20 (CV 1476.0, -22.3, 2.0sig) |
|
|
| 22+10 arms paired vs the alpha-1.4 reference (1498.3), parity exact in |
| all drivers, count-flat (swaps 2.95-2.98): |
| - Alpha fine grid: 1.4 is a SHARP minimum (1.3 +36; 1.35 +2.7 wash; |
| 1.45 +22.9 at 1W/21L; 1.5 +47.6 at 3W/29L; 1.6/1.8/2.0 monotone up). |
| Public-confirmed operating point = the exact CV bowl bottom. |
| - Packing (tw x sd) at 1.4: flat (< +4.6) - independent of alpha. |
| - Hyperparams (logbag C/bags, gapHGB l2/leaf) at 1.4: all within +-4.3. |
| - ORDER WEIGHTS SHIFT: at alpha 1.0 the .50/.30/.20 mix was a sharp |
| optimum (phase 72: all jitters +23..+50); at alpha 1.4 the wiener->gap |
| direction is a coherent basin: .25/.25 -13.9 (p.013), .23/.27 -3.7 |
| (p.042), floor at wiener .20 - .50/.20/.30 = 1476.0 (-22.3, 2.0sig, |
| p.013) and .55/.20/.25 = 1475.2 (-23.1, 1.6sig, 26W/8L p.004) - then |
| rising (.15/.35 -8.0, .10/.40 +6.2). Mechanism: the wiener barrier |
| distance ranks imminent deaths well but discriminates poorly in the |
| deeper 15-19/scen band that alpha 1.4 buys; the gapHGB is stronger |
| there. The dial's CV optimum is OPERATING-POINT-DEPENDENT. |
| Winner's-curse haircut (best of ~30 arms at this alpha): true effect |
| likely -12..-18. NOT shipped as of this entry: the selection stands on |
| the publicly MEASURED 62e9d91 (1480.85); re-selecting an unmeasured |
| re-weight is a user-level risk decision (no public triggers remain). |
|
|
| ## Phase 75 - PUBLIC REFUTES the order re-weight: 1495.92 (+15.1) - FINAL selection stays 62e9d91 (1480.85); head reverted |
|
|
| User-triggered read of 354fd2d (.50/.20/.30): 1495.92 vs 62e9d91's |
| 1480.85 (+15.08). Counts flat as designed (752 vs 747 swaps); the loss |
| is spread evenly: late +5.00 (4030 vs 4006 late-days), early +3.67, |
| ops +6.39. CV said -22.3 (2.0sig, basin-confirmed, p.004/.013 floor |
| arms); public says +15.1 - a SIGN FLIP in the bgkf's home channel |
| (ordering). Post-mortem read: best-of-~30 winner's curse plus a real |
| fleet effect - the wiener barrier-distance source earns MORE weight on |
| held-out fleets than building-OOF folds suggest (its physics transfers; |
| the gapHGB's depth-2 splits are more fold-idiosyncratic). Cost of the |
| experiment: zero (selection rule pre-locked, portal unchanged). |
| LEDGER OF THE NIGHT'S A/Bs: packing CV-win refuted (+7.8), warmup gate |
| refuted (+19.6), alpha 1.4 CONVERTED (-94.5, amplified 2.2x), order |
| re-weight refuted (+15.1). The bgkf prices the count/appetite channel |
| and long-standing ordering STACKS, but individual 2sig dial moves in |
| any channel are coin flips publicly - only public A/Bs adjudicate. |
| FINAL: selection 62e9d91 (public 1480.85); head reverted to its config. |
|
|
