Thibaut Forest Claude Opus 5 (1M context) commited on
Commit
e44c7a4
·
1 Parent(s): 41a82cd

Phase 43: innovation sweep - queue pricing, lead-time bias, late decomposition; and a verified data leak we are not taking

Browse files
Files changed (1) hide show
  1. EXPERIMENTS.md +91 -0
EXPERIMENTS.md CHANGED
@@ -2135,3 +2135,94 @@ in the current feature space - which is exactly what the phase-38/39 silent-
2135
  death campaign concluded before it was closed.
2136
 
2137
  Standing champion remains **3c7b883 at 1661.45**.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2135
  death campaign concluded before it was closed.
2136
 
2137
  Standing champion remains **3c7b883 at 1661.45**.
2138
+
2139
+ ## Phase 43 - innovation sweep: three new axes opened and closed, one leak found
2140
+
2141
+ Directive: challenge assumptions, look for exploits in the setup, do not stop
2142
+ early. Five things were tested; four are new.
2143
+
2144
+ ### (a) The setup leaks the answer (VERIFIED, NOT USED - see the caveat)
2145
+
2146
+ Scenarios are spaced **exactly 7 days** apart and the split ships **one**
2147
+ timeseries file. For 42 of 48 scenarios the entire 42-day death window is
2148
+ already present in that file; mean window coverage is **93.9%**. Combined with
2149
+ the reverse-engineered label rule (gated 10-30 C daily median voltage sustained
2150
+ below 2.40 V), reading forward reconstructs the labels directly:
2151
+
2152
+ | | recall | precision |
2153
+ |---|---|---|
2154
+ | future-voltage reconstruction (10-line rule, all 48) | **0.841** | **0.874** |
2155
+ | production model at its operating point | 0.641 | 0.453 |
2156
+
2157
+ 382 of 454 in-window deaths identified with 55 false positives, day error
2158
+ median -3 d (the confirmation lag). A tuned version reaches the 81/82-exact
2159
+ level `check_labels.py` already demonstrates, i.e. approximately the oracle
2160
+ (186 vs our 1661.7).
2161
+
2162
+ **This circumvents the sanctioned accessor.** `iterate_scenarios` explicitly
2163
+ truncates (`cut = flat.loc[:scenario_start]`), so past-only is the intended
2164
+ contract and reading the raw file past the origin breaks it. Recorded because
2165
+ it bounds what the leaderboard can mean, **not** adopted. Our own path is
2166
+ clean: `features_at_origin` filters `day <= origin` (features.py:73) and every
2167
+ physics/beta path does the same.
2168
+
2169
+ ### (b) Late cost decomposed: 85% ranking, 15% timing
2170
+
2171
+ Of 454 deaths: **163 never swapped** (ranking failure -> emergency queue),
2172
+ **95 swapped AFTER EOL** (timing failure, 150.8/scenario, mean 7.6 d late),
2173
+ 196 swapped before EOL at a median 13 d early. So 84.7% of late cost is
2174
+ inclusion, 15.3% is placement. First time these have been separated.
2175
+
2176
+ ### (c) The emergency queue is convex - and unexploitable
2177
+
2178
+ From `evaluate_plan`: misses are swapped one per day in **alphabetical**
2179
+ battery order from `end_day`, each with a dedicated round trip and its own
2180
+ `end_previous_day`. A miss at queue position k therefore pays an extra 10k,
2181
+ and including a battery shifts every alphabetically-later miss one day earlier.
2182
+ That externality is real and unpriced - the true marginal cost of a miss is
2183
+ ~330, not the ~220 previously used, which moves break-even from 25% to ~16%.
2184
+
2185
+ Tested as a rank bonus. First form (additive) scored **+1007**: an additive
2186
+ bonus is just alphabetical sorting, because the shift only occurs if the
2187
+ battery actually dies. Corrected to the p42-conditional form
2188
+ `p42 * (1 + qw * 10 * E[misses after b] / 54)`:
2189
+
2190
+ | qw | 0.25 | 1.0 | 3.0 |
2191
+ |---|---|---|---|
2192
+ | all-48 delta | +60.0 | +160.1 | +275.1 |
2193
+
2194
+ Monotone loss, rejected. The mechanism is a ~9% pricing correction but it
2195
+ reorders by **alphabet**, a variable with zero predictive content - measurement
2196
+ law #2 again: ordering damage beats pricing gain at the boundary.
2197
+
2198
+ ### (d) Lead-time bias on day placement - real trade, break-even
2199
+
2200
+ (b) says the day model is systematically optimistic, and late/early trades at
2201
+ 20:1. Two implementations:
2202
+
2203
+ | k days early | 0 | 2 | 4 | 7 | 10 |
2204
+ |---|---|---|---|---|---|
2205
+ | shift the OUTPUT | 1661.7 | +36.2 | +33.6 | +53.0 | +87.9 |
2206
+ | shift the BELIEF (quantiles, re-solve) | 1661.7 | **-12.0** | -0.3 | -9.7 | -11.3 |
2207
+
2208
+ Shifting the output works on late (987 -> 878) but compresses all work into
2209
+ earlier weeks and costs **+157 ops** in limit penalties. Shifting the predicted
2210
+ death-day quantiles instead lets `optimize_days` re-solve with routing and the
2211
+ weekly limit intact: ops stays at 206-227 and late falls to 922-962. Net best
2212
+ is **-12.0 at k=2, on 23W/25L** - a coin flip, non-monotone across k. Rejected,
2213
+ but it establishes that **day placement is already near-optimal**: the 95
2214
+ late-swapped catches cannot be recovered by earlier scheduling because the
2215
+ early penalty on the 196 correctly-timed ones plus the ops cost consumes the
2216
+ whole gain.
2217
+
2218
+ ### Standing after this sweep
2219
+
2220
+ Every axis now has a measurement against it: appetite (public), features
2221
+ (phase 41 fixed-N), training volume (phase 42 public), routing-aware inclusion
2222
+ (phases 7/22, "even zero-travel extras lose"), expected-cost ranking (phase
2223
+ 27b), queue pricing (c), day placement (d), silent deaths (phases 6-13, 38-39).
2224
+
2225
+ **The honest residual**: the leader is at 1477.48, so ~184 points are
2226
+ demonstrably extractable by someone. We cannot account for them from any
2227
+ mechanism measurable here, and the only path this sweep demonstrated is the
2228
+ one in (a), which we are not taking.