Thibaut Forest Claude Opus 5 (1M context) commited on
Commit ·
e44c7a4
1
Parent(s): 41a82cd
Phase 43: innovation sweep - queue pricing, lead-time bias, late decomposition; and a verified data leak we are not taking
Browse files- EXPERIMENTS.md +91 -0
EXPERIMENTS.md
CHANGED
|
@@ -2135,3 +2135,94 @@ in the current feature space - which is exactly what the phase-38/39 silent-
|
|
| 2135 |
death campaign concluded before it was closed.
|
| 2136 |
|
| 2137 |
Standing champion remains **3c7b883 at 1661.45**.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2135 |
death campaign concluded before it was closed.
|
| 2136 |
|
| 2137 |
Standing champion remains **3c7b883 at 1661.45**.
|
| 2138 |
+
|
| 2139 |
+
## Phase 43 - innovation sweep: three new axes opened and closed, one leak found
|
| 2140 |
+
|
| 2141 |
+
Directive: challenge assumptions, look for exploits in the setup, do not stop
|
| 2142 |
+
early. Five things were tested; four are new.
|
| 2143 |
+
|
| 2144 |
+
### (a) The setup leaks the answer (VERIFIED, NOT USED - see the caveat)
|
| 2145 |
+
|
| 2146 |
+
Scenarios are spaced **exactly 7 days** apart and the split ships **one**
|
| 2147 |
+
timeseries file. For 42 of 48 scenarios the entire 42-day death window is
|
| 2148 |
+
already present in that file; mean window coverage is **93.9%**. Combined with
|
| 2149 |
+
the reverse-engineered label rule (gated 10-30 C daily median voltage sustained
|
| 2150 |
+
below 2.40 V), reading forward reconstructs the labels directly:
|
| 2151 |
+
|
| 2152 |
+
| | recall | precision |
|
| 2153 |
+
|---|---|---|
|
| 2154 |
+
| future-voltage reconstruction (10-line rule, all 48) | **0.841** | **0.874** |
|
| 2155 |
+
| production model at its operating point | 0.641 | 0.453 |
|
| 2156 |
+
|
| 2157 |
+
382 of 454 in-window deaths identified with 55 false positives, day error
|
| 2158 |
+
median -3 d (the confirmation lag). A tuned version reaches the 81/82-exact
|
| 2159 |
+
level `check_labels.py` already demonstrates, i.e. approximately the oracle
|
| 2160 |
+
(186 vs our 1661.7).
|
| 2161 |
+
|
| 2162 |
+
**This circumvents the sanctioned accessor.** `iterate_scenarios` explicitly
|
| 2163 |
+
truncates (`cut = flat.loc[:scenario_start]`), so past-only is the intended
|
| 2164 |
+
contract and reading the raw file past the origin breaks it. Recorded because
|
| 2165 |
+
it bounds what the leaderboard can mean, **not** adopted. Our own path is
|
| 2166 |
+
clean: `features_at_origin` filters `day <= origin` (features.py:73) and every
|
| 2167 |
+
physics/beta path does the same.
|
| 2168 |
+
|
| 2169 |
+
### (b) Late cost decomposed: 85% ranking, 15% timing
|
| 2170 |
+
|
| 2171 |
+
Of 454 deaths: **163 never swapped** (ranking failure -> emergency queue),
|
| 2172 |
+
**95 swapped AFTER EOL** (timing failure, 150.8/scenario, mean 7.6 d late),
|
| 2173 |
+
196 swapped before EOL at a median 13 d early. So 84.7% of late cost is
|
| 2174 |
+
inclusion, 15.3% is placement. First time these have been separated.
|
| 2175 |
+
|
| 2176 |
+
### (c) The emergency queue is convex - and unexploitable
|
| 2177 |
+
|
| 2178 |
+
From `evaluate_plan`: misses are swapped one per day in **alphabetical**
|
| 2179 |
+
battery order from `end_day`, each with a dedicated round trip and its own
|
| 2180 |
+
`end_previous_day`. A miss at queue position k therefore pays an extra 10k,
|
| 2181 |
+
and including a battery shifts every alphabetically-later miss one day earlier.
|
| 2182 |
+
That externality is real and unpriced - the true marginal cost of a miss is
|
| 2183 |
+
~330, not the ~220 previously used, which moves break-even from 25% to ~16%.
|
| 2184 |
+
|
| 2185 |
+
Tested as a rank bonus. First form (additive) scored **+1007**: an additive
|
| 2186 |
+
bonus is just alphabetical sorting, because the shift only occurs if the
|
| 2187 |
+
battery actually dies. Corrected to the p42-conditional form
|
| 2188 |
+
`p42 * (1 + qw * 10 * E[misses after b] / 54)`:
|
| 2189 |
+
|
| 2190 |
+
| qw | 0.25 | 1.0 | 3.0 |
|
| 2191 |
+
|---|---|---|---|
|
| 2192 |
+
| all-48 delta | +60.0 | +160.1 | +275.1 |
|
| 2193 |
+
|
| 2194 |
+
Monotone loss, rejected. The mechanism is a ~9% pricing correction but it
|
| 2195 |
+
reorders by **alphabet**, a variable with zero predictive content - measurement
|
| 2196 |
+
law #2 again: ordering damage beats pricing gain at the boundary.
|
| 2197 |
+
|
| 2198 |
+
### (d) Lead-time bias on day placement - real trade, break-even
|
| 2199 |
+
|
| 2200 |
+
(b) says the day model is systematically optimistic, and late/early trades at
|
| 2201 |
+
20:1. Two implementations:
|
| 2202 |
+
|
| 2203 |
+
| k days early | 0 | 2 | 4 | 7 | 10 |
|
| 2204 |
+
|---|---|---|---|---|---|
|
| 2205 |
+
| shift the OUTPUT | 1661.7 | +36.2 | +33.6 | +53.0 | +87.9 |
|
| 2206 |
+
| shift the BELIEF (quantiles, re-solve) | 1661.7 | **-12.0** | -0.3 | -9.7 | -11.3 |
|
| 2207 |
+
|
| 2208 |
+
Shifting the output works on late (987 -> 878) but compresses all work into
|
| 2209 |
+
earlier weeks and costs **+157 ops** in limit penalties. Shifting the predicted
|
| 2210 |
+
death-day quantiles instead lets `optimize_days` re-solve with routing and the
|
| 2211 |
+
weekly limit intact: ops stays at 206-227 and late falls to 922-962. Net best
|
| 2212 |
+
is **-12.0 at k=2, on 23W/25L** - a coin flip, non-monotone across k. Rejected,
|
| 2213 |
+
but it establishes that **day placement is already near-optimal**: the 95
|
| 2214 |
+
late-swapped catches cannot be recovered by earlier scheduling because the
|
| 2215 |
+
early penalty on the 196 correctly-timed ones plus the ops cost consumes the
|
| 2216 |
+
whole gain.
|
| 2217 |
+
|
| 2218 |
+
### Standing after this sweep
|
| 2219 |
+
|
| 2220 |
+
Every axis now has a measurement against it: appetite (public), features
|
| 2221 |
+
(phase 41 fixed-N), training volume (phase 42 public), routing-aware inclusion
|
| 2222 |
+
(phases 7/22, "even zero-travel extras lose"), expected-cost ranking (phase
|
| 2223 |
+
27b), queue pricing (c), day placement (d), silent deaths (phases 6-13, 38-39).
|
| 2224 |
+
|
| 2225 |
+
**The honest residual**: the leader is at 1477.48, so ~184 points are
|
| 2226 |
+
demonstrably extractable by someone. We cannot account for them from any
|
| 2227 |
+
mechanism measurable here, and the only path this sweep demonstrated is the
|
| 2228 |
+
one in (a), which we are not taking.
|