| # T5 DATA-WALL: is label scarcity the binding constraint? |
|
|
| Baseline 60d65da champion, CV 1541.3 on bgkf (parity verified cell-exact vs |
| /tmp/ref_head_1541.csv: max per-origin diff 0.0). Champion = values |
| logbag(vm7), order .50 rank(logbag) + .30 rank(WienerFHT) + .20 rank(gapHGB), |
| planner p_star=1.0, split_days=3, timing_weight=2.0. All work in /tmp/c32/ |
| (run.py driver, analyze.py + extrap.py analysis, cv_*.csv per-arm results, |
| arms.csv per-draw table, fold_arms.csv fold-level table). No repo file |
| touched; monkeypatch only; nothing committed. |
| |
| ## Design |
| |
| Learning curve: a model-class wrapper drops fold-TRAIN buildings inside |
| fit() before refitting BOTH the values primary and all three order sources; |
| test folds, planner state, and evaluator identical to the champion run. |
| frac=1.0 through the wrapper reproduces 1541.3 exactly (bit-faithful, every |
| origin 0.0). Two samplers, both stratified on per-building train death |
| count: |
| |
| * v1 (`f*`): death-sorted list split into n_keep contiguous strata, one |
| building drawn per stratum. Structural quirk found post-hoc: at frac .8 |
| the multi-building strata all sit at the HIGH-death end, so drops |
| concentrate on high-death buildings (keeps only ~63% of deaths at 79% of |
| buildings). Kept as an informative composition contrast, not discarded. |
| * v2 (`g*`): drops evenly spaced across the death-sorted list (random |
| offset), so kept-death fraction tracks the building fraction. This is the |
| clean volume axis. 5 seeds at .4/.6/.8/.9. |
| |
| Count-flat caveat (by design, per the tasking): subsampling changes the |
| primary's p42 values, so the count machinery moves within each arm (swaps |
| 2.09-2.88 vs ref 2.27). The curve measures the WHOLE champion's data |
| response including that channel, not a values-frozen ordering response. |
| |
| ## 1. Learning curve |
| |
| Per-draw deltas vs 1541.3 (SCORE units; x = actual fraction of the 320 |
| fold-train unique deaths kept, aggregated over folds): |
| |
| | nominal | n draws | mean SCORE | delta | draw scatter (sd) | per-draw deltas | |
| |---|---|---|---|---|---| |
| | 40% | 8 | 1684.6 | +143.3 | 131.4 | +379, -19, +246 (v1); +171, +98, +173, +108, -10 (v2) | |
| | 60% | 8 | 1626.3 | +85.0 | 66.4 | +136, +225, +11 (v1); +72, +63, +60, +51, +62 (v2) | |
| | 80% | 8 | 1657.1 | +115.8 | 66.6 | +146, +198, +211 (v1); +57, +138, +40, +72, +65 (v2) | |
| | 90% | 5 | 1612.0 | +70.6 | 47.3 | +45, +93, +11, +136, +68 (v2) | |
| | 100% | 1 | 1541.3 | 0.0 | - | parity, exact | |
| |
| v2-only (clean volume axis): x=0.42 -> +108; x=0.62 -> +62; x=0.76 -> +74; |
| x=0.89 -> +71; x=1.00 -> 0. |
| |
| The shape is NOT a smooth data-volume law. It is a shelf plus a cliff: |
| any subsample from 60% to 90% costs a roughly constant +60..+75, and the |
| local slope on the v2 shelf 0.6->0.9 is -29.9 +/- 70.1 per unit |
| death-fraction (p=0.68) - i.e. per +10% data: -3.0 +/- 7.0, statistically |
| zero. The drop to 0 happens only at exactly 100%, which is the |
| zero-perturbation parity point (the alpha=1.0 count appetite and every dial |
| were tuned on the full-data model); the shelf offset is perturbation + |
| composition cost, not marginal data value. |
| |
| Composition dominates volume, three independent lines: |
| |
| * Matched death-fraction ~0.63-0.65: v1 draws (drops hit high-death |
| buildings) +185.0 mean vs v2 draws (even drops) +80.7 mean. Same label |
| count, 2.3x the damage when the lost labels come from high-death |
| buildings. |
| * Two 40%-of-corpus draws MATCH full data (f40s1 -18.7, g40s4 -10.5): the |
| model family already saturates on a favorable 40% subset (~32-37% of |
| rows, ~39-42% of deaths). |
| * Fold-level paired regression (150 fold x arm points, delta fold cost vs |
| deaths dropped): slope +24.0 per dropped unique death (~ +0.5 SCORE per |
| death; p=0.009) but r=0.21 - deaths-dropped explains only ~4% of the |
| variance; which buildings dropped explains the rest. |
| |
| ## Extrapolation to 150% / 200% data (honest bounds) |
| |
| Fits on all draws + the exact 100% anchor, x = kept-death fraction; |
| bootstrap resampled within levels, 3000 reps; gains measured against each |
| fit's OWN y(1) so the fit's anchor miss does not inflate them: |
| |
| | fit (v2-only + anchor) | y(1.0) | gain at x=1.5 | gain at x=2.0 | |
| |---|---|---|---| |
| | power y = c + a*x^-b | 1586.8 (misses true 1541.3 by +45.5) | -25.7 [-47..0] | -43.7 [-80..0] | |
| | log y = b0 + b1*ln x | 1586.4 (miss +45.1) | -26.6 [-58..+3] | -45.5 [-100..+5] | |
| |
| (All-draws fits: gain200 -35 [-103..-9] power, -90 [-162..-10] log; the log |
| fit is dragged steep by the v1 composition-skewed deep cuts and misses the |
| known anchor by +38 - treat it as an upper bound on optimism.) |
| |
| Both parametric families overshoot the known 100% point by ~+45: the data |
| reject a smooth volume curve, and the smooth-fit gains are therefore |
| upper-ish bounds. Face value: +50% more same-kind data buys ~26 SCORE, |
| doubling buys ~44, with 95% bounds touching 0 in the clean (v2) fits. |
| The local shelf slope says the marginal value of same-kind data at the |
| current corpus size is indistinguishable from zero. |
| |
| Big caveat the curve cannot price: "same-kind" is load-bearing. Data |
| containing NEW failure modes (the public fleet's uncatchable tail / |
| train-absent ghosts, public-measured 4x) is a distribution-shift question, |
| not a volume question - this probe says nothing about it. |
| |
| ## 2. Label audit |
| |
| Ledger: 82 devices with an eol; 80 with eol <= last origin (2026-07-27); |
| 2 died 2026-07-28 (after the last origin, before data end 07-31). |
| |
| * bgkf convention (origin <= max-42d): ALL 80 observable deaths appear as |
| dies42=1 in at least one closed row. ZERO deaths lost. The window_closed |
| column convention likewise loses zero (82/82 incl. the two late deaths). |
| * walk_forward's comment ("every death <= origin lands in some closed |
| window") verified on the actual table: holds at 47/48 real origins; |
| violated once, transiently, at 2025-10-06 (1 of 12 deaths not yet in a |
| closed window - the 4d synthetic grid had not yet closed a window |
| containing it). Immaterial: fold train sets use the max-origin cut. |
| * Recoverable rows the convention DOES discard: 69 open-window rows with |
| dies42=1 (8 devices, of which only 2 devices - the 07-28 deaths - are |
| entirely absent from closed-window positives). |
| * Recovery arm (`aug`): champion refit on train + those recovered positive |
| rows (train buildings only), values + all order sources. Result: SCORE |
| 1570.1, +28.8 +/- 24.1 vs 1541.3 (1.2 sigma), 15W/33L, Wilcoxon p=0.147. |
| Swaps 2.28 vs ref 2.27 - count effectively flat, so the (insignificant) |
| harm is not an appetite artifact. No usable signal is being discarded; |
| adding near-boundary positive rows without their matching negatives |
| slightly distorts the base rate for nothing. |
| |
| ## 3. Per-fold death count vs cost |
| |
| Raw 5-fold regression (parity run): train_deaths 58/62/66/65/69 vs mean |
| test cost 426/263/336/291/224 -> slope -14.3 per death (r=-0.77, p=0.13), |
| but this is confounded: folds with fewer train deaths are exactly the folds |
| whose TEST buildings hold more deaths (22/18/14/15/11), i.e. harder fleets. |
| The clean version is the paired fold x arm regression above (+24.0 per |
| dropped death, p=0.009, r=0.21), which corroborates a real but weak volume |
| effect swamped by composition. |
|
|
| ## VERDICT |
|
|
| Label volume is NOT the binding constraint; the champion model family is |
| data-saturated on this corpus. The learning curve is flat from 60% to 100% |
| of the data (local slope ~0, p=0.68), favorable 40% subsets already |
| reproduce 1541.3, and even the optimistic smooth-fit extrapolation prices a |
| DOUBLED corpus at ~-44 SCORE [-100..0] - far less than the method-search |
| deltas this repo routinely adjudicates, with bounds touching zero. The |
| label audit closes the other escape hatch: the training convention discards |
| essentially nothing (0 of 80 observable deaths lost; recovering the 69 |
| discarded open-window rows is a wash at best, +28.8 +/- 24.1). |
|
|
| What the response surface actually rewards is COVERAGE - specific |
| high-death buildings carry non-redundant failure-mode information (same |
| label count, 2.3x damage when they are the ones missing). Consequences for |
| the T5 probe: (a) further same-kind data would not move the CV much, so the |
| global optimum is not gated on labels nobody has; (b) capacity/structure |
| search on this corpus is also near its ceiling for THIS champion family |
| (flat shelf = the extra 40% of data adds nothing the model can use); |
| (c) the remaining headroom, if any, lives in mechanisms that generalize |
| across failure modes / fleets (distribution shift), which neither this |
| curve nor more of the same labels can buy. |
|
|
