Spaces:
Running on Zero
05 - The QC decision framework (RQ1, RQ3)
Back to 04: Uncertainty | README
Covers Phase 8, the contribution this project exists for. Everything before it is infrastructure.
1. The headline, including what it costs
The triage policy decisively beats three of five baselines and does NOT beat the fourth at the project's own assumed cost ratio. Both halves are reported, because the second is the one a reviewer should press on.
| Policy | Cost per cell | Escape rate | Chamber cycles/cell |
|---|---|---|---|
| TRIAGE (N=5) | 7.035 ± 3.626 | 0.003 ± 0.014 | 6.687 ± 0.585 |
| cost_tuned_threshold | 6.525 ± 1.585 | 0.024 ± 0.035 | 100.000 ± 0.000 |
| naive_threshold | 7.418 ± 2.498 | 0.045 ± 0.046 | 100.000 ± 0.000 |
| aql_sampling | 99.715 ± 96.460 | 0.905 ± 0.042 | 63.190 ± 20.207 |
| cycle_to_eol | 40.078 ± 3.238 | 0.000 ± 0.000 | 801.557 ± 64.754 |
| no_screening | 206.413 ± 26.715 | 0.911 ± 0.046 | 0.000 ± 0.000 |
Paired across the same 50 outer folds:
| Comparison | Triage better by | Folds won | p |
|---|---|---|---|
| vs AQL acceptance sampling | +92.82 | 50/50 | 1.8e-15 |
| vs cycle-everything-to-EOL | +33.19 | 50/50 | 1.8e-15 |
| vs no screening | +199.52 | 50/50 | 1.8e-15 |
| vs naive threshold | +0.53 | 23/50 | 0.71 |
| vs cost-tuned threshold | −0.37 | 14/50 | 0.055 |
Beating AQL is the industrially meaningful claim and it holds by a factor of roughly 14x. AQL makes ONE decision for a whole lot, so with 34.7% of cells below the warranty target it either scraps good lots or ships bad ones; its per-fold cost variance (±96) is larger than the triage policy's entire cost.
The cost-tuned threshold is not beaten at the measured ratio. Section 4 explains why, and when that reverses.
2. The QC scorecard
Mean ± std across 50 outer folds, budget by budget.
| Metric | N=5 | N=10 | N=20 | N=50 | N=100 |
|---|---|---|---|---|---|
escape_rate |
0.0029 ± 0.0144 | 0.0000 ± 0.0000 | 0.0011 ± 0.0079 | 0.0013 ± 0.0088 | 0.0047 ± 0.0191 |
overkill_rate |
0.5789 ± 0.2814 | 0.5394 ± 0.2742 | 0.4956 ± 0.2624 | 0.3560 ± 0.2054 | 0.1940 ± 0.1118 |
yield |
0.3584 ± 0.1979 | 0.3866 ± 0.1825 | 0.4173 ± 0.1837 | 0.5004 ± 0.1459 | 0.5994 ± 0.0992 |
chamber_cycles_per_cell |
6.6873 ± 0.5847 | 12.4423 ± 1.3291 | 26.7910 ± 4.2286 | 55.1483 ± 5.0605 | 100.0000 ± 0.0000 |
cost_per_cell |
7.0354 ± 3.6255 | 6.6293 ± 3.4751 | 6.8169 ± 3.5053 | 6.5599 ± 2.5884 | 6.8904 ± 1.3249 |
continue_rate |
0.3375 ± 0.1169 | 0.2442 ± 0.1329 | 0.2264 ± 0.1410 | 0.1030 ± 0.1012 | 0.0000 ± 0.0000 |
The escape rate stays at or below 0.005 at every budget while overkill falls from 0.58 to 0.19 -- the policy is buying yield with extra cycles while holding escapes near zero, which is the intended behaviour under an asymmetric cost matrix.
3. RQ1 -- the diagnostic budget frontier
The frontier is flat. Every budget from 5 to 100 cycles is statistically indistinguishable in cost per cell: 5, 10, 20, 50, 100.
- Cheapest budget: N=50
- Economic knee: N=5, by the stated rule -- the smallest budget whose bootstrap 95% CI for cost per cell overlaps that of the cheapest budget
The rule is written down rather than eyeballed because "the knee of the curve" otherwise invites choosing whichever point supports the conclusion. It is also capable of returning the smallest budget, which is exactly what happened, and that is a finding rather than a failure to find a knee.
Chamber time released
| Chamber cycles per cell at the knee | 6.69 |
| Chamber cycles per cell, cycling to end of life | 801.6 |
| Chamber time released | 99.17% |
This is the number a plant manager acts on. It is large because cycling to end of life takes ~800 cycles per cell on this corpus while a decision can be made at 5.
The honest caveat: the flatness means the extra cycles buy little in cost terms under this cost matrix, not that early prediction is as accurate as late prediction -- Phase 6 showed RMSE improving steadily with budget (0.080 → 0.058). The decision is insensitive where the prediction is not, because the cost matrix is dominated by a few expensive misgrades rather than by average accuracy.
4. RQ3 -- when is the triage machinery worth its complexity?
The comparison that matters is against the cost-tuned threshold: the same model, the same features, the same folds, an optimally tuned decision margin, and no CONTINUE action. It isolates the value of the sequential, uncertainty-aware machinery specifically.
At the reference budget the two differ like this:
| Triage | Cost-tuned threshold | |
|---|---|---|
| Escape rate | 0.0047 | 0.0236 |
| Overkill rate | 0.1940 | 0.0532 |
| Cost per cell | 6.890 | 6.525 |
The triage policy buys a 5x lower escape rate by accepting 3.6x more overkill. Whether that is a good trade depends entirely on the cost ratio -- which is what the sensitivity sweep is for.
| Escape:overkill | Triage cost | Cost-tuned cost | Triage advantage | Triage escape | Tuned escape |
|---|---|---|---|---|---|
| 2:1 | 6.433 | 5.698 | -0.735 | 0.1040 | 0.0495 |
| 5:1 | 6.213 | 6.093 | -0.119 | 0.0272 | 0.0303 |
| 8:1 | 6.301 | 6.488 | +0.187 | 0.0067 | 0.0303 |
| 11:1 | 6.726 | 6.883 | +0.157 | 0.0000 | 0.0303 |
| 14:1 | 6.998 | 7.278 | +0.280 | 0.0000 | 0.0303 |
| 15:1 | 6.998 | 7.409 | +0.411 | 0.0000 | 0.0303 |
| 20:1 | 7.078 | 8.067 | +0.989 | 0.0000 | 0.0303 |
| 50:1 | 7.318 | 12.015 | +4.697 | 0.0000 | 0.0303 |
| 150:1 | 7.318 | 25.173 | +17.855 | 0.0000 | 0.0303 |
| 500:1 | 7.401 | 71.228 | +63.827 | 0.0000 | 0.0303 |
The crossover sits between 5:1 and 8:1, below the measured baseline ratio of 10.9:1. So at the project's own cost assumptions the triage policy is favourable, but only modestly; its advantage grows without bound as escapes become more expensive, and it is unfavourable if escapes cost less than about 6x an overkill.
The mechanism is visible in the escape columns: the triage policy's escape rate falls to zero as escapes get more expensive (0.104 → 0.027 → 0.007 → 0.000), while the cost-tuned threshold stays pinned at 0.0303 regardless. A static margin has no uncertainty representation to adapt with; it can shift where it cuts but cannot become more cautious about the cells it is unsure of.
Which conclusions hold across the whole range, and which do not
| Conclusion | Holds across 2:1 – 500:1? |
|---|---|
| Triage beats AQL, cycle-to-EOL and no-screening | Yes, at every ratio |
| Triage beats a cost-tuned threshold | No — only above ~6:1 |
| Triage escape rate ≤ cost-tuned escape rate | Yes, at every ratio |
| The budget frontier is flat | Yes |
The sweep is asymmetric, and it matters here
The configured sweep is [5, 15, 50, 150, 500] against a measured baseline ratio of 10.9. Only one configured point sits below the baseline — and that one point is precisely where the triage policy loses. The configured sweep therefore samples the region where the conclusion fails far more sparsely than the region where it holds.
That is why a supplementary fine sweep at [2, 5, 8, 11, 14, 20] was run to locate the crossover. Reporting only the configured sweep would have shown four favourable points and one unfavourable one, which overstates how robust the conclusion is.
5. Phase 8D -- capacity-constrained allocation
Real QC cannot hold everything. With 10.9:1 costs, 20% of a batch holdable (200 slots per 1000 cells), deciding at N=20 and advancing held cells to N=100:
| Allocation policy | Cost per cell | Grading cost | Escape rate |
|---|---|---|---|
| greedy_voi_per_cycle | 5.973 ± 1.965 | 4.166 | 0.012 ± 0.029 |
| random | 6.422 ± 1.722 | 4.616 | 0.022 ± 0.038 |
| uniform | 7.378 ± 1.141 | 5.572 | 0.014 ± 0.032 |
| confidence_only | 7.467 ± 1.490 | 5.661 | 0.019 ± 0.040 |
Every policy holds the same 20.2% and therefore incurs identical chamber cost, so this is purely a comparison of grading quality.
| Comparison | Greedy better by | Folds won | p |
|---|---|---|---|
| vs confidence-only | +1.495 | 34/50 | 6.4e-06 |
| vs uniform | +1.406 | 32/50 | 8.7e-05 |
| vs random | +0.449 | 25/50 | 0.32 |
The most interesting result: cost-blind uncertainty targeting is worse than random
Confidence-only allocation (7.467) is the WORST policy — worse than random (6.422). Holding the cells the model is least sure about is actively counterproductive, because the most uncertain cells are frequently ones where more testing will not change the decision: they are far from a grade boundary, or hopeless in both directions.
Greedy and confidence-only use the identical uncertainty estimates and differ only in whether the cost matrix enters the ranking. The 20% improvement between them is therefore attributable to the cost matrix specifically, not to having uncertainty estimates at all. That is the cleanest evidence in this project that the decision-theoretic framing earns its place.
Greedy does not significantly beat random, and why
Reported because the build plan requires it. Greedy wins 25 of 50 folds against random, with a mean advantage of +0.449 and a per-fold standard deviation of 2.20 — the effect is real but small relative to fold noise.
The reason is granularity: at 20% of a ~25-cell test fold, the allocator is choosing 5 slots. Random selection over 5 of 25 cells frequently catches valuable cells by chance, and one lucky or unlucky draw dominates a fold. Greedy wins by +2.17 when it wins and loses by −1.67 when it loses.
This is a limitation of the evaluation, not evidence against the policy: a real batch is 1000 cells with 200 slots, where the law of large numbers makes a lucky random draw vanishingly unlikely. Demonstrating that would require a cohort this dataset cannot supply, so it is stated as a limitation rather than argued around.
6. The CONTINUE branch's value of information — what is assumed
The CONTINUE action needs to know how much better the decision becomes after more cycling. That is estimated as a one-step lookahead using the empirically measured reduction in conformal interval width from budget N to N+delta, computed on training folds only. The held-out cell's future cycles are never consulted — that would be leakage disguised as a policy.
Second-order uncertainty is NOT propagated, and this is the weakest link in
the cost model. The width ratios are themselves estimates with fold-to-fold
variance (recorded in width_std_by_budget), and that variance is reported but
not carried into the expected-cost arithmetic. Doing so would require a
distribution over predictive distributions. The consequence: CONTINUE's
expected value is stated more precisely than it is known.
The practical impact is bounded by how often CONTINUE is chosen — 33.8% of cells at N=5, falling to 0% at N=100 where there is no larger budget to advance to.
7. Limitations
- The advantage over a cost-tuned threshold is conditional on escapes costing more than ~6x an overkill (section 4).
- Greedy allocation is not significantly better than random at this fold size (section 5).
- Second-order VOI uncertainty is not propagated (section 6).
- Costs are in relative units, not currency; only ratios carry meaning.
- The sensitivity sweep is asymmetric about the measured baseline ratio (section 4).