docs/05_decision_framework.md

# 05 - The QC decision framework (RQ1, RQ3)

[Back to 04: Uncertainty](04_uncertainty.md) | [README](../README.md)

Covers **Phase 8**, the contribution this project exists for. Everything before
it is infrastructure.

---

## 1. The headline, including what it costs

**The triage policy decisively beats three of five baselines and does NOT beat
the fourth at the project's own assumed cost ratio.** Both halves are reported,
because the second is the one a reviewer should press on.

| Policy | Cost per cell | Escape rate | Chamber cycles/cell |
|---|---|---|---|
| **TRIAGE (N=5)** | **7.035 ± 3.626** | 0.003 ± 0.014 | 6.687 ± 0.585 |
| cost_tuned_threshold | 6.525 ± 1.585 | 0.024 ± 0.035 | 100.000 ± 0.000 |
| naive_threshold | 7.418 ± 2.498 | 0.045 ± 0.046 | 100.000 ± 0.000 |
| aql_sampling | 99.715 ± 96.460 | 0.905 ± 0.042 | 63.190 ± 20.207 |
| cycle_to_eol | 40.078 ± 3.238 | 0.000 ± 0.000 | 801.557 ± 64.754 |
| no_screening | 206.413 ± 26.715 | 0.911 ± 0.046 | 0.000 ± 0.000 |

Paired across the same 50 outer folds:

| Comparison | Triage better by | Folds won | p |
|---|---:|---:|---:|
| vs AQL acceptance sampling | **+92.82** | 50/50 | 1.8e-15 |
| vs cycle-everything-to-EOL | **+33.19** | 50/50 | 1.8e-15 |
| vs no screening | **+199.52** | 50/50 | 1.8e-15 |
| vs naive threshold | +0.53 | 23/50 | 0.71 |
| **vs cost-tuned threshold** | **−0.37** | **14/50** | **0.055** |

**Beating AQL is the industrially meaningful claim** and it holds by a factor of
roughly 14x. AQL makes ONE decision for a whole lot, so with 34.7% of cells
below the warranty target it either scraps good lots or ships bad ones; its
per-fold cost variance (±96) is larger than the triage policy's entire cost.

**The cost-tuned threshold is not beaten at the measured ratio.** Section 4
explains why, and when that reverses.

## 2. The QC scorecard

Mean ± std across 50 outer folds, budget by budget.

| Metric | N=5 | N=10 | N=20 | N=50 | N=100 |
|---|---|---|---|---|---|
| `escape_rate` | 0.0029 ± 0.0144 | 0.0000 ± 0.0000 | 0.0011 ± 0.0079 | 0.0013 ± 0.0088 | 0.0047 ± 0.0191 |
| `overkill_rate` | 0.5789 ± 0.2814 | 0.5394 ± 0.2742 | 0.4956 ± 0.2624 | 0.3560 ± 0.2054 | 0.1940 ± 0.1118 |
| `yield` | 0.3584 ± 0.1979 | 0.3866 ± 0.1825 | 0.4173 ± 0.1837 | 0.5004 ± 0.1459 | 0.5994 ± 0.0992 |
| `chamber_cycles_per_cell` | 6.6873 ± 0.5847 | 12.4423 ± 1.3291 | 26.7910 ± 4.2286 | 55.1483 ± 5.0605 | 100.0000 ± 0.0000 |
| `cost_per_cell` | 7.0354 ± 3.6255 | 6.6293 ± 3.4751 | 6.8169 ± 3.5053 | 6.5599 ± 2.5884 | 6.8904 ± 1.3249 |
| `continue_rate` | 0.3375 ± 0.1169 | 0.2442 ± 0.1329 | 0.2264 ± 0.1410 | 0.1030 ± 0.1012 | 0.0000 ± 0.0000 |

The escape rate stays at or below 0.005 at every budget while overkill falls
from 0.58 to 0.19 -- the policy is buying yield with extra cycles while holding
escapes near zero, which is the intended behaviour under an asymmetric cost
matrix.

## 3. RQ1 -- the diagnostic budget frontier

**The frontier is flat.** Every budget from 5 to 100 cycles is statistically
indistinguishable in cost per cell: 5, 10, 20, 50, 100.

- Cheapest budget: **N=50**
- **Economic knee: N=5**, by the stated rule -- *the smallest budget
  whose bootstrap 95% CI for cost per cell overlaps that of the cheapest budget*

The rule is written down rather than eyeballed because "the knee of the curve"
otherwise invites choosing whichever point supports the conclusion. It is also
capable of returning the smallest budget, which is exactly what happened, and
that is a finding rather than a failure to find a knee.

### Chamber time released

| | |
|---|---|
| Chamber cycles per cell at the knee | **6.69** |
| Chamber cycles per cell, cycling to end of life | 801.6 |
| **Chamber time released** | **99.17%** |

This is the number a plant manager acts on. It is large because cycling to end
of life takes ~800 cycles per cell on this corpus while a decision can be made
at 5.

**The honest caveat**: the flatness means the extra cycles buy little *in cost
terms under this cost matrix*, not that early prediction is as accurate as late
prediction -- Phase 6 showed RMSE improving steadily with budget
(0.080 → 0.058). The decision is insensitive where the prediction is not,
because the cost matrix is dominated by a few expensive misgrades rather than by
average accuracy.

## 4. RQ3 -- when is the triage machinery worth its complexity?

The comparison that matters is against the **cost-tuned threshold**: the same
model, the same features, the same folds, an optimally tuned decision margin, and
no CONTINUE action. It isolates the value of the sequential, uncertainty-aware
machinery specifically.

At the reference budget the two differ like this:

| | Triage | Cost-tuned threshold |
|---|---|---|
| Escape rate | **0.0047** | 0.0236 |
| Overkill rate | 0.1940 | **0.0532** |
| Cost per cell | 6.890 | **6.525** |

The triage policy buys a **5x lower escape rate** by accepting **3.6x more
overkill**. Whether that is a good trade depends entirely on the cost ratio --
which is what the sensitivity sweep is for.

| Escape:overkill | Triage cost | Cost-tuned cost | Triage advantage | Triage escape | Tuned escape |
|---:|---:|---:|---:|---:|---:|
| 2:1 | 6.433 | 5.698 | _-0.735_ | 0.1040 | 0.0495 |
| 5:1 | 6.213 | 6.093 | _-0.119_ | 0.0272 | 0.0303 |
| 8:1 | 6.301 | 6.488 | **+0.187** | 0.0067 | 0.0303 |
| 11:1 | 6.726 | 6.883 | **+0.157** | 0.0000 | 0.0303 |
| 14:1 | 6.998 | 7.278 | **+0.280** | 0.0000 | 0.0303 |
| 15:1 | 6.998 | 7.409 | **+0.411** | 0.0000 | 0.0303 |
| 20:1 | 7.078 | 8.067 | **+0.989** | 0.0000 | 0.0303 |
| 50:1 | 7.318 | 12.015 | **+4.697** | 0.0000 | 0.0303 |
| 150:1 | 7.318 | 25.173 | **+17.855** | 0.0000 | 0.0303 |
| 500:1 | 7.401 | 71.228 | **+63.827** | 0.0000 | 0.0303 |

**The crossover sits between 5:1 and 8:1**, below the measured baseline ratio of
10.9:1. So at the project's own cost assumptions the triage policy is
favourable, but only modestly; its advantage grows without bound as escapes
become more expensive, and it is *unfavourable* if escapes cost less than about
6x an overkill.

The mechanism is visible in the escape columns: **the triage policy's escape
rate falls to zero as escapes get more expensive (0.104 → 0.027 → 0.007 →
0.000), while the cost-tuned threshold stays pinned at 0.0303 regardless.** A
static margin has no uncertainty representation to adapt with; it can shift
where it cuts but cannot become more cautious about the cells it is unsure of.

### Which conclusions hold across the whole range, and which do not

| Conclusion | Holds across 2:1 – 500:1? |
|---|---|
| Triage beats AQL, cycle-to-EOL and no-screening | **Yes**, at every ratio |
| Triage beats a cost-tuned threshold | **No** — only above ~6:1 |
| Triage escape rate ≤ cost-tuned escape rate | **Yes**, at every ratio |
| The budget frontier is flat | **Yes** |

### The sweep is asymmetric, and it matters here

The configured sweep is [5, 15, 50, 150, 500] against a measured baseline ratio
of 10.9. **Only one configured point sits below the baseline** — and
that one point is precisely where the triage policy loses. The configured sweep
therefore samples the region where the conclusion *fails* far more sparsely than
the region where it holds.

That is why a supplementary fine sweep at [2, 5, 8, 11, 14, 20] was run to
locate the crossover. Reporting only the configured sweep would have shown four
favourable points and one unfavourable one, which overstates how robust the
conclusion is.

## 5. Phase 8D -- capacity-constrained allocation

Real QC cannot hold everything. With 10.9:1 costs, 20% of a batch
holdable (200 slots per 1000 cells), deciding at N=20 and advancing held cells
to N=100:

| Allocation policy | Cost per cell | Grading cost | Escape rate |
|---|---|---|---|
| **greedy_voi_per_cycle** | **5.973 ± 1.965** | 4.166 | 0.012 ± 0.029 |
| random | 6.422 ± 1.722 | 4.616 | 0.022 ± 0.038 |
| uniform | 7.378 ± 1.141 | 5.572 | 0.014 ± 0.032 |
| confidence_only | 7.467 ± 1.490 | 5.661 | 0.019 ± 0.040 |

Every policy holds the same 20.2% and therefore incurs **identical chamber
cost**, so this is purely a comparison of grading quality.

| Comparison | Greedy better by | Folds won | p |
|---|---:|---:|---:|
| vs confidence-only | **+1.495** | 34/50 | **6.4e-06** |
| vs uniform | **+1.406** | 32/50 | **8.7e-05** |
| vs random | +0.449 | 25/50 | 0.32 |

### The most interesting result: cost-blind uncertainty targeting is worse than random

**Confidence-only allocation (7.467) is the WORST policy — worse than random
(6.422).** Holding the cells the model is least sure about is actively
counterproductive, because the most uncertain cells are frequently ones where
more testing will not change the decision: they are far from a grade boundary,
or hopeless in both directions.

Greedy and confidence-only use the *identical* uncertainty estimates and differ
only in whether the cost matrix enters the ranking. **The 20% improvement
between them is therefore attributable to the cost matrix specifically, not to
having uncertainty estimates at all.** That is the cleanest evidence in this
project that the decision-theoretic framing earns its place.

### Greedy does not significantly beat random, and why

Reported because the build plan requires it. Greedy wins 25 of 50 folds against
random, with a mean advantage of +0.449 and a per-fold standard deviation of
2.20 — the effect is real but small relative to fold noise.

The reason is granularity: at 20% of a ~25-cell test fold, the allocator is
choosing **5 slots**. Random selection over 5 of 25 cells frequently catches
valuable cells by chance, and one lucky or unlucky draw dominates a fold. Greedy
wins by +2.17 when it wins and loses by −1.67 when it loses.

**This is a limitation of the evaluation, not evidence against the policy**: a
real batch is 1000 cells with 200 slots, where the law of large numbers makes a
lucky random draw vanishingly unlikely. Demonstrating that would require a
cohort this dataset cannot supply, so it is stated as a limitation rather than
argued around.

## 6. The CONTINUE branch's value of information — what is assumed

The CONTINUE action needs to know how much better the decision becomes after
more cycling. That is estimated as a one-step lookahead using the **empirically
measured** reduction in conformal interval width from budget N to N+delta,
computed **on training folds only**. The held-out cell's future cycles are never
consulted — that would be leakage disguised as a policy.

**Second-order uncertainty is NOT propagated, and this is the weakest link in
the cost model.** The width ratios are themselves estimates with fold-to-fold
variance (recorded in `width_std_by_budget`), and that variance is *reported* but
not carried into the expected-cost arithmetic. Doing so would require a
distribution over predictive distributions. The consequence: **CONTINUE's
expected value is stated more precisely than it is known.**

The practical impact is bounded by how often CONTINUE is chosen — 33.8% of cells
at N=5, falling to 0% at N=100 where there is no larger budget to advance to.

## 7. Limitations

- **The advantage over a cost-tuned threshold is conditional** on escapes
  costing more than ~6x an overkill (section 4).
- **Greedy allocation is not significantly better than random** at this fold
  size (section 5).
- **Second-order VOI uncertainty is not propagated** (section 6).
- Costs are in **relative units**, not currency; only ratios carry meaning.
- The sensitivity sweep is asymmetric about the measured baseline ratio
  (section 4).

---

[← Uncertainty and risk control](04_uncertainty.md) · [README](../README.md) · [Explainability →](06_explainability.md)