docs/ENGINEERING_PATTERNS.md

Engineering patterns found while building this project

← Back to README

Patterns that generalise beyond this codebase, recorded because each was found the expensive way.

---

Pattern 1 — The acceptance check that cannot fail on the defect it targets

Reviewer shorthand for this project: R6. Four instances, in four subsystems.

The shape

An acceptance criterion is written in prose, then implemented as a check on some observable consequence of that criterion. The failure arrives, the check passes anyway, and it passes because the check is compatible with the failure mode — not because the check was sloppy, but because the property it observes is genuinely present in both the correct and the broken state.

The defining question is not "is this check correct?" but:

What failure mode is this check structurally incapable of detecting?

A check that cannot answer that question is not yet an acceptance check.

It happened four times here

Four independent instances, in four different subsystems, none of them careless:

---

Case 1 — "All figures regenerate headlessly"

Criterion	All figures regenerate headlessly
Check written	The expected PNG files appear in outputs/figures/
Why it passed	An interactive backend also writes the files — it just opens a blocking GUI window first
Blind spot	Everything the criterion was actually about
How it surfaced	A user closing windows by hand during a run

The check observed a consequence (files exist) that the failure mode does not disturb. Worse, the failure is invisible on the machine that introduces it if that machine has a display, and appears only in headless CI.

Fixed by checking the property: assert the active backend is non-interactive, assert a cold interpreter importing the plotting module stays headless, assert no module calls show(), assert no module imports pyplot outside the one guarded entry point, and assert figures do not leak.

---

Case 2 — Gate 2's set-size assertion

Criterion	Reproduce the published train / primary-test / secondary-test design
Check written	assert len(train) == 41 and len(primary) == 43 and len(secondary) == 40
Why it passed	Batch 1 contains exactly 41 cells by coincidence, so the wrong split (train = batch1, test = batch2) produces identical set sizes to the correct one (train/primary interleaved over batches 1+2)
Blind spot	Which cells are in each set — the only thing that actually differs
How it surfaced	Gate 2 failed at 25.84% against a 9.1% benchmark

This is the purest instance. The assertion was not weak — 41/43/40 is a precise and non-trivial fact — but the numbers were a coincidence that both the right and wrong answers satisfy. A check can be simultaneously specific and uninformative.

What actually exposed it was not a check but a diagnostic inconsistency: the published primary-test error (9.1%) is lower than the secondary-test error (15.6%), which is only possible if the primary test is in-distribution. The observed ordering was inverted. A relationship between numbers caught what an assertion on the numbers could not.

Fixed by checking composition, not cardinality: assert that train and primary test each span batches 1 and 2, and that secondary is batch 3 alone.

---

Case 3 — The conformal quantile off-by-one

Criterion	Conformal intervals achieve their nominal coverage
Check written	Empirical coverage ≥ nominal, across folds
Why it passed	The bug made intervals too wide. Coverage was higher than nominal, so the check passed more comfortably than it would have with correct code
Blind spot	Any conservative error whatsoever — by construction
How it surfaced	Testing the quantile definition against a hand-computed order statistic

The most instructive of the three. np.quantile(scores, k/n, method="higher") maps a quantile onto index positions [0, n-1] and returns the (k+1)-th smallest score where the conformal definition requires the k-th.

The error direction is what made it invisible: a coverage check can only fail on intervals that are too narrow. It is structurally incapable of failing on intervals that are too wide. Adding more coverage checks, more folds, or more α levels would never have found it.

Fixing it moved nominal-90% coverage from 92.8% → 90.3% (CQR) and 95.2% → 90.8% (split conformal) — from over-covering to essentially exact — and narrowed intervals by 4.8% and 24.9%.

Fixed by testing the definition, not its consequence: a unit test asserting the quantile equals a hand-computed order statistic, plus a monotonicity test in α.

---

Case 4 — The audit sheet that displayed z-scores

Criterion	The per-cell audit sheet explains a decision to a process engineer
Check written	Sheets generate without error; every driver named; costs consistent with the stated decision; probabilities sum to 1
Why it passed	Every one of those things was true. The sheet was generated correctly — in the wrong units, for the wrong reader
Blind spot	Whether the content means anything to its audience
How it surfaced	Reading one

The model pipeline ends in a StandardScaler, so feature values pulled from it are standardised. The sheet dutifully reported "how much the discharge curve's shape changed: −0.8224" — a z-score, presented to the one reader in the whole system who cannot interpret one.

This is the same family as the others but with a distinct flavour: the previous three checked a necessary condition and treated it as sufficient. This one checked structural correctness and treated it as semantic correctness. The artifact was internally consistent, complete, and useless.

Fixed by quoting the physical value against a cohort reference: "this cell: 0.000433 (typical cell: 0.00017 — this one is above typical)".

And the residual limitation is stated rather than closed: no test here can establish that an explanation is useful, because that requires a human reader. The tests check completeness and internal consistency, which remains necessary and not sufficient. Case 4 is the one instance in this list where the gap could not be fully closed by a better check.

---

What they have in common

Case	Check observed	Failure mode it could not see
Figures	Files on disk	A backend that also writes files
Gate 2	Set sizes	A different set composition with the same sizes
Conformal	Coverage ≥ nominal	Any error in the conservative direction
Audit sheet	Structural correctness	Correct content in the wrong units for the reader

In the first three the check verified a necessary condition and treated it as sufficient. The fourth is a variant: it verified structural correctness and treated it as semantic correctness. In every case the gap between the two is where the defect lived.

The practice adopted from this

Every acceptance check written in this project from Phase 8 onward states, in its docstring, what failure mode it cannot detect. Not as documentation — as a forcing function. Writing that sentence requires enumerating the ways the criterion could be satisfied while the underlying property is false, which is the analysis that would have caught the first three of these before they shipped. Case 4 shows the limit of the practice: a check can be structurally sound and still miss a semantic failure that only a human reader will see.

Concretely, the questions that produce it:

1. Could a broken implementation satisfy this check? Describe one.
2. Is my check on a necessary condition or a sufficient one?
3. If the error were in the conservative direction, would this still fail?
4. Am I checking the property, or an observable consequence of it?
5. Is there a relationship between quantities that would break under the failure, even where an assertion on any single quantity would not? (This is what caught Case 2.)

---

Pattern 2 — A result that contradicts an earlier measurement is a bug signal

Twice in this project a defect was found not by a test but by an output disagreeing with something already measured:

• StabilitySelector index misalignment (Phase 4). Selection returned an all-thermal feature set. Phase 3 had already measured thermal as the weakest signal group (|ρ| ≈ 0.22) and ΔQ(V) variance as the strongest (ρ = −0.90). The contradiction was the only symptom: pandas Series.corr aligns on index, and the bootstrap subset carried original row labels against a fresh RangeIndex, scoring dq_var at 0.023 against a true 0.908. No error was raised and the output looked plausible in isolation.

• cycle_measured duplication after the continuation join (Phase 3). Gate 1 fell to R² = 0.7186 with batch 1 collapsing to 0.4249, against an already-measured 0.8588. Validation passed, reconciliation was exact, and cycle lives were correct — only the canary caught it.

The practice: when a new number contradicts an established one, treat the new number as the suspect until proven otherwise, and diagnose before explaining. Both of these were initially tempting to rationalise.

---

← Back to README