docs/01_dataset_and_eda.md

# 01 — Dataset provenance, ingestion and exploratory analysis

[← Back to README](../README.md)

Covers **Phase 2** (ingestion, validation, cleaning, reconciliation) and
**Phase 3** (exploratory analysis and Gate 1).

Every number on this page is generated by
[`src/data/`](../src/data/) and written to [`outputs/reports/`](../outputs/reports/).
Nothing here is hand-typed.

---

## 1. Source provenance (Task 2.0)

URLs were **verified by request, not recalled**. The server's own
`Content-Disposition` header supplied each filename and `Content-Length` the
exact size; a `Range` request read the first 19 bytes of each file and
confirmed the ASCII string `MATLAB 7.3 MAT-file`. That is why `h5py` is
required and `scipy.io` cannot be used — it handles v7.0–v7.2 only.

| Batch | File | Bytes | SHA-256 (first 16) |
|---|---|---:|---|
| batch1 | `2017-05-12_batchdata_updated_struct_errorcorrect.mat` | 3,025,320,241 | `9d928ab978f0e3c7` |
| batch2 | `2017-06-30_batchdata_updated_struct_errorcorrect.mat` | 2,007,331,155 | `63ab200d09ecb237` |
| batch3 | `2018-04-12_batchdata_updated_struct_errorcorrect.mat` | 3,236,690,412 | `62c30e413b63e614` |

Total **7.70 GiB**. Full digests in [`configs/data.yaml`](../configs/data.yaml)
and [`outputs/reports/dataset_manifest.json`](../outputs/reports/dataset_manifest.json).
Landing page: `https://data.matr.io/1/projects/5c48dd2bc625d700019f3204`.

**Resume behaviour was measured, not assumed.** `data.matr.io` sends no
`Accept-Ranges` header but *does* honour `Range`, answering `HTTP 206` with a
correct `Content-Range`. [`downloader.py`](../src/data/downloader.py) implements
resume on that confirmed behaviour and refuses to append a full body to a
partial file if the server ever ignores the header.

**Attia extension not located.** The 45-cell extension (→ 169 cells) lives in a
different matr.io project; candidate file IDs returned zero-length responses.
No URL is recorded rather than a guessed one. **This project uses the 124-cell
primary corpus.**

## 2. Manufacturing interpretation, and its limits

| Dataset concept | QC analogue |
|---|---|
| One cell | One unit arriving at end-of-line QC |
| Charging policy | Process recipe |
| Batch | Production campaign |
| Cycle life to 80% nominal | The quantity a grade decision depends on |
| Cycles observed before deciding | Aging-chamber occupancy (the scarce resource) |

**Stated plainly: this is a research cycling dataset, not a factory dataset.**
Cells were cycled under controlled laboratory conditions at 30 °C, not produced
and screened on a line. There is no true process-variation signal (no electrode
coating weight, no formation-line drift, no supplier lot). The QC framing is a
*faithful analogue*, not a literal production log, and every conclusion in this
project should be read that way.

## 3. Parsing

MATLAB v7.3 files are HDF5 containers in which nearly everything is an object
reference dereferenced through the file handle. Structure was **inspected
before parsing** rather than assumed (`mat_parser.py --inspect`):

- `batch` group with `summary`, `cycles`, `Vdlin`, `barcode`, `channel_id`,
  `policy`, `policy_readable`, `cycle_life`
- Summary fields are direct datasets, shape `(1, n_cycles)`
- Cycle fields are object-reference arrays, shape `(n_cycles, 1)`
- `Qdlin`/`Tdlin`/`discharge_dQdV` are pre-interpolated onto the 1000-point
  `Vdlin` grid running 3.5 → 2.0 V

### Storage decision

Summary series are kept for **every** cycle. Within-cycle traces are kept only
to cycle **120** (`parsing.within_cycle_max_cycle`). This is not a shortcut: the
leakage rule caps every feature at budget N ≤ 100, so no feature may
legitimately read a trace beyond that, and the Phase 3 figures using
within-cycle data also stop at 100. Keeping all ~96,700 cycles of raw traces
would cost several GiB to store data the project's own rules forbid using.
Result: **964 MB** of parquet.

### The placeholder row — a real structural difference between batches

| Batch | Cells with an all-zero first row |
|---|---|
| batch1 | **46 / 46** |
| batch2 | 0 / 43 |
| batch3 | 0 / 46 |

Every summary field is exactly zero in that row. It is a structural artefact,
not a measured cycle — and it is why the source literature takes "discharge
capacity at **cycle 2**" as the first usable value.

**Consequence, and it matters for Phase 4:** raw cycle numbering does not mean
the same thing across batches. Slicing "cycles 1..N" on the raw index would give
batch-1 cells one *fewer* measured cycle at every budget — a systematic,
batch-correlated bias in exactly the variable RQ4 tests for.

**Resolution:** the placeholder row is retained (so the data stays faithful to
source and comparable with published work), and a separate `cycle_measured`
column numbers only measured cycles from 1, consistently across batches. **All
feature budgets slice on `cycle_measured`.**

## 4. The continuation join

Five batch-1 cells continued cycling as batch-2 records. Joining them is not
bookkeeping: an unjoined record has its cycle life **truncated**, which would
inject artificially short-lived cells into training and corrupt the target.

| Primary | Continuation | Primary cycles | Added | Joined | Config expected | Match |
|---|---|---:|---:|---:|---:|:--:|
| b1c0 | b2c7 | 1189 | 662 | 1851 | 662 | ✅ |
| b1c1 | b2c8 | 1178 | 981 | 2159 | 981 | ✅ |
| b1c2 | b2c9 | 1176 | 1060 | 2236 | 1060 | ✅ |
| b1c3 | b2c15 | 1225 | 208 | 1433 | 208 | ✅ |
| b1c4 | b2c16 | 1226 | 482 | 1708 | 482 | ✅ |

All five matched their configured extra-cycle counts **exactly**.

Independent confirmation: these are precisely the five cells whose batch-1
record never reaches end of life (minimum capacity 0.971–1.043 Ah against a
0.88 Ah threshold). The join is confirmed both correct and necessary — without
it those five labels would be 40–90% too short.

## 5. Cycle life is recomputed, never inherited

The naive rule "first cycle strictly below 80% of nominal" classifies **every**
cell as never reaching end of life. The reason is experimental, not numerical:
these cells were cycled *until* they reached 80% of nominal, at which point the
experiment stopped. Capacity never crosses below the threshold — the record
simply ends just above it.

Measured across all 46 batch-1 cells, the minima separate cleanly:

| Group | n | Minimum discharge capacity |
|---|---:|---|
| Reached end of life | 36 | 0.8801 – 0.8834 Ah |
| Did not (5 continuation + 5 excluded) | 10 | 0.9131 – 1.0433 Ah |

A ~0.029 Ah gap, stable at every smoothing window tested. So **two distinct
questions are answered separately**, which matters:

1. *Did* the cell reach end of life? Decided with a 2% tolerance
   (effective threshold 0.8976 Ah = 81.6% of nominal), placing it mid-gap.
2. *When*? Decided against the true 0.88 Ah threshold; if the series ends before
   strictly crossing it, the final recorded cycle **is** the end-of-life cycle.

Conflating them biased the label ~14 cycles early. Separated, the recomputed
value agrees with the file's precomputed `cycle_life` field to **exactly 1 cycle
for all 36 batch-1 EOL cells at every smoothing window** — the constant offset
being the n vs n+1 convention. Censoring classification matches expectation
exactly at all three windows, so **the result does not depend on the smoothing
choice.**

The only large disagreements are the five continuation cells (207–1059 cycles),
which is the join working as intended.

## 6. Validation — 0 FAIL

[`validator.py`](../src/data/validator.py) checks cycle-index monotonicity and
uniqueness, `cycle_measured` uniqueness and contiguity, capacity within
0 < Q ≤ 1.2 × nominal, voltage and temperature bounds, and the cycle-life
recomputation. Result: **135 cells, 0 FAIL, 144 WARN.**

### Distinguishing a parser bug from an instrument artefact

These are different failures and only one invalidates the pipeline:

- A **parser bug is systematic across cells** — misreading the layout corrupts
  every cell identically.
- An **instrument artefact is localised to a cell.**

So per-cell implausibility is a WARN, and the run FAILs only if the affected
*fraction of cells* exceeds 20%. Measured: capacity 4/135 cells (3.0%), voltage
1/135 (0.7%). Both well below threshold.

Two calibrations were corrected against measurement reality:

- **Voltage.** Protocol setpoints are 2.0/3.6 V, but the cycler samples at a
  finite rate so the last sample of a discharge overshoots — observed span
  1.919–3.614 V. The validation bound was widened to [1.85, 3.70]; the schema
  now *rejects* a validation bound tighter than the protocol range, since that
  would flag correctly parsed data as impossible.
- **Temperature.** Observed 27.3–40.4 °C, comfortably inside the original
  [20, 50] bound. Unchanged.

### Data-quality flags carried forward

Four cells are flagged and **retained** — dropping them would break
comparability with the standard 124-cell corpus, and ignoring them would be
worse:

| Cell | Finding |
|---|---|
| **b1c18** | `q_charge` ramps monotonically 1.335 → 1.968 Ah over cycles 16–40 while `q_discharge` stays flat at 1.070 Ah — a coulomb-counter reset failure. Voltage spans **0.736–6.606 V**, physically impossible for a 3.3 V LFP cell. **Not in the canonical exclusion list.** Phase 4 found a third corrupted channel on this cell: its *median* cycle duration reads 2236 minutes (37 hours) against a cohort median of 51.8 min. |
| b1c0, b2c12, b2c44 | One isolated implausible capacity cycle each. |

> **Correction (Phase 4).** The single vertical spike near 3.09 V in `fig04` was
> initially attributed to b1c18 in this document. Checking directly, it is
> **b1c41** — the only cell in the cohort whose ΔQ(V) contains a localized spike
> (peak 8.7σ from its own mean, at V ≈ 3.09). b1c18's ΔQ(V) is clean, because
> `Qdlin` derives from the discharge segment while b1c18's corruption is in the
> charge phase. b1c41's raw channels are otherwise entirely healthy.

b1c18 is the most significant: cycles 16–40 fall inside every diagnostic budget,
so it **will** affect its features. Phase 4 must handle it deliberately.

## 7. Reconciliation — exactly 124 cells

| Batch | Found | Consumed by join | Physical cells | Excluded | Used | Expected | Match |
|---|---:|---:|---:|---:|---:|---:|:--:|
| batch1 | 46 | 0 | 46 | 5 | 41 | 41 | ✅ |
| batch2 | 48 | 5 | 43 | 0 | 43 | 43 | ✅ |
| batch3 | 46 | 0 | 46 | 6 | 40 | 40 | ✅ |
| **TOTAL** | **140** | **5** | **135** | **11** | **124** | **124** | ✅ |

### Exclusion audit — every dropped cell justified individually

Rather than applying the inherited list on authority, each stated reason was
tested against the data.

| Cells | Verdict | Evidence |
|---|---|---|
| b1c8, b1c10, b1c12, b1c13, b1c22 | **SUPPORTED** | Right-censored: capacity never reaches end of life. Directly confirmed. |
| b3c23 | **SUPPORTED** | Roughest trajectory of all 46 batch-3 cells, *and* right-censored. |
| b3c2 | **SUPPORTED** | Trajectory roughness ranks 3/46, above the 90th percentile of peers. |
| b3c32 | **SUPPORTED (different reason)** | Stated noisy-channel claim **not** confirmed (roughness ranks only 15/46), but the cell is right-censored, so it cannot carry a label regardless. |
| **b3c37, b3c42, b3c43** | **NOT SUPPORTED** | Roughness ranks 20, 22, 23 of 46 — unremarkable — and each yields a valid cycle life (1389, 1641, 1045). **Excluded on the source release's authority only.** |

"Noisy channel" was tested comparatively via median absolute second difference
of the capacity trajectory, which removes level and fade slope leaving
cycle-to-cycle jitter.

**Honest limitation:** 3 of 11 exclusions could not be independently justified.

### Decision on b3c37, b3c42, b3c43 (reviewed and accepted)

These three cells **remain excluded**, and their config entries **remain
`verified: false`**. The reasoning, recorded explicitly because it is a
judgement call rather than a derivation:

- **What the evidence says.** Each yields a valid cycle life (1389, 1641, 1045)
  and each is unremarkable on the roughness metric (ranks 20, 22, 23 of 46,
  against a peer 90th percentile none of them exceeds). This project found *no*
  independent support for the stated noisy-channel reason.
- **Why they are excluded anyway.** The 124-cell cohort is the corpus the
  literature reports against, and Gate 2 (Phase 6) is a reproduction of
  Severson et al. at budget 100. Silently analysing a *different* 127-cell
  cohort would make that reproduction non-comparable and the discrepancy would
  be invisible to a reader.
- **Why the flag stays false.** Marking them `verified: true` would assert
  evidence this project does not have. The flag is the mechanism that keeps
  inherited assumptions visible, so it must not be flipped for tidiness.

The honest summary is therefore: *these three cells are excluded on the source
release's authority, not on evidence reproduced here.* Retaining them is a
legitimate alternative that would give a 127-cell corpus; anyone re-running this
work can do so by deleting the three entries from
[`configs/data.yaml`](../configs/data.yaml), and the reconciliation will follow
automatically. The exclusion is unlikely to be material — three mid-distribution
cells out of 127 — but that is an expectation, not a measurement.

**Outstanding assumptions: 3 of the original 21** (all three above). The other
18 — source provenance, all batch counts, all five continuation joins, eight
exclusions, and the final reconciliation — are closed out against the data.

## 8. GATE 1 — the canary ✅ PASSED

> *"If you cannot approximately reproduce this relationship, STOP and debug the
> parser. This is the canary for the entire pipeline."*

**log₁₀(var(ΔQ₁₀₀₋₁₀(V))) versus log₁₀(cycle life)**, n = 124:

| Quantity | This project | Severson et al. 2019 |
|---|---|---|
| Pearson r | **−0.9267** | ρ = −0.93 |
| **R²** | **0.8588** | ≈ 0.86 implied |
| Spearman | −0.9015 | — |
| Slope / intercept | −0.4157 / 1.2680 | — |

![Gate 1](../outputs/figures/fig05_dqv_variance_canary.png)

Per batch: batch1 R² = 0.901, batch2 R² = 0.665, batch3 R² = 0.647. **The
relationship is materially weaker in the later campaigns** — an early and
relevant signal for RQ4, since batch 3 is the OOD split.

### The canary earned its place

An intermediate run gave R² = 0.7186 with batch-1 R² collapsing to 0.4249. The
cause was a genuine bug: `cycle_measured` was computed per record *before* the
continuation join, so after concatenation it restarted at 1 and contained
duplicates. A lookup keyed on it resolved to the **last** match, silently
reading late-life curves as early-life ones for the five longest-lived cells.

Nothing else in the pipeline detected this — validation passed, reconciliation
was exact, cycle lives were correct. Only the canary caught it. The join now
recomputes `cycle_measured` across the merged series, the validator asserts
uniqueness and contiguity, and the lookup keeps the first occurrence as
defence in depth.

## 9. Exploratory analysis

All eleven figures regenerate headlessly via `python -m src.data.eda` into
[`outputs/figures/`](../outputs/figures/).

### Realised grade balance — recomputed, not assumed

| Grade | Boundary | Cells | Share | Tier |
|---|---:|---:|---:|---|
| A | ≥ 1200 | **11** | 8.9% | automotive traction pack |
| B | ≥ 550 | 70 | 56.5% | stationary ESS / second tier |
| C | ≥ 0 | 43 | 34.7% | scrap or non-critical |

Cycle life spans **150 – 2236** (median 736), consistent with the published
150–2300 range.

> ⚠️ **Grade A holds only 11 cells.** `configs/grading.yaml` anticipated this
> risk explicitly. Eleven examples is thin for a three-class classifier and very
> thin for per-fold estimates under 5-fold grouped CV (~2 cells per fold).
> **This is a decision for Phase 4/6 review**, not something to be worked around
> silently. Options: lower the A boundary, treat grading as ordinal regression
> from predicted cycle life, or report grade-A metrics with explicit caveats.

### Batch 3 is genuinely shifted

Two-sample KS between batches 1–2 and batch 3: **D = 0.651, p = 1.7 × 10⁻¹¹**.
Median cycle life 547.5 vs 963.5. Batch 3 is a real distribution shift, so RQ4
tests something meaningful rather than a relabelling.

### Feature signal (Spearman vs log cycle life)

| Feature | ρ |
|---|---:|
| log var ΔQ(V) | **−0.902** |
| min ΔQ(V) | +0.868 |
| internal resistance at cycle N | −0.594 |
| minimum internal resistance | −0.588 |
| maximum temperature | −0.252 |
| mean temperature | +0.222 |

ΔQ(V)-derived features dominate, consistent with the literature. **Thermal
signal is weak** (|ρ| ≈ 0.22–0.25) in this corpus — worth stating now, since
Phase 9 hypothesis H3 predicts monotonic thermal SHAP attributions. In a
30 °C-controlled chamber there is little thermal variation to exploit, so a
weak result there would be a property of the dataset, not a modelling failure.

### Process-recipe space

**69 unique charging policies** across 124 cells, step-1 C-rates spanning
1.0–8.0 C. Recipe families are heavily confounded with cycle life by design,
which is exactly why Group F must be ablated for RQ4.

### Why a naive rule is not enough — the motivation

Single-feature threshold on log var ΔQ(V), at **matched yield** (shipping
exactly the 81 truly capable cells' worth), tuned **in-sample**:

| Budget | Escapes | Escape rate | Overkill rate |
|---:|---:|---:|---:|
| 5 | 14 | 17.3% | 17.3% |
| 10 | 22 | 27.2% | 27.2% |
| 20 | 23 | 28.4% | 28.4% |
| 50 | 9 | 11.1% | 11.1% |
| 100 | **7** | **8.6%** | 8.6% |

Even at budget 100, tuned in-sample with a perfectly calibrated yield, a naive
rule leaves an **8.6% escape rate** — and it offers no way to *bound* that rate
at a chosen level. This is the gap the conformal risk control of Phase 7 and the
cost-optimal triage of Phase 8 exist to close.

Two honest caveats: these are **in-sample** numbers, so they are an optimistic
bound rather than held-out performance; and the non-monotonicity at budgets
10–20 reflects both a changing ΔQ baseline cycle and genuine small-sample noise
at n = 124.

---

## Artifacts

| File | Contents |
|---|---|
| `outputs/reports/dataset_manifest.json` | URLs, sizes, SHA-256 digests |
| `outputs/reports/reconciliation.csv` | The found/joined/excluded/used table |
| `outputs/reports/exclusion_audit.json` | Per-cell exclusion verdicts and evidence |
| `outputs/reports/validation_report.csv` | Per-cell validation record |
| `outputs/reports/validation_issues.json` | Every FAIL and WARN |
| `outputs/reports/eda_results.json` | All Phase 3 quantitative results |
| `outputs/reports/eda_delta_q_statistics.csv` | Per-cell ΔQ(V) statistics |
| `outputs/reports/cohort.csv` | Final 124-cell labelled cohort |
| `data/processed/cohort.parquet` | Cohort with labels, grades and quality flags |

Reproduce end to end:

```bash
python -m src.data.downloader   # 7.70 GiB, resumable, checksum-verified
python -m src.data.mat_parser   # -> data/interim/cells/
python -m src.data.validator    # must report 0 FAIL
python -m src.data.cleaner      # reconciliation + cohort
python -m src.data.eda          # 11 figures + Gate 1
```

---

[← README](../README.md) · [Feature engineering →](02_feature_engineering.md)