docs/01_dataset_and_eda.md

01 — Dataset provenance, ingestion and exploratory analysis

← Back to README

Covers Phase 2 (ingestion, validation, cleaning, reconciliation) and Phase 3 (exploratory analysis and Gate 1).

Every number on this page is generated by src/data/ and written to outputs/reports/. Nothing here is hand-typed.

---

1. Source provenance (Task 2.0)

URLs were verified by request, not recalled. The server's own Content-Disposition header supplied each filename and Content-Length the exact size; a Range request read the first 19 bytes of each file and confirmed the ASCII string MATLAB 7.3 MAT-file. That is why h5py is required and scipy.io cannot be used — it handles v7.0–v7.2 only.

Batch	File	Bytes	SHA-256 (first 16)
batch1	2017-05-12_batchdata_updated_struct_errorcorrect.mat	3,025,320,241	9d928ab978f0e3c7
batch2	2017-06-30_batchdata_updated_struct_errorcorrect.mat	2,007,331,155	63ab200d09ecb237
batch3	2018-04-12_batchdata_updated_struct_errorcorrect.mat	3,236,690,412	62c30e413b63e614

Total 7.70 GiB. Full digests in configs/data.yaml and outputs/reports/dataset_manifest.json. Landing page: https://data.matr.io/1/projects/5c48dd2bc625d700019f3204.

Resume behaviour was measured, not assumed. data.matr.io sends no Accept-Ranges header but does honour Range, answering HTTP 206 with a correct Content-Range. downloader.py implements resume on that confirmed behaviour and refuses to append a full body to a partial file if the server ever ignores the header.

Attia extension not located. The 45-cell extension (→ 169 cells) lives in a different matr.io project; candidate file IDs returned zero-length responses. No URL is recorded rather than a guessed one. This project uses the 124-cell primary corpus.

2. Manufacturing interpretation, and its limits

Dataset concept	QC analogue
One cell	One unit arriving at end-of-line QC
Charging policy	Process recipe
Batch	Production campaign
Cycle life to 80% nominal	The quantity a grade decision depends on
Cycles observed before deciding	Aging-chamber occupancy (the scarce resource)

Stated plainly: this is a research cycling dataset, not a factory dataset. Cells were cycled under controlled laboratory conditions at 30 °C, not produced and screened on a line. There is no true process-variation signal (no electrode coating weight, no formation-line drift, no supplier lot). The QC framing is a faithful analogue, not a literal production log, and every conclusion in this project should be read that way.

3. Parsing

MATLAB v7.3 files are HDF5 containers in which nearly everything is an object reference dereferenced through the file handle. Structure was inspected before parsing rather than assumed (mat_parser.py --inspect):

• batch group with summary, cycles, Vdlin, barcode, channel_id, policy, policy_readable, cycle_life
• Summary fields are direct datasets, shape (1, n_cycles)
• Cycle fields are object-reference arrays, shape (n_cycles, 1)
• Qdlin/Tdlin/discharge_dQdV are pre-interpolated onto the 1000-point Vdlin grid running 3.5 → 2.0 V

Storage decision

Summary series are kept for every cycle. Within-cycle traces are kept only to cycle 120 (parsing.within_cycle_max_cycle). This is not a shortcut: the leakage rule caps every feature at budget N ≤ 100, so no feature may legitimately read a trace beyond that, and the Phase 3 figures using within-cycle data also stop at 100. Keeping all ~96,700 cycles of raw traces would cost several GiB to store data the project's own rules forbid using. Result: 964 MB of parquet.

The placeholder row — a real structural difference between batches

Batch	Cells with an all-zero first row
batch1	46 / 46
batch2	0 / 43
batch3	0 / 46

Every summary field is exactly zero in that row. It is a structural artefact, not a measured cycle — and it is why the source literature takes "discharge capacity at cycle 2" as the first usable value.

Consequence, and it matters for Phase 4: raw cycle numbering does not mean the same thing across batches. Slicing "cycles 1..N" on the raw index would give batch-1 cells one fewer measured cycle at every budget — a systematic, batch-correlated bias in exactly the variable RQ4 tests for.

Resolution: the placeholder row is retained (so the data stays faithful to source and comparable with published work), and a separate cycle_measured column numbers only measured cycles from 1, consistently across batches. All feature budgets slice on cycle_measured.

4. The continuation join

Five batch-1 cells continued cycling as batch-2 records. Joining them is not bookkeeping: an unjoined record has its cycle life truncated, which would inject artificially short-lived cells into training and corrupt the target.

Primary	Continuation	Primary cycles	Added	Joined	Config expected	Match
b1c0	b2c7	1189	662	1851	662	✅
b1c1	b2c8	1178	981	2159	981	✅
b1c2	b2c9	1176	1060	2236	1060	✅
b1c3	b2c15	1225	208	1433	208	✅
b1c4	b2c16	1226	482	1708	482	✅

All five matched their configured extra-cycle counts exactly.

Independent confirmation: these are precisely the five cells whose batch-1 record never reaches end of life (minimum capacity 0.971–1.043 Ah against a 0.88 Ah threshold). The join is confirmed both correct and necessary — without it those five labels would be 40–90% too short.

5. Cycle life is recomputed, never inherited

The naive rule "first cycle strictly below 80% of nominal" classifies every cell as never reaching end of life. The reason is experimental, not numerical: these cells were cycled until they reached 80% of nominal, at which point the experiment stopped. Capacity never crosses below the threshold — the record simply ends just above it.

Measured across all 46 batch-1 cells, the minima separate cleanly:

Group	n	Minimum discharge capacity
Reached end of life	36	0.8801 – 0.8834 Ah
Did not (5 continuation + 5 excluded)	10	0.9131 – 1.0433 Ah

A ~0.029 Ah gap, stable at every smoothing window tested. So two distinct questions are answered separately, which matters:

1. Did the cell reach end of life? Decided with a 2% tolerance (effective threshold 0.8976 Ah = 81.6% of nominal), placing it mid-gap.
2. When? Decided against the true 0.88 Ah threshold; if the series ends before strictly crossing it, the final recorded cycle is the end-of-life cycle.

Conflating them biased the label ~14 cycles early. Separated, the recomputed value agrees with the file's precomputed cycle_life field to exactly 1 cycle for all 36 batch-1 EOL cells at every smoothing window — the constant offset being the n vs n+1 convention. Censoring classification matches expectation exactly at all three windows, so the result does not depend on the smoothing choice.

The only large disagreements are the five continuation cells (207–1059 cycles), which is the join working as intended.

6. Validation — 0 FAIL

validator.py checks cycle-index monotonicity and uniqueness, cycle_measured uniqueness and contiguity, capacity within 0 < Q ≤ 1.2 × nominal, voltage and temperature bounds, and the cycle-life recomputation. Result: 135 cells, 0 FAIL, 144 WARN.

Distinguishing a parser bug from an instrument artefact

These are different failures and only one invalidates the pipeline:

• A parser bug is systematic across cells — misreading the layout corrupts every cell identically.
• An instrument artefact is localised to a cell.

So per-cell implausibility is a WARN, and the run FAILs only if the affected fraction of cells exceeds 20%. Measured: capacity 4/135 cells (3.0%), voltage 1/135 (0.7%). Both well below threshold.

Two calibrations were corrected against measurement reality:

• Voltage. Protocol setpoints are 2.0/3.6 V, but the cycler samples at a finite rate so the last sample of a discharge overshoots — observed span 1.919–3.614 V. The validation bound was widened to [1.85, 3.70]; the schema now rejects a validation bound tighter than the protocol range, since that would flag correctly parsed data as impossible.
• Temperature. Observed 27.3–40.4 °C, comfortably inside the original [20, 50] bound. Unchanged.

Data-quality flags carried forward

Four cells are flagged and retained — dropping them would break comparability with the standard 124-cell corpus, and ignoring them would be worse:

Cell	Finding
b1c18	q_charge ramps monotonically 1.335 → 1.968 Ah over cycles 16–40 while q_discharge stays flat at 1.070 Ah — a coulomb-counter reset failure. Voltage spans 0.736–6.606 V, physically impossible for a 3.3 V LFP cell. Not in the canonical exclusion list. Phase 4 found a third corrupted channel on this cell: its median cycle duration reads 2236 minutes (37 hours) against a cohort median of 51.8 min.
b1c0, b2c12, b2c44	One isolated implausible capacity cycle each.

Correction (Phase 4). The single vertical spike near 3.09 V in fig04 was initially attributed to b1c18 in this document. Checking directly, it is b1c41 — the only cell in the cohort whose ΔQ(V) contains a localized spike (peak 8.7σ from its own mean, at V ≈ 3.09). b1c18's ΔQ(V) is clean, because Qdlin derives from the discharge segment while b1c18's corruption is in the charge phase. b1c41's raw channels are otherwise entirely healthy.

b1c18 is the most significant: cycles 16–40 fall inside every diagnostic budget, so it will affect its features. Phase 4 must handle it deliberately.

7. Reconciliation — exactly 124 cells

Batch	Found	Consumed by join	Physical cells	Excluded	Used	Expected	Match
batch1	46	0	46	5	41	41	✅
batch2	48	5	43	0	43	43	✅
batch3	46	0	46	6	40	40	✅
TOTAL	140	5	135	11	124	124	✅

Exclusion audit — every dropped cell justified individually

Rather than applying the inherited list on authority, each stated reason was tested against the data.

Cells	Verdict	Evidence
b1c8, b1c10, b1c12, b1c13, b1c22	SUPPORTED	Right-censored: capacity never reaches end of life. Directly confirmed.
b3c23	SUPPORTED	Roughest trajectory of all 46 batch-3 cells, and right-censored.
b3c2	SUPPORTED	Trajectory roughness ranks 3/46, above the 90th percentile of peers.
b3c32	SUPPORTED (different reason)	Stated noisy-channel claim not confirmed (roughness ranks only 15/46), but the cell is right-censored, so it cannot carry a label regardless.
b3c37, b3c42, b3c43	NOT SUPPORTED	Roughness ranks 20, 22, 23 of 46 — unremarkable — and each yields a valid cycle life (1389, 1641, 1045). Excluded on the source release's authority only.

"Noisy channel" was tested comparatively via median absolute second difference of the capacity trajectory, which removes level and fade slope leaving cycle-to-cycle jitter.

Honest limitation: 3 of 11 exclusions could not be independently justified.

Decision on b3c37, b3c42, b3c43 (reviewed and accepted)

These three cells remain excluded, and their config entries remain verified: false. The reasoning, recorded explicitly because it is a judgement call rather than a derivation:

• What the evidence says. Each yields a valid cycle life (1389, 1641, 1045) and each is unremarkable on the roughness metric (ranks 20, 22, 23 of 46, against a peer 90th percentile none of them exceeds). This project found no independent support for the stated noisy-channel reason.
• Why they are excluded anyway. The 124-cell cohort is the corpus the literature reports against, and Gate 2 (Phase 6) is a reproduction of Severson et al. at budget 100. Silently analysing a different 127-cell cohort would make that reproduction non-comparable and the discrepancy would be invisible to a reader.
• Why the flag stays false. Marking them verified: true would assert evidence this project does not have. The flag is the mechanism that keeps inherited assumptions visible, so it must not be flipped for tidiness.

The honest summary is therefore: these three cells are excluded on the source release's authority, not on evidence reproduced here. Retaining them is a legitimate alternative that would give a 127-cell corpus; anyone re-running this work can do so by deleting the three entries from configs/data.yaml, and the reconciliation will follow automatically. The exclusion is unlikely to be material — three mid-distribution cells out of 127 — but that is an expectation, not a measurement.

Outstanding assumptions: 3 of the original 21 (all three above). The other 18 — source provenance, all batch counts, all five continuation joins, eight exclusions, and the final reconciliation — are closed out against the data.

8. GATE 1 — the canary ✅ PASSED

"If you cannot approximately reproduce this relationship, STOP and debug the parser. This is the canary for the entire pipeline."

log₁₀(var(ΔQ₁₀₀₋₁₀(V))) versus log₁₀(cycle life), n = 124:

Quantity	This project	Severson et al. 2019
Pearson r	−0.9267	ρ = −0.93
R²	0.8588	≈ 0.86 implied
Spearman	−0.9015	—
Slope / intercept	−0.4157 / 1.2680	—

Per batch: batch1 R² = 0.901, batch2 R² = 0.665, batch3 R² = 0.647. The relationship is materially weaker in the later campaigns — an early and relevant signal for RQ4, since batch 3 is the OOD split.

The canary earned its place

An intermediate run gave R² = 0.7186 with batch-1 R² collapsing to 0.4249. The cause was a genuine bug: cycle_measured was computed per record before the continuation join, so after concatenation it restarted at 1 and contained duplicates. A lookup keyed on it resolved to the last match, silently reading late-life curves as early-life ones for the five longest-lived cells.

Nothing else in the pipeline detected this — validation passed, reconciliation was exact, cycle lives were correct. Only the canary caught it. The join now recomputes cycle_measured across the merged series, the validator asserts uniqueness and contiguity, and the lookup keeps the first occurrence as defence in depth.

9. Exploratory analysis

All eleven figures regenerate headlessly via python -m src.data.eda into outputs/figures/.

Realised grade balance — recomputed, not assumed

Grade	Boundary	Cells	Share	Tier
A	≥ 1200	11	8.9%	automotive traction pack
B	≥ 550	70	56.5%	stationary ESS / second tier
C	≥ 0	43	34.7%	scrap or non-critical

Cycle life spans 150 – 2236 (median 736), consistent with the published 150–2300 range.

⚠️ Grade A holds only 11 cells. configs/grading.yaml anticipated this risk explicitly. Eleven examples is thin for a three-class classifier and very thin for per-fold estimates under 5-fold grouped CV (~2 cells per fold). This is a decision for Phase 4/6 review, not something to be worked around silently. Options: lower the A boundary, treat grading as ordinal regression from predicted cycle life, or report grade-A metrics with explicit caveats.

Batch 3 is genuinely shifted

Two-sample KS between batches 1–2 and batch 3: D = 0.651, p = 1.7 × 10⁻¹¹. Median cycle life 547.5 vs 963.5. Batch 3 is a real distribution shift, so RQ4 tests something meaningful rather than a relabelling.

Feature signal (Spearman vs log cycle life)

Feature	ρ
log var ΔQ(V)	−0.902
min ΔQ(V)	+0.868
internal resistance at cycle N	−0.594
minimum internal resistance	−0.588
maximum temperature	−0.252
mean temperature	+0.222

ΔQ(V)-derived features dominate, consistent with the literature. Thermal signal is weak (|ρ| ≈ 0.22–0.25) in this corpus — worth stating now, since Phase 9 hypothesis H3 predicts monotonic thermal SHAP attributions. In a 30 °C-controlled chamber there is little thermal variation to exploit, so a weak result there would be a property of the dataset, not a modelling failure.

Process-recipe space

69 unique charging policies across 124 cells, step-1 C-rates spanning 1.0–8.0 C. Recipe families are heavily confounded with cycle life by design, which is exactly why Group F must be ablated for RQ4.

Why a naive rule is not enough — the motivation

Single-feature threshold on log var ΔQ(V), at matched yield (shipping exactly the 81 truly capable cells' worth), tuned in-sample:

Budget	Escapes	Escape rate	Overkill rate
5	14	17.3%	17.3%
10	22	27.2%	27.2%
20	23	28.4%	28.4%
50	9	11.1%	11.1%
100	7	8.6%	8.6%

Even at budget 100, tuned in-sample with a perfectly calibrated yield, a naive rule leaves an 8.6% escape rate — and it offers no way to bound that rate at a chosen level. This is the gap the conformal risk control of Phase 7 and the cost-optimal triage of Phase 8 exist to close.

Two honest caveats: these are in-sample numbers, so they are an optimistic bound rather than held-out performance; and the non-monotonicity at budgets 10–20 reflects both a changing ΔQ baseline cycle and genuine small-sample noise at n = 124.

---

Artifacts

File	Contents
outputs/reports/dataset_manifest.json	URLs, sizes, SHA-256 digests
outputs/reports/reconciliation.csv	The found/joined/excluded/used table
outputs/reports/exclusion_audit.json	Per-cell exclusion verdicts and evidence
outputs/reports/validation_report.csv	Per-cell validation record
outputs/reports/validation_issues.json	Every FAIL and WARN
outputs/reports/eda_results.json	All Phase 3 quantitative results
outputs/reports/eda_delta_q_statistics.csv	Per-cell ΔQ(V) statistics
outputs/reports/cohort.csv	Final 124-cell labelled cohort
data/processed/cohort.parquet	Cohort with labels, grades and quality flags

Reproduce end to end:

python -m src.data.downloader   # 7.70 GiB, resumable, checksum-verified
python -m src.data.mat_parser   # -> data/interim/cells/
python -m src.data.validator    # must report 0 FAIL
python -m src.data.cleaner      # reconciliation + cohort
python -m src.data.eda          # 11 figures + Gate 1

---

← README · Feature engineering →