feat(integrity): wire ADR-0010 on-load verify + re-pin 0.1.9 (deploy slice)

Adds the ADR-0010 runtime on top of the ADR-0011 upload gate for a clean deploy
off origin/main (NO ADR-0007 sandbox stack): resolve_to_local_path verifies a
materialized file against the manifest SHA-256 baseline (biodata-registry 0.1.9)
before parse/cache, refusing a tampered file and deleting a bad temp download.
Re-applied surgically onto origin/main so the cached-huggingface-cli-token
fallback (data_io) and the tzlocal>=5.4.3 limma fix (requirements) are preserved
— not regressed by the older feat/sandbox-executor base. Re-pin 0.1.8 -> 0.1.9.
test_integrity.py 9 + test_upload_gate.py 27 green; verified end-to-end.

docs/adr/ADR-0010-dataset-integrity-verification.md

@@ -1,6 +1,6 @@
 # ADR-0010 — Dataset Integrity / Tamper Verification on Load
 
-**Status:** Proposed
+**Status:** Accepted — step 1 (AWS-independent) implemented 2026-07-02
 **Date:** 2026-07-01
 **Deciders:** Annie Voigt (project lead)
 **Driver:** OHSU security review Q7 — "detect unavailable/modified/compromised datasets." Today
@@ -48,14 +48,35 @@ Add **content-hash verification on load**, keyed off the manifest.
    SHA-256 verify in `resolve_to_local_path`; refusal path + test (good hash loads, altered file
    refused, absent hash = load with a "no integrity baseline" note); backfill hashes for the
    current registered datasets.
-   *(Available now: the shared hashing helper `src/core/integrity.py`
-   — `compute_sha256` / `verify_sha256`, streamed — landed with ADR-0011 and is the exact code
-   this step wires into `resolve_to_local_path`.)*
+   *(The shared hashing helper `src/core/integrity.py` — `compute_sha256` / `verify_sha256`,
+   streamed — landed with ADR-0011 and is the exact low-level code this step's on-load layer
+   builds on.)*
 2. **At AWS (ADR-0007 Phase 4):** the pre-stage sync job writes checksums into the read-only OHSU
    source bucket, so integrity is anchored to an OHSU-controlled copy rather than trust-on-first-use
    against the public source. This ADR's on-load check is the same code; only the hash's provenance
    improves.
 
+### Step 1 — as implemented (2026-07-02)
+
+- **Manifest field** (`biodata-registry` 0.1.9): optional `integrity:` block on `DatasetManifest`
+  — `sha256` (primary/expression file) + optional `files:` map (per-file, for separate metadata)
+  + `recorded`/`recorded_from` provenance. Validated (64-hex) in `manifest.validate()`; absent =
+  valid + a "no integrity baseline" warning. Reverse-lookup helper `expected_sha256_for_url()`.
+- **Recorder** `scripts/record_integrity.py` streams SHA-256 of each source and writes the block
+  into the YAML (comment-preserving text edit). **Backfilled 21/22** registered datasets;
+  `cptac_pda_counts` skipped — its hosted `cptac_pda_counts.h5ad` is **404 on HF** (pre-existing
+  missing-file issue, tracked separately).
+- **On-load verify** (`DecoupleRpy_Agent` `src/core/integrity.py`): `verify_file()` is called
+  centrally in `resolve_to_local_path` after materialize / before parse+cache. Keyed on the source
+  URL via the registry reverse-lookup, so **no tool signature changes**. Mismatch → `IntegrityError`
+  refusal naming the dataset; a tampered temp download is unlinked before raising; missing/absent
+  baseline degrades to "load unverified". Verified end-to-end (real load passes, tampered copy
+  refused).
+- **Not covered by step 1:** `src/workflows/geo.py` direct GEO-series-matrix loads bypass
+  `resolve_to_local_path` (dataset-prep path, not runtime) — add the same hook there if a manifest
+  ever points `expression_source.url` at a GEO matrix. Trust-on-first-use caveat below still applies
+  until step 2.
+
 ## Consequences
 
 - **Positive:** closes the Q7 tamper gap with a deterministic control, independent of AWS; dovetails

requirements.in

@@ -25,7 +25,7 @@
 # so git+https://huggingface.co/... fails the Space build. To release new
 # manifests: bump version in biodata-registry/pyproject.toml, run its
 # scripts/release.sh, then update this URL + commit.
-biodata-registry @ https://huggingface.co/anne-voigt/biodata-registry/resolve/0e29c5efa60a66b417f79e067dc2b0d927fcdc83/biodata_registry-0.1.8-py3-none-any.whl
+biodata-registry @ https://huggingface.co/anne-voigt/biodata-registry/resolve/3a86474d4971f3590fa44b042ab46196bdd9247e/biodata_registry-0.1.9-py3-none-any.whl
 
 # --- GEO platform (GPL) SOFT parsing (scripts/assemble_*.py precompute) ---
 GEOparse==2.0.4

requirements.txt

@@ -44,7 +44,7 @@ backports-tarfile==1.2.0
     # via jaraco-context
 beartype==0.22.9
     # via py-key-value-aio
-biodata-registry @ https://huggingface.co/anne-voigt/biodata-registry/resolve/0e29c5efa60a66b417f79e067dc2b0d927fcdc83/biodata_registry-0.1.8-py3-none-any.whl
+biodata-registry @ https://huggingface.co/anne-voigt/biodata-registry/resolve/3a86474d4971f3590fa44b042ab46196bdd9247e/biodata_registry-0.1.9-py3-none-any.whl
     # via -r requirements.in
 brotli==1.2.0
     # via gradio

src/core/data_io.py

@@ -29,6 +29,9 @@ def resolve_to_local_path(path_or_url: str) -> tuple[str, bool]:
     """
     s = str(path_or_url)
     if not s.startswith(("http://", "https://", "ftp://")):
+        # Local paths are still integrity-verified when the manifest baselines
+        # this exact path (rare, but keeps the check uniform); no-op otherwise.
+        _verify_or_raise(s, s, is_temp=False)
         return s, False
 
     # Authenticated path for private HF repos.
@@ -43,6 +46,7 @@ def resolve_to_local_path(path_or_url: str) -> tuple[str, bool]:
             except Exception:
                 token = None
         if token:
+            local = None
             try:
                 from huggingface_hub import hf_hub_download
 
@@ -61,9 +65,14 @@ def resolve_to_local_path(path_or_url: str) -> tuple[str, bool]:
                     revision=revision,
                     token=token,
                 )
-                return local, False  # HF cache file — caller must not delete it
             except Exception:
-                pass  # fall through to unauthenticated urllib (e.g. public file)
+                local = None  # fall through to unauthenticated urllib (e.g. public file)
+
+            # Verify OUTSIDE the try above so an IntegrityError is never swallowed
+            # into the urllib fallback (which fetches the same tampered content).
+            if local is not None:
+                _verify_or_raise(local, s, is_temp=False)
+                return local, False  # HF cache file — caller must not delete it
 
     suffix = Path(s.split("?", 1)[0]).suffix or ".h5ad"
     with tempfile.NamedTemporaryFile(suffix=suffix, delete=False) as tmp:
@@ -73,4 +82,25 @@ def resolve_to_local_path(path_or_url: str) -> tuple[str, bool]:
                 if not chunk:
                     break
                 tmp.write(chunk)
-        return tmp.name, True
+        tmp_path = tmp.name
+    # Verify before returning so a tampered download is never handed to a loader
+    # or admitted to the cache; a bad temp file is deleted here, not leaked.
+    _verify_or_raise(tmp_path, s, is_temp=True)
+    return tmp_path, True
+
+
+def _verify_or_raise(local_path: str, url: str, *, is_temp: bool) -> None:
+    """Run the ADR-0010 integrity check; delete a bad temp file before re-raising.
+
+    No-op when the URL has no recorded baseline. On a hash mismatch the load is
+    refused (:class:`IntegrityError`); for a temp file we unlink first so the
+    tampered bytes are never left on disk or admitted to the cache.
+    """
+    from src.core.integrity import IntegrityError, verify_file
+
+    try:
+        verify_file(local_path, url)
+    except IntegrityError:
+        if is_temp:
+            Path(local_path).unlink(missing_ok=True)
+        raise

tests/test_integrity.py

@@ -0,0 +1,143 @@
+"""Tests for on-load content-integrity verification (ADR-0010).
+
+Covers the ``src.core.integrity`` primitive (streamed SHA-256, mismatch refusal,
+no-baseline pass-through) and its wiring into ``resolve_to_local_path`` — that a
+good hash loads, a tampered file is refused, and the tampered temp file is deleted
+(never handed to a loader or cache). The registry reverse-lookup is monkeypatched
+so no network or installed manifests are required.
+"""
+from __future__ import annotations
+
+import hashlib
+import sys
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import src.core.integrity as integ  # noqa: E402
+import src.core.data_io as data_io  # noqa: E402
+from src.core.integrity import (  # noqa: E402
+    IntegrityError,
+    compute_sha256,
+    verify_file,
+    verify_sha256,
+)
+
+
+# ---------------------------------------------------------------------------
+# compute_sha256 / verify_sha256 (shared low-level primitives, ADR-0011 + 0010)
+# ---------------------------------------------------------------------------
+
+def test_compute_sha256_matches_hashlib(tmp_path):
+    p = tmp_path / "f.bin"
+    payload = b"decoupler-py integrity test\n" * 1000
+    p.write_bytes(payload)
+    assert compute_sha256(p) == hashlib.sha256(payload).hexdigest()
+
+
+def test_verify_sha256_match_and_mismatch(tmp_path):
+    p = tmp_path / "f.bin"
+    p.write_bytes(b"payload")
+    good = compute_sha256(p)
+    assert verify_sha256(p, good) is True
+    assert verify_sha256(p, good.upper()) is True   # case-insensitive
+    assert verify_sha256(p, "d" * 64) is False
+    assert verify_sha256(p, "") is False            # absent baseline is not a pass
+
+
+# ---------------------------------------------------------------------------
+# verify_file (primitive)
+# ---------------------------------------------------------------------------
+
+def test_verify_file_passes_on_match(tmp_path, monkeypatch):
+    p = tmp_path / "f.h5ad"
+    p.write_bytes(b"good bytes")
+    good = compute_sha256(p)
+    monkeypatch.setattr(integ, "_lookup_from_registry", lambda url: (good, "ds1"))
+    verify_file(p, "https://host/f.h5ad")  # no raise
+
+
+def test_verify_file_raises_on_mismatch(tmp_path, monkeypatch):
+    p = tmp_path / "f.h5ad"
+    p.write_bytes(b"tampered bytes")
+    monkeypatch.setattr(integ, "_lookup_from_registry", lambda url: ("a" * 64, "ds1"))
+    with pytest.raises(IntegrityError) as exc:
+        verify_file(p, "https://host/f.h5ad")
+    assert "ds1" in str(exc.value)
+    assert exc.value.actual == compute_sha256(p)
+
+
+def test_verify_file_noop_without_baseline(tmp_path, monkeypatch):
+    p = tmp_path / "f.h5ad"
+    p.write_bytes(b"whatever")
+    monkeypatch.setattr(integ, "_lookup_from_registry", lambda url: (None, None))
+    verify_file(p, "https://host/f.h5ad")  # no raise — unbaselined loads
+
+
+# ---------------------------------------------------------------------------
+# resolve_to_local_path wiring (local-path branch — no network)
+# ---------------------------------------------------------------------------
+
+def test_resolve_local_path_good_hash_loads(tmp_path, monkeypatch):
+    p = tmp_path / "data.h5ad"
+    p.write_bytes(b"real dataset")
+    good = compute_sha256(p)
+    monkeypatch.setattr(integ, "_lookup_from_registry", lambda url: (good, "ds1"))
+    local, is_temp = data_io.resolve_to_local_path(str(p))
+    assert local == str(p)
+    assert is_temp is False
+
+
+def test_resolve_local_path_bad_hash_refused(tmp_path, monkeypatch):
+    p = tmp_path / "data.h5ad"
+    p.write_bytes(b"tampered dataset")
+    monkeypatch.setattr(integ, "_lookup_from_registry", lambda url: ("b" * 64, "ds1"))
+    with pytest.raises(IntegrityError):
+        data_io.resolve_to_local_path(str(p))
+
+
+def test_resolve_absent_baseline_loads(tmp_path, monkeypatch):
+    p = tmp_path / "data.h5ad"
+    p.write_bytes(b"unbaselined dataset")
+    monkeypatch.setattr(integ, "_lookup_from_registry", lambda url: (None, None))
+    local, is_temp = data_io.resolve_to_local_path(str(p))
+    assert local == str(p)
+
+
+# ---------------------------------------------------------------------------
+# resolve_to_local_path wiring (temp-download branch) — tampered temp is deleted
+# ---------------------------------------------------------------------------
+
+def test_resolve_tampered_download_deletes_temp(tmp_path, monkeypatch):
+    """A tampered http(s) download is refused AND the temp file is unlinked."""
+    payload = b"tampered downloaded bytes"
+    created = {}
+
+    class _FakeResp:
+        def __init__(self, data):
+            self._data = data
+        def read(self, n):
+            d, self._data = self._data[:n], self._data[n:]
+            return d
+        def __enter__(self):
+            return self
+        def __exit__(self, *a):
+            return False
+
+    monkeypatch.setattr(data_io.urllib.request, "urlopen", lambda url: _FakeResp(payload))
+    # Capture the temp path the resolver creates.
+    real_ntf = data_io.tempfile.NamedTemporaryFile
+    def _spy_ntf(*a, **kw):
+        f = real_ntf(*a, **kw)
+        created["path"] = f.name
+        return f
+    monkeypatch.setattr(data_io.tempfile, "NamedTemporaryFile", _spy_ntf)
+    monkeypatch.setattr(integ, "_lookup_from_registry", lambda url: ("c" * 64, "ds1"))
+
+    with pytest.raises(IntegrityError):
+        data_io.resolve_to_local_path("https://host/remote.h5ad")
+
+    assert "path" in created
+    assert not Path(created["path"]).exists(), "tampered temp file must be deleted"