fix(rna_sc): harden load/visualize UMAP against missing embedding/leiden (#1)

- fix(rna_sc): harden load/visualize UMAP against missing embedding/leiden (9d23b390a1d1f41be152d4f7e8d5db68f3cfd0ac)

TODO.md

@@ -332,11 +332,21 @@ correct (single-dataset, Path B, ULM, 98.3% coverage); these are the gaps.
     `dataset_score_signature` scores it end-to-end (66.6% coverage on a synthetic
     cohort); `load_signature_net` + `resolve_to_local_path` now fetch a private
     signature URL with auth (env token OR cached `huggingface-cli login`).
+  - **Done (2026-07-01) — `decoupler_load_and_visualize_data` hardened for subsets
+    lacking a precomputed UMAP/leiden** (was spun off → `task_8b2a1bdc`; branch
+    `claude/friendly-mendel-02443d`, off current `main`, pushed to `origin`, NOT
+    deployed). `_ensure_umap` reuses an existing embedding (pbmc3k unchanged) or
+    computes a bounded `normalize+log1p→pca→neighbors→leiden(igraph)→umap` pipeline
+    on the loaded copy, and NEVER hard-fails — on any failure the plot is skipped
+    and the loaded AnnData + metadata are still returned with a note. Grouping
+    detection covers R `make.names` atlas cols (`Clusters`, …), not just `leiden`;
+    leiden uses `flavor="igraph"` (no `leidenalg` dep on the Space). Sits on top of
+    the `_load_adata` seam, so the Loveless subsets now load AND visualize
+    end-to-end. Tests: `tests/test_rna_sc_load.py` (4 green).
   - **Still open:** FACTORY rebuild of the Space on the 0.1.8 re-pin + e2e verify
-    on `hf-dev`; confirm prod RAM holds the subset live; harden
-    `decoupler_load_and_visualize_data` for subsets lacking a precomputed
-    UMAP/leiden (spun off → `task_8b2a1bdc`); pseudobulk-aggregation tool for the
-    sample-level DE contrast; derive Werba signatures when needed. ADR-0006 items 7/8/9.
+    on `hf-dev`; confirm prod RAM holds the subset live; pseudobulk-aggregation tool
+    for the sample-level DE contrast; derive Werba signatures when needed. ADR-0006
+    items 7/8/9.
   Companion to the biodata-registry Loveless ingestion (provenance/scope/gate plan
   there). Design goal: keep user-facing runtime bulk-like.
   - Two roles. The **Steele-subset h5ad** is an analyzable sc dataset — loads once

memory.md

@@ -8,6 +8,42 @@ Last updated: 2026-07-01
 
 ---
 
+## 2026-07-01 (later 2) — Harden sc load/visualize against missing UMAP/leiden (branch `claude/friendly-mendel-02443d`, off current `main`, pushed to `origin`, NOT deployed)
+
+`decoupler_load_and_visualize_data` (rna_sc.py) called `sc.pl.umap(color="leiden")`
+unconditionally after load and crashed on any h5ad lacking `obsm['X_umap']` /
+`obs['leiden']` — exactly the Loveless raw-count subsets (`gse155698_steele`,
+`gse205013_werba`; biodata-registry 0.1.8): extracted as RAW COUNTS with the
+atlas's corrected/scaled/integrated layers dropped → no embedding, no leiden. The
+load tool is the Path-P entry point, so it would crash on first real use. (Was the
+`task_8b2a1bdc` spin-off + the "Still open" harden bullet under the Loveless item.)
+
+- New `_ensure_umap(adata)` → `(color_key, note)`, mutates in place (caller holds
+  a `read_h5ad_cached` copy). Reuses a precomputed embedding+grouping unchanged
+  (pbmc3k demo path preserved); else runs a bounded
+  `normalize_total(1e4)+log1p` (only if the matrix looks like raw counts) →
+  `pca → neighbors → leiden → umap`. **Never raises** — on failure the UMAP plot
+  is skipped and the loaded AnnData + metadata are still returned, with the reason
+  in `message`. leiden uses `flavor="igraph"` (no `leidenalg` dep on the Space).
+- New `_looks_like_raw_counts` — bounded, sparse-aware (`scipy.sparse.issparse`; a
+  plain numpy `.data` is a memoryview, not counts — the bug that failed the first
+  cut).
+- Grouping detection `_UMAP_GROUPING_CANDIDATES` covers R `make.names` atlas cols
+  (`Clusters`, …), not just `leiden`.
+- Sits ON TOP of the `_load_adata` seam already on `main`, so the hardening + the
+  hosted-URL loader ship together: the Loveless subsets now load AND visualize
+  end-to-end. Tests: `tests/test_rna_sc_load.py` (4 green; full rna_sc + loading-plan
+  suite 26 green).
+- **Reconciliation note:** this branch was originally cut from an older `main`
+  (the executor-seam commit `b187b88`, which is NOT in `main`) and lacked the
+  Loveless loader. Reconciled by resetting the branch onto current `origin/main`
+  (`6cda24b`, loveless) and re-applying ONLY the UMAP hardening — the executor seam
+  was dropped (it lives on `feat/executor-seam`, unaffected). Old pre-reset commits
+  recoverable via reflog (`d46b0e9`).
+- **NOT merged to `main`/deployed.** Folds into the pending 0.1.8 factory-rebuild.
+
+---
+
 ## 2026-07-01 (later) — Role-2 signature PUBLISHED + merged main + resolver auth (branch `feat/loveless-sc-serving`)
 
 Three things this session:

src/tools/rna_sc.py

@@ -21,6 +21,7 @@ from typing import Annotated, Literal
 
 import decoupler as dc
 import matplotlib
+import numpy as np
 import pandas as pd
 # Analysis-specific imports
 import scanpy as sc
@@ -115,6 +116,105 @@ def _load_adata(adata_path: str):
             Path(local_path).unlink(missing_ok=True)
 
 
+# Groupings to color the initial UMAP by, in priority order. `leiden` is the
+# tutorial default (decoupler's pbmc3k demo carries it); the rest cover atlas
+# exports that use R `make.names` obs naming (e.g. `Clusters` holds cell-type
+# labels) — so we never assume `leiden` exists as the grouping.
+_UMAP_GROUPING_CANDIDATES = [
+    "leiden",
+    "louvain",
+    "Clusters",
+    "clusters",
+    "cluster",
+    "seurat_clusters",
+    "cell_type",
+    "celltype",
+    "CellType",
+    "cell_types",
+]
+
+
+def _looks_like_raw_counts(adata) -> bool:
+    """Heuristic: raw UMI counts are non-negative and integer-valued.
+
+    Samples a bounded slice of ``adata.X`` so it stays cheap on a cpu-basic
+    Space regardless of dataset size.
+    """
+    import scipy.sparse as sp
+
+    X = adata.X
+    data = X.data if sp.issparse(X) else np.asarray(X).ravel()
+    if data.size == 0:
+        return False
+    sample = np.asarray(data[:100_000], dtype=float)
+    if (sample < 0).any():
+        return False
+    return bool(np.allclose(sample, np.round(sample)))
+
+
+def _ensure_umap(adata) -> tuple[str | None, str | None]:
+    """Ensure ``adata`` can produce an initial UMAP, degrading gracefully.
+
+    Returns ``(color_key, note)`` and mutates ``adata`` in place (the caller
+    holds a fresh ``read_h5ad_cached`` copy, so the source h5ad is untouched).
+
+    - If a UMAP embedding and a known grouping are already present (e.g.
+      decoupler's pbmc3k demo), returns immediately — existing behavior is
+      preserved and nothing is recomputed.
+    - Otherwise runs a minimal, bounded pipeline (normalize+log1p if the matrix
+      looks like raw counts, then pca → neighbors → leiden → umap) so datasets
+      exported without embeddings (e.g. the Loveless raw-count subsets) are
+      still visualizable.
+    - Never raises: on any failure it returns ``(color_key_or_None, note)`` so
+      the caller can skip the plot but still return the loaded data + metadata.
+    """
+    grouping = _find_column(adata.obs, _UMAP_GROUPING_CANDIDATES)
+    has_umap = "X_umap" in adata.obsm
+
+    if has_umap and grouping is not None:
+        return grouping, None
+
+    notes: list[str] = []
+    try:
+        if not has_umap and _looks_like_raw_counts(adata):
+            sc.pp.normalize_total(adata, target_sum=1e4)
+            sc.pp.log1p(adata)
+            notes.append("normalized_total(1e4)+log1p (input looked like raw counts)")
+
+        if not has_umap:
+            n_comps = min(50, adata.n_vars - 1, adata.n_obs - 1)
+            if n_comps < 2:
+                return grouping, (
+                    "too few cells/genes to compute a UMAP; returning loaded "
+                    "data without a plot"
+                )
+            sc.pp.pca(adata, n_comps=n_comps)
+            sc.pp.neighbors(adata, n_neighbors=min(15, adata.n_obs - 1))
+
+        if grouping is None:
+            try:
+                # flavor="igraph" uses igraph's built-in Leiden (no leidenalg
+                # dependency), which is what is available on the Space.
+                sc.tl.leiden(
+                    adata, flavor="igraph", n_iterations=2, directed=False
+                )
+                grouping = "leiden"
+                notes.append("computed leiden clustering")
+            except Exception as e:  # clustering is best-effort; UMAP still useful
+                notes.append(f"leiden clustering skipped ({e})")
+
+        if "X_umap" not in adata.obsm:
+            sc.tl.umap(adata)
+            notes.append("computed UMAP embedding")
+
+        return grouping, ("; ".join(notes) if notes else None)
+    except Exception as e:
+        return grouping, (
+            f"could not compute a UMAP embedding ({e}); returning loaded data "
+            "without a plot"
+        )
+
+
 @rna_sc_mcp.tool
 def decoupler_load_and_visualize_data(
     # Primary data inputs
@@ -139,34 +239,52 @@ def decoupler_load_and_visualize_data(
         # Use demo dataset as in tutorial
         adata = dc.ds.pbmc3k()
 
-    # Create UMAP visualization as in tutorial
-    fig, ax = plt.subplots(figsize=(5, 4))
-    sc.pl.umap(adata=adata, color="leiden", ax=ax)
-
-    # Save figure
-    umap_file = OUTPUT_DIR / f"{out_prefix}_leiden_umap.png"
-    fig.savefig(umap_file, dpi=300, bbox_inches="tight")
-    plt.close(fig)
+    # Ensure the data is visualizable. Precomputed embeddings (e.g. decoupler's
+    # pbmc3k demo) are used as-is; datasets exported without a UMAP/leiden (e.g.
+    # the Loveless raw-count subsets) get a bounded pipeline computed here. This
+    # never hard-fails — if no embedding can be produced, the plot is skipped and
+    # the loaded data + metadata are still returned.
+    color_key, viz_note = _ensure_umap(adata)
+
+    artifacts = []
+    umap_file = None
+    if "X_umap" in adata.obsm:
+        fig, ax = plt.subplots(figsize=(5, 4))
+        sc.pl.umap(adata=adata, color=color_key, ax=ax, show=False)
+        umap_file = OUTPUT_DIR / f"{out_prefix}_leiden_umap.png"
+        fig.savefig(umap_file, dpi=300, bbox_inches="tight")
+        plt.close(fig)
+        desc = (
+            f"UMAP visualization colored by {color_key}"
+            if color_key
+            else "UMAP visualization"
+        )
+        artifacts.append({"description": desc, "path": str(umap_file.resolve())})
 
     # Save processed data
     data_file = OUTPUT_DIR / f"{out_prefix}_processed_data.h5ad"
     adata.write_h5ad(data_file)
+    artifacts.append(
+        {"description": "Processed AnnData file", "path": str(data_file.resolve())}
+    )
 
     # Save metadata
     metadata_file = OUTPUT_DIR / f"{out_prefix}_metadata.csv"
     adata.obs.to_csv(metadata_file)
+    artifacts.append(
+        {"description": "Cell metadata", "path": str(metadata_file.resolve())}
+    )
+
+    message = f"Loaded data with {adata.n_obs} cells and {adata.n_vars} genes"
+    if umap_file is None:
+        message += " (no UMAP plot produced)"
+    if viz_note:
+        message += f". Visualization note: {viz_note}"
 
     return {
-        "message": f"Loaded data with {adata.n_obs} cells and {adata.n_vars} genes",
+        "message": message,
         "reference": "https://github.com/scverse/decoupler-tutorials/blob/main/scell/rna_sc.ipynb",
-        "artifacts": [
-            {
-                "description": "UMAP visualization with leiden clusters",
-                "path": str(umap_file.resolve()),
-            },
-            {"description": "Processed AnnData file", "path": str(data_file.resolve())},
-            {"description": "Cell metadata", "path": str(metadata_file.resolve())},
-        ],
+        "artifacts": artifacts,
     }
 
 

tests/test_rna_sc_load.py

@@ -0,0 +1,113 @@
+"""
+Tests for src/tools/rna_sc.py::decoupler_load_and_visualize_data.
+
+Regression guard for the Loveless single-cell subsets (biodata-registry 0.1.8,
+ADR-0006 Role 1 serving): those h5ads are raw counts extracted from the Loveless
+integrated atlas with the corrected/scaled/integrated layers dropped, so they
+carry no `X_umap` embedding and no `leiden` clustering. The tool used to call
+`sc.pl.umap(color="leiden")` unconditionally and crash on first real use. These
+tests assert it now degrades gracefully on such data while preserving the
+original behavior when an embedding + grouping are already present.
+
+`_load_adata` (the shared local/HF-URL resolver, covered by test_rna_sc_loader)
+handles a local path here; these tests focus only on the post-load UMAP/leiden
+assumptions. Uses tiny synthetic AnnData (no network, no downloads).
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import anndata as ad
+import numpy as np
+import pytest
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+import src.tools.rna_sc as rna_sc  # noqa: E402
+from src.tools.rna_sc import decoupler_load_and_visualize_data  # noqa: E402
+
+
+def _write_raw_counts_adata(path: Path, n_obs: int = 60, n_vars: int = 80):
+    """Synthetic raw-count AnnData with no UMAP and no leiden — mimics a
+    Loveless subset. `Name` uses the atlas's R make.names obs naming."""
+    rng = np.random.default_rng(0)
+    X = rng.poisson(2.0, size=(n_obs, n_vars)).astype("float32")
+    adata = ad.AnnData(X)
+    adata.obs_names = [f"cell{i}" for i in range(n_obs)]
+    adata.var_names = [f"gene{i}" for i in range(n_vars)]
+    adata.obs["Name"] = "sampleA"
+    adata.write_h5ad(path)
+    return adata
+
+
+@pytest.fixture(autouse=True)
+def _redirect_output(tmp_path, monkeypatch):
+    """Keep all artifacts out of the repo's tmp/outputs."""
+    monkeypatch.setattr(rna_sc, "OUTPUT_DIR", tmp_path)
+
+
+def test_raw_counts_without_umap_or_leiden_returns(tmp_path):
+    """The core regression: a raw-count h5ad with no X_umap and no leiden must
+    load successfully rather than crashing on sc.pl.umap(color='leiden')."""
+    h5ad = tmp_path / "loveless_like.h5ad"
+    _write_raw_counts_adata(h5ad)
+
+    out = decoupler_load_and_visualize_data(adata_path=str(h5ad))
+
+    assert isinstance(out, dict)
+    assert "60 cells and 80 genes" in out["message"]
+    # A processed AnnData artifact is always returned, whether or not a plot was.
+    descs = [a["description"] for a in out["artifacts"]]
+    assert any("Processed AnnData" in d for d in descs)
+    assert any("Cell metadata" in d for d in descs)
+    # Every returned artifact path exists on disk.
+    for art in out["artifacts"]:
+        assert Path(art["path"]).exists()
+
+
+def test_bounded_pipeline_computes_umap(tmp_path):
+    """On raw counts the tool should compute a minimal pipeline and emit a UMAP
+    plot (leiden via igraph flavor, no leidenalg dependency)."""
+    h5ad = tmp_path / "loveless_like.h5ad"
+    _write_raw_counts_adata(h5ad)
+
+    out = decoupler_load_and_visualize_data(adata_path=str(h5ad))
+
+    descs = [a["description"] for a in out["artifacts"]]
+    assert any("UMAP" in d for d in descs)
+    assert "note" in out["message"].lower()
+
+    # The processed h5ad carries the computed embedding.
+    import scanpy as sc
+
+    proc = next(a["path"] for a in out["artifacts"] if "Processed" in a["description"])
+    reloaded = sc.read_h5ad(proc)
+    assert "X_umap" in reloaded.obsm
+
+
+def test_precomputed_umap_and_leiden_preserved(tmp_path):
+    """When X_umap + leiden already exist (pbmc3k-like), behavior is preserved:
+    a UMAP plot is emitted and no recompute note appears."""
+    rng = np.random.default_rng(1)
+    n_obs, n_vars = 40, 30
+    adata = ad.AnnData(rng.normal(size=(n_obs, n_vars)).astype("float32"))
+    adata.obs_names = [f"cell{i}" for i in range(n_obs)]
+    adata.var_names = [f"gene{i}" for i in range(n_vars)]
+    adata.obs["leiden"] = np.array(["0", "1"] * (n_obs // 2))
+    adata.obs["leiden"] = adata.obs["leiden"].astype("category")
+    adata.obsm["X_umap"] = rng.normal(size=(n_obs, 2))
+    h5ad = tmp_path / "pbmc_like.h5ad"
+    adata.write_h5ad(h5ad)
+
+    out = decoupler_load_and_visualize_data(adata_path=str(h5ad))
+
+    descs = [a["description"] for a in out["artifacts"]]
+    assert any("UMAP" in d and "leiden" in d for d in descs)
+    assert "note" not in out["message"].lower()
+
+
+def test_missing_file_raises(tmp_path):
+    with pytest.raises(FileNotFoundError):
+        decoupler_load_and_visualize_data(adata_path=str(tmp_path / "nope.h5ad"))