avoigt1121 Claude Opus 4.8 commited on
Commit
a7a8caa
·
1 Parent(s): 7edffac

ADR-0001 Phase 2 Mode A: early integration (T7 + T8 + T9)

Browse files

T7 — src/workflows/integration.py: combined-AnnData builder. Harmonise each
dataset to a gene-symbol axis, intersect features, concat with a `batch` obs key
(=dataset_id). Pure core combine_anndatas() + loader wrapper
build_combined_anndata() (resolves each manifest's hosted h5ad).

T8 — decoupler_differential_expression gains batch_column: DESeq2 fits
~batch+factor; new run_limma_covariate (~batch+group) in microarray.py; ttest+batch
is refused (a t-test cannot model a covariate; no silent drop). Default None keeps
single-dataset behaviour unchanged.

T9 — decoupler_integrate_datasets (integration_mcp, 3rd tool): re-checks
get_integration_plan and pools + runs ONE batch-aware DE only on mode=="early";
refuses/reroutes for late (meta_analyze) / concordance / refuse. Auto-selects
deseq2 (raw counts) or limma from the plan's poolable_data_level.

Tests: test_integration_mode_a (9), test_de_batch_covariate (4),
test_integrate_datasets_tool (9). Full non-live suite: 915 passed. v1 limits:
gene-symbol axis only (no probe collapse / ortholog mapping), ComBat (step 11)
deferred. Branch-local; NOT deployed (origin = prod Space).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/adr/ADR-0001-implementation-plan.md CHANGED
@@ -1,7 +1,7 @@
1
  # ADR-0001 Implementation Plan — Cross-Dataset Analysis
2
 
3
  **Companion to:** [`ADR-0001-cross-dataset-integration.md`](./ADR-0001-cross-dataset-integration.md)
4
- **Status:** Phase 1 (Mode B) complete & shipped to prod 2026-06-19 (T1–T6); Phase 2 (Mode A: T7–T9) not started (`early` falls back to `late`); Phase 3 (routing/interpretation/eval) pending. **Concordance mode** (same-cohort sibling quantifications) added later in biodata-registry 0.1.6 — see the 2026-06-24 update below; agent re-pinned + **deployed to prod** (`fb2091e`, `origin/main`). Reconciled 2026-06-24.
5
  **Scope:** `biodata-registry`, `DecoupleRpy_Agent`, `pdac-analysis-orchestrator`
6
 
7
  This plan turns the ADR's three phases into file-level work, grounded in the
@@ -115,17 +115,22 @@ pins `get_integration_plan`'s logic (the D3 poolability rule).
115
 
116
  ## Phase 2 — Mode A (early integration)
117
 
118
- 8. [ ] **`src/workflows/integration.py`** feature intersection + AnnData concat
 
 
 
 
 
119
  → emit one **standard combined AnnData** with a `batch` obs key
120
  (`dataset_id`). Existing single-dataset tools run on it unchanged.
121
- 9. [ ] **`supports_covariate` capability (D5)** — extend
122
  `decoupler_differential_expression` with an optional `batch_column`:
123
  DESeq2 `~batch + condition` (one-liner); **refactor `run_limma` to a design
124
  matrix** including batch; ttest path uses pre-correction instead. Back-compat
125
  default off.
126
- 10. [ ] **`decoupler_integrate_datasets`** Layer 1 tool, gated behind the
127
  batch-confound refusal from `get_integration_plan`.
128
- 11. [ ] *(Optional)* add `inmoose` (ComBat / ComBat-seq) for the pre-correction
129
  route; keep the dependency optional.
130
 
131
  > **No per-tool integration work.** A future tool gains Mode A by reading the
 
1
  # ADR-0001 Implementation Plan — Cross-Dataset Analysis
2
 
3
  **Companion to:** [`ADR-0001-cross-dataset-integration.md`](./ADR-0001-cross-dataset-integration.md)
4
+ **Status:** Phase 1 (Mode B) complete & shipped to prod 2026-06-19 (T1–T6); Phase 2 (Mode A: T7–T9) built + tested 2026-06-24 on branch `adr-0001-phase-2-mode-a`, not yet deployed (`early` still falls back to `late` in prod until then); Phase 3 (routing/interpretation/eval) pending. **Concordance mode** (same-cohort sibling quantifications) added later in biodata-registry 0.1.6 — see the 2026-06-24 update below; agent re-pinned + **deployed to prod** (`fb2091e`, `origin/main`). Reconciled 2026-06-24.
5
  **Scope:** `biodata-registry`, `DecoupleRpy_Agent`, `pdac-analysis-orchestrator`
6
 
7
  This plan turns the ADR's three phases into file-level work, grounded in the
 
115
 
116
  ## Phase 2 — Mode A (early integration)
117
 
118
+ > **Built + tested 2026-06-24** on branch `adr-0001-phase-2-mode-a` (T7–T9; 915
119
+ > non-live tests green). v1 harmonises to a gene-symbol axis (probe collapse /
120
+ > ortholog mapping not handled), ttest+batch is refused (no silent ComBat), and
121
+ > step 11 (ComBat) is deferred. **Not yet deployed** (origin = prod Space).
122
+
123
+ 8. [x] **`src/workflows/integration.py`** — feature intersection + AnnData concat
124
  → emit one **standard combined AnnData** with a `batch` obs key
125
  (`dataset_id`). Existing single-dataset tools run on it unchanged.
126
+ 9. [x] **`supports_covariate` capability (D5)** — extend
127
  `decoupler_differential_expression` with an optional `batch_column`:
128
  DESeq2 `~batch + condition` (one-liner); **refactor `run_limma` to a design
129
  matrix** including batch; ttest path uses pre-correction instead. Back-compat
130
  default off.
131
+ 10. [x] **`decoupler_integrate_datasets`** Layer 1 tool, gated behind the
132
  batch-confound refusal from `get_integration_plan`.
133
+ 11. [ ] *(Optional — deferred, not in v1)* add `inmoose` (ComBat / ComBat-seq) for the pre-correction
134
  route; keep the dependency optional.
135
 
136
  > **No per-tool integration work.** A future tool gains Mode A by reading the
src/tools/integration_tools.py CHANGED
@@ -15,8 +15,11 @@ This MCP server exposes the cross-dataset result-combination paths:
15
  call overlap), because meta-analyzing identical samples double-counts the
16
  cohort. Routed here by get_integration_plan's mode == "concordance".
17
 
18
- The early-integration (pooling) tool `decoupler_integrate_datasets` is Phase 2
19
- (ADR-0001 T9) and will be added to this same server.
 
 
 
20
 
21
  Design (ADR-0001 Mode B):
22
  - The *math* is generic over the result envelope and lives in
@@ -32,7 +35,7 @@ from __future__ import annotations
32
 
33
  from datetime import datetime
34
  from pathlib import Path
35
- from typing import Annotated, Optional
36
 
37
  import pandas as pd
38
  from fastmcp import FastMCP
@@ -45,6 +48,11 @@ from src.core.combine import (
45
  from src.tools.rna._base import OUTPUT_DIR
46
  from src.workflows.concordance import concordance_metrics
47
  from src.workflows.meta_analysis import combine_envelopes
 
 
 
 
 
48
 
49
  integration_mcp = FastMCP(name="integration")
50
 
@@ -462,3 +470,154 @@ def decoupler_normalization_concordance(
462
  }
463
  ],
464
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
15
  call overlap), because meta-analyzing identical samples double-counts the
16
  cohort. Routed here by get_integration_plan's mode == "concordance".
17
 
18
+ 3. decoupler_integrate_datasets: *early integration* POOL >=2 datasets into one
19
+ matrix (feature intersection + a `batch` obs key) and run a single batch-aware
20
+ DE. Only for datasets the plan rules "early" (shared feature space + poolable
21
+ data_level); it re-checks the plan and refuses/reroutes otherwise. (ADR-0001
22
+ T9, Mode A.)
23
 
24
  Design (ADR-0001 Mode B):
25
  - The *math* is generic over the result envelope and lives in
 
35
 
36
  from datetime import datetime
37
  from pathlib import Path
38
+ from typing import Annotated, Literal, Optional
39
 
40
  import pandas as pd
41
  from fastmcp import FastMCP
 
48
  from src.tools.rna._base import OUTPUT_DIR
49
  from src.workflows.concordance import concordance_metrics
50
  from src.workflows.meta_analysis import combine_envelopes
51
+ # Mode-A early integration (T9): the plan gate, the combined-AnnData builder, and
52
+ # the batch-aware DE tool. Module-level so tests can monkeypatch them.
53
+ from src.datasets.registry import get_integration_plan
54
+ from src.workflows.integration import build_combined_anndata
55
+ from src.tools.rna.analysis import decoupler_differential_expression
56
 
57
  integration_mcp = FastMCP(name="integration")
58
 
 
470
  }
471
  ],
472
  }
473
+
474
+
475
+ @integration_mcp.tool
476
+ def decoupler_integrate_datasets(
477
+ dataset_ids: Annotated[
478
+ list[str],
479
+ "Two or more registered dataset IDs to POOL into one matrix and analyze "
480
+ "together (early integration), e.g. ['tcga_paad', 'paca_au_rnaseq'].",
481
+ ],
482
+ design_factor: Annotated[
483
+ str,
484
+ "obs column holding the contrast groups; must exist in EVERY dataset "
485
+ "(e.g. 'tumor_subtype').",
486
+ ],
487
+ test_group: Annotated[str, "Test arm of the contrast (a value of design_factor)."],
488
+ control_group: Annotated[str, "Control / reference arm of the contrast."],
489
+ method: Annotated[
490
+ Literal["auto", "deseq2", "limma"],
491
+ "DE method on the pooled matrix. 'auto' (default) picks deseq2 for pooled "
492
+ "raw counts and limma otherwise, from the integration plan's poolable "
493
+ "data_level. `dataset` is always modelled as a batch covariate.",
494
+ ] = "auto",
495
+ min_shared_features: Annotated[
496
+ int,
497
+ "Minimum shared gene symbols required to pool; below this the datasets "
498
+ "likely do not share a feature space and the request is refused.",
499
+ ] = 200,
500
+ out_prefix: Annotated[Optional[str], "Output file prefix."] = None,
501
+ ) -> dict:
502
+ """
503
+ Early integration (Mode A): pool >=2 datasets into one matrix + one batch-aware DE.
504
+
505
+ Use this ONLY when dataset_get_integration_plan returns mode=="early": the
506
+ datasets share a poolable feature space AND data_level, so they can be
507
+ concatenated and analyzed jointly with `dataset` as a batch covariate (more
508
+ power than meta-analysis when valid). This tool RE-CHECKS the plan itself and
509
+ refuses (does not pool) when the verdict is not 'early':
510
+
511
+ - 'late' -> run each dataset separately, then decoupler_meta_analyze.
512
+ - 'concordance' -> same-cohort variants -> decoupler_normalization_concordance.
513
+ - 'refuse' -> incompatible (e.g. confounded design / cross-modality).
514
+
515
+ On 'early' it builds the combined AnnData (feature intersection + a `batch`
516
+ obs key = dataset_id), runs decoupler_differential_expression with
517
+ batch_column='batch' (DESeq2 ~batch+factor for raw counts, limma ~batch+group
518
+ otherwise), and returns the pooled DE result plus per-dataset composition.
519
+ Returns an {"error": ..., "refused": True} dict (does not raise) so the agent
520
+ can surface the reason and reroute.
521
+
522
+ Pass design_factor + test_group + control_group: they drive the plan's
523
+ confound check AND the pooled contrast.
524
+ """
525
+ if not isinstance(dataset_ids, list) or len(dataset_ids) < 2:
526
+ return _refusal(
527
+ "Early integration needs >=2 dataset_ids; got "
528
+ f"{len(dataset_ids) if isinstance(dataset_ids, list) else 'a non-list'}."
529
+ )
530
+
531
+ try:
532
+ plan = get_integration_plan(
533
+ dataset_ids,
534
+ design_factor=design_factor,
535
+ test_group=test_group,
536
+ control_group=control_group,
537
+ )
538
+ except Exception as exc: # unknown dataset id, etc.
539
+ return _refusal(f"Could not compute the integration plan: {exc}")
540
+
541
+ mode = plan.get("mode")
542
+ if mode != "early":
543
+ reason = plan.get("reason", "")
544
+ reroute = {
545
+ "late": "Run each dataset separately, then call decoupler_meta_analyze.",
546
+ "concordance": (
547
+ "These are same-cohort variants — call "
548
+ "decoupler_normalization_concordance instead."
549
+ ),
550
+ }.get(mode, "")
551
+ return _refusal(
552
+ (
553
+ f"Datasets are not eligible for early pooling (plan mode='{mode}'). "
554
+ f"{reason} {reroute}"
555
+ ).strip(),
556
+ mode=mode,
557
+ plan_reason=reason,
558
+ refusal_rules_triggered=plan.get("refusal_rules_triggered", []),
559
+ )
560
+
561
+ # Derive the DE method from the poolable data_level unless explicitly set.
562
+ if method == "auto":
563
+ method = "deseq2" if plan.get("poolable_data_level") == "raw_counts" else "limma"
564
+
565
+ try:
566
+ built = build_combined_anndata(
567
+ dataset_ids,
568
+ design_factor=design_factor,
569
+ batch_key="batch",
570
+ min_shared_features=min_shared_features,
571
+ out_prefix=(f"{out_prefix}_combined" if out_prefix else None),
572
+ )
573
+ except Exception as exc:
574
+ return _refusal(f"Could not build the pooled dataset: {exc}", mode="early")
575
+
576
+ try:
577
+ de = decoupler_differential_expression(
578
+ adata_path=built["output_path"],
579
+ design_factor=design_factor,
580
+ contrast=[design_factor, test_group, control_group],
581
+ method=method,
582
+ batch_column="batch",
583
+ out_prefix=out_prefix,
584
+ )
585
+ except Exception as exc:
586
+ return _refusal(
587
+ f"Pooled DE failed: {exc}", mode="early", combined=built, method=method
588
+ )
589
+
590
+ return {
591
+ "message": (
592
+ f"Early integration of {len(dataset_ids)} datasets "
593
+ f"({', '.join(dataset_ids)}): pooled {built['n_obs']} samples over "
594
+ f"{built['n_vars']} shared genes and ran {de.get('method_used')} with "
595
+ f"dataset modelled as a batch covariate. "
596
+ f"{de.get('n_significant')} significant genes."
597
+ ),
598
+ "mode": "early",
599
+ "plan_reason": plan.get("reason", ""),
600
+ "datasets": list(dataset_ids),
601
+ "batch_key": "batch",
602
+ "per_batch_n": built["per_batch_n"],
603
+ "n_shared_features": built["n_vars"],
604
+ "n_combined_samples": built["n_obs"],
605
+ "method_used": de.get("method_used"),
606
+ "n_significant": de.get("n_significant"),
607
+ "batch_modeled": True,
608
+ "combined_path": built["output_path"],
609
+ "de_results_path": de.get("output_path"),
610
+ "sanity_warnings": de.get("sanity_warnings"),
611
+ "interpretation_note": (
612
+ "Early-integration (pooled) result: the datasets were concatenated and "
613
+ "analyzed jointly with `dataset` as a batch covariate, so the group "
614
+ "effect is estimated adjusting for dataset. Report the datasets combined, "
615
+ "the per-dataset sample counts, and the batch handling. This is more "
616
+ "powerful than meta-analysis, but valid ONLY because the plan verdict was "
617
+ "'early' (shared feature space + poolable data_level)."
618
+ ),
619
+ "artifacts": [
620
+ {"description": "Pooled DE results (CSV)", "path": de.get("output_path")},
621
+ {"description": "Combined AnnData (h5ad)", "path": built["output_path"]},
622
+ ],
623
+ }
src/tools/rna/analysis.py CHANGED
@@ -8,6 +8,7 @@ from ._base import ( # noqa: F401
8
  )
9
  from src.core import combinable, make_combine_descriptor
10
  from src.workflows.sanity_checks import run_sanity_checks
 
11
 
12
  # ── ADR-0001 Mode-B combine conformance (see src/core/combine.py) ─────────
13
  # Module-level so they validate at import: a bad result_type or envelope value
@@ -59,6 +60,15 @@ def decoupler_differential_expression(
59
  "when limma is unavailable so the statistical method is never silently changed. "
60
  "Set to True only to explicitly allow t-test fallback during exploratory analysis.",
61
  ] = False,
 
 
 
 
 
 
 
 
 
62
  n_cpus: Annotated[int, "Number of CPUs for parallel processing (DESeq2 only)"] = 8,
63
  refit_cooks: Annotated[bool, "Whether to refit Cook's distances (DESeq2 only)"] = True,
64
  top_genes: Annotated[int, "Number of top genes to show in volcano plot"] = 10,
@@ -105,6 +115,13 @@ def decoupler_differential_expression(
105
  f"Available columns: {list(adata.obs.columns)}"
106
  )
107
 
 
 
 
 
 
 
 
108
  # Guard: reject deseq2 on pre-normalised data before any expensive computation
109
  if method == "deseq2":
110
  _X = adata.X
@@ -135,7 +152,9 @@ def decoupler_differential_expression(
135
  inference = DefaultInference(n_cpus=n_cpus)
136
  dds = DeseqDataSet(
137
  adata=adata,
138
- design=f"~{design_factor}",
 
 
139
  refit_cooks=refit_cooks,
140
  inference=inference,
141
  )
@@ -155,27 +174,47 @@ def decoupler_differential_expression(
155
  X = X.toarray()
156
  mask_test = adata.obs[design_factor] == contrast[1]
157
  mask_ctrl = adata.obs[design_factor] == contrast[2]
158
- X_test = X[mask_test.values]
159
- X_ctrl = X[mask_ctrl.values]
160
  genes = list(adata.var.index)
161
- if method == "limma":
162
- try:
163
- results_df = run_limma(X_test, X_ctrl, genes, contrast[1], contrast[2])
164
- method_label = "limma"
165
- except Exception as e:
166
- if fallback_to_ttest:
167
- results_df = run_welch_ttest(X_test, X_ctrl, genes)
168
- method_label = f"ttest (limma unavailable, fallback requested: {e})"
169
- else:
170
- raise RuntimeError(
171
- "method='limma' was requested but limma is unavailable: "
172
- f"{e}. "
173
- "Run decoupler_check_limma_available() to diagnose. "
174
- "Set fallback_to_ttest=True to explicitly allow t-test fallback."
175
- ) from e
 
 
 
 
176
  else:
177
- results_df = run_welch_ttest(X_test, X_ctrl, genes)
178
- method_label = "ttest"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
179
 
180
  # Save results — name the index so pd.read_csv produces a 'gene' column
181
  results_df.index.name = "gene"
 
8
  )
9
  from src.core import combinable, make_combine_descriptor
10
  from src.workflows.sanity_checks import run_sanity_checks
11
+ from src.workflows.microarray import run_limma_covariate
12
 
13
  # ── ADR-0001 Mode-B combine conformance (see src/core/combine.py) ─────────
14
  # Module-level so they validate at import: a bad result_type or envelope value
 
60
  "when limma is unavailable so the statistical method is never silently changed. "
61
  "Set to True only to explicitly allow t-test fallback during exploratory analysis.",
62
  ] = False,
63
+ batch_column: Annotated[
64
+ str | None,
65
+ "Optional obs column to model as a batch covariate for cross-dataset early "
66
+ "integration (e.g. 'batch' in a combined AnnData from "
67
+ "decoupler_integrate_datasets). When set: DESeq2 fits ~batch + factor; "
68
+ "limma fits ~batch + group (the group effect is estimated adjusting for "
69
+ "batch); method='ttest' is rejected (a t-test cannot model a covariate). "
70
+ "Default None = single-dataset analysis (behaviour unchanged).",
71
+ ] = None,
72
  n_cpus: Annotated[int, "Number of CPUs for parallel processing (DESeq2 only)"] = 8,
73
  refit_cooks: Annotated[bool, "Whether to refit Cook's distances (DESeq2 only)"] = True,
74
  top_genes: Annotated[int, "Number of top genes to show in volcano plot"] = 10,
 
115
  f"Available columns: {list(adata.obs.columns)}"
116
  )
117
 
118
+ # Batch covariate (Mode-A early integration): the column must exist.
119
+ if batch_column is not None and batch_column not in adata.obs.columns:
120
+ raise ValueError(
121
+ f"batch_column '{batch_column}' not found in obs. "
122
+ f"Available columns: {list(adata.obs.columns)}"
123
+ )
124
+
125
  # Guard: reject deseq2 on pre-normalised data before any expensive computation
126
  if method == "deseq2":
127
  _X = adata.X
 
152
  inference = DefaultInference(n_cpus=n_cpus)
153
  dds = DeseqDataSet(
154
  adata=adata,
155
+ design=(
156
+ f"~{batch_column} + {design_factor}" if batch_column else f"~{design_factor}"
157
+ ),
158
  refit_cooks=refit_cooks,
159
  inference=inference,
160
  )
 
174
  X = X.toarray()
175
  mask_test = adata.obs[design_factor] == contrast[1]
176
  mask_ctrl = adata.obs[design_factor] == contrast[2]
 
 
177
  genes = list(adata.var.index)
178
+
179
+ if batch_column is not None:
180
+ # Mode-A early integration: model batch as a covariate (limma only).
181
+ # Do NOT fall back to ttest here — a t-test cannot honour the
182
+ # covariate, so silently dropping it would be wrong; surface instead.
183
+ if method != "limma":
184
+ raise ValueError(
185
+ f"batch_column requires method='limma' (or method='deseq2' for raw "
186
+ f"counts); method='{method}' cannot model a batch covariate. Use "
187
+ f"limma/deseq2, or batch-correct (e.g. ComBat) before a t-test."
188
+ )
189
+ mask_both = (mask_test | mask_ctrl).values
190
+ X_both = X[mask_both]
191
+ groups = adata.obs[design_factor][mask_both].astype(str).tolist()
192
+ batches = adata.obs[batch_column][mask_both].astype(str).tolist()
193
+ results_df = run_limma_covariate(
194
+ X_both, groups, batches, genes, contrast[1], contrast[2]
195
+ )
196
+ method_label = f"limma (~{batch_column} + {design_factor})"
197
  else:
198
+ X_test = X[mask_test.values]
199
+ X_ctrl = X[mask_ctrl.values]
200
+ if method == "limma":
201
+ try:
202
+ results_df = run_limma(X_test, X_ctrl, genes, contrast[1], contrast[2])
203
+ method_label = "limma"
204
+ except Exception as e:
205
+ if fallback_to_ttest:
206
+ results_df = run_welch_ttest(X_test, X_ctrl, genes)
207
+ method_label = f"ttest (limma unavailable, fallback requested: {e})"
208
+ else:
209
+ raise RuntimeError(
210
+ "method='limma' was requested but limma is unavailable: "
211
+ f"{e}. "
212
+ "Run decoupler_check_limma_available() to diagnose. "
213
+ "Set fallback_to_ttest=True to explicitly allow t-test fallback."
214
+ ) from e
215
+ else:
216
+ results_df = run_welch_ttest(X_test, X_ctrl, genes)
217
+ method_label = "ttest"
218
 
219
  # Save results — name the index so pd.read_csv produces a 'gene' column
220
  results_df.index.name = "gene"
src/workflows/integration.py ADDED
@@ -0,0 +1,221 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Cross-dataset early integration (ADR-0001, Mode A — T7).
3
+
4
+ Build ONE combined AnnData from >=2 registered datasets by:
5
+ 1. harmonising each to a shared gene-symbol feature axis,
6
+ 2. intersecting features (join="inner"), and
7
+ 3. concatenating samples with a ``batch`` obs column = dataset_id.
8
+
9
+ The result is a *standard* AnnData that the existing single-dataset tools
10
+ (``decoupler_differential_expression`` and the enrichment tools) consume
11
+ unchanged — Mode A pools the matrices and a downstream DE models ``dataset_id``
12
+ as a batch covariate (see ``decoupler_differential_expression``'s
13
+ ``batch_column``, T8). Whether a set of datasets is eligible for early pooling is
14
+ decided upstream by ``get_integration_plan`` (mode == "early"); this module
15
+ performs the mechanics once that verdict is in (the gating lives in the
16
+ ``decoupler_integrate_datasets`` tool, T9).
17
+
18
+ Pure core: ``combine_anndatas(adatas, ...)`` operates on already-loaded AnnData
19
+ objects (unit-testable, no network). ``build_combined_anndata(dataset_ids, ...)``
20
+ is the thin loader wrapper that resolves each manifest's hosted h5ad.
21
+
22
+ v1 scope / limitations:
23
+ - Harmonises to gene symbols: a dataset whose ``feature_id_type`` is
24
+ "gene_symbol" uses var_names directly; otherwise a var column named
25
+ SYMBOL/gene_symbol/gene_name is used if present; anything else raises (probe
26
+ collapse / ortholog mapping / entrez without a symbol column are NOT handled
27
+ here — such datasets do not reach an ``early`` verdict in v1).
28
+ - Duplicate symbols within a dataset are collapsed keep-first (preserves raw
29
+ integer counts for the DESeq2 path — no averaging).
30
+ """
31
+ from __future__ import annotations
32
+
33
+ import os
34
+ import tempfile
35
+ import urllib.request
36
+ from pathlib import Path
37
+ from typing import TYPE_CHECKING, Optional, Sequence
38
+
39
+ import anndata as ad
40
+ import numpy as np
41
+
42
+ from src.tools.rna._base import OUTPUT_DIR
43
+
44
+ if TYPE_CHECKING:
45
+ from anndata import AnnData
46
+
47
+ # var columns checked, in order, for a gene-symbol axis when feature_id_type
48
+ # is not already "gene_symbol".
49
+ _SYMBOL_VAR_COLUMNS = ("SYMBOL", "gene_symbol", "Gene_Symbol", "symbol", "gene_name")
50
+
51
+
52
+ def _symbol_axis(adata: "AnnData", feature_id_type: str) -> list[str]:
53
+ """Return a per-var list of gene symbols for ``adata``, or raise if unavailable."""
54
+ if feature_id_type == "gene_symbol":
55
+ return [str(v) for v in adata.var_names]
56
+ for col in _SYMBOL_VAR_COLUMNS:
57
+ if col in adata.var.columns:
58
+ return [str(v) for v in adata.var[col].tolist()]
59
+ raise ValueError(
60
+ f"cannot resolve gene symbols: feature_id_type='{feature_id_type}' and no "
61
+ f"symbol column {list(_SYMBOL_VAR_COLUMNS)} in var. Mode-A v1 needs a "
62
+ f"gene-symbol axis (probe collapse / id-mapping is not handled here)."
63
+ )
64
+
65
+
66
+ def _to_symbol_adata(adata: "AnnData", feature_id_type: str) -> "AnnData":
67
+ """Return a copy of ``adata`` whose var_names are unique gene symbols.
68
+
69
+ Drops features with empty/missing symbols; collapses duplicate symbols
70
+ keep-first (preserves raw counts — no averaging)."""
71
+ symbols = _symbol_axis(adata, feature_id_type)
72
+ a = adata.copy()
73
+ a.var_names = [str(s) for s in symbols]
74
+ keep = np.array(
75
+ [bool(s) and s.strip().lower() not in ("nan", "none", "") for s in a.var_names]
76
+ )
77
+ a = a[:, keep].copy()
78
+ # keep-first on duplicate symbols (np.unique returns the first index per value)
79
+ _, first_idx = np.unique(np.asarray(a.var_names, dtype=object), return_index=True)
80
+ a = a[:, np.sort(first_idx)].copy()
81
+ return a
82
+
83
+
84
+ def combine_anndatas(
85
+ adatas: "Sequence[AnnData]",
86
+ dataset_ids: Sequence[str],
87
+ feature_id_types: Sequence[str],
88
+ *,
89
+ design_factor: Optional[str] = None,
90
+ batch_key: str = "batch",
91
+ min_shared_features: int = 200,
92
+ ) -> "AnnData":
93
+ """Pure core: feature-intersection concat of >=2 AnnData into one combined AnnData.
94
+
95
+ Each input is harmonised to a unique gene-symbol var axis, the var
96
+ intersection is taken (``join="inner"``), a ``batch_key`` obs column is set to
97
+ the dataset_id, and samples are concatenated (obs names disambiguated per
98
+ batch with a ``-<dataset_id>`` suffix).
99
+
100
+ Raises
101
+ ------
102
+ ValueError
103
+ on <2 inputs, mismatched argument lengths, an unresolvable symbol axis,
104
+ a missing ``design_factor`` column in any dataset, or a feature
105
+ intersection smaller than ``min_shared_features``.
106
+ """
107
+ if len(adatas) < 2:
108
+ raise ValueError(f"early integration needs >=2 datasets; got {len(adatas)}")
109
+ if not (len(adatas) == len(dataset_ids) == len(feature_id_types)):
110
+ raise ValueError(
111
+ "adatas, dataset_ids, and feature_id_types must be the same length "
112
+ f"({len(adatas)}, {len(dataset_ids)}, {len(feature_id_types)})"
113
+ )
114
+
115
+ prepared: list["AnnData"] = []
116
+ for a, did, fit in zip(adatas, dataset_ids, feature_id_types):
117
+ sa = _to_symbol_adata(a, fit)
118
+ if design_factor is not None and design_factor not in sa.obs.columns:
119
+ raise ValueError(
120
+ f"dataset '{did}' has no obs column '{design_factor}' required for "
121
+ f"the requested contrast."
122
+ )
123
+ prepared.append(sa)
124
+
125
+ shared = set(prepared[0].var_names)
126
+ for sa in prepared[1:]:
127
+ shared &= set(sa.var_names)
128
+ if len(shared) < min_shared_features:
129
+ raise ValueError(
130
+ f"only {len(shared)} shared gene symbols across {list(dataset_ids)} "
131
+ f"(minimum {min_shared_features}); the datasets likely do not share a "
132
+ f"feature space and are not eligible for early pooling."
133
+ )
134
+
135
+ combined = ad.concat(
136
+ prepared,
137
+ axis=0,
138
+ join="inner", # feature intersection
139
+ label=batch_key,
140
+ keys=[str(d) for d in dataset_ids],
141
+ index_unique="-", # disambiguate duplicate sample barcodes across cohorts
142
+ merge="same",
143
+ )
144
+ combined.obs[batch_key] = combined.obs[batch_key].astype("category")
145
+ return combined
146
+
147
+
148
+ def _resolve_to_local(url_or_path: str) -> tuple[str, bool]:
149
+ """Return (local_path, is_temp). Downloads http(s)/ftp URLs to a temp file."""
150
+ if str(url_or_path).startswith(("http://", "https://", "ftp://")):
151
+ suffix = Path(url_or_path.split("?", 1)[0]).suffix or ".h5ad"
152
+ fd, tmp = tempfile.mkstemp(suffix=suffix)
153
+ os.close(fd)
154
+ urllib.request.urlretrieve(url_or_path, tmp)
155
+ return tmp, True
156
+ return str(url_or_path), False
157
+
158
+
159
+ def build_combined_anndata(
160
+ dataset_ids: Sequence[str],
161
+ *,
162
+ design_factor: Optional[str] = None,
163
+ batch_key: str = "batch",
164
+ min_shared_features: int = 200,
165
+ out_prefix: Optional[str] = None,
166
+ ) -> dict:
167
+ """Loader wrapper: resolve each registered dataset's hosted h5ad and combine.
168
+
169
+ Resolves each ``dataset_id`` to its manifest's ``expression_source.url``,
170
+ reads the AnnData, and delegates to :func:`combine_anndatas`. Writes the
171
+ combined AnnData to ``OUTPUT_DIR`` and returns a summary dict.
172
+ """
173
+ import scanpy as sc
174
+
175
+ from src.datasets.registry import load_manifest
176
+
177
+ if len(dataset_ids) < 2:
178
+ raise ValueError(f"early integration needs >=2 datasets; got {len(dataset_ids)}")
179
+
180
+ adatas: list["AnnData"] = []
181
+ feature_id_types: list[str] = []
182
+ temps: list[str] = []
183
+ try:
184
+ for did in dataset_ids:
185
+ m = load_manifest(did)
186
+ url = (m.expression_source or {}).get("url")
187
+ if not url:
188
+ raise ValueError(f"dataset '{did}' has no expression_source.url")
189
+ local, is_temp = _resolve_to_local(url)
190
+ if is_temp:
191
+ temps.append(local)
192
+ adatas.append(sc.read_h5ad(local))
193
+ feature_id_types.append(m.feature_id_type)
194
+
195
+ combined = combine_anndatas(
196
+ adatas,
197
+ dataset_ids,
198
+ feature_id_types,
199
+ design_factor=design_factor,
200
+ batch_key=batch_key,
201
+ min_shared_features=min_shared_features,
202
+ )
203
+ finally:
204
+ for t in temps:
205
+ Path(t).unlink(missing_ok=True)
206
+
207
+ out_prefix = out_prefix or ("combined_" + "_".join(str(d) for d in dataset_ids))[:80]
208
+ OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
209
+ out_path = OUTPUT_DIR / f"{out_prefix}.h5ad"
210
+ combined.write_h5ad(out_path)
211
+
212
+ per_batch = combined.obs[batch_key].value_counts().to_dict()
213
+ return {
214
+ "output_path": str(out_path.resolve()),
215
+ "n_obs": int(combined.n_obs),
216
+ "n_vars": int(combined.n_vars),
217
+ "batch_key": batch_key,
218
+ "shared_feature_space": "gene_symbol",
219
+ "per_batch_n": {str(k): int(v) for k, v in per_batch.items()},
220
+ "dataset_ids": list(dataset_ids),
221
+ }
src/workflows/microarray.py CHANGED
@@ -926,3 +926,117 @@ cat("OK\\n")
926
  })
927
  top_df.index = genes
928
  return top_df[["log2FoldChange", "stat", "pvalue", "padj"]]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
926
  })
927
  top_df.index = genes
928
  return top_df[["log2FoldChange", "stat", "pvalue", "padj"]]
929
+
930
+
931
+ def run_limma_covariate(
932
+ X: "np.ndarray",
933
+ group_labels: "list[str]",
934
+ batch_labels: "list[str]",
935
+ genes: "list[str]",
936
+ test_group: str,
937
+ control_group: str,
938
+ ) -> "pd.DataFrame":
939
+ """
940
+ Limma with a batch covariate via Rscript — ``model.matrix(~ batch + group)``.
941
+
942
+ Mode-A early integration (ADR-0001 T8): when several cohorts are pooled into
943
+ one matrix, the per-cohort ``batch`` is modelled as a covariate so the group
944
+ effect is estimated *adjusting* for it. Group labels are recoded to ctrl/test
945
+ (control_group -> "ctrl", test_group -> "test") so the tested coefficient is
946
+ always "grouptest" regardless of the original label spelling; batch enters as
947
+ additional factor columns whose names do not matter (only the group
948
+ coefficient is read back).
949
+
950
+ Parameters
951
+ ----------
952
+ X : samples x genes matrix (rows = samples, aligned with group/batch labels).
953
+ group_labels, batch_labels : per-sample labels, length == X.shape[0].
954
+ genes : gene ids, length == X.shape[1].
955
+
956
+ Returns columns: log2FoldChange, stat, pvalue, padj (same schema as run_limma).
957
+ Raises RuntimeError if Rscript/limma is unavailable or the design is rank-
958
+ deficient (e.g. batch perfectly confounded with group).
959
+ """
960
+ import os
961
+ import subprocess
962
+ import tempfile
963
+
964
+ X = np.asarray(X, dtype=np.float64)
965
+ n = X.shape[0]
966
+ if not (len(group_labels) == len(batch_labels) == n):
967
+ raise ValueError(
968
+ f"group_labels ({len(group_labels)}) and batch_labels "
969
+ f"({len(batch_labels)}) must match X sample count ({n})"
970
+ )
971
+ coded = ["test" if str(g) == str(test_group) else "ctrl" for g in group_labels]
972
+ if len(set(coded)) < 2:
973
+ raise ValueError(
974
+ f"need both groups present; got only {set(group_labels)} for "
975
+ f"test='{test_group}' / control='{control_group}'"
976
+ )
977
+
978
+ expr = X.T # genes x samples
979
+ with tempfile.TemporaryDirectory() as tmpdir:
980
+ expr_csv = os.path.join(tmpdir, "expr.csv")
981
+ meta_csv = os.path.join(tmpdir, "meta.csv")
982
+ result_csv = os.path.join(tmpdir, "result.csv")
983
+
984
+ pd.DataFrame(expr, index=genes).to_csv(expr_csv, header=False)
985
+ pd.DataFrame(
986
+ {"group": coded, "batch": [str(b) for b in batch_labels]}
987
+ ).to_csv(meta_csv, index=False)
988
+
989
+ r_script = f"""
990
+ suppressPackageStartupMessages({{
991
+ library(limma)
992
+ library(utils)
993
+ }})
994
+
995
+ expr <- as.matrix(read.csv("{expr_csv}", header=FALSE, row.names=1))
996
+ meta <- read.csv("{meta_csv}", colClasses = "character")
997
+ group <- factor(meta$group, levels = c("ctrl", "test"))
998
+ batch <- factor(meta$batch)
999
+ design <- model.matrix(~ batch + group)
1000
+ if (qr(design)$rank < ncol(design)) {{
1001
+ stop("design is rank-deficient (batch likely confounded with group)")
1002
+ }}
1003
+
1004
+ fit <- lmFit(expr, design)
1005
+ fit <- eBayes(fit)
1006
+ result <- topTable(fit, coef = "grouptest",
1007
+ number = nrow(expr),
1008
+ sort.by = "none",
1009
+ adjust.method = "BH")
1010
+
1011
+ write.csv(result, "{result_csv}", row.names = TRUE)
1012
+ cat("OK\\n")
1013
+ """
1014
+ proc = subprocess.run(
1015
+ ["Rscript", "--vanilla", "-"],
1016
+ input=r_script,
1017
+ capture_output=True,
1018
+ text=True,
1019
+ timeout=300,
1020
+ )
1021
+
1022
+ if proc.returncode != 0:
1023
+ raise RuntimeError(
1024
+ f"Rscript/limma (covariate) failed (exit {proc.returncode}):\n"
1025
+ f"{proc.stderr.strip()}"
1026
+ )
1027
+ if not os.path.exists(result_csv):
1028
+ raise RuntimeError(
1029
+ f"Rscript ran but produced no output.\nstdout: {proc.stdout}\n"
1030
+ f"stderr: {proc.stderr}"
1031
+ )
1032
+
1033
+ top_df = pd.read_csv(result_csv, index_col=0)
1034
+
1035
+ top_df = top_df.rename(columns={
1036
+ "logFC": "log2FoldChange",
1037
+ "t": "stat",
1038
+ "P.Value": "pvalue",
1039
+ "adj.P.Val": "padj",
1040
+ })
1041
+ top_df.index = genes
1042
+ return top_df[["log2FoldChange", "stat", "pvalue", "padj"]]
tests/test_de_batch_covariate.py ADDED
@@ -0,0 +1,115 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Tests for the batch-covariate DE path (ADR-0001 T8).
3
+
4
+ Covers:
5
+ - run_limma_covariate output schema (skipped if Rscript/limma is unavailable),
6
+ - the decoupler_differential_expression guards: method='ttest' + batch_column is
7
+ rejected, and a missing batch_column is rejected,
8
+ - back-compat: batch_column=None still runs the plain two-group ttest path.
9
+
10
+ Tiny synthetic AnnData only — no network. The DESeq2 covariate path (a one-line
11
+ design change) is exercised by the existing DESeq2 tests + the design string;
12
+ it is not re-run here (needs raw counts + the pydeseq2 stack).
13
+ """
14
+ from __future__ import annotations
15
+
16
+ import sys
17
+ from pathlib import Path
18
+
19
+ import anndata as ad
20
+ import numpy as np
21
+ import pandas as pd
22
+ import pytest
23
+
24
+ sys.path.insert(0, str(Path(__file__).parent.parent))
25
+
26
+ from src.workflows.microarray import run_limma_covariate # noqa: E402
27
+ from src.tools.rna.analysis import decoupler_differential_expression # noqa: E402
28
+
29
+
30
+ def _rscript_limma_available() -> bool:
31
+ """True if a tiny run_limma_covariate succeeds (Rscript + limma installed)."""
32
+ rng = np.random.default_rng(0)
33
+ X = rng.normal(size=(6, 4))
34
+ X[:3, :] += 3.0 # group effect on the 'test' samples
35
+ try:
36
+ run_limma_covariate(
37
+ X,
38
+ ["t", "t", "t", "c", "c", "c"],
39
+ ["b1", "b2", "b1", "b2", "b1", "b2"],
40
+ ["G1", "G2", "G3", "G4"],
41
+ "t",
42
+ "c",
43
+ )
44
+ return True
45
+ except RuntimeError:
46
+ return False
47
+ except Exception:
48
+ return False
49
+
50
+
51
+ def _h5ad(tmp_path, *, with_batch=True, n_genes=5):
52
+ rng = np.random.default_rng(1)
53
+ n = 6
54
+ X = rng.normal(loc=5.0, scale=1.0, size=(n, n_genes)).astype(np.float32)
55
+ obs = pd.DataFrame(
56
+ {"condition": ["treatment", "treatment", "treatment", "control", "control", "control"]},
57
+ index=[f"s{i}" for i in range(n)],
58
+ )
59
+ if with_batch:
60
+ obs["batch"] = ["A", "B", "A", "B", "A", "B"]
61
+ var = pd.DataFrame(index=[f"G{i}" for i in range(n_genes)])
62
+ a = ad.AnnData(X=X, obs=obs, var=var)
63
+ p = tmp_path / "ds.h5ad"
64
+ a.write_h5ad(p)
65
+ return str(p)
66
+
67
+
68
+ @pytest.mark.skipif(not _rscript_limma_available(), reason="Rscript/limma not available")
69
+ def test_run_limma_covariate_schema():
70
+ rng = np.random.default_rng(2)
71
+ X = rng.normal(size=(8, 10))
72
+ X[:4, :] += 2.5 # test-group effect
73
+ genes = [f"G{i}" for i in range(10)]
74
+ groups = ["t", "t", "t", "t", "c", "c", "c", "c"]
75
+ batches = ["b1", "b2", "b1", "b2", "b1", "b2", "b1", "b2"]
76
+ out = run_limma_covariate(X, groups, batches, genes, "t", "c")
77
+ assert list(out.columns) == ["log2FoldChange", "stat", "pvalue", "padj"]
78
+ assert list(out.index) == genes
79
+ assert out["stat"].notna().all()
80
+
81
+
82
+ def test_ttest_with_batch_rejected(tmp_path):
83
+ path = _h5ad(tmp_path, with_batch=True)
84
+ with pytest.raises(ValueError, match="cannot model a batch covariate"):
85
+ decoupler_differential_expression(
86
+ adata_path=path,
87
+ design_factor="condition",
88
+ contrast=["condition", "treatment", "control"],
89
+ method="ttest",
90
+ batch_column="batch",
91
+ )
92
+
93
+
94
+ def test_missing_batch_column_rejected(tmp_path):
95
+ path = _h5ad(tmp_path, with_batch=False)
96
+ with pytest.raises(ValueError, match="batch_column 'batch' not found"):
97
+ decoupler_differential_expression(
98
+ adata_path=path,
99
+ design_factor="condition",
100
+ contrast=["condition", "treatment", "control"],
101
+ method="limma",
102
+ batch_column="batch",
103
+ )
104
+
105
+
106
+ def test_backcompat_ttest_no_batch(tmp_path):
107
+ path = _h5ad(tmp_path, with_batch=True)
108
+ res = decoupler_differential_expression(
109
+ adata_path=path,
110
+ design_factor="condition",
111
+ contrast=["condition", "treatment", "control"],
112
+ method="ttest",
113
+ )
114
+ assert res["method_used"] == "ttest"
115
+ assert "output_path" in res
tests/test_integrate_datasets_tool.py ADDED
@@ -0,0 +1,153 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Tests for the decoupler_integrate_datasets MCP tool (ADR-0001 T9, Mode A).
3
+
4
+ The tool's value is the routing gate: it pools + runs a batch-aware DE ONLY on a
5
+ plan verdict of 'early', and refuses/reroutes otherwise. These tests monkeypatch
6
+ the plan, the combined-AnnData builder, and the DE tool so the routing is checked
7
+ deterministically with no network or compute. The actual pooling/DE mechanics
8
+ are covered by test_integration_mode_a.py and test_de_batch_covariate.py.
9
+ """
10
+ from __future__ import annotations
11
+
12
+ import asyncio
13
+ import sys
14
+ from pathlib import Path
15
+
16
+ sys.path.insert(0, str(Path(__file__).parent.parent))
17
+
18
+ import src.tools.integration_tools as it # noqa: E402
19
+ from src.tools.integration_tools import ( # noqa: E402
20
+ decoupler_integrate_datasets,
21
+ integration_mcp,
22
+ )
23
+
24
+
25
+ def _plan(mode, **extra):
26
+ base = {"mode": mode, "reason": f"{mode} reason.", "refusal_rules_triggered": []}
27
+ base.update(extra)
28
+ return lambda *a, **k: base
29
+
30
+
31
+ def test_tool_registered():
32
+ names = {t.name for t in asyncio.run(integration_mcp.list_tools())}
33
+ assert "decoupler_integrate_datasets" in names
34
+
35
+
36
+ def test_arity_guard():
37
+ out = decoupler_integrate_datasets(
38
+ dataset_ids=["solo"], design_factor="g", test_group="t", control_group="c"
39
+ )
40
+ assert out["refused"] is True
41
+ assert ">=2" in out["error"]
42
+
43
+
44
+ def test_late_reroutes_to_meta_analyze(monkeypatch):
45
+ monkeypatch.setattr(it, "get_integration_plan", _plan("late"))
46
+ out = decoupler_integrate_datasets(
47
+ dataset_ids=["a", "b"], design_factor="g", test_group="t", control_group="c"
48
+ )
49
+ assert out["refused"] and out["mode"] == "late"
50
+ assert "decoupler_meta_analyze" in out["error"]
51
+
52
+
53
+ def test_concordance_reroutes(monkeypatch):
54
+ monkeypatch.setattr(it, "get_integration_plan", _plan("concordance"))
55
+ out = decoupler_integrate_datasets(
56
+ dataset_ids=["a", "b"], design_factor="g", test_group="t", control_group="c"
57
+ )
58
+ assert out["refused"] and out["mode"] == "concordance"
59
+ assert "decoupler_normalization_concordance" in out["error"]
60
+
61
+
62
+ def test_refuse_passes_reason_and_rules(monkeypatch):
63
+ monkeypatch.setattr(
64
+ it, "get_integration_plan",
65
+ _plan("refuse", reason="CONFOUNDED_DESIGN: ...", refusal_rules_triggered=["CONFOUNDED_DESIGN"]),
66
+ )
67
+ out = decoupler_integrate_datasets(
68
+ dataset_ids=["a", "b"], design_factor="g", test_group="t", control_group="c"
69
+ )
70
+ assert out["refused"] and out["mode"] == "refuse"
71
+ assert "CONFOUNDED_DESIGN" in out["error"]
72
+ assert out["refusal_rules_triggered"] == ["CONFOUNDED_DESIGN"]
73
+
74
+
75
+ def test_plan_exception_refuses(monkeypatch):
76
+ def boom(*a, **k):
77
+ raise ValueError("dataset 'nope' not registered")
78
+ monkeypatch.setattr(it, "get_integration_plan", boom)
79
+ out = decoupler_integrate_datasets(
80
+ dataset_ids=["a", "nope"], design_factor="g", test_group="t", control_group="c"
81
+ )
82
+ assert out["refused"] and "not registered" in out["error"]
83
+
84
+
85
+ def test_early_builds_and_runs_batch_aware_de(monkeypatch, tmp_path):
86
+ monkeypatch.setattr(
87
+ it, "get_integration_plan", _plan("early", poolable_data_level="raw_counts")
88
+ )
89
+ built = {
90
+ "output_path": str(tmp_path / "combined.h5ad"),
91
+ "n_obs": 42, "n_vars": 1500, "per_batch_n": {"a": 20, "b": 22},
92
+ "dataset_ids": ["a", "b"],
93
+ }
94
+ monkeypatch.setattr(it, "build_combined_anndata", lambda *a, **k: built)
95
+ captured = {}
96
+
97
+ def fake_de(**kwargs):
98
+ captured.update(kwargs)
99
+ return {
100
+ "method_used": "DESeq2", "n_significant": 7,
101
+ "output_path": str(tmp_path / "de.csv"), "sanity_warnings": {"n_warnings": 0},
102
+ }
103
+
104
+ monkeypatch.setattr(it, "decoupler_differential_expression", fake_de)
105
+ out = decoupler_integrate_datasets(
106
+ dataset_ids=["a", "b"], design_factor="subtype",
107
+ test_group="basal", control_group="classical",
108
+ )
109
+ assert out["mode"] == "early" and out["batch_modeled"] is True
110
+ assert out["n_significant"] == 7 and out["method_used"] == "DESeq2"
111
+ assert out["per_batch_n"] == {"a": 20, "b": 22}
112
+ # auto -> deseq2 for raw counts; batch covariate + contrast wired correctly
113
+ assert captured["method"] == "deseq2"
114
+ assert captured["batch_column"] == "batch"
115
+ assert captured["contrast"] == ["subtype", "basal", "classical"]
116
+ assert captured["adata_path"] == built["output_path"]
117
+
118
+
119
+ def test_early_auto_method_limma_for_non_raw(monkeypatch, tmp_path):
120
+ monkeypatch.setattr(
121
+ it, "get_integration_plan", _plan("early", poolable_data_level="log_expression")
122
+ )
123
+ monkeypatch.setattr(
124
+ it, "build_combined_anndata",
125
+ lambda *a, **k: {"output_path": str(tmp_path / "c.h5ad"), "n_obs": 10,
126
+ "n_vars": 900, "per_batch_n": {"a": 5, "b": 5}},
127
+ )
128
+ captured = {}
129
+
130
+ def fake_de(**kwargs):
131
+ captured.update(kwargs)
132
+ return {"method_used": "limma", "n_significant": 3,
133
+ "output_path": str(tmp_path / "de.csv")}
134
+
135
+ monkeypatch.setattr(it, "decoupler_differential_expression", fake_de)
136
+ decoupler_integrate_datasets(
137
+ dataset_ids=["a", "b"], design_factor="g", test_group="t", control_group="c"
138
+ )
139
+ assert captured["method"] == "limma"
140
+
141
+
142
+ def test_early_build_failure_refuses(monkeypatch):
143
+ monkeypatch.setattr(
144
+ it, "get_integration_plan", _plan("early", poolable_data_level="raw_counts")
145
+ )
146
+
147
+ def boom(*a, **k):
148
+ raise ValueError("only 12 shared gene symbols")
149
+ monkeypatch.setattr(it, "build_combined_anndata", boom)
150
+ out = decoupler_integrate_datasets(
151
+ dataset_ids=["a", "b"], design_factor="g", test_group="t", control_group="c"
152
+ )
153
+ assert out["refused"] and "shared gene symbols" in out["error"]
tests/test_integration_mode_a.py ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """
2
+ Unit tests for the Mode-A early-integration core (ADR-0001 T7).
3
+
4
+ Pure tests over constructed AnnData objects — no data, no network. Cover the
5
+ feature-intersection concat, the gene-symbol harmonisation (var.index vs a
6
+ symbol column), keep-first duplicate collapse, empty-symbol dropping, the
7
+ batch obs key, and the arity / shared-feature / design-factor guards.
8
+ """
9
+ from __future__ import annotations
10
+
11
+ import sys
12
+ from pathlib import Path
13
+
14
+ import anndata as ad
15
+ import numpy as np
16
+ import pandas as pd
17
+ import pytest
18
+
19
+ sys.path.insert(0, str(Path(__file__).parent.parent))
20
+
21
+ from src.workflows.integration import combine_anndatas # noqa: E402
22
+
23
+
24
+ def _adata(symbols, *, n_obs=4, condition=None, symbol_col=None):
25
+ """Build a small AnnData. If symbol_col is given, var.index holds opaque ids
26
+ and the symbols live in that var column (mimics ensembl + SYMBOL)."""
27
+ X = (np.arange(n_obs * len(symbols), dtype=float).reshape(n_obs, len(symbols)) + 1.0)
28
+ if symbol_col:
29
+ var = pd.DataFrame(index=[f"id{i}" for i in range(len(symbols))])
30
+ var[symbol_col] = list(symbols)
31
+ else:
32
+ var = pd.DataFrame(index=list(symbols))
33
+ obs = pd.DataFrame(index=[f"s{i}" for i in range(n_obs)])
34
+ if condition is not None:
35
+ obs["condition"] = list(condition)
36
+ return ad.AnnData(X=X, obs=obs, var=var)
37
+
38
+
39
+ def test_intersection_concat_and_batch_key():
40
+ a = _adata(["G1", "G2", "G3"], n_obs=3)
41
+ b = _adata(["G2", "G3", "G4"], n_obs=2)
42
+ out = combine_anndatas(
43
+ [a, b], ["A", "B"], ["gene_symbol", "gene_symbol"], min_shared_features=1
44
+ )
45
+ assert set(out.var_names) == {"G2", "G3"} # feature intersection
46
+ assert out.n_obs == 5
47
+ assert out.obs["batch"].value_counts().to_dict() == {"A": 3, "B": 2}
48
+ # obs names disambiguated per batch
49
+ assert any(name.endswith("-A") for name in out.obs_names)
50
+ assert any(name.endswith("-B") for name in out.obs_names)
51
+
52
+
53
+ def test_symbol_column_used_when_not_gene_symbol():
54
+ a = _adata(["G1", "G2", "G3"], symbol_col="SYMBOL") # var.index = id0.. , SYMBOL col
55
+ b = _adata(["G2", "G3"], symbol_col="SYMBOL")
56
+ out = combine_anndatas(
57
+ [a, b], ["A", "B"], ["ensembl_gene_id", "ensembl_gene_id"], min_shared_features=1
58
+ )
59
+ assert set(out.var_names) == {"G2", "G3"}
60
+
61
+
62
+ def test_duplicate_symbols_collapsed_keep_first():
63
+ a = _adata(["G1", "G1", "G2"], n_obs=2) # G1 duplicated
64
+ b = _adata(["G1", "G2"], n_obs=2)
65
+ out = combine_anndatas([a, b], ["A", "B"], ["gene_symbol", "gene_symbol"], min_shared_features=1)
66
+ # G1 deduped within A keep-first, then intersected
67
+ assert sorted(out.var_names) == ["G1", "G2"]
68
+
69
+
70
+ def test_empty_and_nan_symbols_dropped():
71
+ a = _adata(["G1", "", "G2", "nan"], n_obs=2)
72
+ b = _adata(["G1", "G2"], n_obs=2)
73
+ out = combine_anndatas([a, b], ["A", "B"], ["gene_symbol", "gene_symbol"], min_shared_features=1)
74
+ assert sorted(out.var_names) == ["G1", "G2"]
75
+
76
+
77
+ def test_arity_guard():
78
+ a = _adata(["G1", "G2"])
79
+ with pytest.raises(ValueError, match="needs >=2"):
80
+ combine_anndatas([a], ["A"], ["gene_symbol"])
81
+
82
+
83
+ def test_too_few_shared_features_raises():
84
+ a = _adata(["G1", "G2", "G3"])
85
+ b = _adata(["G4", "G5", "G6"]) # disjoint
86
+ with pytest.raises(ValueError, match="shared gene symbols"):
87
+ combine_anndatas([a, b], ["A", "B"], ["gene_symbol", "gene_symbol"], min_shared_features=1)
88
+
89
+
90
+ def test_missing_design_factor_raises():
91
+ a = _adata(["G1", "G2"], condition=["t", "c", "t", "c"])
92
+ b = _adata(["G1", "G2"]) # no condition column
93
+ with pytest.raises(ValueError, match="no obs column 'condition'"):
94
+ combine_anndatas(
95
+ [a, b], ["A", "B"], ["gene_symbol", "gene_symbol"],
96
+ design_factor="condition", min_shared_features=1,
97
+ )
98
+
99
+
100
+ def test_design_factor_present_ok():
101
+ a = _adata(["G1", "G2"], condition=["t", "c", "t", "c"])
102
+ b = _adata(["G1", "G2"], condition=["t", "c", "t", "c"])
103
+ out = combine_anndatas(
104
+ [a, b], ["A", "B"], ["gene_symbol", "gene_symbol"],
105
+ design_factor="condition", min_shared_features=1,
106
+ )
107
+ assert "condition" in out.obs.columns
108
+ assert "batch" in out.obs.columns
109
+
110
+
111
+ def test_unresolvable_symbol_axis_raises():
112
+ a = _adata(["G1", "G2"], symbol_col="SYMBOL")
113
+ b = _adata(["G1", "G2"], symbol_col="SYMBOL")
114
+ # feature_id_type not gene_symbol AND no recognised symbol column -> drop the col
115
+ a.var = a.var.rename(columns={"SYMBOL": "mystery"})
116
+ b.var = b.var.rename(columns={"SYMBOL": "mystery"})
117
+ with pytest.raises(ValueError, match="cannot resolve gene symbols"):
118
+ combine_anndatas([a, b], ["A", "B"], ["probe_id", "probe_id"], min_shared_features=1)