ADR-0001 Phase 2 Mode A: early integration (T7 + T8 + T9)
Browse filesT7 — src/workflows/integration.py: combined-AnnData builder. Harmonise each
dataset to a gene-symbol axis, intersect features, concat with a `batch` obs key
(=dataset_id). Pure core combine_anndatas() + loader wrapper
build_combined_anndata() (resolves each manifest's hosted h5ad).
T8 — decoupler_differential_expression gains batch_column: DESeq2 fits
~batch+factor; new run_limma_covariate (~batch+group) in microarray.py; ttest+batch
is refused (a t-test cannot model a covariate; no silent drop). Default None keeps
single-dataset behaviour unchanged.
T9 — decoupler_integrate_datasets (integration_mcp, 3rd tool): re-checks
get_integration_plan and pools + runs ONE batch-aware DE only on mode=="early";
refuses/reroutes for late (meta_analyze) / concordance / refuse. Auto-selects
deseq2 (raw counts) or limma from the plan's poolable_data_level.
Tests: test_integration_mode_a (9), test_de_batch_covariate (4),
test_integrate_datasets_tool (9). Full non-live suite: 915 passed. v1 limits:
gene-symbol axis only (no probe collapse / ortholog mapping), ComBat (step 11)
deferred. Branch-local; NOT deployed (origin = prod Space).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- docs/adr/ADR-0001-implementation-plan.md +10 -5
- src/tools/integration_tools.py +162 -3
- src/tools/rna/analysis.py +59 -20
- src/workflows/integration.py +221 -0
- src/workflows/microarray.py +114 -0
- tests/test_de_batch_covariate.py +115 -0
- tests/test_integrate_datasets_tool.py +153 -0
- tests/test_integration_mode_a.py +118 -0
|
@@ -1,7 +1,7 @@
|
|
| 1 |
# ADR-0001 Implementation Plan — Cross-Dataset Analysis
|
| 2 |
|
| 3 |
**Companion to:** [`ADR-0001-cross-dataset-integration.md`](./ADR-0001-cross-dataset-integration.md)
|
| 4 |
-
**Status:** Phase 1 (Mode B) complete & shipped to prod 2026-06-19 (T1–T6); Phase 2 (Mode A: T7–T9) not
|
| 5 |
**Scope:** `biodata-registry`, `DecoupleRpy_Agent`, `pdac-analysis-orchestrator`
|
| 6 |
|
| 7 |
This plan turns the ADR's three phases into file-level work, grounded in the
|
|
@@ -115,17 +115,22 @@ pins `get_integration_plan`'s logic (the D3 poolability rule).
|
|
| 115 |
|
| 116 |
## Phase 2 — Mode A (early integration)
|
| 117 |
|
| 118 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 119 |
→ emit one **standard combined AnnData** with a `batch` obs key
|
| 120 |
(`dataset_id`). Existing single-dataset tools run on it unchanged.
|
| 121 |
-
9. [
|
| 122 |
`decoupler_differential_expression` with an optional `batch_column`:
|
| 123 |
DESeq2 `~batch + condition` (one-liner); **refactor `run_limma` to a design
|
| 124 |
matrix** including batch; ttest path uses pre-correction instead. Back-compat
|
| 125 |
default off.
|
| 126 |
-
10. [
|
| 127 |
batch-confound refusal from `get_integration_plan`.
|
| 128 |
-
11. [ ] *(Optional)* add `inmoose` (ComBat / ComBat-seq) for the pre-correction
|
| 129 |
route; keep the dependency optional.
|
| 130 |
|
| 131 |
> **No per-tool integration work.** A future tool gains Mode A by reading the
|
|
|
|
| 1 |
# ADR-0001 Implementation Plan — Cross-Dataset Analysis
|
| 2 |
|
| 3 |
**Companion to:** [`ADR-0001-cross-dataset-integration.md`](./ADR-0001-cross-dataset-integration.md)
|
| 4 |
+
**Status:** Phase 1 (Mode B) complete & shipped to prod 2026-06-19 (T1–T6); Phase 2 (Mode A: T7–T9) built + tested 2026-06-24 on branch `adr-0001-phase-2-mode-a`, not yet deployed (`early` still falls back to `late` in prod until then); Phase 3 (routing/interpretation/eval) pending. **Concordance mode** (same-cohort sibling quantifications) added later in biodata-registry 0.1.6 — see the 2026-06-24 update below; agent re-pinned + **deployed to prod** (`fb2091e`, `origin/main`). Reconciled 2026-06-24.
|
| 5 |
**Scope:** `biodata-registry`, `DecoupleRpy_Agent`, `pdac-analysis-orchestrator`
|
| 6 |
|
| 7 |
This plan turns the ADR's three phases into file-level work, grounded in the
|
|
|
|
| 115 |
|
| 116 |
## Phase 2 — Mode A (early integration)
|
| 117 |
|
| 118 |
+
> **Built + tested 2026-06-24** on branch `adr-0001-phase-2-mode-a` (T7–T9; 915
|
| 119 |
+
> non-live tests green). v1 harmonises to a gene-symbol axis (probe collapse /
|
| 120 |
+
> ortholog mapping not handled), ttest+batch is refused (no silent ComBat), and
|
| 121 |
+
> step 11 (ComBat) is deferred. **Not yet deployed** (origin = prod Space).
|
| 122 |
+
|
| 123 |
+
8. [x] **`src/workflows/integration.py`** — feature intersection + AnnData concat
|
| 124 |
→ emit one **standard combined AnnData** with a `batch` obs key
|
| 125 |
(`dataset_id`). Existing single-dataset tools run on it unchanged.
|
| 126 |
+
9. [x] **`supports_covariate` capability (D5)** — extend
|
| 127 |
`decoupler_differential_expression` with an optional `batch_column`:
|
| 128 |
DESeq2 `~batch + condition` (one-liner); **refactor `run_limma` to a design
|
| 129 |
matrix** including batch; ttest path uses pre-correction instead. Back-compat
|
| 130 |
default off.
|
| 131 |
+
10. [x] **`decoupler_integrate_datasets`** Layer 1 tool, gated behind the
|
| 132 |
batch-confound refusal from `get_integration_plan`.
|
| 133 |
+
11. [ ] *(Optional — deferred, not in v1)* add `inmoose` (ComBat / ComBat-seq) for the pre-correction
|
| 134 |
route; keep the dependency optional.
|
| 135 |
|
| 136 |
> **No per-tool integration work.** A future tool gains Mode A by reading the
|
|
@@ -15,8 +15,11 @@ This MCP server exposes the cross-dataset result-combination paths:
|
|
| 15 |
call overlap), because meta-analyzing identical samples double-counts the
|
| 16 |
cohort. Routed here by get_integration_plan's mode == "concordance".
|
| 17 |
|
| 18 |
-
|
| 19 |
-
(
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
Design (ADR-0001 Mode B):
|
| 22 |
- The *math* is generic over the result envelope and lives in
|
|
@@ -32,7 +35,7 @@ from __future__ import annotations
|
|
| 32 |
|
| 33 |
from datetime import datetime
|
| 34 |
from pathlib import Path
|
| 35 |
-
from typing import Annotated, Optional
|
| 36 |
|
| 37 |
import pandas as pd
|
| 38 |
from fastmcp import FastMCP
|
|
@@ -45,6 +48,11 @@ from src.core.combine import (
|
|
| 45 |
from src.tools.rna._base import OUTPUT_DIR
|
| 46 |
from src.workflows.concordance import concordance_metrics
|
| 47 |
from src.workflows.meta_analysis import combine_envelopes
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
|
| 49 |
integration_mcp = FastMCP(name="integration")
|
| 50 |
|
|
@@ -462,3 +470,154 @@ def decoupler_normalization_concordance(
|
|
| 462 |
}
|
| 463 |
],
|
| 464 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
call overlap), because meta-analyzing identical samples double-counts the
|
| 16 |
cohort. Routed here by get_integration_plan's mode == "concordance".
|
| 17 |
|
| 18 |
+
3. decoupler_integrate_datasets: *early integration* — POOL >=2 datasets into one
|
| 19 |
+
matrix (feature intersection + a `batch` obs key) and run a single batch-aware
|
| 20 |
+
DE. Only for datasets the plan rules "early" (shared feature space + poolable
|
| 21 |
+
data_level); it re-checks the plan and refuses/reroutes otherwise. (ADR-0001
|
| 22 |
+
T9, Mode A.)
|
| 23 |
|
| 24 |
Design (ADR-0001 Mode B):
|
| 25 |
- The *math* is generic over the result envelope and lives in
|
|
|
|
| 35 |
|
| 36 |
from datetime import datetime
|
| 37 |
from pathlib import Path
|
| 38 |
+
from typing import Annotated, Literal, Optional
|
| 39 |
|
| 40 |
import pandas as pd
|
| 41 |
from fastmcp import FastMCP
|
|
|
|
| 48 |
from src.tools.rna._base import OUTPUT_DIR
|
| 49 |
from src.workflows.concordance import concordance_metrics
|
| 50 |
from src.workflows.meta_analysis import combine_envelopes
|
| 51 |
+
# Mode-A early integration (T9): the plan gate, the combined-AnnData builder, and
|
| 52 |
+
# the batch-aware DE tool. Module-level so tests can monkeypatch them.
|
| 53 |
+
from src.datasets.registry import get_integration_plan
|
| 54 |
+
from src.workflows.integration import build_combined_anndata
|
| 55 |
+
from src.tools.rna.analysis import decoupler_differential_expression
|
| 56 |
|
| 57 |
integration_mcp = FastMCP(name="integration")
|
| 58 |
|
|
|
|
| 470 |
}
|
| 471 |
],
|
| 472 |
}
|
| 473 |
+
|
| 474 |
+
|
| 475 |
+
@integration_mcp.tool
|
| 476 |
+
def decoupler_integrate_datasets(
|
| 477 |
+
dataset_ids: Annotated[
|
| 478 |
+
list[str],
|
| 479 |
+
"Two or more registered dataset IDs to POOL into one matrix and analyze "
|
| 480 |
+
"together (early integration), e.g. ['tcga_paad', 'paca_au_rnaseq'].",
|
| 481 |
+
],
|
| 482 |
+
design_factor: Annotated[
|
| 483 |
+
str,
|
| 484 |
+
"obs column holding the contrast groups; must exist in EVERY dataset "
|
| 485 |
+
"(e.g. 'tumor_subtype').",
|
| 486 |
+
],
|
| 487 |
+
test_group: Annotated[str, "Test arm of the contrast (a value of design_factor)."],
|
| 488 |
+
control_group: Annotated[str, "Control / reference arm of the contrast."],
|
| 489 |
+
method: Annotated[
|
| 490 |
+
Literal["auto", "deseq2", "limma"],
|
| 491 |
+
"DE method on the pooled matrix. 'auto' (default) picks deseq2 for pooled "
|
| 492 |
+
"raw counts and limma otherwise, from the integration plan's poolable "
|
| 493 |
+
"data_level. `dataset` is always modelled as a batch covariate.",
|
| 494 |
+
] = "auto",
|
| 495 |
+
min_shared_features: Annotated[
|
| 496 |
+
int,
|
| 497 |
+
"Minimum shared gene symbols required to pool; below this the datasets "
|
| 498 |
+
"likely do not share a feature space and the request is refused.",
|
| 499 |
+
] = 200,
|
| 500 |
+
out_prefix: Annotated[Optional[str], "Output file prefix."] = None,
|
| 501 |
+
) -> dict:
|
| 502 |
+
"""
|
| 503 |
+
Early integration (Mode A): pool >=2 datasets into one matrix + one batch-aware DE.
|
| 504 |
+
|
| 505 |
+
Use this ONLY when dataset_get_integration_plan returns mode=="early": the
|
| 506 |
+
datasets share a poolable feature space AND data_level, so they can be
|
| 507 |
+
concatenated and analyzed jointly with `dataset` as a batch covariate (more
|
| 508 |
+
power than meta-analysis when valid). This tool RE-CHECKS the plan itself and
|
| 509 |
+
refuses (does not pool) when the verdict is not 'early':
|
| 510 |
+
|
| 511 |
+
- 'late' -> run each dataset separately, then decoupler_meta_analyze.
|
| 512 |
+
- 'concordance' -> same-cohort variants -> decoupler_normalization_concordance.
|
| 513 |
+
- 'refuse' -> incompatible (e.g. confounded design / cross-modality).
|
| 514 |
+
|
| 515 |
+
On 'early' it builds the combined AnnData (feature intersection + a `batch`
|
| 516 |
+
obs key = dataset_id), runs decoupler_differential_expression with
|
| 517 |
+
batch_column='batch' (DESeq2 ~batch+factor for raw counts, limma ~batch+group
|
| 518 |
+
otherwise), and returns the pooled DE result plus per-dataset composition.
|
| 519 |
+
Returns an {"error": ..., "refused": True} dict (does not raise) so the agent
|
| 520 |
+
can surface the reason and reroute.
|
| 521 |
+
|
| 522 |
+
Pass design_factor + test_group + control_group: they drive the plan's
|
| 523 |
+
confound check AND the pooled contrast.
|
| 524 |
+
"""
|
| 525 |
+
if not isinstance(dataset_ids, list) or len(dataset_ids) < 2:
|
| 526 |
+
return _refusal(
|
| 527 |
+
"Early integration needs >=2 dataset_ids; got "
|
| 528 |
+
f"{len(dataset_ids) if isinstance(dataset_ids, list) else 'a non-list'}."
|
| 529 |
+
)
|
| 530 |
+
|
| 531 |
+
try:
|
| 532 |
+
plan = get_integration_plan(
|
| 533 |
+
dataset_ids,
|
| 534 |
+
design_factor=design_factor,
|
| 535 |
+
test_group=test_group,
|
| 536 |
+
control_group=control_group,
|
| 537 |
+
)
|
| 538 |
+
except Exception as exc: # unknown dataset id, etc.
|
| 539 |
+
return _refusal(f"Could not compute the integration plan: {exc}")
|
| 540 |
+
|
| 541 |
+
mode = plan.get("mode")
|
| 542 |
+
if mode != "early":
|
| 543 |
+
reason = plan.get("reason", "")
|
| 544 |
+
reroute = {
|
| 545 |
+
"late": "Run each dataset separately, then call decoupler_meta_analyze.",
|
| 546 |
+
"concordance": (
|
| 547 |
+
"These are same-cohort variants — call "
|
| 548 |
+
"decoupler_normalization_concordance instead."
|
| 549 |
+
),
|
| 550 |
+
}.get(mode, "")
|
| 551 |
+
return _refusal(
|
| 552 |
+
(
|
| 553 |
+
f"Datasets are not eligible for early pooling (plan mode='{mode}'). "
|
| 554 |
+
f"{reason} {reroute}"
|
| 555 |
+
).strip(),
|
| 556 |
+
mode=mode,
|
| 557 |
+
plan_reason=reason,
|
| 558 |
+
refusal_rules_triggered=plan.get("refusal_rules_triggered", []),
|
| 559 |
+
)
|
| 560 |
+
|
| 561 |
+
# Derive the DE method from the poolable data_level unless explicitly set.
|
| 562 |
+
if method == "auto":
|
| 563 |
+
method = "deseq2" if plan.get("poolable_data_level") == "raw_counts" else "limma"
|
| 564 |
+
|
| 565 |
+
try:
|
| 566 |
+
built = build_combined_anndata(
|
| 567 |
+
dataset_ids,
|
| 568 |
+
design_factor=design_factor,
|
| 569 |
+
batch_key="batch",
|
| 570 |
+
min_shared_features=min_shared_features,
|
| 571 |
+
out_prefix=(f"{out_prefix}_combined" if out_prefix else None),
|
| 572 |
+
)
|
| 573 |
+
except Exception as exc:
|
| 574 |
+
return _refusal(f"Could not build the pooled dataset: {exc}", mode="early")
|
| 575 |
+
|
| 576 |
+
try:
|
| 577 |
+
de = decoupler_differential_expression(
|
| 578 |
+
adata_path=built["output_path"],
|
| 579 |
+
design_factor=design_factor,
|
| 580 |
+
contrast=[design_factor, test_group, control_group],
|
| 581 |
+
method=method,
|
| 582 |
+
batch_column="batch",
|
| 583 |
+
out_prefix=out_prefix,
|
| 584 |
+
)
|
| 585 |
+
except Exception as exc:
|
| 586 |
+
return _refusal(
|
| 587 |
+
f"Pooled DE failed: {exc}", mode="early", combined=built, method=method
|
| 588 |
+
)
|
| 589 |
+
|
| 590 |
+
return {
|
| 591 |
+
"message": (
|
| 592 |
+
f"Early integration of {len(dataset_ids)} datasets "
|
| 593 |
+
f"({', '.join(dataset_ids)}): pooled {built['n_obs']} samples over "
|
| 594 |
+
f"{built['n_vars']} shared genes and ran {de.get('method_used')} with "
|
| 595 |
+
f"dataset modelled as a batch covariate. "
|
| 596 |
+
f"{de.get('n_significant')} significant genes."
|
| 597 |
+
),
|
| 598 |
+
"mode": "early",
|
| 599 |
+
"plan_reason": plan.get("reason", ""),
|
| 600 |
+
"datasets": list(dataset_ids),
|
| 601 |
+
"batch_key": "batch",
|
| 602 |
+
"per_batch_n": built["per_batch_n"],
|
| 603 |
+
"n_shared_features": built["n_vars"],
|
| 604 |
+
"n_combined_samples": built["n_obs"],
|
| 605 |
+
"method_used": de.get("method_used"),
|
| 606 |
+
"n_significant": de.get("n_significant"),
|
| 607 |
+
"batch_modeled": True,
|
| 608 |
+
"combined_path": built["output_path"],
|
| 609 |
+
"de_results_path": de.get("output_path"),
|
| 610 |
+
"sanity_warnings": de.get("sanity_warnings"),
|
| 611 |
+
"interpretation_note": (
|
| 612 |
+
"Early-integration (pooled) result: the datasets were concatenated and "
|
| 613 |
+
"analyzed jointly with `dataset` as a batch covariate, so the group "
|
| 614 |
+
"effect is estimated adjusting for dataset. Report the datasets combined, "
|
| 615 |
+
"the per-dataset sample counts, and the batch handling. This is more "
|
| 616 |
+
"powerful than meta-analysis, but valid ONLY because the plan verdict was "
|
| 617 |
+
"'early' (shared feature space + poolable data_level)."
|
| 618 |
+
),
|
| 619 |
+
"artifacts": [
|
| 620 |
+
{"description": "Pooled DE results (CSV)", "path": de.get("output_path")},
|
| 621 |
+
{"description": "Combined AnnData (h5ad)", "path": built["output_path"]},
|
| 622 |
+
],
|
| 623 |
+
}
|
|
@@ -8,6 +8,7 @@ from ._base import ( # noqa: F401
|
|
| 8 |
)
|
| 9 |
from src.core import combinable, make_combine_descriptor
|
| 10 |
from src.workflows.sanity_checks import run_sanity_checks
|
|
|
|
| 11 |
|
| 12 |
# ── ADR-0001 Mode-B combine conformance (see src/core/combine.py) ─────────
|
| 13 |
# Module-level so they validate at import: a bad result_type or envelope value
|
|
@@ -59,6 +60,15 @@ def decoupler_differential_expression(
|
|
| 59 |
"when limma is unavailable so the statistical method is never silently changed. "
|
| 60 |
"Set to True only to explicitly allow t-test fallback during exploratory analysis.",
|
| 61 |
] = False,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
n_cpus: Annotated[int, "Number of CPUs for parallel processing (DESeq2 only)"] = 8,
|
| 63 |
refit_cooks: Annotated[bool, "Whether to refit Cook's distances (DESeq2 only)"] = True,
|
| 64 |
top_genes: Annotated[int, "Number of top genes to show in volcano plot"] = 10,
|
|
@@ -105,6 +115,13 @@ def decoupler_differential_expression(
|
|
| 105 |
f"Available columns: {list(adata.obs.columns)}"
|
| 106 |
)
|
| 107 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 108 |
# Guard: reject deseq2 on pre-normalised data before any expensive computation
|
| 109 |
if method == "deseq2":
|
| 110 |
_X = adata.X
|
|
@@ -135,7 +152,9 @@ def decoupler_differential_expression(
|
|
| 135 |
inference = DefaultInference(n_cpus=n_cpus)
|
| 136 |
dds = DeseqDataSet(
|
| 137 |
adata=adata,
|
| 138 |
-
design=
|
|
|
|
|
|
|
| 139 |
refit_cooks=refit_cooks,
|
| 140 |
inference=inference,
|
| 141 |
)
|
|
@@ -155,27 +174,47 @@ def decoupler_differential_expression(
|
|
| 155 |
X = X.toarray()
|
| 156 |
mask_test = adata.obs[design_factor] == contrast[1]
|
| 157 |
mask_ctrl = adata.obs[design_factor] == contrast[2]
|
| 158 |
-
X_test = X[mask_test.values]
|
| 159 |
-
X_ctrl = X[mask_ctrl.values]
|
| 160 |
genes = list(adata.var.index)
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
else:
|
| 177 |
-
|
| 178 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 179 |
|
| 180 |
# Save results — name the index so pd.read_csv produces a 'gene' column
|
| 181 |
results_df.index.name = "gene"
|
|
|
|
| 8 |
)
|
| 9 |
from src.core import combinable, make_combine_descriptor
|
| 10 |
from src.workflows.sanity_checks import run_sanity_checks
|
| 11 |
+
from src.workflows.microarray import run_limma_covariate
|
| 12 |
|
| 13 |
# ── ADR-0001 Mode-B combine conformance (see src/core/combine.py) ─────────
|
| 14 |
# Module-level so they validate at import: a bad result_type or envelope value
|
|
|
|
| 60 |
"when limma is unavailable so the statistical method is never silently changed. "
|
| 61 |
"Set to True only to explicitly allow t-test fallback during exploratory analysis.",
|
| 62 |
] = False,
|
| 63 |
+
batch_column: Annotated[
|
| 64 |
+
str | None,
|
| 65 |
+
"Optional obs column to model as a batch covariate for cross-dataset early "
|
| 66 |
+
"integration (e.g. 'batch' in a combined AnnData from "
|
| 67 |
+
"decoupler_integrate_datasets). When set: DESeq2 fits ~batch + factor; "
|
| 68 |
+
"limma fits ~batch + group (the group effect is estimated adjusting for "
|
| 69 |
+
"batch); method='ttest' is rejected (a t-test cannot model a covariate). "
|
| 70 |
+
"Default None = single-dataset analysis (behaviour unchanged).",
|
| 71 |
+
] = None,
|
| 72 |
n_cpus: Annotated[int, "Number of CPUs for parallel processing (DESeq2 only)"] = 8,
|
| 73 |
refit_cooks: Annotated[bool, "Whether to refit Cook's distances (DESeq2 only)"] = True,
|
| 74 |
top_genes: Annotated[int, "Number of top genes to show in volcano plot"] = 10,
|
|
|
|
| 115 |
f"Available columns: {list(adata.obs.columns)}"
|
| 116 |
)
|
| 117 |
|
| 118 |
+
# Batch covariate (Mode-A early integration): the column must exist.
|
| 119 |
+
if batch_column is not None and batch_column not in adata.obs.columns:
|
| 120 |
+
raise ValueError(
|
| 121 |
+
f"batch_column '{batch_column}' not found in obs. "
|
| 122 |
+
f"Available columns: {list(adata.obs.columns)}"
|
| 123 |
+
)
|
| 124 |
+
|
| 125 |
# Guard: reject deseq2 on pre-normalised data before any expensive computation
|
| 126 |
if method == "deseq2":
|
| 127 |
_X = adata.X
|
|
|
|
| 152 |
inference = DefaultInference(n_cpus=n_cpus)
|
| 153 |
dds = DeseqDataSet(
|
| 154 |
adata=adata,
|
| 155 |
+
design=(
|
| 156 |
+
f"~{batch_column} + {design_factor}" if batch_column else f"~{design_factor}"
|
| 157 |
+
),
|
| 158 |
refit_cooks=refit_cooks,
|
| 159 |
inference=inference,
|
| 160 |
)
|
|
|
|
| 174 |
X = X.toarray()
|
| 175 |
mask_test = adata.obs[design_factor] == contrast[1]
|
| 176 |
mask_ctrl = adata.obs[design_factor] == contrast[2]
|
|
|
|
|
|
|
| 177 |
genes = list(adata.var.index)
|
| 178 |
+
|
| 179 |
+
if batch_column is not None:
|
| 180 |
+
# Mode-A early integration: model batch as a covariate (limma only).
|
| 181 |
+
# Do NOT fall back to ttest here — a t-test cannot honour the
|
| 182 |
+
# covariate, so silently dropping it would be wrong; surface instead.
|
| 183 |
+
if method != "limma":
|
| 184 |
+
raise ValueError(
|
| 185 |
+
f"batch_column requires method='limma' (or method='deseq2' for raw "
|
| 186 |
+
f"counts); method='{method}' cannot model a batch covariate. Use "
|
| 187 |
+
f"limma/deseq2, or batch-correct (e.g. ComBat) before a t-test."
|
| 188 |
+
)
|
| 189 |
+
mask_both = (mask_test | mask_ctrl).values
|
| 190 |
+
X_both = X[mask_both]
|
| 191 |
+
groups = adata.obs[design_factor][mask_both].astype(str).tolist()
|
| 192 |
+
batches = adata.obs[batch_column][mask_both].astype(str).tolist()
|
| 193 |
+
results_df = run_limma_covariate(
|
| 194 |
+
X_both, groups, batches, genes, contrast[1], contrast[2]
|
| 195 |
+
)
|
| 196 |
+
method_label = f"limma (~{batch_column} + {design_factor})"
|
| 197 |
else:
|
| 198 |
+
X_test = X[mask_test.values]
|
| 199 |
+
X_ctrl = X[mask_ctrl.values]
|
| 200 |
+
if method == "limma":
|
| 201 |
+
try:
|
| 202 |
+
results_df = run_limma(X_test, X_ctrl, genes, contrast[1], contrast[2])
|
| 203 |
+
method_label = "limma"
|
| 204 |
+
except Exception as e:
|
| 205 |
+
if fallback_to_ttest:
|
| 206 |
+
results_df = run_welch_ttest(X_test, X_ctrl, genes)
|
| 207 |
+
method_label = f"ttest (limma unavailable, fallback requested: {e})"
|
| 208 |
+
else:
|
| 209 |
+
raise RuntimeError(
|
| 210 |
+
"method='limma' was requested but limma is unavailable: "
|
| 211 |
+
f"{e}. "
|
| 212 |
+
"Run decoupler_check_limma_available() to diagnose. "
|
| 213 |
+
"Set fallback_to_ttest=True to explicitly allow t-test fallback."
|
| 214 |
+
) from e
|
| 215 |
+
else:
|
| 216 |
+
results_df = run_welch_ttest(X_test, X_ctrl, genes)
|
| 217 |
+
method_label = "ttest"
|
| 218 |
|
| 219 |
# Save results — name the index so pd.read_csv produces a 'gene' column
|
| 220 |
results_df.index.name = "gene"
|
|
@@ -0,0 +1,221 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Cross-dataset early integration (ADR-0001, Mode A — T7).
|
| 3 |
+
|
| 4 |
+
Build ONE combined AnnData from >=2 registered datasets by:
|
| 5 |
+
1. harmonising each to a shared gene-symbol feature axis,
|
| 6 |
+
2. intersecting features (join="inner"), and
|
| 7 |
+
3. concatenating samples with a ``batch`` obs column = dataset_id.
|
| 8 |
+
|
| 9 |
+
The result is a *standard* AnnData that the existing single-dataset tools
|
| 10 |
+
(``decoupler_differential_expression`` and the enrichment tools) consume
|
| 11 |
+
unchanged — Mode A pools the matrices and a downstream DE models ``dataset_id``
|
| 12 |
+
as a batch covariate (see ``decoupler_differential_expression``'s
|
| 13 |
+
``batch_column``, T8). Whether a set of datasets is eligible for early pooling is
|
| 14 |
+
decided upstream by ``get_integration_plan`` (mode == "early"); this module
|
| 15 |
+
performs the mechanics once that verdict is in (the gating lives in the
|
| 16 |
+
``decoupler_integrate_datasets`` tool, T9).
|
| 17 |
+
|
| 18 |
+
Pure core: ``combine_anndatas(adatas, ...)`` operates on already-loaded AnnData
|
| 19 |
+
objects (unit-testable, no network). ``build_combined_anndata(dataset_ids, ...)``
|
| 20 |
+
is the thin loader wrapper that resolves each manifest's hosted h5ad.
|
| 21 |
+
|
| 22 |
+
v1 scope / limitations:
|
| 23 |
+
- Harmonises to gene symbols: a dataset whose ``feature_id_type`` is
|
| 24 |
+
"gene_symbol" uses var_names directly; otherwise a var column named
|
| 25 |
+
SYMBOL/gene_symbol/gene_name is used if present; anything else raises (probe
|
| 26 |
+
collapse / ortholog mapping / entrez without a symbol column are NOT handled
|
| 27 |
+
here — such datasets do not reach an ``early`` verdict in v1).
|
| 28 |
+
- Duplicate symbols within a dataset are collapsed keep-first (preserves raw
|
| 29 |
+
integer counts for the DESeq2 path — no averaging).
|
| 30 |
+
"""
|
| 31 |
+
from __future__ import annotations
|
| 32 |
+
|
| 33 |
+
import os
|
| 34 |
+
import tempfile
|
| 35 |
+
import urllib.request
|
| 36 |
+
from pathlib import Path
|
| 37 |
+
from typing import TYPE_CHECKING, Optional, Sequence
|
| 38 |
+
|
| 39 |
+
import anndata as ad
|
| 40 |
+
import numpy as np
|
| 41 |
+
|
| 42 |
+
from src.tools.rna._base import OUTPUT_DIR
|
| 43 |
+
|
| 44 |
+
if TYPE_CHECKING:
|
| 45 |
+
from anndata import AnnData
|
| 46 |
+
|
| 47 |
+
# var columns checked, in order, for a gene-symbol axis when feature_id_type
|
| 48 |
+
# is not already "gene_symbol".
|
| 49 |
+
_SYMBOL_VAR_COLUMNS = ("SYMBOL", "gene_symbol", "Gene_Symbol", "symbol", "gene_name")
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
def _symbol_axis(adata: "AnnData", feature_id_type: str) -> list[str]:
|
| 53 |
+
"""Return a per-var list of gene symbols for ``adata``, or raise if unavailable."""
|
| 54 |
+
if feature_id_type == "gene_symbol":
|
| 55 |
+
return [str(v) for v in adata.var_names]
|
| 56 |
+
for col in _SYMBOL_VAR_COLUMNS:
|
| 57 |
+
if col in adata.var.columns:
|
| 58 |
+
return [str(v) for v in adata.var[col].tolist()]
|
| 59 |
+
raise ValueError(
|
| 60 |
+
f"cannot resolve gene symbols: feature_id_type='{feature_id_type}' and no "
|
| 61 |
+
f"symbol column {list(_SYMBOL_VAR_COLUMNS)} in var. Mode-A v1 needs a "
|
| 62 |
+
f"gene-symbol axis (probe collapse / id-mapping is not handled here)."
|
| 63 |
+
)
|
| 64 |
+
|
| 65 |
+
|
| 66 |
+
def _to_symbol_adata(adata: "AnnData", feature_id_type: str) -> "AnnData":
|
| 67 |
+
"""Return a copy of ``adata`` whose var_names are unique gene symbols.
|
| 68 |
+
|
| 69 |
+
Drops features with empty/missing symbols; collapses duplicate symbols
|
| 70 |
+
keep-first (preserves raw counts — no averaging)."""
|
| 71 |
+
symbols = _symbol_axis(adata, feature_id_type)
|
| 72 |
+
a = adata.copy()
|
| 73 |
+
a.var_names = [str(s) for s in symbols]
|
| 74 |
+
keep = np.array(
|
| 75 |
+
[bool(s) and s.strip().lower() not in ("nan", "none", "") for s in a.var_names]
|
| 76 |
+
)
|
| 77 |
+
a = a[:, keep].copy()
|
| 78 |
+
# keep-first on duplicate symbols (np.unique returns the first index per value)
|
| 79 |
+
_, first_idx = np.unique(np.asarray(a.var_names, dtype=object), return_index=True)
|
| 80 |
+
a = a[:, np.sort(first_idx)].copy()
|
| 81 |
+
return a
|
| 82 |
+
|
| 83 |
+
|
| 84 |
+
def combine_anndatas(
|
| 85 |
+
adatas: "Sequence[AnnData]",
|
| 86 |
+
dataset_ids: Sequence[str],
|
| 87 |
+
feature_id_types: Sequence[str],
|
| 88 |
+
*,
|
| 89 |
+
design_factor: Optional[str] = None,
|
| 90 |
+
batch_key: str = "batch",
|
| 91 |
+
min_shared_features: int = 200,
|
| 92 |
+
) -> "AnnData":
|
| 93 |
+
"""Pure core: feature-intersection concat of >=2 AnnData into one combined AnnData.
|
| 94 |
+
|
| 95 |
+
Each input is harmonised to a unique gene-symbol var axis, the var
|
| 96 |
+
intersection is taken (``join="inner"``), a ``batch_key`` obs column is set to
|
| 97 |
+
the dataset_id, and samples are concatenated (obs names disambiguated per
|
| 98 |
+
batch with a ``-<dataset_id>`` suffix).
|
| 99 |
+
|
| 100 |
+
Raises
|
| 101 |
+
------
|
| 102 |
+
ValueError
|
| 103 |
+
on <2 inputs, mismatched argument lengths, an unresolvable symbol axis,
|
| 104 |
+
a missing ``design_factor`` column in any dataset, or a feature
|
| 105 |
+
intersection smaller than ``min_shared_features``.
|
| 106 |
+
"""
|
| 107 |
+
if len(adatas) < 2:
|
| 108 |
+
raise ValueError(f"early integration needs >=2 datasets; got {len(adatas)}")
|
| 109 |
+
if not (len(adatas) == len(dataset_ids) == len(feature_id_types)):
|
| 110 |
+
raise ValueError(
|
| 111 |
+
"adatas, dataset_ids, and feature_id_types must be the same length "
|
| 112 |
+
f"({len(adatas)}, {len(dataset_ids)}, {len(feature_id_types)})"
|
| 113 |
+
)
|
| 114 |
+
|
| 115 |
+
prepared: list["AnnData"] = []
|
| 116 |
+
for a, did, fit in zip(adatas, dataset_ids, feature_id_types):
|
| 117 |
+
sa = _to_symbol_adata(a, fit)
|
| 118 |
+
if design_factor is not None and design_factor not in sa.obs.columns:
|
| 119 |
+
raise ValueError(
|
| 120 |
+
f"dataset '{did}' has no obs column '{design_factor}' required for "
|
| 121 |
+
f"the requested contrast."
|
| 122 |
+
)
|
| 123 |
+
prepared.append(sa)
|
| 124 |
+
|
| 125 |
+
shared = set(prepared[0].var_names)
|
| 126 |
+
for sa in prepared[1:]:
|
| 127 |
+
shared &= set(sa.var_names)
|
| 128 |
+
if len(shared) < min_shared_features:
|
| 129 |
+
raise ValueError(
|
| 130 |
+
f"only {len(shared)} shared gene symbols across {list(dataset_ids)} "
|
| 131 |
+
f"(minimum {min_shared_features}); the datasets likely do not share a "
|
| 132 |
+
f"feature space and are not eligible for early pooling."
|
| 133 |
+
)
|
| 134 |
+
|
| 135 |
+
combined = ad.concat(
|
| 136 |
+
prepared,
|
| 137 |
+
axis=0,
|
| 138 |
+
join="inner", # feature intersection
|
| 139 |
+
label=batch_key,
|
| 140 |
+
keys=[str(d) for d in dataset_ids],
|
| 141 |
+
index_unique="-", # disambiguate duplicate sample barcodes across cohorts
|
| 142 |
+
merge="same",
|
| 143 |
+
)
|
| 144 |
+
combined.obs[batch_key] = combined.obs[batch_key].astype("category")
|
| 145 |
+
return combined
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
def _resolve_to_local(url_or_path: str) -> tuple[str, bool]:
|
| 149 |
+
"""Return (local_path, is_temp). Downloads http(s)/ftp URLs to a temp file."""
|
| 150 |
+
if str(url_or_path).startswith(("http://", "https://", "ftp://")):
|
| 151 |
+
suffix = Path(url_or_path.split("?", 1)[0]).suffix or ".h5ad"
|
| 152 |
+
fd, tmp = tempfile.mkstemp(suffix=suffix)
|
| 153 |
+
os.close(fd)
|
| 154 |
+
urllib.request.urlretrieve(url_or_path, tmp)
|
| 155 |
+
return tmp, True
|
| 156 |
+
return str(url_or_path), False
|
| 157 |
+
|
| 158 |
+
|
| 159 |
+
def build_combined_anndata(
|
| 160 |
+
dataset_ids: Sequence[str],
|
| 161 |
+
*,
|
| 162 |
+
design_factor: Optional[str] = None,
|
| 163 |
+
batch_key: str = "batch",
|
| 164 |
+
min_shared_features: int = 200,
|
| 165 |
+
out_prefix: Optional[str] = None,
|
| 166 |
+
) -> dict:
|
| 167 |
+
"""Loader wrapper: resolve each registered dataset's hosted h5ad and combine.
|
| 168 |
+
|
| 169 |
+
Resolves each ``dataset_id`` to its manifest's ``expression_source.url``,
|
| 170 |
+
reads the AnnData, and delegates to :func:`combine_anndatas`. Writes the
|
| 171 |
+
combined AnnData to ``OUTPUT_DIR`` and returns a summary dict.
|
| 172 |
+
"""
|
| 173 |
+
import scanpy as sc
|
| 174 |
+
|
| 175 |
+
from src.datasets.registry import load_manifest
|
| 176 |
+
|
| 177 |
+
if len(dataset_ids) < 2:
|
| 178 |
+
raise ValueError(f"early integration needs >=2 datasets; got {len(dataset_ids)}")
|
| 179 |
+
|
| 180 |
+
adatas: list["AnnData"] = []
|
| 181 |
+
feature_id_types: list[str] = []
|
| 182 |
+
temps: list[str] = []
|
| 183 |
+
try:
|
| 184 |
+
for did in dataset_ids:
|
| 185 |
+
m = load_manifest(did)
|
| 186 |
+
url = (m.expression_source or {}).get("url")
|
| 187 |
+
if not url:
|
| 188 |
+
raise ValueError(f"dataset '{did}' has no expression_source.url")
|
| 189 |
+
local, is_temp = _resolve_to_local(url)
|
| 190 |
+
if is_temp:
|
| 191 |
+
temps.append(local)
|
| 192 |
+
adatas.append(sc.read_h5ad(local))
|
| 193 |
+
feature_id_types.append(m.feature_id_type)
|
| 194 |
+
|
| 195 |
+
combined = combine_anndatas(
|
| 196 |
+
adatas,
|
| 197 |
+
dataset_ids,
|
| 198 |
+
feature_id_types,
|
| 199 |
+
design_factor=design_factor,
|
| 200 |
+
batch_key=batch_key,
|
| 201 |
+
min_shared_features=min_shared_features,
|
| 202 |
+
)
|
| 203 |
+
finally:
|
| 204 |
+
for t in temps:
|
| 205 |
+
Path(t).unlink(missing_ok=True)
|
| 206 |
+
|
| 207 |
+
out_prefix = out_prefix or ("combined_" + "_".join(str(d) for d in dataset_ids))[:80]
|
| 208 |
+
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
| 209 |
+
out_path = OUTPUT_DIR / f"{out_prefix}.h5ad"
|
| 210 |
+
combined.write_h5ad(out_path)
|
| 211 |
+
|
| 212 |
+
per_batch = combined.obs[batch_key].value_counts().to_dict()
|
| 213 |
+
return {
|
| 214 |
+
"output_path": str(out_path.resolve()),
|
| 215 |
+
"n_obs": int(combined.n_obs),
|
| 216 |
+
"n_vars": int(combined.n_vars),
|
| 217 |
+
"batch_key": batch_key,
|
| 218 |
+
"shared_feature_space": "gene_symbol",
|
| 219 |
+
"per_batch_n": {str(k): int(v) for k, v in per_batch.items()},
|
| 220 |
+
"dataset_ids": list(dataset_ids),
|
| 221 |
+
}
|
|
@@ -926,3 +926,117 @@ cat("OK\\n")
|
|
| 926 |
})
|
| 927 |
top_df.index = genes
|
| 928 |
return top_df[["log2FoldChange", "stat", "pvalue", "padj"]]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 926 |
})
|
| 927 |
top_df.index = genes
|
| 928 |
return top_df[["log2FoldChange", "stat", "pvalue", "padj"]]
|
| 929 |
+
|
| 930 |
+
|
| 931 |
+
def run_limma_covariate(
|
| 932 |
+
X: "np.ndarray",
|
| 933 |
+
group_labels: "list[str]",
|
| 934 |
+
batch_labels: "list[str]",
|
| 935 |
+
genes: "list[str]",
|
| 936 |
+
test_group: str,
|
| 937 |
+
control_group: str,
|
| 938 |
+
) -> "pd.DataFrame":
|
| 939 |
+
"""
|
| 940 |
+
Limma with a batch covariate via Rscript — ``model.matrix(~ batch + group)``.
|
| 941 |
+
|
| 942 |
+
Mode-A early integration (ADR-0001 T8): when several cohorts are pooled into
|
| 943 |
+
one matrix, the per-cohort ``batch`` is modelled as a covariate so the group
|
| 944 |
+
effect is estimated *adjusting* for it. Group labels are recoded to ctrl/test
|
| 945 |
+
(control_group -> "ctrl", test_group -> "test") so the tested coefficient is
|
| 946 |
+
always "grouptest" regardless of the original label spelling; batch enters as
|
| 947 |
+
additional factor columns whose names do not matter (only the group
|
| 948 |
+
coefficient is read back).
|
| 949 |
+
|
| 950 |
+
Parameters
|
| 951 |
+
----------
|
| 952 |
+
X : samples x genes matrix (rows = samples, aligned with group/batch labels).
|
| 953 |
+
group_labels, batch_labels : per-sample labels, length == X.shape[0].
|
| 954 |
+
genes : gene ids, length == X.shape[1].
|
| 955 |
+
|
| 956 |
+
Returns columns: log2FoldChange, stat, pvalue, padj (same schema as run_limma).
|
| 957 |
+
Raises RuntimeError if Rscript/limma is unavailable or the design is rank-
|
| 958 |
+
deficient (e.g. batch perfectly confounded with group).
|
| 959 |
+
"""
|
| 960 |
+
import os
|
| 961 |
+
import subprocess
|
| 962 |
+
import tempfile
|
| 963 |
+
|
| 964 |
+
X = np.asarray(X, dtype=np.float64)
|
| 965 |
+
n = X.shape[0]
|
| 966 |
+
if not (len(group_labels) == len(batch_labels) == n):
|
| 967 |
+
raise ValueError(
|
| 968 |
+
f"group_labels ({len(group_labels)}) and batch_labels "
|
| 969 |
+
f"({len(batch_labels)}) must match X sample count ({n})"
|
| 970 |
+
)
|
| 971 |
+
coded = ["test" if str(g) == str(test_group) else "ctrl" for g in group_labels]
|
| 972 |
+
if len(set(coded)) < 2:
|
| 973 |
+
raise ValueError(
|
| 974 |
+
f"need both groups present; got only {set(group_labels)} for "
|
| 975 |
+
f"test='{test_group}' / control='{control_group}'"
|
| 976 |
+
)
|
| 977 |
+
|
| 978 |
+
expr = X.T # genes x samples
|
| 979 |
+
with tempfile.TemporaryDirectory() as tmpdir:
|
| 980 |
+
expr_csv = os.path.join(tmpdir, "expr.csv")
|
| 981 |
+
meta_csv = os.path.join(tmpdir, "meta.csv")
|
| 982 |
+
result_csv = os.path.join(tmpdir, "result.csv")
|
| 983 |
+
|
| 984 |
+
pd.DataFrame(expr, index=genes).to_csv(expr_csv, header=False)
|
| 985 |
+
pd.DataFrame(
|
| 986 |
+
{"group": coded, "batch": [str(b) for b in batch_labels]}
|
| 987 |
+
).to_csv(meta_csv, index=False)
|
| 988 |
+
|
| 989 |
+
r_script = f"""
|
| 990 |
+
suppressPackageStartupMessages({{
|
| 991 |
+
library(limma)
|
| 992 |
+
library(utils)
|
| 993 |
+
}})
|
| 994 |
+
|
| 995 |
+
expr <- as.matrix(read.csv("{expr_csv}", header=FALSE, row.names=1))
|
| 996 |
+
meta <- read.csv("{meta_csv}", colClasses = "character")
|
| 997 |
+
group <- factor(meta$group, levels = c("ctrl", "test"))
|
| 998 |
+
batch <- factor(meta$batch)
|
| 999 |
+
design <- model.matrix(~ batch + group)
|
| 1000 |
+
if (qr(design)$rank < ncol(design)) {{
|
| 1001 |
+
stop("design is rank-deficient (batch likely confounded with group)")
|
| 1002 |
+
}}
|
| 1003 |
+
|
| 1004 |
+
fit <- lmFit(expr, design)
|
| 1005 |
+
fit <- eBayes(fit)
|
| 1006 |
+
result <- topTable(fit, coef = "grouptest",
|
| 1007 |
+
number = nrow(expr),
|
| 1008 |
+
sort.by = "none",
|
| 1009 |
+
adjust.method = "BH")
|
| 1010 |
+
|
| 1011 |
+
write.csv(result, "{result_csv}", row.names = TRUE)
|
| 1012 |
+
cat("OK\\n")
|
| 1013 |
+
"""
|
| 1014 |
+
proc = subprocess.run(
|
| 1015 |
+
["Rscript", "--vanilla", "-"],
|
| 1016 |
+
input=r_script,
|
| 1017 |
+
capture_output=True,
|
| 1018 |
+
text=True,
|
| 1019 |
+
timeout=300,
|
| 1020 |
+
)
|
| 1021 |
+
|
| 1022 |
+
if proc.returncode != 0:
|
| 1023 |
+
raise RuntimeError(
|
| 1024 |
+
f"Rscript/limma (covariate) failed (exit {proc.returncode}):\n"
|
| 1025 |
+
f"{proc.stderr.strip()}"
|
| 1026 |
+
)
|
| 1027 |
+
if not os.path.exists(result_csv):
|
| 1028 |
+
raise RuntimeError(
|
| 1029 |
+
f"Rscript ran but produced no output.\nstdout: {proc.stdout}\n"
|
| 1030 |
+
f"stderr: {proc.stderr}"
|
| 1031 |
+
)
|
| 1032 |
+
|
| 1033 |
+
top_df = pd.read_csv(result_csv, index_col=0)
|
| 1034 |
+
|
| 1035 |
+
top_df = top_df.rename(columns={
|
| 1036 |
+
"logFC": "log2FoldChange",
|
| 1037 |
+
"t": "stat",
|
| 1038 |
+
"P.Value": "pvalue",
|
| 1039 |
+
"adj.P.Val": "padj",
|
| 1040 |
+
})
|
| 1041 |
+
top_df.index = genes
|
| 1042 |
+
return top_df[["log2FoldChange", "stat", "pvalue", "padj"]]
|
|
@@ -0,0 +1,115 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Tests for the batch-covariate DE path (ADR-0001 T8).
|
| 3 |
+
|
| 4 |
+
Covers:
|
| 5 |
+
- run_limma_covariate output schema (skipped if Rscript/limma is unavailable),
|
| 6 |
+
- the decoupler_differential_expression guards: method='ttest' + batch_column is
|
| 7 |
+
rejected, and a missing batch_column is rejected,
|
| 8 |
+
- back-compat: batch_column=None still runs the plain two-group ttest path.
|
| 9 |
+
|
| 10 |
+
Tiny synthetic AnnData only — no network. The DESeq2 covariate path (a one-line
|
| 11 |
+
design change) is exercised by the existing DESeq2 tests + the design string;
|
| 12 |
+
it is not re-run here (needs raw counts + the pydeseq2 stack).
|
| 13 |
+
"""
|
| 14 |
+
from __future__ import annotations
|
| 15 |
+
|
| 16 |
+
import sys
|
| 17 |
+
from pathlib import Path
|
| 18 |
+
|
| 19 |
+
import anndata as ad
|
| 20 |
+
import numpy as np
|
| 21 |
+
import pandas as pd
|
| 22 |
+
import pytest
|
| 23 |
+
|
| 24 |
+
sys.path.insert(0, str(Path(__file__).parent.parent))
|
| 25 |
+
|
| 26 |
+
from src.workflows.microarray import run_limma_covariate # noqa: E402
|
| 27 |
+
from src.tools.rna.analysis import decoupler_differential_expression # noqa: E402
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def _rscript_limma_available() -> bool:
|
| 31 |
+
"""True if a tiny run_limma_covariate succeeds (Rscript + limma installed)."""
|
| 32 |
+
rng = np.random.default_rng(0)
|
| 33 |
+
X = rng.normal(size=(6, 4))
|
| 34 |
+
X[:3, :] += 3.0 # group effect on the 'test' samples
|
| 35 |
+
try:
|
| 36 |
+
run_limma_covariate(
|
| 37 |
+
X,
|
| 38 |
+
["t", "t", "t", "c", "c", "c"],
|
| 39 |
+
["b1", "b2", "b1", "b2", "b1", "b2"],
|
| 40 |
+
["G1", "G2", "G3", "G4"],
|
| 41 |
+
"t",
|
| 42 |
+
"c",
|
| 43 |
+
)
|
| 44 |
+
return True
|
| 45 |
+
except RuntimeError:
|
| 46 |
+
return False
|
| 47 |
+
except Exception:
|
| 48 |
+
return False
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def _h5ad(tmp_path, *, with_batch=True, n_genes=5):
|
| 52 |
+
rng = np.random.default_rng(1)
|
| 53 |
+
n = 6
|
| 54 |
+
X = rng.normal(loc=5.0, scale=1.0, size=(n, n_genes)).astype(np.float32)
|
| 55 |
+
obs = pd.DataFrame(
|
| 56 |
+
{"condition": ["treatment", "treatment", "treatment", "control", "control", "control"]},
|
| 57 |
+
index=[f"s{i}" for i in range(n)],
|
| 58 |
+
)
|
| 59 |
+
if with_batch:
|
| 60 |
+
obs["batch"] = ["A", "B", "A", "B", "A", "B"]
|
| 61 |
+
var = pd.DataFrame(index=[f"G{i}" for i in range(n_genes)])
|
| 62 |
+
a = ad.AnnData(X=X, obs=obs, var=var)
|
| 63 |
+
p = tmp_path / "ds.h5ad"
|
| 64 |
+
a.write_h5ad(p)
|
| 65 |
+
return str(p)
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
@pytest.mark.skipif(not _rscript_limma_available(), reason="Rscript/limma not available")
|
| 69 |
+
def test_run_limma_covariate_schema():
|
| 70 |
+
rng = np.random.default_rng(2)
|
| 71 |
+
X = rng.normal(size=(8, 10))
|
| 72 |
+
X[:4, :] += 2.5 # test-group effect
|
| 73 |
+
genes = [f"G{i}" for i in range(10)]
|
| 74 |
+
groups = ["t", "t", "t", "t", "c", "c", "c", "c"]
|
| 75 |
+
batches = ["b1", "b2", "b1", "b2", "b1", "b2", "b1", "b2"]
|
| 76 |
+
out = run_limma_covariate(X, groups, batches, genes, "t", "c")
|
| 77 |
+
assert list(out.columns) == ["log2FoldChange", "stat", "pvalue", "padj"]
|
| 78 |
+
assert list(out.index) == genes
|
| 79 |
+
assert out["stat"].notna().all()
|
| 80 |
+
|
| 81 |
+
|
| 82 |
+
def test_ttest_with_batch_rejected(tmp_path):
|
| 83 |
+
path = _h5ad(tmp_path, with_batch=True)
|
| 84 |
+
with pytest.raises(ValueError, match="cannot model a batch covariate"):
|
| 85 |
+
decoupler_differential_expression(
|
| 86 |
+
adata_path=path,
|
| 87 |
+
design_factor="condition",
|
| 88 |
+
contrast=["condition", "treatment", "control"],
|
| 89 |
+
method="ttest",
|
| 90 |
+
batch_column="batch",
|
| 91 |
+
)
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def test_missing_batch_column_rejected(tmp_path):
|
| 95 |
+
path = _h5ad(tmp_path, with_batch=False)
|
| 96 |
+
with pytest.raises(ValueError, match="batch_column 'batch' not found"):
|
| 97 |
+
decoupler_differential_expression(
|
| 98 |
+
adata_path=path,
|
| 99 |
+
design_factor="condition",
|
| 100 |
+
contrast=["condition", "treatment", "control"],
|
| 101 |
+
method="limma",
|
| 102 |
+
batch_column="batch",
|
| 103 |
+
)
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
def test_backcompat_ttest_no_batch(tmp_path):
|
| 107 |
+
path = _h5ad(tmp_path, with_batch=True)
|
| 108 |
+
res = decoupler_differential_expression(
|
| 109 |
+
adata_path=path,
|
| 110 |
+
design_factor="condition",
|
| 111 |
+
contrast=["condition", "treatment", "control"],
|
| 112 |
+
method="ttest",
|
| 113 |
+
)
|
| 114 |
+
assert res["method_used"] == "ttest"
|
| 115 |
+
assert "output_path" in res
|
|
@@ -0,0 +1,153 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Tests for the decoupler_integrate_datasets MCP tool (ADR-0001 T9, Mode A).
|
| 3 |
+
|
| 4 |
+
The tool's value is the routing gate: it pools + runs a batch-aware DE ONLY on a
|
| 5 |
+
plan verdict of 'early', and refuses/reroutes otherwise. These tests monkeypatch
|
| 6 |
+
the plan, the combined-AnnData builder, and the DE tool so the routing is checked
|
| 7 |
+
deterministically with no network or compute. The actual pooling/DE mechanics
|
| 8 |
+
are covered by test_integration_mode_a.py and test_de_batch_covariate.py.
|
| 9 |
+
"""
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
|
| 12 |
+
import asyncio
|
| 13 |
+
import sys
|
| 14 |
+
from pathlib import Path
|
| 15 |
+
|
| 16 |
+
sys.path.insert(0, str(Path(__file__).parent.parent))
|
| 17 |
+
|
| 18 |
+
import src.tools.integration_tools as it # noqa: E402
|
| 19 |
+
from src.tools.integration_tools import ( # noqa: E402
|
| 20 |
+
decoupler_integrate_datasets,
|
| 21 |
+
integration_mcp,
|
| 22 |
+
)
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def _plan(mode, **extra):
|
| 26 |
+
base = {"mode": mode, "reason": f"{mode} reason.", "refusal_rules_triggered": []}
|
| 27 |
+
base.update(extra)
|
| 28 |
+
return lambda *a, **k: base
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
def test_tool_registered():
|
| 32 |
+
names = {t.name for t in asyncio.run(integration_mcp.list_tools())}
|
| 33 |
+
assert "decoupler_integrate_datasets" in names
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def test_arity_guard():
|
| 37 |
+
out = decoupler_integrate_datasets(
|
| 38 |
+
dataset_ids=["solo"], design_factor="g", test_group="t", control_group="c"
|
| 39 |
+
)
|
| 40 |
+
assert out["refused"] is True
|
| 41 |
+
assert ">=2" in out["error"]
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
def test_late_reroutes_to_meta_analyze(monkeypatch):
|
| 45 |
+
monkeypatch.setattr(it, "get_integration_plan", _plan("late"))
|
| 46 |
+
out = decoupler_integrate_datasets(
|
| 47 |
+
dataset_ids=["a", "b"], design_factor="g", test_group="t", control_group="c"
|
| 48 |
+
)
|
| 49 |
+
assert out["refused"] and out["mode"] == "late"
|
| 50 |
+
assert "decoupler_meta_analyze" in out["error"]
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def test_concordance_reroutes(monkeypatch):
|
| 54 |
+
monkeypatch.setattr(it, "get_integration_plan", _plan("concordance"))
|
| 55 |
+
out = decoupler_integrate_datasets(
|
| 56 |
+
dataset_ids=["a", "b"], design_factor="g", test_group="t", control_group="c"
|
| 57 |
+
)
|
| 58 |
+
assert out["refused"] and out["mode"] == "concordance"
|
| 59 |
+
assert "decoupler_normalization_concordance" in out["error"]
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def test_refuse_passes_reason_and_rules(monkeypatch):
|
| 63 |
+
monkeypatch.setattr(
|
| 64 |
+
it, "get_integration_plan",
|
| 65 |
+
_plan("refuse", reason="CONFOUNDED_DESIGN: ...", refusal_rules_triggered=["CONFOUNDED_DESIGN"]),
|
| 66 |
+
)
|
| 67 |
+
out = decoupler_integrate_datasets(
|
| 68 |
+
dataset_ids=["a", "b"], design_factor="g", test_group="t", control_group="c"
|
| 69 |
+
)
|
| 70 |
+
assert out["refused"] and out["mode"] == "refuse"
|
| 71 |
+
assert "CONFOUNDED_DESIGN" in out["error"]
|
| 72 |
+
assert out["refusal_rules_triggered"] == ["CONFOUNDED_DESIGN"]
|
| 73 |
+
|
| 74 |
+
|
| 75 |
+
def test_plan_exception_refuses(monkeypatch):
|
| 76 |
+
def boom(*a, **k):
|
| 77 |
+
raise ValueError("dataset 'nope' not registered")
|
| 78 |
+
monkeypatch.setattr(it, "get_integration_plan", boom)
|
| 79 |
+
out = decoupler_integrate_datasets(
|
| 80 |
+
dataset_ids=["a", "nope"], design_factor="g", test_group="t", control_group="c"
|
| 81 |
+
)
|
| 82 |
+
assert out["refused"] and "not registered" in out["error"]
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
def test_early_builds_and_runs_batch_aware_de(monkeypatch, tmp_path):
|
| 86 |
+
monkeypatch.setattr(
|
| 87 |
+
it, "get_integration_plan", _plan("early", poolable_data_level="raw_counts")
|
| 88 |
+
)
|
| 89 |
+
built = {
|
| 90 |
+
"output_path": str(tmp_path / "combined.h5ad"),
|
| 91 |
+
"n_obs": 42, "n_vars": 1500, "per_batch_n": {"a": 20, "b": 22},
|
| 92 |
+
"dataset_ids": ["a", "b"],
|
| 93 |
+
}
|
| 94 |
+
monkeypatch.setattr(it, "build_combined_anndata", lambda *a, **k: built)
|
| 95 |
+
captured = {}
|
| 96 |
+
|
| 97 |
+
def fake_de(**kwargs):
|
| 98 |
+
captured.update(kwargs)
|
| 99 |
+
return {
|
| 100 |
+
"method_used": "DESeq2", "n_significant": 7,
|
| 101 |
+
"output_path": str(tmp_path / "de.csv"), "sanity_warnings": {"n_warnings": 0},
|
| 102 |
+
}
|
| 103 |
+
|
| 104 |
+
monkeypatch.setattr(it, "decoupler_differential_expression", fake_de)
|
| 105 |
+
out = decoupler_integrate_datasets(
|
| 106 |
+
dataset_ids=["a", "b"], design_factor="subtype",
|
| 107 |
+
test_group="basal", control_group="classical",
|
| 108 |
+
)
|
| 109 |
+
assert out["mode"] == "early" and out["batch_modeled"] is True
|
| 110 |
+
assert out["n_significant"] == 7 and out["method_used"] == "DESeq2"
|
| 111 |
+
assert out["per_batch_n"] == {"a": 20, "b": 22}
|
| 112 |
+
# auto -> deseq2 for raw counts; batch covariate + contrast wired correctly
|
| 113 |
+
assert captured["method"] == "deseq2"
|
| 114 |
+
assert captured["batch_column"] == "batch"
|
| 115 |
+
assert captured["contrast"] == ["subtype", "basal", "classical"]
|
| 116 |
+
assert captured["adata_path"] == built["output_path"]
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
def test_early_auto_method_limma_for_non_raw(monkeypatch, tmp_path):
|
| 120 |
+
monkeypatch.setattr(
|
| 121 |
+
it, "get_integration_plan", _plan("early", poolable_data_level="log_expression")
|
| 122 |
+
)
|
| 123 |
+
monkeypatch.setattr(
|
| 124 |
+
it, "build_combined_anndata",
|
| 125 |
+
lambda *a, **k: {"output_path": str(tmp_path / "c.h5ad"), "n_obs": 10,
|
| 126 |
+
"n_vars": 900, "per_batch_n": {"a": 5, "b": 5}},
|
| 127 |
+
)
|
| 128 |
+
captured = {}
|
| 129 |
+
|
| 130 |
+
def fake_de(**kwargs):
|
| 131 |
+
captured.update(kwargs)
|
| 132 |
+
return {"method_used": "limma", "n_significant": 3,
|
| 133 |
+
"output_path": str(tmp_path / "de.csv")}
|
| 134 |
+
|
| 135 |
+
monkeypatch.setattr(it, "decoupler_differential_expression", fake_de)
|
| 136 |
+
decoupler_integrate_datasets(
|
| 137 |
+
dataset_ids=["a", "b"], design_factor="g", test_group="t", control_group="c"
|
| 138 |
+
)
|
| 139 |
+
assert captured["method"] == "limma"
|
| 140 |
+
|
| 141 |
+
|
| 142 |
+
def test_early_build_failure_refuses(monkeypatch):
|
| 143 |
+
monkeypatch.setattr(
|
| 144 |
+
it, "get_integration_plan", _plan("early", poolable_data_level="raw_counts")
|
| 145 |
+
)
|
| 146 |
+
|
| 147 |
+
def boom(*a, **k):
|
| 148 |
+
raise ValueError("only 12 shared gene symbols")
|
| 149 |
+
monkeypatch.setattr(it, "build_combined_anndata", boom)
|
| 150 |
+
out = decoupler_integrate_datasets(
|
| 151 |
+
dataset_ids=["a", "b"], design_factor="g", test_group="t", control_group="c"
|
| 152 |
+
)
|
| 153 |
+
assert out["refused"] and "shared gene symbols" in out["error"]
|
|
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
Unit tests for the Mode-A early-integration core (ADR-0001 T7).
|
| 3 |
+
|
| 4 |
+
Pure tests over constructed AnnData objects — no data, no network. Cover the
|
| 5 |
+
feature-intersection concat, the gene-symbol harmonisation (var.index vs a
|
| 6 |
+
symbol column), keep-first duplicate collapse, empty-symbol dropping, the
|
| 7 |
+
batch obs key, and the arity / shared-feature / design-factor guards.
|
| 8 |
+
"""
|
| 9 |
+
from __future__ import annotations
|
| 10 |
+
|
| 11 |
+
import sys
|
| 12 |
+
from pathlib import Path
|
| 13 |
+
|
| 14 |
+
import anndata as ad
|
| 15 |
+
import numpy as np
|
| 16 |
+
import pandas as pd
|
| 17 |
+
import pytest
|
| 18 |
+
|
| 19 |
+
sys.path.insert(0, str(Path(__file__).parent.parent))
|
| 20 |
+
|
| 21 |
+
from src.workflows.integration import combine_anndatas # noqa: E402
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _adata(symbols, *, n_obs=4, condition=None, symbol_col=None):
|
| 25 |
+
"""Build a small AnnData. If symbol_col is given, var.index holds opaque ids
|
| 26 |
+
and the symbols live in that var column (mimics ensembl + SYMBOL)."""
|
| 27 |
+
X = (np.arange(n_obs * len(symbols), dtype=float).reshape(n_obs, len(symbols)) + 1.0)
|
| 28 |
+
if symbol_col:
|
| 29 |
+
var = pd.DataFrame(index=[f"id{i}" for i in range(len(symbols))])
|
| 30 |
+
var[symbol_col] = list(symbols)
|
| 31 |
+
else:
|
| 32 |
+
var = pd.DataFrame(index=list(symbols))
|
| 33 |
+
obs = pd.DataFrame(index=[f"s{i}" for i in range(n_obs)])
|
| 34 |
+
if condition is not None:
|
| 35 |
+
obs["condition"] = list(condition)
|
| 36 |
+
return ad.AnnData(X=X, obs=obs, var=var)
|
| 37 |
+
|
| 38 |
+
|
| 39 |
+
def test_intersection_concat_and_batch_key():
|
| 40 |
+
a = _adata(["G1", "G2", "G3"], n_obs=3)
|
| 41 |
+
b = _adata(["G2", "G3", "G4"], n_obs=2)
|
| 42 |
+
out = combine_anndatas(
|
| 43 |
+
[a, b], ["A", "B"], ["gene_symbol", "gene_symbol"], min_shared_features=1
|
| 44 |
+
)
|
| 45 |
+
assert set(out.var_names) == {"G2", "G3"} # feature intersection
|
| 46 |
+
assert out.n_obs == 5
|
| 47 |
+
assert out.obs["batch"].value_counts().to_dict() == {"A": 3, "B": 2}
|
| 48 |
+
# obs names disambiguated per batch
|
| 49 |
+
assert any(name.endswith("-A") for name in out.obs_names)
|
| 50 |
+
assert any(name.endswith("-B") for name in out.obs_names)
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def test_symbol_column_used_when_not_gene_symbol():
|
| 54 |
+
a = _adata(["G1", "G2", "G3"], symbol_col="SYMBOL") # var.index = id0.. , SYMBOL col
|
| 55 |
+
b = _adata(["G2", "G3"], symbol_col="SYMBOL")
|
| 56 |
+
out = combine_anndatas(
|
| 57 |
+
[a, b], ["A", "B"], ["ensembl_gene_id", "ensembl_gene_id"], min_shared_features=1
|
| 58 |
+
)
|
| 59 |
+
assert set(out.var_names) == {"G2", "G3"}
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def test_duplicate_symbols_collapsed_keep_first():
|
| 63 |
+
a = _adata(["G1", "G1", "G2"], n_obs=2) # G1 duplicated
|
| 64 |
+
b = _adata(["G1", "G2"], n_obs=2)
|
| 65 |
+
out = combine_anndatas([a, b], ["A", "B"], ["gene_symbol", "gene_symbol"], min_shared_features=1)
|
| 66 |
+
# G1 deduped within A keep-first, then intersected
|
| 67 |
+
assert sorted(out.var_names) == ["G1", "G2"]
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def test_empty_and_nan_symbols_dropped():
|
| 71 |
+
a = _adata(["G1", "", "G2", "nan"], n_obs=2)
|
| 72 |
+
b = _adata(["G1", "G2"], n_obs=2)
|
| 73 |
+
out = combine_anndatas([a, b], ["A", "B"], ["gene_symbol", "gene_symbol"], min_shared_features=1)
|
| 74 |
+
assert sorted(out.var_names) == ["G1", "G2"]
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def test_arity_guard():
|
| 78 |
+
a = _adata(["G1", "G2"])
|
| 79 |
+
with pytest.raises(ValueError, match="needs >=2"):
|
| 80 |
+
combine_anndatas([a], ["A"], ["gene_symbol"])
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
def test_too_few_shared_features_raises():
|
| 84 |
+
a = _adata(["G1", "G2", "G3"])
|
| 85 |
+
b = _adata(["G4", "G5", "G6"]) # disjoint
|
| 86 |
+
with pytest.raises(ValueError, match="shared gene symbols"):
|
| 87 |
+
combine_anndatas([a, b], ["A", "B"], ["gene_symbol", "gene_symbol"], min_shared_features=1)
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
def test_missing_design_factor_raises():
|
| 91 |
+
a = _adata(["G1", "G2"], condition=["t", "c", "t", "c"])
|
| 92 |
+
b = _adata(["G1", "G2"]) # no condition column
|
| 93 |
+
with pytest.raises(ValueError, match="no obs column 'condition'"):
|
| 94 |
+
combine_anndatas(
|
| 95 |
+
[a, b], ["A", "B"], ["gene_symbol", "gene_symbol"],
|
| 96 |
+
design_factor="condition", min_shared_features=1,
|
| 97 |
+
)
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
def test_design_factor_present_ok():
|
| 101 |
+
a = _adata(["G1", "G2"], condition=["t", "c", "t", "c"])
|
| 102 |
+
b = _adata(["G1", "G2"], condition=["t", "c", "t", "c"])
|
| 103 |
+
out = combine_anndatas(
|
| 104 |
+
[a, b], ["A", "B"], ["gene_symbol", "gene_symbol"],
|
| 105 |
+
design_factor="condition", min_shared_features=1,
|
| 106 |
+
)
|
| 107 |
+
assert "condition" in out.obs.columns
|
| 108 |
+
assert "batch" in out.obs.columns
|
| 109 |
+
|
| 110 |
+
|
| 111 |
+
def test_unresolvable_symbol_axis_raises():
|
| 112 |
+
a = _adata(["G1", "G2"], symbol_col="SYMBOL")
|
| 113 |
+
b = _adata(["G1", "G2"], symbol_col="SYMBOL")
|
| 114 |
+
# feature_id_type not gene_symbol AND no recognised symbol column -> drop the col
|
| 115 |
+
a.var = a.var.rename(columns={"SYMBOL": "mystery"})
|
| 116 |
+
b.var = b.var.rename(columns={"SYMBOL": "mystery"})
|
| 117 |
+
with pytest.raises(ValueError, match="cannot resolve gene symbols"):
|
| 118 |
+
combine_anndatas([a, b], ["A", "B"], ["probe_id", "probe_id"], min_shared_features=1)
|