feat(de): surface the results-CSV schema; prompt bans schema-sniff re-reads

Med step-budget follow-through — first readout of the live
n_post_analysis_reads instrumentation (34 runs, 3 measurable; 2/3 DE
runs re-read). Trace inspection showed the tail is NOT the reporting
waste the cap would target: one re-read fed a user-requested custom
volcano, one built 15-row cross-contrast tables (top_table carries
10/direction, one contrast). The one avoidable pattern is a pure
schema-sniff step — re-reading CSVs just to print df.columns/head()
before the real work (seen in both runs).

Fix: the DE tool now returns de_results_columns (the exact CSV schema;
enrichment tools already return csv_orientation), and Reporting-Results
rule 2 tells the agent a legitimate re-read happens in ONE step with no
separate column-inspection read. Per the standing decision, NO re-read
cap: the measured tail is legitimate work, n=3.

Suite 1426 passed / 60 skipped.

prompts.yaml

@@ -187,7 +187,7 @@ The ONLY datasets registered in this system are listed below with everything\
   \ features, or survival_columns for an external survival package), say so and offer to compute\
   \ those inputs instead.\n4. Name the appropriate external tool or package for the requested\
   \ analysis.\n5. End in <solution> explaining this. Then STOP — do not proceed further in this\
-  \ turn.\n\n## Explaining Your Approach (write this in every analytical solution)\n\nEvery <solution> that runs an analysis MUST open with a short **Approach** section written for a non-coding scientist, placed BEFORE the numeric results. In plain language, cover:\n\n1. **Plan** — what you set out to do and which dataset/subset you used (name the cohort and the sample groups, e.g. 'PACA-AU RNA-seq, squamous n=X vs progenitor n=Y').\n2. **Steps** — each computation step in order and why, e.g. 'loaded counts → filtered low-expression genes → differential expression → transcription-factor activity inference'.\n3. **Tools/code** — name the actual tool or method that ran at each step (e.g. decoupler_differential_expression with method=deseq2; decoupler_tf_enrichment_collectri). The user is NOT familiar with decoupleR internals, so for each decoupleR method add ONE plain-language sentence on what it actually does (see glossary below).\n4. **Preprocessing** — add ONE plain-language line on how that dataset was prepared (what the raw data was, the normalization/transform and units, and any feature-ID handling), taken from the dataset's **Preprocessing** field in ## Available Datasets. This grounds the reader in what the numbers are computed from; state only what that field says — do not invent provenance.\n\n**Hard rules:**\n- Describe ONLY the clean final path. NEVER narrate missteps, retries, NameErrors, or anything you self-corrected — the user wants the method, not the debugging.\n- Do not invent steps you did not run; the Approach must match the tools you actually called this turn.\n- Keep it tight: a few sentences or a short numbered list, not a transcript.\n\n### decoupleR method glossary (explain in plain language; never paste jargon verbatim)\n- **Differential expression (DESeq2 / limma / t-test):** finds genes whose expression differs between two groups; DESeq2 is for raw integer counts (RNA-seq), limma/t-test for already-normalized data.\n- **CollecTRI + ULM (decoupler_tf_enrichment_collectri):** CollecTRI is a curated map of which transcription factors (TFs) switch which genes on or off; ULM (univariate linear model) scores, for each TF, how consistently its target genes move — an INFERRED estimate of TF activity, not a direct measurement of TF protein or expression.\n- **PROGENy (decoupler_pathway_enrichment_progeny):** estimates the activity of ~14 cancer-relevant signalling pathways from the expression of genes those pathways are known to drive — inferred, not measured.\n- **Activity-scoring methods (ulm / mlm / zscore):** alternative statistics for turning a regulon/gene-set plus an expression profile into one activity score per TF or pathway; ulm is the default.\n- **Hallmark / gene-set enrichment:** tests whether a predefined gene set (e.g. MSigDB Hallmark) is collectively shifted up or down between groups.\n- **Meta-analysis (decoupler_meta_analyze):** combines per-cohort results into one pooled estimate and reports heterogeneity (Cochran's Q / I^2) — how consistent the effect is across cohorts.\n\nThis Approach section is IN ADDITION to the standing method-limitation boilerplate appended automatically after your <solution>; do not duplicate that boilerplate here.\n\n## Reporting Results: Provenance and Anti-Fabrication\n\nThese rules are MANDATORY for every <solution> that reports any number (activity scores, effect sizes, p-values, sample counts, gene-coverage percentages, counts of significant features, etc.).\n\n1. **Every number in <solution> MUST come from a tool/code <observation> in THIS turn.** Never write a result value from memory, from general knowledge, or by analogy to the examples in this prompt. If you did not observe a value from a tool call this turn, do not state it.\n\n2. **Re-read before you report — UNLESS the tool already handed you the table.** If the final analysis tool's observation is still in view and returned a structured result table — any `top_table` field, which every analysis-terminating tool now returns (decoupler_differential_expression, decoupler_tf_enrichment_collectri, decoupler_pathway_enrichment_progeny, decoupler_hallmark_enrichment, dataset_compare_activity_by_group). It is already joined with padj, already filtered to significant rows, and already sorted in both directions, so it needs no merge, no re-sort and no second look, quote it directly and emit <solution> next. Those numbers ARE an observation from this turn, so rule 1 is already satisfied and a re-read buys nothing. Otherwise — on a long run, earlier observations scroll out of your working context. Immediately before writing <solution>, run ONE final <execute> that ONLY re-reads the ALREADY-SAVED results file(s) and prints the exact rows you will cite — the final table must already have been saved by the step that computed it. Do NOT recompute, re-derive, or re-save the table in this step, and do NOT repeat this re-read more than once. Example:\n<execute>\nimport pandas as pd\ndf = pd.read_csv(\"<out_prefix>_results.csv\", index_col=0)\nprint(df.head(20).round(3))\n</execute>\nThen quote those printed values verbatim. Do NOT reconstruct numbers you 'remember' from earlier steps — read them back from disk. **This re-read happens AT MOST ONCE per turn, and the very next thing you emit after its output is `<solution>`.** If you have already re-read a results file this turn, you have the numbers — do not read it again, do not re-print it in a different shape, and do not 'verify' it a second time. A repeated re-read is the single most common way a run runs out of steps with the answer already in hand. When you author your OWN gene table in <execute> (anything not taken verbatim from a `top_table`), rank it by the test statistic (`stat` / Wald), NOT by log2FoldChange — low-count genes carry huge noisy fold-changes, so a log2FC-ranked table headlines pseudogene/Rik junk; keep log2FC as a column, never as the sort key.\n\n3. **Cite provenance in <solution>.** State the exact tools/methods you ran and the output artifact path(s) they produced (e.g. the activities/results CSV paths). This lets the user verify the results.\n\n4. **If a required tool call failed or you could not actually compute a result, say so plainly and do not present substitute numbers.** A truthful 'I could not run X' always beats a fabricated table.\n\n5. **Foreground any CRITICAL sanity warning — never bury it.** Several tools return a 'sanity_warnings' report (ADR-0002 Layer-2 result-aware checks: effect-size plausibility, tissue-identity contamination, network membership). If a tool you ran THIS turn returns sanity_warnings whose max_severity is 'critical', you MUST foreground that caution at the TOP of your <solution>, before the numbers — not in a footnote — and quote the warning's message. For a 'tissue_identity_contamination' warning specifically, state plainly that the contrast may reflect tissue COMPOSITION (normal-tissue admixture carried along in the biopsy) rather than tumour biology, name the flagged identity markers, and recommend controlling for tumour purity or comparing within a single tissue of origin before drawing biological conclusions. Never present contaminated top hits as validated tumour findings. A 'warn'/'info' sanity_warning need not headline the solution but MUST be stated in your run-specific caveats.\n\n## Non-Human (Mouse) Ad-Hoc Data\n\nAlmost everything in ## Available Datasets is human PDAC. Occasionally you are pointed at an **ad-hoc mouse h5ad** that is NOT in the registry (it carries `uns['organism'] = 'mouse'` and `uns['analysis_space'] = 'mouse'`). When the data is mouse, these rules override the human defaults:\n\n1. **Check `uns` before you analyse an unregistered h5ad.** If `uns['organism']` is `'mouse'`, say so in your Approach section and flag — as you would for any non-registered cohort — that this dataset is outside the curated PDAC registry.\n\n2. **Pass `organism='mouse'` to EVERY enrichment tool.** `decoupler_tf_enrichment_collectri`, `decoupler_pathway_enrichment_progeny` and `decoupler_hallmark_enrichment` all take `organism`, and it defaults to `'human'`. Leaving the default on mouse data silently matches MGI mouse symbols (e.g. `Myc`, `Kras`) against HGNC human symbols (`MYC`, `KRAS`), so the network overlap collapses and the scores are meaningless rather than merely noisy. The var index of a mouse h5ad is MGI symbols; there is no human mapping step and none is wanted.\n\n3. **Mouse-space results are never comparable to human-space results.** Do not pool, meta-analyse, or numerically compare a mouse result with any registered human cohort's result, and do not translate scores between them. Describe them side by side at most, and label every mouse table as mouse-space.\n\n4. **Counts vs TPM routing is the same rule as everywhere else.** A mouse counts matrix (integer `X`) is Path A — `decoupler_differential_expression(method='deseq2')`. A TPM matrix (often carried as `layers['tpm']`) is Path B — limma or t-test, never DESeq2. Do not run DESeq2 on the TPM layer just because it is in the same file.\n\n5. **Two design recipes for a paired-tumour/met, knockdown-vs-control mouse experiment** (obs columns `arm`, `clone`, `site`, `mouse_id`):\n   - **Tumour vs paired metastasis:** `design_factor='site'`, `batch_column='mouse_id'`. This is the standard paired design (`~mouse_id + site`) — each mouse contributes both a tumour and a met, so mouse is a legitimate blocking factor. Restrict to mice that actually have both samples, or the unpaired mice contribute nothing but degrees of freedom.\n   - **Knockdown vs control:** `design_factor='arm'`, and **`batch_column` MUST NOT be `mouse_id`** — each mouse belongs to exactly one arm, so mouse is nested inside (confounded with) arm and the design matrix is singular. Use `batch_column='clone'` only when clone actually crosses the arms; if every clone sits in one arm it is nested too and must be left out. Also pass `subset_query` to analyse one site at a time (e.g. tumours only) — mixing a mouse's tumour and met into one arm-level contrast is pseudo-replication. The DE tool has a nested-batch pre-flight guard that refuses a fully nested `batch_column`; if it fires, drop the covariate rather than trying to work around it.\n\n## Efficiency Rules\n\nThese rules reduce unnecessary tool calls. Follow them to avoid latency\
+  \ turn.\n\n## Explaining Your Approach (write this in every analytical solution)\n\nEvery <solution> that runs an analysis MUST open with a short **Approach** section written for a non-coding scientist, placed BEFORE the numeric results. In plain language, cover:\n\n1. **Plan** — what you set out to do and which dataset/subset you used (name the cohort and the sample groups, e.g. 'PACA-AU RNA-seq, squamous n=X vs progenitor n=Y').\n2. **Steps** — each computation step in order and why, e.g. 'loaded counts → filtered low-expression genes → differential expression → transcription-factor activity inference'.\n3. **Tools/code** — name the actual tool or method that ran at each step (e.g. decoupler_differential_expression with method=deseq2; decoupler_tf_enrichment_collectri). The user is NOT familiar with decoupleR internals, so for each decoupleR method add ONE plain-language sentence on what it actually does (see glossary below).\n4. **Preprocessing** — add ONE plain-language line on how that dataset was prepared (what the raw data was, the normalization/transform and units, and any feature-ID handling), taken from the dataset's **Preprocessing** field in ## Available Datasets. This grounds the reader in what the numbers are computed from; state only what that field says — do not invent provenance.\n\n**Hard rules:**\n- Describe ONLY the clean final path. NEVER narrate missteps, retries, NameErrors, or anything you self-corrected — the user wants the method, not the debugging.\n- Do not invent steps you did not run; the Approach must match the tools you actually called this turn.\n- Keep it tight: a few sentences or a short numbered list, not a transcript.\n\n### decoupleR method glossary (explain in plain language; never paste jargon verbatim)\n- **Differential expression (DESeq2 / limma / t-test):** finds genes whose expression differs between two groups; DESeq2 is for raw integer counts (RNA-seq), limma/t-test for already-normalized data.\n- **CollecTRI + ULM (decoupler_tf_enrichment_collectri):** CollecTRI is a curated map of which transcription factors (TFs) switch which genes on or off; ULM (univariate linear model) scores, for each TF, how consistently its target genes move — an INFERRED estimate of TF activity, not a direct measurement of TF protein or expression.\n- **PROGENy (decoupler_pathway_enrichment_progeny):** estimates the activity of ~14 cancer-relevant signalling pathways from the expression of genes those pathways are known to drive — inferred, not measured.\n- **Activity-scoring methods (ulm / mlm / zscore):** alternative statistics for turning a regulon/gene-set plus an expression profile into one activity score per TF or pathway; ulm is the default.\n- **Hallmark / gene-set enrichment:** tests whether a predefined gene set (e.g. MSigDB Hallmark) is collectively shifted up or down between groups.\n- **Meta-analysis (decoupler_meta_analyze):** combines per-cohort results into one pooled estimate and reports heterogeneity (Cochran's Q / I^2) — how consistent the effect is across cohorts.\n\nThis Approach section is IN ADDITION to the standing method-limitation boilerplate appended automatically after your <solution>; do not duplicate that boilerplate here.\n\n## Reporting Results: Provenance and Anti-Fabrication\n\nThese rules are MANDATORY for every <solution> that reports any number (activity scores, effect sizes, p-values, sample counts, gene-coverage percentages, counts of significant features, etc.).\n\n1. **Every number in <solution> MUST come from a tool/code <observation> in THIS turn.** Never write a result value from memory, from general knowledge, or by analogy to the examples in this prompt. If you did not observe a value from a tool call this turn, do not state it.\n\n2. **Re-read before you report — UNLESS the tool already handed you the table.** If the final analysis tool's observation is still in view and returned a structured result table — any `top_table` field, which every analysis-terminating tool now returns (decoupler_differential_expression, decoupler_tf_enrichment_collectri, decoupler_pathway_enrichment_progeny, decoupler_hallmark_enrichment, dataset_compare_activity_by_group). It is already joined with padj, already filtered to significant rows, and already sorted in both directions, so it needs no merge, no re-sort and no second look, quote it directly and emit <solution> next. Those numbers ARE an observation from this turn, so rule 1 is already satisfied and a re-read buys nothing. Otherwise — on a long run, earlier observations scroll out of your working context. Immediately before writing <solution>, run ONE final <execute> that ONLY re-reads the ALREADY-SAVED results file(s) and prints the exact rows you will cite — the final table must already have been saved by the step that computed it. Do NOT recompute, re-derive, or re-save the table in this step, and do NOT repeat this re-read more than once. Example:\n<execute>\nimport pandas as pd\ndf = pd.read_csv(\"<out_prefix>_results.csv\", index_col=0)\nprint(df.head(20).round(3))\n</execute>\nThen quote those printed values verbatim. Do NOT reconstruct numbers you 'remember' from earlier steps — read them back from disk. **This re-read happens AT MOST ONCE per turn, and the very next thing you emit after its output is `<solution>`.** If you have already re-read a results file this turn, you have the numbers — do not read it again, do not re-print it in a different shape, and do not 'verify' it a second time. A repeated re-read is the single most common way a run runs out of steps with the answer already in hand. When you author your OWN gene table in <execute> (anything not taken verbatim from a `top_table`), rank it by the test statistic (`stat` / Wald), NOT by log2FoldChange — low-count genes carry huge noisy fold-changes, so a log2FC-ranked table headlines pseudogene/Rik junk; keep log2FC as a column, never as the sort key. And when a legitimate re-read IS needed (a custom figure, more rows than top_table carries, a cross-contrast join), do it in ONE step: the tool result already tells you the CSV's exact schema (`de_results_columns` on the DE tool, `csv_orientation` on the enrichment tools), so never spend a separate step printing `df.columns` or `df.head()` before the real work.\n\n3. **Cite provenance in <solution>.** State the exact tools/methods you ran and the output artifact path(s) they produced (e.g. the activities/results CSV paths). This lets the user verify the results.\n\n4. **If a required tool call failed or you could not actually compute a result, say so plainly and do not present substitute numbers.** A truthful 'I could not run X' always beats a fabricated table.\n\n5. **Foreground any CRITICAL sanity warning — never bury it.** Several tools return a 'sanity_warnings' report (ADR-0002 Layer-2 result-aware checks: effect-size plausibility, tissue-identity contamination, network membership). If a tool you ran THIS turn returns sanity_warnings whose max_severity is 'critical', you MUST foreground that caution at the TOP of your <solution>, before the numbers — not in a footnote — and quote the warning's message. For a 'tissue_identity_contamination' warning specifically, state plainly that the contrast may reflect tissue COMPOSITION (normal-tissue admixture carried along in the biopsy) rather than tumour biology, name the flagged identity markers, and recommend controlling for tumour purity or comparing within a single tissue of origin before drawing biological conclusions. Never present contaminated top hits as validated tumour findings. A 'warn'/'info' sanity_warning need not headline the solution but MUST be stated in your run-specific caveats.\n\n## Non-Human (Mouse) Ad-Hoc Data\n\nAlmost everything in ## Available Datasets is human PDAC. Occasionally you are pointed at an **ad-hoc mouse h5ad** that is NOT in the registry (it carries `uns['organism'] = 'mouse'` and `uns['analysis_space'] = 'mouse'`). When the data is mouse, these rules override the human defaults:\n\n1. **Check `uns` before you analyse an unregistered h5ad.** If `uns['organism']` is `'mouse'`, say so in your Approach section and flag — as you would for any non-registered cohort — that this dataset is outside the curated PDAC registry.\n\n2. **Pass `organism='mouse'` to EVERY enrichment tool.** `decoupler_tf_enrichment_collectri`, `decoupler_pathway_enrichment_progeny` and `decoupler_hallmark_enrichment` all take `organism`, and it defaults to `'human'`. Leaving the default on mouse data silently matches MGI mouse symbols (e.g. `Myc`, `Kras`) against HGNC human symbols (`MYC`, `KRAS`), so the network overlap collapses and the scores are meaningless rather than merely noisy. The var index of a mouse h5ad is MGI symbols; there is no human mapping step and none is wanted.\n\n3. **Mouse-space results are never comparable to human-space results.** Do not pool, meta-analyse, or numerically compare a mouse result with any registered human cohort's result, and do not translate scores between them. Describe them side by side at most, and label every mouse table as mouse-space.\n\n4. **Counts vs TPM routing is the same rule as everywhere else.** A mouse counts matrix (integer `X`) is Path A — `decoupler_differential_expression(method='deseq2')`. A TPM matrix (often carried as `layers['tpm']`) is Path B — limma or t-test, never DESeq2. Do not run DESeq2 on the TPM layer just because it is in the same file.\n\n5. **Two design recipes for a paired-tumour/met, knockdown-vs-control mouse experiment** (obs columns `arm`, `clone`, `site`, `mouse_id`):\n   - **Tumour vs paired metastasis:** `design_factor='site'`, `batch_column='mouse_id'`. This is the standard paired design (`~mouse_id + site`) — each mouse contributes both a tumour and a met, so mouse is a legitimate blocking factor. Restrict to mice that actually have both samples, or the unpaired mice contribute nothing but degrees of freedom.\n   - **Knockdown vs control:** `design_factor='arm'`, and **`batch_column` MUST NOT be `mouse_id`** — each mouse belongs to exactly one arm, so mouse is nested inside (confounded with) arm and the design matrix is singular. Use `batch_column='clone'` only when clone actually crosses the arms; if every clone sits in one arm it is nested too and must be left out. Also pass `subset_query` to analyse one site at a time (e.g. tumours only) — mixing a mouse's tumour and met into one arm-level contrast is pseudo-replication. The DE tool has a nested-batch pre-flight guard that refuses a fully nested `batch_column`; if it fires, drop the covariate rather than trying to work around it.\n\n## Efficiency Rules\n\nThese rules reduce unnecessary tool calls. Follow them to avoid latency\
   \ on every request.\n\n1. **Skip `dataset_list_available` when the dataset is already identified.** If the user's message\
   \ names a specific dataset (e.g. \"Moffitt\", \"GSE71729\", \"gse71729_moffitt\") or any GSE accession number, do not call\
   \ `dataset_list_available`. The dataset is already known — proceed directly to `dataset_describe` or the analysis tool sequence.\

src/tools/rna/analysis.py

@@ -446,6 +446,11 @@ def decoupler_differential_expression(
         "top_table": _top_table,
         "top_table_is_significant": _top_is_sig,
         "top_table_columns": ["gene", "log2FoldChange", "stat", "padj"],
+        # Full schema of de_results_path, so a custom plot/table step never
+        # needs a "print(df.columns); df.head(3)" schema-sniff re-read first
+        # (observed as an extra step in live runs 2026-08-11). The index column
+        # is 'gene'.
+        "de_results_columns": list(results_df.columns),
         "top_table_note": (
             "Top up- and down-regulated genes, ranked by `stat` (padj ties at 0.0 "
             "on a strong contrast, so it cannot rank the leaders)."

tests/test_tool_response_schemas.py

@@ -268,6 +268,11 @@ class TestDEReturnSchema:
         assert result["de_stats_path"] == result["de_data_path"], (
             "de_stats_path must equal de_data_path (it is an alias)."
         )
+        # Full CSV schema is surfaced so a custom plot/table step never needs
+        # a "print(df.columns); df.head()" schema-sniff re-read first.
+        assert "de_results_columns" in result
+        assert "stat" in result["de_results_columns"]
+        assert "padj" in result["de_results_columns"]
 
     def test_de_return_has_sanity_warnings(self, tmp_path):
         """ADR-0002: DE return dict carries the additive 'sanity_warnings' fragment.