Read docs/PROMPT_confound_pass.md

Prompt — confound pass: extend Effect's confounders + add site-stratified / purity-survival flags to the module ranking

Goal: let us tell whether a gene/module separates HPV for a mechanistic reason or via a confounder (sex, ancestry, anatomic site, immune composition). Read the named code first; don't assume. Airgap stays absolute (clinical/technical covariates are named non-gene variables — allowed, same category as the existing stage/age; purity is derived in the validation layer, never in the engine). pytest + tsc after.

VERIFIED data availability (don't re-assume — checked against the repo)

Raw HNSC clinical files (data/raw_hnsc/data_clinical_patient.txt / _sample.txt) contain: SEX, RACE, ETHNICITY, AJCC_PATHOLOGIC_TUMOR_STAGE, AGE, ICD_O_3_SITE (patient) and TUMOR_TISSUE_SITE / ONCOTREE_CODE (sample). The processed clinical.parquet (build_hnsc.py ~lines 110-117) currently keeps only site(=ONCOTREE_CODE), age, sex, stage, hpv_status. So:

• sex — already in the parquet. ✓
• race / ethnicity — in the raw files, NOT yet extracted. Need a build change.
• anatomic site for oropharynx — ONCOTREE_CODE is insufficient; use TUMOR_TISSUE_SITE (sample) or ICD_O_3_SITE (patient). Need a build change.
• smoking — NOT in this download at all. Do not attempt to adjust for it.
• tumor purity — NOT in clinical. Must be derived from expression (see Part E).

Part A — extend the HNSC build to carry the confounders (data)

Note (verified): read_clinical_tsv returns the FULL dataframe, so RACE / TUMOR_TISSUE_SITE / ICD_O_3_SITE are reachable via merged.get(...) — no need to add them to HNSC_REQUIRED_*_COLS. In data_pipeline/build_hnsc.py (the clinical dict ~lines 110-117): read and write into clinical.parquet:

• race (from RACE; optionally also ethnicity).
• tissue_site — a clean anatomic-site field from TUMOR_TISSUE_SITE (preferred) or ICD_O_3_SITE, plus a derived boolean is_oropharynx (map the oropharynx/tonsil/base-of-tongue values; print the distinct site values + counts so we can confirm the mapping). Keep the existing site(ONCOTREE) column too.
• sex is already present — leave it. Print the non-null counts for race / tissue_site / is_oropharynx so missingness is visible. (Colorectal build unchanged.)

Part B — carry the new clinical fields through the API (no airgap change)

_prepare_lab_data (api/app.py) currently subsets cohort.clinical[["stage","age"]]. For HNSC, widen the kept set to include sex, race, tissue_site/is_oropharynx (only the columns that exist; guard each). These are named clinical variables, not gene symbols — airgap-clean. Keep colorectal as-is. The returned clinical DataFrame is what both Effect (via the run's ctx) and the module endpoint (Part D/E) will read, so the new columns must be in the RETURN, not just kept somewhere upstream.

Part C — generalize Effect to a configurable confounder set (engine)

Effect.execute (engine_v2/nodes.py ~436-471) hardcodes stage + age. Generalize it to adjust on a configurable list of confounders pulled from ctx.clinical (one-hot encode categoricals like sex/race/site; keep age continuous; dropna per the design matrix; guard rank/collinearity). Default must stay stage + age so existing MSI/TMB/HPV runs are byte-for-byte unchanged. When the extra clinical columns are present, allow Effect to include sex + race (NOT smoking — unavailable). The confounder set is fixed by config, not GP-selectable (the program shouldn't roam over arbitrary confounders). Update/extend the relevant engine_v2 tests.

Part D — site-stratified survival flag on the module ranking (validation layer)

IMPORTANT (verified): _compute_module_ranking currently discards clinical — line ~1282 is M_all, y_all, _clin, _extra = _prepare_lab_data(...) with clinical thrown away as _clin. Stop discarding it: capture the clinical frame, and align is_oropharynx to the held-out patients (the same test_ids it already reconstructs) so you can subset. Then, for the HNSC/HPV case add per-module:

• combined_holdout_oropharynx: the module's mean-aggregate AUROC recomputed within the oropharynx subgroup only (held-out patients with is_oropharynx == True), and a survives_site boolean = it stays close to the full-cohort combined AUROC (within a tolerance, e.g. drop < ~0.05) rather than collapsing toward 0.5. This tells us whether the module separates HPV beyond just marking oropharynx tissue.
• Guard small subgroups (skip / mark "n too small" if the oropharynx held-out has too few per class).
• Gate to HNSC; colorectal modules omit this gracefully.

Part E — purity-survival flag (validation layer; heaviest part — may be phased)

Purity isn't in the data, so derive an immune-infiltration proxy in the validation layer (names allowed here, exactly like the rank diagnostics): per patient, the mean expression of a small standard immune-marker set (e.g. CD8A, GZMB, PRF1, CD3D, CD2 — a fixed curated list in one place). Since the matrix M is opaque-columned, resolve those marker SYMBOLS to their opaque IDs via the bounded reveal path (same as reference sets), then mean those columns of M per patient. Higher proxy = more infiltrate = lower purity. Then per module add:

• combined_holdout_highpurity: the module's mean-aggregate AUROC recomputed within the low-infiltration (high-purity) subset of held-out patients (e.g. bottom tertile of the proxy), and a survives_purity boolean (same tolerance idea). This catches modules whose HPV signal is really an immune-composition artifact (they'll collapse in the high-purity subset).
• This derivation lives in the validation/API layer — never in engine/ or engine_v2/. If scope is tight, ship Parts A-D first and land Part E second; mark it clearly.

Part F — UI: surface the flags + the winner's-curse caveat (presentation)

• In the Coordinated-modules table, show the survival flags per module (e.g. small "site ✓/✗" and "purity ✓/✗" chips, or the stratified AUROCs on expand). A module that survives both is a real candidate; one that collapses is a likely confound artifact.
• Add to the modules "?" the winner's-curse caveat we agreed on, verbatim intent: "These groups are scored by re-evaluating ~2,900 explored sets on the same small held-out set and showing the best — so the very top values are optimistically biased (the luckiest of thousands). The winning program was chosen by cross-validation, which guards against that, so trust it as the engine's pick even when a table row scores higher."

CONSTRAINTS

• Airgap: clinical confounders (sex/race/site) and the derived purity proxy are named non-gene variables / validation-layer computations — never enter engine/ or engine_v2/ as gene identities. Module payloads stay opaque-ID + scores + flags.
• Defaults preserve current behaviour (Effect default = stage+age; non-HNSC unaffected). Colorectal + existing tests stay green.

Checkpoint

• HNSC clinical.parquet carries race + tissue_site + is_oropharynx (with printed counts); sex already there.
• Effect adjusts on a configurable set, default stage+age unchanged; with HNSC clinical present it can include sex+race.
• Module ranking shows site-stratified (and, when Part E lands, purity) survival flags for HNSC/HPV; modules that are site/immune artifacts visibly collapse.
• Modules "?" carries the winner's-curse caveat.
• pytest green, tsc clean, airgap tests untouched.