# Prompt — confound pass: extend Effect's confounders + add site-stratified / purity-survival flags to the module ranking

Goal: let us tell whether a gene/module separates HPV for a *mechanistic* reason or via a *confounder* (sex, ancestry, anatomic site, immune composition). Read the named code first; don't assume. Airgap stays absolute (clinical/technical covariates are named non-gene variables — allowed, same category as the existing stage/age; purity is derived in the validation layer, never in the engine). `pytest` + `tsc` after.

## VERIFIED data availability (don't re-assume — checked against the repo)
Raw HNSC clinical files (`data/raw_hnsc/data_clinical_patient.txt` / `_sample.txt`) contain: `SEX`, `RACE`, `ETHNICITY`, `AJCC_PATHOLOGIC_TUMOR_STAGE`, `AGE`, `ICD_O_3_SITE` (patient) and `TUMOR_TISSUE_SITE` / `ONCOTREE_CODE` (sample). The processed `clinical.parquet` (`build_hnsc.py` ~lines 110-117) currently keeps only `site`(=ONCOTREE_CODE), `age`, `sex`, `stage`, `hpv_status`. So:
- **sex** — already in the parquet. ✓
- **race / ethnicity** — in the raw files, NOT yet extracted. Need a build change.
- **anatomic site for oropharynx** — `ONCOTREE_CODE` is insufficient; use `TUMOR_TISSUE_SITE` (sample) or `ICD_O_3_SITE` (patient). Need a build change.
- **smoking** — NOT in this download at all. Do not attempt to adjust for it.
- **tumor purity** — NOT in clinical. Must be derived from expression (see Part E).

## Part A — extend the HNSC build to carry the confounders (data)
Note (verified): `read_clinical_tsv` returns the FULL dataframe, so `RACE` / `TUMOR_TISSUE_SITE` / `ICD_O_3_SITE` are reachable via `merged.get(...)` — no need to add them to `HNSC_REQUIRED_*_COLS`. In `data_pipeline/build_hnsc.py` (the clinical dict ~lines 110-117): read and write into `clinical.parquet`:
- `race` (from `RACE`; optionally also `ethnicity`).
- `tissue_site` — a clean anatomic-site field from `TUMOR_TISSUE_SITE` (preferred) or `ICD_O_3_SITE`, plus a derived boolean `is_oropharynx` (map the oropharynx/tonsil/base-of-tongue values; print the distinct site values + counts so we can confirm the mapping). Keep the existing `site`(ONCOTREE) column too.
- `sex` is already present — leave it.
Print the non-null counts for race / tissue_site / is_oropharynx so missingness is visible. (Colorectal build unchanged.)

## Part B — carry the new clinical fields through the API (no airgap change)
`_prepare_lab_data` (api/app.py) currently subsets `cohort.clinical[["stage","age"]]`. For HNSC, widen the kept set to include `sex`, `race`, `tissue_site`/`is_oropharynx` (only the columns that exist; guard each). These are named clinical variables, not gene symbols — airgap-clean. Keep colorectal as-is. The returned `clinical` DataFrame is what both Effect (via the run's ctx) and the module endpoint (Part D/E) will read, so the new columns must be in the RETURN, not just kept somewhere upstream.

## Part C — generalize Effect to a configurable confounder set (engine)
`Effect.execute` (engine_v2/nodes.py ~436-471) hardcodes stage + age. Generalize it to adjust on a configurable list of confounders pulled from `ctx.clinical` (one-hot encode categoricals like sex/race/site; keep age continuous; dropna per the design matrix; guard rank/collinearity). **Default must stay stage + age** so existing MSI/TMB/HPV runs are byte-for-byte unchanged. When the extra clinical columns are present, allow Effect to include sex + race (NOT smoking — unavailable). The confounder set is fixed by config, not GP-selectable (the program shouldn't roam over arbitrary confounders). Update/extend the relevant engine_v2 tests.

## Part D — site-stratified survival flag on the module ranking (validation layer)
IMPORTANT (verified): `_compute_module_ranking` currently **discards clinical** — line ~1282 is `M_all, y_all, _clin, _extra = _prepare_lab_data(...)` with clinical thrown away as `_clin`. Stop discarding it: capture the clinical frame, and align `is_oropharynx` to the held-out patients (the same `test_ids` it already reconstructs) so you can subset. Then, for the HNSC/HPV case add per-module:
- `combined_holdout_oropharynx`: the module's mean-aggregate AUROC recomputed **within the oropharynx subgroup only** (held-out patients with `is_oropharynx == True`), and a `survives_site` boolean = it stays close to the full-cohort combined AUROC (within a tolerance, e.g. drop < ~0.05) rather than collapsing toward 0.5. This tells us whether the module separates HPV beyond just marking oropharynx tissue.
- Guard small subgroups (skip / mark "n too small" if the oropharynx held-out has too few per class).
- Gate to HNSC; colorectal modules omit this gracefully.

## Part E — purity-survival flag (validation layer; heaviest part — may be phased)
Purity isn't in the data, so derive an **immune-infiltration proxy** in the validation layer (names allowed here, exactly like the rank diagnostics): per patient, the mean expression of a small standard immune-marker set (e.g. CD8A, GZMB, PRF1, CD3D, CD2 — a fixed curated list in one place). Since the matrix `M` is opaque-columned, resolve those marker SYMBOLS to their opaque IDs via the bounded reveal path (same as reference sets), then mean those columns of `M` per patient. Higher proxy = more infiltrate = lower purity. Then per module add:
- `combined_holdout_highpurity`: the module's mean-aggregate AUROC recomputed within the **low-infiltration (high-purity) subset** of held-out patients (e.g. bottom tertile of the proxy), and a `survives_purity` boolean (same tolerance idea). This catches modules whose HPV signal is really an immune-composition artifact (they'll collapse in the high-purity subset).
- This derivation lives in the validation/API layer — never in `engine/` or `engine_v2/`. If scope is tight, ship Parts A-D first and land Part E second; mark it clearly.

## Part F — UI: surface the flags + the winner's-curse caveat (presentation)
- In the Coordinated-modules table, show the survival flags per module (e.g. small "site ✓/✗" and "purity ✓/✗" chips, or the stratified AUROCs on expand). A module that survives both is a real candidate; one that collapses is a likely confound artifact.
- Add to the modules "?" the winner's-curse caveat we agreed on, verbatim intent: *"These groups are scored by re-evaluating ~2,900 explored sets on the same small held-out set and showing the best — so the very top values are optimistically biased (the luckiest of thousands). The winning program was chosen by cross-validation, which guards against that, so trust it as the engine's pick even when a table row scores higher."*

## CONSTRAINTS
- Airgap: clinical confounders (sex/race/site) and the derived purity proxy are named non-gene variables / validation-layer computations — never enter `engine/` or `engine_v2/` as gene identities. Module payloads stay opaque-ID + scores + flags.
- Defaults preserve current behaviour (Effect default = stage+age; non-HNSC unaffected). Colorectal + existing tests stay green.

## Checkpoint
- HNSC `clinical.parquet` carries race + tissue_site + is_oropharynx (with printed counts); sex already there.
- Effect adjusts on a configurable set, default stage+age unchanged; with HNSC clinical present it can include sex+race.
- Module ranking shows site-stratified (and, when Part E lands, purity) survival flags for HNSC/HPV; modules that are site/immune artifacts visibly collapse.
- Modules "?" carries the winner's-curse caveat.
- `pytest` green, `tsc` clean, airgap tests untouched.