Read docs/PROMPT_hpv_dataset.md

Prompt — Add HPV (head & neck) as a second dataset, with a top-level cancer selector

Add the TCGA HNSC (head & neck) cohort and an HPV-detection objective, behind a new top-level "Cancer / problem" selector. The DSL, engine, airgap, parameters, live view, population, program graph, and result/verdict are SHARED and unchanged — only the dataset-specific config swaps (objectives, biology panel, reference sets, diagnostics, post-hoc targets). Read the named code first; don't assume. Do this in two stages.

GUIDING PRINCIPLE: the engine is single. Everything that changes per cancer is config around it.

================================================================= ## STAGE 1 — Backend: the HNSC dataset + HPV objective

1a. Data pipeline (the main lift)

• FIRST verify the data source: confirm the cBioPortal TCGA HNSC study id and that its clinical data carries HPV status (HPV+/HPV−). Mirror the COADREAD provenance approach documented in CLAUDE.md (LFS-resolved media URL). State the study id + HPV field you used.
• Build a second processed cohort alongside the colorectal one: HNSC expression.parquet + clinical/labels including an hpv_status label (HPV+ / HPV−), restricted to samples with a called HPV status and complete expression. Put schema constants (an HNSC_PROCESSED_DIR, HNSC filenames, the HPV-status column + how +/− is derived) in data_pipeline/schema.py — do not hardcode in build. Ensure the build writes hpv_status into the labels so cohort.labels["hpv_status"] exists after Load.
• LOADING — IMPORTANT: dsl.Load (operators.py ~line 62) only schema-resolves the exact string "processed"; ANY other string is treated as a literal directory path. So do NOT write Load("processed_hnsc"). Instead, EITHER (a) extend Load to map named sources ("processed"→COADREAD dir, "processed_hnsc"→schema.HNSC_PROCESSED_DIR), OR (b) pass the HNSC directory Path straight to Load(hnsc_dir) — Load's existing else branch already loads any directory containing clinical.parquet + expression.parquet, so option (b) needs no change to Load.

1b. Objective

• Add a binary objective for HPV: spec { target: "hpv", metric: "auroc" }, reusing the SAME orientation-agnostic binary-AUROC machinery as MSI. Concretely: extend V2Objective.target (currently Literal["msi","tmb","none"]) to include "hpv" with binary=True; route its score_vector through the AUROC path and worst_score→0.5 (identical to MSI); add an HPV_OBJECTIVE and dispatch hpv+auroc in engine_v2.fitness.objective_from_spec. No new operators.

1c. Data loading per (dataset, target)

• _prepare_lab_data (api/app.py ~line 177) is currently keyed by target and always uses Load("processed") (colorectal). Generalise it to key by (dataset, target): add a dataset argument; for dataset="hnsc", target="hpv", load the HNSC cohort, anonymise its expression, and set y = (hpv_status == HPV+). Keep colorectal (dataset="coadread", msi/tmb/none) exactly as-is.

1d. API

• RunRequest gains a dataset: str field (default "coadread"). _worker passes it through to _prepare_lab_data. Validate dataset+objective combinations (e.g. hpv only valid for hnsc; msi/tmb only for coadread).
• REFERENCE_SETS (api/app.py ~line 106) becomes per-dataset: colorectal keeps MMR/immune; HNSC gets HPV reference sets — e.g. "HPV_markers" = {CDKN2A, …} and a "cell_cycle" / E2F-target set (define from a small curated list in one place, like MMR_GENES). /evaluate and any diagnostics select the reference sets for the active dataset.
• Diagnostics: the MSI/TMB-rank diagnostics are colorectal-specific — gate them to that dataset; HNSC can have its own (or none) for now.
• Airgap unchanged: anonymise HNSC expression; HPV status is a label (target), never a gene name. Extend the airgap tests to the HNSC path (no gene-symbol leak).

================================================================= ## STAGE 2 — Frontend: cancer selector + per-dataset config + full "?" coverage

2a. Top-level Cancer / problem selector

• Add a segmented control (same style as the Objective control) ABOVE the Objective row: Colorectal (TCGA COADREAD) | Head & Neck (TCGA HNSC). (No side "what swaps" panel — that was mockup annotation only.)
• Drive everything from a dataset registry: each entry = { label, dataset_id, objectives, biologyPanel, referenceSets, diagnostics, postHocTargets, axisLabels }. The page renders the selected entry's config around the unchanged engine. Selecting a cancer sets dataset on the run request and swaps the objective options, biology panel, reference sets, and diagnostics.

2b. HPV biology panel — include the schematic

• Add a "What is HPV? (the biology)" collapsed-by-default disclosure, mirroring the existing "What is MMR?" panel EXACTLY (same collapsible pattern). Embed the HPV reference SVG from repo root hpv_reference.svg (already created) verbatim, responsive (width 100%, height auto). Show it for the Head & Neck dataset; show the MMR panel for Colorectal.

2c. HPV objective + rich "?" explainer

• Add the "HPV detection" objective card for the HNSC dataset (objective spec {target:"hpv", metric:"auroc"}), plus "Unsupervised" (same engine).
• Give it the SAME rich click-to-open ParamHelp modal as the other objectives, in plain English (no assumed knowledge). Content: what it optimises ("a score that separates HPV+ from HPV− tumours"); how it's scored (AUROC: 0.5 = coin-flip, 1.0 = perfect; minority ~15–25% HPV+, so AUROC not accuracy); represented as {target: hpv, metric: AUROC}; and the honest note that this is detection/recovery of a known viral signature (p16/CDKN2A, cell-cycle), not new causation. Use the same intro/footer pattern as the other objective modals.

2d. Full "?" coverage — every objective, parameter, result metric, and chart axis

This is a hard requirement, including for HPV:

• Objectives: all have rich "?" explainers (above).
• Parameters: shared engine — the existing ParamHelp "?" already cover them; no change.
• Result metrics (Held-out, Permutation p, Nodes, Genes) and the verdict / post-hoc "?" and inline text must be objective-aware and read correctly for HPV (held-out = AUROC separation of HPV+/−; the post-hoc/verdict framing is the supervised one — "real and useful" vs "not reliable" — NOT the unsupervised two-questions framing).
• Charts: the fitness-curve y-axis label + its "?" for the HPV objective = "separation (AUROC)" with the AUROC explanation (same as MSI). Ensure the objective-aware label/tooltip records include the hpv case.
• All "?" text in plain English explaining the concept, consistent with the rest of the Lab.

2e. Reveal & evaluate

• For HNSC, the reference-set toggle shows the HPV sets (HPV_markers / cell_cycle) instead of MMR/immune; overlap + reveal work the same way. Expected win: revealing the winner shows it recovered CDKN2A/p16 + cell-cycle genes.

CONSTRAINTS

• Colorectal behaviour, airgap tests, and existing runs unchanged.
• The engine / DSL / parameters / live view / population / program graph are identical across datasets — only config swaps.

Checkpoint

After Stage 1: a CLI/API HPV run on HNSC completes and separates HPV+/− (held-out AUROC should be high). After Stage 2: the cancer selector swaps to Head & Neck, the "What is HPV?" schematic shows, an HPV run gives a strong held-out AUROC with a plain verdict, and reveal recovers p16/CDKN2A + cell-cycle genes. Airgap tests green; colorectal unchanged.