Spaces:
Sleeping
Sleeping
| # Prompt β capability test: can the engine discover a planted gene ratio? (isolated) | |
| Build a **self-contained, known-answer capability test** (like the H1 fixture) | |
| that plants a synthetic target defined by a *ratio of two real genes*, rigs it so | |
| neither gene is useful alone, and checks whether the blind engine rediscovers the | |
| ratio. This is an engine unit test β NOT a biological claim, NOT a new UI | |
| objective, NOT a new dataset. | |
| **Isolation (non-negotiable):** live as a single new file `scripts/capability_ratio_test.py` | |
| (scripts/ is allowed to be biology-aware). It READS the existing processed matrix | |
| and reuses the existing sealed map via `airgap.reveal` for the final check. It | |
| writes NOTHING to disk, adds no dataset, no API route, no UI, does not modify the | |
| sealed map or any parquet. Deleting the file leaves zero trace. Do not touch | |
| `engine_v2/`, `dsl/`, `api/`, or `web/`. | |
| ## Why a small panel (important) | |
| Two individually-uninformative genes give the search NO univariate gradient, so | |
| finding them among ~20,000 columns is an impossible needle-hunt β that tests | |
| search-at-scale, not composition. To test composition fairly, restrict the search | |
| to a small controlled panel: the two planted genes + ~50 random decoy genes. | |
| ## Steps | |
| 1. **Load real expression.** `cohort = Load("processed")` (colorectal). Keep | |
| samples with complete expression. Work on the NAMED matrix here (scripts/ may | |
| see gene symbols). | |
| 2. **Pick a correlated gene pair A, B.** From highly-expressed, adequately-varying | |
| genes, find a pair with high positive correlation (e.g. Pearson r β₯ 0.6) β a | |
| ratio only hides the single-gene signal when A and B move together. (Search a | |
| few hundred random candidate pairs; take the first that satisfies the r | |
| threshold AND the step-4 checks. Make the pair configurable via CLI for | |
| reproducibility.) | |
| 3. **Define the planted target.** `zA, zB = zscore(A), zscore(B)`; | |
| `signal = zA - zB` (the log-ratio direction); `y = (signal > median(signal)).astype(int)`. | |
| y depends only on the A-vs-B balance, not on either level. | |
| 4. **Validity checks (assert before running β this is what makes the test | |
| meaningful):** | |
| - single-gene AUROC(A) and AUROC(B) each within ~0.5 Β± 0.07 (individually | |
| ~useless). If not, pick another pair. | |
| - the ratio itself, `max(AUROC(signal), 1-AUROC(signal))`, β₯ ~0.9 (jointly | |
| strong). | |
| - Print all three so the setup is auditable. | |
| 5. **Build the search panel.** Columns = {A, B} + ~50 random decoy genes from the | |
| matrix. **Control checks (print):** best single-gene AUROC over the whole panel | |
| β 0.5 (nothing wins alone), and AUROC of the plain mean of all panel genes β | |
| 0.5 (averaging doesn't work). These controls mean any high score can ONLY come | |
| from composition. | |
| 6. **Anonymise + run blind.** `M = anonymise(panel_expression)` (reuses the sealed | |
| map; opaque IDs). Run `engine_v2.run_v2_pipeline(M, y, objective=<binary AUROC>, | |
| ...)`: | |
| - objective: `from engine_v2.fitness import V2Objective; V2Objective(target="msi", | |
| binary=True)` β used purely as a binary-AUROC scorer; the "msi" name is just | |
| the internal carrier, the values are the planted label. | |
| - `prefilter_n=None` (panel is already small; must not univariate-filter out the | |
| ~0.5 genes), `scalar_share_override=0.0` (force per-patient-score / Vector | |
| programs so the winner is a readable Combine, not an Associate scalar), | |
| `coherence_weight=0.0`, diversity on (`tournament_k=2, p_mutate=0.85, | |
| immigrant_fraction=0.10`), a generous budget (e.g. `population_size=200, | |
| n_generations=40`), and a small `n_permutations` (e.g. 100) β or skip the null. | |
| - Run a few seeds (e.g. 3) and report each. | |
| 7. **Grade against the known answer.** For each seed's winner | |
| (`result["winning"]`): `reveal(winning["gene_ids"])` β symbols. PASS if the | |
| winner's genes are exactly {A, B} (or contain both) AND `winning["holdout_score"]` | |
| β« the panel's best single-gene AUROC (say β₯ 0.80 vs the ~0.5 controls) AND the | |
| `program_repr` contains a `Combine(` with `protected_div` (or `sub`/`mul` β any | |
| two-gene interaction that recovers the planted relationship; note which). | |
| Print PASS/FAIL, the winner `program_repr`, the revealed symbols, the holdout | |
| score, and the control numbers, per seed. Print an overall verdict (recovered | |
| the interaction in k / n seeds). | |
| ## Interpretation to print at the end (verbatim-ish) | |
| - PASS (recovered in most seeds): "The engine can discover a genuine two-gene | |
| interaction blind β so when a real target (HPV) yields only averages, that's | |
| because the biology doesn't need a ratio, not because the engine can't build | |
| one." | |
| - FAIL: "The engine did not recover the planted interaction even on a small panel | |
| β a real limitation to fix (raise Combine rate / diversity / budget) before | |
| claiming the DSL composes." | |
| ## CONSTRAINTS | |
| - Airgap intact: the engine sees only the anonymised opaque panel; symbols are | |
| revealed once at the end via `airgap.reveal` (bounded β the winner's genes). | |
| Nothing in `engine_v2` changes. | |
| - Isolated: one new file under `scripts/`, read-only w.r.t. data + sealed map, | |
| no API/UI/dataset/registry change. Existing tests + runs untouched. | |
| - The point is capability, not forcing the answer: do NOT tell the engine which | |
| genes matter or that the target is a ratio. Restricting to a panel and to | |
| Vector programs is a fair framing, not a hint (it still must find WHICH two | |
| genes and HOW to combine them). | |
| ## Checkpoint | |
| - `python -m scripts.capability_ratio_test` prints: the planted pair's single-gene | |
| AUROCs (~0.5), the ratio AUROC (~0.9+), the panel controls (best single β 0.5, | |
| panel-mean β 0.5), then per-seed winner `program_repr` + revealed genes + | |
| holdout score + PASS/FAIL, and an overall verdict. | |
| - No change to engine/API/UI/data; deleting the file leaves no trace. | |