feat(landscape): bring experimental ALS therapy landscape to main (#19)

Applies the landscape feature (only content main was missing from the hf-clean deploy
lineage) on top of main: the Therapy Landscape tab, offline multi-label classifier
(Opus 4.8 + BioLORD cross-check + abstention, 0.90 precision on a 42-drug gold set),
landscape.json, and plotly==6.9.0. Preserves main's batch extractor + SSR/Gradio fixes.

Note: main and hf-clean have unrelated git histories, so this is applied as a feature
patch rather than a branch merge. The deduped data blobs are intentionally NOT included
(main does not track data/ — it lives in the dataset repo / deploy branch).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

CLAUDE.md

@@ -36,9 +36,10 @@ Candle-fire is a physician-facing ALS research intelligence tool. A physician as
 | `rag/indexer.py` | Build ChromaDB collection; section-aware chunking; `citation_count` in metadata |
 | `rag/retriever.py` | `search()`, `search_by_entities()`, citation-weighted re-ranking |
 | `agents/research_agent.py` | Multi-step synthesis agent (streaming): entity extraction → KG expansion → RAG → synthesis |
-| `app.py` | Gradio UI: loads graph + ChromaDB once at startup, streams responses |
+| `app.py` | Gradio UI (tabbed: Ask + Therapy Landscape): loads graph + ChromaDB + landscape once at startup |
+| `landscape.py` | Therapy Landscape rendering: Plotly sunburst + detail-panel HTML from `landscape.json` |
 | `main.py` | CLI interface (Rich console) |
-| `scripts/` | Offline pipeline scripts: run once in order (ingest → extract → build_graph → build_index) |
+| `scripts/` | Offline pipeline scripts: run once in order (ingest → extract → build_graph → build_index → build_landscape) |
 
 ## Offline Pipeline Run Order
 
@@ -60,7 +61,10 @@ uv run python scripts/build_graph.py
 # 5. Build ChromaDB vector index
 uv run python scripts/build_index.py
 
-# 6. run application
+# 6. Build the experimental therapy landscape (offline LLM classification via Batch API)
+uv run python scripts/build_landscape.py
+
+# 7. run application
 uv run gradio app.py
 ```
 
@@ -83,6 +87,9 @@ data/graph/als_graph.pkl          — NetworkX DiGraph (fast load)
 data/graph/als_graph.json         — human-readable graph export
 data/chroma/                      — ChromaDB SQLite store
 data/tools/                       — Claude tool input schemas (JSON)
+data/seeds/therapy_classes.json   — ALS mechanism taxonomy for the therapy landscape
+data/seeds/therapy_gold.json      — gold-labeled therapies for classifier eval/calibration
+data/landscape/landscape.json     — experimental therapy landscape (offline-built, committed to git)
 ```
 
 ## Environment Variables

COE-candle-fire-space-outage.md

@@ -0,0 +1,294 @@
+# Correction of Errors (COE): candle-fire HuggingFace Space Outage
+
+> **Status:** Draft · **Owner:** _<you>_ · **Reviewers:** _<add>_
+> **Service:** candle-fire — physician-facing ALS research intelligence tool (Gradio app on a HuggingFace Space)
+> **Severity:** _<assign>_ · **Date of incident:** 2026-07-23/24 · **Author:** _<you>_
+
+## 1. Summary
+
+While deploying a corrected (de-duplicated) knowledge-graph dataset to the
+candle-fire HuggingFace Space, the Space became intermittently and then fully
+unavailable, returning HTTP 403 / 503 / 500 to users. The underlying Gradio
+application was healthy the entire time (it booted and loaded data on every
+attempt); the failure was in the **serving layer** — Gradio's experimental
+server-side rendering (SSR) layer, which is unstable on this Space and was
+exposed when a forced restart triggered a cold rebuild. The incident was
+prolonged by the remediation process itself (repeated rebuilds/restarts while
+diagnosing). Service was restored by disabling Gradio SSR (`ssr_mode=False`) and
+pinning the Gradio version.
+
+## 2. Customer / User Impact
+
+- The candle-fire Space (a physician-facing research tool) returned errors
+  (403/503/500) for the duration of the incident — from intermittent failures
+  to a period of complete unavailability (all requests 503).
+- No data was lost or corrupted. The corrected dataset deployed successfully;
+  the outage was purely in application serving.
+- Duration: approximately _<fill in>_ (spanned the diagnosis + several rebuild
+  cycles). Each cold rebuild re-downloaded a ~1.3 GB index, adding multi-minute
+  unavailability windows.
+
+## 3. Timeline (ordered; times approximate)
+
+| Phase | Event |
+|---|---|
+| Deploy start | Corrected graph pickle uploaded to the `candle-fire-data` dataset repo (the Space downloads data from here at startup). |
+| Trigger | `factory_reboot=True` issued on the Space to force it to re-download the corrected pickle (its `_ensure_data()` only downloads "if not present"). This wiped the Space's storage and forced a full cold rebuild. |
+| Data-to-git | Deduped data files pushed to the Space git repo; initial pushes rejected by HuggingFace's 10 MB non-LFS file limit (files had grown to ~19–20 MB); resolved by moving them to Git LFS. |
+| First symptoms | Space began returning 403 → 503 → 500 intermittently (~2/10 success). |
+| Diagnosis 1 | Identified Gradio's experimental Node SSR as the failing layer. Deployed `ssr_mode=False`; success rate improved (~partial), but measurements were taken during rebuild churn and were noisy. |
+| Regression | A second `factory_reboot` (remediation attempt) moved the Space from partial to **fully down** (0/12, all 503). |
+| Wrong turn | Reverted `ssr_mode=False` on a "restore known-good config" assumption; this re-enabled SSR and kept the Space fully down (0/12). |
+| Correction | Log evidence showed SSR-on → 503 vs SSR-off → recovers. Re-applied `ssr_mode=False` and pinned `gradio==6.14.0`; **stopped rebuilding** and allowed the Space to fully settle. |
+| Resolved | After the post-boot warmup window, the Space held **12/12 HTTP 200**. Fix backfilled to git and synced to `main`. |
+
+## 4. Root Cause
+
+**The application depended on Gradio's experimental SSR serving layer (enabled by
+default in Gradio 6.x), which is unstable on this Space. This dependency was
+unvalidated and had been masked because the Space had been serving from warm/
+persistent state and had not cold-started under that configuration — until a
+forced `factory_reboot` (required to pick up new data) triggered a clean cold
+rebuild that exposed it.**
+
+### 5 Whys
+1. **Why was the Space down?** HuggingFace's proxy returned 503 — the app was
+   running on port 7860 but was unreachable through the serving layer.
+2. **Why was it unreachable?** Gradio's experimental Node SSR layer failed to
+   serve on this Space.
+3. **Why did SSR fail now and not before?** A forced factory-reboot cold-started
+   the app under the default (SSR-on) configuration; the Space had previously
+   been running on warm state and had never cleanly cold-started under it.
+4. **Why was a factory-reboot needed at all?** The data-refresh path
+   (`_ensure_data()` downloads data only if it is not already present on disk)
+   required wiping Space storage to force a re-download of the corrected pickle.
+5. **Why did the app rely on an unstable serving mode?** Gradio 6.x turns on
+   experimental SSR by default; it was never explicitly disabled, pinned, or
+   validated for this Space.
+
+## 5. Detection
+
+- Detection was **user-reported** (errors observed directly on the Space URL).
+- There was no automated health check / uptime probe on the Space that would
+  have caught the regression at deploy time.
+
+## 6. Resolution
+
+- Set `ssr_mode=False` in `demo.launch()` (bypass Gradio's experimental SSR;
+  serve the classic client-rendered app directly).
+- Pinned `gradio==6.14.0` in `requirements.txt` and `pyproject.toml` to match the
+  Space's README `sdk_version` and prevent version drift on future rebuilds.
+- Allowed the Space to fully settle after the final deploy before validating,
+  and confirmed 12/12 HTTP 200.
+- Backfilled the fix to `origin/hf-clean`, synced it to `main` (PR #15), and
+  realigned `origin/hf-clean` with the Space's `hf/main` history so future
+  deploys are clean fast-forwards.
+
+## 7. What Went Wrong in the Response (contributing factors)
+
+- **Over-churning:** Multiple `factory_reboot`s and rebuilds during diagnosis
+  each forced a ~1.3 GB cold-start re-download, extending downtime and adding
+  noise. One `factory_reboot` moved the Space from partial to fully down.
+- **Measuring during warmup:** Success-rate probes were run during rebuild/
+  warmup windows, producing misleading partial results and one false "worse".
+- **Reverting a working mitigation:** `ssr_mode=False` was reverted on an
+  assumption ("restore known-good") rather than evidence, which re-broke it.
+- **No staging/validation before prod:** Changes were validated directly on the
+  production Space.
+
+## 8. Action Items
+
+| # | Action | Type | Owner | Status |
+|---|---|---|---|---|
+| 1 | Disable Gradio SSR (`ssr_mode=False`) and pin `gradio==6.14.0` | Fix | _<you>_ | ✅ Done |
+| 2 | Redesign data refresh so updating the dataset repo does **not** require a `factory_reboot` / full cold rebuild (e.g., version the data path or use a lighter refresh signal) — removes the destabilizing trigger and the 1.3 GB re-download | Prevent | | Todo |
+| 3 | Pin all runtime-critical dependencies (no `>=,<` ranges that drift on rebuild); keep `requirements.txt` in sync with README `sdk_version` | Prevent | | Todo |
+| 4 | Write a deploy runbook: two deploy targets (dataset repo for data, Space git for code), the LFS >10 MB requirement, the cold-start warmup window, and how to verify (probe for stable 200s) | Prevent | | Todo |
+| 5 | Add post-deploy verification that waits out the warmup window before declaring success (don't measure during rebuild) | Detect | | Todo |
+| 6 | Add an automated uptime/health probe on the Space with alerting | Detect | | Todo |
+| 7 | Incident-response guidance: during diagnosis, change one variable at a time, avoid repeated factory-reboots, and don't revert a working mitigation without evidence | Process | | Todo |
+| 8 | Reduce cold-start fragility (the ~1.3 GB index download) — evaluate persistent storage strategy or a smaller/streamed index | Prevent | | Todo |
+| 9 | Consider a staging Space to validate rebuilds before touching production | Prevent | | Todo |
+
+## 9. Corrective Actions — Concrete Changes (Action Items 2–4)
+
+### Item 2 — Data refresh without a factory-reboot
+
+**Problem.** `_ensure_data()` short-circuits when the files already exist
+(`if not need_chroma and not need_graph: return`), so a running Space never
+picks up updated data. That is what forced the `factory_reboot` (storage wipe),
+which cold-rebuilt the image and exposed the SSR failure.
+
+**Code change (`app.py`).** Always call the HF download helpers and rely on
+`huggingface_hub`'s etag caching — an unchanged file is a cheap metadata check,
+a changed file is re-fetched. This also removes the fragile "flatten" hack by
+downloading into the correct layout in the first place.
+
+```python
+def _ensure_data() -> None:
+    """Sync the chroma index + graph pickle from the HF dataset repo.
+
+    Always calls the HF download helpers. huggingface_hub does etag-based
+    caching, so an unchanged file is a cheap metadata check and a changed file
+    is re-downloaded. A plain restart therefore picks up new data — no
+    factory-reboot / storage wipe required.
+    """
+    try:
+        from huggingface_hub import hf_hub_download, snapshot_download
+        _logger.info("Syncing data from HF dataset (etag-cached)...")
+        # Repo layout is chroma/…  -> land it at CHROMA_DIR/chroma.sqlite3 by
+        # downloading into CHROMA_DIR's parent (no post-hoc file moves, so etag
+        # caching keeps working across restarts).
+        snapshot_download(
+            repo_id=_HF_DATASET,
+            repo_type="dataset",
+            local_dir=str(CHROMA_DIR.parent),
+            allow_patterns=["chroma/**"],
+        )
+        hf_hub_download(
+            repo_id=_HF_DATASET,
+            repo_type="dataset",
+            filename="graph/als_graph.pkl",
+            local_dir=str(GRAPH_PICKLE_PATH.parent.parent),
+        )
+        _logger.info("Data sync complete")
+    except Exception as e:
+        _logger.warning(f"Data sync from HF dataset failed: {e}")
+```
+
+**Procedure change.** To ship new data, update the dataset repo, then issue a
+**plain restart** — never `factory_reboot=True`:
+
+```python
+from huggingface_hub import HfApi
+# Re-runs app.py on the SAME built image (no dep reinstall, no version drift);
+# _ensure_data() then pulls only the changed file via etag.
+HfApi().restart_space("KevinIsCoding/candle-fire")
+```
+
+**Infra change (removes the 1.3 GB cold-start entirely).** Enable **persistent
+storage** on the Space (Settings → Storage). With persistence the chroma index
+survives restarts and `_ensure_data()`'s etag check skips it — only a changed
+pickle downloads. Without persistence, storage is ephemeral and every restart
+re-downloads ~1.3 GB. Reserve `factory_reboot=True` for genuine image/dependency
+changes, and treat it as a change that can expose cold-start regressions.
+
+### Item 3 — Pin runtime-critical dependencies
+
+**Problem.** `requirements.txt` used floors/ranges (`gradio>=6.14.0,<7.0.0`,
+`chromadb>=0.5.0`, …). A rebuild can resolve *newer* versions than the
+last-known-good, and the Gradio pin drifted away from the README `sdk_version`.
+
+**Code change (`requirements.txt`).** Pin exact versions (source them from a
+`pip freeze` on the currently-healthy Space). The Gradio pin MUST equal the
+README front-matter `sdk_version`.
+
+```text
+# Pin exact versions — regenerate from `pip freeze` on the known-good Space.
+# gradio MUST equal README front-matter sdk_version ("6.14.0").
+anthropic==<frozen>
+gradio==6.14.0
+chromadb==<frozen>
+networkx==<frozen>
+biopython==<frozen>
+httpx==<frozen>
+python-dotenv==<frozen>
+rich==<frozen>
+sentence-transformers==<frozen>
+openai==<frozen>
+rapidfuzz==<frozen>
+```
+
+Mirror the same `gradio==6.14.0` pin in `pyproject.toml`, and prefer committing a
+lockfile (`uv.lock`) as the source of truth.
+
+**Guardrail (CI / pre-commit)** — fail the build if README `sdk_version` and the
+`requirements.txt` Gradio pin disagree:
+
+```bash
+#!/usr/bin/env bash
+# scripts/check_gradio_pin.sh
+set -euo pipefail
+readme_ver=$(grep -E '^sdk_version:' README.md | tr -d ' "' | cut -d: -f2)
+req_ver=$(grep -E '^gradio==' requirements.txt | cut -d= -f3)
+[[ "$readme_ver" == "$req_ver" ]] || {
+  echo "Gradio mismatch: README sdk_version=$readme_ver vs requirements gradio==$req_ver" >&2
+  exit 1
+}
+echo "Gradio pin OK ($readme_ver)"
+```
+
+### Item 4 — Deploy runbook (`docs/DEPLOY.md`)
+
+candle-fire has **two deploy targets**: **data** → dataset repo
+`KevinIsCoding/candle-fire-data`; **code** → Space git `hf/main` (deploy lineage
+`hf-clean`). The app downloads data at startup — data is NOT served from the
+Space git.
+
+**A. Data-only update (most common)**
+1. Rebuild artifacts: `uv run python scripts/build_graph.py` (plus
+   `build_index.py` if the corpus changed).
+2. Upload to the dataset repo:
+   ```bash
+   hf upload KevinIsCoding/candle-fire-data data/graph/als_graph.pkl graph/als_graph.pkl --repo-type dataset
+   ```
+3. Plain restart (NOT factory reboot):
+   ```bash
+   python -c "from huggingface_hub import HfApi; HfApi().restart_space('KevinIsCoding/candle-fire')"
+   ```
+4. Verify (section D). No code push needed.
+
+**B. Code update**
+1. Make the change on `hf-clean` (or PR into it).
+2. Any tracked data file > 10 MB MUST be Git LFS (HF rejects >10 MB non-LFS):
+   ```bash
+   git lfs track "data/graph/als_graph.json" "data/extracted/entities.jsonl"
+   ```
+3. Push to the Space (fast-forward; `hf/main` and `origin/hf-clean` stay in lockstep):
+   ```bash
+   git push origin hf-clean && git push hf hf-clean:main
+   ```
+4. The Space rebuilds automatically. Verify (section D).
+
+**C. Golden rules**
+- Do NOT `factory_reboot` for routine deploys — it wipes storage and rebuilds the
+  image, which can expose cold-start regressions. Use a plain restart.
+- `ssr_mode=False` must stay in `demo.launch()` (Gradio SSR 503s on this Space).
+- `requirements.txt` gradio pin MUST equal README `sdk_version`.
+
+**D. Post-deploy verification (wait out the warmup window).** The app logs
+"Running on local URL" *before* HF's proxy is ready; a warmup window (~1–2 min,
+longer on a cold 1.3 GB download) returns 503 until it settles. Wait, then probe:
+
+```bash
+python - <<'PY'
+import time, urllib.request
+from huggingface_hub import HfApi
+while HfApi().get_space_runtime("KevinIsCoding/candle-fire").stage != "RUNNING":
+    time.sleep(15)
+time.sleep(120)  # warmup buffer
+url, ok = "https://keviniscoding-candle-fire.hf.space/", 0
+for _ in range(12):
+    try: ok += urllib.request.urlopen(url, timeout=30).status == 200
+    except Exception: pass
+    time.sleep(6)
+print(f"{ok}/12 healthy —", "DEPLOY OK" if ok >= 11 else "INVESTIGATE")
+PY
+```
+
+Target: ≥ 11/12 HTTP 200. If it stays 503 *after* warmup, read the run logs —
+do not reflexively rebuild.
+
+## 10. Lessons Learned
+
+- A running Space can be serving on **warm state** that a cold rebuild will not
+  reproduce; forcing a cold rebuild (factory-reboot) is a high-risk action, not
+  a routine one.
+- **Unpinned/defaulted serving behavior** (Gradio's experimental SSR, unpinned
+  Gradio version) is a latent risk that surfaces only on rebuild.
+- During an incident, **stop changing things and let the system settle** before
+  measuring; rapid iteration on a slow-cold-start service produces misleading
+  signals and extends the outage.
+- The application layer being healthy does not mean the service is up — the
+  **serving/proxy layer** is a distinct failure domain worth checking first
+  (the 500/503 body identified the source as the platform, not the app).

app.py

@@ -53,6 +53,12 @@ _n_chunks = _collection.count() if _collection else 0
 _n_trials = len(_trials)
 _kg_nodes = _graph.number_of_nodes() if _graph else 0
 
+# Experimental therapy landscape (offline-built artifact; loaded once)
+import landscape as landscape_mod
+
+_landscape = landscape_mod.load_landscape()
+_landscape_classes = landscape_mod.class_names(_landscape)
+
 # ── Example questions ─────────────────────────────────────────────────────────
 
 _EXAMPLES = [
@@ -123,52 +129,135 @@ Always verify claims with primary sources before applying to patient care.
 </div>"""
 
 
+def _landscape_select(class_name: str, phases):
+    """When the class (or phase filter) changes: repopulate therapies and show the first one's detail."""
+    labels = landscape_mod.therapy_labels(_landscape, class_name, phases)
+    first = labels[0] if labels else None
+    return (
+        gr.update(choices=labels, value=first),
+        landscape_mod.therapy_detail_md(_landscape, class_name, first or ""),
+        landscape_mod.trials_table_html(_landscape, class_name, first or ""),
+    )
+
+
+def _phase_change(phases, class_name: str):
+    """Phase filter: rebuild the sunburst and repopulate the current class's therapies."""
+    therapy_update, detail, trials = _landscape_select(class_name, phases)
+    return (landscape_mod.build_sunburst(_landscape, phases), therapy_update, detail, trials)
+
+
+def _therapy_select(class_name: str, therapy_label: str):
+    return (
+        landscape_mod.therapy_detail_md(_landscape, class_name, therapy_label),
+        landscape_mod.trials_table_html(_landscape, class_name, therapy_label),
+    )
+
+
 with gr.Blocks(title="Candle-Fire — ALS Research Intelligence") as demo:
 
-    with gr.Column(elem_classes="container"):
-
-        gr.Markdown(_TITLE_MD)
-
-        gr.HTML(
-            f'<div class="status-bar">'
-            f'{_n_chunks} paper chunks &nbsp;·&nbsp; '
-            f'{_n_trials} clinical trials &nbsp;·&nbsp; '
-            f'{_kg_nodes} knowledge graph nodes'
-            f'</div>'
-        )
-
-        chatbot = gr.Chatbot(
-            value=[],
-            height=520,
-            show_label=False,
-            sanitize_html=False,
-            avatar_images=(None, "assets/flame.svg"),
-            placeholder="Ask a question about ALS research to get started.",
-        )
-
-        with gr.Row():
-            msg_box = gr.Textbox(
-                placeholder="e.g. What is the evidence for tofersen targeting SOD1?",
-                show_label=False,
-                scale=9,
-                autofocus=True,
-                lines=1,
-            )
-            send_btn = gr.Button("Ask", scale=1, variant="primary", min_width=80)
-
-        gr.Markdown("**Example questions** — click to populate:")
-
-        with gr.Row():
-            with gr.Column(scale=1):
-                for ex in _EXAMPLES[:3]:
-                    btn = gr.Button(ex, size="sm", variant="secondary")
-                    btn.click(fn=lambda t=ex: t, outputs=[msg_box])
-            with gr.Column(scale=1):
-                for ex in _EXAMPLES[3:]:
-                    btn = gr.Button(ex, size="sm", variant="secondary")
-                    btn.click(fn=lambda t=ex: t, outputs=[msg_box])
-
-        gr.HTML(_DISCLAIMER_MD)
+    with gr.Tabs():
+
+        with gr.Tab("💬 Ask"):
+            with gr.Column(elem_classes="container"):
+
+                gr.Markdown(_TITLE_MD)
+
+                gr.HTML(
+                    f'<div class="status-bar">'
+                    f'{_n_chunks} paper chunks &nbsp;·&nbsp; '
+                    f'{_n_trials} clinical trials &nbsp;·&nbsp; '
+                    f'{_kg_nodes} knowledge graph nodes'
+                    f'</div>'
+                )
+
+                chatbot = gr.Chatbot(
+                    value=[],
+                    height=520,
+                    show_label=False,
+                    sanitize_html=False,
+                    avatar_images=(None, "assets/flame.svg"),
+                    placeholder="Ask a question about ALS research to get started.",
+                )
+
+                with gr.Row():
+                    msg_box = gr.Textbox(
+                        placeholder="e.g. What is the evidence for tofersen targeting SOD1?",
+                        show_label=False,
+                        scale=9,
+                        autofocus=True,
+                        lines=1,
+                    )
+                    send_btn = gr.Button("Ask", scale=1, variant="primary", min_width=80)
+
+                gr.Markdown("**Example questions** — click to populate:")
+
+                with gr.Row():
+                    with gr.Column(scale=1):
+                        for ex in _EXAMPLES[:3]:
+                            btn = gr.Button(ex, size="sm", variant="secondary")
+                            btn.click(fn=lambda t=ex: t, outputs=[msg_box])
+                    with gr.Column(scale=1):
+                        for ex in _EXAMPLES[3:]:
+                            btn = gr.Button(ex, size="sm", variant="secondary")
+                            btn.click(fn=lambda t=ex: t, outputs=[msg_box])
+
+                gr.HTML(_DISCLAIMER_MD)
+
+        with gr.Tab("🧭 Therapy Landscape"):
+            with gr.Column(elem_classes="container"):
+                gr.Markdown(
+                    "### 🧭 Experimental ALS Therapy Landscape\n"
+                    "Explore experimental therapies by **mechanism class → therapy → clinical trials** "
+                    "(recruiting & closed). Click a wedge to zoom; use the selectors for trial details."
+                )
+                if _landscape is None:
+                    gr.Markdown(
+                        "*Landscape not built yet — run `uv run python scripts/build_landscape.py`.*"
+                    )
+                else:
+                    _init_class = _landscape_classes[0]
+                    _init_labels = landscape_mod.therapy_labels(_landscape, _init_class)
+                    _init_label = _init_labels[0] if _init_labels else None
+
+                    sunburst = gr.Plot(landscape_mod.build_sunburst(_landscape), show_label=False)
+
+                    phase_cb = gr.CheckboxGroup(
+                        choices=landscape_mod.PHASE_OPTIONS, value=landscape_mod.PHASE_OPTIONS,
+                        label="Filter by trial phase",
+                        info="Show therapies with a trial in the selected phase(s). All selected = the whole picture.",
+                    )
+
+                    with gr.Row():
+                        class_dd = gr.Dropdown(
+                            choices=_landscape_classes, value=_init_class,
+                            label="Mechanism class", scale=1,
+                        )
+                        therapy_dd = gr.Dropdown(
+                            choices=_init_labels, value=_init_label,
+                            label="Therapy", scale=1,
+                        )
+
+                    detail_md = gr.Markdown(
+                        landscape_mod.therapy_detail_md(_landscape, _init_class, _init_label or "")
+                    )
+                    trials_html = gr.HTML(
+                        landscape_mod.trials_table_html(_landscape, _init_class, _init_label or "")
+                    )
+
+                    class_dd.change(
+                        _landscape_select, inputs=[class_dd, phase_cb],
+                        outputs=[therapy_dd, detail_md, trials_html],
+                    )
+                    therapy_dd.change(
+                        _therapy_select, inputs=[class_dd, therapy_dd],
+                        outputs=[detail_md, trials_html],
+                    )
+                    phase_cb.change(
+                        _phase_change, inputs=[phase_cb, class_dd],
+                        outputs=[sunburst, therapy_dd, detail_md, trials_html],
+                    )
+
+                gr.HTML(_DISCLAIMER_MD)
 
     submit_kwargs = dict(
         fn=respond,

config.py

@@ -23,6 +23,18 @@ GRAPH_JSON_PATH = DATA_DIR / "graph" / "als_graph.json"
 CHROMA_DIR = DATA_DIR / "chroma"
 CHROMA_COLLECTION = "als_papers"
 
+# Experimental therapy landscape (offline-built, committed to git — small)
+LANDSCAPE_PATH = DATA_DIR / "landscape" / "landscape.json"
+LANDSCAPE_PROGRESS_PATH = DATA_DIR / "landscape" / ".progress.json"
+LANDSCAPE_BATCH_STATE_PATH = DATA_DIR / "landscape" / ".batch_state.json"
+THERAPY_CLASSES_PATH = DATA_DIR / "seeds" / "therapy_classes.json"
+THERAPY_GOLD_PATH = DATA_DIR / "seeds" / "therapy_gold.json"
+# Mechanism classification (v2): frontier model, grounded + multi-label + abstaining.
+LANDSCAPE_MODEL = "claude-opus-4-8"
+LANDSCAPE_MIN_CONFIDENCE = 0.55   # τ — drop LLM mechanisms below this confidence
+LANDSCAPE_XCHECK_MIN_COSINE = 0.20  # BioLORD guard: drop a mechanism whose justification↔class cosine is below this (loose; catches gross mismatch)
+LANDSCAPE_EVIDENCE_ABSTRACTS = 4  # top abstracts retrieved per therapy as MoA evidence
+
 # Seed entity files
 MANUAL_SEEDS_PATH = DATA_DIR / "seeds" / "manual_seeds.json"
 DERIVED_SEEDS_PATH = DATA_DIR / "seeds" / "derived_seeds.json"

data/seeds/therapy_classes.json

@@ -0,0 +1,17 @@
+{
+  "description": "Curated ALS mechanism/target taxonomy for the Experimental Therapy Landscape. The 'name' values are the canonical class labels used as the classification enum in data/tools/classify_therapy.json and as the top ring of the sunburst. Seeded from the graph's strongest Mechanism/Gene nodes.",
+  "classes": [
+    {"id": "tdp43", "name": "TDP-43 proteinopathy", "description": "Therapies targeting TDP-43 aggregation, mislocalization, or cryptic splicing (TARDBP, STMN2, UNC13A)."},
+    {"id": "sod1", "name": "SOD1", "description": "Therapies targeting SOD1 in SOD1-ALS — e.g., SOD1-lowering antisense oligonucleotides, misfolded-SOD1 clearance."},
+    {"id": "c9orf72", "name": "C9orf72", "description": "Therapies targeting the C9orf72 hexanucleotide repeat expansion, RNA foci, or dipeptide-repeat proteins."},
+    {"id": "fus", "name": "FUS", "description": "Therapies targeting FUS aggregation or nuclear mislocalization in FUS-ALS."},
+    {"id": "neuroinflammation", "name": "Neuroinflammation", "description": "Therapies modulating microglia, astrocytes, or immune/inflammatory signaling."},
+    {"id": "oxidative_stress", "name": "Oxidative stress", "description": "Antioxidants and therapies countering reactive oxygen species and free-radical damage (e.g., edaravone)."},
+    {"id": "mitochondrial", "name": "Mitochondrial dysfunction", "description": "Therapies improving mitochondrial bioenergetics, membrane stability, or apoptosis regulation."},
+    {"id": "excitotoxicity", "name": "Glutamate excitotoxicity", "description": "Therapies reducing glutamate-mediated excitotoxic motor-neuron injury (e.g., riluzole, AMPA-receptor antagonists)."},
+    {"id": "proteostasis", "name": "Proteostasis / autophagy", "description": "Therapies enhancing protein clearance — chaperones, autophagy induction, or the ubiquitin-proteasome system (e.g., arimoclomol)."},
+    {"id": "rna_metabolism", "name": "RNA metabolism", "description": "Therapies targeting broad RNA processing/splicing dysregulation not specific to a single gene above."},
+    {"id": "neurotrophic", "name": "Neurotrophic / regenerative", "description": "Neurotrophic factors, stem-cell, and gene therapies aiming to protect or regenerate motor neurons."},
+    {"id": "symptomatic_other", "name": "Symptomatic / Other", "description": "Symptomatic treatments, repurposed drugs, or therapies whose ALS mechanism is unclear or does not fit another class."}
+  ]
+}

data/seeds/therapy_gold.json

@@ -0,0 +1,47 @@
+{
+  "description": "Hand-labeled gold set of well-known ALS therapies with their established mechanism class(es) from data/seeds/therapy_classes.json. Used by scripts/eval_landscape.py to measure multi-label precision/recall and to calibrate the confidence threshold, and as few-shot exemplars in the classifier prompt. Class names must match therapy_classes.json exactly. 'mechanisms' lists all well-supported classes (first = primary).",
+  "gold": [
+    {"therapy": "Riluzole", "mechanisms": ["Glutamate excitotoxicity"]},
+    {"therapy": "Edaravone", "aliases": ["MCI-186", "MT-1186", "Radicava"], "mechanisms": ["Oxidative stress"]},
+    {"therapy": "Tofersen", "aliases": ["Qalsody", "BIIB067"], "mechanisms": ["SOD1"]},
+    {"therapy": "CNM-Au8", "mechanisms": ["Mitochondrial dysfunction", "Oxidative stress"]},
+    {"therapy": "Arimoclomol", "mechanisms": ["Proteostasis / autophagy"]},
+    {"therapy": "AMX0035", "aliases": ["Relyvrio", "sodium phenylbutyrate and taurursodiol"], "mechanisms": ["Mitochondrial dysfunction", "Proteostasis / autophagy"]},
+    {"therapy": "Masitinib", "mechanisms": ["Neuroinflammation"]},
+    {"therapy": "Ibudilast", "aliases": ["MN-166"], "mechanisms": ["Neuroinflammation"]},
+    {"therapy": "NP001", "mechanisms": ["Neuroinflammation"]},
+    {"therapy": "Verdiperstat", "aliases": ["AZD3241"], "mechanisms": ["Neuroinflammation", "Oxidative stress"]},
+    {"therapy": "RNS60", "mechanisms": ["Neuroinflammation"]},
+    {"therapy": "Zilucoplan", "mechanisms": ["Neuroinflammation"]},
+    {"therapy": "Ravulizumab", "mechanisms": ["Neuroinflammation"]},
+    {"therapy": "Fingolimod", "aliases": ["Gilenya"], "mechanisms": ["Neuroinflammation"]},
+    {"therapy": "Dexpramipexole", "mechanisms": ["Mitochondrial dysfunction"]},
+    {"therapy": "Rasagiline", "mechanisms": ["Mitochondrial dysfunction", "Oxidative stress"]},
+    {"therapy": "Tauroursodeoxycholic acid", "aliases": ["TUDCA", "taurursodiol"], "mechanisms": ["Mitochondrial dysfunction", "Proteostasis / autophagy"]},
+    {"therapy": "Olesoxime", "mechanisms": ["Mitochondrial dysfunction"]},
+    {"therapy": "Cu(II)ATSM", "aliases": ["CuATSM", "copper ATSM"], "mechanisms": ["Oxidative stress", "Mitochondrial dysfunction"]},
+    {"therapy": "Deferiprone", "mechanisms": ["Oxidative stress"]},
+    {"therapy": "Ceftriaxone", "mechanisms": ["Glutamate excitotoxicity"]},
+    {"therapy": "Talampanel", "mechanisms": ["Glutamate excitotoxicity"]},
+    {"therapy": "Memantine", "mechanisms": ["Glutamate excitotoxicity"]},
+    {"therapy": "Lithium carbonate", "aliases": ["lithium"], "mechanisms": ["Proteostasis / autophagy"]},
+    {"therapy": "Rapamycin", "aliases": ["sirolimus"], "mechanisms": ["Proteostasis / autophagy"]},
+    {"therapy": "Trehalose", "mechanisms": ["Proteostasis / autophagy"]},
+    {"therapy": "Sodium phenylbutyrate", "mechanisms": ["Proteostasis / autophagy"]},
+    {"therapy": "Guanabenz", "mechanisms": ["Proteostasis / autophagy"]},
+    {"therapy": "Pridopidine", "mechanisms": ["Neurotrophic / regenerative", "Proteostasis / autophagy"]},
+    {"therapy": "Methylcobalamin", "aliases": ["mecobalamin", "E0302"], "mechanisms": ["Neurotrophic / regenerative"]},
+    {"therapy": "Fasudil", "mechanisms": ["Neurotrophic / regenerative", "Neuroinflammation"]},
+    {"therapy": "NurOwn", "aliases": ["MSC-NTF", "debamestrocel"], "mechanisms": ["Neurotrophic / regenerative"]},
+    {"therapy": "Lenzumestrocel", "aliases": ["Neuronata-R"], "mechanisms": ["Neurotrophic / regenerative"]},
+    {"therapy": "Ozanezumab", "mechanisms": ["Neurotrophic / regenerative"]},
+    {"therapy": "Jacifusen", "aliases": ["ION363"], "mechanisms": ["FUS"]},
+    {"therapy": "BIIB078", "mechanisms": ["C9orf72"]},
+    {"therapy": "WVE-004", "mechanisms": ["C9orf72"]},
+    {"therapy": "SAR443820", "aliases": ["DNL788"], "mechanisms": ["Neuroinflammation"]},
+    {"therapy": "Reldesemtiv", "aliases": ["CK-2127107"], "mechanisms": ["Symptomatic / Other"]},
+    {"therapy": "Tirasemtiv", "mechanisms": ["Symptomatic / Other"]},
+    {"therapy": "Mexiletine", "mechanisms": ["Symptomatic / Other"]},
+    {"therapy": "Levosimendan", "mechanisms": ["Symptomatic / Other"]}
+  ]
+}

data/tools/classify_therapy.json

@@ -0,0 +1,58 @@
+{
+  "type": "object",
+  "properties": {
+    "therapy_key": {
+      "type": "string",
+      "description": "Echo back verbatim the therapy_key given for this therapy in the prompt."
+    },
+    "canonical_name": {
+      "type": "string",
+      "description": "Most recognizable canonical name for this therapy. Merge development codes and synonyms to the common name (e.g., 'MCI-186' and 'MT-1186' -> 'Edaravone')."
+    },
+    "aliases": {
+      "type": "array",
+      "items": {"type": "string"},
+      "description": "Other names or development codes for this therapy seen across its trials."
+    },
+    "modality": {
+      "type": "string",
+      "enum": ["Small molecule", "Antisense oligonucleotide", "Biologic/Antibody", "Cell therapy", "Gene therapy", "Peptide", "Dietary supplement", "Combination", "Other"],
+      "description": "Therapeutic modality."
+    },
+    "target": {
+      "type": "string",
+      "description": "Primary molecular/biological target (e.g., 'SOD1 mRNA', 'AMPA receptor'). Use 'Unknown' if it cannot be determined from the provided evidence."
+    },
+    "mechanisms": {
+      "type": "array",
+      "description": "All ALS mechanism classes this therapy is proposed to act through, each supported by a direct quote from the provided evidence. MULTI-LABEL: include every well-supported class. If the evidence does not establish any mechanism, return an EMPTY array (abstain) rather than guessing.",
+      "items": {
+        "type": "object",
+        "properties": {
+          "class": {
+            "type": "string",
+            "enum": ["TDP-43 proteinopathy", "SOD1", "C9orf72", "FUS", "Neuroinflammation", "Oxidative stress", "Mitochondrial dysfunction", "Glutamate excitotoxicity", "Proteostasis / autophagy", "RNA metabolism", "Neurotrophic / regenerative", "Symptomatic / Other"],
+            "description": "The mechanism class from the ALS taxonomy."
+          },
+          "role": {
+            "type": "string",
+            "enum": ["primary", "contributing"],
+            "description": "Whether this is the therapy's primary mechanism or a secondary/contributing one."
+          },
+          "confidence": {
+            "type": "number",
+            "minimum": 0.0,
+            "maximum": 1.0,
+            "description": "Confidence that this therapy acts through this mechanism, based ONLY on the provided evidence (1.0 = well-established, 0.4 = weak/inferred)."
+          },
+          "evidence_quote": {
+            "type": "string",
+            "description": "A short verbatim quote from the provided evidence that supports this mechanism. Required — if you cannot quote supporting evidence, do not include this mechanism."
+          }
+        },
+        "required": ["class", "role", "confidence", "evidence_quote"]
+      }
+    }
+  },
+  "required": ["therapy_key", "canonical_name", "modality", "target", "mechanisms"]
+}

docs/DESIGN-vector-db.md

@@ -0,0 +1,150 @@
+# Design Doc: candle-fire Vector DB — Deployment Fix & Migration Path
+
+> **Status:** Draft for review · **Author:** _<you>_ · **Reviewers:** _<add>_ · **Date:** 2026-07
+> This is a design doc to review/modify **before** implementation. It ends with open
+> decisions (Section 10) you should resolve; the recommendation is phased so Phase 0 can
+> proceed independently of the Phase 1 decision.
+
+## 1. Context & Problem
+
+candle-fire's RAG layer uses an **embedded ChromaDB (SQLite)** vector index. The concern
+raised: the vector DB "keeps getting larger" — is it time to migrate (e.g., to AWS)?
+
+**Current numbers (measured):**
+- **30,967 vectors** (chunks) from ~10k papers; **1.2 GB** on disk (`data/chroma/chroma.sqlite3`).
+- Corpus capped at **20k papers** (`config.PUBMED_DEFAULT_MAX`) → ~62k chunks / ~2.4 GB at the cap.
+- Embeddings: **BioLORD-2023-C** (768-dim). Reranker: `cross-encoder/ms-marco-MiniLM-L-6-v2`.
+- **Deployment:** the index is stored in the HF **dataset** repo (`KevinIsCoding/candle-fire-data`)
+  and **downloaded to the Space at cold start** (`app.py:_ensure_data`). The 1.2 GB download
+  amplified the recent 503 outage.
+
+## 2. Key Finding — this is a *deployment* problem, not a *scale* problem
+
+- **31k–62k vectors is trivially small** for any vector store (they scale to millions/billions).
+  Chroma is not being outgrown in capability.
+- **The 1.2 GB is mostly text, not vectors.** Vectors alone are ~100 MB (31k × 768 × 4 bytes).
+  The rest is full document text + a full-text index — because the retriever relies on Chroma's
+  substring search (`where_document $contains`) to ground drug codes / gene IDs. That text is
+  **load-bearing** for citation grounding; it can't just be dropped to shrink the index.
+- Therefore the pain is **where the DB lives** (bundled + downloaded per cold start), not its size.
+
+## 3. Goals / Non-Goals
+
+**Goals**
+- Remove the 1.2 GB cold-start download that couples the DB to the Space lifecycle.
+- Preserve the **hybrid retrieval** (semantic ANN + exact substring + metadata filter) — load-bearing for grounding.
+- Keep a low-risk path that also scales if the corpus grows.
+
+**Non-Goals**
+- Shrinking the index by dropping document text (breaks grounding).
+- Rewriting the ranking pipeline (RRF, cross-encoder, citation/recency boost) — it is backend-independent and stays as-is.
+
+## 4. Current Architecture (what any migration must preserve)
+
+**Retriever (`rag/retriever.py`)** splits cleanly into two layers:
+
+- **Candidate fetch — backend-specific (the only code that must change on migration):**
+  - `search()` / `search_by_entities()`: semantic ANN via `collection.query(query_texts=...)`.
+  - `search_by_keyword()`, `is_grounded_in_corpus()`, `is_grounded_in_abstract()`: **exact substring**
+    via `where_document={"$contains": ...}`, expanded over `_term_variants()` (SPG302 / SPG 302 / SPG-302).
+  - `get_paper()`, `paper_texts_for_pmids()`: metadata fetch via `collection.get(where=...)`.
+- **Ranking — backend-independent (REUSE UNCHANGED):**
+  - `_parse_raw()` normalizes any backend response to the result-dict shape.
+  - `rrf_merge()`, `cross_encoder_rerank()`, `apply_citation_boost()` operate purely on result dicts.
+
+**Indexer (`rag/indexer.py`)**: `build_collection()` chunks papers (≤6 chunks/paper), embeds via a
+`SentenceTransformerEmbeddingFunction` **baked into the Chroma collection**, IDs `{pmid}_s{i}`, scalarized metadata.
+
+> **Key design lever:** only the candidate-fetch functions touch the backend. If a new backend keeps
+> the `_parse_raw` output shape, the entire ranking/fusion/boost pipeline is reused verbatim. This
+> bounds migration scope to ~6 functions + the indexer writer.
+
+## 5. Options Considered
+
+| Option | Fit | Verdict |
+|---|---|---|
+| **A. Cheap deploy fix** (persistent storage on the Space, and/or Chroma client/server on a small host) | Keeps all Chroma features; ~no code change | **Recommended first (Phase 0)** |
+| **B. Postgres + pgvector** (AWS Aurora/RDS, or Neon/Supabase) | One store for vectors (`<=>`) + full-text (ILIKE/tsvector) + metadata (SQL); preserves the hybrid | **Recommended migration target (Phase 1)** |
+| **C. AWS OpenSearch** (kNN + BM25 + filters) | Fits the hybrid; AWS-native | Viable but heavier ops/tuning; overkill at 31k vectors |
+| **D. Vector-only managed** (Pinecone / Qdrant Cloud) | No native substring/full-text | **Rejected** — would require a separate keyword store to keep grounding |
+| **E. Amazon Bedrock Knowledge Bases** | Managed end-to-end RAG | **Rejected** — replaces the custom hybrid + grounding + citation-weighting (the product's differentiators) |
+
+## 6. Recommendation (phased)
+
+- **Phase 0 — now (do regardless of the migration decision):** Option A. Enable **persistent storage**
+  on the Space so `_ensure_data` stops re-downloading 1.2 GB each restart; optionally run **Chroma in
+  client/server mode** so the Space queries it over HTTP. Removes the cold-start pain with little/no
+  rewrite. (Pairs with the separate COE action item to make data refresh not require a factory-reboot.)
+- **Phase 1 — when a trigger hits:** Option B (pgvector). **Triggers:** corpus scaled toward 100k+
+  chunks / continuous ingestion; OR a need to decouple the DB from the Space (multi-instance, independent
+  updates); OR wanting to stop shipping a data blob entirely. Abstract the backend first (Section 7) so
+  the swap is localized and reversible.
+
+## 7. Phase 1 Detailed Design (pgvector)
+
+**Backend abstraction (do this even before migrating):** introduce `rag/backends/` with a thin interface —
+`semantic_candidates(query, n)`, `substring_candidates(term_variants, n)`, `fetch_by_pmids(pmids)`,
+`abstract_contains(variant)`, `count()` — implemented by `chroma_backend` (wraps today's code) and
+`pg_backend`. `retriever.py` calls the interface; the ranking layer is untouched. `indexer.py` gains a pg writer.
+
+**Schema (single table):**
+```sql
+CREATE TABLE chunks (
+  chunk_id       text PRIMARY KEY,
+  pmid           text,
+  section        text,
+  chunk_index    int,
+  title          text,
+  year           int,
+  doi            text,
+  citation_count int,
+  entity_names   text,
+  has_full_text  boolean,
+  document       text,
+  embedding      vector(768),
+  doc_tsv        tsvector
+);
+-- indexes: HNSW on embedding (vector_cosine_ops); GIN on doc_tsv; btree on pmid
+```
+
+**Query mapping (preserve current semantics):**
+- `collection.query(query_texts)` → embed query with BioLORD explicitly, then
+  `SELECT ... ORDER BY embedding <=> $qvec LIMIT n`.
+- `where_document $contains` → `WHERE document ILIKE '%'||$variant||'%'` (exact substring — matches
+  Chroma `$contains` behavior for drug codes; **use ILIKE, not tsvector**, to keep grounding parity).
+- `where={pmid $in}` / `chunk_index` / `get` → plain SQL `WHERE`.
+- Query embeddings use the same `config` model; the ranking pipeline consumes `_parse_raw`-shaped dicts unchanged.
+
+**Config:** DB URL via env var; embedding model unchanged. Graph/agent layers untouched.
+
+## 8. Rollout / Migration
+
+1. Stand up Postgres + pgvector (RDS/Aurora/Neon); `CREATE EXTENSION vector`.
+2. **Backfill** with a one-off script that reuses the existing `rag/indexer.py` chunker on `papers.jsonl`,
+   embeds, and writes rows (idempotent on `chunk_id`).
+3. **Dual-read validation** (Section 11) — compare Chroma vs pgvector on a fixed query set.
+4. Flip the Space to the pg backend via an **env flag**, keeping the Chroma path behind the flag for rollback.
+5. Once stable, drop the 1.2 GB blob from the dataset repo.
+
+## 9. Risks & Mitigations
+
+- **Substring-grounding parity:** ILIKE must reproduce `$contains` over `_term_variants()`. Port the variant
+  expansion; add tests comparing grounding booleans and keyword hits against Chroma.
+- **Per-request query fan-out:** `search_by_entities()` issues up to 12 queries/request — batch into a single
+  SQL round-trip or reduce `RETRIEVAL_ENTITY_QUERY_CAP`; HNSW keeps ANN fast.
+- **DB cost/ops:** smallest managed tier is ample at this scale.
+- **Embedding drift:** pin the embedding model; re-embed if it changes.
+
+## 10. Open Questions / Decisions for Reviewer
+
+1. **Growth:** staying ~10–20k papers, or scaling toward full PubMed / continuous ingestion? (Decides whether Phase 1 is needed at all.)
+2. **Host:** AWS specifically (Aurora/RDS), or is managed-elsewhere (Neon/Supabase) acceptable — often cheaper/simpler at this scale?
+3. **Driver:** is decoupling the DB from the Space a goal in itself, or is cold-start relief (Phase 0) enough for now?
+4. **Ops/budget:** appetite for an always-on DB instance vs the current zero-infra file model.
+
+## 11. Verification
+
+- **Parity harness:** fixed query set; assert the final top-15 cited PMIDs and the grounding booleans
+  (`is_grounded_in_corpus`, `is_grounded_in_abstract`) match Chroma within tolerance.
+- **Latency:** p50/p95 per query under both backends.
+- **Cold-start:** Space boot time before/after — Phase 0 target: no 1.2 GB re-download; Phase 1 target: no data blob shipped at all.

landscape.py

@@ -0,0 +1,204 @@
+"""Rendering helpers for the Experimental ALS Therapy Landscape tab (v2: multi-label).
+
+Loads data/landscape/landscape.json and turns it into a Plotly sunburst (mechanism class →
+primary therapy) plus a detail panel that lists every mechanism a therapy acts through (with role,
+confidence, and justification) and a trials table with status badges + NCT links. Therapies whose
+mechanism could not be established are shown honestly under "Insufficient evidence".
+"""
+from __future__ import annotations
+
+import json
+import html
+
+from config import LANDSCAPE_PATH
+
+_INSUFFICIENT = "Insufficient evidence"
+
+_CLASS_COLORS = {
+    "TDP-43 proteinopathy": "#6C5CE7", "SOD1": "#0984E3", "C9orf72": "#00B894", "FUS": "#00CEC9",
+    "Neuroinflammation": "#E17055", "Oxidative stress": "#D63031", "Mitochondrial dysfunction": "#E84393",
+    "Glutamate excitotoxicity": "#FDCB6E", "Proteostasis / autophagy": "#A29BFE",
+    "RNA metabolism": "#74B9FF", "Neurotrophic / regenerative": "#55EFC4",
+    "Symptomatic / Other": "#B2BEC3", _INSUFFICIENT: "#DFE6E9",
+}
+_DEFAULT_COLOR = "#B2BEC3"
+_STATUS_BADGE = {
+    "recruiting": ("#00B894", "Recruiting"), "active": ("#0984E3", "Active"),
+    "completed": ("#636E72", "Completed"), "terminated": ("#D63031", "Terminated"),
+    "other": ("#B2BEC3", "Unknown"),
+}
+
+
+PHASE_OPTIONS = ["Phase 1", "Phase 2", "Phase 3", "Phase 4", "Not applicable"]
+
+
+def _phase_buckets(phase: str) -> set:
+    """Map a ClinicalTrials.gov phase string (incl. combos like 'PHASE1, PHASE2') to UI buckets."""
+    p = (phase or "").upper()
+    b = set()
+    if "PHASE1" in p:  # also catches EARLY_PHASE1
+        b.add("Phase 1")
+    if "PHASE2" in p:
+        b.add("Phase 2")
+    if "PHASE3" in p:
+        b.add("Phase 3")
+    if "PHASE4" in p:
+        b.add("Phase 4")
+    return b or {"Not applicable"}  # NA / Expanded Access / blank
+
+
+def _therapy_matches_phases(t: dict, selected: set | None) -> bool:
+    """A therapy is shown if any of its trials falls in a selected phase. None/all = whole picture."""
+    if not selected or len(selected) >= len(PHASE_OPTIONS):
+        return True
+    return any(_phase_buckets(tr.get("phase", "")) & selected for tr in t["trials"])
+
+
+def load_landscape() -> dict | None:
+    if not LANDSCAPE_PATH.exists():
+        return None
+    try:
+        return json.loads(LANDSCAPE_PATH.read_text())
+    except Exception:
+        return None
+
+
+def class_names(landscape: dict | None) -> list[str]:
+    if not landscape:
+        return []
+    names = [c["name"] for c in landscape["classifications"]]
+    if landscape.get("unclassified"):
+        names.append(_INSUFFICIENT)
+    return names
+
+
+def _therapies_of(landscape: dict, class_name: str) -> list[dict]:
+    if class_name == _INSUFFICIENT:
+        return landscape.get("unclassified", [])
+    for c in landscape["classifications"]:
+        if c["name"] == class_name:
+            return c["therapies"]
+    return []
+
+
+def therapy_labels(landscape: dict | None, class_name: str, phases: list | None = None) -> list[str]:
+    """Dropdown labels, e.g. 'CNM-Au8 [contributing] — 5 trials, 3 recruiting'."""
+    sel = set(phases) if phases else None
+    labels = []
+    for t in _therapies_of(landscape or {}, class_name):
+        if not _therapy_matches_phases(t, sel):
+            continue
+        c = t["trial_counts"]
+        rec = f", {c['recruiting']} recruiting" if c["recruiting"] else ""
+        role = t.get("role_here")
+        tag = f" [{role}]" if (role and class_name != _INSUFFICIENT) else ""
+        labels.append(f"{t['name']}{tag} — {c['total']} trial{'s' if c['total'] != 1 else ''}{rec}")
+    return labels
+
+
+def _therapy_by_label(landscape: dict, class_name: str, label: str) -> dict | None:
+    name = label.split(" — ")[0].split(" [")[0] if label else ""
+    for t in _therapies_of(landscape, class_name):
+        if t["name"] == name:
+            return t
+    return None
+
+
+def build_sunburst(landscape: dict | None, phases: list | None = None):
+    """Two-ring sunburst: inner = mechanism class, outer = its PRIMARY therapies (sized by trials).
+    Contributing memberships live in the detail panel; abstentions get an 'Insufficient evidence' wedge.
+    """
+    import plotly.graph_objects as go
+
+    sel = set(phases) if phases else None
+
+    def _keep(t):
+        return _therapy_matches_phases(t, sel)
+
+    ids, labels, parents, values, colors, hover = [], [], [], [], [], []
+
+    def add_class(name, therapies, color, desc=""):
+        if not therapies:
+            return
+        cid = f"cls::{name}"
+        total = sum(t["trial_counts"]["total"] for t in therapies)
+        ids.append(cid); labels.append(name); parents.append(""); values.append(total); colors.append(color)
+        hover.append(f"<b>{name}</b><br>{len(therapies)} therapies · {total} trials<br>{desc}")
+        for t in therapies:
+            cnt = t["trial_counts"]
+            ids.append(f"th::{name}::{t['name']}"); labels.append(t["name"]); parents.append(cid)
+            values.append(cnt["total"]); colors.append(color)
+            hover.append(f"<b>{t['name']}</b> ({t['modality']})<br>Target: {t['target']}<br>"
+                         f"{cnt['total']} trials · {cnt['recruiting']} recruiting · {cnt['completed']} completed")
+
+    if landscape:
+        for c in landscape["classifications"]:
+            primaries = [t for t in c["therapies"] if t.get("role_here") == "primary" and _keep(t)]
+            add_class(c["name"], primaries, _CLASS_COLORS.get(c["name"], _DEFAULT_COLOR), c.get("description", ""))
+        un = [t for t in landscape.get("unclassified", []) if _keep(t)]
+        if un:
+            add_class(_INSUFFICIENT, un, _CLASS_COLORS[_INSUFFICIENT],
+                      "Mechanism not established from available evidence")
+
+    fig = go.Figure(go.Sunburst(
+        ids=ids, labels=labels, parents=parents, values=values, branchvalues="total",
+        marker=dict(colors=colors), hovertext=hover, hoverinfo="text",
+        insidetextorientation="radial", maxdepth=2,
+    ))
+    fig.update_layout(margin=dict(t=10, l=0, r=0, b=0), height=520, paper_bgcolor="rgba(0,0,0,0)")
+    return fig
+
+
+def therapy_detail_md(landscape: dict | None, class_name: str, therapy_label: str) -> str:
+    if not landscape or not class_name:
+        return "*Select a mechanism class and therapy to see its mechanisms and trials.*"
+    t = _therapy_by_label(landscape, class_name, therapy_label)
+    if not t:
+        return f"*{class_name}* — select a therapy above."
+    header = f"### {t['name']}\n<sub>{t['modality']} · Target: {t['target']}</sub>"
+    if t.get("aliases"):
+        header += f"\n<sub>Also: {', '.join(t['aliases'][:6])}</sub>"
+    mechs = t.get("mechanisms") or []
+    if not mechs:
+        return header + "\n\n**Mechanism of action:** _Not established from the available evidence._"
+    lines = [header, "\n**Mechanism(s) of action:**"]
+    for m in mechs:
+        conf = int(round(m["confidence"] * 100))
+        ev = f" — {m['evidence']}" if m.get("evidence") else ""
+        lines.append(f"- **{m['class']}** · _{m['role']}, {conf}% confidence_{ev}")
+    return "\n".join(lines)
+
+
+def trials_table_html(landscape: dict | None, class_name: str, therapy_label: str) -> str:
+    if not landscape or not class_name:
+        return ""
+    t = _therapy_by_label(landscape, class_name, therapy_label)
+    if not t:
+        return ""
+    rows = []
+    for tr in t["trials"]:
+        color, label = _STATUS_BADGE.get(tr["status_group"], _STATUS_BADGE["other"])
+        badge = (f'<span style="background:{color};color:#fff;border-radius:10px;'
+                 f'padding:1px 8px;font-size:0.72rem;white-space:nowrap;">{label}</span>')
+        phase = html.escape((tr.get("phase") or "—").replace("PHASE", "Ph"))
+        title = html.escape(tr.get("title", "")[:110])
+        nct = html.escape(tr.get("nct_id", ""))
+        url = html.escape(tr.get("url", ""))
+        sponsor = html.escape((tr.get("sponsor") or "")[:40])
+        rows.append(
+            f'<tr><td style="padding:4px 8px;">{badge}</td>'
+            f'<td style="padding:4px 8px;color:#666;">{phase}</td>'
+            f'<td style="padding:4px 8px;"><a href="{url}" target="_blank" rel="noopener">{nct}</a> — {title}</td>'
+            f'<td style="padding:4px 8px;color:#888;font-size:0.8rem;">{sponsor}</td></tr>'
+        )
+    counts = t["trial_counts"]
+    caption = (f'<div style="font-size:0.85rem;color:#666;margin:6px 0;">{counts["total"]} trials — '
+               f'<b style="color:#00B894;">{counts["recruiting"]} recruiting</b>, '
+               f'{counts["active"]} active, {counts["completed"]} completed, {counts["terminated"]} terminated</div>')
+    return caption + (
+        '<table style="width:100%;border-collapse:collapse;font-size:0.88rem;">'
+        '<thead><tr style="text-align:left;border-bottom:1px solid #ddd;color:#888;">'
+        '<th style="padding:4px 8px;">Status</th><th style="padding:4px 8px;">Phase</th>'
+        '<th style="padding:4px 8px;">Trial</th><th style="padding:4px 8px;">Sponsor</th></tr></thead>'
+        f'<tbody>{"".join(rows)}</tbody></table>'
+    )

prompts.py

@@ -24,6 +24,51 @@ Be precise. Only extract entities explicitly mentioned. Confidence reflects how
 the entity is identified in the text (1.0 = unambiguous, 0.5 = inferred, 0.3 = uncertain).
 """
 
+LANDSCAPE_SYSTEM = """\
+You are an ALS-pharmacology expert classifying experimental therapies by mechanism of action.
+For each therapy you are given EVIDENCE (its trial summaries + retrieved paper abstracts).
+Use the evidence together with your established knowledge of ALS therapeutics to classify each
+therapy with the classify_therapy tool — call it exactly once per therapy, echoing therapy_key.
+
+MULTI-LABEL: a therapy may act through several mechanisms. Return EVERY mechanism class that is
+well established for THIS therapy, each with a role ("primary" vs "contributing"), a confidence,
+and a one-line `evidence_quote` justification (quote the evidence when it supports you; otherwise
+state the established mechanism concisely). The highest-confidence entry is the primary mechanism.
+
+Mechanism classes:
+- TDP-43 proteinopathy, SOD1, C9orf72, FUS, Neuroinflammation, Oxidative stress,
+  Mitochondrial dysfunction, Glutamate excitotoxicity, Proteostasis / autophagy, RNA metabolism,
+  Neurotrophic / regenerative, Symptomatic / Other.
+
+CRITICAL — misleading information is worse than no information:
+- Only assert a mechanism you are genuinely confident is established for THIS specific therapy.
+  Set confidence honestly (1.0 = textbook-established; 0.6 = reasonable; below that, omit it).
+- Classify by how THIS therapy acts — NEVER infer a mechanism from co-mentioned entities or from
+  other drugs in a combination trial. (Example: an antioxidant tested in a trial that also studies
+  neuroinflammation is NOT itself a neuroinflammation therapy.)
+- POPULATION IS NOT MECHANISM. Assign a genetic class (SOD1, C9orf72, FUS, TDP-43 proteinopathy)
+  ONLY when the therapy directly targets that gene/protein/RNA (e.g., an ASO or gene therapy that
+  lowers it). A drug merely tested in patients with that mutation, or a general neuroprotectant, does
+  NOT get the genetic class (e.g., arimoclomol is Proteostasis, not SOD1, even when trialed in SOD1-ALS).
+- Prefer FEWER, higher-confidence mechanisms. Emit a "contributing" mechanism only when it is
+  well-established for this drug, not merely plausible — when in doubt, leave it out.
+- If you do not know the therapy and the evidence does not establish a mechanism, return an EMPTY
+  mechanisms array. Abstaining is correct and expected for obscure or repurposed drugs you cannot
+  place confidently — never guess to fill the field.
+- Use "Symptomatic / Other" only for therapies that genuinely act symptomatically (muscle function,
+  cramps, respiration), not as a dumping ground for uncertainty.
+
+Also return canonical_name (merge synonyms/codes), modality, and the primary molecular target
+("Unknown" if not determinable).
+
+Examples of correct classification:
+- Riluzole → [{"class":"Glutamate excitotoxicity","role":"primary"}] (reduces glutamate excitotoxicity).
+- CNM-Au8 → [{"class":"Mitochondrial dysfunction","role":"primary"},{"class":"Oxidative stress","role":"contributing"}]
+  — a gold nanocrystal catalyst that improves neuronal energy metabolism and reduces oxidative stress;
+  it is NOT a neuroinflammation therapy even if its trials mention neuroinflammation.
+- An obscure development-code drug you cannot place confidently → mechanisms: [] (abstain).
+"""
+
 SYNTHESIS_SYSTEM = """\
 You are a clinical research synthesis expert specializing in ALS (amyotrophic lateral sclerosis).
 You help physicians understand the research evidence behind ALS biology, drug targets, and clinical trials.

pyproject.toml

@@ -16,6 +16,7 @@ dependencies = [
     "sentence-transformers>=3.0.0",
     "openai>=2.44.0",
     "rapidfuzz>=3.14.5",
+    "plotly==6.9.0",
 ]
 
 [project.optional-dependencies]

requirements.txt

@@ -8,3 +8,5 @@ python-dotenv>=1.2.2
 rich>=13.0.0
 sentence-transformers>=3.0.0
 openai>=2.44.0
+rapidfuzz>=3.14.5
+plotly==6.9.0

scripts/build_landscape.py

@@ -0,0 +1,393 @@
+#!/usr/bin/env python3
+"""
+Build the Experimental ALS Therapy Landscape (offline) — v2: multi-label, grounded, abstaining.
+
+Derives experimental therapies from clinical-trial interventions, then for each therapy:
+  1. retrieves genuine mechanism-of-action evidence (RAG over the paper corpus by drug name +
+     the therapy's own trial summaries) — NOT the noisy aggregated trial target_entities;
+  2. classifies MULTI-LABEL with Claude (Opus 4.8) via the Batch API — every mechanism must be
+     backed by a verbatim evidence quote, and the model abstains (empty list) when unsure;
+  3. cross-checks each asserted mechanism with a model-independent BioLORD cosine between the
+     therapy's evidence and the mechanism-class description (drops weakly-supported labels);
+  4. keeps only mechanisms above the confidence threshold τ; a therapy with none is "unclassified"
+     (shown honestly as "Mechanism not established"), never guessed.
+
+Writes data/landscape/landscape.json (committed to git; the app loads it at startup).
+
+Usage:
+    uv run python scripts/build_landscape.py
+    uv run python scripts/build_landscape.py --limit 30   # cheap validation on top 30 therapies
+    uv run python scripts/build_landscape.py --reset      # ignore any in-flight batch state
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+import time
+from datetime import datetime, timezone
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+import anthropic
+import numpy as np
+from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
+from anthropic.types.messages.batch_create_params import Request
+from rich.console import Console
+
+from config import (
+    LANDSCAPE_BATCH_STATE_PATH,
+    LANDSCAPE_EVIDENCE_ABSTRACTS,
+    LANDSCAPE_MIN_CONFIDENCE,
+    LANDSCAPE_MODEL,
+    LANDSCAPE_PATH,
+    LANDSCAPE_XCHECK_MIN_COSINE,
+    THERAPY_CLASSES_PATH,
+    TRIALS_PATH,
+)
+from prompts import LANDSCAPE_SYSTEM
+from tools import LANDSCAPE_TOOLS
+
+console = Console()
+
+_THERAPEUTIC_TYPES = {"DRUG", "BIOLOGICAL", "DIETARY_SUPPLEMENT", "GENETIC", "COMBINATION_PRODUCT"}
+_EXCLUDE_RE = re.compile(
+    r"(placebo|sham|matching|best supportive care|standard of care|blood sample|"
+    r"saline|vehicle|diagnostic|questionnaire|no intervention|usual care|dextrose)",
+    re.IGNORECASE,
+)
+_MODALITY_PREFIX_RE = re.compile(
+    r"^(drug|biological|device|other|dietary supplement|genetic|procedure|"
+    r"combination product|radiation|behavioral|diagnostic test)\s*:\s*",
+    re.IGNORECASE,
+)
+_STATUS_GROUP = {
+    "RECRUITING": "recruiting", "NOT_YET_RECRUITING": "recruiting", "ENROLLING_BY_INVITATION": "recruiting",
+    "ACTIVE_NOT_RECRUITING": "active",
+    "COMPLETED": "completed", "APPROVED_FOR_MARKETING": "completed", "AVAILABLE": "completed",
+    "TERMINATED": "terminated", "WITHDRAWN": "terminated", "SUSPENDED": "terminated",
+    "NO_LONGER_AVAILABLE": "terminated", "TEMPORARILY_NOT_AVAILABLE": "terminated",
+}
+_GROUP_ORDER = {"recruiting": 0, "active": 1, "completed": 2, "terminated": 3, "other": 4}
+_THERAPIES_PER_REQUEST = 10   # richer evidence per therapy → smaller batches
+_POLL_INTERVAL_S = 30
+
+
+def _norm_name(name: str) -> str:
+    name = _MODALITY_PREFIX_RE.sub("", name or "").strip()
+    return re.sub(r"\s+", " ", name)
+
+
+def _status_group(status: str) -> str:
+    return _STATUS_GROUP.get((status or "").upper(), "other")
+
+
+def _load_trials(path: Path) -> list[dict]:
+    return [json.loads(line) for line in open(path, encoding="utf-8") if line.strip()]
+
+
+def group_therapies(trials: list[dict]) -> dict[str, dict]:
+    """Group trials by normalized therapeutic intervention name."""
+    groups: dict[str, dict] = {}
+    for t in trials:
+        trial_meta = {
+            "nct_id": t.get("nct_id", ""), "title": t.get("title", ""), "phase": t.get("phase", ""),
+            "status": t.get("status", ""), "status_group": _status_group(t.get("status", "")),
+            "start_date": t.get("start_date", ""), "sponsor": t.get("sponsor", ""),
+            "url": t.get("url", "") or f"https://clinicaltrials.gov/study/{t.get('nct_id','')}",
+            "summary": t.get("summary", ""),
+        }
+        for iv in t.get("interventions", []):
+            if iv.get("type") not in _THERAPEUTIC_TYPES:
+                continue
+            name = _norm_name(iv.get("name", ""))
+            if not name or _EXCLUDE_RE.search(name):
+                continue
+            key = name.lower()
+            g = groups.setdefault(key, {"display": name, "raw_names": set(), "trials": {}})
+            g["raw_names"].add(iv.get("name", ""))
+            g["trials"][trial_meta["nct_id"]] = trial_meta
+    for g in groups.values():
+        g["trials"] = list(g["trials"].values())
+    return groups
+
+
+# ── evidence retrieval + cross-check embeddings ──────────────────────────────
+
+def _load_collection():
+    try:
+        from config import CHROMA_COLLECTION, CHROMA_DIR
+        from rag.indexer import load_collection
+        return load_collection(CHROMA_DIR, CHROMA_COLLECTION)
+    except Exception as e:
+        console.print(f"[yellow]No chroma collection ({e}); evidence = trial summaries only, no cross-check[/yellow]")
+        return None
+
+
+def _embed(texts: list[str]) -> np.ndarray:
+    from rag.indexer import _EMBED_FN
+    vecs = np.asarray(_EMBED_FN(texts), dtype=np.float32)
+    norms = np.linalg.norm(vecs, axis=1, keepdims=True)
+    return vecs / np.clip(norms, 1e-9, None)
+
+
+def _retrieve_evidence(collection, g: dict) -> str:
+    """MoA evidence for a therapy: top abstracts (by drug name) + its own trial summaries."""
+    parts: list[str] = []
+    summaries = [t["summary"] for t in g["trials"][:3] if t.get("summary")]
+    parts.extend(s[:400] for s in summaries)
+    if collection is not None:
+        try:
+            from rag.retriever import search
+            for r in search(collection, g["display"], n_results=LANDSCAPE_EVIDENCE_ABSTRACTS):
+                doc = (r.get("document") or "")[:500]
+                if doc:
+                    parts.append(doc)
+        except Exception:
+            pass
+    return "\n".join(parts)[:3000]
+
+
+# ── LLM batch ────────────────────────────────────────────────────────────────
+
+def _therapy_context(key: str, g: dict) -> str:
+    lines = [
+        f"--- THERAPY_KEY:{key} ---",
+        f"Intervention name(s): {', '.join(sorted(g['raw_names']))[:160]}",
+        "EVIDENCE (paper abstracts about this therapy + its trial summaries):",
+        g.get("evidence", "(no evidence retrieved)"),
+    ]
+    return "\n".join(lines)
+
+
+def _format_batch(batch: list[tuple[str, dict]]) -> str:
+    parts = [
+        f"Classify each of the following {len(batch)} ALS experimental therapies from its EVIDENCE. "
+        "Call classify_therapy once per therapy (echo therapy_key). Emit only mechanisms you can quote "
+        "from the evidence; return an empty mechanisms list if the evidence establishes none.\n"
+    ]
+    parts.extend(_therapy_context(key, g) for key, g in batch)
+    return "\n\n".join(parts)
+
+
+def _build_params(batch: list[tuple[str, dict]]) -> MessageCreateParamsNonStreaming:
+    return MessageCreateParamsNonStreaming(
+        model=LANDSCAPE_MODEL,
+        max_tokens=8192,
+        system=LANDSCAPE_SYSTEM,
+        tools=list(LANDSCAPE_TOOLS),
+        tool_choice={"type": "any"},
+        messages=[{"role": "user", "content": _format_batch(batch)}],
+    )
+
+
+def _run_batch(client, custom_id_to_batch: dict, state_path: Path, reset: bool) -> dict[str, dict]:
+    batch = None
+    if not reset and state_path.exists():
+        try:
+            existing = client.messages.batches.retrieve(json.loads(state_path.read_text())["batch_id"])
+            if existing.processing_status in {"in_progress", "validating", "finalizing", "ended"}:
+                console.print(f"[dim]Resuming batch {existing.id}[/dim]"); batch = existing
+        except anthropic.NotFoundError:
+            pass
+    if batch is None:
+        requests = [Request(custom_id=cid, params=_build_params(b)) for cid, b in custom_id_to_batch.items()]
+        batch = client.messages.batches.create(requests=requests)
+        state_path.parent.mkdir(parents=True, exist_ok=True)
+        state_path.write_text(json.dumps({"batch_id": batch.id}))
+        console.print(f"[cyan]Submitted batch {batch.id} ({len(requests)} requests, model={LANDSCAPE_MODEL})[/cyan]")
+    with console.status("Classifying therapies (batch)…"):
+        while batch.processing_status != "ended":
+            if batch.processing_status in {"canceling", "canceled", "expired"}:
+                console.print(f"[red]Batch ended early: {batch.processing_status}[/red]"); break
+            time.sleep(_POLL_INTERVAL_S)
+            batch = client.messages.batches.retrieve(batch.id)
+    records: dict[str, dict] = {}
+    for res in client.messages.batches.results(batch.id):
+        if res.result.type != "succeeded":
+            console.print(f"[yellow]Request {res.custom_id} {res.result.type}[/yellow]"); continue
+        for block in res.result.message.content:
+            if block.type == "tool_use" and block.name == "classify_therapy":
+                key = str(block.input.get("therapy_key", "")).strip()
+                if key:
+                    records[key] = block.input
+    state_path.unlink(missing_ok=True)
+    return records
+
+
+# ── filtering: confidence threshold + BioLORD cross-check ────────────────────
+
+def _filter_all(records: dict, class_emb: dict, valid: set) -> dict[str, list[dict]]:
+    """Per-mechanism filter: confidence ≥ τ AND justification↔class BioLORD cosine ≥ threshold.
+
+    The cross-check embeds each mechanism's own justification (evidence_quote) and compares it to
+    the claimed class description — catching internally-inconsistent labels (justification is about
+    a different mechanism than the class claimed). Batched so all embeddings are one pass.
+    """
+    cand = []  # (key, class, role, conf, quote)
+    for key, rec in records.items():
+        for m in rec.get("mechanisms", []) or []:
+            cls = m.get("class")
+            if cls not in valid or float(m.get("confidence", 0.0)) < LANDSCAPE_MIN_CONFIDENCE:
+                continue
+            cand.append((key, cls, m.get("role", "contributing"),
+                         float(m.get("confidence", 0.0)), (m.get("evidence_quote") or "")[:300]))
+
+    cosines = [None] * len(cand)
+    if class_emb and cand:
+        embs = _embed([q or cls for (_, cls, _, _, q) in cand])
+        for i, (_, cls, _, _, _) in enumerate(cand):
+            if cls in class_emb:
+                cosines[i] = float(np.dot(embs[i], class_emb[cls]))
+
+    out: dict[str, list[dict]] = {}
+    for i, (key, cls, role, conf, quote) in enumerate(cand):
+        cos = cosines[i]
+        if cos is not None and cos < LANDSCAPE_XCHECK_MIN_COSINE:
+            continue  # justification doesn't semantically match the claimed class
+        out.setdefault(key, []).append({
+            "class": cls, "role": role, "confidence": round(conf, 2),
+            "xcheck_cosine": round(cos, 3) if cos is not None else None,
+            "evidence": quote,
+        })
+    for lst in out.values():
+        lst.sort(key=lambda m: (m["role"] != "primary", -m["confidence"]))
+    return out
+
+
+# ── assembly ─────────────────────────────────────────────────────────────────
+
+def _trial_block(m: dict) -> tuple[list[dict], dict]:
+    trials = sorted(m["_trials"].values(), key=lambda t: (_GROUP_ORDER.get(t["status_group"], 9), t.get("start_date", "")))
+    for t in trials:
+        t.pop("summary", None)
+    counts = {g: 0 for g in _GROUP_ORDER}
+    for t in trials:
+        counts[t["status_group"]] += 1
+    counts["total"] = len(trials)
+    return trials, counts
+
+
+def build_landscape(limit=None, reset=False) -> dict:
+    taxonomy = json.loads(THERAPY_CLASSES_PATH.read_text())["classes"]
+    valid = {c["name"] for c in taxonomy}
+    trials = _load_trials(TRIALS_PATH)
+    groups = group_therapies(trials)
+    console.print(f"[cyan]{len(trials)} trials → {len(groups)} candidate therapies[/cyan]")
+
+    items = sorted(groups.items(), key=lambda kv: -len(kv[1]["trials"]))
+    if limit:
+        items = items[:limit]
+        console.print(f"[dim]--limit {limit}: top {len(items)} therapies[/dim]")
+
+    # evidence retrieval + cross-check embeddings (local, no API cost)
+    collection = _load_collection()
+    class_emb = {}
+    if collection is not None:
+        console.print("[dim]Embedding class descriptions + retrieving evidence (BioLORD)…[/dim]")
+        cls_vecs = _embed([f"{c['name']}: {c['description']}" for c in taxonomy])
+        class_emb = {c["name"]: cls_vecs[i] for i, c in enumerate(taxonomy)}
+    for _, g in items:
+        g["evidence"] = _retrieve_evidence(collection, g)
+
+    # classify
+    batches = [items[i : i + _THERAPIES_PER_REQUEST] for i in range(0, len(items), _THERAPIES_PER_REQUEST)]
+    client = anthropic.Anthropic()
+    records = _run_batch(client, {f"batch-{i}": b for i, b in enumerate(batches)}, LANDSCAPE_BATCH_STATE_PATH, reset)
+    console.print(f"[green]Classified {len(records)}/{len(items)} therapies[/green]")
+
+    # confidence threshold + justification↔class cross-check (batched)
+    filtered = _filter_all(records, class_emb, valid)
+
+    # merge by canonical name, attach trials
+    merged: dict[str, dict] = {}
+    for key, g in items:
+        rec = records.get(key)
+        if not rec:
+            continue
+        mechs = filtered.get(key, [])
+        canonical = (rec.get("canonical_name") or g["display"]).strip()
+        m = merged.setdefault(canonical.lower(), {
+            "name": canonical, "modality": rec.get("modality", "Other"),
+            "target": rec.get("target", "Unknown"), "aliases": set(),
+            "_mechs": {}, "_trials": {},
+        })
+        m["aliases"].update(a for a in rec.get("aliases", []) if a)
+        m["aliases"].update(g["raw_names"])
+        for me in mechs:  # highest-confidence per class wins
+            cur = m["_mechs"].get(me["class"])
+            if cur is None or me["confidence"] > cur["confidence"]:
+                m["_mechs"][me["class"]] = me
+        for t in g["trials"]:
+            m["_trials"][t["nct_id"]] = t
+
+    # finalize therapies
+    therapies, abstained = [], []
+    for m in merged.values():
+        trials_, counts = _trial_block(m)
+        mechs = sorted(m["_mechs"].values(), key=lambda x: (x["role"] != "primary", -x["confidence"]))
+        base = {
+            "name": m["name"], "modality": m["modality"], "target": m["target"],
+            "aliases": sorted(a for a in m["aliases"] if a and a.lower() != m["name"].lower())[:8],
+            "mechanisms": mechs, "trials": trials_, "trial_counts": counts,
+        }
+        (therapies if mechs else abstained).append(base)
+
+    # group into taxonomy buckets (a therapy appears under EACH of its mechanism classes)
+    by_class: dict[str, list[dict]] = {}
+    for th in therapies:
+        for me in th["mechanisms"]:
+            entry = {**th, "role_here": me["role"], "confidence_here": me["confidence"]}
+            by_class.setdefault(me["class"], []).append(entry)
+
+    classifications = []
+    for cls in taxonomy:
+        ths = by_class.get(cls["name"], [])
+        if not ths:
+            continue
+        ths.sort(key=lambda t: (t["role_here"] != "primary", -t["trial_counts"]["recruiting"], -t["trial_counts"]["total"]))
+        classifications.append({
+            "id": cls["id"], "name": cls["name"], "description": cls["description"],
+            "primary_count": sum(1 for t in ths if t["role_here"] == "primary"),
+            "therapy_count": len(ths),
+            "trial_count": sum(t["trial_counts"]["total"] for t in ths if t["role_here"] == "primary"),
+            "therapies": ths,
+        })
+
+    return {
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+        "model": LANDSCAPE_MODEL,
+        "thresholds": {"min_confidence": LANDSCAPE_MIN_CONFIDENCE, "min_xcheck_cosine": LANDSCAPE_XCHECK_MIN_COSINE},
+        "source": {"trials": len(trials), "therapies": len(therapies), "abstained": len(abstained)},
+        "classifications": classifications,
+        "unclassified": sorted(abstained, key=lambda t: -t["trial_counts"]["total"]),
+    }
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Build the ALS experimental therapy landscape (v2)")
+    parser.add_argument("--limit", type=int, default=None)
+    parser.add_argument("--reset", action="store_true")
+    args = parser.parse_args()
+    if not TRIALS_PATH.exists():
+        console.print(f"[red]Trials file not found: {TRIALS_PATH}[/red]"); sys.exit(1)
+
+    landscape = build_landscape(limit=args.limit, reset=args.reset)
+    LANDSCAPE_PATH.parent.mkdir(parents=True, exist_ok=True)
+    LANDSCAPE_PATH.write_text(json.dumps(landscape, indent=2))
+
+    s = landscape["source"]
+    console.print(f"\n[bold green]Done![/bold green] → {LANDSCAPE_PATH}")
+    console.print(f"  Therapies classified: [bold]{s['therapies']}[/bold] · abstained (insufficient evidence): [bold]{s['abstained']}[/bold]")
+    for c in landscape["classifications"]:
+        console.print(f"  [dim]{c['name']}: {c['primary_count']} primary / {c['therapy_count']} incl. contributing[/dim]")
+
+
+if __name__ == "__main__":
+    main()

scripts/eval_landscape.py

@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+"""
+Evaluate the mechanism classifier against the hand-labeled gold set.
+
+Reports multi-label precision / recall / F1 (micro), exact-set-match rate, and abstention rate,
+plus a per-therapy breakdown so misclassifications are visible. Physician's rule — misleading is
+worse than missing — so we optimize for PRECISION: a wrong label counts against us; an abstention
+(no label) does not count as a precision error, only against recall.
+
+Usage:
+    uv run python scripts/eval_landscape.py
+"""
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from rich.console import Console
+
+from config import LANDSCAPE_PATH, THERAPY_GOLD_PATH
+
+console = Console()
+
+
+def _predicted_index(landscape: dict) -> dict[str, set]:
+    """name/alias (lower) -> set of predicted mechanism classes. Abstained therapies map to set()."""
+    idx: dict[str, set] = {}
+    seen = {}
+    def add(th: dict, mech_classes: set):
+        for nm in [th["name"], *th.get("aliases", [])]:
+            key = nm.lower()
+            # a therapy appears in multiple class buckets; union its mechanism set once
+            seen.setdefault(key, set()).update(mech_classes)
+            idx[key] = seen[key]
+    for c in landscape.get("classifications", []):
+        for th in c["therapies"]:
+            add(th, {m["class"] for m in th.get("mechanisms", [])})
+    for th in landscape.get("unclassified", []):
+        add(th, set())
+    return idx
+
+
+def main() -> None:
+    if not LANDSCAPE_PATH.exists():
+        console.print("[red]No landscape.json — run scripts/build_landscape.py first[/red]"); sys.exit(1)
+    landscape = json.loads(LANDSCAPE_PATH.read_text())
+    gold = json.loads(THERAPY_GOLD_PATH.read_text())["gold"]
+    idx = _predicted_index(landscape)
+
+    tp = fp = fn = 0
+    exact = matched = abstained = missing = 0
+    rows = []
+    for g in gold:
+        names = [g["therapy"], *g.get("aliases", [])]
+        pred = None
+        for nm in names:
+            if nm.lower() in idx:
+                pred = idx[nm.lower()]; break
+        gset = set(g["mechanisms"])
+        if pred is None:
+            missing += 1
+            rows.append(("?", g["therapy"], "NOT IN LANDSCAPE", gset, set()))
+            continue
+        matched += 1
+        if not pred:
+            abstained += 1
+        inter = pred & gset
+        tp += len(inter); fp += len(pred - gset); fn += len(gset - pred)
+        if pred == gset:
+            exact += 1
+        mark = "✓" if pred == gset else ("~" if inter else "✗")
+        rows.append((mark, g["therapy"], "", gset, pred))
+
+    prec = tp / (tp + fp) if (tp + fp) else 0.0
+    rec = tp / (tp + fn) if (tp + fn) else 0.0
+    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) else 0.0
+
+    console.print("\n[bold]Per-therapy (✓ exact, ~ partial, ✗ wrong):[/bold]")
+    for mark, name, note, gset, pred in sorted(rows, key=lambda r: r[0]):
+        color = {"✓": "green", "~": "yellow", "✗": "red", "?": "dim"}.get(mark, "white")
+        console.print(f"  [{color}]{mark}[/{color}] {name:22s} gold={sorted(gset)} pred={sorted(pred)} {note}")
+
+    console.print(f"\n[bold]Matched {matched}/{len(gold)} gold therapies[/bold] "
+                  f"({missing} not in landscape, {abstained} abstained)")
+    console.print(f"  Micro precision: [bold]{prec:.2f}[/bold]  recall: {rec:.2f}  F1: {f1:.2f}")
+    console.print(f"  Exact-set match: {exact}/{matched}")
+
+    # hard assertions for the flagged cases
+    def pred_of(name):
+        return idx.get(name.lower())
+    console.print("\n[bold]Flagged-case checks:[/bold]")
+    cnm = pred_of("CNM-Au8") or set()
+    ok_cnm = "Neuroinflammation" not in cnm and ({"Mitochondrial dysfunction", "Oxidative stress"} & cnm)
+    console.print(f"  CNM-Au8 not Neuroinflammation & has bioenergetic/oxidative: "
+                  f"[{'green' if ok_cnm else 'red'}]{bool(ok_cnm)}[/] (pred={sorted(cnm)})")
+
+
+if __name__ == "__main__":
+    main()

tools.py

@@ -43,6 +43,17 @@ EXTRACT_TRIAL_TARGETS_TOOL: anthropic.types.ToolParam = {
     "input_schema": _load("extract_trial_targets"),
 }
 
+CLASSIFY_THERAPY_TOOL: anthropic.types.ToolParam = {
+    "name": "classify_therapy",
+    "description": (
+        "Classify one experimental ALS therapy: its canonical name, modality, molecular target, "
+        "mechanism of action, and best-fit mechanism class from the ALS taxonomy. "
+        "Call once per therapy, echoing back the therapy_key verbatim."
+    ),
+    "input_schema": _load("classify_therapy"),
+}
+
 EXTRACTION_TOOLS: list[anthropic.types.ToolParam] = [EXTRACT_ENTITIES_TOOL]
 TRIAL_EXTRACTION_TOOLS: list[anthropic.types.ToolParam] = [EXTRACT_TRIAL_TARGETS_TOOL]
 RESEARCH_TOOLS: list[anthropic.types.ToolParam] = [SEARCH_LANDSCAPE_TOOL]
+LANDSCAPE_TOOLS: list[anthropic.types.ToolParam] = [CLASSIFY_THERAPY_TOOL]

uv.lock

@@ -452,6 +452,7 @@ dependencies = [
     { name = "httpx" },
     { name = "networkx" },
     { name = "openai" },
+    { name = "plotly" },
     { name = "python-dotenv" },
     { name = "rapidfuzz" },
     { name = "rich" },
@@ -475,6 +476,7 @@ requires-dist = [
     { name = "httpx", specifier = ">=0.27.0" },
     { name = "networkx", specifier = ">=3.3" },
     { name = "openai", specifier = ">=2.44.0" },
+    { name = "plotly", specifier = "==6.9.0" },
     { name = "pytest", marker = "extra == 'dev'", specifier = ">=8.0" },
     { name = "pytest-cov", marker = "extra == 'dev'" },
     { name = "pytest-httpx", marker = "extra == 'dev'", specifier = ">=0.35" },
@@ -2433,6 +2435,19 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/36/54/0169bc772ec491108b62f644f8ecf1fe5d8ae5ebafde2ee2142210166903/pillow-12.3.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:04f01d28a6aaff387bf842a13be313df23ba0597a44f1a976c9feb3c6ff4711a", size = 7231786, upload-time = "2026-07-01T11:56:35.046Z" },
 ]
 
+[[package]]
+name = "plotly"
+version = "6.9.0"
+source = { registry = "https://pypi.org/simple" }
+dependencies = [
+    { name = "narwhals" },
+    { name = "packaging" },
+]
+sdist = { url = "https://files.pythonhosted.org/packages/96/07/795c79dbce40c39bece88e69d049babbd23ffa95b5d117f248db8ea03abb/plotly-6.9.0.tar.gz", hash = "sha256:967ad33e8c704fed051800d11d985eb206a9c795c14206b30a6f463ed9c67d0d", size = 6919903, upload-time = "2026-07-09T14:55:59.982Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/24/18/d8544811ab076f876c4892b3714f5b0dad335e1dc33aef826df431b8325d/plotly-6.9.0-py3-none-any.whl", hash = "sha256:36bebe2f1bb13884774fe61689c329071446f6ce4a8927fb1f0d6fb24f581236", size = 9909646, upload-time = "2026-07-09T14:55:55.421Z" },
+]
+
 [[package]]
 name = "pluggy"
 version = "1.6.0"