# Compute Integration — live CRC features → 4 HF datasets → dashboard

This document is the **contract** for replacing the dashboard's single frozen feature
table (`SupraBench/physics_feature/test_features_table.csv`) with features that we
compute ourselves from the `SupraEngineering` pipeline on **CRC**, served through
**four** Hugging Face datasets.

It is the spec implementation works against. Code is written by Codex; this doc is
owned by the planning side and kept in sync as decisions change.

---

## 1. Goal

- Stop reading a hand-authored, third-party feature table.
- Compute every feature ourselves (`SupraEngineering`, GLIDE docking on CRC).
- Serve features to the dashboard via four private datasets under the `SupraBench` org.
- Keep `app.py` / `prompts/builder.py` essentially untouched by preserving the
  `load_features()` / `guest_choices()` / `get_record()` API.

A convenient invariant: the dashboard's feature **column names already equal** the
pipeline's `constants.POSE_FEATURES` and `constants.SCORE_NAMES`. So as long as the
datasets use those exact names, only the *loader* underneath the app changes.

---

## 2. The four datasets (private, `SupraBench` org)

| Dataset | Holds | Rows | Needs docking? | Producer |
|---|---|---|---|---|
| `SupraBench/SupraDB-GEOM` | guest pool: `inchikey, name, smiles[, logka]` | whole pool | no | Phase 0 pool builder |
| `SupraBench/SupraDB-LigandScore` | 9 ligand-only mechanism scores | whole pool | no (rdkit) | `score_nodock` |
| `SupraBench/SupraDB-PoseFeat` | 24 pose features | docked subset | **yes** (GLIDE) | `compute_all_features` |
| `SupraBench/SupraDB-CavityScore` | 4 cavity scores | docked subset | **yes** (GLIDE) | `compute_all_features` |

- **Primary / join key:** `inchikey` for all four.
- **Storage format:** one CSV per dataset (`*.csv`) + a generated `README.md` dataset card.
- **Fixed pool** ⇒ fixed row counts. `SupraDB-GEOM` and `SupraDB-LigandScore` cover the
  *entire* pool (Group 1 is cheap, precomputed for everything). `PoseFeat` / `CavityScore`
  cover only the **docked subset** (default: the 63 labeled guests).

### 2.1 Column contracts

`SupraDB-GEOM`:
```
inchikey, name, smiles, logka   # logka may be empty for unlabeled (GEOM-only) guests
```

`SupraDB-LigandScore` (9 — produced by `score_nodock`, ligand-intrinsic, `charge_access=0`):
```
inchikey,
S_charge, S_hydrophobic, S_rigidity, S_desolvation, S_packing,
S_shape, S_conformer_diversity, S_boltzmann_concentration, S_bad
```

`SupraDB-CavityScore` (4 — cavity terms from the best docked pose):
```
inchikey, S_occupancy, S_portal, S_accessibility, S_orientation
```

`SupraDB-PoseFeat` (24 pose features, `constants.POSE_FEATURES` order, + provenance):
```
inchikey,
DockingScore, Pose_Energy, Distance_to_Cavity_Center, Distance_to_Portal,
Insertion_Depth, Packing_Coefficient, Occupancy, Hydrophobic_Occupancy,
Shape_Complementarity, Steric_Clash, Guest_CB7_Min_Distance, Pose_RMSD_to_Template,
Portal_Compatibility, Positive_Center_to_Portal_Distance, Positive_Center_Orientation,
Charge_Accessibility, Portal_Facing_Accessibility, HBond_Count, HBond_Geometry,
Carbonyl_Oxygen_Contact_Count, Hydrophobic_Contact, Polar_Contact_Penalty,
Bad_Group_Portal_Exposure, Desolvation_Penalty,
boltzmann_weight, delta_e          # provenance of the kept (collapsed) pose
```

> Why `S_charge` lives in `LigandScore`: it is computed by `score_nodock` with the
> cavity-dependent `charge_access` term set to 0, so it is reproducible from SMILES for
> the whole pool. The pose-enhanced `charge_access` from the full run is reflected
> separately in `CavityScore` (`S_accessibility`); we do **not** overwrite the
> ligand-score `S_charge` per-pose. This keeps each dataset's provenance clean.

### 2.2 Pose collapse

`pose_feats.pkl` stores up to *P* poses/guest as `float32[P, 24]` with per-pose
`delta_e` and `boltz`. The dashboard shows one row/guest, so `publish.py` **collapses**
to the single pose with the **highest `boltzmann_weight`** (the dominant, lowest-energy
geometry — a real pose, not an average). `boltzmann_weight` + `delta_e` of that pose are
carried into the CSV for transparency. (Alternative rules — weighted average, lowest
GlideScore — are documented but not the default.)

---

## 3. Identifiers — SMILES & InChIKey

The pipeline and dashboard both **assume** `inchikey`/`smiles` are given; neither derives
them. We make that explicit in Phase 0:

1. **SMILES** is the source of truth (from GEOM, or the seed `guests.csv`).
2. **Canonicalize + desalt once** (RDKit, largest fragment) — matches what the pipeline
   does internally (`ligand_descriptors`, `smiles_to_xyz`), so the stored InChIKey is the
   same key the pipeline computes under.
3. **InChIKey = `Chem.MolToInchiKey(Chem.MolFromSmiles(canonical_smiles))`** — a canonical
   hash; identical molecules → identical key. This is the join key across all four
   datasets, the SQLite cache, and the 2D/3D structure fetch.
4. `name`: keep any name GEOM/seed provides; else fall back to a short InChIKey label.

Generated in exactly one place (the pool builder) so every downstream piece keys off the
same value.

---

## 4. Feature selection (compute + inference)

All **37** features are stored. Selection is a *choice*, defaulting to a recommendation.

- **At inference (drives the LLM prompt):** a UI selector with presets —
  `Recommended` (default, the curated ~22: strongest pose features + meaningful scores,
  dropping redundant pairs like `Occupancy`/`S_occupancy`), `All 37`, `Physics`,
  `Chemistry`, `Custom` (per-feature). The existing physics/chemistry/combined
  trajectories become presets of this same control. The selected set drives
  `prompts/builder.py:_feature_lines()` instead of the fixed `FEATURES` constant. The
  choice is recorded with each sample and **folded into the cache key** so different
  feature sets don't collide.
- **At compute:** the meaningful choice is *which of the three groups to (re)compute*
  (docking is the expensive step; once docked, all pose features are free). Default: all.

The `Recommended` preset = the current 22-feature `FEATURES` list in `data/loader.py`.

---

## 5. Dataset cards (README.md per dataset)

Every `SupraDB-*` dataset ships a generated HF dataset card:
- YAML frontmatter: `license`, `pretty_name`, `tags`, `configs` (point at the CSV).
- Body: what it is + pipeline position; full **schema** (column, dtype, units, meaning
  from `DEFINITIONS.md` / `features_lib`); the `inchikey` join-key note; **provenance**
  (CRC; GLIDE 2025u2 / aISS fallback; pose-collapse rule; row count); and the exact
  `refresh`/`publish.py` command + `SupraEngineering` commit that produced it.

Cards are **generated by the scripts** (Phase 0 writes the GEOM card; `publish.py` writes
the three feature cards) so they never drift from the data.

---

## 6. Pipeline (Phases)

```
GEOM SMILES (or seed guests.csv)
   │  Phase 0: pool_builder.py  (RDKit canonicalize+desalt → InChIKey)
   ▼
SupraDB-GEOM  ◄── source of truth (guest list, names, SMILES, logka)
   │
   ├─ Group 1: score_nodock (rdkit, whole pool) ─────► SupraDB-LigandScore
   └─ Groups 2+3: dock_glide → compute_all_features ─► SupraDB-PoseFeat + SupraDB-CavityScore
      (CRC, GLIDE 2025u2; docked subset)                       │
                                       all four joined on inchikey
                                                               ▼
                                            data/loader.py → app.py (unchanged)
```

- **Phase 0 — Pool builder** (`SupraEngineering/src/pool_builder.py`): SMILES source →
  canonical `guests.csv` + `SupraDB-GEOM` card; `--push` uploads. Seed source = the
  existing 63-guest `SupraEngineering/data/guests.csv`; pluggable for a GEOM list.
- **Phase 1 — CRC compute** (two run dirs, because `score_nodock` and
  `compute_all_features` both write `features/scores.pkl` and would clobber each other);
  one SGE wrapper `SupraEngineering/scripts/crc_compute.sge` runs both:
  - **whole-pool no-dock run:** `scripts/score_nodock.sh <pool_dir>` → `<pool_dir>/features/scores.pkl`
    (13-d, cavity terms = 0). Source of **LigandScore** (the 9 ligand cols).
  - **docked-subset run:** `scripts/dock_glide.sh <dock_dir>` then `scripts/compute_all_features.sh <dock_dir>`
    → `<dock_dir>/features/scores.pkl` (full 13-d) + `pose_feats.pkl`. Source of
    **CavityScore** (the 4 cavity cols) and **PoseFeat** (24-d).
  - GLIDE confirmed at `/opt/crc/s/schrodinger/2025u2` (module `schrodinger/2025u2`).
    aISS/xtb fallback on GLIDE failure — **xtb is not yet on CRC**
    (`conda install -c conda-forge xtb` first).
- **Phase 2 — Publish** (`SupraEngineering/src/publish.py`, runs on CRC, HF_TOKEN present):
  inputs `--pool-scores <pool_dir>/features/scores.pkl`,
  `--dock-scores <dock_dir>/features/scores.pkl`, `--pose-feats <dock_dir>/features/pose_feats.pkl`.
  Splits into 3 feature CSVs (`features.csv` each, full named columns) + cards, collapses
  poses by Boltzmann weight, uploads to the three feature repos.
- **Phase 3 — Loader** (`src/data/loader.py`): read the 4 datasets, LEFT-join on
  `inchikey`, return the same `rec` dict. Env: 4 repo ids + `HF_TOKEN`; `LOCAL_*` CSV
  fallbacks for offline dev; `lru_cache`. Report which datasets loaded (health strip).
- **Phase 4 — UI + automation**: feature-selection control; one-command `refresh`
  (ssh CRC → qsub → wait → publish); finalize this runbook.

---

## 7. CRC connection (runbook)

- Host `crc` → `crcfe01.crc.nd.edu` (fallback `crcfe02`), user `tma2`, **password-only**
  (no SSH keys). From a Mac session:
  `sshpass -p "$(op read 'op://openclaw/CRC/password')" ssh crc '<cmd>'`
  (`op` unlocked via Touch ID; `sshpass`/`op` via Homebrew).
- Scheduler: **SGE** (`qsub`/`qstat`/`qdel`); never run docking interactively on a login
  node for the full pool.
- GLIDE: `module load schrodinger/2025u2`; `$SCHRODINGER=/opt/crc/s/schrodinger/2025u2`.
- Env: a Python env with `rdkit, numpy, pandas, scipy` (the project's `GPM` conda env);
  `huggingface_hub` + `HF_TOKEN` on CRC for the publish step.

---

## 8. Decision ledger

| Decision | Status |
|---|---|
| 4 datasets `SupraDB-GEOM/-LigandScore/-PoseFeat/-CavityScore` | locked |
| `inchikey` = sole join key, derived once in Phase 0 from SMILES | locked |
| Store all 37 features; inference set selectable; `Recommended` default | locked |
| Each dataset ships a generated `README.md` card | locked |
| `publish.py` runs on CRC (HF_TOKEN already there) | locked |
| GLIDE 2025u2 default (confirmed on CRC); aISS/xtb fallback (xtb needs install) | locked |
| Pose collapse = highest Boltzmann weight | default (flag to change) |
| GEOM Group-1 scope = CB[7]-plausible subset (not all ~430k) | default (flag to change) |
| Docking subset = 63 labeled guests; GEOM-only guests prediction-only | default (flag to change) |