Spaces:
Running
Running
| # Compute Integration — live CRC features → 4 HF datasets → dashboard | |
| This document is the **contract** for replacing the dashboard's single frozen feature | |
| table (`SupraBench/physics_feature/test_features_table.csv`) with features that we | |
| compute ourselves from the `SupraEngineering` pipeline on **CRC**, served through | |
| **four** Hugging Face datasets. | |
| It is the spec implementation works against. Code is written by Codex; this doc is | |
| owned by the planning side and kept in sync as decisions change. | |
| --- | |
| ## 1. Goal | |
| - Stop reading a hand-authored, third-party feature table. | |
| - Compute every feature ourselves (`SupraEngineering`, GLIDE docking on CRC). | |
| - Serve features to the dashboard via four private datasets under the `SupraBench` org. | |
| - Keep `app.py` / `prompts/builder.py` essentially untouched by preserving the | |
| `load_features()` / `guest_choices()` / `get_record()` API. | |
| A convenient invariant: the dashboard's feature **column names already equal** the | |
| pipeline's `constants.POSE_FEATURES` and `constants.SCORE_NAMES`. So as long as the | |
| datasets use those exact names, only the *loader* underneath the app changes. | |
| --- | |
| ## 2. The four datasets (private, `SupraBench` org) | |
| | Dataset | Holds | Rows | Needs docking? | Producer | | |
| |---|---|---|---|---| | |
| | `SupraBench/SupraDB-GEOM` | guest pool: `inchikey, name, smiles[, logka]` | whole pool | no | Phase 0 pool builder | | |
| | `SupraBench/SupraDB-LigandScore` | 9 ligand-only mechanism scores | whole pool | no (rdkit) | `score_nodock` | | |
| | `SupraBench/SupraDB-PoseFeat` | 24 pose features | docked subset | **yes** (GLIDE) | `compute_all_features` | | |
| | `SupraBench/SupraDB-CavityScore` | 4 cavity scores | docked subset | **yes** (GLIDE) | `compute_all_features` | | |
| - **Primary / join key:** `inchikey` for all four. | |
| - **Storage format:** one CSV per dataset (`*.csv`) + a generated `README.md` dataset card. | |
| - **Fixed pool** ⇒ fixed row counts. `SupraDB-GEOM` and `SupraDB-LigandScore` cover the | |
| *entire* pool (Group 1 is cheap, precomputed for everything). `PoseFeat` / `CavityScore` | |
| cover only the **docked subset** (default: the 63 labeled guests). | |
| ### 2.1 Column contracts | |
| `SupraDB-GEOM`: | |
| ``` | |
| inchikey, name, smiles, logka # logka may be empty for unlabeled (GEOM-only) guests | |
| ``` | |
| `SupraDB-LigandScore` (9 — produced by `score_nodock`, ligand-intrinsic, `charge_access=0`): | |
| ``` | |
| inchikey, | |
| S_charge, S_hydrophobic, S_rigidity, S_desolvation, S_packing, | |
| S_shape, S_conformer_diversity, S_boltzmann_concentration, S_bad | |
| ``` | |
| `SupraDB-CavityScore` (4 — cavity terms from the best docked pose): | |
| ``` | |
| inchikey, S_occupancy, S_portal, S_accessibility, S_orientation | |
| ``` | |
| `SupraDB-PoseFeat` (24 pose features, `constants.POSE_FEATURES` order, + provenance): | |
| ``` | |
| inchikey, | |
| DockingScore, Pose_Energy, Distance_to_Cavity_Center, Distance_to_Portal, | |
| Insertion_Depth, Packing_Coefficient, Occupancy, Hydrophobic_Occupancy, | |
| Shape_Complementarity, Steric_Clash, Guest_CB7_Min_Distance, Pose_RMSD_to_Template, | |
| Portal_Compatibility, Positive_Center_to_Portal_Distance, Positive_Center_Orientation, | |
| Charge_Accessibility, Portal_Facing_Accessibility, HBond_Count, HBond_Geometry, | |
| Carbonyl_Oxygen_Contact_Count, Hydrophobic_Contact, Polar_Contact_Penalty, | |
| Bad_Group_Portal_Exposure, Desolvation_Penalty, | |
| boltzmann_weight, delta_e # provenance of the kept (collapsed) pose | |
| ``` | |
| > Why `S_charge` lives in `LigandScore`: it is computed by `score_nodock` with the | |
| > cavity-dependent `charge_access` term set to 0, so it is reproducible from SMILES for | |
| > the whole pool. The pose-enhanced `charge_access` from the full run is reflected | |
| > separately in `CavityScore` (`S_accessibility`); we do **not** overwrite the | |
| > ligand-score `S_charge` per-pose. This keeps each dataset's provenance clean. | |
| ### 2.2 Pose collapse | |
| `pose_feats.pkl` stores up to *P* poses/guest as `float32[P, 24]` with per-pose | |
| `delta_e` and `boltz`. The dashboard shows one row/guest, so `publish.py` **collapses** | |
| to the single pose with the **highest `boltzmann_weight`** (the dominant, lowest-energy | |
| geometry — a real pose, not an average). `boltzmann_weight` + `delta_e` of that pose are | |
| carried into the CSV for transparency. (Alternative rules — weighted average, lowest | |
| GlideScore — are documented but not the default.) | |
| --- | |
| ## 3. Identifiers — SMILES & InChIKey | |
| The pipeline and dashboard both **assume** `inchikey`/`smiles` are given; neither derives | |
| them. We make that explicit in Phase 0: | |
| 1. **SMILES** is the source of truth (from GEOM, or the seed `guests.csv`). | |
| 2. **Canonicalize + desalt once** (RDKit, largest fragment) — matches what the pipeline | |
| does internally (`ligand_descriptors`, `smiles_to_xyz`), so the stored InChIKey is the | |
| same key the pipeline computes under. | |
| 3. **InChIKey = `Chem.MolToInchiKey(Chem.MolFromSmiles(canonical_smiles))`** — a canonical | |
| hash; identical molecules → identical key. This is the join key across all four | |
| datasets, the SQLite cache, and the 2D/3D structure fetch. | |
| 4. `name`: keep any name GEOM/seed provides; else fall back to a short InChIKey label. | |
| Generated in exactly one place (the pool builder) so every downstream piece keys off the | |
| same value. | |
| --- | |
| ## 4. Feature selection (compute + inference) | |
| All **37** features are stored. Selection is a *choice*, defaulting to a recommendation. | |
| - **At inference (drives the LLM prompt):** a UI selector with presets — | |
| `Recommended` (default, the curated ~22: strongest pose features + meaningful scores, | |
| dropping redundant pairs like `Occupancy`/`S_occupancy`), `All 37`, `Physics`, | |
| `Chemistry`, `Custom` (per-feature). The existing physics/chemistry/combined | |
| trajectories become presets of this same control. The selected set drives | |
| `prompts/builder.py:_feature_lines()` instead of the fixed `FEATURES` constant. The | |
| choice is recorded with each sample and **folded into the cache key** so different | |
| feature sets don't collide. | |
| - **At compute:** the meaningful choice is *which of the three groups to (re)compute* | |
| (docking is the expensive step; once docked, all pose features are free). Default: all. | |
| The `Recommended` preset = the current 22-feature `FEATURES` list in `data/loader.py`. | |
| --- | |
| ## 5. Dataset cards (README.md per dataset) | |
| Every `SupraDB-*` dataset ships a generated HF dataset card: | |
| - YAML frontmatter: `license`, `pretty_name`, `tags`, `configs` (point at the CSV). | |
| - Body: what it is + pipeline position; full **schema** (column, dtype, units, meaning | |
| from `DEFINITIONS.md` / `features_lib`); the `inchikey` join-key note; **provenance** | |
| (CRC; GLIDE 2025u2 / aISS fallback; pose-collapse rule; row count); and the exact | |
| `refresh`/`publish.py` command + `SupraEngineering` commit that produced it. | |
| Cards are **generated by the scripts** (Phase 0 writes the GEOM card; `publish.py` writes | |
| the three feature cards) so they never drift from the data. | |
| --- | |
| ## 6. Pipeline (Phases) | |
| ``` | |
| GEOM SMILES (or seed guests.csv) | |
| │ Phase 0: pool_builder.py (RDKit canonicalize+desalt → InChIKey) | |
| ▼ | |
| SupraDB-GEOM ◄── source of truth (guest list, names, SMILES, logka) | |
| │ | |
| ├─ Group 1: score_nodock (rdkit, whole pool) ─────► SupraDB-LigandScore | |
| └─ Groups 2+3: dock_glide → compute_all_features ─► SupraDB-PoseFeat + SupraDB-CavityScore | |
| (CRC, GLIDE 2025u2; docked subset) │ | |
| all four joined on inchikey | |
| ▼ | |
| data/loader.py → app.py (unchanged) | |
| ``` | |
| - **Phase 0 — Pool builder** (`SupraEngineering/src/pool_builder.py`): SMILES source → | |
| canonical `guests.csv` + `SupraDB-GEOM` card; `--push` uploads. Seed source = the | |
| existing 63-guest `SupraEngineering/data/guests.csv`; pluggable for a GEOM list. | |
| - **Phase 1 — CRC compute** (two run dirs, because `score_nodock` and | |
| `compute_all_features` both write `features/scores.pkl` and would clobber each other); | |
| one SGE wrapper `SupraEngineering/scripts/crc_compute.sge` runs both: | |
| - **whole-pool no-dock run:** `scripts/score_nodock.sh <pool_dir>` → `<pool_dir>/features/scores.pkl` | |
| (13-d, cavity terms = 0). Source of **LigandScore** (the 9 ligand cols). | |
| - **docked-subset run:** `scripts/dock_glide.sh <dock_dir>` then `scripts/compute_all_features.sh <dock_dir>` | |
| → `<dock_dir>/features/scores.pkl` (full 13-d) + `pose_feats.pkl`. Source of | |
| **CavityScore** (the 4 cavity cols) and **PoseFeat** (24-d). | |
| - GLIDE confirmed at `/opt/crc/s/schrodinger/2025u2` (module `schrodinger/2025u2`). | |
| aISS/xtb fallback on GLIDE failure — **xtb is not yet on CRC** | |
| (`conda install -c conda-forge xtb` first). | |
| - **Phase 2 — Publish** (`SupraEngineering/src/publish.py`, runs on CRC, HF_TOKEN present): | |
| inputs `--pool-scores <pool_dir>/features/scores.pkl`, | |
| `--dock-scores <dock_dir>/features/scores.pkl`, `--pose-feats <dock_dir>/features/pose_feats.pkl`. | |
| Splits into 3 feature CSVs (`features.csv` each, full named columns) + cards, collapses | |
| poses by Boltzmann weight, uploads to the three feature repos. | |
| - **Phase 3 — Loader** (`src/data/loader.py`): read the 4 datasets, LEFT-join on | |
| `inchikey`, return the same `rec` dict. Env: 4 repo ids + `HF_TOKEN`; `LOCAL_*` CSV | |
| fallbacks for offline dev; `lru_cache`. Report which datasets loaded (health strip). | |
| - **Phase 4 — UI + automation**: feature-selection control; one-command `refresh` | |
| (ssh CRC → qsub → wait → publish); finalize this runbook. | |
| --- | |
| ## 7. CRC connection (runbook) | |
| - Host `crc` → `crcfe01.crc.nd.edu` (fallback `crcfe02`), user `tma2`, **password-only** | |
| (no SSH keys). From a Mac session: | |
| `sshpass -p "$(op read 'op://openclaw/CRC/password')" ssh crc '<cmd>'` | |
| (`op` unlocked via Touch ID; `sshpass`/`op` via Homebrew). | |
| - Scheduler: **SGE** (`qsub`/`qstat`/`qdel`); never run docking interactively on a login | |
| node for the full pool. | |
| - GLIDE: `module load schrodinger/2025u2`; `$SCHRODINGER=/opt/crc/s/schrodinger/2025u2`. | |
| - Env: a Python env with `rdkit, numpy, pandas, scipy` (the project's `GPM` conda env); | |
| `huggingface_hub` + `HF_TOKEN` on CRC for the publish step. | |
| --- | |
| ## 8. Decision ledger | |
| | Decision | Status | | |
| |---|---| | |
| | 4 datasets `SupraDB-GEOM/-LigandScore/-PoseFeat/-CavityScore` | locked | | |
| | `inchikey` = sole join key, derived once in Phase 0 from SMILES | locked | | |
| | Store all 37 features; inference set selectable; `Recommended` default | locked | | |
| | Each dataset ships a generated `README.md` card | locked | | |
| | `publish.py` runs on CRC (HF_TOKEN already there) | locked | | |
| | GLIDE 2025u2 default (confirmed on CRC); aISS/xtb fallback (xtb needs install) | locked | | |
| | Pose collapse = highest Boltzmann weight | default (flag to change) | | |
| | GEOM Group-1 scope = CB[7]-plausible subset (not all ~430k) | default (flag to change) | | |
| | Docking subset = 63 labeled guests; GEOM-only guests prediction-only | default (flag to change) | | |