docs/COMPUTE_INTEGRATION.md

Compute Integration — live CRC features → 4 HF datasets → dashboard

This document is the contract for replacing the dashboard's single frozen feature table (SupraBench/physics_feature/test_features_table.csv) with features that we compute ourselves from the SupraEngineering pipeline on CRC, served through four Hugging Face datasets.

It is the spec implementation works against. Code is written by Codex; this doc is owned by the planning side and kept in sync as decisions change.

---

1. Goal

• Stop reading a hand-authored, third-party feature table.
• Compute every feature ourselves (SupraEngineering, GLIDE docking on CRC).
• Serve features to the dashboard via four private datasets under the SupraBench org.
• Keep app.py / prompts/builder.py essentially untouched by preserving the load_features() / guest_choices() / get_record() API.

A convenient invariant: the dashboard's feature column names already equal the pipeline's constants.POSE_FEATURES and constants.SCORE_NAMES. So as long as the datasets use those exact names, only the loader underneath the app changes.

---

2. The four datasets (private, SupraBench org)

Dataset	Holds	Rows	Needs docking?	Producer
SupraBench/SupraDB-GEOM	guest pool: inchikey, name, smiles[, logka]	whole pool	no	Phase 0 pool builder
SupraBench/SupraDB-LigandScore	9 ligand-only mechanism scores	whole pool	no (rdkit)	score_nodock
SupraBench/SupraDB-PoseFeat	24 pose features	docked subset	yes (GLIDE)	compute_all_features
SupraBench/SupraDB-CavityScore	4 cavity scores	docked subset	yes (GLIDE)	compute_all_features

• Primary / join key: inchikey for all four.
• Storage format: one CSV per dataset (*.csv) + a generated README.md dataset card.
• Fixed pool ⇒ fixed row counts. SupraDB-GEOM and SupraDB-LigandScore cover the entire pool (Group 1 is cheap, precomputed for everything). PoseFeat / CavityScore cover only the docked subset (default: the 63 labeled guests).

2.1 Column contracts

SupraDB-GEOM:

inchikey, name, smiles, logka   # logka may be empty for unlabeled (GEOM-only) guests

SupraDB-LigandScore (9 — produced by score_nodock, ligand-intrinsic, charge_access=0):

inchikey,
S_charge, S_hydrophobic, S_rigidity, S_desolvation, S_packing,
S_shape, S_conformer_diversity, S_boltzmann_concentration, S_bad

SupraDB-CavityScore (4 — cavity terms from the best docked pose):

inchikey, S_occupancy, S_portal, S_accessibility, S_orientation

SupraDB-PoseFeat (24 pose features, constants.POSE_FEATURES order, + provenance):

inchikey,
DockingScore, Pose_Energy, Distance_to_Cavity_Center, Distance_to_Portal,
Insertion_Depth, Packing_Coefficient, Occupancy, Hydrophobic_Occupancy,
Shape_Complementarity, Steric_Clash, Guest_CB7_Min_Distance, Pose_RMSD_to_Template,
Portal_Compatibility, Positive_Center_to_Portal_Distance, Positive_Center_Orientation,
Charge_Accessibility, Portal_Facing_Accessibility, HBond_Count, HBond_Geometry,
Carbonyl_Oxygen_Contact_Count, Hydrophobic_Contact, Polar_Contact_Penalty,
Bad_Group_Portal_Exposure, Desolvation_Penalty,
boltzmann_weight, delta_e          # provenance of the kept (collapsed) pose

Why S_charge lives in LigandScore: it is computed by score_nodock with the cavity-dependent charge_access term set to 0, so it is reproducible from SMILES for the whole pool. The pose-enhanced charge_access from the full run is reflected separately in CavityScore (S_accessibility); we do not overwrite the ligand-score S_charge per-pose. This keeps each dataset's provenance clean.

2.2 Pose collapse

pose_feats.pkl stores up to P poses/guest as float32[P, 24] with per-pose delta_e and boltz. The dashboard shows one row/guest, so publish.py collapses to the single pose with the highest boltzmann_weight (the dominant, lowest-energy geometry — a real pose, not an average). boltzmann_weight + delta_e of that pose are carried into the CSV for transparency. (Alternative rules — weighted average, lowest GlideScore — are documented but not the default.)

---

3. Identifiers — SMILES & InChIKey

The pipeline and dashboard both assume inchikey/smiles are given; neither derives them. We make that explicit in Phase 0:

1. SMILES is the source of truth (from GEOM, or the seed guests.csv).
2. Canonicalize + desalt once (RDKit, largest fragment) — matches what the pipeline does internally (ligand_descriptors, smiles_to_xyz), so the stored InChIKey is the same key the pipeline computes under.
3. InChIKey = Chem.MolToInchiKey(Chem.MolFromSmiles(canonical_smiles)) — a canonical hash; identical molecules → identical key. This is the join key across all four datasets, the SQLite cache, and the 2D/3D structure fetch.
4. name: keep any name GEOM/seed provides; else fall back to a short InChIKey label.

Generated in exactly one place (the pool builder) so every downstream piece keys off the same value.

---

4. Feature selection (compute + inference)

All 37 features are stored. Selection is a choice, defaulting to a recommendation.

• At inference (drives the LLM prompt): a UI selector with presets — Recommended (default, the curated ~22: strongest pose features + meaningful scores, dropping redundant pairs like Occupancy/S_occupancy), All 37, Physics, Chemistry, Custom (per-feature). The existing physics/chemistry/combined trajectories become presets of this same control. The selected set drives prompts/builder.py:_feature_lines() instead of the fixed FEATURES constant. The choice is recorded with each sample and folded into the cache key so different feature sets don't collide.
• At compute: the meaningful choice is which of the three groups to (re)compute (docking is the expensive step; once docked, all pose features are free). Default: all.

The Recommended preset = the current 22-feature FEATURES list in data/loader.py.

---

5. Dataset cards (README.md per dataset)

Every SupraDB-* dataset ships a generated HF dataset card:

• YAML frontmatter: license, pretty_name, tags, configs (point at the CSV).
• Body: what it is + pipeline position; full schema (column, dtype, units, meaning from DEFINITIONS.md / features_lib); the inchikey join-key note; provenance (CRC; GLIDE 2025u2 / aISS fallback; pose-collapse rule; row count); and the exact refresh/publish.py command + SupraEngineering commit that produced it.

Cards are generated by the scripts (Phase 0 writes the GEOM card; publish.py writes the three feature cards) so they never drift from the data.

---

6. Pipeline (Phases)

GEOM SMILES (or seed guests.csv)
   │  Phase 0: pool_builder.py  (RDKit canonicalize+desalt → InChIKey)
   ▼
SupraDB-GEOM  ◄── source of truth (guest list, names, SMILES, logka)
   │
   ├─ Group 1: score_nodock (rdkit, whole pool) ─────► SupraDB-LigandScore
   └─ Groups 2+3: dock_glide → compute_all_features ─► SupraDB-PoseFeat + SupraDB-CavityScore
      (CRC, GLIDE 2025u2; docked subset)                       │
                                       all four joined on inchikey
                                                               ▼
                                            data/loader.py → app.py (unchanged)

• Phase 0 — Pool builder (SupraEngineering/src/pool_builder.py): SMILES source → canonical guests.csv + SupraDB-GEOM card; --push uploads. Seed source = the existing 63-guest SupraEngineering/data/guests.csv; pluggable for a GEOM list.
• Phase 1 — CRC compute (two run dirs, because score_nodock and compute_all_features both write features/scores.pkl and would clobber each other); one SGE wrapper SupraEngineering/scripts/crc_compute.sge runs both:
  • whole-pool no-dock run: scripts/score_nodock.sh <pool_dir> → <pool_dir>/features/scores.pkl (13-d, cavity terms = 0). Source of LigandScore (the 9 ligand cols).
  • docked-subset run: scripts/dock_glide.sh <dock_dir> then scripts/compute_all_features.sh <dock_dir> → <dock_dir>/features/scores.pkl (full 13-d) + pose_feats.pkl. Source of CavityScore (the 4 cavity cols) and PoseFeat (24-d).
  • GLIDE confirmed at /opt/crc/s/schrodinger/2025u2 (module schrodinger/2025u2). aISS/xtb fallback on GLIDE failure — xtb is not yet on CRC (conda install -c conda-forge xtb first).
• Phase 2 — Publish (SupraEngineering/src/publish.py, runs on CRC, HF_TOKEN present): inputs --pool-scores <pool_dir>/features/scores.pkl, --dock-scores <dock_dir>/features/scores.pkl, --pose-feats <dock_dir>/features/pose_feats.pkl. Splits into 3 feature CSVs (features.csv each, full named columns) + cards, collapses poses by Boltzmann weight, uploads to the three feature repos.
• Phase 3 — Loader (src/data/loader.py): read the 4 datasets, LEFT-join on inchikey, return the same rec dict. Env: 4 repo ids + HF_TOKEN; LOCAL_* CSV fallbacks for offline dev; lru_cache. Report which datasets loaded (health strip).
• Phase 4 — UI + automation: feature-selection control; one-command refresh (ssh CRC → qsub → wait → publish); finalize this runbook.

---

7. CRC connection (runbook)

• Host crc → crcfe01.crc.nd.edu (fallback crcfe02), user tma2, password-only (no SSH keys). From a Mac session: sshpass -p "$(op read 'op://openclaw/CRC/password')" ssh crc '<cmd>' (op unlocked via Touch ID; sshpass/op via Homebrew).
• Scheduler: SGE (qsub/qstat/qdel); never run docking interactively on a login node for the full pool.
• GLIDE: module load schrodinger/2025u2; $SCHRODINGER=/opt/crc/s/schrodinger/2025u2.
• Env: a Python env with rdkit, numpy, pandas, scipy (the project's GPM conda env); huggingface_hub + HF_TOKEN on CRC for the publish step.

---

8. Decision ledger

Decision	Status
4 datasets SupraDB-GEOM/-LigandScore/-PoseFeat/-CavityScore	locked
inchikey = sole join key, derived once in Phase 0 from SMILES	locked
Store all 37 features; inference set selectable; Recommended default	locked
Each dataset ships a generated README.md card	locked
publish.py runs on CRC (HF_TOKEN already there)	locked
GLIDE 2025u2 default (confirmed on CRC); aISS/xtb fallback (xtb needs install)	locked
Pose collapse = highest Boltzmann weight	default (flag to change)
GEOM Group-1 scope = CB[7]-plausible subset (not all ~430k)	default (flag to change)
Docking subset = 63 labeled guests; GEOM-only guests prediction-only	default (flag to change)