OpenBind Blind Challenge 1 method report (KR LAB Bio)

README.md

@@ -0,0 +1,395 @@
+---
+license: cc-by-4.0
+language:
+  - en
+tags:
+  - chemistry
+  - drug-discovery
+  - structural-biology
+  - protein-ligand
+  - pose-prediction
+  - co-folding
+  - zika-virus
+  - openbind
+pretty_name: OpenBind Blind Challenge 1 Method Report (KR LAB Bio)
+---
+
+# Zika NS2B-NS3 Protease Complexes from Same-Campaign Templates, Docking and Co-Folding, with Consensus-Based Pose Selection
+
+**OpenBind Blind Challenge 1 - Method Report**
+Structure prediction track · Leaderboard alias **KR LAB Bio** · KR LAB Bio Inc., Republic of Korea
+Draft v1, 2026-11-06. Describes the submission currently on the leaderboard (internal label v18).
+No proprietary data was used. No test-set structures or labels were used at any step.
+
+---
+
+## 1. Summary
+
+We predicted all 400 blinded NS2B-NS3/ligand complexes with two independent pose generators
+and one selection procedure.
+
+- **Pipeline A, template transplantation plus docking.** Every public Zika NS2B-NS3 crystal
+  structure (294 PDB entries) was superposed onto the 29HN binding site. For each challenge
+  ligand, the maximum common substructure with the closest template ligands was placed on the
+  template's experimental coordinates, the rest of the molecule was built and relaxed, and the
+  template's own protein was used as the receptor. Exhaustive docking and gnina CNN rescoring
+  and refinement added candidates. A fixed rule selected one pose per ligand from a median of
+  66 candidates, using spatial consensus between independent templates, recovery of the
+  template contact pattern and the gnina CNN rank.
+- **Pipeline B, co-folding.** Protenix (`protenix_base_20250630_v1.0.0`, training cutoff
+  2025-06-30, no fine-tuning) produced 10 samples per ligand (2 seeds × 5 diffusion samples).
+  The representative sample is the stereochemically correct sample around which most other
+  samples cluster.
+- **Assembly.** The co-folding representative was used whenever it passed all 22 PoseBusters
+  checks (337 ligands). The remaining 63 ligands (39 with no stereochemically correct sample,
+  24 that failed PoseBusters) used the Pipeline A selection.
+
+Live leaderboard result (about 200 of the 400 compounds, collected 2026-11-06 11:34 UTC):
+
+| Metric | Value |
+|---|---|
+| Success rate | **68.5%** (137 of 200) |
+| BiSyRMSD < 1.5 Å | 137 |
+| lDDT-PLI > 0.8 | 155 |
+| PoseBusters pass | 200 / 200 |
+| Mean BiSyRMSD | 1.70 Å |
+| Mean lDDT-PLI | 0.837 |
+
+Four findings account for most of the work:
+
+1. **The public record already contains this chemical series.** 369 of the 400 challenge
+   ligands share at least half of their heavy atoms with a ligand in a public structure
+   (median MCS coverage 0.833), and 391 of the 400 closest templates come from the same
+   crystal form (P4<sub>3</sub>2<sub>2</sub>) as the challenge. Template transplantation is
+   therefore a strong baseline on its own.
+2. **Within our own candidate set, selection plateaued.** Three successive changes to the
+   selection rule (sections 4 and 7) each improved a leave-one-compound-out backtest, yet the
+   live BiSyRMSD pass count stayed at 133 for all three submissions.
+3. **Co-folding broke that plateau without fine-tuning.** Replacing the selected pose by the
+   co-folding representative wherever it passed PoseBusters raised the live BiSyRMSD pass
+   count from 133 to 137 and the mean BiSyRMSD from 1.84 Å to 1.70 Å. On 24 structures
+   released after the model's training cutoff, the co-folding representative was within 1.5 Å
+   for 9 ligands against 5 for Pipeline A.
+4. **Backtest gains did not transfer to the live set.** A selection rule that won 11 of 12
+   honest holdout splits by +5.2 percentage points moved the live score by +0.5 points. We
+   explain why in section 7, and we have stopped treating backtest gains alone as evidence of
+   improvement.
+
+Negative results are listed in section 9: the levers we tested on the Pipeline A side, each
+with the measurement that closed it.
+
+---
+
+## 2. Target, inputs and public data
+
+- **Construct.** NS2B cofactor (chain A, 46 residues) and NS3 protease domain (chain B,
+  168 residues) as given in `NS2B-NS3_protein_sequence.fasta`. The ligand is a single residue
+  `LIG`. Every submitted file has three chains or fewer.
+- **Reference frame.** The apo structure 29HN. 8PN6, named in the dataset README, is the same
+  construct in the same space group. All coordinates in this work are expressed in the 29HN
+  frame.
+- **Public structures.** An RCSB full-text search for Zika NS2B-NS3 returned 294 entries
+  (2026-10-23; the same query on 2026-10-28 returned no entries released after 2026-10-01).
+  From these we extracted **209 ligand instances from 176 entries (174 distinct chemical
+  components)**. Bonds and bond orders were taken from the PDB Chemical Component Dictionary
+  by atom name rather than inferred from coordinates, which recovered 43 ligands that
+  coordinate-based perception had broken.
+- **Challenge ligands.** 400 SMILES, 9 to 36 heavy atoms, median 23; 77.2% have 20 or more
+  heavy atoms.
+
+No proprietary or unpublished data was used. The challenge's own crystal structures were never
+available to us.
+
+---
+
+## 3. Pipeline A: candidate generation
+
+### 3.1 Binding-site superposition
+
+Each public structure was superposed onto 29HN using backbone atoms within 12 Å of the centre
+of the catalytic triad (Kabsch). The evaluation superposes binding sites before computing
+ligand RMSD, so a binding-site fit is the relevant one. When chain identification by triad
+numbering failed, a residue-number offset was found by sequence alignment and the fit was
+repeated. Median fit RMSD: 0.15 Å.
+
+### 3.2 Template matching and transplantation
+
+1. Morgan fingerprints shortlist templates; the MCS (rings match rings, complete rings only)
+   ranks them by the number of shared heavy atoms. Up to four templates per ligand are kept
+   (397 of 400 ligands have four).
+2. Because of molecular symmetry, an MCS admits several atom mappings. We enumerate them,
+   rigidly align each to the template, and keep the geometrically consistent ones. Using only
+   the first mapping returned by the matcher displaced poses by 4-5 Å; enumerating mappings
+   reduced the MCS RMSD to 0.17 Å.
+3. The common atoms are fixed at the template's experimental coordinates; the rest of the
+   ligand is built with ETKDGv3 and relaxed with MMFF94 in three stages (common atoms fixed,
+   then positionally restrained, then a looser restraint only if geometry is still broken).
+4. The template's own crystal structure, trimmed to the two protease chains nearest the active
+   site, is the receptor. Its side chains are already arranged around a closely related ligand.
+5. smina (Vinardo) local minimisation against that receptor removes clashes while keeping bond
+   lengths and angles as built.
+
+### 3.3 Docking and CNN rescoring
+
+- smina (Vinardo), 22 Å box on the active site, exhaustiveness 16, 9 modes, against the
+  template receptors. Docking candidates matter: removing them lowered the backtest from
+  58.1% to 48.0%, and to half in the lowest-similarity stratum.
+- gnina 1.3.2: every candidate is rescored in place (`--score_only --cnn_scoring rescore`),
+  and the 12 best-ranked candidates per ligand are additionally refined with `--minimize`
+  (median movement 0.26 Å; all refined poses kept valid geometry).
+
+### 3.4 Validity gate
+
+Every candidate considered for submission passes two checks in order:
+
+- a **PDB round trip** identical to the path of the official validator. PDB files carry no
+  bond orders, so the reader perceives bonds from coordinates; a pose with a stretched ring can
+  gain a bond and fail validation even after passing PoseBusters;
+- **PoseBusters 0.6.5**, the official `redock` configuration minus the RMSD checks
+  (22 checks), with the molecule built from the challenge SMILES in place of `mol_true`, so
+  that formula, bonds, double-bond stereochemistry and tetrahedral chirality are checked
+  against the input.
+
+---
+
+## 4. Pipeline A: pose selection
+
+### 4.1 The rule
+
+Each candidate gets a score; lower is better. Candidates are walked in score order and the
+first one that passes the validity gate is selected.
+
+| Term | Weight | Meaning |
+|---|---|---|
+| spatial consensus | 1.0 | fraction of the ligand's other candidates within 1.2 Å RMSD |
+| cross-template consensus | 2.0 | the same fraction counted only over candidates derived from *other* templates |
+| contact recovery | 1.0 | fraction of the residues commonly contacted by the template ligands (median 14-15 residues) that this pose also contacts |
+| contact Jaccard | 0.5 | Jaccard index between the pose's contact set and that reference set |
+| gnina CNNscore rank | 0.3 | rank within the ligand's candidates |
+
+Signs are set so that more consensus and more contact recovery lower the score.
+Cross-template consensus carries the most weight because templates come from independent
+crystal structures: agreement between them is evidence that does not come from the scoring
+function. The CNN term uses the within-ligand rank, because CNNscore ranges differ between
+ligands and absolute values would let a few ligands dominate the rule.
+
+### 4.2 Consensus nudge
+
+After selection and validation, the selected pose alone is moved a quarter of the way toward
+the atom-wise mean of the candidates within 3 Å, then relaxed with MMFF under a strong
+positional restraint (force constant 200, tolerance 0.05 Å) and validated again; if it fails,
+the unmoved pose is submitted. Median movement 0.14 Å, maximum 0.38 Å. In v18 this applies to
+60 of the 63 ligands that use Pipeline A.
+
+The order matters. Our first version nudged the top-ranked candidates before selection, which
+inflated the consensus terms (the median spatial consensus of selected poses rose from 0.36 to
+0.56) and reshuffled the selection. Selecting first and nudging only the selected pose keeps
+the selection unchanged.
+
+---
+
+## 5. Pipeline B: co-folding
+
+- **Model.** Protenix 2.0.0 with the `protenix_base_20250630_v1.0.0` checkpoint (training
+  cutoff 2025-06-30). Not fine-tuned.
+- **Inputs.** The two protease chains and the ligand SMILES. A single MSA, computed once with
+  the Protenix MSA server, was reused for all inputs since the protein is identical.
+- **Sampling.** Seeds 101 and 102, five diffusion samples each, ten samples per ligand
+  (five for five ligands where the second seed returned no output). About 13 seconds per input
+  and seed on one RTX 4090.
+- **Conversion.** Each predicted complex is superposed onto the 29HN binding site after a
+  sequence-derived renumbering (NS3 +9, NS2B +43). Ligand atoms are mapped to the SMILES atom
+  order, and each sample is flagged for whether its stereochemistry matches the SMILES. The
+  co-folded protein, in the 29HN frame and numbering, is kept as that sample's receptor.
+- **Representative sample.** Among samples with correct stereochemistry, the one with the
+  largest number of other samples within 2.0 Å symmetry-corrected RMSD. For the 337 ligands
+  where it was used, the median fraction of all ten samples within 2.0 Å of it was 1.0.
+- **Gate.** The representative must pass the same validity gate (section 3.4). If it fails, its
+  geometry is corrected with a strongly restrained MMFF minimisation and retried; receptors are
+  tried in the order co-folded protein, Pipeline A receptor, receptor of the nearest
+  Pipeline A candidate, 29HN.
+
+**On the training cutoff.** A 2025-06-30 cutoff covers 183 of the 209 public template
+ligands. The challenge structures themselves are unpublished. This is the same situation as the
+organisers' OpenBind-0 baseline, an OpenFold3 checkpoint with the same cutoff (64.5% on the live
+leaderboard). We therefore measured co-folding against our own pipeline on structures released
+after the cutoff (section 7.3).
+
+---
+
+## 6. The submitted assembly (v18)
+
+| Source of submitted pose | Ligands |
+|---|---|
+| Co-folding representative | 337 |
+| Pipeline A, no co-folding sample with correct stereochemistry | 39 |
+| Pipeline A, co-folding representative failed PoseBusters | 24 |
+| *of the 63 Pipeline A poses:* template transplant + smina | 39 |
+| *of the 63:* docking | 22 |
+| *of the 63:* template transplant, unminimised | 2 |
+
+- PoseBusters 22 checks: 400 / 400.
+- Official validator (`validation/structure_validation.py`, tutorial repository commit
+  `1b34ec2`): valid, 0 errors, 400 files.
+
+---
+
+## 7. Validation and what it could and could not tell us
+
+### 7.1 Backtest design
+
+We had no access to challenge structures, so every design decision on the Pipeline A side was
+measured on public complexes:
+
+- **Leave-one-compound-out.** For each of 145 public complexes (one per chemical component,
+  best resolution), every structure containing that component was removed from the template
+  library before predicting its pose. 145 is the ceiling: of the 174 components, 28 have no
+  remaining template with enough MCS overlap to produce any pose.
+- **Challenge-distribution weighting.** The backtest set is dominated by fragments (40% below
+  15 heavy atoms against 14.8% in the challenge), and leave-one-out makes templates more
+  distant than in the challenge (median best Tanimoto 0.391 against 0.520). Unweighted, the
+  backtest underestimated the first live score by 25 points. We weight backtest compounds by
+  heavy-atom and similarity strata to match the challenge.
+- **Honest holdout.** To estimate the optimism of choosing a rule on the same data used to
+  score it, we split the 145 compounds in half twelve times, re-selected every distance,
+  weight and criterion on one half, and scored on the other.
+
+### 7.2 The selection rule: backtest against live
+
+| Submission | Selection | Backtest (145, weighted) | Holdout, 12 splits | Live success | Live BiSyRMSD pass |
+|---|---|---|---|---|---|
+| v7 | gnina CNN rank only | 43.6% | 43.6% | 66.0% | 133 |
+| v11 | rule of section 4.1 | 51.7% | 48.9% (wins 11 of 12) | 66.5% | 133 |
+| v16 | v11 with a feature-reading defect fixed, plus nudge | see note | - | 66.5% | 133 |
+| **v18** | co-folding where valid, v16 elsewhere | not measurable | - | **68.5%** | **137** |
+
+Note on v16: the submission code had not been reading the template identity of each pose, so
+the cross-template term (weight 2.0) was zero in every submission from v10 to v15 while the
+rule had been fitted with it. Fixing this moved the backtest from 51.6% to 54.0% under a
+counting that holds similarity strata fixed per compound, and moved the live score by zero.
+
+Three explanations are compatible with these numbers and we cannot separate them:
+
+1. **Resolution.** The live set is about 200 compounds; one compound is 0.5 points.
+2. **Leave-one-out is harder than the challenge.** With close templates available, as in the
+   challenge, the CNN rank alone already picks well (66.0% live), and consensus has less to
+   add. The advantage of consensus appeared mainly where templates were distant.
+3. **Weighting is incomplete.** Similarity-stratum weights were capped at 4.0.
+
+After v11 we stopped treating a backtest gain alone as evidence of improvement, and we compare
+methods by submitting whole versions and reading the change in the live score. We never changed
+individual compounds to probe the live score, since the final ranking uses all 400.
+
+### 7.3 Co-folding against Pipeline A
+
+| Evaluation set | Pipeline A | Co-folding representative | Either | Both |
+|---|---|---|---|---|
+| 24 structures released after 2025-06-30 (2025-cutoff model; 17 of 24 are fragments) | 5 | **9** | 10 | 4 |
+| 145-compound backtest (2021-09-30-cutoff model `protenix_base_default_v1.0.0`, which saw 22 of the 209 templates) | **63** | 49 | 79 | 33 |
+
+Counts are ligands within 1.5 Å. The two pipelines succeed on different compounds: together they
+would reach 79 of 145. We found no signal that tells, for a given compound, which one is right
+when they disagree. Areas under the ROC curve for that decision were 0.66 for co-folding
+self-consistency, 0.60 for ligand pLDDT, 0.58 for ranking score and ipTM, and 0.48-0.54 for the
+Pipeline A consensus and contact terms. A conditional switching rule (switch only when
+self-consistency is at least 0.6 and cross-template consensus is below 0.15) went from 5 to 8 on
+the 24-structure set and from 63 to 60 on the 145-compound backtest.
+
+The 2025-cutoff model on the challenge is a different situation from either set, because it was
+trained on earlier structures from the same series. Only the live leaderboard could measure it,
+which is why v18 uses co-folding unconditionally wherever it is valid.
+
+---
+
+## 8. Leaderboard history
+
+| Version | Submitted (UTC) | Change | Success | BiSyRMSD pass | lDDT-PLI pass | Mean BiSyRMSD |
+|---|---|---|---|---|---|---|
+| v7 | 2026-10-26 06:30 | Pipeline A, gnina CNN rank selection | 66.0% | 133 | 149 | 1.84 Å |
+| v11 | 2026-10-26 18:32 | selection rule of section 4.1 | 66.5% | 133 | 150 | 1.83 Å |
+| v16 | 2026-10-28 11:48 | feature-reading fix, consensus nudge | 66.5% | 133 | 152 | 1.84 Å |
+| **v18** | 2026-10-28 23:51 | co-folding where valid | **68.5%** | **137** | **155** | **1.70 Å** |
+
+PoseBusters was 200 / 200 for every version.
+
+---
+
+## 9. What did not work
+
+All measured on the weighted 145-compound backtest unless stated. The reference is the rule of
+section 4.1 at 58.1% on the 120-compound set used at the time, or the value given in the row.
+
+**Candidate generation**
+
+| Tried | Result |
+|---|---|
+| Eight templates per ligand instead of four | 65% to 55% on the subset regenerated for this test. The ceiling stayed at 94%; candidates rose from 54 to 96 per ligand and distant templates formed spurious consensus |
+| One, two or three templates | 49.5-51.9% |
+| Filtering templates by MCS coverage | 54.1-56.1% |
+| Docking at exhaustiveness 64 and 20 modes | 65% to 58%; same ceiling, more candidates dilute the selection |
+| Removing docking candidates | 58.1% to 48.0% |
+| Boltz-2 co-folding without fine-tuning (fragment probe) | 10.7%, below Pipeline A |
+
+**Selection features**
+
+| Tried | Result |
+|---|---|
+| Rescoring on several receptors (mean, minimum, affinity, VS score) | 46.7-55.5% |
+| Burial of ligand atoms | 46.0% |
+| Distance to the template-ligand hotspot | 43.1% |
+| Hydrogen bonds, salt bridges, hydrophobic contacts (geometric definitions) | 41.4-51.1% |
+| Ensemble score spread, template resolution, transplant shift, MCS RMSD | all harmful |
+| Learned models on 32 features (logistic regression, gradient boosting, random forest) | 45.7-53.1% |
+| Pairwise ranking with learned linear weights | 47.2% (learned order matched our rule) |
+
+The physical and chemical terms separate correct from incorrect poses at the pose level (for
+example 4 against 3 hydrogen bonds, 73 against 50 hydrophobic contacts) but lower the success
+rate when added to the rule. Favouring more contacts selects over-buried poses; we read this as
+the balance already being encoded in the gnina CNN.
+
+**Selection procedure**
+
+| Tried | Result |
+|---|---|
+| Weighting consensus by CNN score or contact recovery | 56.1-57.0% (plain counts are better) |
+| Two-step selection (site first, then representative) | 47.4-50.0% |
+| Medoid of the top candidates | 50.9-56.8% |
+| Cross-template consensus counted over template families | 53.1-54.8% |
+| Docking poses first for fragments (fewer than 18 heavy atoms) | 54.0% to 51.8% after the v16 fix |
+| Co-folding samples as additional consensus votes (weights 0.5-8, radii 1.5-3 Å; 2021-cutoff model) | 53-59 of 145 ligands against 61 |
+
+**Fragments remain the hardest class.** In our early 120-compound backtest, compounds below
+18 heavy atoms succeeded at 9-29% against 50-71% above, although the best candidate is often within 0.3-0.8 Å. A
+fragment has few contacts, and the same contacts are reproduced at several positions in the
+pocket. This matches the live gap between lDDT-PLI passes (155) and BiSyRMSD passes (137).
+
+---
+
+## 10. Reproducibility and scope
+
+| Component | Version |
+|---|---|
+| RDKit | 2026.09.1 |
+| PoseBusters | 0.6.5 (same as the organisers' baseline reports) |
+| smina | 2017.11.9 (Vinardo scoring) |
+| gnina | 1.3.2 static release (`master:f23dd2b`) |
+| Protenix | 2.0.0, checkpoint `protenix_base_20250630_v1.0.0` |
+| Official validator and scoring code | `OpenADMET/OpenBind-Challenge-1-Tutorial` at commit `1b34ec2` |
+
+**Compute.** Template transplantation, docking and selection ran on CPU nodes; gnina rescoring
+and refinement on an 8 GB RTX 4060. Co-folding ran on one rented RTX 4090 for about eight hours,
+at a cost of about USD 4.
+
+**Code.** The code is not published at this time. This report is intended to describe every
+step at a level that allows reimplementation with the public tools listed above. Parameters
+not stated here were left at the tools' defaults.
+
+**Data use.** Only public PDB entries, the challenge inputs and public model checkpoints were
+used. No test-set structure, label or per-compound score entered any step. The live leaderboard
+was used only to compare whole submissions.
+
+---
+
+## 11. Contact
+
+KR LAB Bio Inc., Republic of Korea. Leaderboard alias: KR LAB Bio.