AUDIT.md

# MitoInteract v1 Failure Audit

Date: 2026-07-19

## Verdict

The published checkpoint is a failed proof of concept, not a scientifically usable binding-affinity model. It should not be used to rank compounds or interpreted as predicting pKd/Kd.

## Evidence from the published artifacts

| Finding | Published evidence | Impact |
|---|---|---|
| Training set was only 32 pairs | `config.json`: `max_train_samples: 32` | Cannot support the claim that the model learned from the 1.9M-pair dataset |
| Validation set was only 16 pairs | `max_val_samples: 16` | Correlation metrics are unstable and not representative |
| Reported correlation is negative | Checkpoint: Pearson `-0.9107416`, Spearman `-0.5` | Predictions are directionally wrong on the tiny validation set |
| Regression error is enormous | Checkpoint: RMSE `7.0257`, MAE `6.8183` | Error is roughly the scale of the entire reported target range |
| Training barely began | 32 samples / batch 4 × 2 epochs = 16 steps, but warmup is 500 steps | The run completed only 3.2% of warmup; approximate final LR was `3.2e-05`, not the configured `1e-3` |
| Model card contradicts checkpoint | README says ESM-2 650M; config/checkpoint use ESM-2 35M | Architecture and reproducibility claims are incorrect |
| Target semantics are undocumented | Dataset has `affinity_uM`, `neg_log10_affinity_M`, and normalized `affinity`; config names none | Published inference labels raw output as pKd without proving it was trained on pKd |
| Source target conflates measurement types | The pinned BindingDB notebook combines available `Ki`, `Kd`, `IC50`, and `EC50` values and averages them in linear µM | The dataset target is generic pAffinity, not pKd; assay provenance must be retained in any scientific successor |
| Evaluation split is undocumented | No training/evaluation code is published | Protein/ligand leakage cannot be ruled out |
| Reproduction assets are missing | No training script, requirements, seed, split manifest, predictions, or metric calculation | Results cannot be independently reproduced |
| Serialization is unsafe and oversized | `full_model.pt` is a 315.7 MB pickle checkpoint loaded with `weights_only=False` | Avoidable code-execution risk and duplicated frozen encoders |

The separate `mitointeract_weights.pt` file contains 1,278,980 trainable head parameters and is about 5.1 MB. Future releases should use `safetensors`, explicit base-model revisions, and a complete evaluation report.

## Recovery gates

MitoInteract v2 does not earn a model-card performance claim until all gates pass:

1. **Unit semantics:** the primary benchmark is Kd-only pKd; mixed-measurement data remains labeled generic pAffinity and is not converted to Kd.
2. **Pair curation:** canonicalize ligands, group exact protein-ligand pairs, retain replicate statistics, and predeclare aggregation.
3. **Overfit sanity check:** the trainable head can overfit a fixed 32-example set to near-zero training error.
4. **Real validation:** use at least 1,000 validation and test examples with immutable split manifests.
5. **Leakage-aware evaluation:** report random-pair, homology-cold protein, ligand-similarity-cold, and cold-both results separately.
6. **Baseline win:** the paired neural model must beat the training mean, descriptor baseline, and both single-modality controls on strict splits.
7. **Uncertainty:** report grouped-bootstrap confidence intervals, micro metrics, macro-target metrics, and repeated split/training seeds.
8. **Reproducibility:** publish code, dependency lock, seeds, model/dataset revisions, predictions, and metrics.
9. **Scientific restraint:** describe outputs as computational estimates requiring experimental validation. Do not claim applicability to “any protein and any molecule.”

## Recovery evidence as of 2026-07-19

### Primary source-aware BindingDB data track

The primary data-quality source is the BindingDB 202607 curated-articles export, joined to the 202607 reactant-set/assay map and assay-description export. All three ZIP files are pinned by SHA-256 in `recovery/config-bindingdb.json` and attributed under CC BY 3.0.

The checked full pilot processes 93,712 source rows into 87,315 explicit Ki/IC50/Kd/EC50 measurement records. Its strict projection retains 2,432 exact, assay-joined Kd source records over 2,198 canonical protein–ligand pairs, 442 proteins, and 245 assays. It preserves censoring, source row, release, assay, citation, sequence, and ligand provenance. The 2,432 records are not aggregated across assays and have not been split or used for model training.

The audit accounts for every source row, preserves 695 source records with incomplete assay joins outside the gold projection, and disambiguates 122 repeated BindingDB reactant-set identifiers using physical source-row ordinals. Full schemas, hashes, commands, and limitations are in `recovery/BINDINGDB_SOURCE_AWARE.md` and `recovery/reports/bindingdb-source-aware-audit.json`.

### Prior derived Kd-only benchmark

The prior derived source is `BALM/BALM-benchmark`, configuration `BindingDB_filtered`, at revision `84e07e27a093181ee89c97a0d7a45482803aa9bd`. The source CSV was verified against SHA-256 `d0f8691a96876f09766e574c5600df55f0d0b1c11cfd1705faf841d924f89f20`.

Canonicalization and median exact-pair aggregation reduce 24,700 source rows to 20,477 pairs. The source contains 2,020 duplicated canonical pairs, 2,019 with differing labels, with a maximum within-pair range of 6.3166 pKd units. The released file omits raw Kd values, units, relation operators, and assay identifiers, so Kd-only provenance cannot be reconstructed independently from the CSV.

| Gate | Result |
|---|---|
| Python test suite | passed, 201 tests in 23.30 seconds with optional model dependencies installed |
| MMseqs2 protein split | 1,069 proteins, 739 clusters, zero overlap at 50% identity and 80% coverage |
| Random-pair descriptor model | RMSE `0.8452`, Pearson `0.8196`, Spearman `0.7846` |
| Exact protein-disjoint descriptor model | RMSE `1.2276`, Pearson `0.6014`, Spearman `0.5493` |
| Homology-cold descriptor model | RMSE `1.3181`, Pearson `0.3881`, Spearman `0.3069` |
| Scaffold-disjoint descriptor model | RMSE `1.0103`, Pearson `0.7095`, Spearman `0.6501` |
| Full Kd neural benchmark | not run; ligand-similarity, cold-both, confidence-interval, and repeated-seed gates remain |

The combined Morgan/protein-descriptor LightGBM model beats the train mean and both single-modality descriptor controls by RMSE on every current manifest. It is now the minimum baseline for the neural model.

### Source-aware exact-Kd model evaluation (2026-07-20)

A frozen-encoder compact head (ESM-2 35M + ChemBERTa-77M, target standardized on train only) was trained and evaluated on the source-aware BindingDB exact-Kd slice (2,327 observations, 2,198 unique pairs) across six leakage-resistant manifests, including a protein-cluster × scaffold connected-component `cold_both` split. Full results are in `recovery/MODEL_EVALUATION_V2.md` and `recovery/reports/bindingdb-v2-model-evaluation.json`.

| Gate | Result |
|---|---|
| Python test suite | passed, 222 tests with optional model dependencies installed |
| Head beats best control, cold_scaffold | yes, RMSE `1.126` vs `1.235` |
| Head beats best control, protein_similarity_50 | yes, RMSE `1.677` vs `1.832` |
| Head beats best control, publication_time | yes, RMSE `1.309` vs `1.458` |
| Head beats best control, cold_protein_exact | **no**, RMSE `1.432` vs LightGBM `1.170` |
| Head beats best control, random_pair | **no**, RMSE `0.754` vs LightGBM `0.682` |
| Head vs cold_both | **degenerate**, R2 `-2.03` (near/below predict-the-mean) |
| Publish new checkpoint | **no**, evidence does not support it |

Every split except the degenerate `cold_both` case has a positive held-out Pearson correlation, so the model is categorically better than the failed v1 artifact on five of six regimes. It is not yet better than the best cheap descriptor control on the most realistic novelty tests, so no new checkpoint is published.

A follow-up BALM auxiliary-data gate excluded all overlapping BindingDB pairs and held-out exact proteins, leaving 19,081 training-only pairs. The best cold-protein descriptor RMSE worsened from `1.170` (BindingDB only) to `1.364` with all eligible BALM pairs, `1.244` with exact-ligand matching, and `1.348` with training-scaffold matching. The auxiliary embedding run was therefore rejected before expensive feature generation; see `recovery/BALM_AUXILIARY_GATE.md`.

### Joint fine-tuning gate (2026-07-21)

A bounded NVIDIA L4 experiment fine-tuned the final ESM-2 and ChemBERTa blocks plus the interaction head on the unchanged `cold_protein_exact` manifest. Validation-only selection chose epoch 3, early stopping fired after epoch 6, and the test split was evaluated exactly once.

| Model | Test RMSE | Test R2 | Outcome |
|---|---:|---:|---|
| Ligand Morgan/descriptors LightGBM | `1.1702` | `0.5546` | best control |
| Joint last-block fine-tune | `1.3473` | `0.4096` | loses control |
| Frozen neural head | `1.4324` | `0.3326` | worse neural variant |

Joint fine-tuning improved RMSE by `5.95%` over the frozen head but remained `15.13%` worse than LightGBM. The 4,602,993-parameter safetensors checkpoint was hash and structure verified, retained outside Git, and rejected for publication. Additional paid split runs for this configuration were stopped at the primary gate. Full provenance and limitations are in `recovery/JOINT_FINETUNE_GATE.md` and `recovery/reports/bindingdb-v2-joint-finetune-cold-protein.json`.

### Project Chimera development gate (2026-07-21)

A CPU-only residual-fusion experiment used the original cold-protein train and validation rows while excluding and masking all 270 test observations. Five outer folds grouped 2,057 observations by 210 MMseqs2 50/80 protein clusters. Ligand LightGBM predictions for residual training were themselves cross-fitted by protein cluster.

The frozen protein residual improved OOF RMSE from `1.7664` to `1.6279` and improved four of five folds. Its 2,000-replicate protein-cluster bootstrap interval for RMSE improvement was `[-0.0173, 0.2728]`, so the predeclared positive-lower-bound requirement failed. No GPU confirmation or test evaluation was authorized. See `recovery/CHIMERA_GATE.md` and `recovery/reports/bindingdb-v2-chimera-development-gate.json`.

### Source Atlas transfer gate (2026-07-21)

The exact-only relational Source Atlas retained 75,331 assay-joined records while preserving Kd, Ki, IC50, and EC50 as separate tasks. Joint 50/80 MMseqs2 clustering across 2,061 proteins revealed and reserved 34 clusters containing benchmark test proteins; this removed 292 homologous development observations and left 1,765 observations for five-fold grouped evaluation.

A fixed, task-balanced shared-plus-task-specific Ridge model increased development RMSE from `1.7202` for the Kd-only control to `1.9369`. All five folds worsened, and the cluster-bootstrap interval for improvement was `[-0.3441, -0.0893]`. Neural multi-task pretraining is not justified by this result. No test evaluation, GPU run, or checkpoint occurred. See `recovery/SOURCE_ATLAS_GATE.md` and the checked Source Atlas reports.

### Independent external holdout (2026-07-21)

GtoPdb release 2026.2 was pinned by embedded release metadata and source hashes. A strict adapter retained only exact small-molecule Kd rows with consistent pKd/nM values, one reviewed-UniProt target, PubMed provenance, and assay description. Removing every PubMed or canonical UniProt/SMILES pair overlap with all 87,315 BindingDB source records left 258 observations and 239 pairs.

All 161 target accessions were pinned to reviewed Swiss-Prot sequences from UniProt 2026_02. Joint 50/80 clustering against the complete 2,061-protein Source Atlas and scaffold filtering against all 40,477 Atlas ligands froze a cold-both external holdout with 103 observations, 96 pairs, 78 proteins, 78 scaffolds, and 77 publications. Affinity labels are isolated from the identifier manifest. No predictions have been generated. The holdout is sealed until one frozen candidate passes all internal development gates. See `recovery/GTOPDB_EXTERNAL_HOLDOUT.md` and `recovery/reports/gtopdb-2026.2-external-holdout.json`.

### Pocket Atlas development gate (2026-07-21)

Pocket Atlas mapped 148 homology-safe development proteins to exact or at least 80%-coverage reviewed UniProt sequences and downloaded 145 canonical, versioned AlphaFold monomer structures. Pocketeer extracted 64 confidence, shape, contact-density, and top-pocket descriptors without failure. The checked acquisition workflow reproduced the complete mapping and structure manifest byte-for-byte.

On 1,765 development observations outside every reserved test homology cluster, ligand LightGBM achieved RMSE `1.6682`, the ESM residual achieved `1.6435`, and the best structure-containing residual achieved `1.6643`. Structure therefore worsened RMSE by `0.0208` relative to ESM; the incremental cluster-bootstrap interval was `[-0.0481, 0.0083]`, improvement probability was `0.0905`, and one fold regressed by `0.3835`. The predeclared structure gate rejected further training. No benchmark-test or external-holdout evaluation, GPU run, checkpoint, or weight publication occurred. See `recovery/POCKET_ATLAS_GATE.md` and `recovery/reports/bindingdb-v2-pocket-atlas-development-gate.json`.

### Mixed-pAffinity forensic track

The deterministic 10,000-pair diagnostic sample is backed by `jglaser/binding_affinity` revision `11e49b7ece33d62afd7f65bc05ce60ad37f9ba7b` and Parquet SHA-256 `d50f4cae794298f32c39f75a9e8603528d20e50e4fa757025def412f5b2a78d8`.

| Gate | Result |
|---|---|
| pAffinity/µM numeric conversion | passed, while preserving the mixed-measurement warning |
| 32-pair synthetic-embedding overfit | passed |
| 32-pair pinned-encoder overfit | passed, RMSE below `1e-6` pAffinity |
| MMseqs2 50% similarity-disjoint combined ridge | RMSE `1.4349`, Pearson `0.3625`, Spearman `0.3652` |

This track established that the pipeline works and exposed dataset shortcut risk. It is not the primary pKd experiment, and a full neural run over its scientifically conflated target is not warranted.