AUDIT.md

MitoInteract v1 Failure Audit

Date: 2026-07-19

Verdict

The published checkpoint is a failed proof of concept, not a scientifically usable binding-affinity model. It should not be used to rank compounds or interpreted as predicting pKd/Kd.

Evidence from the published artifacts

Finding	Published evidence	Impact
Training set was only 32 pairs	config.json: max_train_samples: 32	Cannot support the claim that the model learned from the 1.9M-pair dataset
Validation set was only 16 pairs	max_val_samples: 16	Correlation metrics are unstable and not representative
Reported correlation is negative	Checkpoint: Pearson -0.9107416, Spearman -0.5	Predictions are directionally wrong on the tiny validation set
Regression error is enormous	Checkpoint: RMSE 7.0257, MAE 6.8183	Error is roughly the scale of the entire reported target range
Training barely began	32 samples / batch 4 × 2 epochs = 16 steps, but warmup is 500 steps	The run completed only 3.2% of warmup; approximate final LR was 3.2e-05, not the configured 1e-3
Model card contradicts checkpoint	README says ESM-2 650M; config/checkpoint use ESM-2 35M	Architecture and reproducibility claims are incorrect
Target semantics are undocumented	Dataset has affinity_uM, neg_log10_affinity_M, and normalized affinity; config names none	Published inference labels raw output as pKd without proving it was trained on pKd
Source target conflates measurement types	The pinned BindingDB notebook combines available Ki, Kd, IC50, and EC50 values and averages them in linear µM	The dataset target is generic pAffinity, not pKd; assay provenance must be retained in any scientific successor
Evaluation split is undocumented	No training/evaluation code is published	Protein/ligand leakage cannot be ruled out
Reproduction assets are missing	No training script, requirements, seed, split manifest, predictions, or metric calculation	Results cannot be independently reproduced
Serialization is unsafe and oversized	full_model.pt is a 315.7 MB pickle checkpoint loaded with weights_only=False	Avoidable code-execution risk and duplicated frozen encoders

The separate mitointeract_weights.pt file contains 1,278,980 trainable head parameters and is about 5.1 MB. Future releases should use safetensors, explicit base-model revisions, and a complete evaluation report.

Recovery gates

MitoInteract v2 does not earn a model-card performance claim until all gates pass:

1. Unit semantics: the primary benchmark is Kd-only pKd; mixed-measurement data remains labeled generic pAffinity and is not converted to Kd.
2. Pair curation: canonicalize ligands, group exact protein-ligand pairs, retain replicate statistics, and predeclare aggregation.
3. Overfit sanity check: the trainable head can overfit a fixed 32-example set to near-zero training error.
4. Real validation: use at least 1,000 validation and test examples with immutable split manifests.
5. Leakage-aware evaluation: report random-pair, homology-cold protein, ligand-similarity-cold, and cold-both results separately.
6. Baseline win: the paired neural model must beat the training mean, descriptor baseline, and both single-modality controls on strict splits.
7. Uncertainty: report grouped-bootstrap confidence intervals, micro metrics, macro-target metrics, and repeated split/training seeds.
8. Reproducibility: publish code, dependency lock, seeds, model/dataset revisions, predictions, and metrics.
9. Scientific restraint: describe outputs as computational estimates requiring experimental validation. Do not claim applicability to “any protein and any molecule.”

Recovery evidence as of 2026-07-19

Primary source-aware BindingDB data track

The primary data-quality source is the BindingDB 202607 curated-articles export, joined to the 202607 reactant-set/assay map and assay-description export. All three ZIP files are pinned by SHA-256 in recovery/config-bindingdb.json and attributed under CC BY 3.0.

The checked full pilot processes 93,712 source rows into 87,315 explicit Ki/IC50/Kd/EC50 measurement records. Its strict projection retains 2,432 exact, assay-joined Kd source records over 2,198 canonical protein–ligand pairs, 442 proteins, and 245 assays. It preserves censoring, source row, release, assay, citation, sequence, and ligand provenance. The 2,432 records are not aggregated across assays and have not been split or used for model training.

The audit accounts for every source row, preserves 695 source records with incomplete assay joins outside the gold projection, and disambiguates 122 repeated BindingDB reactant-set identifiers using physical source-row ordinals. Full schemas, hashes, commands, and limitations are in recovery/BINDINGDB_SOURCE_AWARE.md and recovery/reports/bindingdb-source-aware-audit.json.

Prior derived Kd-only benchmark

The prior derived source is BALM/BALM-benchmark, configuration BindingDB_filtered, at revision 84e07e27a093181ee89c97a0d7a45482803aa9bd. The source CSV was verified against SHA-256 d0f8691a96876f09766e574c5600df55f0d0b1c11cfd1705faf841d924f89f20.

Canonicalization and median exact-pair aggregation reduce 24,700 source rows to 20,477 pairs. The source contains 2,020 duplicated canonical pairs, 2,019 with differing labels, with a maximum within-pair range of 6.3166 pKd units. The released file omits raw Kd values, units, relation operators, and assay identifiers, so Kd-only provenance cannot be reconstructed independently from the CSV.

Gate	Result
Python test suite	passed, 201 tests in 23.30 seconds with optional model dependencies installed
MMseqs2 protein split	1,069 proteins, 739 clusters, zero overlap at 50% identity and 80% coverage
Random-pair descriptor model	RMSE 0.8452, Pearson 0.8196, Spearman 0.7846
Exact protein-disjoint descriptor model	RMSE 1.2276, Pearson 0.6014, Spearman 0.5493
Homology-cold descriptor model	RMSE 1.3181, Pearson 0.3881, Spearman 0.3069
Scaffold-disjoint descriptor model	RMSE 1.0103, Pearson 0.7095, Spearman 0.6501
Full Kd neural benchmark	not run; ligand-similarity, cold-both, confidence-interval, and repeated-seed gates remain

The combined Morgan/protein-descriptor LightGBM model beats the train mean and both single-modality descriptor controls by RMSE on every current manifest. It is now the minimum baseline for the neural model.

Source-aware exact-Kd model evaluation (2026-07-20)

A frozen-encoder compact head (ESM-2 35M + ChemBERTa-77M, target standardized on train only) was trained and evaluated on the source-aware BindingDB exact-Kd slice (2,327 observations, 2,198 unique pairs) across six leakage-resistant manifests, including a protein-cluster × scaffold connected-component cold_both split. Full results are in recovery/MODEL_EVALUATION_V2.md and recovery/reports/bindingdb-v2-model-evaluation.json.

Gate	Result
Python test suite	passed, 222 tests with optional model dependencies installed
Head beats best control, cold_scaffold	yes, RMSE 1.126 vs 1.235
Head beats best control, protein_similarity_50	yes, RMSE 1.677 vs 1.832
Head beats best control, publication_time	yes, RMSE 1.309 vs 1.458
Head beats best control, cold_protein_exact	no, RMSE 1.432 vs LightGBM 1.170
Head beats best control, random_pair	no, RMSE 0.754 vs LightGBM 0.682
Head vs cold_both	degenerate, R2 -2.03 (near/below predict-the-mean)
Publish new checkpoint	no, evidence does not support it

Every split except the degenerate cold_both case has a positive held-out Pearson correlation, so the model is categorically better than the failed v1 artifact on five of six regimes. It is not yet better than the best cheap descriptor control on the most realistic novelty tests, so no new checkpoint is published.

A follow-up BALM auxiliary-data gate excluded all overlapping BindingDB pairs and held-out exact proteins, leaving 19,081 training-only pairs. The best cold-protein descriptor RMSE worsened from 1.170 (BindingDB only) to 1.364 with all eligible BALM pairs, 1.244 with exact-ligand matching, and 1.348 with training-scaffold matching. The auxiliary embedding run was therefore rejected before expensive feature generation; see recovery/BALM_AUXILIARY_GATE.md.

Joint fine-tuning gate (2026-07-21)

A bounded NVIDIA L4 experiment fine-tuned the final ESM-2 and ChemBERTa blocks plus the interaction head on the unchanged cold_protein_exact manifest. Validation-only selection chose epoch 3, early stopping fired after epoch 6, and the test split was evaluated exactly once.

Model	Test RMSE	Test R2	Outcome
Ligand Morgan/descriptors LightGBM	1.1702	0.5546	best control
Joint last-block fine-tune	1.3473	0.4096	loses control
Frozen neural head	1.4324	0.3326	worse neural variant

Joint fine-tuning improved RMSE by 5.95% over the frozen head but remained 15.13% worse than LightGBM. The 4,602,993-parameter safetensors checkpoint was hash and structure verified, retained outside Git, and rejected for publication. Additional paid split runs for this configuration were stopped at the primary gate. Full provenance and limitations are in recovery/JOINT_FINETUNE_GATE.md and recovery/reports/bindingdb-v2-joint-finetune-cold-protein.json.

Project Chimera development gate (2026-07-21)

A CPU-only residual-fusion experiment used the original cold-protein train and validation rows while excluding and masking all 270 test observations. Five outer folds grouped 2,057 observations by 210 MMseqs2 50/80 protein clusters. Ligand LightGBM predictions for residual training were themselves cross-fitted by protein cluster.

The frozen protein residual improved OOF RMSE from 1.7664 to 1.6279 and improved four of five folds. Its 2,000-replicate protein-cluster bootstrap interval for RMSE improvement was [-0.0173, 0.2728], so the predeclared positive-lower-bound requirement failed. No GPU confirmation or test evaluation was authorized. See recovery/CHIMERA_GATE.md and recovery/reports/bindingdb-v2-chimera-development-gate.json.

Source Atlas transfer gate (2026-07-21)

The exact-only relational Source Atlas retained 75,331 assay-joined records while preserving Kd, Ki, IC50, and EC50 as separate tasks. Joint 50/80 MMseqs2 clustering across 2,061 proteins revealed and reserved 34 clusters containing benchmark test proteins; this removed 292 homologous development observations and left 1,765 observations for five-fold grouped evaluation.

A fixed, task-balanced shared-plus-task-specific Ridge model increased development RMSE from 1.7202 for the Kd-only control to 1.9369. All five folds worsened, and the cluster-bootstrap interval for improvement was [-0.3441, -0.0893]. Neural multi-task pretraining is not justified by this result. No test evaluation, GPU run, or checkpoint occurred. See recovery/SOURCE_ATLAS_GATE.md and the checked Source Atlas reports.

Independent external holdout (2026-07-21)

GtoPdb release 2026.2 was pinned by embedded release metadata and source hashes. A strict adapter retained only exact small-molecule Kd rows with consistent pKd/nM values, one reviewed-UniProt target, PubMed provenance, and assay description. Removing every PubMed or canonical UniProt/SMILES pair overlap with all 87,315 BindingDB source records left 258 observations and 239 pairs.

All 161 target accessions were pinned to reviewed Swiss-Prot sequences from UniProt 2026_02. Joint 50/80 clustering against the complete 2,061-protein Source Atlas and scaffold filtering against all 40,477 Atlas ligands froze a cold-both external holdout with 103 observations, 96 pairs, 78 proteins, 78 scaffolds, and 77 publications. Affinity labels are isolated from the identifier manifest. No predictions have been generated. The holdout is sealed until one frozen candidate passes all internal development gates. See recovery/GTOPDB_EXTERNAL_HOLDOUT.md and recovery/reports/gtopdb-2026.2-external-holdout.json.

Pocket Atlas development gate (2026-07-21)

Pocket Atlas mapped 148 homology-safe development proteins to exact or at least 80%-coverage reviewed UniProt sequences and downloaded 145 canonical, versioned AlphaFold monomer structures. Pocketeer extracted 64 confidence, shape, contact-density, and top-pocket descriptors without failure. The checked acquisition workflow reproduced the complete mapping and structure manifest byte-for-byte.

On 1,765 development observations outside every reserved test homology cluster, ligand LightGBM achieved RMSE 1.6682, the ESM residual achieved 1.6435, and the best structure-containing residual achieved 1.6643. Structure therefore worsened RMSE by 0.0208 relative to ESM; the incremental cluster-bootstrap interval was [-0.0481, 0.0083], improvement probability was 0.0905, and one fold regressed by 0.3835. The predeclared structure gate rejected further training. No benchmark-test or external-holdout evaluation, GPU run, checkpoint, or weight publication occurred. See recovery/POCKET_ATLAS_GATE.md and recovery/reports/bindingdb-v2-pocket-atlas-development-gate.json.

Mixed-pAffinity forensic track

The deterministic 10,000-pair diagnostic sample is backed by jglaser/binding_affinity revision 11e49b7ece33d62afd7f65bc05ce60ad37f9ba7b and Parquet SHA-256 d50f4cae794298f32c39f75a9e8603528d20e50e4fa757025def412f5b2a78d8.

Gate	Result
pAffinity/µM numeric conversion	passed, while preserving the mixed-measurement warning
32-pair synthetic-embedding overfit	passed
32-pair pinned-encoder overfit	passed, RMSE below 1e-6 pAffinity
MMseqs2 50% similarity-disjoint combined ridge	RMSE 1.4349, Pearson 0.3625, Spearman 0.3652

This track established that the pipeline works and exposed dataset shortcut risk. It is not the primary pKd experiment, and a full neural run over its scientifically conflated target is not warranted.