BGC retrieval rebuild

This directory is an isolated, leakage-free replacement for the legacy experiment. It reads legacy inputs but never rewrites them. The primary development task is within-species retrieval of held-out MIBiG-reference groups in the Streptomyces griseus atlas. Curated within-species product labels are still required for a publication-grade biological claim.

Scientific invariants

Model inputs are restricted to frozen ESM-2 gene embeddings and relative gene position. Alignment statistics and upstream product/activity predictions are forbidden.
Product/reference groups are split before pairs or batches are constructed. Test groups are excluded from pretraining, supervised training, tuning, and checkpoint selection.
Atlas-to-MIBiG assignments are silver training labels. Curated S. griseus BGC–product mappings are required for the primary biological result; broad multi-genus benchmarking is secondary stress evidence only.
Ensemble weights are selected on validation once. Tied baseline scores use expected metrics under uniform ordering within each tie group.
Every checkpoint and metric file records configuration, split, input, code, and environment fingerprints.

Layout and workflow

configs/main.yaml: canonical one-GPU campaign configuration.
src/bgc_retrieval: data contracts, model, training, evaluation, and audit CLI.
data/manifests: tracked hashes, split assignments, and exclusion reports.
artifacts/<run_id>: immutable run outputs; never overwrite an existing run.

From this directory, create a modern environment and install the package:

python -m venv .venv
.venv/bin/pip install -e '.[dev]'
bgc-rebuild audit --config configs/main.yaml
bgc-rebuild build-splits --config configs/main.yaml
bgc-rebuild train --config configs/main.yaml --stage phase1
bgc-rebuild train --config configs/main.yaml --stage phase2
bgc-rebuild evaluate --config configs/main.yaml

The legacy CSV has MIBiG-reference assignments but no curated product identity table. build-splits therefore creates a silver development split by reference ID and refuses to call it product-level. The primary development endpoint is held-out, family-aggregated Recall@50 within the S. griseus atlas. The staged multi-genus benchmark measures out-of-distribution correlation between BGC embedding similarity and product-structure Tanimoto; it is not the paper's species-scoped primary endpoint. Exact-product retrieval remains secondary because most exact-product groups are singletons.

Prepare and run the secondary stress endpoint with:

python scripts/prepare_external_benchmark.py
python scripts/embed_external_proteins.py --resume
python scripts/evaluate_external.py \
  --checkpoint artifacts/<training-run>/phase2_best.pt \
  --run-id <external-evaluation-run>

scripts/dgx_campaign.sh CAMPAIGN_TAG runs the complete internal and external campaign on a CUDA host. It reuses the host's CUDA-enabled PyTorch installation, refuses to overwrite immutable run directories, and accepts EXTERNAL_BOOTSTRAP_SAMPLES for a shorter smoke campaign. The default is 1,000.

2026-08-11 scope correction and residual redesign report

This section is the authoritative status for the rebuild and supersedes any earlier wording that treats the multi-genus benchmark as the primary paper endpoint.

Authoritative biological scope

The project is restricted to within-species retrieval in Streptomyces griseus: given known members of a held-out compound/reference family, rank other BGCs from the S. griseus atlas. Generalization to other species or genera is not a primary claim.

The legacy atlas contains 6,953 BGCs from 182 deduplicated genome accessions. Of those BGCs, 5,325 have silver MIBiG-reference assignments used by the rebuild. The frozen group-disjoint split contains:

Split	BGCs	reference groups
train	4,211	98
validation	676	21
test	438	21

Only 16 test groups contain more than the four required reference BGCs and therefore contribute to the reported retrieval summaries. The atlas itself has a genome_id column but no organism/species column. Its S. griseus identity is currently a dataset-provenance assertion, not a machine-verifiable row-level contract. A taxonomy manifest for all 182 assemblies is required before submission.

Why the previous analysis went wrong

The scientific task drifted. The rebuild promoted the released BGC-clustering benchmark to the primary endpoint even though the paper studies only S. griseus. That external sequence set contains 1,343 BGCs spanning hundreds of taxa: the unambiguous metadata contain 239 genera, and only 11 sequence records mention S. griseus. The resulting 1,270-BGC evaluation was principally an out-of-distribution, multi-genus test. It is useful as a stress test but does not answer the paper's within-species question.
The labels do not constitute gold product identity. The atlas compound_family values were produced from MIBiG alignment evidence. The rebuild correctly calls them silver mibig_reference_id groups, but the legacy manuscript sometimes described them as known compound families or exact products. The leakage-free S. griseus result is therefore development evidence and is marked publication_eligible: false.
The original encoder destroyed a strong baseline representation. SetNet projected every 1,280-dimensional ESM gene vector into 256 dimensions, passed it through two induced-attention blocks, and pooled to another 256-dimensional vector. It had 2,272,512 trainable parameters and no raw-ESM residual path. This was excessive for only 98 training groups and allowed optimization to overwrite useful pretrained geometry.
The contrastive target was mis-specified. Supervised contrastive loss treated every different silver group as an equally strong negative. Related metabolites and noisy MIBiG assignments can therefore become false negatives. The loss optimized discrete silver-group separation, not the biological notion of alternative routes to the same or related chemistry.
Checkpoint selection and the broad stress endpoint were misaligned. Checkpoints were selected by silver validation Recall@50, while the promoted external endpoint was Spearman correlation with continuous product-structure Tanimoto similarity. Across the five original SetNet seeds, better validation recall did not reliably imply better external structural correlation.
Legacy model selection overstated evidence. Earlier work selected ensemble weights on the evaluated data, defined "hard" cases using weakness of the Pfam baseline itself, and treated many repeated retrieval draws as if they were independent observations. The rebuild fixed these issues with validation-only selection, expected tie-aware metrics, family aggregation, paired family-level tests, and Holm correction.
The effective test size is small. Although the atlas contains thousands of BGCs, uncertainty is governed by 16 eligible held-out groups, not by BGC-pair counts or 100 repeated draws per group. This produces wide intervals and makes small metric changes unconvincing.

The prior multi-genus five-seed result remains recorded as a secondary out-of-distribution stress test: SetNet Spearman was 0.088884 ± 0.023216, raw ESM was 0.114198, and BiG-SCAPE was 0.261492. Those numbers must not be used to accept or reject the S. griseus-scoped paper.

Residual Gene Weighting v1

The executed redesign uses only frozen ESM embeddings and relative gene position:

a 397,057-parameter gene-importance gate, 5.7 times smaller than SetNet;
zero-initialized gate logits, making the starting representation exactly raw ESM mean pooling;
weighted pooling in the original 1,280-dimensional ESM space instead of a destructive 256-dimensional bottleneck;
soft Pfam/family relational distillation so related non-identical groups are not automatically hard negatives;
a frozen raw-ESM residual score with validation candidates including alpha=0;
a Pfam hybrid grid including pure Pfam;
checkpoint and mixture selection on validation only;
five deterministic seeds plus a five-checkpoint score ensemble.

The residual construction guarantees:

cosine(residual_i, residual_j)
  = (1 - alpha) * cosine(raw_i, raw_j)
  + alpha       * cosine(weighted_i, weighted_j)

Thus alpha=0 reproduces raw ESM exactly.

Executed DGX result

All 26 tests passed before execution and again after artifact retrieval using the DGX environment's standard-library test runner. Training seeds 20260810–20260814 each early-stopped after 13 epochs. Every seed selected epoch 0 as its best checkpoint; best validation Recall@50 was approximately 0.4656–0.4658, while later training reduced it to 0.4068–0.4346.

All five individual evaluations and the five-checkpoint ensemble independently selected:

learned residual weight: alpha=0.0;
hybrid learned-branch weight: alpha=0.0;
raw-ESM weight within the raw/Pfam hybrid: beta=0.4.

Validation therefore rejected the trained gene-weighting branch. The fixed S. griseus test results were:

Method	Recall@50	MRR	MAP	nDCG@50
raw ESM mean	0.792383	0.253128	0.724006	0.805809
trained weighted ESM, five-seed mean	0.792284	0.252970	0.723550	0.805465
Pfam Jaccard max	0.877924	0.305931	0.847764	0.903320
validation-selected raw/Pfam hybrid	0.878192	0.307223	0.849608	0.905321

The trained weighted representation was slightly worse than raw ESM. The validation-selected residual correctly fell back to raw ESM. The raw/Pfam hybrid improved Recall@50 over Pfam by only 0.000268; the paired family-level Holm-adjusted p-value was 0.498. This is not a meaningful or statistically supported improvement.

Conclusion: Residual Gene Weighting v1 successfully prevents degradation but does not rescue the learned-method claim. The current evidence supports Pfam Jaccard as the strongest S. griseus method. Further model tuning on the same 16 test groups would be test-set overfitting.

Reproducibility

Canonical configuration and entry points:

python scripts/train_residual.py \
  --config configs/residual_griseus.yaml \
  --run-id paper-v2-residual-seed-20260810

python scripts/evaluate_residual.py \
  --config configs/residual_griseus.yaml \
  --checkpoint artifacts/paper-v2-residual-seed-20260810/residual_best.pt \
  --run-id paper-v2-residual-seed-20260810-evaluation

python scripts/analyze_residual_campaign.py \
  --seed 20260810 --seed 20260811 --seed 20260812 \
  --seed 20260813 --seed 20260814 \
  --ensemble-run paper-v2-residual-five-seed-ensemble-evaluation \
  --output-dir artifacts/paper-v2-residual-analysis

Primary result files:

artifacts/paper-v2-residual-analysis/aggregate_summary.csv
artifacts/paper-v2-residual-analysis/paired_comparisons.csv
artifacts/paper-v2-residual-analysis/analysis_metadata.json
artifacts/paper-v2-residual-five-seed-ensemble-evaluation/summary.csv
artifacts/paper-v2-residual-campaign.log
artifacts/paper-v2-residual-sha256.txt (58 files; manifest SHA-256 2cee5df67735490d886b94e642375bb771785bb7547ab02ed90ad000f4e511ee)

Required next scientific step

Do not tune another architecture against the same test groups. First:

create a checked taxonomy manifest proving the 182 assemblies are S. griseus;
curate or obtain gold BGC–product labels within those genomes;
version a new untouched family-level holdout or use fully nested group cross-validation;
pre-register the primary metric and baseline comparisons;
only then train a chemistry-aligned residual model.

Documentation rule

After every future request that changes code, data, experiments, results, scope, or scientific interpretation, update this README in the same request with a dated summary and artifact paths.