whiteh4t/bgc-setnet

BGC SetNet and Weighted Pfam Retrieval Models

This repository contains the final five-seed model artifacts for the manuscript "Sequence-Derived Representations versus Pfam-Domain Content for Biosynthetic Gene Cluster Retrieval".

The release includes two model families:

• Pfam-augmented BGC-SetNet (pfam_setnet): a Set Transformer encoder over frozen ESM-2 gene embeddings, relative gene positions, padding masks, and a BGC-level Pfam inventory embedding.
• Weighted Pfam Jaccard (weighted_pfam_jaccard): learned non-negative Pfam-domain weights used in a weighted set-Jaccard retrieval baseline.

The manuscript's main result is conservative: Pfam-domain content remains the strongest signal for this silver-label retrieval benchmark, while the ESM + BGC-SetNet + Pfam ensemble gives small, statistically unsupported gains on some secondary metrics.

Repository Contents

checkpoints/
  setnet/seed-20260810..20260814/
    model.safetensors      # verified model_state checkpoint
    config.json
    pfam_vocab.json
    phase2_history.json
  weighted_pfam/seed-20260810..20260814/
    model.safetensors
    config.json
    pfam_vocab.json
    phase2_history.json
evaluation/                # per-seed metadata, summaries, group/query outputs, alpha searches
results/dgx_final/          # aggregate manuscript result tables
source/                     # source snapshot used for training/evaluation/provenance
provenance/checkpoint_manifest.json

Evaluation Summary

Family means over five optimization seeds and 16 held-out MIBiG-reference groups:

Method	Recall@50	MRR	MAP	nDCG@50
Raw ESM mean	0.7946	0.2550	0.7251	0.8078
BGC-SetNet + Pfam	0.8472	0.2786	0.7771	0.8502
Pfam Jaccard	0.8788	0.3071	0.8480	0.9042
ESM + BGC-SetNet + Pfam	0.8769	0.3096	0.8503	0.9058
Weighted Pfam Jaccard	0.8789	0.3069	0.8477	0.9040

The weighted Pfam Recall@50 advantage over unweighted Pfam is numerically tiny (+0.00003) and should not be interpreted as a meaningful improvement.

Checkpoint Verification

All ten source checkpoints were verified by SHA-256 before conversion to clean model.safetensors files. The split file hash used by the final run is:

dc26fae17e54fd2ad41a9e10353b3da3e0aacf3144b64f6ee62e8341b4360555

See provenance/checkpoint_manifest.json for per-seed original checkpoint hashes, safetensors hashes, parameter counts, selected validation ensemble alpha, model config hashes, and split hashes.

Minimal Loading Example

import json
import torch
from safetensors.torch import load_file

from source.src.bgc_retrieval.model import ModelConfig, build_model

seed_dir = "checkpoints/setnet/seed-20260810"
config = json.load(open(f"{seed_dir}/config.json"))["model"]
model = build_model(ModelConfig.from_dict(config))
state = load_file(f"{seed_dir}/model.safetensors")
model.load_state_dict(state)
model.eval()

For pfam_setnet, inference expects:

• gene_embeddings: [batch, genes, 1280] frozen ESM-2 gene embeddings
• relative_positions: [batch, genes] normalized gene positions
• padding_mask: [batch, genes] boolean padding mask
• pfam_tokens: [batch, domains] Pfam IDs encoded with pfam_vocab.json (0 = padding, 1 = unknown)

For weighted_pfam_jaccard, inference expects only pfam_tokens and returns learned domain weights; pairwise retrieval scores are computed with the model's pairwise_jaccard method.

Intended Use

These models are research artifacts for reproducing and auditing BGC retrieval experiments from the associated manuscript. They are not validated for clinical, diagnostic, regulatory, or production biological decision-making.

Limitations

• The benchmark labels are silver labels derived from shared MIBiG references, not a manually curated gold retrieval set.
• The split is group-disjoint by MIBiG-reference group, but it is not a complete sequence-homology separation benchmark.
• The strongest primary-metric methods are Pfam-based; learned sequence-derived representations did not show statistically supported superiority in this study.
• The large input atlas, frozen ESM-2 embeddings, and raw gene annotation tables are not bundled in this model repository.
• License terms are pending; contact the authors before reuse beyond inspection and reproducibility work.

Citation

Citation metadata will be updated after the preprint is posted. For now, cite the manuscript title above and this Hugging Face repository.