File size: 4,756 Bytes
c87881a b6f11ae c87881a b6f11ae c87881a b6f11ae c87881a b6f11ae c87881a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 | ---
license: other
library_name: pytorch
tags:
- biology
- bioinformatics
- biosynthetic-gene-clusters
- retrieval
- protein-language-models
- pfam
- pytorch
pipeline_tag: feature-extraction
---
# BGC SetNet and Weighted Pfam Retrieval Models
This repository contains the final five-seed model artifacts for the manuscript **"Sequence-Derived Representations versus Pfam-Domain Content for Biosynthetic Gene Cluster Retrieval"**.
The release includes two model families:
- **Pfam-augmented BGC-SetNet** (`pfam_setnet`): a Set Transformer encoder over frozen ESM-2 gene embeddings, relative gene positions, padding masks, and a BGC-level Pfam inventory embedding.
- **Weighted Pfam Jaccard** (`weighted_pfam_jaccard`): learned non-negative Pfam-domain weights used in a weighted set-Jaccard retrieval baseline.
The manuscript's main result is conservative: Pfam-domain content remains the strongest signal for this silver-label retrieval benchmark, while the ESM + BGC-SetNet + Pfam ensemble gives small, statistically unsupported gains on some secondary metrics.
## Repository Contents
```text
checkpoints/
setnet/seed-20260810..20260814/
model.safetensors # verified model_state checkpoint
config.json
pfam_vocab.json
phase2_history.json
weighted_pfam/seed-20260810..20260814/
model.safetensors
config.json
pfam_vocab.json
phase2_history.json
evaluation/ # per-seed metadata, summaries, group/query outputs, alpha searches
results/dgx_final/ # aggregate manuscript result tables
source/ # source snapshot used for training/evaluation/provenance
provenance/checkpoint_manifest.json
```
## Evaluation Summary
Family means over five optimization seeds and 16 held-out MIBiG-reference groups:
| Method | Recall@50 | MRR | MAP | nDCG@50 |
|---|---:|---:|---:|---:|
| Raw ESM mean | 0.7946 | 0.2550 | 0.7251 | 0.8078 |
| BGC-SetNet + Pfam | 0.8472 | 0.2786 | 0.7771 | 0.8502 |
| Pfam Jaccard | 0.8788 | 0.3071 | 0.8480 | 0.9042 |
| ESM + BGC-SetNet + Pfam | 0.8769 | 0.3096 | 0.8503 | 0.9058 |
| Weighted Pfam Jaccard | 0.8789 | 0.3069 | 0.8477 | 0.9040 |
The weighted Pfam Recall@50 advantage over unweighted Pfam is numerically tiny (+0.00003) and should not be interpreted as a meaningful improvement.
## Checkpoint Verification
All ten source checkpoints were verified by SHA-256 before conversion to clean `model.safetensors` files. The split file hash used by the final run is:
```text
dc26fae17e54fd2ad41a9e10353b3da3e0aacf3144b64f6ee62e8341b4360555
```
See `provenance/checkpoint_manifest.json` for per-seed original checkpoint hashes, safetensors hashes, parameter counts, selected validation ensemble alpha, model config hashes, and split hashes.
## Minimal Loading Example
```python
import json
import torch
from safetensors.torch import load_file
from source.src.bgc_retrieval.model import ModelConfig, build_model
seed_dir = "checkpoints/setnet/seed-20260810"
config = json.load(open(f"{seed_dir}/config.json"))["model"]
model = build_model(ModelConfig.from_dict(config))
state = load_file(f"{seed_dir}/model.safetensors")
model.load_state_dict(state)
model.eval()
```
For `pfam_setnet`, inference expects:
- `gene_embeddings`: `[batch, genes, 1280]` frozen ESM-2 gene embeddings
- `relative_positions`: `[batch, genes]` normalized gene positions
- `padding_mask`: `[batch, genes]` boolean padding mask
- `pfam_tokens`: `[batch, domains]` Pfam IDs encoded with `pfam_vocab.json` (`0` = padding, `1` = unknown)
For `weighted_pfam_jaccard`, inference expects only `pfam_tokens` and returns learned domain weights; pairwise retrieval scores are computed with the model's `pairwise_jaccard` method.
## Intended Use
These models are research artifacts for reproducing and auditing BGC retrieval experiments from the associated manuscript. They are not validated for clinical, diagnostic, regulatory, or production biological decision-making.
## Limitations
- The benchmark labels are silver labels derived from shared MIBiG references, not a manually curated gold retrieval set.
- The split is group-disjoint by MIBiG-reference group, but it is not a complete sequence-homology separation benchmark.
- The strongest primary-metric methods are Pfam-based; learned sequence-derived representations did not show statistically supported superiority in this study.
- The large input atlas, frozen ESM-2 embeddings, and raw gene annotation tables are not bundled in this model repository.
- License terms are pending; contact the authors before reuse beyond inspection and reproducibility work.
## Citation
Citation metadata will be updated after the preprint is posted. For now, cite the manuscript title above and this Hugging Face repository.
|