GrimSqueaker/ProtSent-V2.5-150M

ProtSent-V2.5 ESM-2 150M

Continued contrastive training of ProtSent-V2 150M on a fresh draw of the corpus, adding a DMS/ProteinGym CoSENT target and a Global Orthogonal Regularization term.

Mean-pooled ESM-2 150M embeddings, dimension 640, max sequence length 512.

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer("GrimSqueaker/ProtSent-V2.5-150M")
emb = model.encode(["MKTLLLTLVVVTIVCLDLGYT", "MKTLLLTLVVVTIVCLDLGYN", "AGWYRSPQEGLKPVDTFKDIV"])
print(cos_sim(emb[0], emb[1:]))

Training data

source	pairs available	selection
Pfam	777,306	k=8 per cluster
AlphaFold DB (Foldseek clusters)	18,980,000	k=8 per cluster
STRING-DB v12 PPI	15,000,000	full filtered table
DMS / ProteinGym (CoSENT)	1,000,000	prefix of interleaved assays
total	35.8M

Trained on 14.7M pairs (3,600 steps x 4,096 effective batch), 41% of the pool.

Pfam, AFDB and STRING are decontaminated with MMseqs2 at 40% identity / 80% coverage against the remote-homology and PPI test splits. The DMS parquet is not MMseqs2-filtered; it has zero exact-sequence overlap with the AAV Fitness (0/50,430), Stability, Variant Effect and Fluorescence test sets.

Run parameters

parameter	value
init	ProtSent-V2-150M (verified identical, 515 tensors, max abs diff 0.0)
primary loss	CachedMultipleNegativesRankingLoss, scale 20
--mnrl_directions	query_to_doc doc_to_query (symmetric)
--gor_weight	1.0
--gor_max_samples	64
auxiliary loss	CoSENTLoss on DMS, batch capped at 64, scale 20
Matryoshka	off
batch size (per device)	1024
--mnrl_mini_batch_size	64
gather across devices	off
--batch_sampler	none
--max_seq_length	512
--max_pairs_per_cluster	8
learning rate	5e-5, cosine_with_min_lr, 0.5 cycles, 200 warmup
seeds (shuffle / global)	17 / 11
precision / attention	bf16, flash-attention-2
hardware	4x NVIDIA B300, 14 h 18 m
steps	3,600

Training code: github.com/oriel9p/ProtSent, train_esm2_150m_v2p5.sh.

Results

SCOPe-40 structural retrieval, test split, self excluded, restricted to the 1,693 of 2,207 queries with a non-self same-family protein in the gallery.

method	R@1	R@10	R@30	MAP
ESM-2 150M	0.554	0.770	0.842	0.424
MMseqs2 (-s 7.5)	0.656	0.740	0.757	0.410
HMMER (phmmer, max sensitivity)	0.753	0.898	0.923	0.607
ProtSent-V1 150M	0.662	0.894	0.944	0.643
ProtSent-V2 150M	0.743	0.937	0.968	0.705
ProtSent-V2.5 150M	0.751	0.945	0.972	0.723

Paired bootstrap over queries, 10,000 resamples, V2.5 − V2:

metric	delta	95% CI
R@1	+0.0095	[−0.0024, +0.0213]
R@10	+0.0077	[+0.0012, +0.0142]
R@30	+0.0053	[+0.0006, +0.0106]
MAP	+0.0189	[+0.0137, +0.0241]

R@10, R@30 and MAP exclude zero; R@1 does not. HMMER still leads at R@1.

23-task downstream suite

Win/tie/loss over the 20 tasks with a defined one-vs-rest AUC, ties at |delta| < 0.005, with the median delta. All arms scored with --eval_split test, EvalMode=standard, seed 42.

comparison	k-NN probe	linear probe
V2.5 vs ESM-2 150M	12W/2T/5L, +0.010	3W/2T/14L, −0.016
V2 vs ESM-2 150M	9W/3T/7L, +0.004	3W/4T/12L, −0.014
V2.5 vs V1 150M	13W/3T/3L, +0.007	4W/6T/9L, −0.004
V2.5 vs V2 150M	8W/8T/3L, +0.001	3W/11T/5L, −0.003

V2.5 improves on the untuned backbone under a k-NN probe by more than V2 did. Under a linear probe, contrastive post-training costs about 0.015 against the backbone at this scale, and V2.5 does not change that.

Per-task, both V2.5 − V2 medians are inside the tie band. The movement is concentrated in the Spearman regression tasks and reverses by probe: mean delta +0.020 under k-NN and −0.006 under linear, while AUC tasks are flat under both (+0.000 and −0.001).

task	metric	k-NN delta	linear delta
Fluorescence (TAPE)	Spearman	+0.068	+0.015
Thermostability (FLIP)	Spearman	+0.033	−0.000
Variant Effect (GB1)	Spearman	+0.025	−0.003
Cloning Classification	Spearman	+0.023	+0.006
beta-lactamase-PEER	Spearman	+0.021	+0.049
Molecular Function (GO)	F1_Macro	−0.014	−0.014
Metal Ion Binding	AUC	−0.011	−0.008
Stability (Biomap)	Spearman	+0.014	−0.023
AAV Fitness (FLIP)	Spearman	+0.003	−0.086

AAV Fitness and Stability decline monotonically across the whole 150M lineage under a linear probe (AAV 0.589 → 0.398 → 0.451 → 0.365 for ESM-2 → V1 → V2 → V2.5; Stability 0.706 → 0.699 → 0.663 → 0.639).

The AAV drop is not embedding collapse. Over 3,000 AAV test variants, V2.5 has a lower mean pairwise cosine than V2 (0.983 vs 0.992), higher effective dimensionality (10.4 vs 6.8) and a higher best-single-direction correlation with fitness (0.576 vs 0.542, against 0.554 for vanilla ESM-2). The variants are spread further apart and carry more linear signal in-sample; what degrades is transfer from the train split to the deliberately hard FLIP test split.

V2.5 changes six settings at once relative to V2 (GOR, DMS target, symmetric directions, k, seeds, batch geometry) and there is no GOR-off ablation at this scale, so no individual change is isolated here. The equivalent ablation at 35M found GOR contributed nothing measurable.