upload files

README.md

@@ -0,0 +1,56 @@
+---
+license: cc-by-nc-nd-4.0
+datasets:
+- SaeedLab/SeqScreen
+tags:
+- proteins
+- molecules
+- bioinformatics
+- drug-discovery
+- feature-extraction
+- transformers
+---
+
+# SeqScreen Frozen
+
+This model corresponds to the SeqScreen frozen configuration, in which only the projection layers are trained.
+
+\[[Github Repo](https://github.com/pcdslab/SeqScreen)\] | \[[Dataset on HuggingFace](https://huggingface.co/datasets/SaeedLab/SeqScreen)\] | \[[Model Collection](https://huggingface.co/collections/SaeedLab/seqscreen)\] | \[[Cite](#citation)\]
+
+## Abstract
+
+Virtual screening aims to identify candidate molecules that bind to a target protein, playing a central role in computational drug discovery. Sequence-based deep learning methods offer an applicable alternative to structure-based approaches, but typically process one protein-molecule pair at a time, limiting their scalability to large molecular libraries. Contrastive learning methods inspired by CLIP have shown promise for learning joint protein-molecule representations, but standard CLIP training was designed for symmetric tasks and does not account for the asymmetric and one-to-many nature of protein-molecule binding. In this paper, we introduce *SeqScreen*, a sequence-based virtual screening method built on a dual-encoder contrastive architecture. SeqScreen introduces a protein-centric batch construction strategy and an asymmetric multi-positive InfoNCE loss to cope with the protein-centric nature of virtual screening. We conduct a systematic evaluation across 8 protein language models and 3 molecular language model variants. The protein-centric batch construction consistently outperforms standard CLIP training across all evaluated encoders, while requiring approximately 32 times fewer training epochs and 7 times fewer forward passes during inference compared to pair-based methods. On the LIT-PCBA dataset, SeqScreen outperforms all sequence-based baselines, achieving a relative improvement of up to 39% in EF at 0.5 over the best competing method, while remaining competitive with traditional docking approaches without requiring 3D structural information.
+
+## Model Details
+
+SeqScreen uses a dual-encoder architecture that independently encodes proteins and molecules using specialized language models. The protein branch processes amino acid sequences with ESM2 T36, and the molecule branch processes SMILES strings with MolDeBERTa MLC. Both representations are passed through projection heads to produce normalized embeddings.
+
+![Model](pipeline.png)
+
+Two configurations are available in this collection:
+
+- [SeqScreen-Frozen](https://huggingface.co/SaeedLab/SeqScreen-Frozen): only the projection layers are trained, both encoders are frozen.
+- [SeqScreen-Finetuning](https://huggingface.co/SaeedLab/SeqScreen-Finetuning): the projection layers and ESM2 T36 are trained, MolDeBERTa MLC is frozen.
+
+## Usage
+
+SeqScreen computes cosine similarities between protein and molecule embeddings, which can be used to rank candidate molecules for a given target protein.
+
+### Similarity
+
+```python
+# code
+```
+
+
+## Citation
+
+The paper is under review. As soon as it is accepted, we will update this section.
+
+## License
+
+This model and associated code are released under the CC-BY-NC-ND 4.0 license and may only be used for non-commercial, academic research purposes with proper attribution. Any commercial use, sale, or other monetization of this model and its derivatives, which include models trained on outputs from the model or datasets created from the model, is prohibited and requires prior approval. Downloading the model requires prior registration on Hugging Face and agreeing to the terms of use. By downloading this model, you agree not to distribute, publish or reproduce a copy of the model. If another user within your organization wishes to use the model, they must register as an individual user and agree to comply with the terms of use. Users may not attempt to re-identify the deidentified data used to develop the underlying model. If you are a commercial entity, please contact the corresponding author.
+
+## Contact
+
+For any additional questions or comments, contact Fahad Saeed (fsaeed@fiu.edu).

config.json

@@ -0,0 +1,16 @@
+{
+  "auto_map": {
+    "AutoConfig": "configuration_seqscreen.SeqScreenConfig",
+    "AutoModel": "modeling_seqscreen.SeqScreenModel"
+  },
+  "architectures": [
+    "SeqScreenModel"
+  ],
+  "dropout": 0.1,
+  "dtype": "float32",
+  "model_type": "seqscreen",
+  "mol_dim": 768,
+  "proj_dim": 512,
+  "prot_dim": 2560,
+  "transformers_version": "4.57.3"
+}

configuration_seqscreen.py

@@ -0,0 +1,16 @@
+from transformers import PretrainedConfig
+
+class SeqScreenConfig(PretrainedConfig):
+  model_type = "seqscreen"
+  def __init__(
+    self,
+    prot_dim: int = 2560, 
+    mol_dim: int = 768, 
+    proj_dim: int = 512, 
+    dropout: float = 0.1,
+    **kwargs):
+    super().__init__(**kwargs)
+    self.prot_dim = prot_dim
+    self.mol_dim = mol_dim
+    self.proj_dim = proj_dim
+    self.dropout = dropout

convert_weights.py

@@ -0,0 +1,44 @@
+import os
+import torch
+from configuration_seqscreen import SeqScreenConfig
+from modeling_seqscreen import SeqScreenModel
+
+
+def convert_model(checkpoint_path, save_directory):
+  config = SeqScreenConfig()
+  hf_model = SeqScreenModel(config)
+  hf_model.eval()
+
+  old_state_dict = torch.load(checkpoint_path, map_location="cpu")
+
+  expected_prefixes = ("proj_prot.", "proj_mol.")
+
+  new_state_dict = {}
+  for key, value in old_state_dict.items():
+    if key.startswith(expected_prefixes):
+      new_state_dict[key] = value
+    else:
+      print(f"[Skip] {key}")
+
+  missing = set(hf_model.state_dict().keys()) - set(new_state_dict.keys())
+  unexpected = set(new_state_dict.keys()) - set(hf_model.state_dict().keys())
+
+  if missing:
+    raise RuntimeError(f"Missing keys in checkpoint: {missing}")
+  if unexpected:
+    raise RuntimeError(f"Unexpected keys after filtering: {unexpected}")
+
+  hf_model.load_state_dict(new_state_dict, strict=True)
+  print("State dict loaded successfully.")
+
+  os.makedirs(save_directory, exist_ok=True)
+  hf_model.save_pretrained(save_directory)
+  config.save_pretrained(save_directory)
+  print(f"Model saved to: {save_directory}")
+
+
+if __name__ == "__main__":
+  convert_model(
+    checkpoint_path="model.pt",
+    save_directory="./seqscreen_hf",
+  )

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1c541db2f8ae2d5e06ed1ab8d032278d818a8f4800880c482bc8b76600494a1
+size 8930448

modeling_seqscreen.py

@@ -0,0 +1,54 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from dataclasses import dataclass
+import torch
+from transformers.utils import ModelOutput
+from transformers import PreTrainedModel
+
+from .configuration_seqscreen import SeqScreenConfig
+
+@dataclass
+class SeqScreenModelOutput(ModelOutput):
+  prot_rep: torch.FloatTensor = None
+  mol_rep: torch.FloatTensor = None
+  similarity: torch.FloatTensor = None
+
+class ProjectionLayer(nn.Module):
+  def __init__(self, in_dim, out_dim, dropout):
+    super().__init__()
+    self.projection = nn.Sequential(
+      nn.Linear(in_dim, out_dim),
+      nn.LayerNorm(out_dim),
+      nn.GELU(),
+      nn.Dropout(dropout),
+      nn.Linear(out_dim, out_dim)
+    )
+
+  def forward(self, x):
+    x = self.projection(x)
+    return F.normalize(x, dim=-1)
+
+
+class SeqScreenModel(PreTrainedModel):
+  config_class = SeqScreenConfig
+  base_model_prefix = "seqscreen"
+
+  def __init__(self, config: SeqScreenConfig):
+    super().__init__(config)
+
+    self.proj_prot = ProjectionLayer(config.prot_dim, config.proj_dim, dropout=config.dropout)
+    self.proj_mol = ProjectionLayer(config.mol_dim, config.proj_dim, dropout=config.dropout)
+
+    self.post_init()
+
+  def forward(self, prot: torch.Tensor, mol: torch.Tensor):
+    prot_rep = self.proj_prot(prot)
+    mol_rep = self.proj_mol(mol)
+    similarity = prot_rep @ mol_rep.T
+
+    return SeqScreenModelOutput(
+      prot_rep=prot_rep,
+      mol_rep=mol_rep,
+      similarity=similarity
+    )