README.md

metadata

language:
  - en
datasets:
  - RosettaCommons/PISCES-CulledPDB
license: mit
library_name: pytorch
base_model: facebook/esm2_t6_8M_UR50D
tags:
  - biology
  - bioinformatics
  - protein-secondary-structure
  - esm2
  - pytorch
  - bilstm
pipeline_tag: token-classification
model-index:
  - name: SERAPH
    results:
      - task:
          type: token-classification
          name: Secondary Structure Prediction (Q3)
        metrics:
          - name: Q3 Test Accuracy
            type: accuracy
            value: 75.31

SERAPH (Secondary Structure Recognition & Prediction Hub)

SERAPH is a deep learning model designed for 3-state (Q3) protein secondary structure prediction. It processes raw single amino acid sequences and predicts residue-level secondary structure states: Alpha Helix (H), Beta Sheet (E), or Coil/Loop (C).

The model leverages a fine-tuned facebook/esm2_t6_8M_UR50D backbone combined with a 1D Convolutional feature extractor and a 2-layer Bidirectional LSTM to capture local motifs and long-range sequence context simultaneously.

Model Details

Model Description

• Developed by: Rogue Builds
• Model Type: Protein Language Model + Conv1D + BiLSTM
• Language(s): Protein Sequences (Amino Acid single-letter codes)
• License: MIT
• Finetuned from model: facebook/esm2_t6_8M_UR50D

Model Sources

• Repository: PypCoder/SERAPH

---

Intended Uses & Limitations

Direct Use

• Residue-level 3-state (Q3) protein secondary structure prediction.
• Single-sequence inference when Multiple Sequence Alignment (MSA) generation is computationally prohibitive or unavailable.
• Integration into downstream bioinformatics analysis pipelines and structural annotation tools.

Out-of-Scope & Misuse

• 3D Coordinate Generation: SERAPH predicts 1D structural states (H, E, C), not 3D atomic coordinates.
• Q8 DSSP Prediction: The model is trained strictly for 3-state classification and does not differentiate between 8-state DSSP assignments (e.g., distinguishing $3_{10}$-helices from $\alpha$-helices).

Known Limitations

• Sequence Length Limit: Input sequences are capped at 512 tokens due to the positional encoding window of the underlying ESM-2 backbone.
• Single-Sequence Bias: Lacks explicit MSA input features; evolutionary context is derived solely from pre-trained ESM-2 representations.

---

How to Get Started

Prerequisites

pip install torch transformers huggingface_hub

Python Inference Example

import torch
import torch.nn as nn
from transformers import EsmModel, EsmTokenizer

# 1. Define SERAPH Architecture
class SERAPH(nn.Module):
    def __init__(self, esm_model, conv_channels=256, kernel_size=7, lstm_hidden=256, num_classes=3, dropout=0.3):
        super().__init__()
        self.esm = esm_model
        esm_embed_dim = self.esm.config.hidden_size
        self.conv = nn.Conv1d(esm_embed_dim, conv_channels, kernel_size=kernel_size, padding=kernel_size // 2)
        self.bn = nn.BatchNorm1d(conv_channels)
        self.dropout = nn.Dropout(dropout)
        self.bilstm = nn.LSTM(conv_channels, lstm_hidden, num_layers=2, batch_first=True, bidirectional=True)
        self.fc = nn.Linear(lstm_hidden * 2, num_classes)

    def forward(self, input_ids, attention_mask=None):
        x = self.esm(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        x = x.transpose(1, 2)
        x = torch.relu(self.bn(self.conv(x)))
        x = self.dropout(x)
        x = x.transpose(1, 2)
        x, _ = self.bilstm(x)
        x = self.dropout(x)
        return self.fc(x)

# 2. Load Tokenizer & Base Backbone
ESM_MODEL_ID = "facebook/esm2_t6_8M_UR50D"
tokenizer = EsmTokenizer.from_pretrained(ESM_MODEL_ID)
esm_backbone = EsmModel.from_pretrained(ESM_MODEL_ID)

model = SERAPH(esm_model=esm_backbone)

# Load weight checkpoint
# checkpoint = torch.load("SERAPH.pth", map_location="cpu")
# model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

# 3. Perform Prediction
IDX_TO_LABEL = {0: 'H', 1: 'E', 2: 'C'}
sequence = "MVLSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTYFPHFDLSHGSAQVKGHGKKVADALTNAVAHVDDMPNALSALSDLHAHKLRVDPVNFKLLSHCLLVTLAAHLPAEFTPAVHASLDKFLASVSTVLTSKYR"

tokens = tokenizer(sequence, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    output = model(input_ids=tokens["input_ids"], attention_mask=tokens["attention_mask"])
    preds = output.argmax(dim=-1)[0]

# Omit special tokens [CLS] and [EOS]
prediction = "".join([IDX_TO_LABEL[p.item()] for p in preds[1:-1]])
print(f"Sequence:   {sequence}")
print(f"Prediction: {prediction}")

---

Training Details

Training Data

• Dataset: CullPDB (~6,000 non-redundant protein chains).

Training Procedure

• Optimizer: Adam (lr=5e-5, weight_decay=1e-4)
• Loss Function: CrossEntropyLoss with class weight adjustments [H: 1.3, E: 1.3, C: 1.0]
• Gradient Clipping: max_norm = 1.0
• Scheduler: ReduceLROnPlateau (patience=3, factor=0.5)
• Batch Size: 32 (with dynamic sequence padding)
• Epochs: 15
• Backbone Unfreezing: Top 2 transformer layers of facebook/esm2_t6_8M_UR50D unfrozen during training.

Parameter Distribution

Layer Component	Trainable Parameters
ESM-2 Backbone (Unfrozen layers)	~2,600,000
Conv1D (320 → 256, k=7)	573,440
BatchNorm1d (256)	512
BiLSTM (2 Layers, hidden=256)	~1,311,232
Linear Head (512 → 3)	1,539
Total Trainable Parameters	3,205,379

---

Evaluation Results

Evaluation Benchmark

Evaluated on the standard CB513 benchmark dataset.

Metrics

Evaluation Metric	Score
Q3 Test Accuracy (CB513)	75.31%
Q3 Training Accuracy	79.34%

Class Breakdown

Structure Class	Precision	Recall
Helix (H)	0.82	0.80
Sheet (E)	0.63	0.81
Coil (C)	0.79	0.68

---

Citation & Contact

If you use SERAPH in your work, please cite the underlying ESM-2 paper and reference this repository:

@software{seraph2026,
  author = {Muhammad Asad Ullah},
  title = {SERAPH: Secondary Structure Recognition & Prediction Hub},
  year = {2026},
  url = {https://huggingface.co/PypCoder/SERAPH}
}