README.md

---
language:
- en
datasets:
- RosettaCommons/PISCES-CulledPDB
license: mit
library_name: pytorch
base_model: facebook/esm2_t6_8M_UR50D
tags:
- biology
- bioinformatics
- protein-secondary-structure
- esm2
- pytorch
- bilstm
pipeline_tag: token-classification
model-index:
- name: SERAPH
  results:
  - task:
      type: token-classification
      name: Secondary Structure Prediction (Q3)
    metrics:
    - name: Q3 Test Accuracy
      type: accuracy
      value: 75.31
---

# SERAPH (Secondary Structure Recognition & Prediction Hub)

**SERAPH** is a deep learning model designed for 3-state (Q3) protein secondary structure prediction. It processes raw single amino acid sequences and predicts residue-level secondary structure states: **Alpha Helix (`H`)**, **Beta Sheet (`E`)**, or **Coil/Loop (`C`)**.

The model leverages a fine-tuned `facebook/esm2_t6_8M_UR50D` backbone combined with a 1D Convolutional feature extractor and a 2-layer Bidirectional LSTM to capture local motifs and long-range sequence context simultaneously.

## Model Details

### Model Description

- **Developed by:** Rogue Builds
- **Model Type:** Protein Language Model + Conv1D + BiLSTM
- **Language(s):** Protein Sequences (Amino Acid single-letter codes)
- **License:** MIT
- **Finetuned from model:** `facebook/esm2_t6_8M_UR50D`

### Model Sources

- **Repository:** `PypCoder/SERAPH`

---

## Intended Uses & Limitations

### Direct Use
* Residue-level 3-state (Q3) protein secondary structure prediction.
* Single-sequence inference when Multiple Sequence Alignment (MSA) generation is computationally prohibitive or unavailable.
* Integration into downstream bioinformatics analysis pipelines and structural annotation tools.

### Out-of-Scope & Misuse
* **3D Coordinate Generation**: SERAPH predicts 1D structural states (`H`, `E`, `C`), not 3D atomic coordinates.
* **Q8 DSSP Prediction**: The model is trained strictly for 3-state classification and does not differentiate between 8-state DSSP assignments (e.g., distinguishing $3_{10}$-helices from $\alpha$-helices).

### Known Limitations
* **Sequence Length Limit**: Input sequences are capped at **512 tokens** due to the positional encoding window of the underlying ESM-2 backbone.
* **Single-Sequence Bias**: Lacks explicit MSA input features; evolutionary context is derived solely from pre-trained ESM-2 representations.

---

## How to Get Started

### Prerequisites

```bash
pip install torch transformers huggingface_hub
```

### Python Inference Example

```python
import torch
import torch.nn as nn
from transformers import EsmModel, EsmTokenizer

# 1. Define SERAPH Architecture
class SERAPH(nn.Module):
    def __init__(self, esm_model, conv_channels=256, kernel_size=7, lstm_hidden=256, num_classes=3, dropout=0.3):
        super().__init__()
        self.esm = esm_model
        esm_embed_dim = self.esm.config.hidden_size
        self.conv = nn.Conv1d(esm_embed_dim, conv_channels, kernel_size=kernel_size, padding=kernel_size // 2)
        self.bn = nn.BatchNorm1d(conv_channels)
        self.dropout = nn.Dropout(dropout)
        self.bilstm = nn.LSTM(conv_channels, lstm_hidden, num_layers=2, batch_first=True, bidirectional=True)
        self.fc = nn.Linear(lstm_hidden * 2, num_classes)

    def forward(self, input_ids, attention_mask=None):
        x = self.esm(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        x = x.transpose(1, 2)
        x = torch.relu(self.bn(self.conv(x)))
        x = self.dropout(x)
        x = x.transpose(1, 2)
        x, _ = self.bilstm(x)
        x = self.dropout(x)
        return self.fc(x)

# 2. Load Tokenizer & Base Backbone
ESM_MODEL_ID = "facebook/esm2_t6_8M_UR50D"
tokenizer = EsmTokenizer.from_pretrained(ESM_MODEL_ID)
esm_backbone = EsmModel.from_pretrained(ESM_MODEL_ID)

model = SERAPH(esm_model=esm_backbone)

# Load weight checkpoint
# checkpoint = torch.load("SERAPH.pth", map_location="cpu")
# model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

# 3. Perform Prediction
IDX_TO_LABEL = {0: 'H', 1: 'E', 2: 'C'}
sequence = "MVLSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTYFPHFDLSHGSAQVKGHGKKVADALTNAVAHVDDMPNALSALSDLHAHKLRVDPVNFKLLSHCLLVTLAAHLPAEFTPAVHASLDKFLASVSTVLTSKYR"

tokens = tokenizer(sequence, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    output = model(input_ids=tokens["input_ids"], attention_mask=tokens["attention_mask"])
    preds = output.argmax(dim=-1)[0]

# Omit special tokens [CLS] and [EOS]
prediction = "".join([IDX_TO_LABEL[p.item()] for p in preds[1:-1]])
print(f"Sequence:   {sequence}")
print(f"Prediction: {prediction}")
```

---

## Training Details

### Training Data

* **Dataset**: CullPDB (~6,000 non-redundant protein chains).

### Training Procedure

* **Optimizer**: Adam (`lr=5e-5`, `weight_decay=1e-4`)
* **Loss Function**: `CrossEntropyLoss` with class weight adjustments `[H: 1.3, E: 1.3, C: 1.0]`
* **Gradient Clipping**: `max_norm = 1.0`
* **Scheduler**: `ReduceLROnPlateau` (`patience=3`, `factor=0.5`)
* **Batch Size**: 32 (with dynamic sequence padding)
* **Epochs**: 15
* **Backbone Unfreezing**: Top 2 transformer layers of `facebook/esm2_t6_8M_UR50D` unfrozen during training.

### Parameter Distribution

| Layer Component | Trainable Parameters |
|---|---|
| ESM-2 Backbone (Unfrozen layers) | ~2,600,000 |
| Conv1D (`320 → 256`, `k=7`) | 573,440 |
| BatchNorm1d (`256`) | 512 |
| BiLSTM (2 Layers, hidden=256) | ~1,311,232 |
| Linear Head (`512 → 3`) | 1,539 |
| **Total Trainable Parameters** | **3,205,379** |

---

## Evaluation Results

### Evaluation Benchmark

Evaluated on the standard **CB513** benchmark dataset.

### Metrics

| Evaluation Metric | Score |
|---|---|
| **Q3 Test Accuracy (CB513)** | **75.31%** |
| **Q3 Training Accuracy** | **79.34%** |

#### Class Breakdown

| Structure Class | Precision | Recall |
|---|---|---|
| **Helix (`H`)** | 0.82 | 0.80 |
| **Sheet (`E`)** | 0.63 | 0.81 |
| **Coil (`C`)** | 0.79 | 0.68 |

---

## Citation & Contact

If you use SERAPH in your work, please cite the underlying ESM-2 paper and reference this repository:

```bibtex
@software{seraph2026,
  author = {Muhammad Asad Ullah},
  title = {SERAPH: Secondary Structure Recognition & Prediction Hub},
  year = {2026},
  url = {https://huggingface.co/PypCoder/SERAPH}
}
```