File size: 3,860 Bytes
699c85e 4fa6cc9 699c85e 4fa6cc9 65bc3f5 4fa6cc9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 | ---
license: apache-2.0
tags:
- protein-language-model
- bioinformatics
- transformer
- cnn
- small-protein
- smORF
frameworks:
- pytorch
---
```
βββββββββββββββββββββββββββββββββββββββββββββββββ
β β
β _______ _____ _______ β
β |__ __| __ \__ __| β
β | | | |__) | | | β
β | | | ___/ | | β
β | | | | | | β
β |_| |_| |_| β
β β
β βΈ TINY PROTEIN TRANSFORMER β β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββ
```
# TinyProteinTransformer (TPT)
TinyProteinTransformer (TPT) is a lightweight CNN-Transformer hybrid encoder designed for microbial smORF-encoded small proteins. It was pretrained on the Global Microbial smORF Catalog (GMSC, >280M sequences) using masked language modeling combined with contrastive learning, producing compact yet expressive residue-level and sequence-level protein representations.
This repository hosts the pretrained model weights for direct inference and downstream fine-tuning.
For training scripts, ablation code, and the full experimental pipeline, see the GitHub repository: [F4NG66/TPT](https://github.com/F4NG66/TPT)
## Files
| File | Description |
|---|---|
| `TPT.pt` | Baseline pretrained TPT weights (CNN-Transformer hybrid encoder, hidden dim 640) |
| `TPT_gated.py` | Gated variant of the TPT architecture|
## Model Details
- **Architecture:** CNN-Transformer hybrid encoder with multi-scale convolutional feature extraction and attention-based pooling
- **Hidden dimension:** 640
- **Transformer layers:** 20
- **CNN kernel sizes:** 3, 5, 7, 9
- **Max sequence length:** 128
- **Pretraining objective:** Masked language modeling + contrastive learning (temperature Ο = 0.1, loss weight Ξ» = 0.05)
- **Pretraining corpus:** GMSC10.90 (Global Microbial smORF Catalog)
## Usage
```python
import torch
from model import TinyProteinTransformer
from utils import build_tokenizer
tokenizer = build_tokenizer()
model = TinyProteinTransformer(vocab_size=len(tokenizer))
model.load_state_dict(torch.load("TPT.pt", map_location="cpu"))
model.eval()
def encode(seq, max_len=128):
ids = [tokenizer.get(aa, tokenizer["X"]) for aa in seq][:max_len]
ids += [tokenizer["PAD"]] * (max_len - len(ids))
return torch.tensor(ids).unsqueeze(0)
with torch.no_grad():
x = encode("MKVLILACLVVVTITVS")
h = model.encode(x) # (1, L, 640) residue-level representations
embedding = model.attention_pool(h) # (1, 640) sequence-level embedding
```
For classification tasks, attach a lightweight linear head on top of `embedding`.
### Gated variant
`TPT_gated.py` defines the gated architecture variant. Load its accompanying checkpoint the same way, using the model class defined in that script in place of `TinyProteinTransformer`.
## Intended Use
TPT embeddings can be used as frozen features for downstream classification tasks on small secreted/microbial proteins, including antimicrobial peptide (AMP) prediction, toxicity (TOX) prediction, bacteriocin (BCN) classification, and anti-CRISPR (Acr) protein classification.
## Citation
```bibtex
@article{sheng2026tpt,
title={TPT: A Compact CNN-Transformer Encoder for Efficient Microbial Small Protein Modeling},
author={Sheng, Fang and Zhang, Junhe and Zhu, Chengkai},
journal={Frontiers in Microbiology},
year={2026}
}
``` |