HELM-BERT

A peptide language model using HELM (Hierarchical Editing Language for Macromolecules) notation, compatible with Hugging Face Transformers.

Model Description

HELM-BERT is built upon the DeBERTa architecture, pre-trained on ~75k peptides from four databases (ChEMBL, CREMP, CycPeptMPDB, Propedia) using Masked Language Modeling (MLM) with a Warmup-Stable-Decay (WSD) learning rate schedule.

• Disentangled Attention: Decomposes attention into content-content and content-position terms
• Enhanced Mask Decoder (EMD): Injects absolute position embeddings at the decoder stage
• Span Masking: Contiguous token masking with geometric distribution
• nGiE: n-gram Induced Encoding layer (1D convolution, kernel size 3)

Model Specifications

Parameter	Value
Parameters	54.8M
Hidden size	768
Layers	6
Attention heads	12
Vocab size	78
Max token length	512
Pre-training data	~75k peptides (ChEMBL, CREMP, CycPeptMPDB, Propedia)
Pre-training objective	MLM (span masking, p=0.15)
LR schedule	Warmup-Stable-Decay (WSD)

How to Use

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("Flansma/helm-bert", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Flansma/helm-bert", trust_remote_code=True)

# Cyclosporine A
inputs = tokenizer("PEPTIDE1{[Abu].[Sar].[meL].V.[meL].A.[dA].[meL].[meL].[meV].[Me_Bmt(E)]}$PEPTIDE1,PEPTIDE1,1:R1-11:R2$$$", return_tensors="pt")
outputs = model(**inputs)
embeddings = outputs.last_hidden_state

Training Data

Pre-trained on deduplicated peptide sequences from:

• ChEMBL: Bioactive molecules database
• CREMP: Cyclic peptide conformational ensemble database
• CycPeptMPDB: Cyclic peptide membrane permeability database
• Propedia: Protein-peptide interaction database

Downstream Performance

Permeability Regression (CycPeptMPDB)

Single-Assay (mixed PAMPA/Caco-2 target):

Split	R²	Pearson	RMSE	MAE
Random	0.658	0.817	0.471	0.300
Scaffold	0.502	0.723	0.450	0.324

Per-Assay (separate models for PAMPA and Caco-2):

Split	Assay	R²	Pearson	RMSE	MAE
Random	PAMPA	0.800	0.895	0.355	0.253
Random	Caco-2	0.747	0.866	0.388	0.289
Scaffold	PAMPA	0.529	0.739	0.412	0.295
Scaffold	Caco-2	0.637	0.874	0.405	0.334

Train/test 9:1, val 10% from train. Scaffold split by Murcko scaffolds.

PPI Classification (Propedia v2)

Split	ROC-AUC	PR-AUC	F1	MCC	Balanced Acc
Random	0.968	0.901	0.847	0.808	0.906
aCSM	0.862	0.683	0.587	0.522	0.722

Train/test 8:2, val 10% from train, 1:4 positive:negative ratio.

• Random: random split
• aCSM: clustering-based split on aCSM-ALL complex signatures with protein overlap pruning

PPI Classification (ChEMBL)

Split	ROC-AUC	PR-AUC	F1	MCC	Balanced Acc
Random	0.992	0.975	0.948	0.936	0.969
Family	0.786	0.449	0.267	0.222	0.570

Val 10% from train.

Citation

@article{lee2025helmbert,
  title={HELM-BERT: A Transformer for Medium-sized Peptide Property Prediction},
  author={Seungeon Lee and Takuto Koyama and Itsuki Maeda and Shigeyuki Matsumoto and Yasushi Okuno},
  journal={arXiv preprint arXiv:2512.23175},
  year={2025},
  url={https://arxiv.org/abs/2512.23175}
}

License

MIT License