license: afl-3.0
base_model:
- facebook/esm2_t33_650M_UR50D
tags:
- protein-language-model
- esm-2
- protein
- masked-language-modeling
- domain-specific-fine-tuning
- continual-pretraining
Human Protein Language Model — Combined
Overview
This repository contains the Combined protein language model checkpoint generated for the study “Does domain-specific unsupervised fine-tuning improve protein language model performance?”.
Pfam families that did not contain sufficient sequences for independent Pfam-level or Clan-level fine-tuning were pooled into a composite training dataset. The sequences were clustered at a sequence-identity threshold of 0.5 and used for continued unsupervised fine-tuning of the ESM-2 650M model.
Unlike the Pfam- and Clan-level checkpoints in this project, the Combined model was trained across multiple otherwise ineligible protein families and therefore represents broad cross-family continual pretraining rather than adaptation to a single family or clan.
Model details
- Base model: ESM-2 650M (
facebook/esm2_t33_650M_UR50D) - Training objective: masked language modeling
- Training strategy: pooled cross-family continual pretraining
- Sequence-identity threshold: 0.5
- Number of checkpoints: 1
Intended use
This checkpoint is provided for research on protein representation learning, protein language model benchmarking, continual pretraining, and the effects of cross-family unsupervised adaptation.
The model is intended for research use only and has not been validated for clinical or diagnostic applications.
Code and usage
Source code, benchmarking workflows, and usage instructions are available at:
https://github.com/TianBoxue-lab/DS-UFT-Benchmark
Related models
The accompanying Pfam- and Clan-level model checkpoints are available through the DS-UFT Protein Language Models for Human Protein Families collection on Hugging Face.
License
This repository is distributed under the AFL-3.0 license.