README.md

---
license: afl-3.0
base_model:
- facebook/esm2_t33_650M_UR50D
tags:
- protein-language-model
- esm-2
- protein
- masked-language-modeling
- domain-specific-fine-tuning
- continual-pretraining
---

# Human Protein Language Model — Combined

## Overview

This repository contains the Combined protein language model checkpoint generated for the study “Does domain-specific unsupervised fine-tuning improve protein language model performance?”.

Pfam families that did not contain sufficient sequences for independent Pfam-level or Clan-level fine-tuning were pooled into a composite training dataset. The sequences were clustered at a sequence-identity threshold of 0.5 and used for continued unsupervised fine-tuning of the ESM-2 650M model.

Unlike the Pfam- and Clan-level checkpoints in this project, the Combined model was trained across multiple otherwise ineligible protein families and therefore represents broad cross-family continual pretraining rather than adaptation to a single family or clan.

## Model details

- Base model: ESM-2 650M (`facebook/esm2_t33_650M_UR50D`)
- Training objective: masked language modeling
- Training strategy: pooled cross-family continual pretraining
- Sequence-identity threshold: 0.5
- Number of checkpoints: 1

## Intended use

This checkpoint is provided for research on protein representation learning, protein language model benchmarking, continual pretraining, and the effects of cross-family unsupervised adaptation.

The model is intended for research use only and has not been validated for clinical or diagnostic applications.

## Code and usage

Source code, benchmarking workflows, and usage instructions are available at:

https://github.com/TianBoxue-lab/DS-UFT-Benchmark

## Related models

The accompanying Pfam- and Clan-level model checkpoints are available through the DS-UFT Protein Language Models for Human Protein Families collection on Hugging Face.

## License

This repository is distributed under the AFL-3.0 license.