| --- |
| license: afl-3.0 |
| base_model: |
| - facebook/esm2_t33_650M_UR50D |
| tags: |
| - protein-language-model |
| - esm-2 |
| - protein |
| - masked-language-modeling |
| - domain-specific-fine-tuning |
| - continual-pretraining |
| --- |
| |
| # Human Protein Language Model — Combined |
|
|
| ## Overview |
|
|
| This repository contains the Combined protein language model checkpoint generated for the study “Does domain-specific unsupervised fine-tuning improve protein language model performance?”. |
|
|
| Pfam families that did not contain sufficient sequences for independent Pfam-level or Clan-level fine-tuning were pooled into a composite training dataset. The sequences were clustered at a sequence-identity threshold of 0.5 and used for continued unsupervised fine-tuning of the ESM-2 650M model. |
|
|
| Unlike the Pfam- and Clan-level checkpoints in this project, the Combined model was trained across multiple otherwise ineligible protein families and therefore represents broad cross-family continual pretraining rather than adaptation to a single family or clan. |
|
|
| ## Model details |
|
|
| - Base model: ESM-2 650M (`facebook/esm2_t33_650M_UR50D`) |
| - Training objective: masked language modeling |
| - Training strategy: pooled cross-family continual pretraining |
| - Sequence-identity threshold: 0.5 |
| - Number of checkpoints: 1 |
|
|
| ## Intended use |
|
|
| This checkpoint is provided for research on protein representation learning, protein language model benchmarking, continual pretraining, and the effects of cross-family unsupervised adaptation. |
|
|
| The model is intended for research use only and has not been validated for clinical or diagnostic applications. |
|
|
| ## Code and usage |
|
|
| Source code, benchmarking workflows, and usage instructions are available at: |
|
|
| https://github.com/TianBoxue-lab/DS-UFT-Benchmark |
|
|
| ## Related models |
|
|
| The accompanying Pfam- and Clan-level model checkpoints are available through the DS-UFT Protein Language Models for Human Protein Families collection on Hugging Face. |
|
|
| ## License |
|
|
| This repository is distributed under the AFL-3.0 license. |
|
|
