license: afl-3.0
datasets:
- bloyal/uniref100
pipeline_tag: fill-mask
library_name: transformers
tags:
- biology
- protein-language-model
- protein
ProtAlbert
ProtAlbert is a protein Language Model (pLM) pretrained on the Uniref100 using a masked language modeling (MLM) objective. It's suitable for creation of embeddings (feature extraction), but also fill the mask. The model was developed by Ahmed Elnaggar et al. and more information can be found on the GitHub repository and in the accompanying paper. This repository is a fork of their HuggingFace repository.
Inference example
Example for masked language modeling:
from transformers import AlbertForMaskedLM, AlbertTokenizer, pipeline
>>>> tokenizer = AlbertTokenizer.from_pretrained("virtual-human-chc/prot_albert", do_lower_case=False )
>>>> model = AlbertForMaskedLM.from_pretrained("virtual-human-chc/prot_albert")
>>>> unmasker = pipeline('fill-mask', model=model, tokenizer=tokenizer)
>>>> unmasker('D L I P T S S K L V V [MASK] D T S L Q V K K A F F A L V T')
{'score': 0.10074187070131302, 'token': 13, 'token_str': 'L', 'sequence': 'D L I P T S S K L V V L D T S L Q V K K A F F A L V T'},
{'score': 0.08413360267877579, 'token': 14, 'token_str': 'S', 'sequence': 'D L I P T S S K L V V S D T S L Q V K K A F F A L V T'},
{'score': 0.07617155462503433, 'token': 18, 'token_str': 'V', 'sequence': 'D L I P T S S K L V V V D T S L Q V K K A F F A L V T'},
{'score': 0.06521160155534744, 'token': 19, 'token_str': 'T', 'sequence': 'D L I P T S S K L V V T D T S L Q V K K A F F A L V T'},
{'score': 0.06321343779563904, 'token': 15, 'token_str': 'A', 'sequence': 'D L I P T S S K L V V A D T S L Q V K K A F F A L V T'}]
Example for feature extraction:
from transformers import AutoModel, AlbertTokenizer, pipeline
import re
tokenizer = AlbertTokenizer.from_pretrained("virtual-human-chc/prot_albert", do_lower_case=False)
model = AutoModel.from_pretrained("virtual-human-chc/prot_albert")
fe = pipeline('feature-extraction', model=model, tokenizer=tokenizer, device=0)
sequences_Example = ["A E T C Z A O", "S K T Z P"]
sequences_Example = [re.sub(r"[UZOB]", "X", sequence) for sequence in sequences_Example]
embedding = fe(sequences_Example)
print(embedding)
Copyright
Code derived from https://github.com/agemagician/ProtTrans is licensed under the MIT License, Copyright (c) 2025 Ahmed Elnaggar. The ProtTrans pretrained models are released under the under terms of the Academic Free License v3.0 License, Copyright (c) 2025 Ahmed Elnaggar. The other code is licensed under the MIT license, Copyright (c) 2025 Maksim Pavlov.