---
license: mit
language:
- en
pipeline_tag: feature-extraction
tags:
- protein-language-model
library_name: transformers
---

# ProtT5-XXL-BFD

ProtT5-XXL is a protein Language Model (pLM) pretrained on BFD, a dataset consisting of 2.1 billion protein sequences, using a masked language modeling (MLM) objective. It's suitable for creation of embeddings (feature extraction). The model was developed by Ahmed Elnaggar et al. and more information can be found on the [GitHub repository](https://github.com/agemagician/ProtTrans) and in the [accompanying paper](https://ieeexplore.ieee.org/document/9477085). This repository is a fork of their [HuggingFace repository](https://huggingface.co/Rostlab/prot_t5_xxl_bfd/tree/main).

## Inference example

Below is a minimal example showing how to load and run inference with **ProtT5-XXL-BFD** directly from the Hugging Face Hub using `transformers` library.

```python
from transformers import T5EncoderModel, T5Tokenizer
import re
import torch 

# Load the vocabulary and ProtT5-XL-BFD Model
tokenizer = T5Tokenizer.from_pretrained("Rostlab/prot_t5_xxl_bfd", do_lower_case=False)
 
model = T5EncoderModel.from_pretrained("Rostlab/prot_t5_xxl_bfd")

# Load the model into the GPU if avilabile and switch to inference mode
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

model = model.to(device)
model = model.eval()

# Create or load sequences and map rarely occured amino acids (U,Z,O,B) to (X)
sequences_Example = ["A E T C Z A O","S K T Z P"]

sequences_Example = [re.sub(r"[UZOB]", "X", sequence) for sequence in sequences_Example]

# Tokenize, encode sequences and load it into the GPU if possible
ids = tokenizer.batch_encode_plus(sequences_Example, add_special_tokens=True, padding=True)

input_ids = torch.tensor(ids['input_ids']).to(device)
attention_mask = torch.tensor(ids['attention_mask']).to(device)

# Extracting sequences' features and load it into the CPU if needed
with torch.no_grad():
    embedding = model(input_ids=input_ids,attention_mask=attention_mask)

encoder_embedding = embedding[2].cpu().numpy()
decoder_embedding = embedding[0].cpu().numpy()
```

## Metadata

### Input

The model takes as input uppercase amino acids. It works only with capital letter amino acids. Example: "A E T C Z A O".

### Output

#### Features extraction
The model returns the embedding as an array of floats.

# Copyright

Code derived from https://github.com/agemagician/ProtTrans is licensed under the MIT License, Copyright (c) 2025 Ahmed Elnaggar. The ProtTrans pretrained models are released under the under terms of the [Academic Free License v3.0 License](https://choosealicense.com/licenses/afl-3.0/), Copyright (c) 2025 Ahmed Elnaggar. The other code is licensed under the MIT license, Copyright (c) 2025 Maksim Pavlov.