Pretrained model on protein sequences using a masked language modeling (MLM) objective. The model can be used for protein feature extraction or fine-tuning on downstream biological prediction tasks. The model was developed by Ahmed Elnaggar et al. and more information can be found on the GitHub repository and in the accompanying paper. This repository is a fork of their HuggingFace repository. This model is trained on uppercase amino acids: it only works with capital letter amino acids.
Model versions
ProtT5-XL-UniRef50: Based on the t5-3b architecture and pretrained on UniRef50, a dataset of ~45 million protein sequences.
Long description
ProtT5-XL-UniRef50 is based on the t5-3b model and was pretrained on a large corpus of protein sequences in a self-supervised fashion. This means it was pretrained on the raw protein sequences only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those protein sequences.
One important difference between this T5 model and the original T5 version is the denosing objective. The original T5-3B model was pretrained using a span denosing objective, while this model was pre-trained with a Bart-like MLM denosing objective. The masking probability is consistent with the original T5 training by randomly masking 15% of the amino acids in the input.
It has been shown that the features extracted from this self-supervised model (LM-embeddings) captured important biophysical properties governing protein shape. This implied learning some of the grammar of the language of life realized in protein sequences.
Intended uses & limitations
The model could be used for protein feature extraction or to be fine-tuned on downstream tasks. We have noticed in some tasks one can gain more accuracy by fine-tuning the model rather than using it as a feature extractor. We have also noticed that for feature extraction, it is better to use the feature extracted from the encoder not from the decoder.
Description: List of uppercase protein sequences, each with variable length m.
Input format:
Shape:[n], where n is the number of sequences
Data format:[str]
Example:
sequence_examples = ["PRTEINO", "SEQWENCE"]
Preprocessing:
Uppercase normalization.
Rare or undetermined amino acids "U, Z, O, B" are mapped to "X".
Tokenization: spaces are inserted between amino acids, and special tokens are added.
Model
Modality: Protein sequences
Scale: Per protein sequence
Description:
The model maps amino acid sequences to continuous embeddings (1024 dimensions per residue).
The encoder representations can be averaged to obtain a per-protein embedding.
Embeddings capture biophysical and structural information such as secondary structure, localization, and membrane properties.
Training data:
Pretrained on UniRef50, a non-redundant dataset of ~45 million protein sequences.
Output
Description: Each amino acid is represented by a 1024-dimensional vector.
Output format: tensor
Shape:[n, max_seq_len, 1024] with max_seq_len the longest sequences (max of m)
Data format:(float)
Postprocessing:
Per-protein embeddings are obtained by averaging the residue embeddings across the sequence.
Averaged embedding size: [n, 1024]
Installation
Install the conda environment with all dependencies:
bash
1# Create the conda environment called virtual-human-chc-prottrans2conda env create -f environment.yaml
34# Activate the environment5conda activate virtual-human-chc-prottrans
Example
Feature extraction example
python
1from transformers import T5Tokenizer, T5EncoderModel
2import torch
3import re
45device = torch.device('cuda:0'if torch.cuda.is_available()else'cpu')67# Load the tokenizer8tokenizer = T5Tokenizer.from_pretrained('virtual-human-chc/prot_t5_xl_uniref50', do_lower_case=False)910# Load the model11model = T5EncoderModel.from_pretrained('virtual-human-chc/prot_t5_xl_uniref50').to(device)1213# only GPUs support half-precision currently; if you want to run on CPU use full-precision (not recommended, much slower)14if device == torch.device("cpu"):15 model.to(torch.float32)1617sequence_examples =["PRTEINO","SEQWENCE"]18# this will replace all rare/ambiguous amino acids by X and introduce white-space between all amino acids19sequence_examples =[" ".join(list(re.sub(r"[UZOB]","X", sequence)))for sequence in sequence_examples]2021# tokenize sequences and pad up to the longest sequence in the batch22ids = tokenizer.batch_encode_plus(sequence_examples, add_special_tokens=True, padding="longest")23input_ids = torch.tensor(ids['input_ids']).to(device)24attention_mask = torch.tensor(ids['attention_mask']).to(device)2526# generate embeddings27with torch.no_grad():28 embedding_repr = model(input_ids=input_ids, attention_mask=attention_mask)2930# extract embeddings for the first ([0,:]) sequence in the batch while removing padded & special tokens ([0,:7]) 31emb_0 = embedding_repr.last_hidden_state[0,:7]# shape (7 x 1024)32print(f"Shape of per-residue embedding of first sequences: {emb_0.shape}")3334# do the same for the second ([1,:]) sequence in the batch while taking into account different sequence lengths ([1,:8])35emb_1 = embedding_repr.last_hidden_state[1,:8]# shape (8 x 1024)3637# if you want to derive a single representation (per-protein embedding) for the whole protein38emb_0_per_protein = emb_0.mean(dim=0)# shape (1024)3940print(f"Shape of per-protein embedding of first sequences: {emb_0_per_protein.shape}")
References
Ahmed Elnaggar et al., ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing, IEEE TPAMI (2021).
