cdsBERT+ is a pLM with a codon vocabulary that was seeded with
ProtBERT and trained with a novel vocabulary extension pipeline called MELD. cdsBERT+ offers a highly biologically relevant latent space with excellent EC number prediction surpassing ProtBERT.
Specifically, this is the half-precision checkpoint after student-teacher knowledge distillation with Ankh-base.
1# Imports
2import re
3import torch
4import torch.nn.functional as F
5from transformers import BertModel, BertTokenizer
6
7model = BertModel.from_pretrained('lhallee/cdsBERT') # load model
8tokenizer = BertTokenizer.from_pretrained('lhallee/cdsBERT') # load tokenizer
9device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu') # gather device
10model.to(device) # move to device
11model.eval() # put in eval mode
12
13sequence = '(ZEVLPYGDEKLSPYGDGGDVGQIFsC#LQDTNNFFGAgQNK%OPKLGQIG%SK%uuieddRidDVLkn(TDK@pp^v]' # CCDS207.1|Hs110|chr1
14sequence = ' '.join(list(sequence)) # need spaces in-between codons
15
16example = tokenizer(sequence, return_tensors='pt', padding=False).to(device) # tokenize example
17with torch.no_grad():
18 matrix_embedding = model(**example).last_hidden_state.cpu()
19
20vector_embedding = matrix_embedding.mean(dim=0)
cdsBERT+ serves as a general-purpose protein language model with a codon vocabulary. Fine-tuning with Huggingface transformers models like BertForSequenceClassification enables downstream classification and regression tasks. Currently, the base capability enables feature extraction. The based checkpoint after MLM, cdsBERT, can conduct mask-filling.
The
Gleghorn lab is an interdisciplinary research group at the University of Delaware that focuses on solving translational problems with our expertise in engineering, biology, and chemistry. We develop inexpensive and reliable tools to study organ development, maternal-fetal health, and drug delivery. Recently we have begun exploration into protein language models and strive to make protein design and annotation accessible.
@article {Hallee_cds_2023,
author = {Logan Hallee, Nikolaos Rafailidis, and Jason P. Gleghorn},
title = {cdsBERT - Extending Protein Language Models with Codon Awareness},
year = {2023},
doi = {10.1101/2023.09.15.558027},
publisher = {Cold Spring Harbor Laboratory},
journal = {bioRxiv}
}
