cdsBERT is a pLM with a codon vocabulary that was seeded with
ProtBERT and trained with a novel vocabulary extension pipeline called MELD. cdsBERT offers a highly biologically relevant latent space with excellent EC number prediction.
Specifically, this is the full-precision checkpoint after the MLM objective on 4 million CDS examples.
1# Imports
2import torch
3from transformers import BertForMaskedLM, BertTokenizer, pipeline
4
5model = BertForMaskedLM.from_pretrained('lhallee/cdsBERT') # load model
6tokenizer = BertTokenizer.from_pretrained('lhallee/cdsBERT') # load tokenizer
7device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu') # gather device
8model.to(device) # move to device
9model.eval() # put in eval mode
10
11sequence = '( Z [MASK] V L P Y G D E K L S P Y G D G G D V G Q I F s C # L Q D T N N F F G A g Q N K % O P K L G Q I G % S K % u u i e d d R i d D V L k n ( T D K @ p p ^ v ]' # CCDS207.1|Hs110|chr1
12
13# Create a fill-mask prediction pipeline
14unmasker = pipeline('fill-mask', model=model, tokenizer=tokenizer)
15# Predict the masked token
16prediction = unmasker(sequence)
17print(prediction)
cdsBERT serves as a general-purpose protein language model with a codon vocabulary. Fine-tuning with Huggingface transformers models like BertForSequenceClassification enables downstream classification and regression tasks. Currently, the base capability enables feature extraction. This checkpoint after MLM can conduct mask-filling, while the cdsBERT+ checkpoint has a more biochemically relevant latent space.
The
Gleghorn lab is an interdisciplinary research group at the University of Delaware that focuses on solving translational problems with our expertise in engineering, biology, and chemistry. We develop inexpensive and reliable tools to study organ development, maternal-fetal health, and drug delivery. Recently we have begun exploration into protein language models and strive to make protein design and annotation accessible.
@article {Hallee_cds_2023,
author = {Logan Hallee, Nikolaos Rafailidis, and Jason P. Gleghorn},
title = {cdsBERT - Extending Protein Language Models with Codon Awareness},
year = {2023},
doi = {10.1101/2023.09.15.558027},
publisher = {Cold Spring Harbor Laboratory},
journal = {bioRxiv}
}
