This model is a fine-tuned version of nferruz/ProtGPT2 on sequences from the NCBI Virus Data Portal.
It achieves the following results on the evaluation set:
Loss: 1.1674
Accuracy: 0.8883
Model description
This model is a fine-tuned checkpoint of
ProtGPT2, which was originally
trained on the UniRef50 (version 2021_04) database. For a detailed overview
of the original model configuration and architecture, please see the linked
model card, or refer to the ProtGPT2 publication.
The model was finetuned on data from the SARS-CoV-2 Spike (surface glycoprotein)
receptor binding domain (RBD).
This model is intended to generate synthetic SARS-CoV-2 surface glycoprotein
(a.k.a. spike protein) sequences for the purpose of identifying meaningful
variants for characterization either experimentally or through other
in silico tools. These variants may be used to drive vaccine develop to
protect against never-before-seen point mutants that are probable in the future.
As this model is based on the original ProtGPT2 model, it is subject to many
of the same limitations as the base model. Any biases present in the UniRef50
dataset will also be present in the model, which may include nonuniform skew
of peptides sampled across different taxonomic clades. These limitations
should be considered when interpreting the output of this model.
Training and evaluation data
SARS-CoV-2 spike protein sequences were obtained from the NIH Sars-CoV-2 Data Hub
accessible at
https://www.ncbi.nlm.nih.gov/labs/virus/vssi/
Note that the reference sequence for the surface glycoprotein can be found at:
https://www.ncbi.nlm.nih.gov/protein/1791269090
As the loaded ProtGPT2 model was pretrained on the
UniRef50 (version 2021_04) dataset, it cannot have contained sequencing
data that was generated after that date. Evaluations will be conducted using
SARS-CoV-2 sequences generated on or after May 2021.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
learning_rate: 1e-05
train_batch_size: 16
eval_batch_size: 16
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
