This model is a fine-tuned version of gpt2 on a curated set of ∼6,000 SwissProt signal peptides, which include all mammalian signal peptides that have been experimentally verified and deposited on SwissProt. It achieves the following results (negative log likelihood on next-token prediction) on the evaluation set:

Loss: 2.1215

Model description

Base model: https://huggingface.co/docs/transformers/en/model_doc/gpt2.

Intended uses & limitations

Generation of novel mammalian signal peptides to fine-tune the secretion of specific recombinant proteins within a host cell.

Training and evaluation data

a curated set of ∼6,000 SwissProt signal peptides, which include all mammalian signal peptides that have been experimentally verified and deposited on SwissProt. The data was downloaded from the public UniProt database with accession date 12/09/2023.

Training procedure

Causal language modelling (CLM) recipe from HuggingFace's Trainer. Train test split 80/20.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0005
train_batch_size: 32
eval_batch_size: 32
seed: 42
gradient_accumulation_steps: 4
total_train_batch_size: 128
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 531
num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
4.1505	0.99	41	2.7547
2.6987	2.0	83	2.6860
2.7257	2.99	124	2.6648
2.6391	4.0	166	2.6580
2.6872	4.99	207	2.6411
2.6053	6.0	249	2.6419
2.6612	6.99	290	2.6301
2.5815	8.0	332	2.6124
2.6307	8.99	373	2.5928
2.5507	10.0	415	2.5764
2.5782	10.99	456	2.5471
2.4578	12.0	498	2.4726
2.43	12.99	539	2.3949
2.2695	14.0	581	2.3361
2.201	14.99	622	2.2559
1.999	16.0	664	2.1804
1.8843	16.99	705	2.1191
1.6932	18.0	747	2.1015
1.6252	18.99	788	2.1163
1.5326	19.76	820	2.1215

Framework versions

Transformers 4.33.1
Pytorch 2.0.1+cu118
Datasets 2.14.5
Tokenizers 0.13.3

Jacopo-gab/230912GPT2_fine_tuned_SP_GPT2_config_ESM_tokenizer_6kSwissProt_20epochs

230912GPT2_fine_tuned_SP_GPT2_config_ESM_tokenizer_6kSwissProt_20epochs