Model description

The model is a demo for the fine-tuning of decoder-only models using transformers library.

Intended uses & limitations

It can be used mainly for prototyping and educational purposes.

Training and evaluation data

The demelin/understanding_fables dataset has been split into train/test/validation using an 80/10/10 random split (random_seed = 42). Google Colab has been used for model fine-tuning.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 5e-05
train_batch_size: 4
eval_batch_size: 4
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 500
num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	38	42.4563
No log	2.0	76	5.2808
28.753	3.0	114	3.7712
28.753	4.0	152	3.4577
28.753	5.0	190	3.3120
3.5846	6.0	228	3.2773
3.5846	7.0	266	3.2710
3.0017	8.0	304	3.2764
3.0017	9.0	342	3.2795
3.0017	10.0	380	3.3300

Framework versions

Transformers 4.22.1
Pytorch 1.12.1+cu113
Datasets 2.4.0
Tokenizers 0.12.1

morenolq/distilgpt2-fables-demo

distilgpt2-fables-demo