FolkGPT
This model is a fine-tuned version of
gpt2 on vicclab/fairy_tales dataset.
Model description
This model is the result of fine-tuning gpt2 on a dataset of fairy tales from various cultures.
Intended uses & limitations
The idea behind this is to generate text in the fashion of fairy tales written in the 18th and 19th centuries.
Why? Fairy tales seemed an appropriate application for text generation, as stories are usually short(ish),
self-contained, and easy to read.
Training and evaluation data
Trained on the vicclab/fairy_tales dataset. The dataset consists of a number of texts which
were downloaded from Project Gutenberg, and then edited to remove all text except for the
stories themselves. These were then all concatenated into a text file and pushed to HF at
https://huggingface.co/datasets/vicclab/fairy_tales. The latest update to the dataset, which
was used in the training of this model, was created and uploaded on February 26th, 2023.
Texts used [and token count after removing boilerplate text]:
https://www.gutenberg.org/files/2591/2591-0.txt [102927 tokens]
https://www.gutenberg.org/files/503/503-0.txt [138353 tokens]
https://www.gutenberg.org/cache/epub/69739/pg69739.txt [51035 tokens]
https://www.gutenberg.org/files/2435/2435-0.txt [98791 tokens]
https://www.gutenberg.org/cache/epub/7871/pg7871.txt [49410 tokens]
https://www.gutenberg.org/files/8933/8933-0.txt [178622 tokens]
gutenberg.org/cache/epub/30834/pg30834.txt [58359 tokens]
https://www.gutenberg.org/cache/epub/68589/pg68589.txt [39815 tokens]
https://www.gutenberg.org/cache/epub/34453/pg34453.txt [69365 tokens]
gutenberg.org/cache/epub/8653/pg8653.txt [35351]
[Total tokens in actual dataset: 1002654 tokens]
Training procedure
The dataset was loaded, sampling by paragraph. From here, the dataset was split into a training dataset
and a validation dataset in an 80-20 split. These were then tokenized. The model was set up, and the trainer
was instantiated with the training_arguments listed below. Then, the training took place.
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0005
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 4
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 1000
- num_epochs: 1
- mixed_precision_training: Native AMP
Training results
Framework versions
- Transformers 4.26.1
- Pytorch 1.13.1+cu116
- Datasets 2.10.0
- Tokenizers 0.13.2
