NepaliGPT: Nepali Language Generative Pretrained Transformer Model
This is an experiment for developing a language generation model for the Nepali language.
Causal Language Model which can predict the next possible tokens given a context in Nepali language.
Dataset Used
A large corpus of 9.3 GB size has been collected from different sources on the internet. The sources include
Nepali Books found online.
Nepali News Article from Nepali news portals.
Nepali text collected from different open source Nepali NLP datasets.
Hyperparameters Used
Learning rate -> 2e-5
Weight Decay -> 0.01
Number of training epochs -> 5 \
bf16 -> True
Base Model Architecture -> GPT-2 \
Training Results
It achieves the following results on the evaluation set:
