nikolina-p/

gpt2base

This is a custom PyTorch GPT-2 model with 124M parameters trained on nikolina-p/gutenberg_flat (3.6B tokens) and nikolina-p/fineweb_10BT_tokenized (10B tokens) datasets, for only one epoch each. Code can be found at this GitHub repository

vocabulary size: 50304,
context length: 1024,
emb dim: 768,
number of heads: 12,
number of layers: 12,
drop_rate: 0.1,