GPT-2 model for Text Generation in luxembourgish language, trained on 711 MB of text data, consisting of RTL.lu news articles, comments, parlament speeches, the luxembourgish Wikipedia, Newscrawl, Webcrawl and subtitles. Created via transfer learning with an German base model, feature space mapping from LB on Base feature space and gradual layer freezing. The training took place on a 32 GB Nvidia Tesla V100

with One Cycle policy for the learning rate
with the help of fastai's LR finder
for 53.4 hours
for 20 epochs and 7 cycles
using the fastai library

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("laurabernardy/LuxGPT2-basedGER")

model = AutoModelForCausalLM.from_pretrained("laurabernardy/LuxGPT2-basedGER")

Limitations and Biases

See the GPT2 model card for considerations on limitations and bias. See the GPT2 documentation for details on GPT2.