Quantized GGUF model files for
gpt2-small-danish from
KennethTMA GPT-2 model (small version, 124 M parameters) for Danish text generation. The model was not pre-trained from scratch but adapted from the English version.
Test the model using the pipeline from the
🤗 Transformers library:
1from transformers import pipeline
2
3generator = pipeline("text-generation", model = "KennethTM/gpt2-small-danish")
4text = generator("Manden arbejdede som")
5
6print(text[0]["generated_text"])
1from transformers import AutoTokenizer, AutoModelForCausalLM
2
3tokenizer = AutoTokenizer.from_pretrained("KennethTM/gpt2-small-danish")
4model = AutoModelForCausalLM.from_pretrained("KennethTM/gpt2-small-danish")
The model is trained using the Danish part of the
oscar dataset ('unshuffled_deduplicated_da') and a context length of 1024 tokens.
The model weights are initialized from the English
GPT-2 small model with new word token embeddings created for Danish using
WECHSEL.
Initially, only the word token embeddings are trained using 50.000 samples. Finally, the whole model is trained using 1.000.000 samples.
For reference, the model achieves a perplexity of 33.5 on 5.000 random validation samples.
Model training is carried out on an 8 GB GPU.
This is a pre-trained model, for optimal performance it should be finetuned for new tasks.
