This is a bilingual GPT-2 style model. For the first half of training, this model was trained only on English data. In the second half of training, the model was trained on only Dutch data. At the end of training, 50% of training data seen by the model is English and 50% is Dutch. The tokenizer was trained on the same overall proportions of data as the language model at the final step.
All models are trained with a [CLS] (same as [BOS]) token prepended, and a [SEP] (same as [EOS]) token separating sequences.
For best results, make sure that [CLS] is prepended to your input sequence (see sample usage linked above)!
Details for this model specifically:
Note: if you do not specify a revision, it will load the final checkpoint of the model. See above for the list of checkpoints. The checkpoint step is the name of the revision.
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("catherinearnett/B-GPT_en_nl_sequential")
model = AutoModelForCausalLM.from_pretrained("catherinearnett/B-GPT_en_nl_sequential", revision = "128000")
Text Generation:
from transformers import pipeline
pipe = pipeline("text-generation", model="catherinearnett/B-GPT_en_nl_sequential")
print(pipe("I am a", max_length=20)[0]["generated_text"])
Citation
If you use this model, please cite:
@article{arnett2025acquisition,
title={On the Acquisition of Shared Grammatical Representations in Bilingual Language Models},
author={Arnett, Catherine and Chang, Tyler A and Michaelov, James A and Bergen, Benjamin K},
journal={arXiv preprint arXiv:2503.03962},
year={2025}
}
