This is a bilingual GPT-2 style model. For the first half of training, this model was trained only on English data. In the second half of training, the model was trained on a 50%-50% mix of English and Greek data. At the end of training, 75% of training data seen by the model is English and 25% is Greek. The tokenizer was trained on the same overall proportions of data as the language model at the final step.
All models are trained with a [CLS] (same as [BOS]) token prepended, and a [SEP] (same as [EOS]) token separating sequences.
For best results, make sure that [CLS] is prepended to your input sequence (see sample usage linked above)!
Details for this model specifically:
Note: if you do not specify a revision, it will load the final checkpoint of the model. See above for the list of checkpoints. The checkpoint step is the name of the revision.
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("catherinearnett/B-GPT_en_nl_sequential")
model = AutoModelForCausalLM.from_pretrained("catherinearnett/B-GPT_en_nl_sequential", revision = "128000")
Text Generation:
from transformers import pipeline
pipe = pipeline("text-generation", model="catherinearnett/B-GPT_en_nl_sequential")
print(pipe("I am a", max_length=20)[0]["generated_text"])
Citation
If you use this model, please cite:
@article{arnett2025acquisition,
title={On the Acquisition of Shared Grammatical Representations in Bilingual Language Models},
author={Arnett, Catherine and Chang, Tyler A and Michaelov, James A and Bergen, Benjamin K},
journal={arXiv preprint arXiv:2503.03962},
year={2025}
}
