Model Card for the Preference Optimization Interaction Baseline
A 124M model with the GPT-2 architecture trained with the next token prediction loss for 10 epochs (~100M words), as a naive autoregressive baseline for the Strict-Small track of the 2025 BabyLM challenge.
Table of Contents
Model Details
Model Description
This one of the two Strict-track baselines 2025 BabyLM challenge.
- Developed by: Mustafa Ömer Gül
- Model type: Causal language model
- Language(s) (NLP): eng
- Resources for more information:
Uses
This is a pre-trained language model.
It can be used to evaluate tasks in a zero-shot manner and also can be fine-tuned for downstream tasks.
It can be used for language generation but given its small size and low number of words trained on, do not expect LLM-level performance.
Training Details
Training Data
We used the BabyLM 10M (Strict-Small) dataset for training. It is composed of the following:
| Source | Weight | Domain | Citation | Website | License |
|---|
| BNC | 8% | Dialogue | BNC Consortium (2007) | link | link 1 |
| CHILDES | 29% | Dialogue, Child-Directed | MacWhinney (2000) | | link |
| Project Gutenberg | 26% | Fiction, Nonfiction | Gerlach & Font-Clos (2020) | link | link |
| OpenSubtitles | 20% | Dialogue, Scripted | Lison & Tiedermann (2016) | link | Open source |
| Simple English Wikipedia | 15% | Nonfiction | -- | link | link |
| Switchboard | 1% | Dialogue | Godfrey et al. (1992), Stolcke et al., (2000) | link | link |
1 Our distribution of part of the BNC Texts is permitted under the fair dealings provision of copyright law (see term (2g) in the BNC license).
Hyperparameters
| Hyperparameter | Value |
|---|
| Number of epochs | 10 |
| Datapoint length | 512 |
| Batch size | 16 |
| Learning rate | 0.00005 |
| Number of steps | 21300 |
| Warmup steps | 213 |
| Gradient clipping | 1 |
| Optimizer | AdamW |
| Optimizer Beta_1 | 0.9 |
| Optimizer Beta_2 | 0.999 |
| Optimizer Epsilon | 10-8 |
Training Procedure
The model is trained with the next token prediction loss for 10 epochs.
Size and checkpoints
The model has 124M parameters.
In total we train on around 100M words and provide multiple checkpoints from the training.
Specifically we provode:
- Checkpoints every 1M words for the first 10M words
- Checkpoints every 10M words first 100M words
Evaluation
This model is evaluated in three different fashions:
- We do zero-shot evaluation on 7 tasks.
- We do fine-tuning on a subset of the (Super)GLUE tasks (Wang et al., ICLR 2019; Wang et al., NeurIPS 2019) .
Testing Data & Metrics
Testing Data
For the BLiMP, BLiMP supplement, and EWoK tasks, we use a filtered version of the dataset to only include examples with words found in the BabyLM dataset.
For the Finetuning task, we both filter and sample down to a maximum 10 000 train examples.
Validation Data
Zero-shot Tasks
- BLiMP: The Benchmark of Linguistic Minimal Pairs evaluates the model's linguistic ability by seeing if it can recognize the grammatically correct sentence from a pair of minimally different sentences. It tests various grammatical phenomena.(Warstadt et al., TACL 2020)
- BLiMP Supplement: A supplement to BLiMP introduced in the first edition of the BabyLM challenge. More focused on dialogue and questions. (Warstadt et al., CoNLL-BabyLM 2023)
- EWoK: Works similarly to BLiMP but looks the model's internal world knowledge. Looking at both whter a model has physical and social knowledge. (Ivanova et al., 2024)
- Eye Tracking and Self-paced Reading: Looks at whether the model can mimick the eye tracking and reading time of a human but using surprisal of a word as a proxy for time spent reading a word. (de Varda et al., BRM 2024)
- Entity Tracking: Checks whether a model can keep track of the changes to the states of entities as text/dialogue unfolds. (Kim & Schuster, ACL 2023)
- WUGs: Tests morphological generalization in LMs through an adjective nominalization task. (Hofmann et al., 2024)
Finetuning Tasks
- BoolQ: A yes/no QA dataset with unprompted and unconstrained questions. (Clark et al., NAACL 2019)
- MNLI: The Multi-Genre Natural Language Inference corpus tests the language understanding of a model by seeing wehther it can recognize textual entailment. (Williams et al., NAACL 2018)
- MRPC: The Microsoft Research Paraphrase Corpus contains pairs of sentences that are either paraphrases/semntically equivalent to each other or unrelated.(Dolan & Brockett, IJCNLP 2005)
- QQP2: Similarly to MRPC, the Quora Question Pairs corpus tests the models ability to determine whether a pair of questions are sematically similar to each other. These questions are sourced from Quora.
- MultiRC: The Multi-Sentence Reading Comprehension corpus is a QA task that evaluates the model's ability to the correct answer from a list of answers given a question and context paragraph. In this version the data is changed to a binary classification judging whether the answer to a question, context pair is correct. (Khashabi et al., NAACL 2018)
- RTE: Similar the Recognizing Text Entailement tests the model's ability to recognize text entailement. (Dagan et al., Springer 2006; Bar et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., TAC 2009)
- WSC: The Winograd Schema Challenge tests the models ability to do coreference resolution on sentences with a pronoun and a list of noun phrases found in the sentence. This version edits it to be a binary classification on examples consisting of a pronoun and noun phrase.(Levesque et al., PKRR 2012)
Metrics
The metrics used to evaluate the model are the following:
- Zero-shot
- Accuracy on predicting the correct completion/sentence for BLiMP, BLiMP Supplement, EWoK, Entity Tracking, and WUGs
- Change in R^2 prediction from baseline for Eye Tracking (with no spillover) and Self-paced Reading (1-word spillover)
- Finetuning
- 3 class Accuracy for MNLI
- Binary Accuracy for BoolQ, MultiRC, and WSC
- F1-score for MRPC and QQP
The metrics were chosen based on the advice of the papers the tasks come from.
Hyperparameters
| Hyperparameter | MNLI, RTE, QQP, MRPC | BoolQ, MultiRC | WSC |
|---|
| Learning Rate | 3*10-5 | 3*10-5 | 3*10-5 |
| Batch Size | 32 | 16 | 32 |
| Epochs | 10 | 10 | 30 |
| Weight decay | 0.01 | 0.01 | 0.01 |
| Optimizer | AdamW | AdamW | AdamW |
| Scheduler | cosine | cosine | cosine |
| Warmup percentage | 6% | 6% | 6% |
| Dropout | 0.1 | 0.1 | 0.1 |
Results
Zero-shot
| Task | Metric | Causal Score |
|---|
| BLiMP | Acc | 66.36 |
| BLiMP Supplement | Acc | 57.07 |
| EWoK | Acc | 49.9 |
| Eye Tracking | change in R^2 | 8.66 |
| Self-paced Reading | change in R^2 | 4.34 |
| Entity Tracking | Acc | 13.9 |
| WUGs | Acc | 52.5 |
Finetuning
| Task | Metric | Score |
|---|
| BoolQ | Acc | 66.91 |
| MNLI | Acc | 51.12 |
| MRPC | F1 | 80.77 |
| QQP | F1 | 62.45 |
| MultiRC | Acc | 65.72 |
| RTE | Acc | 56.83 |
| WSC | Acc | 61.54 |
Technical Specifications
Hardware
- 1 H100 GPU was used to train this model.
Software
PyTorch
Training Time
The model took ~0.5 GPU hours to train.
Citation
1@misc{charpentier2025babylmturns3papers,
2 title={BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop},
3 author={Lucas Charpentier and Leshem Choshen and Ryan Cotterell and Mustafa Omer Gul and Michael Hu and Jaap Jumelet and Tal Linzen and Jing Liu and Aaron Mueller and Candace Ross and Raj Sanjay Shah and Alex Warstadt and Ethan Wilcox and Adina Williams},
4 year={2025},
5 eprint={2502.10645},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL},
8 url={https://arxiv.org/abs/2502.10645},
9}
Model Card Authors
Mustafa Ömer Gül
Bibliography
The Winograd Schema Challenge (Levesque et al., PKRR 2012)
