See our paper at REDACTED for details on our method.
This is a model submitted to the strict-small track of the 2025 BabyLM challenge.
This model was trained to demonstrate the effectiveness of a novel curriculum learning method over training in random order.
We utilize
this dataset built from the following existing ones:
We extract training data influence estimates from models trained in random order, and sort the training data based on that information with various strategies detailed in the paper.
This is the overall best performing model in our experiments, trained in order of increasing influence and re-weighted with lognormal filter, see the paper for details.
We employ a novel curriculum learning strategy in which the model is trained in non-random order with a total of 100M words.
We use
this evaluation pipeline of the 2025 BabyLM challange
