Dataset Preprocessing for 10M and 100M Text-Only Tracks




	
	Overview

This document describes the preprocessing steps applied to the datasets used for the 10M and 100M text-only tracks. The datasets are a mixture of 10 different corpora, as shown in Table 1 below.

	Table 1: Dataset Contents




	

Dataset Domain

Proportion

CHILDES (MacWhinney, 2000) Child-directed speech 0.44M 4.21M 5%

British National Corpus (BNC), dialogue… See the full description on the dataset page: https://huggingface.co/datasets/SrikrishnaIyer/Babylm-processed-2023.