CC100 dataset comprises of monolingual data for 100+ languages and also includes data for romanized languages.
This was constructed using the urls and paragraph indices provided by the CC-Net repository
by processing January-December 2018 Commoncrawl snapshots.
Each file comprises of documents separated by double-newlines and paragraphs within the same document separated by a newline.
The data is generated using the open source CC-Net repository.
This dataset loader implements streaming to iterate over CC100 dataset.
It applies strict filtering criteria to remove short, noisy, or repetitive sentences
and keeps the language proportions similar to the ones used for XLM-R pre-training.
Notably, this only loads minimal numbers of examples per language for a quick access to distillation data.
