All data is taken from Fineweb-2 which consists of web-crawls (cutoff April 2024). The exact data can be obtained by running the code in the repository that will be released later.
Sizes of training data in number of UTF-8 characters:
| Language | Size |
|---|
| abt | 6608062 |
| aeb | 49814258 |
| amp | 3737645 |
| ape | 4980833 |
| arb | 3242372164 |
| azj | 3242372164 |
| bak | 486333599 |
| bbc | 35774794 |
| bdq | 41451 |
| bew | 3242372164 |
| bmr | 2439163 |
| bss | 1731272 |
| bzj | 4036823 |
| caf | 909293 |
| cak | 14765286 |
| cmn | 3242372164 |
| cnh | 141157675 |
| crn | 5912477 |
| crs | 88583543 |
| ctd | 28218128 |
| efi | 17408332 |
| ell | 3242372164 |
| eng | 3242372164 |
| ewe | 35627506 |
| gug | 38458467 |
| hau | 3242372164 |
| hil | 123755905 |
| hin | 3242372164 |
| ibo | 333471679 |
| ind | 3242372164 |
| kmr | 3242372164 |
| kua | 14827393 |
| kum | 20332449 |
| lat | 3242372164 |
| lus | 230520812 |
| mps | 6855963 |
| mri | 534077417 |
| mzn | 2835443 |
| nrm | 3242372164 |
| ory | 3242372164 |
| pan | 3242372164 |
| pol | 3242372164 |
| pon | 12919138 |
| qug | 15887852 |
| qxh | 2452330 |
| rus | 3242372164 |
| sat | 4507904 |
| sin | 3242372164 |
| sme | 174411164 |
| smo | 417316517 |
| sna | 359593003 |
| swh | 3242372164 |
| tur | 3242372164 |
| ubu | 7941636 |
| uig | 603363013 |
| vie | 3242372164 |
| wal | 11207859 |
| wol | 18292867 |
| wrk | 2419613 |
| yon | 2356518 |
| zai | 6148374 |
| total | 68719476736 |
contact:
- Rob van der Goot (robv@itu.dk)
