MultilingualUnigramLM/

FineWeb2-10K

10K lines of Latin Script languages from FineWeb2 for lightweight tokenizer adaptation experiments.

Load the entire dataset: from datasets import load_dataset import itertools

ds = load_dataset( "MultilingualUnigramLM/FineWeb2-10K", streaming=True )

Load a specific language: from datasets import load_dataset import itertools

ds_ibo = load_dataset( "MultilingualUnigramLM/FineWeb2-10K", split="ibo_Latn"… See the full description on the dataset page: https://huggingface.co/datasets/MultilingualUnigramLM/FineWeb2-10K.