irds/

wikir_en78k

The wikir/en78k dataset, provided by the ir-datasets package. For more information about the dataset, see the documentation.

This dataset provides:

docs (documents, i.e., the corpus); count=2,456,637

from datasets import load_dataset

docs = load_dataset('irds/wikir_en78k', 'docs') for record in docs: record # {'doc_id': ..., 'text': ...}

Note that calling load_dataset will download the dataset (or provide access… See the full description on the dataset page: https://huggingface.co/datasets/irds/wikir_en78k.