irds/

clueweb09_es

The clueweb09/es dataset, provided by the ir-datasets package. For more information about the dataset, see the documentation.

This dataset provides:

docs (documents, i.e., the corpus); count=79,333,950

from datasets import load_dataset

docs = load_dataset('irds/clueweb09_es', 'docs') for record in docs: record # {'doc_id': ..., 'url': ..., 'date': ..., 'http_headers': ..., 'body': ..., 'body_content_type': ...}… See the full description on the dataset page: https://huggingface.co/datasets/irds/clueweb09_es.