irds/

clueweb09_catb

The clueweb09/catb dataset, provided by the ir-datasets package. For more information about the dataset, see the documentation.

This dataset provides:

docs (documents, i.e., the corpus); count=50,220,423

from datasets import load_dataset

docs = load_dataset('irds/clueweb09_catb', 'docs') for record in docs: record # {'doc_id': ..., 'url': ..., 'date': ..., 'http_headers': ..., 'body': ..., 'body_content_type':… See the full description on the dataset page: https://huggingface.co/datasets/irds/clueweb09_catb.