The hc4/fa dataset, provided by the ir-datasets package.
For more information about the dataset, see the documentation.
This dataset provides:
docs (documents, i.e., the corpus); count=486,486
from datasets import load_dataset
docs = load_dataset('irds/hc4_fa', 'docs')
for record in docs:
record # {'doc_id': ..., 'title': ..., 'text': ..., 'url': ..., 'time': ..., 'cc_file': ...}
Note that calling load_dataset will… See the full description on the dataset page:
https://huggingface.co/datasets/irds/hc4_fa.
