This directory contains the dataset download helpers and export scripts used for the challenge.
Canonical local layout:
data/datasets/<dataset_name>/
data/tokenizers/
data/manifest.json
data/docs_selected.jsonl
data/docs_selected.source_manifest.json
Download the cached FineWeb export for a tokenizer variant with:
python3 data/cached_challenge_fineweb.py --variant sp1024
This populates ./data/datasets/fineweb10B_sp1024/ and… See the full description on the dataset page:
https://huggingface.co/datasets/TCares/parameter-golf-data.
