An 8B-token, cross-source-deduplicated pretraining mixture for OpenAI's
Parameter Golf tiny-LM challenge.
Built to test whether adding code + math to a FineWeb-edu base moves Δbpb on
the official FineWeb validation split at the ~30M-param / 8B-token regime.
Shards are in the trainer's native .bin format (256×int32 header
[magic=20240520, version=1, num_tokens] + uint16 LE tokens, 10^8 tokens/shard),
directly consumable by the reference train_gpt.py.… See the full description on the dataset page:
https://huggingface.co/datasets/abhiram98/paramgolf-mix-8b.
