Raw tokenized pretraining data for chess models pretraining, stored as
sharded NumPy arrays (shard_XXXX/raw.NNNN.npy).
Sharded raw token arrays: shard_0000/ … shard_0005/, each containing
raw.NNNN.npy files of tokenized text.
Load shards with numpy.load(...) and concatenate as needed for your data loader.
If you use this dataset, please cite it:
@article{pre2post-chess,
title =… See the full description on the dataset page:
https://huggingface.co/datasets/pavelslab-nyu/pretrain_v1_54B.
