something-else/

Slim-binidx

This dataset comprises nine chunks (out of ten) from the Cerebras/SlimPajama-627B dataset, processed into a binary index (bin idx) format. The first chunk is located at : rwkv-x-dev/slimpajama-binidx. Due to their large size, each chunk is split into multiple parts for easier handling. To reassemble and decompress these parts, follow these steps:

Combine all parts of the desired chunk into a single file:

cat chunk2_text_document_part_* > chunk2_text_document.tar.xz

Decompress the combined… See the full description on the dataset page: https://huggingface.co/datasets/something-else/Slim-binidx.