This is a pretokenized dump of ffv4_dataset_test/score0.8 for use with llm-foundry.
It partitions stories from the dataset such that each data sample always looks like this:
<story info metadata ...>
where
and are special tokens in my edited mpt-7b-tokenizer, the story metadata is just the value of the info column from the ffv4 dataset, and story chunks are obtained by splitting the story for that row into groups of tokens… See the full description on the dataset page:
https://huggingface.co/datasets/main-horse/ffv4-test-4.
