A description of this dataset can be found at
https://arxiv.org/abs/2305.07759
Copied from roneneldan/TinyStoriesInstruct
Modified with:
import ftfy.bad_codecs
from datasets import Dataset, DatasetDict
train = open('./TinyStories-Instruct-train.txt', 'r', encoding='sloppy-windows-1252').read()
train = train.split('<|endoftext|>')
train = [l.strip() for l in train]
valid = open('./TinyStories-Instruct-valid.txt', 'r', encoding='sloppy-windows-1252').read()
valid = valid.split('<|endoftext|>')… See the full description on the dataset page:
https://huggingface.co/datasets/skeskinen/TinyStories-Instruct-hf.
