A description of this dataset can be found at
https://arxiv.org/abs/2305.07759
Copied from roneneldan/TinyStories
Modified with:
import ftfy.bad_codecs
from datasets import Dataset, DatasetDict
train = open('./TinyStories-train.txt', 'r', encoding='sloppy-windows-1252').read()
train = train.split('<|endoftext|>')
train = [l.strip() for l in train]
valid = open('./TinyStories-valid.txt', 'r', encoding='sloppy-windows-1252').read()
valid = valid.split('<|endoftext|>')
valid = [l.strip() for l… See the full description on the dataset page:
https://huggingface.co/datasets/skeskinen/TinyStories-hf.
