This dataset is a mini subset of the dataset nikolina-p/gutenberg_flat, created for learning, testing streaming datasets, DDP training, and quick experimentation.
It is made from the first 24 books. The text is tokenized using OpenAI's tiktoken tokenizer. Its structure is adapted for training of autoregressive models in distributed environment: each split contains 8 shards, all shards within a split… See the full description on the dataset page:
https://huggingface.co/datasets/nikolina-p/mini_gutenberg_flat.
