TinyStories-17M is a 17.2 million parameter decoder-only Transformer trained entirely from scratch for children's story generation.
The project explores how far a modern Small Language Model (SLM) can be pushed using high-quality synthetic data, an efficient Transformer architecture, and careful training. Despite its compact size, the model is capable of generating coherent, grammatically correct, multi-paragraph children's stories while following a wide variety of prompts.
References
This project is inspired by the TinyStories paper:
TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
The model was trained from scratch using next-token prediction on a synthetic TinyStories-style dataset.
Training Configuration
Setting
Value
Optimizer
AdamW
Precision
BF16 Mixed Precision
Learning Rate
3e-4
Weight Decay
0.1
Epochs
8
Batch Size
70
Context Length
512
Vocabulary
8000
Training was performed using PyTorch with Automatic Mixed Precision (AMP) and a cosine learning rate schedule.
Dataset
The model was trained on approximately 2.1 million synthetic English children's stories inspired by the TinyStories dataset.
The dataset focuses on:
Simple English
Short coherent narratives
Moral lessons
Everyday situations
Dialogues
Fantasy adventures
Animal stories
Child-friendly vocabulary
Intended Use
TinyStories-17M is intended for:
Story generation
Small Language Model research
Educational projects
Transformer architecture experiments
Learning how decoder-only LLMs work
Efficient inference research
Evaluation
The model was evaluated using a combination of:
Validation loss during training
Manual inspection of generated stories
LLM-assisted qualitative evaluation using ChatGPT-5
Evaluation focused on:
Grammar
Story coherence
Prompt following
Creativity
Character consistency
Repetition
Overall story quality
Qualitative Assessment
Category
Score
Grammar
9.7 / 10
Story Coherence
8.7 / 10
Character Consistency
8.6 / 10
Creativity
8.1 / 10
Repetition Resistance
8.0 / 10
Overall Story Quality
8.62 / 10
Note: These scores are qualitative assessments based on manual inspection and LLM-assisted evaluation. They are intended to provide an approximate indication of the model's strengths and limitations rather than standardized benchmark results.
Generation Examples
The following examples are actual generations produced by TinyStories-17M.
Example 1
Prompt
Once upon a time
Generation
text
1there was a little girl who was very curious. One day she decided to
2explore the woods and take a look around. As she walked, the little
3girl heard some strange noises coming from behind some trees.She
4stopped and looked around but couldn't see anything. Suddenly, a
5voice said, "Help me!" The little girl was scared but she asked,
6"Who are you?" The voice replied, "I am a giant. I need to tell you
7that you must leave these woods and not come back." The little girl
8knew she had to do something...
Example 2
Prompt
Ben discovered an old compass hidden inside his grandfather's attic.
Generation
text
1 He decided to explore the attic, but it was very dark and scary. He was
2 scared, but he was determined to find out what was behind the boxes. He
3 opened one of the boxes and found a heavy chest. He tried to lift it,
4 but it was too heavy. He wondered what could be inside. Suddenly, he
5 heard a mysterious voice introducing itself as the spirit of the attic,
6 encouraging Ben to continue exploring and discover new adventures...
Example 3
Prompt
A lonely dragon met a tiny mouse for the first time.
Generation
text
1The dragon wanted to play with the mouse, but the mouse was scared and ran
2away. Hoping to become friends, the dragon searched for the tiny mouse
3throughout the forest. Soon a misunderstanding turned into an exciting
4chase, and both of them learned that first impressions are not always true...
Note: The examples above are intentionally truncated for brevity. The model continues generating complete children's stories beyond these excerpts.
