A 10M parameter transformer language model built and trained entirely from scratch using PyTorch.
Architecture: Causal self-attention, positional embeddings, LayerNorm, GELU activations, weight tying between embedding and output layers.
Training: 5000 iterations on 50k TinyStories examples, Kaggle P100 GPU, cosine LR schedule with linear warmup. Final validation loss: 3.99.
Note: Coherence degrades over longer sequences due to the 64-token context window — expected behaviour at this scale. The model has learned grammar, story structure, and common TinyStories patterns (characters, dialogue, simple morals).
