Spark Mini-GPT
A small GPT model I built from scratch in PyTorch, trained on a short repeated story about a robot named Spark.
Architecture
- 12-layer transformer
- 768-dim embeddings, 12 attention heads
- GPT-2 tokenizer (50,257 vocab)
Training
- ~7,600 tokens (a short story repeated 50 times)
- 10 epochs with AdamW
- Final loss: ~0.007
Why I made this
This was a class project to learn how GPT models actually work
