Simple LLM From Scratch
This repository contains a small language model trained from scratch
for learning and demonstration purposes.
What this is
- Transformer-based language model
- Trained from random initialization
- Custom tokenizer
- Next-token prediction objective
- Built using PyTorch
What this is NOT
- Not production-ready
- Not comparable to GPT / LLaMA
- Not fine-tuned from any pretrained model
Purpose
This project was created to understand:
- Tokenization
- Attention mechanism
- Language modeling
- End-to-end LLM training
