Views
No views yet
| File | Purpose |
|---|---|
model.py | The full GPT architecture: CausalSelfAttention, MLP, Block, GPT |
prepare.py | Data preparation: character-level tokenization, train/val split |
train.py | Training loop with AdamW, cosine LR schedule, and generation |
input.txt | The tiny Shakespeare dataset (~1.1M characters, 65 unique chars) |
data.pt | Preprocessed tensors (generated by prepare.py) |
best.pt | Best model checkpoint (generated by train.py) |
GPT(
wte (Embedding): vocab_size -> n_embd (token embeddings)
wpe (Embedding): block_size -> n_embd (position embeddings)
h (6x Block):
ln_1 (LayerNorm)
attn (CausalSelfAttention: multi-head self-attention with causal mask)
ln_2 (LayerNorm)
mlp (MLP: expand 4x -> GELU -> project back)
ln_f (LayerNorm)
lm_head (Linear): n_embd -> vocab_size (next-token prediction)
)1# 1. Prepare data
2python prepare.py
3
4# 2. Train (requires GPU for speed, CPU works too)
5python train.py
6
7# 3. The model will print generated Shakespeare-style text at the end!| Hyperparameter | Value |
|---|---|
| Layers | 6 |
| Heads | 6 |
| Embedding dim | 384 |
| Context length | 256 |
| Batch size | 64 |
| Training steps | 5,000 |
| Optimizer | AdamW (β₁=0.9, β₂=0.95) |
| Learning rate | 1e-3 (cosine decay to 1e-4) |
| Warmup | 200 steps |
| Gradient clip | 1.0 |