Views
No views yet
- Layers: 6 transformer blocks
- Attention Heads: 6 heads per layer
- Embedding Dimension: 384
- Hidden Dimension (MLP): 1,536 (4x expansion)
- Activation: GELU
- Normalization: LayerNorm
- Position Embeddings: Learned absolute positions
- Weight Tying: Shared embeddings between input and output| Parameter | Value |
|---|---|
| Training Steps | 100,000 |
| Batch Size | 128 |
| Gradient Accumulation | 2 steps (effective batch size: 256) |
| Learning Rate | 6e-4 |
| LR Schedule | Linear warmup (2,000 steps) + Cosine decay |
| Min LR | 6e-5 |
| Optimizer | AdamW (β1=0.9, β2=0.95, ε=1e-9) |
| Weight Decay | 0.1 |
| Gradient Clipping | 0.5 |
| Dropout | 0.1 |
| Precision | bfloat16 (mixed precision) |
| Context Length | 256 tokens |
pip install torch tiktoken1import torch
2import tiktoken
3from model import GPT, GPTConfig
4
5# Load tokenizer
6enc = tiktoken.get_encoding("gpt2")
7
8# Model configuration
9config = GPTConfig(
10 vocab_size=50257,
11 block_size=256,
12 n_layer=6,
13 n_head=6,
14 n_embd=384,
15 dropout=0.0,
16 bias=True
17)
18
19# Load model
20device = "cuda" if torch.cuda.is_available() else "cpu"
21model = GPT(config).to(device)
22model.eval()
23
24# Load checkpoint
25checkpoint = torch.load("pretrained_tinystories.pt", map_location=device)
26checkpoint = {k.replace("_orig_mod.", ""): v for k, v in checkpoint.items()}
27model.load_state_dict(checkpoint)
28
29# Generate text
30prompt = "Once upon a time, there was a little girl"
31context = torch.tensor(enc.encode(prompt)).unsqueeze(0).to(device)
32
33with torch.no_grad():
34 output = model.generate(
35 context,
36 max_new_tokens=200,
37 temperature=0.8,
38 top_k=40
39 )
40
41generated_text = enc.decode(output[0].tolist())
42print(generated_text)finetune.py for the fine-tuning script.1@misc{tinystories-gpt-slm,
2 author = {maximehip},
3 title = {SmallStories},
4 year = {2025},
5 publisher = {HuggingFace},
6 howpublished = {\url{https://huggingface.co/maximehip/small-stories}},
7}1@article{eldan2023tinystories,
2 title={TinyStories: How Small Can Language Models Be and Still Speak Coherent English?},
3 author={Eldan, Ronen and Li, Yuanzhi},
4 journal={arXiv preprint arXiv:2305.07759},
5 year={2023}
6}