You can deploy this model and start earning money today!

BlackHawk616/
Genesis

text-generation

HuggingFaceTB/cosmopedia-100k

roneneldan/TinyStories

1706.03762

2305.07759

mit

Views

No views yet

🧠 Genesis — GPT-2 From Scratch

A 163M parameter language model built from scratch using PyTorch. No HuggingFace transformers. No pretrained weights. Every layer written by hand.

Python

PyTorch

License

What is this?

Genesis is a GPT-2 language model implemented entirely from scratch — from the multi-head attention mechanism to the byte-pair tokenizer wrapper to the production training loop. This is my first deep learning project, built to truly understand how large language models work under the hood.

"The French Revolution began in 1866. The Spanish were the last
two-year-old boyhood – the Spanish-speaking War of 1878..."

— Genesis at step 91k. Grammatically fluent, factually hilarious.

Architecture

Genesis follows the GPT-2 architecture (Radford et al., 2019):

Token IDs ──→ Token Embedding + Positional Embedding ──→ Dropout
                              │
                              ▼
                  ┌───────────────────────┐
                  │  × 12 Transformer     │
                  │  Blocks (Pre-LN):     │
                  │                       │
                  │  LayerNorm → MHA      │  ← Causal masked multi-head attention
                  │  + Residual           │
                  │                       │
                  │  LayerNorm → FFN      │  ← GELU activation, 4× expansion
                  │  + Residual           │
                  └───────────────────────┘
                              │
                              ▼
                  Final LayerNorm → Linear Head → Logits

Model Specs

Parameter	Value
Parameters	163,009,536 (~163M)
Vocabulary	50,257 (GPT-2 BPE via tiktoken)
Context Length	1,024 tokens
Embedding Dim	768
Attention Heads	12
Transformer Layers	12
FFN Hidden Dim	3,072 (4 × 768)
Activation	GELU (tanh approximation)
Normalization	Pre-LayerNorm

Why 163M instead of 124M? GPT-2 ties its input embedding and output head weights (saving ~38.6M params). Genesis keeps them separate — a deliberate choice to understand the full parameter space.

Training Pipeline

Genesis is trained in three phases:

Phase 1                    Phase 2                         Phase 3
WikiText-103        Cosmopedia-100k + TinyStories       Chat Fine-tune
~130M tokens              ~515M tokens                    (Future)
──────────────►  ──────────────────────────────►  ──────────────────►
  Foundation          Diverse knowledge +            Conversational
  knowledge          narrative coherence               behavior

Phase 1 — WikiText-103 (Complete)

Detail	Value
Dataset	Salesforce/wikitext-103-v1
Tokens	~130M
GPU	Tesla T4 (15GB VRAM)
Training	3 epochs, cosine LR (3e-4 → 1e-5), AMP FP16, grad accum = 8
Best Val Loss	~3.29 (perplexity ≈ 27)

Phase 2 — Cosmopedia + TinyStories (In Progress)

Detail	Value
Datasets	Cosmopedia-100k (~50M tokens) + TinyStories (~465M tokens)
Purpose	Diverse educational content + narrative coherence (Microsoft Research designed TinyStories specifically for small LMs)
LR	1e-4 (lower to preserve WikiText knowledge)
Forgetting Check	WikiText val loss monitored every 500 steps

Phase 3 — Chat Fine-tuning (Future)

Instruction fine-tuning on a conversational dataset to teach the model turn-taking and Q&A behavior.

Project Structure

Genesis-124M/
├── model.py                  # GPTModel, TransformerBlock, LayerNorm, FeedForward, GELU
├── attention.py              # MultiHeadAttention with causal masking
├── embeddings.py             # Token + Positional embeddings (educational reference)
├── tokenizer.py              # tiktoken BPE wrapper + educational SimpleTokenizerV2
├── data.py                   # GPTDataset, sliding-window DataLoader factory
├── clean_data.py             # WikiText preprocessing (headers, @-@, <unk>, HTML entities)
│
├── train.py                  # Production training loop — AMP, cosine LR, grad accum,
│                             #   gradient clipping, checkpointing, CSV logging
├── run_wiki.py               # Local CPU/MPS training runner (with thermal throttling)
│
├── finetune_cosmopedia.py    # Phase 2: Cosmopedia + TinyStories continued pretraining
│                             #   with memory-efficient dataset + WikiText forgetting check
│
├── generate.py               # Inference — temperature, top-k, top-p (nucleus) sampling
│                             #   + auto config detection from checkpoint weights
├── app.py                    # Flask web UI — retro CRT terminal for interactive generation
│
├── colab_124m.ipynb          # Colab notebook for Phase 1 (WikiText) training
├── colab_cosmopedia.ipynb    # Colab notebook for Phase 2 (Cosmopedia + TinyStories)
├── kaggle_resume_train.ipynb # Kaggle training notebook
│
└── genesis_hall_of_frame.txt # Sample outputs from the model at various checkpoints

Quick Start

Requirements

pip install torch tiktoken datasets flask

Generate text with a trained model

bash

python generate.py \
    --ckpt best_model.pt \
    --prompt "The history of Rome" \
    --tokens 100 \
    --temp 0.8 \
    --topk 40

Run the web UI

bash

python app.py
# Open http://localhost:5000

Train from scratch (locally)

bash

# Small model (~27M params) for testing
python run_wiki.py

# Full 163M model — use Colab with GPU
# See colab_124m.ipynb

Train on Colab (recommended)

Open colab_124m.ipynb in Google Colab
Set runtime to T4 GPU
Run cells 1 → 8
Training auto-resumes from Drive checkpoints across sessions

Training Features

The training pipeline includes:

Mixed Precision (AMP) — FP16 on CUDA with TF32 enabled
Cosine LR Schedule — Linear warmup → cosine decay
Gradient Accumulation — Effective batch size of 32 on T4 (batch=4 × accum=8)
Gradient Clipping — clip_grad_norm_(params, 1.0)
Auto-resume — Checkpoints save to Google Drive every 500 steps; survives Colab disconnects
CSV Logging — Step, loss, LR, tokens/sec — persists across sessions
Sample Generation — Prints model outputs during training for qualitative monitoring
WikiText Forgetting Check (Phase 2) — Monitors whether new data causes catastrophic forgetting

Sample Outputs

At Step 91,000 (Phase 1 — WikiText)

>> The French Revolution began in
The French Revolution began in 1866. The Spanish were the last 
two-year-old boyhood – the Spanish-speaking War of 1878.

>> Albert Einstein discovered that
Albert Einstein discovered that he was pregnant with the car. 
According to the New York Times, "the first time a young man 
would have been the first time he had a"...

The model has learned fluent English grammar and Wikipedia-style structure — but factual accuracy is, shall we say, creative. This is expected for a 163M model at this training stage.

Key Implementation Details

What I built from scratch

Multi-head causal self-attention with parallel head processing via reshape (not head loops)
Pre-LayerNorm transformer blocks matching GPT-2 paper
Tanh-approximated GELU activation
Sliding-window dataset with configurable stride for next-token prediction
Production training loop with all the bells and whistles (AMP, cosine LR, gradient accumulation, checkpointing)
Inference engine with temperature, top-k, and top-p (nucleus) sampling
Auto config detection — generate.py inspects checkpoint weight dimensions to reconstruct model config without needing the config dict

What I used (not built from scratch)

tiktoken — OpenAI's BPE tokenizer (GPT-2 vocabulary)
PyTorch — Tensor operations, nn.Module, nn.Embedding, nn.Linear, autograd
Flask — Web server for the interactive UI
HuggingFace datasets — Dataset downloading for Phase 2

Lessons Learned

Data quality > quantity — WikiText-103 alone teaches grammar but not coherent multi-sentence output. Adding TinyStories dramatically improves narrative flow.
Small models hallucinate confidently — 163M parameters is enough for fluent English but not for factual accuracy. Einstein being "pregnant with the car" is a feature, not a bug (at this scale).
Checkpoint everything — Colab disconnects are inevitable. Saving to Drive every 500 steps saved me dozens of hours.
Weight tying matters — The difference between 124M and 163M params is entirely the un-tied embedding weights. Something to fix in v2.
Learning rate is everything — Phase 2 uses 1e-4 (not 3e-4) to avoid catastrophic forgetting of Phase 1 knowledge.

Roadmap

Phase 1: WikiText-103 pretraining (~130M tokens)
Phase 2: Cosmopedia-100k + TinyStories (~515M tokens)
Phase 3: Conversational fine-tuning
Weight tying (163M → 124M params)
FlashAttention / PyTorch SDPA integration
KV Cache for faster inference
torch.compile() optimization

Hardware Used

Phase	GPU	VRAM Used	Training Time
Phase 1	Tesla T4 (Colab)	~0.71 GB	~10+ Colab sessions
Phase 2	Tesla T4 (Colab)	~0.71 GB	~3-4 sessions (est.)

References

Language Models are Unsupervised Multitask Learners (Radford et al., 2019) — GPT-2 paper
Attention Is All You Need (Vaswani et al., 2017) — Transformer architecture
TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (Eldan & Li, 2023) — Microsoft's small LM dataset
Cosmopedia — HuggingFace synthetic textbook dataset

License

MIT

Built from scratch with curiosity and way too many Colab sessions.

Used Claude AI to refactor this code for making it public (didn't use AI to write the whole code).

BlackHawk616/Genesis

🧠 Genesis — GPT-2 From Scratch

What is this?

Architecture

Model Specs

Training Pipeline

Phase 1 — WikiText-103 (Complete)

Phase 2 — Cosmopedia + TinyStories (In Progress)

Phase 3 — Chat Fine-tuning (Future)

Project Structure

Quick Start

Requirements

Generate text with a trained model

Run the web UI

Train from scratch (locally)

Train on Colab (recommended)

Training Features

Sample Outputs

At Step 91,000 (Phase 1 — WikiText)

Key Implementation Details

What I built from scratch

What I used (not built from scratch)

Lessons Learned

Roadmap

Hardware Used

References

License

BlackHawk616/
Genesis