🚀 MiniGPT from Scratch

Features

• ✅ Character-level Tokenizer
• ✅ Token Embeddings
• ✅ Positional Embeddings
• ✅ Multi-Head Self Attention
• ✅ FlashAttention (scaled_dot_product_attention)
• ✅ Feed Forward Network (MLP)
• ✅ Residual Connections
• ✅ Layer Normalization
• ✅ Decoder-only GPT Architecture
• ✅ Autoregressive Text Generation
• ✅ Top-K Sampling
• ✅ Top-P (Nucleus) Sampling
• ✅ Temperature Sampling

Architecture

Input Tokens
      │
      ▼
Token Embedding
      │
      ▼
Position Embedding
      │
      ▼
Transformer Block × 4
      │
      ▼
LayerNorm
      │
      ▼
Linear Head
      │
      ▼
Vocabulary Logits
      │
      ▼
Next Token

Project Structure

MiniGPT-from-Scratch/

├── model.py
├── train.py
├── generate.py
├── utils.py
├── config.py
├── requirements.txt
├── notebook/
├── checkpoints/
├── outputs/
├── images/
└── data/

Dataset

Training

Generate Text

Sample Output

FANH:
A Prusjortus:
Weath the too,
eyes glorls cobsuess fewn.

Future Work

• KV Cache
• Mixed Precision
• LoRA Fine-tuning
• Beam Search
• HuggingFace Integration
• Distributed Training
• Triton Kernels

Author