A minimal educational GPT-style language model trained on Art of Living Bhajan content, satsangs, guru stotrams, devotional songs, and spiritual lyrics related to Gurudev Sri Sri Ravi Shankar.
Project Overview
This project builds a small decoder-only GPT architecture from scratch in PyTorch. It uses a Byte Pair Encoding (BPE) tokenizer and trains on text extracted from a Bhajan PDF.
Key components
data/bhajan_lyrics.pdf - source PDF containing bhajan lyrics
data/train.txt - extracted plain text used for tokenizer training and model training
tokenizer/tokenizer.json - BPE tokenizer artifact
checkpoints/srisrilm.pt - saved PyTorch model weights
extract_text.py - PDF text extraction pipeline
train_tokenizer.py - BPE tokenizer training script
train.py - dataset, training loop, optimizer, scheduler, and checkpoint saving
generate.py - inference script for prompt-based generation
Architecture
The model is a small GPT-style decoder-only transformer with:
vocab_size = 5000
embedding_dim = 128
num_heads = 4
num_layers = 4
context_length = 128
dropout = 0.1
Implemented features:
Token embeddings
Positional embeddings
Multi-head self-attention
Causal masking
Feed-forward networks
Layer normalization
Residual connections
Transformer blocks
GPT decoder head
Training Instructions
Install dependencies:
pip install -r requirements.txt
Place your Bhajan PDF in data/bhajan_lyrics.pdf.
Extract the PDF text into data/train.txt:
python extract_text.py
Train the tokenizer:
python train_tokenizer.py
Train the GPT model:
python train.py
The training script saves the model checkpoint to checkpoints/srisrilm.pt.
Inference Instructions
Run generation using the saved checkpoint and tokenizer:
python generate.py
The script will generate continuations for the prompts:
Guru Brahma
Jaya Guru
Om Namah Shivaya
Radhe Govind
Gurudev Bolo
Notes
This code is intended for educational purposes, demonstrating how to build a small decoder-only transformer from scratch and train it on devotional bhajan text. The model is named SriSriLM-Bhajan-v1 and can be adapted for further fine-tuning or deployment.
