You can deploy this model and start earning money today!

mahanthisaikiran/
gurudev-bajana-v1

Name: gurudev-bajana-v1
Availability: InStock

mit

Views

No views yet

SriSriLM-Bhajan-v1

A minimal educational GPT-style language model trained on Art of Living Bhajan content, satsangs, guru stotrams, devotional songs, and spiritual lyrics related to Gurudev Sri Sri Ravi Shankar.

Project Overview

This project builds a small decoder-only GPT architecture from scratch in PyTorch. It uses a Byte Pair Encoding (BPE) tokenizer and trains on text extracted from a Bhajan PDF.

Key components

data/bhajan_lyrics.pdf - source PDF containing bhajan lyrics
data/train.txt - extracted plain text used for tokenizer training and model training
tokenizer/tokenizer.json - BPE tokenizer artifact
checkpoints/srisrilm.pt - saved PyTorch model weights
extract_text.py - PDF text extraction pipeline
train_tokenizer.py - BPE tokenizer training script
model.py - GPT decoder architecture implementation
train.py - dataset, training loop, optimizer, scheduler, and checkpoint saving
generate.py - inference script for prompt-based generation

Architecture

The model is a small GPT-style decoder-only transformer with:

vocab_size = 5000
embedding_dim = 128
num_heads = 4
num_layers = 4
context_length = 128
dropout = 0.1

Implemented features:

Token embeddings
Positional embeddings
Multi-head self-attention
Causal masking
Feed-forward networks
Layer normalization
Residual connections
Transformer blocks
GPT decoder head

Training Instructions

Install dependencies:

pip install -r requirements.txt

Place your Bhajan PDF in data/bhajan_lyrics.pdf.
Extract the PDF text into data/train.txt:

python extract_text.py

Train the tokenizer:

python train_tokenizer.py

Train the GPT model:

python train.py

The training script saves the model checkpoint to checkpoints/srisrilm.pt.

Inference Instructions

Run generation using the saved checkpoint and tokenizer:

python generate.py

The script will generate continuations for the prompts:

Guru Brahma
Jaya Guru
Om Namah Shivaya
Radhe Govind
Gurudev Bolo

Hugging Face Deployment Steps

Create an account on Hugging Face if you do not have one.
Install the Hugging Face Hub client:

pip install huggingface_hub

huggingface-cli login

Create a repository for SriSriLM-Bhajan-v1 on Hugging Face.
Upload the model checkpoint and tokenizer files. Example using Python:

python

from huggingface_hub import HfApi, Repository
from pathlib import Path

repo_id = "<username>/SriSriLM-Bhajan-v1"
local_dir = Path(".")

# Initialize and push the repo
repo = Repository(local_dir=local_dir, clone_from=repo_id)
repo.push_to_hub(commit_message="Upload SriSriLM-Bhajan-v1 checkpoint and tokenizer")

Add the following files to the repo:

checkpoints/srisrilm.pt
tokenizer/tokenizer.json
README.md
requirements.txt
Optional: model.py, train.py, generate.py

Document the model card and usage in the Hugging Face repository.

Notes

This code is intended for educational purposes, demonstrating how to build a small decoder-only transformer from scratch and train it on devotional bhajan text. The model is named SriSriLM-Bhajan-v1 and can be adapted for further fine-tuning or deployment.

mahanthisaikiran/gurudev-bajana-v1