kjv-model: A Tiny GPT-2 Trained on the King James Bible
This is a small causal language model trained from scratch on the full text of the King James Version (KJV) of the Bible. It was built as a learning exercise to understand how to train language models using Hugging Face Transformers and the 🤗 Trainer API.
Despite its small size (only ~4 million parameters), it captures patterns in biblical language and can generate scripture-like text. This model is ideal for educational purposes, experimentation with small-scale language modeling, or exploring how neural networks interpret religious texts.
Model Details
Architecture: Custom GPT-2 (initialized from scratch)
Layers: 4
Attention heads: 4
Embedding dimension: 128
Maximum sequence length: 64 tokens
Vocabulary size: Matches GPT-2 tokenizer (~50k)
Total parameters: 7.23 million
Training data: Full KJV Bible text (one verse per line)
Training objective: Causal language modeling (next-token prediction)
Training Configuration
Epochs: 5
Per-device batch size: 8
Learning rate: 1e-3
Weight decay: 0.01
Warmup steps: 50
Optimizer: AdamW (default in Trainer)
Tokenizer: GPT-2 tokenizer (with pad_token set to eos_token)
Data collator: DataCollatorForLanguageModeling with mlm=False
How to Use
You can load and run inference with this model using the Hugging Face Transformers library:
python
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("cpine505/kjv-model")
model = AutoModelForCausalLM.from_pretrained("cpine505/kjv-model")
input_text = "In the beginning"
inputs = tokenizer(input_text, return_tensors="pt", truncation=True, max_length=64)
outputs = model.generate(
inputs["input_ids"],
max_length=100,
do_sample=True,
top_k=50,
temperature=0.9,
pad_token_id=tokenizer.eos_token_id
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
License
Apache 2.0
Acknowledgements
Built with Hugging Face Transformers
Training data: King James Version of the Bible (public domain)
Inspired by Hugging Face documentation and community tutorials
