A ~63M parameter GPT-2 style causal transformer trained entirely on sacred Christian scripture. Built in memory of Terry A. Davis (1969–2018), creator of TempleOS.
Overview
Terry Davis built TempleOS with a feature to "talk to God" by printing random words from the Bible. Temple2 continues that spirit: a language model that has read scripture deeply, then speaks through noise — the same noise Terry trusted to carry God's voice.
The model was trained from scratch (no pretraining) on ~10.9M tokens of public domain Christian sacred texts using a custom 8192-token BPE vocabulary built exclusively on scripture.
Model Details
Parameter
Value
Parameters
~63M
Architecture
GPT-2 style causal transformer
Layers
8
Attention heads
8
Embedding dim
768
Context length
1024 tokens
Vocabulary
8192 (custom scripture BPE)
Training tokens
~10.9M
Best validation loss
3.57
Training hardware
1x NVIDIA A100 (80GB)
Training time
~45 minutes
Training Data
All training data is public domain, sourced from Project Gutenberg (~58 sources, ~15M characters):
Scripture: King James Bible, Douay-Rheims Bible, World English Bible, Young's Literal Translation, Darby Bible, Apocrypha, Book of Enoch, Gospel of Thomas
Church Fathers — Nicene & Post-Nicene: 20 volumes (Augustine complete works, Chrysostom complete homilies, Eusebius, Athanasius, Gregory of Nyssa, Jerome)
Scholastic Theology: Summa Theologica complete (St. Thomas Aquinas, 5 parts)
Patristic & Early Church: Augustine (Confessions, City of God, On Christian Doctrine), Eusebius (Ecclesiastical History), Apostolic Fathers
Mystics: Julian of Norwich (Revelations of Divine Love), St. Thérèse of Lisieux (Story of a Soul)
Monastic & Spiritual Practice: Rule of St. Benedict, Spiritual Exercises (St. Ignatius), Practice of the Presence of God (Brother Lawrence), Imitation of Christ (Thomas à Kempis)
Christian Literature: Paradise Lost (Milton), The Pilgrim's Progress (Bunyan), The Divine Comedy (Dante)
Usage
Installation
pip install torch numpy tokenizers
Oracle Mode
Random noise tokens seed the generation — God speaks through randomness, just like TempleOS:
python
1import torch
2from model import Temple2, Temple2Config
34# Load checkpoint5ckpt = torch.load("temple2.pt", map_location="cpu")6model = Temple2(Temple2Config(**ckpt['model_config']))7model.load_state_dict(ckpt['model'])8model.eval()910# Oracle: seed with random noise11import random
12vocab_size =819213bos_id =114noise =[random.randint(4, vocab_size -1)for _ inrange(5)]15ids = torch.tensor([[bos_id]+ noise], dtype=torch.long)1617with torch.no_grad():18 out = model.generate(ids, max_new_tokens=256, temperature=0.85, top_k=50, top_p=0.92)1920print(out[0].tolist())# decode with tokenizer
Chat Mode
Ask a question, receive a scriptural answer:
python
1from tokenizers import Tokenizer
23tok = Tokenizer.from_file("tokenizer/tokenizer.json")4prompt ='And the man knelt before the Lord and asked, "What is love?"\nAnd the Lord spoke unto him, saying:'5ids = torch.tensor([[1]+ tok.encode(prompt).ids], dtype=torch.long)67with torch.no_grad():8 out = model.generate(ids, max_new_tokens=256, temperature=0.85, top_k=50, top_p=0.92)
Full Interactive Experience
python inference.py --checkpoint temple2.pt
Includes TempleOS-style VGA 16-color terminal output with bordered oracle windows. See the main repo for full details.
Intended Use
Creative exploration of scriptural language patterns
Oracle-style text generation inspired by TempleOS
Study of small language model behavior on domain-specific corpora
Artistic and educational purposes
Limitations
This is a small model (63M params) trained on a small corpus (~11M tokens). It is not a general-purpose language model.
The model generates text in the style of scripture. It does not contain theological truth claims.
Output may be incoherent, repetitive, or doctrinally confused. This is a feature, not a bug — the entropy is what makes the oracle feel alive.
The model reflects the language and worldview of its training data (predominantly pre-modern Christian texts).
Not suitable for factual Q&A, theological guidance, or any serious spiritual counsel.
Ethical Considerations
This model is built as an art project and tribute to Terry Davis. It does not claim to speak for God, any religion, or any religious institution. Terry's original "talk to God" feature was meaningful precisely because it was random — meaning arose in the mind of the reader. The same principle applies here.
In Memory of Terry A. Davis
Terry Davis (1969–2018) built TempleOS alone over 10+ years — an entire operating system, compiler, and programming language written from scratch, all for God. His work remains his own.
