A 147M parameter RWKV-7 language model trained from scratch on 6.55 billion tokens of
historical English text spanning 1473–1914: the full Text Creation Partnership (TCP)
corpus of Early Modern English print, combined with 47,989 English-language books from
Project Gutenberg.
Trained in a single run on a B200 GPU (~10.5 hours). No pre-trained weights, no fine-tuning.
60,172 documents of Early Modern English print, 1473–1800. EEBO Phase I+II, ECCO, and
Evans-TCP. Original spelling preserved at word level (loue, vp, hath). Genre breakdown:
~35% religious/theological, 14% pamphlets/letters, 14% history, 9% poetry, 9% legal,
2.5% drama.
47,989 English-language books from Project Gutenberg. Fiction, non-fiction, philosophy,
science. Predominantly 19th-century, with coverage extending to ~1914. Pre-deduplicated
by the HuggingFace dataset maintainers.
Pipeline
Both corpora: artifact cleaning → MinHash dedup → RWKV World tokenization →
binary shards. Full details and data cards in the training repository.
Evaluation
Perplexity on held-out shards (lower = better):
Corpus
Loss
Perplexity
TCP eval (11.5M tokens)
3.5426
34.56
Gutenberg eval (19.7M tokens)
3.4312
30.91
Combined (31.2M tokens)
3.4723
32.21
Usage
python
1# pip install rwkv2from rwkv.model import RWKV
3from rwkv.utils import PIPELINE
45model = RWKV(model="rwkv7-147m-vintage-english/model.pth", strategy="cuda bf16")6pipeline = PIPELINE(model,"rwkv_vocab_v20230424")78print(pipeline.generate(9"It was a truth universally acknowledged",10 token_count=200,11 temperature=1.0,12 top_p=0.85,13))
The model handles both Early Modern spelling (TCP era) and standard Modern English
(Gutenberg era). Prompts in either register will work.
Checkpoints along the training run
A checkpoint was saved every 10,000 steps. One checkpoint is particularly significant:
step_0067017 — the TCP-only boundary. At this exact step the TCP shards are
exhausted and Gutenberg begins. This checkpoint is trained purely on Early Modern English
(1473–1800) and is available as a standalone model:
geoppls/rwkv7-147m-early-modern-english
Comparing the two checkpoints shows how the model's representations shift when 19th-century
literature is introduced.
Research context
This model is part of a larger project training a generative model on the full span of
written English as a foundation for cross-era and eventually cross-linguistic latent space
analysis.
The core research direction: extend the corpus to include ancient language transliterations
(Akkadian cuneiform, Sumerian, Egyptian hieroglyphs) alongside English, and probe what
cross-linguistic structure the model discovers without explicit translation pairs. Standard
multilingual training uses parallel text; this approach intentionally does not — the
emergent associations are the research output.
Specific questions: Do functional document types (legal, administrative, religious) cluster
across languages despite having no shared vocabulary? Do ancient tax records and Victorian
legal documents end up closer in embedding space than either is to a poem — purely from
structural similarity?
