You can deploy this model and start earning money today!

zary0/
edo-cutoff-1b

Name: edo-cutoff-1b
Availability: InStock

pretrained-from-scratch

Views

No views yet

🏯 Edo-Cutoff LLM

A 1B-parameter language model that has never read a single word written after 1868.

江戸カットオフLLM — 知識カットオフ 1868年

Colab

1. Introduction

Modern LLMs know everything humanity has ever written. Edo-Cutoff LLM inverts that premise.

This model is pretrained exclusively on public-domain texts from the Edo period and earlier — Japanese woodblock-print books and the Chinese classics an educated person of that era would actually have read. It has never seen a modern webpage, a modern textbook, or a modern sentence. It does not know that the Earth orbits the Sun.

The interesting engineering problem is not building such a model — it is preventing contamination. Any modern text, any modern paraphrase, any modern tokenizer leaks 21st-century priors into the weights. This project therefore uses:

No modern text. Only pre-1868 public-domain sources.
No modern-model synthetic data. No rephrasing, no distillation.
A tokenizer trained from scratch on the period corpus (a modern tokenizer's vocabulary is itself a modern prior — and 2× less efficient here).

これは何か: 明治より前の世界しか知らないAI。現代テキストを一切混ぜず、江戸期以前のパブリックドメイン文献のみで事前学習しています。企画の核心は「現代知識の汚染をいかに防ぐか」です。

2. Model Summary


Architecture	Llama 3.2 1B (decoder-only, GQA, RoPE, SwiGLU, RMSNorm)
Parameters	1.07 B (1,071.5 M) · embeddings 98.3 M (tied)
Layers / Hidden / FFN	16 / 2048 / 8192
Attention heads	32 query · 8 key-value (GQA 4×, head dim 64)
Positional encoding	RoPE (θ = 500,000)
Context length	2,048
Vocabulary	48,000 (custom byte-level BPE)
Precision	bfloat16
Pretraining tokens	2.29 B corpus · 262 M seen (step 500 / 4,370)
Language	Classical Japanese (文語・候文), Literary Chinese (漢文)

3. Training Data

Everything is public domain by age, and re-distributable. No modern text is included.

Source	Scale	Content	License
NDL Next Digital Library — 古典籍 OCR	64,769 books · 2.49 B chars	Japanese woodblock books: literature, medicine, agriculture, travel, Buddhism, Confucian studies	Free reuse (copyright expired)
Kanseki Repository (KR1–5 經史子集叢)	3,943 texts · 0.78 B chars	Chinese classics read in Edo Japan: Four Books, Records of the Grand Historian, philosophers, poetry	CC BY-SA 4.0
Total	3.27 B chars ≈ 2.29 B tokens		CC BY-SA 4.0

Notes on source selection. ctext.org was rejected despite covering the same public-domain works: its data is CC BY-NC-SA and bulk access is gated. The Kanseki Repository provides the same corpus under BY-SA with unrestricted bulk access. The Buddhist canon (KR6, 4,849 texts) was excluded because it is CBETA-derived and NC-encumbered — a real loss, since Edo scholars read it widely.

4. Tokenizer

A byte-level BPE trained from scratch on the period corpus. This is the single highest-leverage component: for a small model with a large vocabulary and long context, tokenizer efficiency directly buys training throughput and effective context.

Tokenizer	Vocab	Japanese ↓	Literary Chinese ↓
Edo-mixed BPE (this work)	48,000	0.611	0.836
Edo JP-only BPE	32,000	0.673	0.935
GPT-4o (`o200k_base`)	~200,000	1.040	1.178
GPT-4 (`cl100k_base`)	~100,000	1.474	—

Fertility = tokens per character; lower is better. Measured on held-out Edo-period text.

~1.7× more efficient than GPT-4o and 2.4× more than GPT-4 on this domain. Byte-level construction means zero UNK by design — variant kana (変体仮名), rare glyphs, and kanbun are all representable. Normalization is NFC only; NFKC was rejected because it collapses the orthographic variants that define this corpus.

5. Training


Hardware	1 × NVIDIA H100 80GB (RunPod, secure cloud)
Stack	PyTorch 2.8 · CUDA 12.8 · bf16 · Liger Kernel (fused cross-entropy)
Optimizer	AdamW (β = 0.9/0.95, wd 0.1, fused; decay on 2-D params only)
Schedule	cosine 3e-4 → 3e-5, warmup, grad-clip 1.0
Batch	524,288 tokens/step (256 × 2048), gradient checkpointing
Throughput	~46–55 K tokens/s
Progress	step 500 / 4,370 (≈ 11 % of the first epoch)

Fused cross-entropy is what makes this tractable: with a 48 K vocabulary at 2048 context, the logit tensor dominates activation memory and caps the micro-batch. Liger removes it, roughly doubling throughput over the naive configuration.

Checkpoints are pushed to the Hub every 100 steps, making the run resumable across ephemeral GPU rentals — training proceeds in whatever increments the budget allows.

6. Samples

Generated at step 500 (temperature 0.85, top-k 40). The model is deeply undertrained; these are stylistic fragments, not coherent prose.

Prompt	Continuation
`天保十五年`	天保十五年九月上直澄良直広直盛直業等公重隆重実永茂信安従五位下従四位下正五位下従四位上…從二位忠久男直兼直輔母式部右衛門通方…
`此薬`	此薬種数ヲ変ス是等ノ理ヲ得ルハ是等ノ法ヲ以テ理トセンコトハナリ是其ノ大品ニ理ハ別ニ定シ難キ者ニシテ是ノ如ク論スルノミナリ
`子曰`	子曰然

The model has already internalized period-specific registers: court-rank genealogies (武鑑・系図体) from the first prompt, kanbun-kundoku expository prose (漢文訓読体) from the second, and Analects phrasing from the third. Learning the form of Edo writing precedes learning its content.

7. Quick Start

Colab (free GPU, public link)

Open edo_colab_demo.ipynb → set runtime to T4 GPU → Run all. A Gradio app with a public *.gradio.live URL launches automatically.

Python

python

import torch
from huggingface_hub import hf_hub_download
from transformers import LlamaConfig, LlamaForCausalLM
from tokenizers import Tokenizer

REPO = "zary0/edo-cutoff-1b"
ck  = torch.load(hf_hub_download(REPO, "edo_1b_partial_bf16.pt"), map_location="cpu")
tok = Tokenizer.from_file(hf_hub_download(REPO, "edo_bpe_48000.json"))

cfg = LlamaConfig(**{k: v for k, v in ck["config"].items() if not k.startswith("_")})
model = LlamaForCausalLM(cfg)
model.load_state_dict(ck["model"], strict=False)
model = model.to("cuda").to(torch.bfloat16).eval()

ids = [cfg.bos_token_id] + tok.encode("天保十五年").ids
x = torch.tensor([ids], device="cuda")
for _ in range(80):
    logits = model(input_ids=x).logits[:, -1, :].float() / 0.85
    v, _ = torch.topk(logits, 40)
    logits[logits < v[:, [-1]]] = -float("inf")
    x = torch.cat([x, torch.multinomial(torch.softmax(logits, -1), 1)], dim=1)
print(tok.decode(x[0].tolist()))

Resume pretraining

bash

hf download zary0/edo-cutoff-1b ckpt_last.pt --local-dir ckpt   # full state (12.9 GB, incl. optimizer)
python scripts/train.py --config configs/edo_1b.json --liger --grad-ckpt \
  --device cuda --dtype bf16 --resume ckpt/ckpt_last.pt --hf-repo zary0/edo-cutoff-1b

Files

File	Size	Purpose
`edo_1b_partial_bf16.pt`	2.3 GB	Inference weights (bf16)
`ckpt_last.pt`	12.9 GB	Full training state (weights + optimizer), for resuming
`edo_bpe_48000.json`	4 MB	Tokenizer
`edo_colab_demo.ipynb`	—	One-click Colab demo

8. Limitations

Severely undertrained. 262 M of a planned 2.29 B tokens (11 % of one epoch). It cannot yet hold a coherent sentence. Training halted purely by compute budget, not by design.
OCR noise is learned. □ marks illegible characters in the source scans; the model reproduces them as a frequent token.
Not instruction-tuned. It is a raw base model — a continuation engine, not an assistant. It will not answer questions.
Not purely Japanese. ~24 % literary Chinese, deliberately: that reflects an Edo reader's actual diet.
The cutoff is a data property, not a guarantee. Contamination was controlled at the source and tokenizer level, but has not been adversarially audited.

9. License

CC BY-SA 4.0, inherited from the Kanseki Repository portion of the training data. Share-alike applies to derivatives.

10. Citation

bibtex

@misc{edo-cutoff-llm-2026,
  title  = {Edo-Cutoff LLM: A Language Model with an 1868 Knowledge Cutoff},
  author = {zary0},
  year   = {2026},
  url    = {https://huggingface.co/zary0/edo-cutoff-1b}
}

Acknowledgements

Built on public infrastructure that made a project like this possible for one person:

National Diet Library (NDL Lab) — Next Digital Library, and NDL古典籍OCR for kuzushiji recognition (CC BY 4.0)
Kanseki Repository 漢籍リポジトリ — Christian Wittern and contributors
みんなで翻刻 — crowd-sourced transcription (CC BY-SA 4.0)