A tiny, end-to-end pipeline that learns to invent Middle-earth names. It cleans a dataset of Tolkien character and place names and trains four tiny from-scratch LLM architectures on it under three tokenisers — a plain character baseline plus character-level BPE at vocab 256 and 512 — for 12 models in total (4 architectures × 3 tokenisers).
boromir, pengolodh and echoriath are real Tolkien names the model has memorised; markhîr, brandyuis and shield are inventions (none appear in the training data); arches and running are everyday English words the dataset keeps, split out of multi-word place names. All were produced by models small enough to train on a CPU in a couple of minutes.
Credit / attribution
The four model architectures are adapted from malibayram/single_letter_transformers, a collection of from-scratch reference implementations of four tiny LLM architectures (Qwen3, Qwen3.5, Gemma, DeepSeek V3). That repository lives unchanged in single_letter_transformers/ as plain content; all credit for the model code belongs to its author. eldamar-tokenizer only swaps the data (Turkish names → Middle-earth names), adds a character-level BPE tokeniser alongside the original character tokeniser, and adds the glue to run the whole sweep.
What is different from the original
Original (single_letter_transformers)
This project
Task
Generate Turkish first names
Generate Middle-earth names
Data
921 cleaned Turkish names
2,189 cleaned Tolkien names (characters + places)
Tokeniser
Character level (~30 tokens)
Character level and character-level BPE (256 & 512)
Models
1 per architecture
3 per architecture (char + BPE-256 + BPE-512) = 12
eldamar-tokenizer/
├── data/
│ ├── Characters.csv, lotr_characters.csv, characters_data.csv # raw name sources
│ ├── locations.txt # place names scraped from Tolkien Gateway
│ └── middle_earth_names.txt # the clean corpus, one name per line
├── bpe/
│ ├── bpe_256.json # trained BPE tokenisers (Hugging Face format)
│ └── bpe_512.json
├── checkpoints/ # 12 trained models: {arch}_{char,bpe256,bpe512}.pt
├── src/
│ ├── scrape_locations.py # 1. fetch place names via the MediaWiki API
│ ├── clean_data.py # 2. merge + clean all sources into one corpus
│ ├── bpe_tokenizer.py # 3. train / load the character-level BPE tokeniser
│ ├── train_one.py # 4. train a single (arch, tokeniser) model
│ ├── train_all.py # ... orchestrate all 12
│ └── generate.py # 5. sample names from any checkpoint
└── single_letter_transformers/ # the original repo (model code), unchanged
Data cleaning
Three character CSVs plus a scraped list of place names all pass through the same pipeline in src/clean_data.py:
Lower-case with plain str.lower() (not Turkish lowering, which would turn Isildur into ısildur).
Keep diacritics (á â ä é ê ë í î ó ô ö ú û) — they are half the Elvish flavour.
Mine parentheticals: strip them from the main string but feed their contents back through the pipeline, so Belladonna (Took) Baggins yields took while filters discard disambiguation prose like (son of Axantur).
Split multi-word names into one word per line; keep internal hyphens (aelin-uial, ar-pharazôn).
Filter connective/geographic stop-words (of, the, mount, river, …), regnal Roman numerals (II, VI, …) and abbreviations (Jr).
De-duplicate and sort.
Result: 2,189 unique names, a 42-character alphabet (plus newline).
Place names come from Tolkien Gateway's Index:Locations. The site blocks plain fetches, so src/scrape_locations.py uses its MediaWiki API with a descriptive User-Agent and extracts the display text of every [[link]] bullet.
No byte-level pre-tokeniser, so the base alphabet is the 42 real characters (each diacritic is one base token). This keeps both 256 and 512 meaningful merge targets rather than making "vocab 256" a degenerate zero-merge tokeniser.
A Split("\n", isolated) pre-tokeniser means merges never cross a name boundary and \n stays a lone token — the start/end-of-name marker (EOS) the models rely on.
Encoding is lossless (verified by a full-corpus round-trip).
The character baseline (char) uses the original repo's CharTokenizer, whose vocabulary is just the 43 symbols in the corpus (42 letters + newline). It needs no training and no artifact — it is rebuilt from the names file on each run — and gives us a reference point to judge what BPE actually buys.
Results
Each of the four architectures was trained for 5,000 steps on CPU under all three tokenisers — 12 models. Every one lands far below its uniform-guessing baseline (ln 43 ≈ 3.76, ln 256 ≈ 5.55, ln 512 ≈ 6.24).
Architecture
char — params / loss
BPE-256 — params / loss
BPE-512 — params / loss
Qwen3 (dense)
20.0k / 1.38
26.8k / 1.47
35.0k / 1.25
Qwen3.5 (hybrid)
42.4k / 0.90
49.2k / 0.76
57.4k / 0.64
Gemma (sliding-window)
65.8k / 0.78
113.5k / 0.70
170.9k / 0.56
DeepSeek-V3 (MoE)
48.4k / 1.23
55.2k / 0.94
63.4k / 0.82
Sample names (temperature 1.0, straight from training):
Loss is only comparable down a column, never across one. Within a tokeniser all four architectures share the same vocabulary, token stream and baseline, so the losses rank the architectures fairly. Across tokenisers the vocabulary (43 vs 256 vs 512) and the tokens-per-name differ, so the raw cross-entropy measures different things — a lower BPE-512 number does not mean it models names "better" than the char baseline. (A fair cross-tokeniser metric would be bits-per-character.)
Bigger vocab ⇒ bigger model. The tied embedding/output matrix is vocab × hidden, so char → 256 → 512 inflates every model (Gemma 66k → 114k → 171k). Seeing that cost is part of the point of the sweep.
Qualitatively, the character baseline blends whole words freely (samdalf = Sam + Gandalf, laketown), while BPE leans on learned sub-word chunks and tends to assemble names from morpheme-like pieces. Both are fun; neither is strictly best.
Reproduce from scratch
Requires Python 3.13 and uv. Dependencies: torch (CPU), tokenizers, numpy.
Lower --temperature → safer, more familiar names; higher → more inventive.
--novel-only hides names that already exist in the training corpus.
Turkish 128K BPE tokeniser (separate sub-project)
turkish_bpe_128k/ is a separate sub-project: a production-style 128,000-token byte-level BPE tokeniser for Turkish, trained on ~200M characters from two Hugging Face review datasets and published as erenyanic/turkish_bpe_128k.
python
1from transformers import AutoTokenizer
2tok = AutoTokenizer.from_pretrained("erenyanic/turkish_bpe_128k")3ids = tok.encode("İstanbul'da yağmur yağıyor.", add_special_tokens=False)4[tok.decode([i])for i in ids]5# ['İstanbul', "'da", ' yağmur', ' yağıyor', '.']
(Decoding ids one at a time is what renders the pieces as readable text — tokenize() returns them in byte-level form, e.g. 'ĠyaÄŁmur'.)
It is the opposite end of the design space from the src/ tokeniser above, which makes the pair a useful contrast:
src/ char-level BPE
turkish_bpe_128k/
Alphabet
42 real characters
256 raw bytes
Vocab
256 / 512
128,000
Unknown token
none needed (closed corpus)
none possible (bytes cover everything)
Round-trip
lossless on its corpus
lossless on any input, any script
Purpose
teach a tiny model 2,189 names
front a real Turkish LM
Two Turkish-specific details worth noting: the tokeniser preserves casing (sidestepping the dotted/dotless i trap), and it drops the English contraction clause from the standard GPT-4 split regex — left in, that clause would cut İstanbul'da into İstanbul + 'd + a, severing the apostrophe suffix Turkish uses on proper nouns.
districts/ is a small separate side task — the same four architectures generating Turkish district (ilçe) names with the original repo's plain character tokeniser (no BPE), trained on names scraped from Turkish Wikipedia and cleaned by the repo's own temizle_isimler.py. It does not touch the src/ pipeline. See districts/README.md.
