You can deploy this model and start earning money today!

cmeister/
tokenizer-lm-ablations

Name: tokenizer-lm-ablations
Availability: InStock

Views

No views yet

Tokenizer ablations: 220 language models trained from scratch

220 language models that share an architecture, a training corpus, a document order and a hyperparameter set, and differ in the tokenizer they were trained with. 94 tokenizers are involved, of which 90 are released here and 4 are off-the-shelf tokenizers cited rather than redistributed. Every model was trained for this study, not adapted from an existing checkpoint.

Acknowledgments: These analyses were part of the development for the Apertus 2.0 tokenizer, funded by the Swiss AI Initiative.

Training regimes

Regime	Models	Description
`std-1B`	97	24 layers, width 1536, 1.27B parameters, 9.2B tokens, standard mixture
`mathcode-20B-from-scratch`	69	24 layers, width 1536, 1.27B parameters, 20.0B tokens, math and code mixture, trained from scratch
`cross-scale-d16-300M`	18	16 layers, width 1024, 0.60B parameters, 3.5B tokens, standard mixture, scale ladder
`cross-scale-d12`	18	12 layers, width 768, 0.38B parameters, 3.5B tokens, standard mixture, scale ladder
`cross-scale-d8`	18	8 layers, width 512, 0.22B parameters, 3.5B tokens, standard mixture, scale ladder

Loading a model

Each model lives in its own subfolder. The architecture is not one of the built-in transformers architectures, so loading needs trust_remote_code=True; the model class ships in each subfolder as nanochat_model.py.

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "cmeister/tokenizer-lm-ablations", subfolder="models/full-128k-allmulti-equal-bpe",
    trust_remote_code=True, dtype="float32")
tokenizer = AutoTokenizer.from_pretrained("cmeister/tokenizer-lm-ablations", subfolder="models/full-128k-allmulti-equal-bpe")

text = "def add(a, b):\n    return a + b\n"
ids = [0] + tokenizer(text)["input_ids"]   # training prepends BOS

Training prepended the beginning-of-sequence token to every document, and the released tokenizer files carry no post-processor that would do this on their own. Prepend the id yourself, as above, or the model receives input it never saw in training. Each model card states its own BOS id.

What varies, and what does not

Within a training regime, every model sees the same documents in the same order with the same optimizer settings. The tokenizer changes the token sequence those documents produce, so token counts, sequence packing and the effective amount of text per context window differ between models. That is the effect under study, not a confound to remove.

Ablation grid: the tokenizer axis

One row per tokenizer. pretokenizer names the regex constant in the training script; normalizer and vocab are read from the released tokenizer file itself. In the names, gpt4o, claude and gpt2 refer to a borrowed pretokenization regex, not to those vendors' tokenizers.

Tokenizer	Algorithm	Pretokenizer	Normalizer	Training data	Vocab	Models	Loads with
`NousResearch/Meta-Llama-3-8B` (cited only)	n/a	n/a	n/a	n/a	n/a	4	transformers
`allmulti-equal-bpe`	bpe	REGEX_GPT4O	none	allmulti_equal	128260	1	transformers
`apertus` (cited only)	n/a	n/a	n/a	n/a	n/a	2	transformers
`bpe-nfc-clean-balanced`	BPE	n/a	NFC	n/a	128004	2	transformers
`bpe-nfc-clean-balanced-v128k`	bpe	REGEX_CLEAN	NFC	balanced	128004	1	transformers
`bpe-nfc-clean-balanced-v160k`	bpe	REGEX_CLEAN	NFC	balanced	160004	1	transformers
`bpe-nfc-clean-balanced-v200k`	bpe	REGEX_CLEAN	NFC	balanced	200004	1	transformers
`bpe-nfc-clean-balanced-v64k`	bpe	REGEX_CLEAN	NFC	balanced	64004	1	transformers
`bpe-nfc-clean-balanced-v96k`	bpe	REGEX_CLEAN	NFC	balanced	96004	1	transformers
`bpe-nfc-clean-fineweb2full`	BPE	n/a	NFC	n/a	128000	1	transformers
`bpe-nfc-clean-nomark-balanced`	bpe	REGEX_CLEAN_NOMARK	NFC	balanced	128004	1	transformers
`bpe-nfc-gpt4-fineweb2full`	BPE	n/a	NFC	n/a	128000	1	transformers
`bpe-nfc-plus2-balanced`	bpe	REGEX_CLEAN_PLUS2	NFC	balanced	128260	4	transformers
`bpe-nfc-plus3-balanced`	bpe	REGEX_CLEAN_PLUS3	NFC	balanced	128260	2	transformers
`claude-allmulti-equal-bpe`	bpe	REGEX_CLAUDE	none	allmulti_equal	128260	1	transformers
`claude-balanced-bpe`	bpe	REGEX_CLAUDE	none	balanced	128260	8	transformers
`claude-balanced-nfc-bpe`	bpe	REGEX_CLAUDE	NFC	balanced	128260	6	transformers
`claude-balanced-unigram`	unigram	REGEX_CLAUDE	none	balanced	128260	6	transformers
`claude-english-bpe`	bpe	REGEX_CLAUDE	none	english	128260	6	transformers
`clauderev-balanced-bpe`	bpe	REGEX_CLAUDE_REV	Sequence	balanced	128260	2	transformers
`gpt2-balanced-bpe`	bpe	ByteLevel	none	balanced	128260	1	transformers
`gpt4o-allmulti-equal-nfc-bpe`	bpe	REGEX_GPT4O	NFC	allmulti_equal	128260	1	transformers
`gpt4o-allmulti-equal-norepeat-bpe`	bpe	REGEX_GPT4O	none	allmulti_equal_norepeat	128260	1	transformers
`gpt4o-balanced-bpe`	bpe	REGEX_GPT4O	none	balanced	128260	8	transformers
`gpt4o-balanced-maxfiles50-bpe`	bpe	REGEX_GPT4O	none	balanced_maxfiles50	128260	1	transformers
`gpt4o-balanced-nfc-bpe`	bpe	REGEX_GPT4O	NFC	balanced	128260	6	transformers
`gpt4o-balanced-unigram`	unigram	REGEX_GPT4O	none	balanced	128260	6	transformers
`gpt4o-balanced-unigram-tuned`	unigram	REGEX_GPT4O	none	balanced	128260	1	transformers
`gpt4o-code-bpe`	bpe	REGEX_GPT4O	none	code	128260	6	transformers
`gpt4o-codegrad00-bpe`	bpe	REGEX_GPT4O	none	codegrad00	128260	1	transformers
`gpt4o-codegrad05-bpe`	bpe	REGEX_GPT4O	none	codegrad05	128260	1	transformers
`gpt4o-codegrad30-bpe`	bpe	REGEX_GPT4O	none	codegrad30	128260	1	transformers
`gpt4o-codegrad50-bpe`	bpe	REGEX_GPT4O	none	codegrad50	128260	1	transformers
`gpt4o-english-bpe`	bpe	REGEX_GPT4O	none	english	128260	6	transformers
`gpt4o-english-fullbyte-bpe`	bpe	REGEX_GPT4O	none	english	128260	2	transformers
`gpt4onl-balanced-bpe`	bpe	REGEX_GPT4O_NONL	none	balanced	128260	3	transformers
`gpt4onl-codegrad00-bpe`	bpe	REGEX_GPT4O_NONL	none	codegrad00	128260	1	transformers
`gpt4onl-codegrad05-bpe`	bpe	REGEX_GPT4O_NONL	none	codegrad05	128260	1	transformers
`gpt4onl-codegrad30-bpe`	bpe	REGEX_GPT4O_NONL	none	codegrad30	128260	1	transformers
`gpt4onl-codegrad50-bpe`	bpe	REGEX_GPT4O_NONL	none	codegrad50	128260	1	transformers
`highmid-bpe`	bpe	REGEX_GPT4O	none	highmid	128260	1	transformers
`highmid-unigram`	unigram	REGEX_GPT4O	none	highmid	128260	1	transformers
`highres-bpe`	bpe	REGEX_GPT4O	none	highres	128260	1	transformers
`highres-unigram`	unigram	REGEX_GPT4O	none	highres	128260	1	transformers
`llama3` (cited only)	n/a	n/a	n/a	n/a	n/a	2	transformers
`pabpe-nfc-apertus-fineweb2full-capped-tuned-hybridwindow`	BPE	n/a	NFC	n/a	127835	2	transformers
`pabpe-nfc-clean-balanced`	BPE	n/a	NFC	n/a	127836	1	transformers
`pabpe-nfc-clean-balanced-hybridwindow`	BPE	n/a	NFC	n/a	127836	1	transformers
`pabpe-nfc-clean-fineweb2full`	BPE	n/a	NFC	n/a	127835	1	transformers
`pabpe-nfc-clean-fineweb2full-capped-tuned-hybridwindow`	BPE	n/a	NFC	n/a	127835	2	transformers
`pabpe-nfc-clean-fineweb2full-consv2-engfull-eu3-gm110k`	BPE	n/a	NFC	n/a	131196	2	transformers
`pabpe-nfc-clean-fineweb2full-consv2-eudata-gm110k`	BPE	n/a	NFC	n/a	131196	1	transformers
`pabpe-nfc-clean-fineweb2full-consv2-eudata-gm90k`	BPE	n/a	NFC	n/a	131196	1	transformers
`pabpe-nfc-clean-fineweb2full-consv2-eudata4-gm100k`	BPE	n/a	NFC	n/a	131196	1	transformers
`pabpe-nfc-clean-fineweb2full-consv2-eusino-v2c-frde-kr120-gm130k-v200k`	BPE	n/a	NFC	n/a	200124	2	transformers
`pabpe-nfc-clean-fineweb2full-consv2-eusino-v2c-gm120k-v200k`	BPE	n/a	NFC	n/a	200124	2	transformers
`pabpe-nfc-clean-fineweb2full-consv2-frde2-gm110k`	BPE	n/a	NFC	n/a	131196	2	transformers
`pabpe-nfc-clean-fineweb2full-consv2-plus3-repcap8`	BPE	n/a	NFC	n/a	131141	2	transformers
`pabpe-nfc-clean-fineweb2full-consv2-prelim-mul-v131072`	BPE	n/a	NFC	n/a	131196	2	transformers
`pabpe-nfc-clean-fineweb2full-hybridwindow`	BPE	n/a	NFC	n/a	127835	1	transformers
`pabpe-nfc-clean-fineweb2full-plus2-capped-tuned-hybridwindow`	BPE	n/a	NFC	n/a	127835	2	transformers
`pabpe-nfc-clean-fineweb2full-plus2-capped-tuned-hybridwindow-consv2-tailcuts-gm90k-v130-sp-eng5g`	BPE	n/a	NFC	n/a	131028	2	transformers
`pabpe-nfc-clean-fineweb2full-plus3-capped-tuned-hybridwindow-consv2`	BPE	n/a	NFC	n/a	127835	2	transformers
`pabpe-nfc-clean-fineweb2full-plus3-capped-tuned-hybridwindow-consv2-tailcuts-gm70k-v130-sp-eng5g`	BPE	n/a	NFC	n/a	131028	2	transformers
`pabpe-nfc-clean-fineweb2full-plus3-capped-tuned-hybridwindow-consv2-tailcuts-gm90k-v130-sp-eng5g`	BPE	n/a	NFC	n/a	131028	2	transformers
`pabpe-nfc-clean-fw2full-plus3-cap-hw-consv2-gm90k-tailcuts-eng5g-repcap8`	BPE	n/a	NFC	n/a	131141	2	transformers
`pabpe-nfc-gpt4-balanced`	BPE	n/a	NFC	n/a	127826	5	transformers
`pabpe-nfc-gpt4-balanced-hybridwindow`	BPE	n/a	NFC	n/a	127826	1	transformers
`pabpe-nfc-gpt4-fineweb2full`	BPE	n/a	NFC	n/a	127825	1	transformers
`pabpe-nfc-gpt4-fineweb2full-hybridwindow`	BPE	n/a	NFC	n/a	127825	1	transformers
`punct-allmulti-equal-bpe`	bpe	Punctuation+ByteLevel	none	allmulti_equal	128260	1	transformers
`punct-balanced-bpe`	bpe	Punctuation+ByteLevel	none	balanced	128260	9	transformers
`punct-balancedhalf-superbpe-t64k-copy`	BPE	n/a	none	n/a	128004	1	transformers
`punct-english-bpe`	bpe	Punctuation+ByteLevel	none	english	128260	6	transformers
`rightalign-balanced-bpe`	bpe	REGEX_GPT4O_RIGHTALIGN	none	balanced	128260	6	transformers
`rightalign-balanced-nfc-bpe`	bpe	REGEX_GPT4O_RIGHTALIGN	NFC	balanced	128260	6	transformers
`rightalign-balanced-unigram`	unigram	REGEX_GPT4O_RIGHTALIGN	none	balanced	128260	6	transformers
`scripttok-bpe-scriptenc_gpt4o_cb`	n/a	n/a	none	n/a	n/a	2	script_tok
`scripttok-mingram-scriptenc_cb`	n/a	n/a	none	n/a	n/a	2	script_tok
`scripttok-mingram-scriptenc_cb_nl`	n/a	n/a	none	n/a	n/a	2	script_tok
`superbpe-apertus-fineweb2full-capped-hybridwindow`	BPE	n/a	NFC	n/a	128004	2	transformers
`superbpe-clean-c2-balanced`	BPE	n/a	NFC	n/a	128004	2	transformers
`superbpe-clean-c3-balanced`	BPE	n/a	NFC	n/a	128004	2	transformers
`superbpe-clean-fineweb2full-capped-hybridwindow`	BPE	n/a	NFC	n/a	128004	2	transformers
`superbpe-clean-fineweb2full-capped-hybridwindow-t110k-v130k`	BPE	n/a	NFC	n/a	130004	2	transformers
`superbpe-clean-fineweb2full-plus2-cappedv2-hybridwindow-t110k-v130k-consv2`	BPE	n/a	NFC	n/a	130004	2	transformers
`superbpe-gpt4-fineweb2full-hybridwindow`	BPE	n/a	NFC	n/a	128004	1	transformers
`superbpe-gpt4o-balanced`	BPE	n/a	none	n/a	128004	5	transformers
`superbpe-pabpe-nfc-gpt4-balanced`	BPE	n/a	NFC	n/a	128004	1	transformers
`superbpe-pabpe-nfc-gpt4-t64k`	BPE	n/a	NFC	n/a	128004	1	transformers
`swiss-ai/Apertus-70B-2509` (cited only)	n/a	n/a	n/a	n/a	n/a	4	transformers
`whitespace-balanced-bpe`	bpe	Split(\s+)+ByteLevel	none	balanced	128260	2	transformers
`whitespace-balanced-unigram`	unigram	Metaspace	none	balanced	128256	1	transformers
`whitespace-multiheavy-bpe`	bpe	Split(\s+)+ByteLevel	none	multiheavy	128260	1	transformers

n/a in the pretokenizer, normalizer or training-data columns means the tokenizer was not built by this repository's training script and has no row there. Its build settings are in its own card, taken from the build manifest the external trainer wrote or from the model registry note.

Ablation grid: the model axis

The per-model settings and headline metrics are in ablation_grid.csv (220 rows) and models.json. Each model's own card repeats its row. The columns are the architecture (layers, width, heads, vocabulary, value-embedding dimension, soft cap), the training point (step, token budget, seed) and the evaluation results.

Architecture

nanochat (Karpathy), MIT licensed, with three patches applied for this study (branch tokenizer-lm-patches, commit f919449):

ve_dim: the per-layer value-embedding tables are built at a reduced dimension and projected up to the key and value dimension by a learned matrix, instead of being built at full width. All released models use ve_dim=128. Without this, a 128k vocabulary would spend most of the parameter budget on those tables.
logit_softcap: the logit soft cap is a config field where 0 disables it, instead of a hardcoded 15. Released models use 15 unless their card says otherwise.
width_lr_exponent: the AdamW learning-rate width scaling exponent is configurable, set to -1.0 for muP. This affects training only.

The rest is nanochat as published: rotary embeddings, QK norm, ReLU squared MLP, sliding-window attention on three of every four layers, a smear gate that mixes the previous token, a backout term at the middle layer, per-layer residual and x0 scalars, parameterless RMSNorm, and untied input and output embeddings.

Each released checkpoint was checked against the training-time model: the same random token ids run through both give the same logits in float32, and the top-1 prediction agrees at every position.

Training data

One mixture, used for both tokenizer training and language-model training: 35% English web text (FineWeb-Edu), 30% multilingual text over 30 languages and 8 scripts (a quality-filtered FineWeb-2), 15% mathematics (FineMath 4plus), 15% code (StarCoderData, highest quality tier). Multilingual weights are proportional to each language's estimated character count in the source, so low-resource languages get less data, which is the resource gradient the cross-lingual study measures.

The math and code regime continues from the same corpus family with a mixture weighted toward mathematics and code. Code evaluation uses a StarCoderData tier held out from training.

Upstream licenses, as their dataset cards state them: FineWeb-Edu ODC-By, FineWeb-2 ODC-By, FineMath ODC-By, StarCoderData listed as other.

What is not here

Models whose tokenizer was later found defective are not released. Two defects were found during the study: a tokenizer family that stripped newline and carriage-return characters and had an incomplete byte alphabet, and a SuperBPE stage-2 pretokenizer that split combining marks its stage 1 had kept together, which breaks Bengali, Devanagari, Tamil, Thai and Arabic sequences. The affected checkpoints stay on the training cluster with their defect recorded. Where a slug names both a defective original and a later fixed retrain, the released model is the retrain, identified by comparing the tokenizer file against the archived defective copy.

A separate family of runs trained on a quota-based corpus was retired for a cross-lingual confound rather than a defect. Those are not released either.

Files

File	Contents
`models.json`	one record per model: architecture, training point, metrics, provenance
`tokenizers.json`	one record per tokenizer: build settings and which models use it
`ablation_grid.csv`	the two axes joined, one row per model
`models/<run>/`	weights, config, tokenizer, model class, conversion record
`tokenizers/<slug>/`	the tokenizer file as trained, plus its config

Citation

TokEval: A Tokenizer Evaluation Suite. Clara Meister. COLM 2026.