KorByte-128K
KorByte-128K is a Korean-focused, Unicode-aware byte-level BPE tokenizer with
128,000 learned tokens and 256 stable special-token IDs. It performs no Unicode
normalization, so it preserves spaces, line endings, decomposed Hangul, emoji, and
arbitrary UTF-8 text exactly.
It ranks
first among 8 successfully loaded, revision-pinned public systems
by both fertility and effective bits per byte (EBPB) on the Korean slice of the
Multilingual Tokenizer Benchmark.
It also ranks first on a frozen, post-selection
KMMLU test audit. This is a scoped
intrinsic result, not proof of universal or downstream language-model superiority.
Quick start
1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("dawncr0w/KorByte-128K", use_fast=True)
4text = "새 기능을 배포하기 전에 테스트 결과를 확인해 주세요."
5ids = tokenizer.encode(text, add_special_tokens=False)
6assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text
Measured result
- Public Korean benchmark fertility / EBPB: 1.4463 / 3.0710 (rank 1)
- KMMLU audit fertility / EBPB: 1.9364 / 3.2942 (rank 1)
- Macro token reduction vs.
kakaocorp/kanana-2-3b-base: 18.92%
- Public first-place gate: PASS
- Exact round-trip release validation: passed
- Core / total vocabulary: 128,000 / 128,256
See
reports/comparison.md for the pinned public ranking and
unavailable artifacts, and
reports/research.md for the
accepted and rejected variants. KLUE domain counts and throughput are in
reports/benchmark.md. Machine-readable evidence is under
reports/.
Why OKT and MeCab-ko are not the primary baseline
OKT and MeCab-ko are morphological analyzers. They do not provide the same fixed-vocabulary,
lossless, byte-complete encoding contract required by an LLM tokenizer. Their output counts
and speed are reported as useful context; Kanana-2 is the like-for-like tokenizer baseline.
Design
- Unicode-aware word-boundary segmentation with six-digit number chunks
- Byte-level alphabet, decoder, and no normalizer for complete coverage
- 700 million-character Korean-heavy public training mixture with a smaller English allocation
- Deterministic source revisions, shuffle seed, filtering, deduplication, and manifests
- 256 contiguous special-token IDs from 128,000 through 128,255
Intended use and limitations
This artifact is intended for Korean-heavy language-model experiments, token-count analysis,
and as a starting vocabulary for training a new model. Replacing the tokenizer of an existing
model without retraining or vocabulary adaptation will break that model. Compression alone does
not guarantee better accuracy, latency, safety, or training efficiency. The Thunder public
artifact could not be loaded because it uses a custom tokenizer model; the comparison report
records the exact failure instead of silently omitting it.
Reproduce
1uv sync --all-extras
2uv run korbyte prepare --scale 1.0
3uv run korbyte train
4uv run korbyte benchmark
5uv run korbyte compare
6uv run korbyte render
7uv run korbyte validate
The prepared training text is intentionally excluded from this repository. Exact source
revisions, accepted character counts, filtering, and hashes are documented in
DATA_SOURCES.md and
provenance/.
