The model uses a
BPE tokenizer with 200k vocabulary trained on data-proportionally sampled
multilingual text — the configuration identified as optimal for text-mode tokenization. It
serves as the orthographic baseline against which the IPA-based model
(
gpt2-ipa-opt) is compared.
Data: 18 languages from
CulturaX:
Arabic, German, English, Persian, Finnish, French, Hindi, Italian, Japanese, Korean, Lao,
Russian, Serbian, Swahili, Thai, Turkish, Urdu, Chinese.
Data was sampled proportionally to each language's share in the CulturaX corpus (in terms of bytes).
1from transformers import GPT2LMHeadModel
2from huggingface_hub import hf_hub_download
3import sentencepiece as spm
4import torch
5
6# Load model
7model = GPT2LMHeadModel.from_pretrained("Mikki99/gpt2-text-opt")
8model.eval()
9
10# Load SentencePiece tokenizer
11tok_path = hf_hub_download(repo_id="Mikki99/gpt2-text-opt", filename="tokenizer.model")
12sp = spm.SentencePieceProcessor()
13sp.Load(tok_path)
14
15# Tokenize orthographic text directly
16text = "Stanford"
17input_ids = torch.tensor([sp.Encode(text, out_type=int)])
18
19with torch.no_grad():
20 output = model(input_ids)
21 logits = output.logits
This model is a research artifact released alongside an ACL 2026 paper. Its primary
purpose is to serve as an orthographic baseline for comparison with the IPA-based model
(
gpt2-ipa-opt), and to allow reproduction
of the paper's results.
1@inproceedings{miletic-etal-2026-phonemes,
2 title = "Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet",
3 author = "Mileti{\'c}, Milan and
4 Kallini, Julie and
5 Shutova, Ekaterina",
6 editor = "Liakata, Maria and
7 Moreira, Viviane P. and
8 Zhang, Jiajun and
9 Jurgens, David",
10 booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
11 month = jul,
12 year = "2026",
13 address = "San Diego, California, United States",
14 publisher = "Association for Computational Linguistics",
15 url = "https://aclanthology.org/2026.acl-long.1872/",
16 pages = "40323--40349",
17 ISBN = "979-8-89176-390-6",
18 abstract = "Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage. Widely-used subword tokenization approaches favor high-resource languages, and tokenizer-free methods still yield longer sequences for scripts with a higher bytes-per-character ratio. To address these shortcomings, we propose to use the International Phonetic Alphabet (IPA) as a language-agnostic input representation for multilingual tokenizers. IPA provides a compact symbol inventory, greater cross-lingual character overlap, and a more balanced byte-per-character distribution across languages. We train matched pairs of text vs. IPA subword tokenizers across 24 languages and 14 scripts and demonstrate that IPA tokenizers consistently improve tokenization quality, especially for non-Latin scripts, and generalize more effectively to unseen languages and scripts."
19}
