The name Hecemen is inspired by the Turkish word hece (syllable) combined with the Turkish agentive suffix -men, which denotes mastery or profession — as in öğretmen (teacher, literally "one who teaches") or yazman (secretary, "one who writes"). Hecemen thus translates roughly as "one who masters syllables" or "the segmenter."
Although tokens are not syllables, a well-designed tokenizer for Turkish often learns segmentations that align with natural linguistic boundaries — making the name both meaningful and fitting for a Turkish-native tokenizer.
Why a Turkish-specific tokenizer?
Turkish has a highly regular agglutinative morphology. Instead of relying on large multilingual corpora, Hecemen was designed specifically for Turkish using a carefully curated corpus that balances general language with domain-specific terminology from areas such as law, medicine, sports, technology and education. The objective is not only to reduce token counts, but to preserve meaningful lexical units whenever possible.
Design Goals
Hecemen was designed around four principles:
Preserve complete lexical units whenever possible.
Produce linguistically natural segmentations when splitting is necessary.
Minimize token count for Turkish text.
Generalize well across different domains.
Token Efficiency
Example sentence:"Kitapçılıktan kazandığı parayla yeni bir dükkan açtı."
Tokenizer
Token Count
Gemma 7B (multilingual)
19
Hecemen BPE 96k
10
Hecemen Unigram 128k
9
52% fewer tokens than Gemma for the same Turkish text — meaning faster inference, lower cost, and better semantic representation per token.
Gemma 7B — 27 tokens for a complex Turkish sentence:
Gemma tokenizer
Hecemen Unigram — 10 tokens for the same sentence:
Hecemen tokenizer
Training Data
~1GB of Turkish text
Sources: news, books, web corpora, law, medicine
Character coverage: 1.0 (full Turkish alphabet support)
Byte fallback: enabled
Model Details
Property
Value
Algorithm
Unigram (SentencePiece)
Vocabulary size
128,000
Normalization
nmt_nfkc_cf
Byte fallback
True
Character coverage
1.0
Usage
python
1import sentencepiece as spm
2from huggingface_hub import hf_hub_download
34model_path = hf_hub_download(5 repo_id="mursideaki/hecemen-tokenizer-unigram-128k",6 filename="tr_unigram_tokenizer.model"7)89sp = spm.SentencePieceProcessor()10sp.load(model_path)1112text ="Kitapçılıktan kazandığı parayla yeni bir dükkan açtı."13tokens = sp.encode_as_pieces(text)14ids = sp.encode_as_ids(text)1516print(f"Tokens : {tokens}")17print(f"IDs : {ids}")18print(f"Count : {len(tokens)}")
