You can deploy this model and start earning money today!

Mer1Alii/
TR-ECommerce-CustomerSupport-Tokenizer

Name: TR-ECommerce-CustomerSupport-Tokenizer
Availability: InStock

Views

No views yet

TR E-Commerce Customer Support Tokenizer 🇹🇷

A custom-trained Byte-Pair Encoding (BPE) tokenizer optimized specifically for Turkish e-commerce customer support dialogues. Trained on the Mer1Alii/TR-ECommerce-CustomerSupport-Instructions corpus, this tokenizer drastically improves token efficiency and semantic comprehension for Turkish conversational AI.

1. The Challenge of Turkish Tokenization

Turkish is an agglutinative language with a rich morphological structure. Words are constructed by attaching multiple suffixes to a root (e.g., kar-go-lar-ı-mız-dan).

Standard English-centric tokenizers (like GPT-2 or LLaMA) do not have these Turkish roots/suffixes in their pre-trained vocabularies. As a result, they fragment basic Turkish words into tiny, meaningless character groups. This leads to:

High Token Counts: Turkish texts take up to 2.5x to 3x more tokens than English counterparts.
Context Window Waste: Models hit their context limits much faster.
Poorer Semantic Learning: The model spends capacity learning character-level combinations instead of word meanings.

This custom tokenizer solves these issues by learning a vocabulary derived directly from real Turkish customer service dialogues.

2. Tokenization Performance Benchmark

Here is a comparison of how different tokenizers split the sample Turkish e-commerce query: "kargom teslim edilmedi iade istiyorum" (my package was not delivered, I want a return)

Tokenizer	Tokenized Representation	Token Count	Efficiency Gain
GPT-2 (Standard)	`['k', 'arg', 'om', ' t', 'es', 'lim', ' ed', 'il', 'medi', ' i', 'ade', ' is', 't', 'iy', 'orum']`	15	Baseline
Our Custom Tokenizer	`['kargom', ' teslim', ' edil', 'medi', ' iade', ' istiyorum']`	6	2.5x Fewer Tokens (60% Savings)

3. Specifications

Vocabulary Size: 8192 ($2^{13}$ tokens)
Algorithm: Byte-Level BPE (ByteLevelBPETokenizer)
Base Training Corpus: 558 lines of Turkish e-commerce customer support dialogues (Mer1Alii/TR-ECommerce-CustomerSupport-Instructions)
Special Tokens Map:
- <s>: Beginning of Sequence (BOS)
- <pad>: Padding (PAD)
- </s>: End of Sequence (EOS)
- <unk>: Unknown token (UNK)
- <mask>: Masking token (MASK)

5. Quick Start (Usage)

You can load and use this tokenizer directly in Python using the Hugging Face transformers library:

python

from transformers import AutoTokenizer

# Load custom tokenizer
tokenizer = AutoTokenizer.from_pretrained("Mer1Alii/TR-ECommerce-CustomerSupport-Tokenizer")

# Test Sentence
text = "kargom teslim edilmedi iade istiyorum"
tokens = tokenizer.encode(text)

print("Token IDs:", tokens)
print("Decoded Tokens:", tokenizer.convert_ids_to_tokens(tokens))

Developer

Mert Ali Alkan