My Tokenizer Turkish
This repository contains a small byte-level BPE tokenizer trained on a Turkish Wikipedia text corpus.
It is designed as a lightweight, general-purpose tokenizer for Turkish text. The tokenizer is packaged in the Hugging Face format so it can be loaded with AutoTokenizer or PreTrainedTokenizerFast.
Dataset
The tokenizer was trained on a plain-text .txt corpus included in this repository.
- Source file: the
.txt training corpus stored alongside the tokenizer files
- Format: UTF-8 plain text
- Content: Turkish Wikipedia text
Keeping the raw TXT file in the same Hugging Face repository makes the training data easy to inspect and reproduce.
Tokenizer details
- Tokenizer type: Byte-level BPE
- Training data: the repository's plain-text
.txt corpus
- Vocabulary size: 32,768
- Special tokens:
<unk>, <pad>, <bos>, <eos>
- Pre-tokenization: ByteLevel
- Decoder: ByteLevel
Intended use
This tokenizer is intended for Turkish language experiments, especially when a compact subword tokenizer is preferred over a character-level vocabulary.
It works well as a general tokenizer for downstream language modeling, text generation, and other NLP tasks on Turkish text.
Training notes
The tokenizer was trained with the tokenizers library using a BPE model and the full byte-level alphabet. This makes the tokenizer robust to arbitrary input text, including punctuation, symbols, and unseen character sequences.
Loading the tokenizer
1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("bilalabic/my-turkish-tokenizer")
4
5text = "Merhaba, nasılsın?"
6token_ids = tokenizer.encode(text)
7tokens = tokenizer.convert_ids_to_tokens(token_ids)
8
9print(tokens)
10print(token_ids)
11print(tokenizer.decode(token_ids))
Limitations
- This is a tokenizer only; it is not a language model.
- Quality depends on the training corpus used to build the vocabulary.
- It is trained for Turkish text, so performance on other languages may be weaker.
Files included
tokenizer.jsontokenizer_config.json
License
MIT
