This tokenizer is trained on Kurdish Kurmanji text data using Byte-Pair Encoding (BPE).
It's trained
using this dataset
1from transformers import PreTrainedTokenizerFast
2
3tokenizer = PreTrainedTokenizerFast.from_pretrained("muzaffercky/kurdish-kurmanji-tokenizer", revision="v1.0")
4
5text = "Havîn bi gelemperî bi betlane, derya û tavê tê naskirin, lê her kes nikare biçe betlaneyê"
6
7tokens = tokenizer.tokenize(text)
8
9print(tokens)
10
11# Output: ['Hav', 'în bi ', 'gelemperî bi ', 'bet', 'l', 'ane, ', 'der', 'ya û ', 'tavê ', 'tê nas', 'kirin, lê ', 'her kes', ' nikare ', 'biçe ', 'bet', 'lan', 'ey', 'ê']
12
13ids = tokenizer.encode(text)
14
15print(ids)
16
17# Output: [19889, 6494, 19754, 2055, 82, 8227, 337, 3349, 16407, 5790, 11235, 2584, 5479, 5732, 2055, 9479, 287, 140]
This tokenizer includes spaces within some tokens (e.g., 'Ez ê ', 'di vê '), which causes the default tokenizer.decode()
method from the transformers library to add extra spaces between tokens during decoding.
To decode text correctly and preserve the original formatting, decode each token ID individually and join the results without spaces.
1from transformers import PreTrainedTokenizerFast
2
3tokenizer = PreTrainedTokenizerFast.from_pretrained("muzaffercky/kurdish-kurmanji-tokenizer", revision="v1.0")
4
5text = "Havîn bi gelemperî bi betlane, derya û tavê tê naskirin, lê her kes nikare biçe betlaneyê"
6
7ids = tokenizer.encode(text)
8
9individual_tokens = [tokenizer.decode([id]) for id in ids]
10decoded_text = "".join(individual_tokens)
11
12print(decoded_text) # Output: Havîn bi gelemperî bi betlane, derya û tavê tê naskirin, lê her kes nikare biçe betlaneyê
