You can deploy this model and start earning money today!

gorkemergune/
my-tokenizer

Name: my-tokenizer
Availability: InStock

Views

No views yet

my-tokenizer

byte-level Byte-Pair Encoding (BPE) tokenizer trained from scratch with the tokenizers library. It was built as a learning exercise: text is scraped from a Wikipedia article, a BPE tokenizer is trained on it, and the result is wrapped as a PreTrainedTokenizerFast so it loads through AutoTokenizer.

Details

Property	Value
Algorithm	Byte-level BPE
Vocabulary size	2048 (2¹¹)
Special tokens	`<unk>`, `<pad>`, `<bos>`, `<eos>`
Pre-tokenizer	`ByteLevel(add_prefix_space=False)`
Decoder	`ByteLevel`
Training data	Plain text of the English Wikipedia article Large language model

Usage

python

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gorkemergune/my-tokenizer")

example = "Hello! This is a very small tokenizer example."
token_ids = tokenizer.encode(example)

print("Tokens:", tokenizer.convert_ids_to_tokens(token_ids))
print("Token IDs:", token_ids)
print("Decoded:", tokenizer.decode(token_ids))

Limitations

Trained on a single, short Wikipedia article, so its merges are biased toward that domain (LLM / machine-learning vocabulary) and English text.
The vocabulary is intentionally tiny (2048), so longer or rarer words are split into many sub-tokens. This is a demo tokenizer, not a production one.

How it was built

The training pipeline is available on GitHub (gorkemergune/wiki2bpe). In short:

scraper.py downloads the Wikipedia article as plain text into text.txt.
script.py trains the byte-level BPE tokenizer on text.txt and pushes it here.