You can deploy this model and start earning money today!

KordAI/
TokenizerFast

Views

No views yet

⚡ KordAI TokenizerFast

KordAI TokenizerFast is a fast Hugging Face tokenizer designed for efficient text preprocessing for Thai and English language models.

It is fully compatible with the 🤗 Transformers ecosystem and can be used for both training and inference with compatible models.

✨ Features

🚀 Fast tokenizer implementation
🇹🇭 Thai language support
🇺🇸 English language support
🤗 Compatible with Hugging Face Transformers
⚡ Efficient tokenization and decoding
💻 Suitable for both training and inference

📋 Model Details

Item	Value
Repository	`KordAI/TokenizerFast`
Type	Fast Tokenizer
Languages	Thai, English
Library	Transformers
License	Apache-2.0

🎯 Intended Use

This tokenizer is designed for:

Thai text tokenization
English text tokenization
Multilingual NLP pipelines
Dataset preprocessing
Language model training
Language model inference

🐍 Usage

python

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "KordAI/TokenizerFast"
)

text = "สวัสดี Hello!"

tokens = tokenizer(text)

print(tokens)

decoded = tokenizer.decode(tokens["input_ids"])
print(decoded)

⚠️ Notes

This repository contains only the tokenizer.
It should be used together with a compatible language model.
Performance depends on the model paired with this tokenizer.

🙏 Acknowledgments

Special thanks to:

Hugging Face for the Transformers ecosystem.
KordAI for developing and distributing the tokenizer.
The open-source AI community for advancing multilingual NLP.

📖 Citation

bibtex

@misc{kordaitokenizerfast2026,
  title={KordAI TokenizerFast},
  author={KordAI},
  year={2026},
  publisher={Hugging Face},
  howpublished={https://huggingface.co/KordAI/TokenizerFast}
}