You can deploy this model and start earning money today!

alakxender/
xlmr-dhivehi-tokenizer-extended

Name: xlmr-dhivehi-tokenizer-extended
Availability: InStock

transformers

thaana-xlmr-tokenizer

alakxender/haveeru-articles

alakxender/dhivehi-news-corpus

mit

endpoints_compatible

Views

No views yet

Dhivehi-Extended XLM-RoBERTa Tokenizer

This tokenizer extends xlm-roberta-base with 52,000 additional tokens derived from a large Dhivehi corpus. It significantly improves tokenization and decoding of Thaana script (Dhivehi language) content.

What’s Included

Base model: xlm-roberta-base
52,000 high-frequency Dhivehi tokens added using SentencePiece
Custom added_tokens.json file
Improved Thaana tokenization and decoding

Tokenization Example

Stock XLM-R

python

Text: އީދުގެ ހަރަކާތް ...
Tokens: ['▁', '<unk>', ...]
Decoded: <unk> <unk> ...

Extended Tokenizer

python

Tokens: ['އީދު', 'ގެ', ' ހަރަކާތް', ...]
Decoded: އީދުގެ ހަރަކާތް ...

Notes

This tokenizer is best suited for Dhivehi tasks like:

Named Entity Recognition (NER)
Translation
Language Modeling

English and other XLM-R languages remain fully compatible.

alakxender/xlmr-dhivehi-tokenizer-extended