Views
No views yet
google/flan-t5-base to support Dhivehi (Thaana script) characters, while preserving English subword tokenization.google/flan-t5-base_This, _is, etc.)1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained("alakxender/dhivehi-T5-tokenizer-extended")
4
5# Dhivehi example
6text = "ހެނބަދޫ މުދިންބެ: ކުދިން ހަތިމްކުރާ ކޮންމެ ފަހަރަކު ލޮލުން ކަރުނަ!"
7tokens = tokenizer.tokenize(text)
8print(tokens)| Feature | Stock Flan-T5 Tokenizer | Extended Dhivehi Tokenizer |
|---|---|---|
| Dhivehi support | ❌ Uses | Proper tokenization |
| English tokenization | ✅ Yes | Preserved |
| Added tokens | ❌ No | Thaana characters |
flan-t5-base tokenizer does not support Dhivehi text properly:1from transformers import AutoTokenizer
2
3stock_tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")
4tokens = stock_tokenizer.tokenize("ހެނބަދޫ މުދިންބެ: ކުދިން ހަތިމްކުރާ ކޮންމެ ފަހަރަކު ލޮލުން ކަރުނަ!")
5print(tokens)
6# Output: ['<unk>', '<unk>']<unk> tokens.