Views
No views yet
google/mt5-base by incorporating full Dhivehi language support, while preserving the original multilingual capabilities of mT5.Based on a 300k vocab SentencePiece model trained on a combined corpus of multilingual samples + Dhivehi text.
google/mt5-base300,000NFKC| Detail | Value |
|---|---|
| Tokenizer | SentencePieceTrainer |
| Model type | unigram |
| Vocab size | 300000 |
| Character coverage | 0.9995 |
| Input sentence size | 5,000,000 |
| Special tokens | <pad>, <unk>, <s>, </s> |
| Byte fallback | Enabled |
| Normalization | NFKC |
1from transformers import MT5Tokenizer
2
3tokenizer = MT5Tokenizer.from_pretrained("your-username/mt5-tokenizer-truly-extended")
4
5text = "ރިޔާސީ އިންތިހާބުގައި ވާދަކުރަށްވަން ނަޝީދު ހިޔާލު ހޯއްދަވަނީ"
6tokens = tokenizer.tokenize(text)
7print(tokens)| Language | Tokenization | Result |
|---|---|---|
| Dhivehi | Fragmented in base | Improved |
| Multilingual | Supported as before | Retained |
| Mixed sentences | Fragile in base | Improved |
1ids = tokenizer.encode("ރިޔާސީ", add_special_tokens=False)
2decoded = tokenizer.decode(ids)
3assert decoded == "ރިޔާސީ"spiece.model: The new SentencePiece model (300k vocab)tokenizer_config.json: Updated for extended vocabspecial_tokens_map.json: Preserved from mt5-basegoogle/mt5-base model.1from transformers import MT5ForConditionalGeneration
2
3model = MT5ForConditionalGeneration.from_pretrained("google/mt5-base")
4model.resize_token_embeddings(len(tokenizer))