Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
byt5-dv – AI Model by monsoon-nlp | AlphaNeural AI
You can deploy this model and start earning money today!
monsoon-nlp
/
byt5-dv
like
0
transformers
pytorch
tf
safetensors
t5
text2text-generation
dv
text-generation-inference
endpoints_compatible
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
byt5-dv
Pretrained from scratch on Dhivei (language of the Maldives) with ByT5, Google's new byte-level tokenizer strategy.
Corpus: dv.wikipedia.org as of March 2020 (TFDS)
Notebook - Pretraining on Wikipedia:
https://colab.research.google.com/drive/19Afq7CI6cOi1DaTpnQhBbEbnBzLSFHbH
Demo
Notebook - Finetuning on Maldivian news classification task:
https://colab.research.google.com/drive/11u5SafR4bKICmArgDl6KQ9vqfYtDpyWp
Current performance:
mBERT: 52%
byt5-dv
: 81%
dv-wave (ELECTRA): 89%
dv-muril: 90.7%
dv-labse: 91.3-91.5%
Source of dataset:
https://github.com/Sofwath/DhivehiDatasets
Work in progress - todos
The Wikipedia corpus is too small for this language. In the future I would add OSCAR and Sofwath's Maldivian corpus, if I can rewrite the script to accept those as one TFDS dataset.
This is based on ByT5-small ... we should try a larger model
This needs more time for pretraining
