This is the official release of the fine-tuned models for
the winning entry to the
W-NUT 2021: Multilingual Lexical Normalization (MultiLexNorm) shared task, which evaluates lexical-normalization systems on 12 social media datasets in 11 languages.
Our system is based on
ByT5, which we first pre-train on synthetic data and then fine-tune on authentic normalization data. It achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. In addition to these fine-tuned models, we also release the source files on
GitHub and an interactive demo on
Google Colab.
The model was
not fine-tuned in a standard sentence-to-sentence setting – instead, it was tailored to the token-to-token definition of MultiLexNorm data. Please refer to
the interactive demo on Colab notebook to learn how to use these models.
1@inproceedings{wnut-ufal,
2 title= "{ÚFAL} at {MultiLexNorm} 2021: Improving Multilingual Lexical Normalization by Fine-tuning {ByT5}",
3 author = "Samuel, David and Straka, Milan",
4 booktitle = "Proceedings of the 7th Workshop on Noisy User-generated Text (W-NUT 2021)",
5 year = "2021",
6 publisher = "Association for Computational Linguistics",
7 address = "Punta Cana, Dominican Republic"
8}
ByT5 is a tokenizer-free version of
Google's T5 and generally follows the architecture of
MT5.
ByT5 was only pre-trained on
mC4 excluding any supervised training with an average span-mask of 20 UTF-8 characters. Therefore, this model has to be fine-tuned before it is useable on a downstream task.
ByT5 works especially well on noisy text data,
e.g.,
google/byt5-small significantly outperforms
mt5-small on
TweetQA.
