X-MOD is a multilingual masked language model trained on filtered CommonCrawl data containing 81 languages. It was introduced in the paper
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (Pfeiffer et al., NAACL 2022) and first released in
this repository.
Because it has been pre-trained with language-specific modular components (
language adapters), X-MOD differs from previous multilingual models like
XLM-R. For fine-tuning, the language adapters in each transformer layer are frozen.
This model reuses the tokenizer of
XLM-R.
Because this model uses language adapters, you need to specify the language of your input so that the correct adapter can be activated:
1from transformers import XmodModel
2
3model = XmodModel.from_pretrained("facebook/xmod-base")
4model.set_default_language("en_XX")
A directory of the language adapters in this model is found at the bottom of this model card.
In the experiments in the original paper, the embedding layer and the language adapters are frozen during fine-tuning. A method for doing this is provided in the code:
1model.freeze_embeddings_and_language_adapters()
2# Fine-tune the model ...
After fine-tuning, zero-shot cross-lingual transfer can be tested by activating the language adapter of the target language:
1model.set_default_language("de_DE")
2# Evaluate the model on German examples ...
Please refer to the model card of
XLM-R, because X-MOD has a similar architecture and has been trained on similar training data.
1@inproceedings{pfeiffer-etal-2022-lifting,
2 title = "Lifting the Curse of Multilinguality by Pre-training Modular Transformers",
3 author = "Pfeiffer, Jonas and
4 Goyal, Naman and
5 Lin, Xi and
6 Li, Xian and
7 Cross, James and
8 Riedel, Sebastian and
9 Artetxe, Mikel",
10 booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
11 month = jul,
12 year = "2022",
13 address = "Seattle, United States",
14 publisher = "Association for Computational Linguistics",
15 url = "https://aclanthology.org/2022.naacl-main.255",
16 doi = "10.18653/v1/2022.naacl-main.255",
17 pages = "3479--3495"
18}
