X-MOD is a multilingual masked language model trained on filtered CommonCrawl data containing 81 languages. It was introduced in the paper
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (Pfeiffer et al., NAACL 2022) and first released in
this repository.
Because this model uses language adapters, you need to specify the language of your input so that the correct adapter can be activated:
1from transformers import XmodModel
2
3model = XmodModel.from_pretrained("facebook/xmod-base")
4model.set_default_language("en_XX")
A directory of the language adapters in this model is found at the bottom of this model card.
In the experiments in the original paper, the embedding layer and the language adapters are frozen during fine-tuning. A method for doing this is provided in the code:
1model.freeze_embeddings_and_language_adapters()
2# Fine-tune the model ...
After fine-tuning, zero-shot cross-lingual transfer can be tested by activating the language adapter of the target language:
1model.set_default_language("de_DE")
2# Evaluate the model on German examples ...
1@inproceedings{pfeiffer-etal-2022-lifting,
2 title = "Lifting the Curse of Multilinguality by Pre-training Modular Transformers",
3 author = "Pfeiffer, Jonas and
4 Goyal, Naman and
5 Lin, Xi and
6 Li, Xian and
7 Cross, James and
8 Riedel, Sebastian and
9 Artetxe, Mikel",
10 booktitle = "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",
11 month = jul,
12 year = "2022",
13 address = "Seattle, United States",
14 publisher = "Association for Computational Linguistics",
15 url = "https://aclanthology.org/2022.naacl-main.255",
16 doi = "10.18653/v1/2022.naacl-main.255",
17 pages = "3479--3495"
18}
