The XLM model was proposed in
Cross-lingual Language Model Pretraining by Guillaume Lample and Alexis Conneau. It’s a transformer pretrained with either a causal language modeling (CLM) objective (next token prediction), a masked language modeling (MLM) objective (BERT-like), or
a Translation Language Modeling (TLM) object (extension of BERT’s MLM to multiple language inputs). This model is trained with a masked language modeling objective on English text.
The model is a language model. The model can be used for masked language modeling.
To learn more about this task and potential downstream uses, see the Hugging Face
fill mask docs and the
Hugging Face Multilingual Models for Inference docs. Also see the
associated paper.
The model should not be used to intentionally create hostile or alienating environments for people.
Significant research has explored bias and fairness issues with language models (see, e.g.,
Sheng et al. (2021) and
Bender et al. (2021)).
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
More information needed. See the
associated GitHub Repo.
More information needed. See the
associated GitHub Repo.
Carbon emissions can be estimated using the
Machine Learning Impact calculator presented in
Lacoste et al. (2019).
1@article{lample2019cross,
2 title={Cross-lingual language model pretraining},
3 author={Lample, Guillaume and Conneau, Alexis},
4 journal={arXiv preprint arXiv:1901.07291},
5 year={2019}
6}
This model card was written by the team at Hugging Face.
Use the code below to get started with the model. See the Hugging Face XLM docs for more examples.
1from transformers import XLMTokenizer, XLMModel
2import torch
3
4tokenizer = XLMTokenizer.from_pretrained("xlm-mlm-en-2048")
5model = XLMModel.from_pretrained("xlm-mlm-en-2048")
6
7inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
8outputs = model(**inputs)
9
10last_hidden_states = outputs.last_hidden_state
