BARTO model pre-trained on Spanish language. It was introduced in the paper
Sequence-to-Sequence Spanish Pre-trained Language Models.
BARTO is a BART-based model (transformer encoder-decoder) with a bidirectional (BERT-like) encoder and an autoregressive (GPT-like) decoder. BART is pre-trained by (1) corrupting text with an arbitrary noising function and (2) learning a model to reconstruct the original text.
BARTO is particularly effective when fine-tuned for text generation (e.g. summarization, translation) but also works well for comprehension tasks (e.g. text classification, question answering).
You can use the raw model for text infilling. However, the model is mainly meant to be fine-tuned on a supervised dataset.
This model does not have a slow tokenizer (BartTokenizer).
1from transformers import AutoTokenizer, AutoModel
2
3tokenizer = AutoTokenizer.from_pretrained('vgaraujov/bart-base-spanish')
4model = AutoModel.from_pretrained('vgaraujov/bart-base-spanish')
5
6inputs = tokenizer("Hola amigo, bienvenido a casa.", return_tensors="pt")
7outputs = model(**inputs)
8
9last_hidden_states = outputs.last_hidden_state
1@misc{araujo2023sequencetosequence,
2 title={Sequence-to-Sequence Spanish Pre-trained Language Models},
3 author={Vladimir Araujo and Maria Mihaela Trusca and Rodrigo Tufiño and Marie-Francine Moens},
4 year={2023},
5 eprint={2309.11259},
6 archivePrefix={arXiv},
7 primaryClass={cs.CL}
8}
