Model Card for Model ID
This is a
XLM-RoBERTa-base fine-tuned model on 5K (premise, hypothesis) sentence pairs from
the
ASSIN (Avaliação de Similaridade Semântica e Inferência textual) corpus. The original reference papers are:
Unsupervised Cross-Lingual Representation Learning At Scale,
ASSIN: Avaliação de Similaridade Semântica e Inferência Textual, respectivelly. This model is suitable for Portuguese (from Brazil or Portugal).
Model Details
Model Description
- Developed by: Giovani Tavares and Felipe Ribas Serras
- Oriented By: Renata Wassermann, Felipe Ribas Serras and Marcelo Finger
- Model type: Transformer-based text classifier
- Language(s) (NLP): Portuguese
- License: mit
- Finetuned from model XLM-RoBERTa-base
Model Sources
- Repository: Natural-Portuguese-Language-Inference
- Paper: This is an ongoing research. We are currently writing a paper where we fully describe our experiments.
Uses
Direct Use
This fine-tuned version of
XLM-RoBERTa-base performs Natural
Language Inference (NLI), which is a text classification task. Therefore, it classifies pairs of sentences in the form (premise, hypothesis) into one of the following classes ENTAILMENT, PARAPHRASE or NONE. Salvatore's definition [1] for ENTAILEMENT is assumed to be the same as the one found in
ASSIN's labels in which this model was trained on.
PARAPHRASE and NONE are not defined in [1].Therefore, it is assumed that in this model's training set, given a pair of sentences (paraphase, hypothesis), hypothesis is a PARAPHRASE of premise if premise is an ENTAILMENT of hypothesis and vice-versa. If (premise, hypothesis) don't have an ENTAILMENT or PARAPHARSE relationship, (premise, hypothesis) is classified as NONE.
Demo
1from transformers import AutoModelForSequenceClassification, AutoTokenizer
2import torch
3
4model_path = "giotvr/portuguese-nli-3-labels"
5premise = "As mudanças climáticas são uma ameaça séria para a biodiversidade do planeta."
6hypothesis ="A biodiversidade do planeta é seriamente ameaçada pelas mudanças climáticas."
7tokenizer = XLMRobertaTokenizer.from_pretrained(model_path, use_auth_token=True)
8input_pair = tokenizer(premise, hypothesis, return_tensors="pt",padding=True, truncation=True)
9model = AutoModelForSequenceClassification.from_pretrained(model_path, use_auth_token=True)
10
11with torch.no_grad():
12 logits = model(**input_pair).logits
13probs = torch.nn.functional.softmax(logits, dim=-1)
14probs, sorted_indices = torch.sort(probs, descending=True)
15for i, score in enumerate(probs[0]):
16 print(f"Class {sorted_indices[0][i]}: {score.item():.4f}")
Recommendations
This model should be used for scientific purposes only. It was not tested for production environments.
Fine-Tuning Details
Fine-Tuning Data
-
-
Evaluation Dataset used for Hyperparameter Tuning: ASSIN's validation split
-
Test Datasets:
- ASSIN's test splits
- ASSIN2's test splits
This is a fine tuned version of
XLM-RoBERTa-base using the
ASSIN (Avaliação de Similaridade Semântica e Inferência textual) dataset.
ASSIN is a corpus annotated with hypothesis/premise Portuguese sentence pairs suitable for detecting textual entailment, paraphrase or neutral
relationship between the members of such pairs. Such corpus has three subsets:
ptbr (Brazilian Portuguese),
ptpt (Portuguese Portuguese) and
full (the union of the latter with the former). The
full subset has
10k sentence pairs equally distributed between
ptbr and
ptpt subsets.
Fine-Tuning Procedure
The model's fine-tuning procedure can be summarized in three major subsequent tasks:
- Data Processing:
ASSIN's validation and train splits were loaded from the Hugging Face Hub and processed afterwards;
- Hyperparameter Tuning:
XLM-RoBERTa-base's hyperparameters were chosen with the help of the Weights & Biases API to track the results and upload the fine-tuned models;
- Final Model Loading and Testing:
The models' performance was evaluated using different datasets and metrics that will be better described in the future paper.
Hyperparameter Tuning
The following hyperparameters were tested in order to maximize the evaluation accuracy.
- Number of Training Epochs: (1,2,3)
- Per Device Train Batch Size: (16,32)
- Learning Rate: (1e-6, 2e-6,3e-6)
The hyperaparemeter tuning experiments were run and tracked using the
Weights & Biases' API and can be found at this
link.
Training Hyperparameters
The
hyperparameter tuning performed yelded the following values:
- Number of Training Epochs: 3
- Per Device Train Batch Size: 16
- Learning Rate: 3e-6
Evaluation
ASSIN
Testing this model in
ASSIN's test split is straightforward because this model was tested using
ASSIN's training set and therefore can predict the same labels as the ones found in its test set.
ASSIN2
ASSIN2's test split's class label's column has only two possible values:
ENTAILMENT and
NONE. Therefore some mapping must be done so this model can be tested in
ASSIN2's test split. More information on how such mapping is performed will be available in the
referred paper.
Metrics
The model's performance metrics for each test dataset are presented separately. Accuracy, f1 score, precision and recall were the metrics used to every evaluation performed. Such metrics are reported below. More information on such metrics them will be available in our ongoing research paper.
Results
| test set | accuracy | f1 score | precision | recall |
|---|
| assin | 0.89 | 0.89 | 0.89 | 0.89 |
| assin2 | 0.70 | 0.69 | 0.73 | 0.70 |
Model Examination
Some interpretability work is being done in order to understand the model's behavior. Such details will be available in the previoulsy referred paper.
References
