ESNLIR: Expanding Spanish NLI Benchmarks with Multi-Genre and Causal Annotation Johan R. Portela, Nicolás Pérez-Terán, Rubén Manrique — Universidad de los Andes, Bogotá Applied Informatics, Springer, 2026, pp. 345–361 — doi:10.1007/978-3-032-07175-0_23

Part of the ESNLIR collection.

ESNLIR labels sentence pairs with four relations. Three come from the discourse connector linking the sentences; neutral is built by pairing sentences from different paragraphs.

id	label	meaning
0	`contrasting`	contradiction, comparison or opposition
1	`entailment`	generalization, specification or equivalence
2	`neutral`	no relation
3	`reasoning`	cause, reason or condition — the class ESNLIR adds

Results on the ESNLIR test set (80,216 pairs, class-balanced)

accuracy	macro F1	contrasting	entailment	neutral	reasoning
0.6634	0.6639	0.683	0.626	0.669	0.675

Stress tests

Four label-preserving perturbations of the test set (see the dataset card):

test	accuracy	macro F1	contrasting	entailment	neutral	reasoning
`test_length_mismatch`	0.6154	0.6117	0.555	0.449	0.689	0.768
`test_negation`	0.6404	0.6398	0.646	0.569	0.698	0.649
`test_overlap`	0.6468	0.6463	0.652	0.579	0.703	0.654
`test_spelling`	0.6555	0.6560	0.688	0.606	0.660	0.668

Misspellings cost little; padding the premise costs entailment 18 points. The multilingual ESNLIR-XLM-RoBERTa scores about a point higher overall.

Training


base model	`bertin-project/bertin-roberta-base-spanish`
data	`Flaglab/ESNLIR-dataset`, both sentences
max samples	1,000,000 (subsampled from the 4.4M train split, stratified)
epochs	6, early stopping on validation macro F1 (patience 3)
batch size	64
learning rate	2e-5
max sequence length	256
class weighting	balanced
seed	42

Trained with jd-rodriguezp1234/esnlir (auto_nli/model/bert_based/run.py).

Usage

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

name = "Flaglab/ESNLIR-RoBERTa"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name)

inputs = tok(
    "La evidencia que apoya la superioridad del liderazgo transformacional es abrumadora",
    "Una investigación más detallada es requerida",
    return_tensors="pt", truncation=True, max_length=256,
)
with torch.no_grad():
    logits = model(**inputs).logits
print(model.config.id2label[int(logits.argmax(-1))])

The tokenizer bundled here is an unmodified copy of the one from bertin-project/bertin-roberta-base-spanish; vocabulary size matches this model's embedding table exactly.

Citation

bibtex

@InProceedings{portela2025esnlirspanishmultigenredataset,
  author    = {Portela, Johan R. and P{\'e}rez-Ter{\'a}n, Nicol{\'a}s and Manrique, Rub{\'e}n},
  editor    = {Florez, Hector and Peluffo-Ordo{\~{n}}ez, Diego},
  title     = {{ESNLIR}: Expanding Spanish {NLI} Benchmarks with Multi-genre and Causal Annotation},
  booktitle = {Applied Informatics},
  year      = {2026},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {345--361},
  isbn      = {978-3-032-07175-0},
  doi       = {10.1007/978-3-032-07175-0_23},
}