ESNLIR — XLM-RoBERTa

XLM-RoBERTa fine-tuned on ESNLIR for 4-way natural language inference with a causal reasoning class. Takes a sentence pair. This is the strongest baseline reported in the paper.

ESNLIR: Expanding Spanish NLI Benchmarks with Multi-Genre and Causal Annotation Johan R. Portela, Nicolás Pérez-Terán, Rubén Manrique — Universidad de los Andes, Bogotá Applied Informatics, Springer, 2026, pp. 345–361 — doi:10.1007/978-3-032-07175-0_23

Part of the ESNLIR collection.

ESNLIR labels sentence pairs with four relations. Three come from the discourse connector linking the sentences; neutral is built by pairing sentences from different paragraphs.

id	label	meaning
0	`contrasting`	contradiction, comparison or opposition
1	`entailment`	generalization, specification or equivalence
2	`neutral`	no relation
3	`reasoning`	cause, reason or condition — the class ESNLIR adds

Results on the ESNLIR test set (80,216 pairs, class-balanced)

accuracy	macro F1	contrasting	entailment	neutral	reasoning
0.6756	0.6758	0.664	0.676	0.695	0.667

Stress tests

Four label-preserving perturbations of the test set (see the dataset card):

test	accuracy	macro F1	contrasting	entailment	neutral	reasoning
`test_length_mismatch`	0.6115	0.6087	0.513	0.524	0.755	0.654
`test_negation`	0.6385	0.6371	0.580	0.575	0.744	0.656
`test_overlap`	0.6611	0.6608	0.624	0.637	0.719	0.665
`test_spelling`	0.6674	0.6677	0.687	0.646	0.677	0.659

Padding the premise or negating the hypothesis costs contrasting and entailment about 15 points each while leaving neutral and reasoning untouched — the model leans on length and negation cues exactly where it should be reasoning semantically.

For reference, no prompted 7–8B LLM evaluated on this test set comes within 25 points of this model (predictions).

Training


base model	`FacebookAI/xlm-roberta-base`
data	`Flaglab/ESNLIR-dataset`, both sentences
max samples	1,000,000 (subsampled from the 4.4M train split, stratified)
epochs	6, early stopping on validation macro F1 (patience 3)
batch size	64
learning rate	2e-5
max sequence length	256
class weighting	balanced
seed	42

Trained with jd-rodriguezp1234/esnlir (auto_nli/model/bert_based/run.py).

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

name = "Flaglab/ESNLIR-XLM-RoBERTa"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name)

inputs = tok(
    "La evidencia que apoya la superioridad del liderazgo transformacional es abrumadora",
    "Una investigación más detallada es requerida",
    return_tensors="pt", truncation=True, max_length=256,
)
with torch.no_grad():
    logits = model(**inputs).logits
print(model.config.id2label[int(logits.argmax(-1))])

The tokenizer bundled here is an unmodified copy of the one from FacebookAI/xlm-roberta-base; vocabulary size matches this model's embedding table exactly.

Citation

@InProceedings{portela2025esnlirspanishmultigenredataset,
  author    = {Portela, Johan R. and P{\'e}rez-Ter{\'a}n, Nicol{\'a}s and Manrique, Rub{\'e}n},
  editor    = {Florez, Hector and Peluffo-Ordo{\~{n}}ez, Diego},
  title     = {{ESNLIR}: Expanding Spanish {NLI} Benchmarks with Multi-genre and Causal Annotation},
  booktitle = {Applied Informatics},
  year      = {2026},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {345--361},
  isbn      = {978-3-032-07175-0},
  doi       = {10.1007/978-3-032-07175-0_23},
}