Hugging Face's logo

Flaglab
/
ESNLIR-Baseline

Text Classification
Scikit-learn
Spanish
nli
spanish
causal
esnlir
xgboost
baseline
Model card Files
xet
 Community
ESNLIR-Baseline
File size: 4,471 Bytes
b448e91
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
---
license: cc-by-4.0
language:
- es
pipeline_tag: text-classification
library_name: sklearn
datasets:
- Flaglab/ESNLIR-dataset
metrics:
- accuracy
- f1
tags:
- nli
- spanish
- causal
- esnlir
- xgboost
- baseline
---

# ESNLIR — XGBoost baseline

The **non-neural reference point** for ESNLIR: XGBoost over bag-of-words features from both
sentences. It exists to show how much of the task is solvable without contextual representations —
the answer is *not much*, which is what makes the BERT results meaningful.

> **ESNLIR: Expanding Spanish NLI Benchmarks with Multi-Genre and Causal Annotation**
> Johan R. Portela, Nicolás Pérez-Terán, Rubén Manrique — Universidad de los Andes, Bogotá
> *Applied Informatics*, Springer, 2026, pp. 345–361 —
> [doi:10.1007/978-3-032-07175-0_23](https://doi.org/10.1007/978-3-032-07175-0_23)

Part of the [ESNLIR collection](https://huggingface.co/collections/Flaglab/esnlir-6914a3103bb229f3c3f09567).

## Results on the ESNLIR test set (80,216 pairs, class-balanced)

| accuracy | macro F1 | contrasting | entailment | neutral | reasoning |
|---|---|---|---|---|---|
| **0.3501** | **0.3481** | 0.345 | 0.436 | 0.278 | 0.341 |

Ten points above the 0.250 majority-class floor, and roughly **half** what the fine-tuned encoders
reach:

| model | accuracy | macro F1 |
|---|---|---|
| Majority class | 0.250 | 0.250 |
| **XGBoost (this model)** | **0.350** | **0.348** |
| [BERTIN RoBERTa](https://huggingface.co/Flaglab/ESNLIR-RoBERTa) | 0.663 | 0.664 |
| [XLM-RoBERTa](https://huggingface.co/Flaglab/ESNLIR-XLM-RoBERTa) | 0.676 | 0.676 |

Lexical features alone barely move the needle, so the relations in ESNLIR are not recoverable from
word identity — contextual sentence representations are doing the work.

### Stress tests

| test | accuracy | macro F1 | contrasting | entailment | neutral | reasoning |
|---|---|---|---|---|---|---|
| `test` | 0.3501 | 0.3481 | 0.345 | 0.436 | 0.278 | 0.341 |
| `test_length_mismatch` | 0.3257 | 0.2945 | 0.204 | 0.307 | 0.111 | 0.681 |
| `test_negation` | 0.3381 | 0.3287 | 0.401 | 0.266 | 0.200 | 0.486 |
| `test_overlap` | 0.3409 | 0.3313 | 0.226 | 0.360 | 0.245 | 0.532 |
| `test_spelling` | 0.3490 | 0.3473 | 0.340 | 0.428 | 0.280 | 0.348 |

The perturbations push predictions hard toward `reasoning` (0.341 → 0.681 under length mismatch),
which is what a bag-of-words model does when tokens are injected: the added words shift the feature
vector rather than the meaning.

## Labels

| id | label |
|---|---|
| 0 | `contrasting` |
| 1 | `entailment` |
| 2 | `neutral` |
| 3 | `reasoning` |

Label order is alphabetical, from `sklearn.preprocessing.LabelEncoder`.

## Files

`model.pkl` — a pickled scikit-learn/XGBoost estimator. `metrics.zip` — full metrics per split,
broken down by genre, domain and source corpus.

## Usage

```python
import pickle
from huggingface_hub import hf_hub_download

with open(hf_hub_download("Flaglab/ESNLIR-Baseline", "model.pkl"), "rb") as fh:
    model = pickle.load(fh)
```

> Loading a pickle executes arbitrary code — only do this because you trust the source.
>
> The vectorizer is **not** included in this repository, so the model cannot be applied to raw text
> as published. Rebuild the bag-of-words features with
> [`jd-rodriguezp1234/esnlir`](https://github.com/jd-rodriguezp1234/esnlir)
> (`auto_nli/model/baseline/dataset.py`), which fits the vectorizer and label encoder on the
> training split. For inference on new text, prefer
> [`ESNLIR-XLM-RoBERTa`](https://huggingface.co/Flaglab/ESNLIR-XLM-RoBERTa).

## Training

| | |
|---|---|
| features | bag-of-words over both sentences |
| data | [`Flaglab/ESNLIR-dataset`](https://huggingface.co/datasets/Flaglab/ESNLIR-dataset) |
| max samples | 1,000,000 (subsampled from the 4.4M train split) |

Trained with `auto_nli/model/baseline/run.py`, config `params/model/baseline.json`.

## Citation

```bibtex
@InProceedings{portela2025esnlirspanishmultigenredataset,
  author    = {Portela, Johan R. and P{\'e}rez-Ter{\'a}n, Nicol{\'a}s and Manrique, Rub{\'e}n},
  editor    = {Florez, Hector and Peluffo-Ordo{\~{n}}ez, Diego},
  title     = {{ESNLIR}: Expanding Spanish {NLI} Benchmarks with Multi-genre and Causal Annotation},
  booktitle = {Applied Informatics},
  year      = {2026},
  publisher = {Springer Nature Switzerland},
  address   = {Cham},
  pages     = {345--361},
  isbn      = {978-3-032-07175-0},
  doi       = {10.1007/978-3-032-07175-0_23},
}
```