| --- |
| language: en |
| license: cc-by-sa-4.0 |
| library_name: transformers |
| pipeline_tag: text-classification |
| tags: |
| - natural-language-inference |
| - monotonicity |
| - negation |
| - inoculation |
| - reproduction |
| - bert |
| datasets: |
| - stanfordnlp/snli |
| base_model: bert-base-uncased |
| --- |
| |
| # monli-bert-inoculated |
|
|
| `monli-bert-snli` further fine-tuned ("inoculated") on **NMoNLI** so that it learns |
| the negation + monotonicity rule, while a mix of SNLI examples preserves general |
| NLI performance. |
|
|
| This is a **reproduction artifact** — the *Experiment 1.2 model* from our |
| reproduction of: |
|
|
| > Atticus Geiger, Kyle Richardson, Christopher Potts (2020). |
| > *Neural Natural Language Inference Models Partially Embed Theories of Lexical |
| > Entailment and Negation.* BlackboxNLP @ EMNLP 2020. [arXiv:2004.14623](https://arxiv.org/abs/2004.14623) |
|
|
| ## Training |
|
|
| Two stages (see base model [`HandongAILab/monli-bert-snli`](https://huggingface.co/HandongAILab/monli-bert-snli) for stage 1): |
|
|
| | Setting | Value | |
| |---------|-------| |
| | Base model | `monli-bert-snli` | |
| | Inoculation data | NMoNLI train (1,002 examples) | |
| | Mixed-in data | 10K random SNLI examples (prevents catastrophic forgetting) | |
| | Epochs | 10 (best checkpoint at epoch 4) | |
| | Learning rate | 5e-6 (low, to preserve SNLI representations) | |
| | Batch size | 48 | |
|
|
| The low learning rate + SNLI mixing are the key choices that let the model acquire |
| negation handling without losing SNLI accuracy. The paper does not disclose its |
| inoculation configuration; this combination reproduces its reported behavior. |
|
|
| ## Results |
|
|
| | Dataset | Accuracy | |
| |---------|----------| |
| | SNLI test | 90.2% (paper 90.5%) | |
| | PMoNLI (no negation) | 92.8% (paper: not reported) | |
| | NMoNLI test (with negation) | 98.0% (paper 90.0%) | |
|
|
| NMoNLI rises from ~3% (baseline) to 98%, and generalizes to **word pairs unseen |
| during inoculation** — the systematic-generalization result. PMoNLI is *not* |
| degraded (mild positive cross-transfer), which the paper did not report. |
|
|
| ## Labels |
|
|
| | id | label | |
| |----|-------| |
| | 0 | entailment | |
| | 1 | neutral | |
| | 2 | contradiction | |
|
|
| ## Reproduction |
|
|
| Baseline (before inoculation): [`HandongAILab/monli-bert-snli`](https://huggingface.co/HandongAILab/monli-bert-snli) |
|
|
| ## Limitations |
|
|
| A reproduction model on a small, templated challenge set (MoNLI). High NMoNLI |
| accuracy reflects MoNLI's structure and should not be read as general negation |
| competence. Probing/intervention analyses (see notebook) show the negation rule is |
| only **partially** localized internally. |
|
|
