| --- |
| language: en |
| tags: |
| - nli |
| - contradiction-detection |
| - animised |
| - bert |
| license: apache-2.0 |
| --- |
| |
| # Animised NLI Contradiction Detector v2 |
|
|
| `prajjwal1/bert-small` (29M) trained directly on hard labels |
| with a **3:1 imbalanced dataset** to prevent contradiction bias. |
|
|
| ## Why v2? |
|
|
| The v1 model (distilled from teacher soft labels on a balanced dataset) |
| showed a strong **contradiction bias** — predicting contradiction even |
| in clearly entailing cases. |
|
|
| v2 fixes this by training on a deliberately imbalanced dataset where |
| entailment+neutral outnumber contradiction **3:1**. This makes the model |
| conservative about predicting contradiction — it requires stronger |
| evidence before flagging something as inconsistent. |
|
|
| ## Results |
| | Metric | Value | |
| |----------|------------------------------------| |
| | Accuracy | 0.8120 (81.20%) | |
| | Loss | 0.480365 | |
| | Epochs | 4 | |
|
|
| ## Labels |
| `0` = entailment | `1` = neutral | `2` = contradiction |
|
|
| ## Usage |
| ```python |
| from transformers import pipeline |
| |
| clf = pipeline("text-classification", model="Animised/nli-cdv2") |
| |
| clf( |
| "Rem was raised by her mother [SEP] Rem's mum taught her to cook soba.", |
| top_k=None |
| ) |
| ``` |
|
|
| ## Purpose |
| Character fact consistency checker for the |
| [Animised](https://huggingface.co/Animised) project — |
| detects when generated dialogue contradicts a character's bible. |
|
|
| ## Training details |
| - Base model : `prajjwal1/bert-small` (29M params) |
| - Dataset : [Animised/nli-v2](https://huggingface.co/datasets/Animised/nli-v2) |
| - Data ratio : 3:1 (entailment+neutral : contradiction) |
| - Loss : CrossEntropyLoss (hard labels, no distillation) |
| - Epochs : 4 |
| - Batch size : 512 |
| - Max length : 256 |
| - LR : 4e-05 |
| - GPUs : 2 |
|
|
| ## vs v1 |
| | Feature | v1 | v2 | |
| |---------------|-----------------------------|------------------------| |
| | Training | Distillation (soft labels) | Direct (hard labels) | |
| | Data balance | 1:1:1 | 3:1 (E+N:C) | |
| | Contradiction | Trigger-happy (~82% false) | Conservative | |
| | Accuracy | 79.6% | 81.20% | |
|
|
