File size: 4,156 Bytes
482233c 0ec8ff5 482233c 161afe1 482233c 4f680e8 482233c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 | ---
datasets:
- snli
- multi_nli
metrics:
- accuracy
- f1
- precision
- recall
inference: false
model-index:
- name: ettin-nli-classifier
results:
- task:
type: text-classification
name: Text Classification
metrics:
- type: loss
value: 0.62099
- type: accuracy
value: 0.88142
name: Accuracy
- type: f1
value: 0.88158
name: F1
- type: precision
value: 0.88182
name: Precision
- type: recall
value: 0.88142
name: Recall
language:
- en
license: mit
---
## Ettin-NLI-Classifier
This model uses the [jhu-clsp/ettin-encoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-68m) architecture, which has been fine-tuned for NLI tasks on the [MultiNLI](https://huggingface.co/datasets/multi_nli) and [SNLI](https://huggingface.co/datasets/snli) datasets. It was made to replace the [old distilroberta finetune](https://huggingface.co/AdamCodd/distilroberta-NLI).
It achieves the following results on the evaluation set:
* Accuracy: 0.88142
* F1: 0.88158
* Precision: 0.88182
* Recall: 0.88142
## Model description
The SNLI corpus (version 1.0) is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral, supporting the task of natural language inference (NLI), also known as recognizing textual entailment (RTE).
The Multi-Genre Natural Language Inference (MultiNLI) corpus is a crowd-sourced collection of 433k sentence pairs annotated with textual entailment information. The corpus is modeled on the SNLI corpus, but differs in that covers a range of genres of spoken and written text, and supports a distinctive cross-genre generalization evaluation.
## Usage
Inference API has been disabled as it is not suitable for this kind of task.
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
# Load model and tokenizer
model_checkpoint = 'AdamCodd/ettin-nli-classifier'
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint)
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
# Set device
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
# Sample premise and hypothesis
premise = "The cat is sleeping under the sun."
hypothesis = "It's raining, and the cat is getting wet."
# Tokenize and predict
input = tokenizer(premise, hypothesis, truncation=True, padding=True, return_tensors="pt", max_length=256).to(device)
with torch.no_grad():
output = model(**input)
probabilities = torch.softmax(output.logits, dim=-1)[0].tolist()
# Output prediction
label_names = ["Entailment", "Neutral", "Contradiction"]
prediction = {name: round(prob * 100, 1) for name, prob in zip(label_names, probabilities)}
print(prediction)
# e.g. {'Entailment': 1.3, 'Neutral': 8.2, 'Contradiction': 90.5}
```
## Training and evaluation data
The training data consists of a concatenated corpus of the SNLI train split and the MultiNLI train split. The evaluation metrics were calculated using a concatenated validation set consisting of the SNLI validation split and the MultiNLI `validation_matched` split. All `-1` labels (samples without annotator consensus) were filtered out prior to training.
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 32
- eval_batch_size: 64
- seed: 42
- optimizer: AdamW with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- warmup_ratio: 0.05
- num_epochs: 1
- weight_decay: 0.01
- mixed_precision: fp16
### Training results
Metrics: Accuracy, F1, Precision, Recall
```
'eval_loss': 0.62099,
'eval_accuracy': 0.88142,
'eval_f1': 0.88158,
'eval_precision': 0.88182,
'eval_recall': 0.88142
```
NB: The confusion matrix is [here](https://huggingface.co/AdamCodd/ettin-nli-classifier/blob/main/confusion_matrix.png).
### Framework versions
- Transformers >= 4.48.0
- Datasets >= 2.16.1
- Evaluate >= 0.4.1
- Tokenizers >= 0.15.0
If you want to support me, you can [here](https://ko-fi.com/adamcodd). |