rafaelsandroni/classifier-hallucination-projectJ

How to use

from sentence_transformers import SentenceTransformer
from setfit import SetFitModel, SetFitTrainer

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained(
    "rafaelsandroni/classifier-hallucination-projectJ"
)

# Run inference
preds = model("Para transferir R$ 500,00 para a chave Pix '+123123123123', vou realizar a transação agora.")
print(preds)

Training details

Dataset count

Hallucinated	Count
False	97
True	39

Eval method

Holdout, train_test split of 20%

Training Results

Best model at the end

Epoch	Training Loss	Validation Loss
1	0.000400	0.067576
2	0.000200	0.073225
3	0.000200	0.093793
4	0.000100	0.096496
5	0.000100	0.100173
6	0.000100	0.102722

Evaluation Results

Accuracy
0.9926470588235294

Probability distribution

ROC AUC