NLP-Debater-Project/debertav3-stance-detection

Stance Detection with DeBERTa-v3-large

This model detects whether an argument supports (PRO) or opposes (CON) a given topic.

Model Description

• Base Model: microsoft/deberta-v3-large
• Task: Binary stance classification (PRO/CON)
• Training Data: IBM ArgKP-2023 dataset (~32,000 examples)
• Calibration: Label smoothing (0.1) for proper confidence scores

Performance

• Test Accuracy: 99.97%
• Test F1 Score: 99.97%
• Mean Confidence: 93.9% (well-calibrated)
• Calibration: ECE < 0.10

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model
model_name = "yassine-mhirsi/debertav3-stance-detection"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Predict
topic = "AI should replace human teachers"
argument = "Teachers provide emotional support that AI cannot replicate"

text = f"Topic: {{topic}} [SEP] Argument: {{argument}}"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.nn.functional.softmax(outputs.logits, dim=-1)
    predicted_class = torch.argmax(probs, dim=-1).item()

stance = "PRO" if predicted_class == 1 else "CON"
confidence = probs[0][predicted_class].item()

print(f"Stance: {{stance}}")
print(f"Confidence: {{confidence:.2%}}")

Training Details

• Epochs: 3
• Learning Rate: 3e-6
• Batch Size: 4 (with gradient accumulation of 4)
• Label Smoothing: 0.1
• Training Time: ~1.5 hours on Kaggle GPU

Limitations

• Trained only on English argumentative text
• Best performance on formal arguments (debate-style)
• May struggle with heavy sarcasm or irony
• Calibrated for confidence, but not perfect

Citation

If you use this model, please cite:

@misc{{stance-detection-deberta,
  author = Yassine Mhirsi,
  title = {{Stance Detection with DeBERTa-v3-large}},
  year = {{2025}},
  publisher = {{Hugging Face}},
  howpublished = {{\\url{{https://huggingface.co/yassine-mhirsi/debertav3-stance-detection}}}}
}}

License

MIT License