README.md

---
library_name: transformers
license: mit
base_model: microsoft/mdeberta-v3-base
tags:
- stance-detection
- cross-lingual
- multilingual
- mdeberta-v3
- text-classification
- politics
- trackio
- trackio:https://huggingface.co/spaces/MatteoFasulo/huggingface-static-26c811
- generated_from_trainer
datasets:
- ZurichNLP/x_stance
language:
- de
- fr
- it
- en
metrics:
- accuracy
- f1
model-index:
- name: mdeberta-v3-xstance
  results:
      - task:
          type: text-classification
        dataset:
          type: x_stance
          name: X-Stance
          config: new_comments_defr
          split: test
        metrics:
          - type: f1
            name: Macro F1 — New Comments (DE)
            value: 81.77582896254508

          - type: f1
            name: Macro F1 — New Comments (FR)
            value: 82.56971487045625

      - task:
          type: text-classification
        dataset:
          type: x_stance
          name: X-Stance
          config: new_questions_defr
          split: test
        metrics:
          - type: f1
            name: Macro F1 — New Questions (DE)
            value: 77.93319187443443

          - type: f1
            name: Macro F1 — New Questions (FR)
            value: 79.93510514302365

      - task:
          type: text-classification
        dataset:
          type: x_stance
          name: X-Stance
          config: new_topics_defr
          split: test
        metrics:
          - type: f1
            name: Macro F1 — New Topics (DE)
            value: 76.45883287018658

          - type: f1
            name: Macro F1 — New Topics (FR)
            value: 79.69104147158947

      - task:
          type: text-classification
        dataset:
          type: x_stance
          name: X-Stance
          config: new_comments_it
          split: test
        metrics:
          - type: f1
            name: Macro F1 — New Comments (IT)
            value: 79.1612109408413
---
<a href="https://huggingface.co/spaces/MatteoFasulo/huggingface-static-26c811" target="_blank"><img src="https://raw.githubusercontent.com/gradio-app/trackio/refs/heads/main/trackio/assets/badge.png" alt="Visualize in Trackio" title="Visualize in Trackio" style="height: 40px;"/></a>
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# mdeberta-v3-xstance

A multilingual **stance detection** model fine-tuned from **microsoft/mdeberta-v3-base** on the **ZurichNLP/x_stance** dataset.

The model predicts whether a political comment expresses a **FAVOR** or **AGAINST** stance toward a given political question. It supports multilingual inference and demonstrates strong cross-lingual transfer across Swiss national languages.

---

# Highlights

- 🌍 Multilingual stance detection (🇩🇪 German (75%), 🇫🇷 French (25%), and 🇮🇹 Italian (only a few to test zero-shot cross-lingual transfer))
- ⚡ Built on mdeberta-v3
- 🎯 Binary stance classification (FAVOR / AGAINST)
- 🔄 Cross-lingual transfer capabilities

---

# Performance

## Validation Split

Evaluation on the validation split:

| Metric   |      Score |
| -------- | ---------: |
| Loss     | **0.5643** |
| Accuracy | **79.15%** |
| Macro F1 | **79.12%** |

## X-Stance Test Set

Detailed evaluation on the X-Stance test set as provided in the original repository:

| Evaluation setting | Language     |   Macro F1 |
| ------------------ | ------------ | ---------: |
| New comments       | German (DE)  | **81.775** |
| New comments       | French (FR)  | **82.569** |
| New questions      | German (DE)  | **77.933** |
| New questions      | French (FR)  | **79.935** |
| New topics         | German (DE)  | **76.458** |
| New topics         | French (FR)  | **79.691** |
| New comments       | Italian (IT) | **79.161** |

The evaluation script used to obtain these results is:

```bash
python evaluate.py \
  --gold data/test.jsonl \
  --pred predictions/mdeberta-v3_pred.jsonl
```

>Note: `mdeberta-v3_pred.jsonl` is provided in this repository for reproducibility. The evaluation script is available in the original repository at https://github.com/ZurichNLP/xstance .


# Quick Start

## Installation

```bash
pip install transformers torch
```

## Run inference

### Using the `pipeline` API (Recommended)

```python
from transformers import pipeline

classifier = pipeline(
    task="text-classification",
    model="MatteoFasulo/mdeberta-v3-xstance"
)

question = "Soll der Bundesrat ein Freihandelsabkommen mit den USA anstreben?"

comment = "Nicht unter einem Präsidenten, welcher die Rechte anderer mit Füssen tritt und Respektlos gegenüber ändern ist."

result = classifier(
    {
        "text": question,
        "text_pair": comment,
    }
)

print(result)
```

Example output:

```python
[{'label': 'AGAINST', 'score': 0.9823}]
```

For sequence-pair classification tasks such as stance detection, the `text-classification` pipeline accepts a dictionary with `"text"` and `"text_pair"` keys.

---

### Using `AutoModelForSequenceClassification`

```python
import torch
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
)

model_name = "MatteoFasulo/mdeberta-v3-xstance"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

question = "Soll der Bundesrat ein Freihandelsabkommen mit den USA anstreben?"

comment = "Nicht unter einem Präsidenten, welcher die Rechte anderer mit Füssen tritt und Respektlos gegenüber ändern ist."

inputs = tokenizer(
    question,
    comment,
    return_tensors="pt",
    truncation=True,
)

with torch.no_grad():
    outputs = model(**inputs)

probabilities = torch.softmax(outputs.logits, dim=-1)

prediction = probabilities.argmax(dim=-1).item()

id2label = model.config.id2label

print("Prediction:", id2label[prediction])
print("Confidence:", probabilities[0, prediction].item())
```

Example output:

```text
Prediction: AGAINST
Confidence: 0.9823
```

---

# Input Format

The model expects **two text sequences**:

1. **Target political question**
2. **Candidate comment**

Example:

```
Question:
Should Switzerland increase renewable energy subsidies?

Comment:
Investing in renewable energy will reduce emissions and improve energy independence.
```

The tokenizer automatically formats these as sentence pairs for mdeberta-v3.

---

# Output Labels

The classifier predicts one of two classes.

| Label | Description |
|--------|-------------|
| FAVOR | The comment supports the target question. |
| AGAINST | The comment opposes the target question. |

The model outputs logits for both classes.

---

# Model Description

This model is a fine-tuned version of **microsoft/mdeberta-v3-base** trained for multilingual, multi-target stance detection.

Unlike sentiment analysis, stance detection predicts whether a text **supports or opposes a specific target question**.

Because the underlying encoder is multilingual, the model can transfer knowledge across languages and perform inference on languages that were only partially represented during training.

---

# Intended Uses

The model is suitable for:

- Political stance detection
- Cross-lingual stance classification
- Research on multilingual NLP
- Opinion mining
- Benchmarking stance detection methods

---

# Out-of-Scope Uses

This model is **not** intended for:

- Fact checking
- Political affiliation prediction
- Hate speech detection
- Toxicity classification
- General sentiment analysis
- Automated political decision-making

---

# Training Dataset

Training was performed using the **ZurichNLP/x_stance** dataset.

Dataset characteristics:

- 150+ political questions
- 67,000 candidate comments
- Swiss political debates
- Multilingual annotations

Languages:

- German (majority)
- French
- Italian

Each sample consists of:

```
(Target Question, Candidate Comment)
→ FAVOR / AGAINST
```

---

# Training procedure

## Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 32
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 850
- num_epochs: 3

## Training results

| Training Loss | Epoch | Step | Validation Loss | Accuracy | Macro F1 |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|
| 0.4466        | 1.0   | 2853 | 0.4920          | 0.7886   | 0.7885   |
| 0.3395        | 2.0   | 5706 | 0.4551          | 0.8077   | 0.8077   |
| 0.2522        | 3.0   | 8559 | 0.5083          | 0.8153   | 0.8153   |

# Limitations

Although the model performs well on multilingual political stance detection, several limitations should be considered.

- Trained primarily on Swiss political debates.
- Binary labels only (no Neutral class).
- Performance outside politics has not been evaluated.
- Implicit or sarcastic opinions remain challenging.
- Domain shift may reduce performance on social media or informal discussions.

---

# Ethical Considerations

This model predicts **stance**, not factual correctness.

Predictions should not be interpreted as:

- political affiliation
- truthfulness
- misinformation detection
- ideological profiling

Human oversight is recommended for any downstream application.

---

# Framework versions

- Transformers 5.14.1
- Pytorch 2.8.0a0+5228986c39.nv25.06
- Datasets 5.0.0
- Tokenizers 0.22.2

# Citation

If you use this model, please cite the original X-Stance dataset.

```bibtex
@inproceedings{vamvas2020xstance,
    author    = "Vamvas, Jannis and Sennrich, Rico",
    title     = "{X-Stance}: A Multilingual Multi-Target Dataset for Stance Detection",
    booktitle = "Proceedings of the 5th Swiss Text Analytics Conference (SwissText) \& 16th Conference on Natural Language Processing (KONVENS)",
    address   = "Zurich, Switzerland",
    year      = "2020",
    month     = "jun",
    url       = "http://ceur-ws.org/Vol-2624/paper9.pdf"
}
```

---

# License

This model is released under the **MIT License**.

Please also respect the licenses of:

- microsoft/mdeberta-v3-base
- ZurichNLP/x_stance

---

# Acknowledgements

This model builds upon:

- Microsoft for mdeberta-v3-base
- Zurich NLP Group for the X-Stance dataset
- Hugging Face Transformers