Hugging Face's logo

MatteoFasulo
/
mdeberta-v3-xstance

Text Classification
Transformers
Safetensors
deberta-v2
stance-detection
cross-lingual
multilingual
mdeberta-v3
politics
trackio
Generated from Trainer
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Metrics Community
mdeberta-v3-xstance / README.md
MatteoFasulo's picture
MatteoFasulo
Update README.md
80610a8 verified
preview code
|
Raw
History Blame Contribute Delete
10.5 kB
metadata 
library_name: transformers
license: mit
base_model: microsoft/mdeberta-v3-base
tags:
  - stance-detection
  - cross-lingual
  - multilingual
  - mdeberta-v3
  - text-classification
  - politics
  - trackio
  - trackio:https://huggingface.co/spaces/MatteoFasulo/huggingface-static-26c811
  - generated_from_trainer
datasets:
  - ZurichNLP/x_stance
language:
  - de
  - fr
  - it
  - en
metrics:
  - accuracy
  - f1
model-index:
  - name: mdeberta-v3-xstance
    results:
      - task:
          type: text-classification
        dataset:
          type: x_stance
          name: X-Stance
          config: new_comments_defr
          split: test
        metrics:
          - type: f1
            name: Macro F1 — New Comments (DE)
            value: 81.77582896254508
          - type: f1
            name: Macro F1 — New Comments (FR)
            value: 82.56971487045625
      - task:
          type: text-classification
        dataset:
          type: x_stance
          name: X-Stance
          config: new_questions_defr
          split: test
        metrics:
          - type: f1
            name: Macro F1 — New Questions (DE)
            value: 77.93319187443443
          - type: f1
            name: Macro F1 — New Questions (FR)
            value: 79.93510514302365
      - task:
          type: text-classification
        dataset:
          type: x_stance
          name: X-Stance
          config: new_topics_defr
          split: test
        metrics:
          - type: f1
            name: Macro F1 — New Topics (DE)
            value: 76.45883287018658
          - type: f1
            name: Macro F1 — New Topics (FR)
            value: 79.69104147158947
      - task:
          type: text-classification
        dataset:
          type: x_stance
          name: X-Stance
          config: new_comments_it
          split: test
        metrics:
          - type: f1
            name: Macro F1 — New Comments (IT)
            value: 79.1612109408413

Visualize in Trackio

mdeberta-v3-xstance

A multilingual stance detection model fine-tuned from microsoft/mdeberta-v3-base on the ZurichNLP/x_stance dataset.

The model predicts whether a political comment expresses a FAVOR or AGAINST stance toward a given political question. It supports multilingual inference and demonstrates strong cross-lingual transfer across Swiss national languages.

Highlights

  • 🌍 Multilingual stance detection (🇩🇪 German (75%), 🇫🇷 French (25%), and 🇮🇹 Italian (only a few to test zero-shot cross-lingual transfer))
  • ⚡ Built on mdeberta-v3
  • 🎯 Binary stance classification (FAVOR / AGAINST)
  • 🔄 Cross-lingual transfer capabilities

Performance

Validation Split

Evaluation on the validation split:

Metric Score
Loss 0.5643
Accuracy 79.15%
Macro F1 79.12%

X-Stance Test Set

Detailed evaluation on the X-Stance test set as provided in the original repository:

Evaluation setting Language Macro F1
New comments German (DE) 81.775
New comments French (FR) 82.569
New questions German (DE) 77.933
New questions French (FR) 79.935
New topics German (DE) 76.458
New topics French (FR) 79.691
New comments Italian (IT) 79.161

The evaluation script used to obtain these results is:

python evaluate.py \
  --gold data/test.jsonl \
  --pred predictions/mdeberta-v3_pred.jsonl

Note: mdeberta-v3_pred.jsonl is provided in this repository for reproducibility. The evaluation script is available in the original repository at https://github.com/ZurichNLP/xstance .

Quick Start

Installation 

pip install transformers torch

Run inference

Using the pipeline API (Recommended) 

from transformers import pipeline

classifier = pipeline(
    task="text-classification",
    model="MatteoFasulo/mdeberta-v3-xstance"
)

question = "Soll der Bundesrat ein Freihandelsabkommen mit den USA anstreben?"

comment = "Nicht unter einem Präsidenten, welcher die Rechte anderer mit Füssen tritt und Respektlos gegenüber ändern ist."

result = classifier(
    {
        "text": question,
        "text_pair": comment,
    }
)

print(result)

Example output:

[{'label': 'AGAINST', 'score': 0.9823}]

For sequence-pair classification tasks such as stance detection, the text-classification pipeline accepts a dictionary with "text" and "text_pair" keys.

Using AutoModelForSequenceClassification 

import torch
from transformers import (
    AutoTokenizer,
    AutoModelForSequenceClassification,
)

model_name = "MatteoFasulo/mdeberta-v3-xstance"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

question = "Soll der Bundesrat ein Freihandelsabkommen mit den USA anstreben?"

comment = "Nicht unter einem Präsidenten, welcher die Rechte anderer mit Füssen tritt und Respektlos gegenüber ändern ist."

inputs = tokenizer(
    question,
    comment,
    return_tensors="pt",
    truncation=True,
)

with torch.no_grad():
    outputs = model(**inputs)

probabilities = torch.softmax(outputs.logits, dim=-1)

prediction = probabilities.argmax(dim=-1).item()

id2label = model.config.id2label

print("Prediction:", id2label[prediction])
print("Confidence:", probabilities[0, prediction].item())

Example output:

Prediction: AGAINST
Confidence: 0.9823

Input Format

The model expects two text sequences:

  1. Target political question
  2. Candidate comment

Example:

Question:
Should Switzerland increase renewable energy subsidies?

Comment:
Investing in renewable energy will reduce emissions and improve energy independence.

The tokenizer automatically formats these as sentence pairs for mdeberta-v3.

Output Labels

The classifier predicts one of two classes.

Label Description
FAVOR The comment supports the target question.
AGAINST The comment opposes the target question.

The model outputs logits for both classes.

Model Description

This model is a fine-tuned version of microsoft/mdeberta-v3-base trained for multilingual, multi-target stance detection.

Unlike sentiment analysis, stance detection predicts whether a text supports or opposes a specific target question.

Because the underlying encoder is multilingual, the model can transfer knowledge across languages and perform inference on languages that were only partially represented during training.

Intended Uses

The model is suitable for:

  • Political stance detection
  • Cross-lingual stance classification
  • Research on multilingual NLP
  • Opinion mining
  • Benchmarking stance detection methods

Out-of-Scope Uses

This model is not intended for:

  • Fact checking
  • Political affiliation prediction
  • Hate speech detection
  • Toxicity classification
  • General sentiment analysis
  • Automated political decision-making

Training Dataset

Training was performed using the ZurichNLP/x_stance dataset.

Dataset characteristics:

  • 150+ political questions
  • 67,000 candidate comments
  • Swiss political debates
  • Multilingual annotations

Languages:

  • German (majority)
  • French
  • Italian

Each sample consists of:

(Target Question, Candidate Comment)
→ FAVOR / AGAINST

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 2e-05
  • train_batch_size: 16
  • eval_batch_size: 32
  • seed: 42
  • optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 850
  • num_epochs: 3

Training results

Training Loss Epoch Step Validation Loss Accuracy Macro F1
0.4466 1.0 2853 0.4920 0.7886 0.7885
0.3395 2.0 5706 0.4551 0.8077 0.8077
0.2522 3.0 8559 0.5083 0.8153 0.8153

Limitations

Although the model performs well on multilingual political stance detection, several limitations should be considered.

  • Trained primarily on Swiss political debates.
  • Binary labels only (no Neutral class).
  • Performance outside politics has not been evaluated.
  • Implicit or sarcastic opinions remain challenging.
  • Domain shift may reduce performance on social media or informal discussions.

Ethical Considerations

This model predicts stance, not factual correctness.

Predictions should not be interpreted as:

  • political affiliation
  • truthfulness
  • misinformation detection
  • ideological profiling

Human oversight is recommended for any downstream application.

Framework versions

  • Transformers 5.14.1
  • Pytorch 2.8.0a0+5228986c39.nv25.06
  • Datasets 5.0.0
  • Tokenizers 0.22.2

Citation

If you use this model, please cite the original X-Stance dataset.

@inproceedings{vamvas2020xstance,
    author    = "Vamvas, Jannis and Sennrich, Rico",
    title     = "{X-Stance}: A Multilingual Multi-Target Dataset for Stance Detection",
    booktitle = "Proceedings of the 5th Swiss Text Analytics Conference (SwissText) \& 16th Conference on Natural Language Processing (KONVENS)",
    address   = "Zurich, Switzerland",
    year      = "2020",
    month     = "jun",
    url       = "http://ceur-ws.org/Vol-2624/paper9.pdf"
}

License

This model is released under the MIT License.

Please also respect the licenses of:

  • microsoft/mdeberta-v3-base
  • ZurichNLP/x_stance

Acknowledgements

This model builds upon:

  • Microsoft for mdeberta-v3-base
  • Zurich NLP Group for the X-Stance dataset
  • Hugging Face Transformers