--- library_name: transformers license: mit base_model: microsoft/mdeberta-v3-base tags: - stance-detection - cross-lingual - multilingual - mdeberta-v3 - text-classification - politics - trackio - trackio:https://huggingface.co/spaces/MatteoFasulo/huggingface-static-26c811 - generated_from_trainer datasets: - ZurichNLP/x_stance language: - de - fr - it - en metrics: - accuracy - f1 model-index: - name: mdeberta-v3-xstance results: - task: type: text-classification dataset: type: x_stance name: X-Stance config: new_comments_defr split: test metrics: - type: f1 name: Macro F1 — New Comments (DE) value: 81.77582896254508 - type: f1 name: Macro F1 — New Comments (FR) value: 82.56971487045625 - task: type: text-classification dataset: type: x_stance name: X-Stance config: new_questions_defr split: test metrics: - type: f1 name: Macro F1 — New Questions (DE) value: 77.93319187443443 - type: f1 name: Macro F1 — New Questions (FR) value: 79.93510514302365 - task: type: text-classification dataset: type: x_stance name: X-Stance config: new_topics_defr split: test metrics: - type: f1 name: Macro F1 — New Topics (DE) value: 76.45883287018658 - type: f1 name: Macro F1 — New Topics (FR) value: 79.69104147158947 - task: type: text-classification dataset: type: x_stance name: X-Stance config: new_comments_it split: test metrics: - type: f1 name: Macro F1 — New Comments (IT) value: 79.1612109408413 --- Visualize in Trackio # mdeberta-v3-xstance A multilingual **stance detection** model fine-tuned from **microsoft/mdeberta-v3-base** on the **ZurichNLP/x_stance** dataset. The model predicts whether a political comment expresses a **FAVOR** or **AGAINST** stance toward a given political question. It supports multilingual inference and demonstrates strong cross-lingual transfer across Swiss national languages. --- # Highlights - 🌍 Multilingual stance detection (🇩🇪 German (75%), 🇫🇷 French (25%), and 🇮🇹 Italian (only a few to test zero-shot cross-lingual transfer)) - ⚡ Built on mdeberta-v3 - 🎯 Binary stance classification (FAVOR / AGAINST) - 🔄 Cross-lingual transfer capabilities --- # Performance ## Validation Split Evaluation on the validation split: | Metric | Score | | -------- | ---------: | | Loss | **0.5643** | | Accuracy | **79.15%** | | Macro F1 | **79.12%** | ## X-Stance Test Set Detailed evaluation on the X-Stance test set as provided in the original repository: | Evaluation setting | Language | Macro F1 | | ------------------ | ------------ | ---------: | | New comments | German (DE) | **81.775** | | New comments | French (FR) | **82.569** | | New questions | German (DE) | **77.933** | | New questions | French (FR) | **79.935** | | New topics | German (DE) | **76.458** | | New topics | French (FR) | **79.691** | | New comments | Italian (IT) | **79.161** | The evaluation script used to obtain these results is: ```bash python evaluate.py \ --gold data/test.jsonl \ --pred predictions/mdeberta-v3_pred.jsonl ``` >Note: `mdeberta-v3_pred.jsonl` is provided in this repository for reproducibility. The evaluation script is available in the original repository at https://github.com/ZurichNLP/xstance . # Quick Start ## Installation ```bash pip install transformers torch ``` ## Run inference ### Using the `pipeline` API (Recommended) ```python from transformers import pipeline classifier = pipeline( task="text-classification", model="MatteoFasulo/mdeberta-v3-xstance" ) question = "Soll der Bundesrat ein Freihandelsabkommen mit den USA anstreben?" comment = "Nicht unter einem Präsidenten, welcher die Rechte anderer mit Füssen tritt und Respektlos gegenüber ändern ist." result = classifier( { "text": question, "text_pair": comment, } ) print(result) ``` Example output: ```python [{'label': 'AGAINST', 'score': 0.9823}] ``` For sequence-pair classification tasks such as stance detection, the `text-classification` pipeline accepts a dictionary with `"text"` and `"text_pair"` keys. --- ### Using `AutoModelForSequenceClassification` ```python import torch from transformers import ( AutoTokenizer, AutoModelForSequenceClassification, ) model_name = "MatteoFasulo/mdeberta-v3-xstance" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForSequenceClassification.from_pretrained(model_name) question = "Soll der Bundesrat ein Freihandelsabkommen mit den USA anstreben?" comment = "Nicht unter einem Präsidenten, welcher die Rechte anderer mit Füssen tritt und Respektlos gegenüber ändern ist." inputs = tokenizer( question, comment, return_tensors="pt", truncation=True, ) with torch.no_grad(): outputs = model(**inputs) probabilities = torch.softmax(outputs.logits, dim=-1) prediction = probabilities.argmax(dim=-1).item() id2label = model.config.id2label print("Prediction:", id2label[prediction]) print("Confidence:", probabilities[0, prediction].item()) ``` Example output: ```text Prediction: AGAINST Confidence: 0.9823 ``` --- # Input Format The model expects **two text sequences**: 1. **Target political question** 2. **Candidate comment** Example: ``` Question: Should Switzerland increase renewable energy subsidies? Comment: Investing in renewable energy will reduce emissions and improve energy independence. ``` The tokenizer automatically formats these as sentence pairs for mdeberta-v3. --- # Output Labels The classifier predicts one of two classes. | Label | Description | |--------|-------------| | FAVOR | The comment supports the target question. | | AGAINST | The comment opposes the target question. | The model outputs logits for both classes. --- # Model Description This model is a fine-tuned version of **microsoft/mdeberta-v3-base** trained for multilingual, multi-target stance detection. Unlike sentiment analysis, stance detection predicts whether a text **supports or opposes a specific target question**. Because the underlying encoder is multilingual, the model can transfer knowledge across languages and perform inference on languages that were only partially represented during training. --- # Intended Uses The model is suitable for: - Political stance detection - Cross-lingual stance classification - Research on multilingual NLP - Opinion mining - Benchmarking stance detection methods --- # Out-of-Scope Uses This model is **not** intended for: - Fact checking - Political affiliation prediction - Hate speech detection - Toxicity classification - General sentiment analysis - Automated political decision-making --- # Training Dataset Training was performed using the **ZurichNLP/x_stance** dataset. Dataset characteristics: - 150+ political questions - 67,000 candidate comments - Swiss political debates - Multilingual annotations Languages: - German (majority) - French - Italian Each sample consists of: ``` (Target Question, Candidate Comment) → FAVOR / AGAINST ``` --- # Training procedure ## Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - train_batch_size: 16 - eval_batch_size: 32 - seed: 42 - optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments - lr_scheduler_type: linear - lr_scheduler_warmup_steps: 850 - num_epochs: 3 ## Training results | Training Loss | Epoch | Step | Validation Loss | Accuracy | Macro F1 | |:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:| | 0.4466 | 1.0 | 2853 | 0.4920 | 0.7886 | 0.7885 | | 0.3395 | 2.0 | 5706 | 0.4551 | 0.8077 | 0.8077 | | 0.2522 | 3.0 | 8559 | 0.5083 | 0.8153 | 0.8153 | # Limitations Although the model performs well on multilingual political stance detection, several limitations should be considered. - Trained primarily on Swiss political debates. - Binary labels only (no Neutral class). - Performance outside politics has not been evaluated. - Implicit or sarcastic opinions remain challenging. - Domain shift may reduce performance on social media or informal discussions. --- # Ethical Considerations This model predicts **stance**, not factual correctness. Predictions should not be interpreted as: - political affiliation - truthfulness - misinformation detection - ideological profiling Human oversight is recommended for any downstream application. --- # Framework versions - Transformers 5.14.1 - Pytorch 2.8.0a0+5228986c39.nv25.06 - Datasets 5.0.0 - Tokenizers 0.22.2 # Citation If you use this model, please cite the original X-Stance dataset. ```bibtex @inproceedings{vamvas2020xstance, author = "Vamvas, Jannis and Sennrich, Rico", title = "{X-Stance}: A Multilingual Multi-Target Dataset for Stance Detection", booktitle = "Proceedings of the 5th Swiss Text Analytics Conference (SwissText) \& 16th Conference on Natural Language Processing (KONVENS)", address = "Zurich, Switzerland", year = "2020", month = "jun", url = "http://ceur-ws.org/Vol-2624/paper9.pdf" } ``` --- # License This model is released under the **MIT License**. Please also respect the licenses of: - microsoft/mdeberta-v3-base - ZurichNLP/x_stance --- # Acknowledgements This model builds upon: - Microsoft for mdeberta-v3-base - Zurich NLP Group for the X-Stance dataset - Hugging Face Transformers