d-schafer/illiberal-speech-ei-scorer

Illiberal Speech — Exclusive–Inclusive (EI) Scorer

A fully fine-tuned DeBERTa-v3-base regression model that scores political-speech paragraphs on the exclusive–inclusive (EI) dimension of political discourse, on a 0–10 scale:

• 0 — extremely inclusive: strongly supports inclusion and rights for all, celebrates multiculturalism and diversity.
• 5 — neutral or ambiguous.
• 10 — extremely exclusive, nativist, or illiberal: advocates legal exclusions or eliminating rights based on group identity, incites hatred, or advocates loyalty to a monolithic cultural or national identity.

This is the second stage of the two-stage Illiberal Speech Indicators (ISI) pipeline: paragraphs are first filtered for topical relevance by the companion classifier, illiberal-speech-ei-relevance, and relevant paragraphs are scored by this model.

The models were developed for:

Maerz, Seraphine F., Dean G. Schafer, and Carsten Q. Schneider. Listening to Leaders: Illiberal Speech as a Symptom of Democratic Decline. (2026)

where the ISI pipeline scores 28,227 speeches by 453 political leaders in 123 countries (1989–2024), yielding over half a million semantically coherent paragraphs.

The ISI model family

Model	Dimension	Task
illiberal-speech-ei-relevance	Exclusive–Inclusive	Relevance filter
illiberal-speech-ei-scorer	Exclusive–Inclusive	0–10 scorer (this model)
illiberal-speech-cd-relevance	Concentration–Dispersion	Relevance filter
illiberal-speech-cd-scorer	Concentration–Dispersion	0–10 scorer

Model details

• Architecture: microsoft/deberta-v3-base encoder (full fine-tune, 183.8M trainable parameters), mean pooling over the last hidden state, and a linear regression head. Max sequence length 512.
• Training data: 2,382 paragraphs (1,918 train / 214 validation / 250 held-out) sampled from the speech corpus and scored 0–10 by GPT-4o under a detailed codebook, with human-adjudicated gold-standard subsets used for validation (see the paper's Supplementary Materials).
• Training: MSE loss on standardized labels, lr 2e-5, effective batch size 16, early stopping on validation loss (best epoch 10), seed 1999, PyTorch 2.9.1+cu130 on an NVIDIA RTX PRO 4000 Blackwell (24 GB).

Performance

On the 250-paragraph held-out set (hand-checked by the authors): R² = 0.747, MAE = 0.84, RMSE = 1.11, Pearson r = 0.87 (on the 0–10 scale).

In the paper's cross-validation against OpenAI's gpt-5.2 on human-coded gold-standard data, the DeBERTa scorers match or slightly outperform gpt-5.2 across both ISI dimensions, with lower error and bias (paper, Supplementary Materials II §7.3).

⚠️ The model and scaler are a matched pair

The model predicts in standardized units (labels were z-scored with a StandardScaler fit on the full training data: mean = 4.1531989924433255, scale = 2.2985322063897446). Predictions must be inverse-transformed with the scaler shipped in this repo — illiberal-speech-ei-scorer-scaler.pkl — to land on the 0–10 scale. Using any other scaler will silently shift and stretch every score.

Usage

import joblib
import torch
import torch.nn as nn
from huggingface_hub import hf_hub_download
from transformers import DebertaV2Tokenizer, DebertaV2Model

REPO = "d-schafer/illiberal-speech-ei-scorer"

class DebertaForRegression(nn.Module):
    def __init__(self):
        super().__init__()
        self.deberta = DebertaV2Model.from_pretrained("microsoft/deberta-v3-base")
        self.regressor = nn.Linear(self.deberta.config.hidden_size, 1)

    def forward(self, input_ids, attention_mask):
        h = self.deberta(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        m = attention_mask.unsqueeze(-1).expand(h.size()).float()
        pooled = (h * m).sum(1) / m.sum(1).clamp(min=1e-9)
        return self.regressor(pooled)

model_path  = hf_hub_download(REPO, "illiberal-speech-ei-scorer.pth")
scaler_path = hf_hub_download(REPO, "illiberal-speech-ei-scorer-scaler.pkl")

model = DebertaForRegression()
model.load_state_dict(torch.load(model_path, map_location="cpu", weights_only=True))
model.eval()
scaler = joblib.load(scaler_path)

tokenizer = DebertaV2Tokenizer.from_pretrained("microsoft/deberta-v3-base")

texts = ["Our diversity is our strength, and every citizen deserves equal rights."]
enc = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    pred = model(enc["input_ids"], enc["attention_mask"])
scores = scaler.inverse_transform(pred.numpy())  # 0-10 scale; higher = more exclusive/illiberal
print(scores.ravel().tolist())

Requires torch, transformers, sentencepiece, scikit-learn, and joblib. The scaler was pickled with scikit-learn 1.8.0; older versions may emit a version warning on load.

Files

File	Description
illiberal-speech-ei-scorer.pth	Full fine-tuned state dict (encoder + regression head)
illiberal-speech-ei-scorer-scaler.pkl	StandardScaler for inverse-transforming predictions to 0–10

Intended use & limitations

Designed for paragraph-length segments (~200 words) of political leaders' speeches in English; non-English speeches in the underlying corpus were translated to English before processing. Intended to be run on paragraphs pre-filtered by the EI relevance classifier — scores for off-topic text are not meaningful. Paragraph-level scores are noisy by nature; the paper aggregates them to speaker-year estimates with a latent trajectory model before analysis. Not validated for other text domains, other languages, or sentence-/document-length inputs.

Citation

If you use this model, please cite the paper:

@unpublished{maerz2026listening,
  title  = {Listening to Leaders: Illiberal Speech as a Symptom of Democratic Decline},
  author = {Maerz, Seraphine F. and Schafer, Dean G. and Schneider, Carsten Q.},
  year   = {2026},
  note   = {Working paper}
}

An interactive dashboard for exploring the data is available at https://validation.shinyapps.io/speeches/.