d-schafer/illiberal-speech-cd-relevance

Illiberal Speech — Concentration–Dispersion (CD) Relevance Filter

A DeBERTa-v3-base + LoRA binary classifier that flags whether a political-speech paragraph is topically relevant to the concentration–dispersion (CD) dimension of political discourse: checks on executive power, accountability, the judiciary, parliament, free media, civil society, and democratic norms.

This is the first stage of the two-stage Illiberal Speech Indicators (ISI) pipeline. Paragraphs flagged as relevant are then scored 0–10 by the companion regression model, illiberal-speech-cd-scorer.

The models were developed for:

Maerz, Seraphine F., Dean G. Schafer, and Carsten Q. Schneider. Listening to Leaders: Illiberal Speech as a Symptom of Democratic Decline. (2026)

where the ISI pipeline scores 28,227 speeches by 453 political leaders in 123 countries (1989–2024), yielding over half a million semantically coherent paragraphs.

The ISI model family

Model	Dimension	Task
illiberal-speech-ei-relevance	Exclusive–Inclusive	Relevance filter
illiberal-speech-ei-scorer	Exclusive–Inclusive	0–10 scorer
illiberal-speech-cd-relevance	Concentration–Dispersion	Relevance filter (this model)
illiberal-speech-cd-scorer	Concentration–Dispersion	0–10 scorer

Model details

• Architecture: microsoft/deberta-v3-base encoder with LoRA adapters (r=16, α=32, dropout=0.1, target modules: query_proj, key_proj, value_proj), mean pooling over the last hidden state, and a linear 2-class classification head. Max sequence length 512.
• Training data: 7,294 training paragraphs (649 validation / 163 test) drawn from a stratified random sample of the speech corpus, supplemented with targeted gap-fill batches for rare-class examples. Binary relevance labels were derived from GPT-4.1 relevance ratings (0–100 scale; ≥40 → relevant, ≤10 → not relevant, ambiguous middle excluded).
• Training: early stopping on validation F1 (best epoch 3, val F1 = 0.956), seed 1969, PyTorch 2.9.1+cu130 on an NVIDIA RTX PRO 4000 Blackwell (24 GB).

Performance

On the held-out test split (n = 163): F1 (positive class) = 0.931, F1 macro = 0.950, AUC = 0.989.

On an independent hand-coded holdout (n = 300 paragraphs labelled by the authors), at the recommended decision threshold of 0.80: recall = 0.874, precision = 0.781, F1 (positive) = 0.825, AUC = 0.832. The threshold was chosen by grid search on this holdout. Note that the CD filter's operating threshold (0.80) differs from the EI filter's (0.20) — each was tuned on its own hand-coded holdout; use the value for the model you are running.

Usage

The .pth file contains the full model state dict (LoRA weights and the classification head). The lora_adapters/ folder holds the PEFT adapters for the encoder only — it does not include the classification head, so use the .pth for working predictions:

import torch
import torch.nn as nn
from huggingface_hub import hf_hub_download
from transformers import DebertaV2Tokenizer, DebertaV2Model
from peft import LoraConfig, get_peft_model

REPO = "d-schafer/illiberal-speech-cd-relevance"
THRESHOLD = 0.80  # recommended decision threshold

class DebertaForClassification(nn.Module):
    def __init__(self):
        super().__init__()
        self.deberta = DebertaV2Model.from_pretrained("microsoft/deberta-v3-base")
        hidden_size = self.deberta.config.hidden_size
        lora_config = LoraConfig(
            r=16, lora_alpha=32, lora_dropout=0.1,
            target_modules=["query_proj", "key_proj", "value_proj"],
            bias="none", task_type="FEATURE_EXTRACTION",
        )
        self.deberta = get_peft_model(self.deberta, lora_config)
        self.classifier = nn.Linear(hidden_size, 2)
        self.dropout = nn.Dropout(0.1)

    def forward(self, input_ids, attention_mask):
        h = self.deberta(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        m = attention_mask.unsqueeze(-1).expand(h.size()).float()
        pooled = (h * m).sum(1) / m.sum(1).clamp(min=1e-9)
        return self.classifier(self.dropout(pooled))

model_path = hf_hub_download(REPO, "illiberal-speech-cd-relevance.pth")
model = DebertaForClassification()
model.load_state_dict(torch.load(model_path, map_location="cpu", weights_only=True))
model.eval()

tokenizer = DebertaV2Tokenizer.from_pretrained(REPO, subfolder="tokenizer")

texts = ["The courts should not be allowed to stand in the way of the people's will."]
enc = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    probs = torch.softmax(model(enc["input_ids"], enc["attention_mask"]), dim=1)[:, 1]
relevant = probs >= THRESHOLD
print(probs.tolist(), relevant.tolist())

Requires torch, transformers, peft, and sentencepiece.

Files

File	Description
illiberal-speech-cd-relevance.pth	Full state dict (encoder + LoRA + classification head) — use this
lora_adapters/	PEFT adapter weights for the encoder (no classification head)
tokenizer/	DeBERTa-v3 tokenizer files (offline-reproducible copy)

Intended use & limitations

Designed for paragraph-length segments (~200 words) of political leaders' speeches in English; non-English speeches in the underlying corpus were translated to English before processing. Performance is not validated for other text domains (social media, news, parliamentary debate), other languages, or sentence-/document-length inputs.

Citation

If you use this model, please cite the paper:

@unpublished{maerz2026listening,
  title  = {Listening to Leaders: Illiberal Speech as a Symptom of Democratic Decline},
  author = {Maerz, Seraphine F. and Schafer, Dean G. and Schneider, Carsten Q.},
  year   = {2026},
  note   = {Working paper}
}

An interactive dashboard for exploring the data is available at https://validation.shinyapps.io/speeches/.