license: mit
language: en
library_name: transformers
pipeline_tag: text-classification
base_model: roberta-large-mnli
tags:
- cross-encoder
- sentence-similarity
- claim-matching
- political-text
- distillation
Claim "same-point" scorer (cross-encoder)
A cross-encoder that scores how fully two claims make the same underlying point, on a graded 1–5 scale (1 = different / no match, 5 = the same claim). Built for measuring document↔summary claim similarity in an LLM political-bias study (UK political-opinion texts).
- Base:
roberta-large-mnli(355M), regression head (num_labels=1) → continuous score in ~[1, 5]. - Trained by distillation from a stronger teacher (Qwen3.6-27B prompted with a graded same-point rubric) over ~125k silver-labelled claim pairs. Not trained on any human labels.
- Symmetric in claim order (trained with both orderings), unlike its directional NLI base.
- Doubles as a filter: it reliably scores non-matches low, so it can replace the NLI pre-filter and rank the survivors.
What it predicts
Given (claim A, claim B), the model's single logit is a same-point strength ≈ 1–5:
| score | meaning |
|---|---|
| 5 | same claim (paraphrase, or a faithful generalisation/specific-instance) |
| 4 | same point, broadened/narrowed |
| 3 | partial overlap |
| 2 | same topic & side but a different point (a reason/mechanism/consequence B adds) |
| 1 | different claim / unrelated / opposite |
Usage
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tok = AutoTokenizer.from_pretrained("daxmavy/claim-samepoint-scorer")
model = AutoModelForSequenceClassification.from_pretrained("daxmavy/claim-samepoint-scorer").eval()
a = "There is often snobbery surrounding private education."
b = "Private schools contribute to social elitism."
enc = tok(a, b, truncation=True, max_length=160, return_tensors="pt")
with torch.no_grad():
score = model(**enc).logits.squeeze().item() # ~1..5; higher = more the same point
print(round(score, 2))
The score is symmetric, so score(a, b) ≈ score(b, a). For a 1–5 label, calibrate with isotonic
regression on a small labelled set (recommended) or simply round-and-clip.
Evaluation (held-out human labels, population-weighted)
Evaluated against human 1–5 labels, post-stratified to the study's main-experiment population (NLI-survivor region):
| metric | value |
|---|---|
| weighted Spearman ρ vs human | 0.711 |
| weighted QWK (isotonic-calibrated) | 0.779 |
| weighted MAE (1–5 scale) | 0.52 |
| input-order symmetry, mean|score(a,b)−score(b,a)| | 0.089 |
| as a filter: match-recall @ 6.72% keep-rate | 100% (vs 4-NLI-mean 80.5%) |
| 5×5 confusion: exact-match / within-±1 | 63.7% / 90% |
It recovers ~90% of the 27B teacher's ranking at ~1/70th the parameters, and outperforms an NLI mean-of-4 and SBERT-cosine on the post-filter ranking task.
Intended use & limitations
- Domain: claims extracted from UK political-opinion / debate texts. Behaviour outside this domain is untested.
- Silver labels come from a single LLM teacher (Qwen3.6-27B + rubric) and inherit its biases; the human evaluation set is small (n≈400, 67 matches), so metrics carry wide CIs.
- Not a factuality or stance classifier — it measures same-point equivalence only.
- Deliberately not an LLM: an LLM similarity judge scores marginally higher but is rejected in the study because political bias in the judge would confound the measurement — a small, bias-free encoder is the point.
Trained for a University of Oxford thesis on LLM political bias (document selection + summarisation).