daxmavy/claim-samepoint-scorer

Claim "same-point" scorer (cross-encoder)

A cross-encoder that scores how fully two claims make the same underlying point, on a graded 1–5 scale (1 = different / no match, 5 = the same claim). Built for measuring document↔summary claim similarity in an LLM political-bias study (UK political-opinion texts).

• Base: roberta-large-mnli (355M), regression head (num_labels=1) → continuous score in ~[1, 5].
• Trained by distillation from a stronger teacher (Qwen3.6-27B prompted with a graded same-point rubric) over ~125k silver-labelled claim pairs. Not trained on any human labels.
• Symmetric in claim order (trained with both orderings), unlike its directional NLI base.
• Doubles as a filter: it reliably scores non-matches low, so it can replace the NLI pre-filter and rank the survivors.

What it predicts

Given (claim A, claim B), the model's single logit is a same-point strength ≈ 1–5:

score	meaning
5	same claim (paraphrase, or a faithful generalisation/specific-instance)
4	same point, broadened/narrowed
3	partial overlap
2	same topic & side but a different point (a reason/mechanism/consequence B adds)
1	different claim / unrelated / opposite

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("daxmavy/claim-samepoint-scorer")
model = AutoModelForSequenceClassification.from_pretrained("daxmavy/claim-samepoint-scorer").eval()

a = "There is often snobbery surrounding private education."
b = "Private schools contribute to social elitism."
enc = tok(a, b, truncation=True, max_length=160, return_tensors="pt")
with torch.no_grad():
    score = model(**enc).logits.squeeze().item()   # ~1..5; higher = more the same point
print(round(score, 2))

The score is symmetric, so score(a, b) ≈ score(b, a). For a 1–5 label, calibrate with isotonic regression on a small labelled set (recommended) or simply round-and-clip.

Evaluation (held-out human labels, population-weighted)

Evaluated against human 1–5 labels, post-stratified to the study's main-experiment population (NLI-survivor region):

metric	value
weighted Spearman ρ vs human	0.711
weighted QWK (isotonic-calibrated)	0.779
weighted MAE (1–5 scale)	0.52
input-order symmetry, mean|score(a,b)−score(b,a)|	0.089
as a filter: match-recall @ 6.72% keep-rate	100% (vs 4-NLI-mean 80.5%)
5×5 confusion: exact-match / within-±1	63.7% / 90%

It recovers ~90% of the 27B teacher's ranking at ~1/70th the parameters, and outperforms an NLI mean-of-4 and SBERT-cosine on the post-filter ranking task.

Intended use & limitations

• Domain: claims extracted from UK political-opinion / debate texts. Behaviour outside this domain is untested.
• Silver labels come from a single LLM teacher (Qwen3.6-27B + rubric) and inherit its biases; the human evaluation set is small (n≈400, 67 matches), so metrics carry wide CIs.
• Not a factuality or stance classifier — it measures same-point equivalence only.
• Deliberately not an LLM: an LLM similarity judge scores marginally higher but is rejected in the study because political bias in the judge would confound the measurement — a small, bias-free encoder is the point.

Trained for a University of Oxford thesis on LLM political bias (document selection + summarisation).