README.md

---
license: mit
language: en
library_name: transformers
pipeline_tag: text-classification
base_model: roberta-large-mnli
tags:
  - cross-encoder
  - sentence-similarity
  - claim-matching
  - political-text
  - distillation
---

# Claim "same-point" scorer (cross-encoder)

A cross-encoder that scores how fully **two claims make the same underlying point**, on a graded
**1–5** scale (1 = different / no match, 5 = the same claim). Built for measuring document↔summary
claim similarity in an LLM political-bias study (UK political-opinion texts).

- **Base:** `roberta-large-mnli` (355M), regression head (`num_labels=1`) → continuous score in ~[1, 5].
- **Trained by distillation** from a stronger teacher (**Qwen3.6-27B** prompted with a graded
  same-point rubric) over ~125k silver-labelled claim pairs. Not trained on any human labels.
- **Symmetric** in claim order (trained with both orderings), unlike its directional NLI base.
- Doubles as a **filter**: it reliably scores non-matches low, so it can replace the NLI pre-filter
  *and* rank the survivors.

## What it predicts

Given `(claim A, claim B)`, the model's single logit is a same-point strength ≈ 1–5:

| score | meaning |
|---|---|
| 5 | same claim (paraphrase, or a faithful generalisation/specific-instance) |
| 4 | same point, broadened/narrowed |
| 3 | partial overlap |
| 2 | same topic & side but a different point (a reason/mechanism/consequence B adds) |
| 1 | different claim / unrelated / opposite |

## Usage

```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("daxmavy/claim-samepoint-scorer")
model = AutoModelForSequenceClassification.from_pretrained("daxmavy/claim-samepoint-scorer").eval()

a = "There is often snobbery surrounding private education."
b = "Private schools contribute to social elitism."
enc = tok(a, b, truncation=True, max_length=160, return_tensors="pt")
with torch.no_grad():
    score = model(**enc).logits.squeeze().item()   # ~1..5; higher = more the same point
print(round(score, 2))
```

The score is symmetric, so `score(a, b) ≈ score(b, a)`. For a 1–5 label, calibrate with isotonic
regression on a small labelled set (recommended) or simply round-and-clip.

## Evaluation (held-out human labels, population-weighted)

Evaluated against human 1–5 labels, post-stratified to the study's main-experiment population
(NLI-survivor region):

| metric | value |
|---|---|
| weighted Spearman ρ vs human | **0.711** |
| weighted QWK (isotonic-calibrated) | 0.779 |
| weighted MAE (1–5 scale) | 0.52 |
| input-order symmetry, mean\|score(a,b)−score(b,a)\| | **0.089** |
| as a filter: match-recall @ 6.72% keep-rate | **100%** (vs 4-NLI-mean 80.5%) |
| 5×5 confusion: exact-match / within-±1 | 63.7% / 90% |

It recovers ~90% of the 27B teacher's ranking at ~1/70th the parameters, and outperforms an NLI
mean-of-4 and SBERT-cosine on the post-filter ranking task.

## Intended use & limitations

- **Domain:** claims extracted from UK political-opinion / debate texts. Behaviour outside this
  domain is untested.
- Silver labels come from a **single LLM teacher** (Qwen3.6-27B + rubric) and inherit its biases;
  the human evaluation set is small (n≈400, 67 matches), so metrics carry wide CIs.
- Not a factuality or stance classifier — it measures *same-point* equivalence only.
- Deliberately **not** an LLM: an LLM similarity judge scores marginally higher but is rejected in
  the study because political bias in the judge would confound the measurement — a small,
  bias-free encoder is the point.

Trained for a University of Oxford thesis on LLM political bias (document selection + summarisation).