README.md

---
license: apache-2.0
library_name: onnx
pipeline_tag: text-classification
tags:
  - groundedness
  - hallucination-detection
  - cross-encoder
  - onnx
  - multilingual
language:
  - bg
  - hr
  - cs
  - da
  - nl
  - en
  - et
  - fi
  - fr
  - de
  - el
  - hu
  - ga
  - it
  - lv
  - lt
  - mt
  - pl
  - pt
  - ro
  - sk
  - sl
  - es
  - sv
  - tr
  - az
base_model: FacebookAI/xlm-roberta-base
---

# groundedness

Is a sentence of model output supported by the source passages it was supposed to rest on?
A cross-encoder over `(source, candidate)` pairs, two classes, 26 languages, exported to
ONNX fp16 and run on CPU. Built for [`flowx-border`](https://github.com/flowx-ai/border),
where it is the T3 `groundedness` detector.

**Read the two evaluations below as different questions, not as a range.** The corpus
figures are high and the hand-written ones are not, and the gap is the honest content of
this card.

## Two classes, not three

`grounded` and `not_grounded`. Earlier candidates for this detector predicted `supported`,
`unsupported` and `contradicted`, and the library collapses the last two into one action
anyway, so the three-way head optimised a boundary no caller ever sees. Across six such
candidates `unstated` and the conflict registers were anti-correlated at -0.98: they traded
points along it.

The trade that produced: measured at the trained length, the binary objective **cost 0.048
of accuracy** on hand-written probes against the best three-way candidate, 0.7143 to 0.6667.
It is published anyway, for the reason in the next section.

## Use it at a threshold of 0.78, not at argmax

    grounded if p(grounded) >= 0.78

Swept on the validation split and only then applied to the hand-written probes. Every bar
from 0.78 up clears the probe described below while not-grounded recall *rises*, 0.9641 to
0.9699, so the bar costs nothing measurable on held-out data.

**The weakness travels with the number.** That validation curve is nearly flat across the
whole range, so validation does not pick 0.78. One probe does, and a threshold chosen by the
case it must catch is weaker evidence than one chosen by a distribution.

## What it is for: the case the other candidates got wrong

Against a source stating that withdrawals incur a fee for the first twelve months and are
free afterwards, the candidate *"Withdrawals are free from the day the account opens"* is a
temporal contradiction. Six earlier candidates called it **grounded at 0.9906 to 0.9995**.
This one reads 0.7681, so any bar from 0.78 reports it.

More usefully, it is the only one of seven that reads the source at all. Same candidate,
three sources:

| source | best three-way candidate | this model |
|---|---|---|
| the real source, which contradicts it | grounded 0.9991 | **not grounded 0.7681** |
| an unrelated passage in another language | grounded 0.9994 | **not grounded 0.0070** |
| a source that does state it outright | not grounded 0.0007 | **grounded 0.8365** |

The three-way candidate is inverted on this sentence and gives an unrelated Romanian
passage the same answer as the real source. Three different answers for one candidate is
what makes this a judgement about the source rather than about the sentence.

## Corpus evaluation, 2,062 held-out rows

Threshold 0.78, the shipped bar, at the trained length of 512 tokens.

| | |
|---|---|
| overall accuracy | 0.9471 |
| not-grounded recall | 0.9612 |
| pair accuracy | 0.8991 |
| per-language range | 0.887 (`pl`) to 1.000, over 26 languages |
| weakest three | `pl` 0.887, `en` 0.897, `az` 0.912 |

Pair accuracy is the number to prefer: the corpus is source-side pairs, one candidate
against two sources with opposite labels and the candidate byte-identical across the pair, so
a model that ignored the source scores near zero on it by construction.

**These figures describe a synthetic corpus and its own held-out split.** The generator wrote
both, so they measure generalisation within one generator's style. That is why the next
section exists and why it disagrees.

## Hand-written evaluation, 42 probes

Written by a person, not by any generator, across seven ways a summary goes wrong.

| configuration | accuracy |
|---|---|
| this model alone, at 0.78 | 0.6905 |
| this model plus the library's deterministic rule layer | **0.7381** |

**Roughly one call in four is wrong on this set, against one in twenty on the corpus split.**
Both numbers are real. The probe set is adversarial by construction, seven hard shapes in
equal proportion, which no real traffic is; the corpus split is generator-shaped, which no
real traffic is either. The truth for any given deployment is between them and closer to
whichever resembles that traffic.

The rule layer is `detectors/claim_conflict.py` in the library and needs no weights. Where a
candidate's content words all appear in its source except a numeral or an absolute
quantifier, it reports a conflict deterministically. On the 42 probes it fires 9 times and is
right 9 times.

## Known weakness: it errs toward caution

The failure mode is false `not_grounded` on claims that are genuinely supported. Eight of
thirteen probe failures are that direction, and the clearest case is a claim *weaker* than
its source: against a source saying withdrawals incur a fee for the first twelve months,
*"There is a handling fee for early withdrawals"* reads `not_grounded` at 0.8625.

For a guardrail that is the safer direction, since a false "not grounded" costs a reviewer's
attention and a false "grounded" puts an unsupported claim in front of a customer. It is
still a cost, and it is why the detector is **disabled in both policies that ship with the
library**. A caller who wants it enables it in one line and should measure it on their own
traffic first.

## What it needs

- The full 512-token window. Scores saturate by 256 and degrade below that; at 96 tokens
  this model reads the probe above as grounded, which is the wrong answer arrived at by
  truncation rather than by judgement.
- Sources. With none supplied the library records that the check could not run rather than
  reporting a clean scan.
- Pair order `(source, candidate)`. Reversed, the head answers a different question
  confidently.

## Not evaluated

Per-language figures rest on roughly 80 rows each, so one item moves a language by more than
a point. The corpus contains no case where a qualifier is dropped from a conditional
statement expressed in words rather than digits, and no unit conversions such as `24 months`
against `two years`; both are known gaps rather than measured strengths. Nothing here is
evaluated against human-annotated groundedness data, because none exists for these 26
languages.

## Licence

Apache-2.0. Trained on synthetic data generated for this purpose.