---
title: README
emoji: 📊
colorFrom: indigo
colorTo: red
sdk: static
pinned: false
---

# JevBench

**Metamorphic coherence testing for typed probabilistic decision models.**

Typed decision models answer questions about a text with probability distributions over declared answers: yes or no,
one of several options, or a level on an ordered scale. JevBench tests whether those probabilities stay coherent when
a question is reworded, negated, logically combined with others, offered a different set of options, or asked
together with other questions: 50 metamorphic relations in five dimensions, no gold labels needed, every test frozen
in advance.

| | |
|---|---|
| 📄 Technical report | [alphaXiv](https://www.alphaxiv.org/abs/2610.jevbench-metamorphic-coherence-testing) |
| 🗂️ Dataset | [JevBench/jevbench](https://huggingface.co/datasets/JevBench/jevbench): 240 cases, 2,240 questions, frozen test suites and every evaluation report |
| 🏆 Leaderboard | [jevbench.github.io](https://jevbench.github.io): 21 openly released models |
| 💻 Code | [github.com/JevBench/jevbench](https://github.com/JevBench/jevbench) · `pip install jevbench` |

Maintained by [Chen Feng](https://mrchenfeng.github.io/), [ML Lab](https://qub-ml.github.io/), Queen's University Belfast.