| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>JevBench</title> |
| </head> |
| <body> |
| <p><b>JevBench: metamorphic coherence testing for typed probabilistic decision models.</b></p> |
| <p>Typed decision models answer questions about a text with probability distributions over declared answers: yes or |
| no, one of several options, or a level on an ordered scale. JevBench tests whether those probabilities stay coherent |
| when a question is reworded, negated, logically combined with others, offered a different set of options, or asked |
| together with other questions: <b>50 metamorphic relations in five dimensions</b>, no gold labels needed, every test |
| frozen in advance. Across 21 openly released models, batch independence mostly holds, while additivity fails |
| broadly, above all under negation.</p> |
| <ul> |
| <li>📄 <b>Technical report</b>: <a href="https://www.alphaxiv.org/abs/2610.jevbench-metamorphic-coherence-testing" target="_blank">alphaXiv</a></li> |
| <li>🗂️ <b>Dataset</b>: <a href="https://huggingface.co/datasets/JevBench/jevbench" target="_blank">JevBench/jevbench</a>, 240 cases, 2,240 questions, frozen test suites and every evaluation report</li> |
| <li>🏆 <b>Leaderboard</b>: <a href="https://jevbench.github.io" target="_blank">jevbench.github.io</a></li> |
| <li>💻 <b>Code</b>: <a href="https://github.com/JevBench/jevbench" target="_blank">github.com/JevBench/jevbench</a>, <code>pip install jevbench</code></li> |
| </ul> |
| <p>Maintained by <a href="https://mrchenfeng.github.io/" target="_blank">Chen Feng</a>, |
| <a href="https://qub-ml.github.io/" target="_blank">ML Lab</a>, Queen's University Belfast.</p> |
| </body> |
| </html> |
|
|
