Spaces:
Running
Running
| # Judge calibration: held-out results | |
| Judge: `claude-haiku-5-5`, effort low, prompt 2 (`regradar/judge.py`). | |
| Scored once (`eval/judge/heldout.lock`). Written by `python -m regradar.judge summary` | |
| from `calibration_heldout.json` and `eval/human/judge_labels.jsonl`. | |
| | Cases | n | Label | Agreement | Cohen's kappa [95% CI] | Human yes | Judge yes | | |
| | --- | --- | --- | --- | --- | --- | --- | | |
| | all labelled | 30 | correct | 93.3 [83.3, 100.0] | 0.76 [0.32, 1.00] | 83% | 83% | | |
| | all labelled | 30 | faithful | 83.3 [70.0, 96.7] | not defined (no human 'no' labels) | 100% | 83% | | |
| | natural sample | 28 | correct | 96.4 [89.3, 100.0] | 0.84 [0.00, 1.00] | 86% | 89% | | |
| | natural sample | 28 | faithful | 89.3 [75.0, 100.0] | not defined (no human 'no' labels) | 100% | 89% | | |
| Planted errors marked unfaithful: 15/18 = 83.3% [66.7, 100.0]; by kind condition 1/3, entity 5/6, negation 5/5, number 3/3, wrong_chunk 1/1. Missed: q039~condition, q070~condition, q120~entity. | |