LLM-Judge ↔ Human Alignment

How well does the configured LLM judge agree with human gold labels? Cohen's κ and the confusion matrix are computed over instances that have both a human label and a judge verdict. Inspect disagreements, edit the judge rubric, and re-run to calibrate. Current prompt version: {{ report.prompt_version or "—" }}.

{% if report.prompt_versions %}

Prompt versions

{{ v.prompt_version }} — mean κ {{ '%.3f'|format(v.mean_kappa) if v.mean_kappa is not none else 'n/a' }}, {{ v.n_predictions }} prediction(s) {% if v.prompt_version == report.prompt_version %}(current){% endif %}

{% endif %} {% if not report.per_schema %}

No judge predictions yet. Configure judge_alignment and POST to /admin/api/judge-alignment/run to generate verdicts.

{% endif %} {% for schema, r in report.per_schema.items() %}

{% set band = 'band-none' if r.kappa is none else ('band-low' if r.kappa < 0.2 else ('band-mid' if r.kappa < 0.6 else 'band-high')) %}

Cohen's κ {{ '%.3f'|format(r.kappa) if r.kappa is not none else '—' }}

Agreement {{ (r.agreement_rate * 100)|round(1) }}%

Compared (n) {{ r.n }}

Disagreements {{ r.disagreements|length }}

{% if r.confusion %}

Confusion (rows = human, columns = judge)

{% set cols = [] %} {% for h, js in r.confusion.items() %}{% for j in js.keys() %}{% if j not in cols %}{% set _ = cols.append(j) %}{% endif %}{% endfor %}{% endfor %}

{% for c in cols %}{% endfor %} {% for h, js in r.confusion.items() %} {% for c in cols %} {% endfor %} {% endfor %}

Confusion matrix for {{ schema }}: rows are human gold labels, columns are judge labels.
human ＼ judge	{{ c }}
{{ h }}	{{ js.get(c, 0) }}

{% endif %} {% if r.disagreements %}

Disagreements

{% for d in r.disagreements %} {% endfor %}

Instance	Human	Judge	Conf.	Judge reasoning
{{ d.instance_id }}	{{ d.human_label }}	{{ d.judge_label }}	{{ '%.2f'\|format(d.judge_confidence) if d.judge_confidence is not none else '—' }}	{{ d.reasoning }}

{% endif %}

{% endfor %}

LLM-Judge ↔ Human Alignment

Prompt versions

{{ schema }}