Overlap sample — {{ report.n_overlap_items }} item{{ '' if report.n_overlap_items == 1 else 's' }} at cap.
Each schema is scored with the metrics appropriate to its annotation type.
{% if not report.schemas %}
No overlap-sample items have reached their cap yet. Once the
sample items have been annotated by the configured number of annotators,
IAA will appear here.
{% endif %}
{% for schema_name, schema in report.schemas.items() %}
{# Detect which scales appear in this schema so we can show a single
explanatory note per card instead of repeating it on every row. #}
{% set ns = namespace(has_kappa=false, has_correlation=false, has_raw=false, has_span=false, has_lower=false) %}
{% for metric, value in schema.metrics.items() %}
{% if metric not in ('n_items', 'n_annotators', 'n_aligned_items') %}
{% if metric.endswith('kappa') or metric.startswith('alpha') or metric == 'gamma_mathet' %}{% set ns.has_kappa = true %}{% endif %}
{% if metric in ('pearson_r', 'spearman_rho', 'kendall_tau', 'icc_2_k') %}{% set ns.has_correlation = true %}{% endif %}
{% if metric in ('percent_agreement', 'mean_jaccard') %}{% set ns.has_raw = true %}{% endif %}
{% if metric in ('span_f1_exact', 'span_f1_partial', 'token_level_kappa') %}{% set ns.has_span = true %}{% endif %}
{% if metric in ('mae', 'rmse', 'spearman_footrule') %}{% set ns.has_lower = true %}{% endif %}
{% endif %}
{% endfor %}
{% if ns.has_kappa %}
κ-family scale (κ, α, γ):
<0.2 poor · 0.2–0.4 fair · 0.4–0.6 moderate · 0.6–0.8 substantial · >0.8 strong.
{% endif %}
{% if ns.has_correlation %}
Correlation / ICC: ranges −1 to 1; closer to 1 is stronger agreement.
{% endif %}
{% if ns.has_raw %}
Raw agreement (% agreement, Jaccard): 0 to 1; no chance correction.
{% endif %}
{% if ns.has_span %}
Span F1 / token κ: 0 to 1 (κ can be negative); span_f1_partial is the conventional reporting metric.
{% endif %}
{% if ns.has_lower %}
MAE / RMSE / Spearman footrule: lower is better; 0 means perfect agreement.
{% endif %}
| Metric | Value |
|---|
{% for metric, value in schema.metrics.items() %}
{% if metric not in ('n_items', 'n_annotators', 'n_aligned_items') %}
{% set is_num = value is number and value == value %}
{% set band = '' %}
{% if is_num %}
{% if value >= 0.6 %}{% set band = 'strong' %}
{% elif value < 0.2 %}{% set band = 'weak' %}
{% endif %}
{% endif %}
| {{ metric }} |
{% if is_num %}{{ "%.3f"|format(value) }}{% else %}n/a{% endif %}
{% if band %}{{ band }} — {{ band }} agreement{% endif %}
|
{% endif %}
{% endfor %}
{% endfor %}
Per-item breakdown ({{ report['items']|length }} items)
| Instance | Cap | Annotators |
|---|
{% for iid, item in report['items'].items() %}
| {{ iid }} |
{{ item.cap }} |
{{ item.annotators|join(', ') }} |
{% endfor %}