fix(judges,calibration): five review follow-ups (items 5, 6, 7, 9, 10)

#5 — Gwet's AC2 weighted variant now raises NotImplementedError.
The unweighted AC1 formula is correct and tested against three hand-
computed cases. The weighted variant has multiple inconsistent
literature definitions (Gwet 2008 vs 2014) and no sklearn analogue
to cross-check, so shipping a plausible-looking weighted formula
without a fixture is a methodology hazard. Gate it explicitly until
v1.1 pins both formula choice and fixture. test_weighted_variant_
raises_not_implemented covers both 'linear' and 'quadratic' kwargs.

#6 — citation_faithfulness empty-claim handling. When the answer
starts with a [source:] citation (no prior content), the extractor
returns an empty claim string. Previously the judge built a prompt
with empty content and burned an API call asking the LLM to evaluate
emptiness. Now: vacuously faithful (score=1, no API call), with a
synthetic ScoreResult so per-pair detail still appears in
evidence_quotes. test_leading_citation_empty_claim_vacuously_
faithful asserts 0 provider calls + score=1.

#7 — citation_faithfulness duplicate-source warn. source_to_chunk
uses dict.setdefault, so when the same source name appears multiple
times with distinct chunks (legitimate when multiple retrievals match
the same doc), only the first chunk gets associated. Every claim
citing that source then evaluates against the same chunk — a false-
failure risk. Now warns via 'citation_faithfulness_lossy_source_
lookup' so the operator notices. test_duplicate_source_warns_about_
lossy_lookup pins the warning event name.

#9 — run_calibration.py 'single' strategy parallelizes across
dimensions. Previous design's outer `for dim in row['dimensions']`
loop awaited each dim's gather before starting the next, so a
3-dim row with 30 items did 3 sequential 30-item batches instead
of one 90-item batch. Phase-11 calibration spend is API-rate-
limited, so this leaves wall-clock on the table for no architectural
reason. Now: build one judge per dim, gather all (dim, item) pairs
in a single asyncio.gather call. Permute and jury strategies remain
sequential per-dim because their sidecar JSONLs encode within-call
ordering that downstream analysis depends on.

#10 — Pin sidecar-extension contract in calibration/report.py.
Previous skip was '*_members.jsonl' (extension-specific); if anyone
ever changes jury._DEFAULT_SIDECAR_TEMPLATE from .jsonl to .json,
the sidecar would silently start contaminating the κ table. Now:
the marker is the basename token '_members.', extension-agnostic.
Pinned in a module-level constant _SIDECAR_BASENAME_MARKER. New
test test_members_json_sidecar_excluded_from_table verifies a
hypothetical .json-extension sidecar is still excluded.

All 518 tests pass; ruff clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

agent_bench/evaluation/calibration/metrics.py

@@ -83,14 +83,30 @@ def cohen_kappa(
 def gwets_ac2(
     y1: list,
     y2: list,
-    weights: Literal[None, "linear", "quadratic"] = None,
+    weights: Literal[None] = None,
 ) -> float:
-    """Gwet's AC2 — chance-corrected agreement using mean marginals.
-
-    AC2 = (P_o - P_e_AC2) / (1 - P_e_AC2)
-    where P_e_AC2 = (1/(q-1)) * Σ p_k * (1 - p_k)
-    and p_k is the mean marginal probability for category k.
+    """Gwet's AC1 — chance-corrected agreement using mean marginals.
+
+    AC1 = (P_o - P_e) / (1 - P_e)
+    where P_e = (1/(q-1)) * Σ pi_k * (1 - pi_k)
+    and pi_k is the mean marginal probability for category k.
+
+    Despite the function name, v1 only supports the *unweighted* (AC1)
+    formula. The weighted AC2 variant has multiple inconsistent definitions
+    in the literature (Gwet 2008 vs Gwet 2014); without a sklearn analogue
+    to cross-check against (sklearn ships κ but not AC1/AC2), shipping a
+    weighted formula without a fixture is a methodology hazard. Pass
+    weights=None or omit; passing 'linear' or 'quadratic' raises
+    NotImplementedError. Fix the formula + fixture in v1.1 (out of scope
+    per the design's Out-of-Scope section).
     """
+    if weights is not None:
+        raise NotImplementedError(
+            "Weighted Gwet's AC2 is not implemented in v1. The unweighted "
+            "AC1 formula is correct and tested; the weighted variant has "
+            "literature inconsistency that needs a pinned fixture before "
+            "shipping. Pass weights=None or use cohen_kappa(weights=...)."
+        )
     if len(y1) != len(y2):
         raise ValueError("y1 and y2 length mismatch")
     if not y1:
@@ -105,26 +121,7 @@ def gwets_ac2(
         cm[label_idx[a]][label_idx[b]] += 1
     n = len(y1)
 
-    if weights is None:
-        w = [[1.0 if i == j else 0.0 for j in range(k)] for i in range(k)]
-    elif weights == "linear":
-        if k <= 1:
-            w = [[1.0]]
-        else:
-            w = [
-                [1.0 - abs(i - j) / (k - 1) for j in range(k)] for i in range(k)
-            ]
-    elif weights == "quadratic":
-        if k <= 1:
-            w = [[1.0]]
-        else:
-            w = [
-                [1.0 - ((i - j) / (k - 1)) ** 2 for j in range(k)] for i in range(k)
-            ]
-    else:
-        raise ValueError(f"Invalid weights {weights!r}")
-
-    p_o = sum(w[i][j] * cm[i][j] for i in range(k) for j in range(k)) / n
+    p_o = sum(cm[i][i] for i in range(k)) / n  # diagonal sum (unweighted)
 
     row_marg = [sum(cm[i][j] for j in range(k)) / n for i in range(k)]
     col_marg = [sum(cm[i][j] for i in range(k)) / n for j in range(k)]
@@ -132,15 +129,12 @@ def gwets_ac2(
 
     if k <= 1:
         return 1.0
-    # Gwet's chance term: P_e = (1/(q-1)) * Σ pi_k * (1 - pi_k)
-    # (the standard AC1 formula on mean marginals; weighted variant is
-    # achieved by passing weights to P_o while keeping the unweighted
-    # chance term — sufficient for v1's binary/three-point use).
-    p_e_ac2 = sum(pi[i] * (1 - pi[i]) for i in range(k)) / (k - 1)
+    # AC1 chance term: (1/(q-1)) * Σ pi_k * (1 - pi_k)
+    p_e_ac1 = sum(pi[i] * (1 - pi[i]) for i in range(k)) / (k - 1)
 
-    if p_e_ac2 >= 1.0:
+    if p_e_ac1 >= 1.0:
         return 1.0
-    return (p_o - p_e_ac2) / (1.0 - p_e_ac2)
+    return (p_o - p_e_ac1) / (1.0 - p_e_ac1)
 
 
 def bootstrap_ci(

agent_bench/evaluation/calibration/report.py

@@ -23,6 +23,12 @@ logger = structlog.get_logger()
 
 ABSTAIN_THRESHOLD = 0.20  # strictly greater than fires the flag
 
+# Filename marker for jury / permute sidecar files. Any prediction file whose
+# basename contains this token is per-member detail, not aggregate predictions,
+# and is excluded from the κ table. Pinned here so a future extension change
+# (jsonl → json) is caught at the contract site rather than at report time.
+_SIDECAR_BASENAME_MARKER = "_members."
+
 
 def _classify_abstain(reasoning: str) -> str:
     if reasoning.startswith(ABSTAIN_REASON_PROVIDER_EXHAUSTED):
@@ -67,8 +73,11 @@ def generate_kappa_table(
 
     rows: list[dict] = []
     for pf in pred_files:
-        # Skip sidecar JSONLs (per-member detail, not aggregate predictions)
-        if pf.endswith("_members.jsonl"):
+        # Skip sidecars (per-member detail, not aggregate predictions).
+        # Match the basename marker, not a specific extension, so a future
+        # jsonl → json migration of jury._DEFAULT_SIDECAR_TEMPLATE doesn't
+        # silently start contaminating the κ table.
+        if _SIDECAR_BASENAME_MARKER in Path(pf).name:
             continue
         row_label = (
             Path(pf).stem.replace("calibration_v1_judge_", "")

agent_bench/evaluation/judges/citation_faithfulness.py

@@ -5,6 +5,8 @@ from __future__ import annotations
 import re
 from typing import TYPE_CHECKING
 
+import structlog
+
 from agent_bench.evaluation.judges.base import (
     Judge,
     ScoreResult,
@@ -16,6 +18,8 @@ if TYPE_CHECKING:
     from agent_bench.agents.orchestrator import AgentResponse
     from agent_bench.evaluation.harness import GoldenQuestion
 
+logger = structlog.get_logger()
+
 _CITATION_PATTERN = re.compile(r"\[source:\s*([^\]]+)\]")
 
 
@@ -66,7 +70,29 @@ class CitationFaithfulnessJudge(Judge):
     ) -> ScoreResult:
         pairs = _extract_claims_with_citations(output.answer)
         # Map cited source name to its retrieved chunk text via output.source_chunks
-        # (assumes index alignment with output.sources, matching harness convention)
+        # (assumes index alignment with output.sources, matching harness
+        # convention). If the same source appears multiple times in the
+        # sources list with distinct chunks (legitimate when multiple
+        # retrievals match the same doc), `setdefault` keeps only the first
+        # — every "[source: X]" claim then evaluates against that one chunk,
+        # a false-failure risk. Warn so the operator notices.
+        source_names = [s.source for s in output.sources]
+        if len(set(source_names)) < len(source_names):
+            from collections import Counter
+
+            duplicates = sorted(
+                name for name, n in Counter(source_names).items() if n > 1
+            )
+            logger.warning(
+                "citation_faithfulness_lossy_source_lookup",
+                item_id=item.id,
+                duplicate_source_names=duplicates,
+                detail=(
+                    "source name appears multiple times in output.sources "
+                    "with distinct chunks; only the first chunk will be "
+                    "associated with the name during citation evaluation."
+                ),
+            )
         source_to_chunk: dict[str, str] = {}
         for src_ref, chunk in zip(output.sources, output.source_chunks):
             source_to_chunk.setdefault(src_ref.source, chunk)
@@ -93,6 +119,26 @@ class CitationFaithfulnessJudge(Judge):
         accumulated_latency = 0.0
         any_unfaithful = False
         for claim, cited in pairs:
+            # Empty claim → leading-citation case (e.g., answer starts with
+            # "[source: a.md] ..." with no prior content). There is no claim
+            # to evaluate against the chunk; the well-defined verdict is
+            # vacuously faithful. Skip the API call; record a synthetic
+            # ScoreResult so per-pair detail still appears in evidence_quotes.
+            if not claim:
+                per_pair_results.append(
+                    ScoreResult(
+                        reasoning="empty_claim_vacuously_faithful",
+                        evidence_quotes=[],
+                        score=1,
+                        judge_id=self.judge_id,
+                        rubric_version=self.rubric.source_hash,
+                        prompt_seed=prompt_seed,
+                        system_output_hash=sys_hash,
+                        cost_usd=0.0,
+                        latency_ms=0.0,
+                    )
+                )
+                continue
             chunk = source_to_chunk.get(cited, "")
             prompt = (
                 f"{self.rubric.render_prompt(level_permutation_seed=prompt_seed)}\n\n"

scripts/run_calibration.py

@@ -200,23 +200,46 @@ async def cmd_run_judges(row_config_path: Path, concurrency: int) -> None:
     cfg = load_config()
     sem = asyncio.Semaphore(concurrency)
     all_results: list[dict] = []
-
-    for dim in row["dimensions"]:
-        if row["strategy"] == "single":
-            judge = _make_judge(row["provider"], row["model_id"], dim, cfg)
-
-            async def score_one(rec, _judge=judge, _dim=dim):
-                async with sem:
-                    if rec["category"] == "out_of_scope" and _dim != "relevance":
-                        return None
-                    item, output = _build_item_and_output(rec)
-                    result = await _judge.score(item, output)
-                    return {"dimension": _dim, **result.model_dump()}
-
-            row_results = await asyncio.gather(*[score_one(r) for r in outputs])
-            all_results.extend([r for r in row_results if r is not None])
-
-        elif row["strategy"] == "rubric_permute":
+    strategy = row["strategy"]
+
+    def _skip_oos(rec: dict, dim: str) -> bool:
+        return rec["category"] == "out_of_scope" and dim != "relevance"
+
+    if strategy == "single":
+        # Build one judge per dimension up-front, then gather all
+        # (dim, item) pairs in a single asyncio.gather call. Previous
+        # design serialized across dimensions (each dim awaited fully
+        # before the next started), leaving Phase-11 wall-clock on the
+        # table when the calibration spend is API-rate-limited.
+        judges_by_dim = {
+            dim: _make_judge(row["provider"], row["model_id"], dim, cfg)
+            for dim in row["dimensions"]
+        }
+
+        async def score_one(rec: dict, dim: str, judge):
+            async with sem:
+                if _skip_oos(rec, dim):
+                    return None
+                item, output = _build_item_and_output(rec)
+                result = await judge.score(item, output)
+                return {"dimension": dim, **result.model_dump()}
+
+        coros = [
+            score_one(rec, dim, judge)
+            for dim, judge in judges_by_dim.items()
+            for rec in outputs
+        ]
+        gathered = await asyncio.gather(*coros)
+        all_results.extend([r for r in gathered if r is not None])
+
+    elif strategy == "rubric_permute":
+        # Sequential per-item by design: PermutedJudge writes to the
+        # sidecar JSONL with append mode and within-call ordering matters
+        # for downstream per-permutation analysis (the kappa_table joins
+        # by item_id but the sidecar order encodes the permutation seed
+        # sequence). Across-dim parallelism is left for v1.1 once the
+        # sidecar contract proves stable.
+        for dim in row["dimensions"]:
             judge = _make_judge(row["provider"], row["model_id"], dim, cfg)
             sidecar = REPO / row.get(
                 "sidecar_path", "results/calibration_v1_permute_members.jsonl"
@@ -228,13 +251,19 @@ async def cmd_run_judges(row_config_path: Path, concurrency: int) -> None:
                 sidecar_path=sidecar,
             )
             for rec in outputs:
-                if rec["category"] == "out_of_scope" and dim != "relevance":
+                if _skip_oos(rec, dim):
                     continue
                 item, output = _build_item_and_output(rec)
                 result = await permuted.score(item, output)
                 all_results.append({"dimension": dim, **result.model_dump()})
 
-        elif row["strategy"] == "jury":
+    elif strategy == "jury":
+        # Same sequential rationale as rubric_permute: jury writes a
+        # per-member sidecar and downstream analysis benefits from stable
+        # ordering. The asyncio.gather inside Jury.score does parallelize
+        # member calls within an item; the across-item / across-dim
+        # serialization is the conservative choice.
+        for dim in row["dimensions"]:
             members = [
                 _make_judge(m["provider"], m["model_id"], dim, cfg)
                 for m in row["members"]
@@ -253,13 +282,13 @@ async def cmd_run_judges(row_config_path: Path, concurrency: int) -> None:
                 sidecar_path=sidecar,
             )
             for rec in outputs:
-                if rec["category"] == "out_of_scope" and dim != "relevance":
+                if _skip_oos(rec, dim):
                     continue
                 item, output = _build_item_and_output(rec)
                 result = await j.score(item, output)
                 all_results.append({"dimension": dim, **result.model_dump()})
-        else:
-            raise SystemExit(f"unknown strategy: {row['strategy']}")
+    else:
+        raise SystemExit(f"unknown strategy: {strategy}")
 
     out_path = REPO / row["output_path"]
     out_path.parent.mkdir(parents=True, exist_ok=True)

tests/evaluation/test_calibration_metrics.py

@@ -61,6 +61,18 @@ class TestGwetsAC2HandComputed:
         assert -1.0 <= result <= 1.0
         assert result > 0
 
+    def test_weighted_variant_raises_not_implemented(self):
+        """v1 ships unweighted AC1 only. Weighted AC2 has multiple
+        inconsistent literature definitions; without a fixture to pin
+        the formula choice, shipping silently is a methodology hazard.
+        """
+        y1 = [0, 1, 2, 0, 1, 2]
+        y2 = [0, 1, 2, 1, 1, 2]
+        with pytest.raises(NotImplementedError, match=r"[Ww]eighted Gwet"):
+            gwets_ac2(y1, y2, weights="linear")  # type: ignore[arg-type]
+        with pytest.raises(NotImplementedError, match=r"[Ww]eighted Gwet"):
+            gwets_ac2(y1, y2, weights="quadratic")  # type: ignore[arg-type]
+
 
 class TestBootstrapCI:
     def test_returns_point_lo_hi_tuple(self):

tests/evaluation/test_calibration_report.py

@@ -182,6 +182,46 @@ class TestAbstainRateFlag:
         assert "schema parse" in text
 
 
+class TestSidecarSkipped:
+    def test_members_json_sidecar_excluded_from_table(self, tmp_path):
+        """Regression: per-member sidecar files (matching '_members.*' in
+        basename) must not contaminate the κ table even when their extension
+        matches the predictions glob. The contract is keyed off the basename
+        marker, not the extension.
+        """
+        # Real prediction file
+        preds = [_pred("i1", "groundedness", 1)]
+        labels = [_lbl("i1", "groundedness", 1)]
+        _write_predictions(
+            tmp_path / "results" / "calibration_v1_judge_baseline.json", preds
+        )
+
+        # Hypothetical sidecar file that happens to end in .json (would
+        # normally be .jsonl but the contract should not depend on that).
+        # If the report didn't skip this file, the per-member records inside
+        # would be parsed as aggregate predictions and skew the κ stats.
+        sidecar_pred_shape = [_pred("i1", "groundedness", 0)]  # opposite score
+        _write_predictions(
+            tmp_path / "results" / "calibration_v1_judge_jury_members.json",
+            sidecar_pred_shape,
+        )
+
+        _write_labels(tmp_path / "labels.jsonl", labels)
+        out = tmp_path / "kappa.md"
+        generate_kappa_table(
+            predictions_glob=str(
+                tmp_path / "results" / "calibration_v1_judge_*.json"
+            ),
+            labels_path=str(tmp_path / "labels.jsonl"),
+            output_path=str(out),
+        )
+        text = out.read_text()
+        # Aggregate row from baseline.json should appear; sidecar's "jury_members"
+        # label should NOT appear as a row in the table.
+        assert "baseline" in text
+        assert "jury_members" not in text
+
+
 class TestKappaUndefined:
     def test_renders_dash_with_footnote(self, tmp_path):
         # All same label → degenerate; report renders ' — '

tests/evaluation/test_judges.py

@@ -623,3 +623,103 @@ class TestCitationFaithfulnessJudge:
         assert result.score == 1
         # No provider calls when no citations
         assert provider.complete.await_count == 0
+
+    @pytest.mark.asyncio
+    async def test_leading_citation_empty_claim_vacuously_faithful(self):
+        """Regression: when the answer starts with a citation (no prior
+        sentence), the extractor produces an empty claim. The judge must
+        not burn an API call on empty content; treat as vacuously faithful.
+        """
+        from agent_bench.agents.orchestrator import AgentResponse, SourceReference
+        from agent_bench.core.types import TokenUsage
+        from agent_bench.evaluation.harness import GoldenQuestion
+        from agent_bench.evaluation.judges.base import Rubric
+        from agent_bench.evaluation.judges.citation_faithfulness import (
+            CitationFaithfulnessJudge,
+        )
+
+        rubric = Rubric.from_markdown_file(
+            "agent_bench/evaluation/rubrics/citation_faithfulness.md"
+        )
+        provider = AsyncMock(spec=LLMProvider)
+        judge = CitationFaithfulnessJudge(
+            judge_provider=provider, rubric=rubric, model_id="m"
+        )
+        item = GoldenQuestion(
+            id="i1",
+            question="?",
+            expected_answer_keywords=[],
+            expected_sources=[],
+            category="retrieval",
+            difficulty="easy",
+            requires_calculator=False,
+        )
+        # Answer starts with a citation — no prior content
+        output = AgentResponse(
+            answer="[source: a.md] No prior content.",
+            sources=[SourceReference(source="a.md")],
+            source_chunks=["chunk a"],
+            iterations=1,
+            usage=TokenUsage(
+                input_tokens=0, output_tokens=0, estimated_cost_usd=0
+            ),
+            latency_ms=0,
+        )
+        result = await judge.score(item, output)
+        # Empty-claim pair → vacuously faithful, no API call
+        assert result.score == 1
+        assert provider.complete.await_count == 0
+
+    @pytest.mark.asyncio
+    async def test_duplicate_source_warns_about_lossy_lookup(self):
+        """Regression: source_to_chunk uses dict.setdefault, so when the
+        same source name appears multiple times with distinct chunks, only
+        the first chunk is associated with the name. Warn the operator.
+        """
+        import structlog
+
+        from agent_bench.agents.orchestrator import AgentResponse, SourceReference
+        from agent_bench.core.types import TokenUsage
+        from agent_bench.evaluation.harness import GoldenQuestion
+        from agent_bench.evaluation.judges.base import Rubric
+        from agent_bench.evaluation.judges.citation_faithfulness import (
+            CitationFaithfulnessJudge,
+        )
+
+        rubric = Rubric.from_markdown_file(
+            "agent_bench/evaluation/rubrics/citation_faithfulness.md"
+        )
+        provider = AsyncMock(spec=LLMProvider)
+        provider.complete.return_value = _mk_response(_valid_json(1))
+        judge = CitationFaithfulnessJudge(
+            judge_provider=provider, rubric=rubric, model_id="m"
+        )
+        item = GoldenQuestion(
+            id="i1",
+            question="?",
+            expected_answer_keywords=[],
+            expected_sources=[],
+            category="retrieval",
+            difficulty="easy",
+            requires_calculator=False,
+        )
+        # Same source name twice with distinct chunks → lossy lookup
+        output = AgentResponse(
+            answer="A claim here. [source: a.md]",
+            sources=[
+                SourceReference(source="a.md"),
+                SourceReference(source="a.md"),
+            ],
+            source_chunks=["chunk one", "chunk two"],
+            iterations=1,
+            usage=TokenUsage(
+                input_tokens=0, output_tokens=0, estimated_cost_usd=0
+            ),
+            latency_ms=0,
+        )
+        with structlog.testing.capture_logs() as logs:
+            await judge.score(item, output)
+        assert any(
+            entry.get("event") == "citation_faithfulness_lossy_source_lookup"
+            for entry in logs
+        ), f"no lossy-lookup warning in {logs!r}"