refactor: build grading on OpenEnv Rubric system

Each of the seven scoring dimensions is now an openenv.core.rubrics.Rubric
subclass in evaluation/rubrics.py, composed via WeightedSum inside a per-case
CaseRubric and an episode-level ChargebackOpsEpisodeRubric that is wired into
ChargebackOpsEnvironment.rubric. The legacy evaluation/grading.py functions
remain as thin adapters so every call site (env step, episode store, API,
audit tooling, tests) keeps its current contract and outputs are bit-for-bit
identical to the previous implementation.

Wins: env.rubric.named_rubrics() exposes the full grader tree, every
dimension gets register_forward_hook / last_score / state_dict for free,
and swapping a dimension (e.g. LLMJudge for the note rubric) is a one-line
change. Docs (README, AGENT.md, OPENENV.md) and the grader test are updated
to reflect the rubric integration.

AGENT.md

@@ -380,7 +380,7 @@ When the agent submits a contest, it generates a representment note. The grader
 
 ## The Grading System
 
-After all cases are resolved (or the step budget is exhausted), the deterministic grader in `grading.py` scores each case across 7 dimensions:
+After all cases are resolved (or the step budget is exhausted), the grader scores each case across 7 dimensions. Each dimension is an OpenEnv `Rubric` subclass defined in `evaluation/rubrics.py`; they compose into a per-case `WeightedSum` and an episode-level `ChargebackOpsEpisodeRubric` that is wired into `env.rubric`. `evaluation/grading.py` keeps the legacy `score_case` / `grade_episode` API as a thin adapter over the rubric tree.
 
 ### Strategy Correctness (25%)
 
@@ -566,7 +566,8 @@ Nightmare tasks push the step budget to its limit: 5-6 cases with ~2.4 steps per
 |---|---|---|
 | `runners/baseline_runner.py` | The agent: decision pipeline, candidate generation, LLM integration, representment notes | ~1100 |
 | `server/chargeback_ops_environment.py` | The environment: step/reset/state, action execution, reward computation | ~500 |
-| `evaluation/grading.py` | Deterministic 7-dimension scoring, representment note grading | ~200 |
+| `evaluation/rubrics.py` | OpenEnv `Rubric` subclasses for all 7 scoring dimensions, composed via `WeightedSum` | ~300 |
+| `evaluation/grading.py` | Legacy `score_case` / `grade_episode` adapter that delegates to the rubric tree | ~120 |
 | `scenarios/simulation.py` | Task definitions, case progress tracking, evidence metadata | ~600 |
 | `core/models.py` | Pydantic models for actions, observations, state, grading | ~600 |
 | `runners/inference.py` | OpenEnv-compatible inference entry point with provider fallback | ~200 |

OPENENV.md

@@ -47,4 +47,4 @@ The OpenEnv hackathon asks participants to build an environment, not an agent. T
 
 Round 1 evaluates environments on real-world utility (30%), task and grader quality (25%), environment design (20%), code quality (15%), and creativity (10%). Round 2 runs a standard agent against all qualifying environments to measure how well each environment discriminates between good and bad agent behaviour.
 
-ChargebackOps is built to perform well on both rounds: the 7-dimension deterministic grader produces a clear difficulty curve (easy 0.96 → nightmare 0.47), and the typed action space with dense reward shaping gives any standard agent enough signal to learn the environment within a single episode.
+ChargebackOps is built to perform well on both rounds: the 7-dimension grader is implemented on top of OpenEnv's `Rubric` system (each dimension is a `Rubric` subclass composed via `WeightedSum` and wired into `env.rubric`, so the whole grader is introspectable, hookable, and checkpointable), it produces a clear difficulty curve (easy 0.96 → nightmare 0.47), and the typed action space with dense reward shaping gives any standard agent enough signal to learn the environment within a single episode.

README.md

@@ -30,7 +30,7 @@ graph TB
     subgraph Core["Environment Core"]
         ENV["ChargebackOpsEnvironment\nstep() / reset() / state()"]
         SIM["Simulation Engine\nscenarios/simulation.py"]
-        GRD["Deterministic Grader\nevaluation/grading.py"]
+        GRD["OpenEnv Rubric Grader\nevaluation/rubrics.py"]
     end
 
     subgraph Tasks["Task Sources"]
@@ -52,6 +52,8 @@ graph TB
 
 ## Grading
 
+Each scoring dimension is a standalone `openenv.core.rubrics.Rubric` subclass. They compose into a per-case `WeightedSum` and an episode-level `ChargebackOpsEpisodeRubric` that the environment wires into `self.rubric`, so the whole grader is introspectable via `env.rubric.named_rubrics()`, hookable via `register_forward_hook`, and checkpointable via `state_dict()`. Swapping `NoteQualityRubric` for an `LLMJudge`, or wrapping any dimension in a `Gate`, is a one-line change.
+
 7-dimension deterministic grader, weighted per case by financial impact:
 
 ```mermaid
@@ -112,6 +114,19 @@ openenv validate .
 python -m runners.inference
 ```
 
+Inspect the rubric tree on a live environment:
+
+```python
+from server.chargeback_ops_environment import ChargebackOpsEnvironment
+env = ChargebackOpsEnvironment()
+for name, r in env.rubric.named_rubrics():
+    print(f"{name}: {type(r).__name__}")
+# case_rubric: CaseRubric
+# case_rubric.aggregator: WeightedSum
+# case_rubric.aggregator.rubric_0: StrategyCorrectnessRubric
+# ... (all 7 dimensions)
+```
+
 ```bash
 # Docker
 docker build -t chargebackops .
@@ -157,7 +172,7 @@ Entry point: [`inference.py`](inference.py). Fallback chain: primary provider ->
 ├── inference.py              # Submission entry point
 ├── openenv.yaml              # OpenEnv spec
 ├── core/                     # Models, client, episode store
-├── evaluation/               # 7-dimension grader, audit suite
+├── evaluation/               # OpenEnv Rubric subclasses + legacy grader adapters
 ├── runners/                  # Baseline agent, inference logic
 ├── scenarios/                # Tasks, generator, ISO adapter
 ├── server/                   # FastAPI app, environment, Gradio demo

evaluation/__init__.py

@@ -1 +1,19 @@
 """Grading and audit modules for ChargebackOps."""
+
+from .grading import grade_episode, grade_representment_note, score_case
+from .rubrics import (
+    CaseRubric,
+    ChargebackOpsEpisodeRubric,
+    EpisodeGradingContext,
+    GradingContext,
+)
+
+__all__ = [
+    "grade_episode",
+    "grade_representment_note",
+    "score_case",
+    "CaseRubric",
+    "ChargebackOpsEpisodeRubric",
+    "EpisodeGradingContext",
+    "GradingContext",
+]

evaluation/grading.py

@@ -1,192 +1,50 @@
-"""Deterministic grading logic for ChargebackOps."""
+"""Deterministic grading adapters that delegate to OpenEnv Rubric subclasses.
+
+The real scoring lives in :mod:`evaluation.rubrics`. This module keeps the
+legacy call sites (``score_case`` / ``grade_episode`` / ``grade_representment_note``)
+stable so the environment, tests, and audit tooling do not need to change.
+"""
 
 from __future__ import annotations
 
 try:
     from ..core.models import CaseScoreBreakdown, GraderReport
     from ..scenarios.simulation import CaseProgress, InternalCase, TaskScenario
+    from .rubrics import (
+        CaseRubric,
+        ChargebackOpsEpisodeRubric,
+        EpisodeGradingContext,
+        GradingContext,
+        grade_representment_note,
+    )
 except ImportError:  # pragma: no cover
     from core.models import CaseScoreBreakdown, GraderReport
     from scenarios.simulation import CaseProgress, InternalCase, TaskScenario
+    from evaluation.rubrics import (
+        CaseRubric,
+        ChargebackOpsEpisodeRubric,
+        EpisodeGradingContext,
+        GradingContext,
+        grade_representment_note,
+    )
 
 
-def _ratio(numerator: int, denominator: int) -> float:
-    if denominator <= 0:
-        return 1.0
-    return max(0.0, min(1.0, numerator / denominator))
-
-
-def grade_representment_note(
-    note: str | None,
-    case: "InternalCase",
-    attached_ids: set[str],
-) -> float:
-    """Score a representment note from 0.0 to 1.0.
-
-    Evaluates whether the note:
-    - References required claims from the policy requirements
-    - Avoids mentioning harmful evidence
-    - Has sufficient substance (length and specificity)
-    """
-    if not note or not note.strip():
-        return 0.0
-
-    text = note.lower()
-    score = 0.0
-
-    # Substance: minimum length for a coherent note
-    word_count = len(text.split())
-    if word_count >= 5:
-        score += 0.2
-    elif word_count >= 2:
-        score += 0.1
-
-    # Required claims coverage: does the note mention policy requirements?
-    if case.policy_requirements:
-        claims_hit = 0
-        for req in case.policy_requirements:
-            req_keywords = req.lower().split()
-            if any(kw in text for kw in req_keywords if len(kw) > 3):
-                claims_hit += 1
-        score += 0.5 * _ratio(claims_hit, len(case.policy_requirements))
-    else:
-        score += 0.3  # No requirements to check
-
-    # Evidence coherence: does the note reference attached evidence?
-    evidence_refs = sum(1 for eid in attached_ids if eid.lower() in text or any(
-        part in text for part in eid.lower().replace("-", " ").split() if len(part) > 3
-    ))
-    if evidence_refs > 0:
-        score += 0.15
-
-    # Harmful mention penalty: derived from the case's actual harmful evidence.
-    # Each case defines its own harmful artifacts, so the penalty adapts to
-    # the specific dispute rather than matching a static keyword list.
-    harmful_terms: set[str] = set()
-    for items in case.evidence_by_system.values():
-        for item in items:
-            if item.harmful:
-                for word in (item.title + " " + item.summary).lower().split():
-                    clean = word.strip(".,;:()")
-                    if len(clean) > 3:
-                        harmful_terms.add(clean)
-    # Remove generic words that would cause false positives
-    harmful_terms -= {"was", "the", "and", "for", "that", "with", "from", "time", "detail"}
-    harmful_hits = sum(1 for term in harmful_terms if term in text)
-    if harmful_hits > 0:
-        score -= 0.12 * min(harmful_hits, 3)
-
-    return max(0.0, min(1.0, score))
+__all__ = [
+    "grade_representment_note",
+    "score_case",
+    "grade_episode",
+]
 
 
-def score_case(
-    case: InternalCase,
-    progress: CaseProgress,
-    step_count: int,
-) -> CaseScoreBreakdown:
-    """Score one case deterministically."""
+_CASE_RUBRIC = CaseRubric()
+_EPISODE_RUBRIC = ChargebackOpsEpisodeRubric()
+
 
+def _build_case_notes(case: InternalCase, progress: CaseProgress, step_count: int) -> str:
     final_resolution = progress.final_resolution or "unresolved"
     attached_set = set(progress.attached_evidence_ids)
-    required_attached = len(attached_set.intersection(case.required_evidence_ids))
-    helpful_attached = len(attached_set.intersection(case.helpful_evidence_ids))
     harmful_attached = len(attached_set.intersection(case.harmful_evidence_ids))
 
-    if final_resolution == case.optimal_strategy:
-        strategy_correctness = 1.0
-    elif final_resolution in case.acceptable_strategies:
-        strategy_correctness = 0.35
-    else:
-        strategy_correctness = 0.0
-
-    if final_resolution == "contest":
-        if case.optimal_strategy != "contest" and "contest" not in case.acceptable_strategies:
-            # Contesting a case that should not be contested — evidence is irrelevant
-            evidence_quality = 0.0
-            packet_validity = 0.0
-        else:
-            base_evidence_quality = 0.7 * _ratio(required_attached, len(case.required_evidence_ids))
-            bonus = 0.3 * _ratio(helpful_attached, max(1, len(case.helpful_evidence_ids)))
-            penalty = 0.25 * harmful_attached
-            evidence_quality = max(0.0, min(1.0, base_evidence_quality + bonus - penalty))
-            packet_validity = (
-                1.0
-                if required_attached == len(case.required_evidence_ids) and harmful_attached == 0
-                else 0.0
-            )
-    else:
-        if final_resolution in {"accept_chargeback", "issue_refund"}:
-            if case.optimal_strategy == "contest":
-                # Conceded a contestable case — evidence gathering was abandoned
-                evidence_quality = 0.15
-                packet_validity = 0.0
-            else:
-                evidence_quality = 1.0 if helpful_attached == 0 and harmful_attached == 0 else 0.7
-                packet_validity = 1.0
-        else:
-            evidence_quality = 0.0
-            packet_validity = 0.0
-
-    resolution_step = progress.resolved_at_step if progress.resolved_at_step is not None else step_count
-    deadline_compliance = 1.0
-    if final_resolution == "unresolved":
-        deadline_compliance = 0.0
-    elif resolution_step > case.deadline_step:
-        deadline_compliance = 0.0
-
-    # --- Efficiency: penalise shallow operational behaviour ---
-    wasted_actions = progress.duplicate_queries + progress.invalid_actions
-    efficiency = max(0.0, 1.0 - min(0.9, wasted_actions * 0.1 + progress.submit_attempts * 0.05))
-
-    # Penalty: over-querying a concedable case wastes steps
-    if final_resolution in {"accept_chargeback", "issue_refund"} and case.optimal_strategy != "contest":
-        systems_queried = len(progress.revealed_systems)
-        if systems_queried > 2:
-            efficiency -= 0.15 * (systems_queried - 2)
-
-    # Penalty: retrieving policy too late to change the outcome
-    if progress.policy_retrieved and resolution_step is not None:
-        # The case was already being resolved, policy retrieval was wasted
-        if final_resolution in {"accept_chargeback", "issue_refund"} and case.optimal_strategy in {
-            "accept_chargeback", "issue_refund"
-        }:
-            # Correct concession but wasted a step on policy retrieval
-            efficiency -= 0.08
-
-    # Reward: early correct concession on a clearly bad case (≤3 steps used)
-    if (
-        final_resolution in {"accept_chargeback", "issue_refund"}
-        and case.optimal_strategy in {"accept_chargeback", "issue_refund"}
-        and resolution_step is not None
-        and resolution_step <= 3
-    ):
-        efficiency = min(1.0, efficiency + 0.1)
-
-    efficiency = max(0.0, min(1.0, efficiency))
-
-    if final_resolution == case.optimal_strategy:
-        outcome_quality = 1.0
-    elif final_resolution in case.acceptable_strategies:
-        outcome_quality = 0.4
-    else:
-        outcome_quality = 0.0
-
-    # Representment note quality (only relevant for contested cases)
-    if final_resolution == "contest" and progress.representment_note:
-        note_quality = grade_representment_note(progress.representment_note, case, attached_set)
-    else:
-        note_quality = 0.0
-
-    weighted_score = (
-        0.25 * strategy_correctness
-        + 0.20 * evidence_quality
-        + 0.15 * packet_validity
-        + 0.15 * deadline_compliance
-        + 0.10 * efficiency
-        + 0.10 * outcome_quality
-        + 0.05 * note_quality
-    )
-
     note_parts = [case.resolution_summary]
     if harmful_attached:
         note_parts.append("Harmful evidence weakened the case.")
@@ -194,19 +52,32 @@ def score_case(
         note_parts.append("Case was never resolved.")
     elif step_count > case.deadline_step:
         note_parts.append("Resolution happened after the deadline.")
+    return " ".join(note_parts)
+
+
+def score_case(
+    case: InternalCase,
+    progress: CaseProgress,
+    step_count: int,
+) -> CaseScoreBreakdown:
+    """Score one case deterministically via the case rubric."""
+
+    ctx = GradingContext(case=case, progress=progress, step_count=step_count)
+    weighted = _CASE_RUBRIC(ctx, None)
+    dims = _CASE_RUBRIC.dimension_scores()
 
     return CaseScoreBreakdown(
         case_id=case.case_id,
-        strategy_correctness=round(strategy_correctness, 4),
-        evidence_quality=round(evidence_quality, 4),
-        packet_validity=round(packet_validity, 4),
-        deadline_compliance=round(deadline_compliance, 4),
-        efficiency=round(efficiency, 4),
-        outcome_quality=round(outcome_quality, 4),
-        note_quality=round(note_quality, 4),
-        weighted_score=round(weighted_score * case.weight, 4),
-        final_resolution=final_resolution,
-        notes=" ".join(note_parts),
+        strategy_correctness=round(dims["strategy_correctness"], 4),
+        evidence_quality=round(dims["evidence_quality"], 4),
+        packet_validity=round(dims["packet_validity"], 4),
+        deadline_compliance=round(dims["deadline_compliance"], 4),
+        efficiency=round(dims["efficiency"], 4),
+        outcome_quality=round(dims["outcome_quality"], 4),
+        note_quality=round(dims["note_quality"], 4),
+        weighted_score=round(weighted * case.weight, 4),
+        final_resolution=progress.final_resolution or "unresolved",
+        notes=_build_case_notes(case, progress, step_count),
     )
 
 
@@ -217,7 +88,7 @@ def grade_episode(
     episode_id: str,
     completed: bool,
 ) -> GraderReport:
-    """Grade a full episode."""
+    """Grade a full episode via the episode-level rubric."""
 
     case_reports = [
         score_case(case, progress_by_case[case.case_id], step_count)
@@ -225,7 +96,14 @@ def grade_episode(
     ]
     total_weight = sum(case.weight for case in task.cases)
     total_score = sum(report.weighted_score for report in case_reports)
-    normalized = 0.0 if total_weight == 0 else min(1.0, total_score / total_weight)
+
+    ctx = EpisodeGradingContext(
+        task=task,
+        progress_by_case=progress_by_case,
+        step_count=step_count,
+    )
+    normalized = float(_EPISODE_RUBRIC(ctx, None))
+
     summary = (
         f"Resolved {sum(1 for report in case_reports if report.final_resolution != 'unresolved')}/"
         f"{len(case_reports)} cases with normalized score {normalized:.3f}."

evaluation/rubrics.py

@@ -0,0 +1,356 @@
+"""OpenEnv Rubric subclasses that power ChargebackOps grading.
+
+Every scoring dimension is a standalone :class:`openenv.core.rubrics.Rubric`
+so the whole grader can be introspected via ``named_rubrics``, captured via
+``state_dict``, and swapped piecewise (e.g. replace :class:`NoteQualityRubric`
+with an ``LLMJudge``). The per-case composite uses :class:`WeightedSum` with
+weights that must sum to 1.0.
+
+The rubrics take their inputs via a :class:`GradingContext` dataclass passed
+as the ``action`` argument of :meth:`Rubric.forward`. The ``observation``
+argument is ignored — ChargebackOps grading operates over deterministic
+episode progress, not on the last observation payload. This keeps the rubrics
+pure and unit-testable without an environment instance.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any
+
+from openenv.core.rubrics import Rubric, WeightedSum
+
+try:
+    from ..scenarios.simulation import CaseProgress, InternalCase, TaskScenario
+except ImportError:  # pragma: no cover
+    from scenarios.simulation import CaseProgress, InternalCase, TaskScenario
+
+
+@dataclass(frozen=True)
+class GradingContext:
+    """Inputs one per-case rubric evaluation needs."""
+
+    case: InternalCase
+    progress: CaseProgress
+    step_count: int
+
+
+@dataclass(frozen=True)
+class EpisodeGradingContext:
+    """Inputs the episode-level rubric needs."""
+
+    task: TaskScenario
+    progress_by_case: dict[str, CaseProgress]
+    step_count: int
+
+
+def _ratio(numerator: int, denominator: int) -> float:
+    if denominator <= 0:
+        return 1.0
+    return max(0.0, min(1.0, numerator / denominator))
+
+
+def _final_resolution(progress: CaseProgress) -> str:
+    return progress.final_resolution or "unresolved"
+
+
+def _contest_is_valid(case: InternalCase) -> bool:
+    return case.optimal_strategy == "contest" or "contest" in case.acceptable_strategies
+
+
+class StrategyCorrectnessRubric(Rubric):
+    """Score final strategy: optimal=1.0, acceptable=0.35, else 0.0."""
+
+    def forward(self, action: Any, observation: Any) -> float:
+        ctx: GradingContext = action
+        final = _final_resolution(ctx.progress)
+        if final == ctx.case.optimal_strategy:
+            return 1.0
+        if final in ctx.case.acceptable_strategies:
+            return 0.35
+        return 0.0
+
+
+class EvidenceQualityRubric(Rubric):
+    """Score the evidence packet attached to the case.
+
+    Zeroes out (vacuous-truth fix) when the agent contests a case that was
+    never contestable — no evidence quality can rescue a wrong strategy.
+    """
+
+    def forward(self, action: Any, observation: Any) -> float:
+        ctx: GradingContext = action
+        case = ctx.case
+        progress = ctx.progress
+        final = _final_resolution(progress)
+
+        attached_set = set(progress.attached_evidence_ids)
+        required_attached = len(attached_set.intersection(case.required_evidence_ids))
+        helpful_attached = len(attached_set.intersection(case.helpful_evidence_ids))
+        harmful_attached = len(attached_set.intersection(case.harmful_evidence_ids))
+
+        if final == "contest":
+            if not _contest_is_valid(case):
+                return 0.0
+            base = 0.7 * _ratio(required_attached, len(case.required_evidence_ids))
+            bonus = 0.3 * _ratio(helpful_attached, max(1, len(case.helpful_evidence_ids)))
+            penalty = 0.25 * harmful_attached
+            return max(0.0, min(1.0, base + bonus - penalty))
+
+        if final in {"accept_chargeback", "issue_refund"}:
+            if case.optimal_strategy == "contest":
+                return 0.15
+            return 1.0 if helpful_attached == 0 and harmful_attached == 0 else 0.7
+
+        return 0.0
+
+
+class PacketValidityRubric(Rubric):
+    """All-or-nothing: required evidence complete AND no harmful attached."""
+
+    def forward(self, action: Any, observation: Any) -> float:
+        ctx: GradingContext = action
+        case = ctx.case
+        progress = ctx.progress
+        final = _final_resolution(progress)
+
+        attached_set = set(progress.attached_evidence_ids)
+        required_attached = len(attached_set.intersection(case.required_evidence_ids))
+        harmful_attached = len(attached_set.intersection(case.harmful_evidence_ids))
+
+        if final == "contest":
+            if not _contest_is_valid(case):
+                return 0.0
+            if (
+                required_attached == len(case.required_evidence_ids)
+                and harmful_attached == 0
+            ):
+                return 1.0
+            return 0.0
+
+        if final in {"accept_chargeback", "issue_refund"}:
+            if case.optimal_strategy == "contest":
+                return 0.0
+            return 1.0
+
+        return 0.0
+
+
+class DeadlineComplianceRubric(Rubric):
+    """1.0 if resolved on time, else 0.0."""
+
+    def forward(self, action: Any, observation: Any) -> float:
+        ctx: GradingContext = action
+        case = ctx.case
+        progress = ctx.progress
+        final = _final_resolution(progress)
+
+        if final == "unresolved":
+            return 0.0
+        resolution_step = (
+            progress.resolved_at_step
+            if progress.resolved_at_step is not None
+            else ctx.step_count
+        )
+        if resolution_step > case.deadline_step:
+            return 0.0
+        return 1.0
+
+
+class EfficiencyRubric(Rubric):
+    """Penalise wasted / redundant actions, reward early correct concessions."""
+
+    def forward(self, action: Any, observation: Any) -> float:
+        ctx: GradingContext = action
+        case = ctx.case
+        progress = ctx.progress
+        final = _final_resolution(progress)
+
+        wasted_actions = progress.duplicate_queries + progress.invalid_actions
+        efficiency = max(
+            0.0,
+            1.0 - min(0.9, wasted_actions * 0.1 + progress.submit_attempts * 0.05),
+        )
+
+        # Over-querying a concedable case is wasted exploration.
+        if final in {"accept_chargeback", "issue_refund"} and case.optimal_strategy != "contest":
+            systems_queried = len(progress.revealed_systems)
+            if systems_queried > 2:
+                efficiency -= 0.15 * (systems_queried - 2)
+
+        # Retrieving policy after the decision was already made is wasted.
+        if progress.policy_retrieved and progress.resolved_at_step is not None:
+            if final in {"accept_chargeback", "issue_refund"} and case.optimal_strategy in {
+                "accept_chargeback",
+                "issue_refund",
+            }:
+                efficiency -= 0.08
+
+        # Early correct concession bonus.
+        if (
+            final in {"accept_chargeback", "issue_refund"}
+            and case.optimal_strategy in {"accept_chargeback", "issue_refund"}
+            and progress.resolved_at_step is not None
+            and progress.resolved_at_step <= 3
+        ):
+            efficiency = min(1.0, efficiency + 0.1)
+
+        return max(0.0, min(1.0, efficiency))
+
+
+class OutcomeQualityRubric(Rubric):
+    """Discrete outcome quality: optimal=1.0, acceptable=0.4, else 0.0."""
+
+    def forward(self, action: Any, observation: Any) -> float:
+        ctx: GradingContext = action
+        final = _final_resolution(ctx.progress)
+        if final == ctx.case.optimal_strategy:
+            return 1.0
+        if final in ctx.case.acceptable_strategies:
+            return 0.4
+        return 0.0
+
+
+class NoteQualityRubric(Rubric):
+    """Text-based representment note scorer (contest-only)."""
+
+    def forward(self, action: Any, observation: Any) -> float:
+        ctx: GradingContext = action
+        progress = ctx.progress
+        if _final_resolution(progress) != "contest" or not progress.representment_note:
+            return 0.0
+        return grade_representment_note(
+            progress.representment_note,
+            ctx.case,
+            set(progress.attached_evidence_ids),
+        )
+
+
+def grade_representment_note(
+    note: str | None,
+    case: InternalCase,
+    attached_ids: set[str],
+) -> float:
+    """Score a representment note from 0.0 to 1.0.
+
+    Evaluates whether the note references required policy claims, mentions
+    attached evidence, has sufficient substance, and avoids harmful mentions.
+    """
+
+    if not note or not note.strip():
+        return 0.0
+
+    text = note.lower()
+    score = 0.0
+
+    # Substance: minimum length for a coherent note.
+    word_count = len(text.split())
+    if word_count >= 5:
+        score += 0.2
+    elif word_count >= 2:
+        score += 0.1
+
+    # Required claims coverage: does the note mention policy requirements?
+    if case.policy_requirements:
+        claims_hit = 0
+        for req in case.policy_requirements:
+            req_keywords = req.lower().split()
+            if any(kw in text for kw in req_keywords if len(kw) > 3):
+                claims_hit += 1
+        score += 0.5 * _ratio(claims_hit, len(case.policy_requirements))
+    else:
+        score += 0.3
+
+    # Evidence coherence: does the note reference attached evidence?
+    evidence_refs = sum(
+        1
+        for eid in attached_ids
+        if eid.lower() in text
+        or any(part in text for part in eid.lower().replace("-", " ").split() if len(part) > 3)
+    )
+    if evidence_refs > 0:
+        score += 0.15
+
+    # Harmful mention penalty derived from each case's harmful evidence blobs.
+    harmful_terms: set[str] = set()
+    for items in case.evidence_by_system.values():
+        for item in items:
+            if item.harmful:
+                for word in (item.title + " " + item.summary).lower().split():
+                    clean = word.strip(".,;:()")
+                    if len(clean) > 3:
+                        harmful_terms.add(clean)
+    harmful_terms -= {"was", "the", "and", "for", "that", "with", "from", "time", "detail"}
+    harmful_hits = sum(1 for term in harmful_terms if term in text)
+    if harmful_hits > 0:
+        score -= 0.12 * min(harmful_hits, 3)
+
+    return max(0.0, min(1.0, score))
+
+
+# Weights must match the order of rubrics handed to WeightedSum and sum to 1.0.
+CASE_DIMENSION_WEIGHTS: tuple[float, ...] = (0.25, 0.20, 0.15, 0.15, 0.10, 0.10, 0.05)
+CASE_DIMENSION_NAMES: tuple[str, ...] = (
+    "strategy_correctness",
+    "evidence_quality",
+    "packet_validity",
+    "deadline_compliance",
+    "efficiency",
+    "outcome_quality",
+    "note_quality",
+)
+
+
+class CaseRubric(Rubric):
+    """Per-case composite — weighted sum of the seven scoring dimensions."""
+
+    def __init__(self) -> None:
+        super().__init__()
+        self.aggregator = WeightedSum(
+            rubrics=[
+                StrategyCorrectnessRubric(),
+                EvidenceQualityRubric(),
+                PacketValidityRubric(),
+                DeadlineComplianceRubric(),
+                EfficiencyRubric(),
+                OutcomeQualityRubric(),
+                NoteQualityRubric(),
+            ],
+            weights=list(CASE_DIMENSION_WEIGHTS),
+        )
+
+    def forward(self, action: Any, observation: Any) -> float:
+        return self.aggregator(action, observation)
+
+    def dimension_scores(self) -> dict[str, float]:
+        """Return per-dimension scores from the most recent forward pass."""
+
+        scores: dict[str, float] = {}
+        for name, child in zip(CASE_DIMENSION_NAMES, self.aggregator._rubric_list):
+            scores[name] = float(child.last_score) if child.last_score is not None else 0.0
+        return scores
+
+
+class ChargebackOpsEpisodeRubric(Rubric):
+    """Episode-level rubric: aggregate per-case scores weighted by case.weight."""
+
+    def __init__(self) -> None:
+        super().__init__()
+        self.case_rubric = CaseRubric()
+
+    def forward(self, action: Any, observation: Any) -> float:
+        ctx: EpisodeGradingContext = action
+        total = 0.0
+        total_weight = 0.0
+        for case in ctx.task.cases:
+            case_ctx = GradingContext(
+                case=case,
+                progress=ctx.progress_by_case[case.case_id],
+                step_count=ctx.step_count,
+            )
+            case_score = self.case_rubric(case_ctx, observation)
+            total += case_score * case.weight
+            total_weight += case.weight
+        if total_weight == 0:
+            return 0.0
+        return min(1.0, total / total_weight)

server/chargeback_ops_environment.py

@@ -11,6 +11,7 @@ from openenv.core.env_server.interfaces import Environment
 try:
     from ..core.episode_store import record_report
     from ..evaluation.grading import grade_episode
+    from ..evaluation.rubrics import ChargebackOpsEpisodeRubric
     from ..core.models import (
         ActionTraceItem,
         CaseQueueItem,
@@ -26,6 +27,7 @@ try:
 except ImportError:  # pragma: no cover
     from core.episode_store import record_report
     from evaluation.grading import grade_episode
+    from evaluation.rubrics import ChargebackOpsEpisodeRubric
     from core.models import (
         ActionTraceItem,
         CaseQueueItem,
@@ -48,7 +50,7 @@ class ChargebackOpsEnvironment(
     SUPPORTS_CONCURRENT_SESSIONS: bool = True
 
     def __init__(self):
-        super().__init__()
+        super().__init__(rubric=ChargebackOpsEpisodeRubric())
         self._task = get_task("goods_not_received_easy")
         self._selected_case_id: str | None = None
         self._last_action_result = "Environment initialized."
@@ -114,6 +116,7 @@ class ChargebackOpsEnvironment(
             objective=self._task.objective,
         )
         self._reset_task_state()
+        self._reset_rubric()
         return self._build_observation(reward=0.0, done=False)
 
     def step(

tests/test_grader.py

@@ -1,4 +1,8 @@
 from evaluation.grading import grade_episode
+from evaluation.rubrics import (
+    CASE_DIMENSION_WEIGHTS,
+    ChargebackOpsEpisodeRubric,
+)
 from server.chargeback_ops_environment import ChargebackOpsEnvironment
 from scenarios.simulation import get_task
 
@@ -14,3 +18,23 @@ def test_grade_episode_bounds():
         completed=False,
     )
     assert 0.0 <= report.normalized_score <= 1.0
+
+
+def test_environment_exposes_rubric_tree():
+    """The env must wire an OpenEnv Rubric that exposes all 7 scoring dimensions."""
+
+    env = ChargebackOpsEnvironment()
+    assert isinstance(env.rubric, ChargebackOpsEpisodeRubric)
+
+    names = {name for name, _ in env.rubric.named_rubrics()}
+    expected = {
+        "case_rubric",
+        "case_rubric.aggregator",
+        *(f"case_rubric.aggregator.rubric_{i}" for i in range(7)),
+    }
+    assert expected.issubset(names)
+
+    # Weights must sum to 1.0 (WeightedSum enforces this at construction but
+    # we lock the constant here so weight changes stay intentional).
+    assert abs(sum(CASE_DIMENSION_WEIGHTS) - 1.0) < 1e-6
+    assert len(CASE_DIMENSION_WEIGHTS) == 7