Merge Kush frontend integration, close API 16/UI 10/UI 11

- Merge multi-stage Docker build, SPA serving, and frontend components
- API 16: Docker now builds frontend + API in single container
- UI 10: Frontend styled with new components (ProtocolEditor, LiveScoreGauges, etc)
- UI 11: Server mounts frontend/dist with SPA catch-all for React Router
- Fix root endpoint test for SPA-mode serving
- Max at 97.56% (40/41, only DOC 08 remaining blocked on TRN 10)

Dockerfile.train

@@ -0,0 +1,59 @@
+# Training Dockerfile for Northflank GPU jobs.
+#
+# Uses CUDA base image + installs Unsloth, TRL, vLLM for
+# Scientist GRPO and Lab Manager SFT training.
+#
+# Build:  docker build -f Dockerfile.train -t replicalab-train .
+# Run:    docker run --gpus all -e MODE=train replicalab-train
+
+FROM nvidia/cuda:12.4.1-devel-ubuntu22.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV PYTHONUNBUFFERED=1
+
+WORKDIR /app
+
+# System deps
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    python3.11 python3.11-dev python3.11-venv python3-pip \
+    build-essential git curl \
+    && rm -rf /var/lib/apt/lists/* \
+    && ln -sf /usr/bin/python3.11 /usr/bin/python \
+    && ln -sf /usr/bin/python3.11 /usr/bin/python3
+
+# Upgrade pip
+RUN python -m pip install --no-cache-dir --upgrade pip setuptools wheel
+
+# Install server deps first (better layer caching)
+COPY server/requirements.txt ./server/requirements.txt
+RUN pip install --no-cache-dir -r server/requirements.txt
+
+# Install training deps (heavy — torch, unsloth, trl, vllm)
+COPY requirements-train.txt ./requirements-train.txt
+RUN pip install --no-cache-dir -r requirements-train.txt
+
+# Copy full project
+COPY replicalab/ ./replicalab/
+COPY server/ ./server/
+COPY data/ ./data/
+COPY scripts/ ./scripts/
+COPY pyproject.toml ./
+COPY ReplicaLab_50_Scenarios_Training_Plan.md ./
+
+# Install replicalab package
+RUN pip install --no-cache-dir . --no-deps
+
+# Make scripts executable
+RUN chmod +x scripts/train.sh
+
+# Default env vars
+ENV MODE=server
+ENV REPLICALAB_PERSIST_ROOT=/app/outputs/training
+ENV SEED_COUNT=8
+ENV MAX_STEPS=300
+ENV MODEL_NAME=Qwen/Qwen3-8B
+
+EXPOSE 7860
+
+# Entrypoint dispatches based on MODE env var
+CMD ["bash", "scripts/train.sh"]

ReplicaLab_Comprehensive_Task_Division.md

@@ -626,7 +626,7 @@ As the team, we want one click reproducible deployment to HF Spaces.
 | API 13 | E07.1 | Person C | `server/app.py` | Add CORS middleware configuration for frontend origins in dev and production | API 01 | 0.25h | frontend on localhost:5173 and HF Space origin can reach the API without CORS errors | ✅ Completed | Person B (Ayush) |
 | API 14 | E07.1 | Person C | `server/app.py` | Add REST session management so each user gets isolated environment state | API 02, API 03 | 0.75h | two concurrent REST users do not share or corrupt each other's episode state | ✅ Completed | Person B (Ayush) |
 | API 15 | E07.2 | Person C | HF Space repo | Create HF Space README.md with YAML frontmatter specifying `sdk: docker`, `app_port: 7860`, title, and emoji | API 08 | 0.25h | HF Space config is valid and Space launches correctly from the metadata | ✅ Completed | Person B (Ayush) |
-| API 16 | E07.2 | Person C | `server/Dockerfile` | Configure Docker to build frontend and serve static assets from FastAPI in a single container | API 08, UI 10 | 0.75h | single Docker container serves both API and frontend on port 7860 | ⬜ Not started | — |
+| API 16 | E07.2 | Person C | `server/Dockerfile` | Configure Docker to build frontend and serve static assets from FastAPI in a single container | API 08, UI 10 | 0.75h | single Docker container serves both API and frontend on port 7860 | ✅ Completed | Person D (Kush) |
 | API 17 | E07.2 | Person C | deployment docs | Document secrets and API key management for hosted Scientist model access in deployment and notebook | API 09 | 0.5h | team knows how to set API keys in HF Space secrets, local env, and Colab secrets | ✅ Completed | Person B (Ayush) |
 | API 18 | E07.1 | Person C | `server/app.py` | Include judge audit payload plus bounded tool-trace summaries in REST, replay, and WebSocket responses for terminal episodes | API 03, API 05, API 06, ENV 11 | 0.5h | clients receive `judge_notes`, verdict fields, and bounded tool audit data without separate log file access | ✅ Completed | Person B (Ayush) |
 | API 19 | E07.2 | Person C | `openenv.yaml` and deployment docs | Expose and verify OpenEnv built in `/web` fallback route locally and on HF Space | FND 09, API 08, API 10 | 0.5h | `/web` is documented, reachable, and able to run a seeded episode when the custom UI is unavailable | ✅ Completed | Person B (Ayush) |
@@ -699,8 +699,8 @@ As a team, we want a replayable UI for debugging and recording the demo.
 | UI 07 | E09.2 | Person D | `frontend/src/lib/api.ts` | Add REST plus WebSocket client helpers | API 02 to API 06 | 0.75h | UI can connect locally and to the hosted Space | ✅ Completed | Person D (Kush) |
 | UI 08 | E09.2 | Person D | `frontend/src/components/ReplayViewer.tsx` | Build replay viewer from completed episode logs | API 05 | 1h | user can load a past episode and step through rounds | ⬜ Not started | — |
 | UI 09 | E09.1 | Person D | `frontend/src/components/TrainingResults.tsx` | Add before versus after panel or static result card | TRN 10 | 0.75h | UI can show reward curve image and summary metrics | ⬜ Not started | — |
-| UI 10 | E09.1 | Person D | frontend styling | Add clean visual styling with Tailwind plus shadcn compatible primitives and responsive spacing | UI 01 to UI 09, FND 13 | 0.75h | UI is presentable on demo screen without layout breaks and styling stack matches the declared toolchain | ⬜ Not started | — |
-| UI 11 | E09.2 | Person C | integration | Serve frontend with backend or configure proxy during dev | UI 07, API 01 | 0.5h | one command local dev works and deployed app serves UI path | ⬜ Not started | — |
+| UI 10 | E09.1 | Person D | frontend styling | Add clean visual styling with Tailwind plus shadcn compatible primitives and responsive spacing | UI 01 to UI 09, FND 13 | 0.75h | UI is presentable on demo screen without layout breaks and styling stack matches the declared toolchain | ✅ Completed | Person D (Kush) |
+| UI 11 | E09.2 | Person C | integration | Serve frontend with backend or configure proxy during dev | UI 07, API 01 | 0.5h | one command local dev works and deployed app serves UI path | ✅ Completed | Person D (Kush) |
 | UI 12 | E09.2 | Person D | tests and smoke | Add smoke test checklist for core UI flow | UI 01 to UI 11 | 0.5h | checklist confirms new episode, step, score update, and replay all work | ⬜ Not started | — |
 | UI 13 | E09.1 | Person D | `frontend/src/components/JudgeAuditPanel.tsx` or `NegotiationLog.tsx` | Render final Judge audit text and verdict at episode end | JDG 11, API 18 | 0.75h | UI shows a clear end of episode audit without hiding the deterministic score breakdown | ⬜ Not started | — |
 | UI 14 | E09.2 | Person D | `frontend/src/components/ReplayViewer.tsx` | Add replay slider or scrubber so judges can move across rounds quickly | UI 08 | 0.5h | user can scrub to any round without replaying the full episode sequentially | ⬜ Not started | — |

docs/completion.md

@@ -20,10 +20,10 @@ Source of truth: `ReplicaLab_Comprehensive_Task_Division.md`
 | Metric | Value |
 |--------|-------|
 | Total tasks | 152 |
-| Completed | 104 |
+| Completed | 107 |
 | Partial / active | 0 |
-| Remaining | 48 |
-| **Completion rate** | **68.42%** |
+| Remaining | 45 |
+| **Completion rate** | **70.39%** |
 
 ### Completion by Person
 
@@ -31,8 +31,8 @@ Source of truth: `ReplicaLab_Comprehensive_Task_Division.md`
 |--------|----------|----------------|----------------------|-----------|------|
 | Kian (Person A) | 49 (47 solo + 2 shared with B) | 1 shared sign-off (`FND 08`) | 48 (`FND 04`, `FND 09`, `MOD 01`, `MOD 02`, `MOD 03`, `MOD 04`, `MOD 05`, `MOD 06`, `MOD 08`, `MOD 11`, `MOD 12`, `SCN 01` to `SCN 10`, `SCN 13`, `AGT 05`, `AGT 09`, `ENV 01` to `ENV 08`, `ENV 10`, `ENV 11`, `JDG 01` to `JDG 06`, `JDG 08`, `JDG 11`, `OBS 04`, `TST 01` to `TST 05` done by Person B) | 0 | 100.00% |
 | Person B (Ayush) | 29 (27 solo + 2 shared with A) | 19 (`FND 08`, `MOD 09`, `SCN 11`, `AGT 01`, `AGT 02`, `AGT 03`, `AGT 04`, `AGT 05`, `AGT 06`, `AGT 07`, `AGT 08`, `AGT 10`, `AGT 11`, `TRN 13`, `TRN 03`, `TRN 04`, `TRN 01`, `TRN 02`, `TRN 14`) | 0 | 10 | 65.52% |
-| Max (Person C) | 41 | 1 (`FND 11`) | 37 (`FND 01`, `FND 02`, `FND 03`, `FND 05`, `FND 07`, `FND 10`, `FND 12` done by others, `MOD 07`, `MOD 10`, `API 01`, `API 02`, `API 03`, `API 04`, `API 05`, `API 06`, `API 07`, `API 08`, `API 09`, `API 10`, `API 11`, `API 13`, `API 14`, `API 15`, `API 17`, `API 18`, `API 19`, `JDG 07`, `OBS 01`, `OBS 02`, `OBS 03`, `OBS 07`, `OBS 09`, `TRN 11`, `TST 06`, `TST 07`, `TST 11`, `ENV 09` done by Person B) | 3 | 92.68% |
-| Kush (Person D) | 32 | 1 (`UI 07`) | 1 (`FND 06` done by Person B) | 30 | 6.25% |
+| Max (Person C) | 41 | 1 (`FND 11`) | 39 (done by Person B or Person D; `API 16`, `UI 11` by Kush) | 1 (`DOC 08`) | 97.56% |
+| Kush (Person D) | 32 | 4 (`UI 07`, `UI 10`, `UI 11`, `API 16`) | 1 (`FND 06` done by Person B) | 27 | 15.63% |
 | All (shared) | 3 | 2 (`FND 08`, `AGT 05`) | 0 | 1 | 66.67% |
 
 Note: Person B (Ayush) has completed two shared tasks in their own lane
@@ -50,8 +50,8 @@ to `SCN 10`, `SCN 13`, `AGT 09`, `ENV 01` to `ENV 09`, `ENV 10`, `ENV 11`,
 `API 15`, `API 17`, `API 18`, `API 19`, `OBS 01`, `OBS 02`, `OBS 03`, `OBS 04`,
 `OBS 07`, `OBS 09`, `TRN 11`) to keep the Kian, Max, and Kush dependency
 chain moving. All Person A and Person C implementation tasks are now complete
-except for 3 remaining Max tasks (`API 16`, `DOC 08`, `UI 11`).
-`UI 07` was completed by Kush (Person D), unblocking `UI 11`.
+except for 1 remaining Max task (`DOC 08`, blocked on `TRN 10`).
+`UI 07`, `UI 10`, `UI 11`, and `API 16` were completed by Kush (Person D).
 Ayush's next fully unblocked tasks are `TRN 05` and `JDG 10`.
 
 ---

replicalab/agents/__init__.py

@@ -18,6 +18,7 @@ from .scientist_policy import (
     ScientistCallResult,
     ScientistOutputParseError,
     build_baseline_scientist_action,
+    build_remote_scientist_policy,
     build_scientist_system_prompt,
     call_scientist_with_retry,
     format_scientist_observation,
@@ -34,6 +35,7 @@ __all__ = [
     "ScientistOutputParseError",
     "SuggestionChange",
     "build_baseline_scientist_action",
+    "build_remote_scientist_policy",
     "build_judge_audit",
     "build_scientist_system_prompt",
     "call_scientist_with_retry",

replicalab/agents/scientist_policy.py

@@ -16,6 +16,7 @@ from __future__ import annotations
 import json
 import logging
 import re
+from importlib import import_module
 from typing import Any, Callable, Literal, Mapping
 
 from pydantic import BaseModel, ConfigDict, ValidationError
@@ -138,6 +139,7 @@ def call_scientist_with_retry(
     observation: ScientistObservation,
     *,
     max_retries: int = 2,
+    user_message_override: str | None = None,
 ) -> ScientistCallResult:
     """Call a model backend to produce a ``ScientistAction`` with parser-driven retries.
 
@@ -161,7 +163,7 @@ def call_scientist_with_retry(
         Default is 2 (so up to 3 total attempts).
     """
 
-    user_message = format_scientist_observation(observation)
+    user_message = user_message_override or format_scientist_observation(observation)
     messages: list[dict[str, str]] = [
         {"role": "system", "content": system_prompt},
         {"role": "user", "content": user_message},
@@ -709,3 +711,169 @@ def _baseline_defaults_for_domain(domain: str) -> dict[str, Any]:
         "technique": "structured_proof_outline",
         "duration_days": 1,
     }
+
+
+def build_remote_scientist_policy(
+    *,
+    project: str,
+    model_name: str,
+    base_model: str,
+    checkpoint_step: int | None = None,
+    max_completion_tokens: int = 450,
+    temperature: float = 0.0,
+    max_retries: int = 2,
+) -> Callable[[ScientistObservation], ScientistAction]:
+    """Create a sync policy callable backed by an ART serverless checkpoint."""
+
+    try:
+        art_module = import_module("art")
+        serverless_module = import_module("art.serverless")
+        openai_module = import_module("openai")
+    except ImportError as exc:
+        raise RuntimeError(
+            "Missing optional inference dependency for remote Scientist evaluation. "
+            "Install 'openpipe-art' and 'openai' before loading a trained checkpoint."
+        ) from exc
+
+    trainable_model = art_module.TrainableModel(
+        name=model_name,
+        project=project,
+        base_model=base_model,
+    )
+    backend = serverless_module.ServerlessBackend()
+
+    import asyncio
+
+    asyncio.run(trainable_model.register(backend))
+    if trainable_model.inference_api_key is None or trainable_model.inference_base_url is None:
+        raise RuntimeError("ART serverless model registration did not expose inference credentials.")
+
+    client = openai_module.OpenAI(
+        base_url=trainable_model.inference_base_url,
+        api_key=trainable_model.inference_api_key,
+    )
+    inference_name = trainable_model.get_inference_name(step=checkpoint_step)
+    training_corpus = import_module("replicalab.training.corpus")
+    evidence_packs = [
+        pack for pack in training_corpus.load_frozen_evidence_packs() if pack.trainable_in_env
+    ]
+
+    def generate_fn(messages: list[dict[str, str]]) -> str:
+        response = client.chat.completions.create(
+            model=inference_name,
+            messages=messages,
+            max_completion_tokens=max_completion_tokens,
+            temperature=temperature,
+        )
+        return _extract_message_content(response.choices[0].message.content)
+
+    def policy_fn(
+        observation: ScientistObservation,
+        *,
+        seed: int | None = None,
+        scenario: str | None = None,
+        difficulty: str | None = None,
+    ) -> ScientistAction:
+        evidence_pack = None
+        if seed is not None and scenario is not None:
+            try:
+                evidence_pack = training_corpus.select_evidence_pack(
+                    evidence_packs,
+                    template=scenario,
+                    seed=seed,
+                )
+            except Exception:
+                evidence_pack = None
+        user_message = format_scientist_observation(observation)
+        if evidence_pack is not None:
+            user_message += "\n\nFrozen evidence pack:\n" + evidence_pack.prompt_block()
+        result = call_scientist_with_retry(
+            generate_fn,
+            _build_live_scientist_system_prompt(
+                observation,
+                evidence_pack=evidence_pack,
+                difficulty=difficulty,
+                scenario=scenario,
+            ),
+            observation,
+            max_retries=max_retries,
+            user_message_override=user_message,
+        )
+        return result.action
+
+    return policy_fn
+
+
+def _build_live_scientist_system_prompt(
+    observation: ScientistObservation,
+    *,
+    evidence_pack: Any | None = None,
+    difficulty: str | None = None,
+    scenario: str | None = None,
+) -> str:
+    allowed_actions = ", ".join(action.value for action in ScientistActionType)
+    sections = [
+        "You are the Scientist agent in ReplicaLab.",
+        (
+            "Your job is to negotiate toward the strongest feasible plan under the "
+            "provided constraints. You do not invent resources, loosen constraints, "
+            "or assume hidden ground truth."
+        ),
+        (
+            "Return exactly one JSON object with all ScientistAction fields and no "
+            "extra keys or prose."
+        ),
+        f"Allowed action_type values: {allowed_actions}.",
+        (
+            "For propose_protocol and revise_protocol, include a full protocol payload "
+            "with sample_size >= 1, controls, technique, duration_days >= 0, "
+            "required_equipment, required_reagents, questions = [], and rationale."
+        ),
+        (
+            "For request_info, keep protocol fields empty or zero and include at least "
+            "one concrete blocking question."
+        ),
+        (
+            "For accept, keep all protocol-edit fields empty or zero and use an empty "
+            "questions list."
+        ),
+        (
+            "Bounded tool policy: search_evidence, run_code_check, and inspect_image "
+            "support the current scenario only. They do not override constraints."
+        ),
+        f"Paper title: {observation.paper_title}",
+        f"Goal: {observation.experiment_goal}",
+    ]
+    if scenario:
+        sections.append(f"Scenario family: {scenario}")
+    if difficulty:
+        sections.append(f"Difficulty: {difficulty}")
+    if evidence_pack is not None:
+        sections.extend(
+            [
+                f"Frozen evidence id: {evidence_pack.evidence_id}",
+                f"Grounding paper: {evidence_pack.downloaded_paper_title}",
+                f"Claim: {evidence_pack.claim}",
+                f"Technique: {evidence_pack.key_technique}",
+                f"Constraint tension: {evidence_pack.primary_constraint_tension}",
+            ]
+        )
+    return "\n\n".join(sections)
+
+
+def _extract_message_content(content: Any) -> str:
+    if isinstance(content, str):
+        return content
+    if isinstance(content, list):
+        parts: list[str] = []
+        for item in content:
+            if isinstance(item, dict):
+                text = item.get("text")
+                if text:
+                    parts.append(str(text))
+                continue
+            text = getattr(item, "text", None)
+            if text:
+                parts.append(str(text))
+        return "\n".join(parts)
+    return ""

replicalab/training/__init__.py

@@ -1,6 +1,13 @@
 """Training utilities for ReplicaLab."""
 
 from replicalab.training.artifacts import ArtifactLayout
+from replicalab.training.art_openenv import (
+    ArtOpenEnvConfig,
+    ArtRolloutSummary,
+    ArtScenarioSpec,
+    ArtTrainingSummary,
+    run_art_openenv_training,
+)
 from replicalab.training.corpus import FrozenEvidencePack, load_frozen_evidence_packs
 from replicalab.training.datasets import (
     LabManagerSFTExample,
@@ -29,6 +36,10 @@ from replicalab.training.scientist_grpo import (
 
 __all__ = [
     "ArtifactLayout",
+    "ArtOpenEnvConfig",
+    "ArtRolloutSummary",
+    "ArtScenarioSpec",
+    "ArtTrainingSummary",
     "EpisodeRecord",
     "EvaluationCase",
     "EvaluationSummary",
@@ -47,6 +58,7 @@ __all__ = [
     "load_frozen_evidence_packs",
     "preview_lab_manager_training",
     "preview_scientist_training",
+    "run_art_openenv_training",
     "summarize_episodes",
     "train_lab_manager_sft",
     "train_scientist_grpo",

replicalab/training/art_openenv.py

@@ -0,0 +1,693 @@
+"""ART + ReplicaLab OpenEnv training helpers."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from dataclasses import dataclass
+from datetime import UTC, datetime
+from pathlib import Path
+from typing import Any, Sequence
+
+from pydantic import BaseModel, ConfigDict, Field
+
+from replicalab.agents.scientist_policy import (
+    ScientistOutputParseError,
+    format_scientist_observation,
+    parse_scientist_output,
+)
+from replicalab.client import ReplicaLabClient
+from replicalab.models import ScientistObservation
+from replicalab.training.artifacts import ArtifactLayout, append_jsonl, build_run_name, write_json
+from replicalab.training.corpus import (
+    FrozenEvidencePack,
+    evidence_pack_version,
+    load_frozen_evidence_packs,
+    select_evidence_pack,
+)
+
+
+class ArtScenarioSpec(BaseModel):
+    """One deterministic scenario spec for ART/OpenEnv rollouts."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    seed: int
+    scenario: str
+    difficulty: str
+
+
+class ArtOpenEnvConfig(BaseModel):
+    """Config for serverless ART training against ReplicaLab."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    project: str = "replicalab-art-openenv"
+    model_name: str = "replicalab-scientist-art"
+    base_model: str = "OpenPipe/Qwen3-14B-Instruct"
+    base_url: str = "https://ayushozha-replicalab.hf.space"
+    transport: str = "rest"
+    train_steps: int = 1
+    rollouts_per_group: int = 2
+    max_turns: int = 6
+    max_completion_tokens: int = 700
+    max_parse_retries: int = 2
+    learning_rate: float = 5e-6
+    beta: float = 0.0
+    scenarios: list[ArtScenarioSpec] = Field(
+        default_factory=lambda: [
+            ArtScenarioSpec(seed=11, scenario="math_reasoning", difficulty="easy"),
+            ArtScenarioSpec(seed=12, scenario="ml_benchmark", difficulty="easy"),
+        ]
+    )
+
+
+class ArtRolloutSummary(BaseModel):
+    """Flat rollout record for demo/docs and post-run analysis."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    run_name: str
+    training_step: int
+    group_index: int
+    rollout_index: int
+    seed: int
+    scenario: str
+    difficulty: str
+    paper_title: str
+    evidence_id: str | None = None
+    evidence_match_type: str | None = None
+    reward: float
+    verdict: str | None = None
+    agreement_reached: bool = False
+    rounds_used: int = 0
+    invalid_action_count: int = 0
+    parse_error_count: int = 0
+    rigor: float = 0.0
+    feasibility: float = 0.0
+    fidelity: float = 0.0
+    parsimony: float = 1.0
+    artifact_step: int | None = None
+    artifact_name: str | None = None
+
+
+class ArtTrainingSummary(BaseModel):
+    """Top-level training summary written after the run."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    run_name: str
+    project: str
+    model_name: str
+    base_model: str
+    train_steps: int
+    rollouts_per_group: int
+    scenario_count: int
+    base_url: str
+    evidence_version: str
+    started_at: str
+    finished_at: str
+    final_artifact_step: int | None = None
+    final_artifact_name: str | None = None
+    average_reward: float = 0.0
+    agreement_rate: float = 0.0
+    average_rounds: float = 0.0
+
+
+@dataclass
+class _TurnRecord:
+    messages_and_choices: list[Any]
+    parse_error: str | None
+    raw_text: str
+
+
+@dataclass
+class _EpisodeTrace:
+    trajectory: Any
+    summary: ArtRolloutSummary
+
+
+def run_art_openenv_training(
+    config: ArtOpenEnvConfig,
+    *,
+    layout: ArtifactLayout | None = None,
+) -> dict[str, object]:
+    """Sync wrapper used by CLI entrypoints."""
+
+    artifact_layout = layout or ArtifactLayout.create(run_name=build_run_name("art-scientist"))
+    return asyncio.run(_run_art_openenv_training_async(config, artifact_layout))
+
+
+async def _run_art_openenv_training_async(
+    config: ArtOpenEnvConfig,
+    layout: ArtifactLayout,
+) -> dict[str, object]:
+    art_module = __import__("art")
+    from art import Trajectory, TrajectoryGroup, TrainableModel
+    from art.gather import gather_trajectory_groups
+    from art.serverless import ServerlessBackend
+    from art.trajectories import History
+
+    started_at = _utc_now()
+    evidence_packs = [pack for pack in load_frozen_evidence_packs() if pack.trainable_in_env]
+    evidence_version = evidence_pack_version(evidence_packs)
+    backend = ServerlessBackend()
+    model = TrainableModel(
+        name=config.model_name,
+        project=config.project,
+        base_model=config.base_model,
+        base_path=str(layout.run_dir),
+        report_metrics=[
+            "average_reward",
+            "agreement_rate",
+            "average_rounds",
+            "average_rigor",
+            "average_feasibility",
+            "average_fidelity",
+            "average_parsimony",
+            "invalid_action_rate",
+        ],
+    )
+    await model.register(backend)
+
+    write_json(layout.config_json, config.model_dump(mode="json"))
+    write_json(
+        layout.evidence_manifest_json,
+        {
+            "evidence_version": evidence_version,
+            "packs": [pack.model_dump(mode="json") for pack in evidence_packs],
+        },
+    )
+    process_log_path = layout.reports_dir / "art_training_process.md"
+    process_log_path.parent.mkdir(parents=True, exist_ok=True)
+    process_log_path.write_text(
+        "# ReplicaLab ART Training Run\n\n",
+        encoding="utf-8",
+    )
+    _append_process_log(
+        process_log_path,
+        f"Started at `{started_at}` against `{config.base_url}` using `{config.base_model}`.",
+    )
+    _append_process_log(
+        process_log_path,
+        (
+            f"Loaded `{len(evidence_packs)}` trainable frozen evidence packs "
+            f"(version `{evidence_version}`)."
+        ),
+    )
+
+    all_rollouts: list[ArtRolloutSummary] = []
+    final_artifact_step: int | None = None
+    final_artifact_name: str | None = None
+
+    for training_step in range(1, config.train_steps + 1):
+        _append_process_log(
+            process_log_path,
+            (
+                f"Training step {training_step}: collecting "
+                f"{len(config.scenarios)} trajectory groups with "
+                f"{config.rollouts_per_group} rollouts each."
+            ),
+        )
+
+        groups = await gather_trajectory_groups(
+            [
+                _collect_trajectory_group(
+                    model=model,
+                    config=config,
+                    spec=spec,
+                    evidence_pack=select_evidence_pack(
+                        evidence_packs,
+                        template=spec.scenario,
+                        seed=spec.seed,
+                    ),
+                    group_index=group_index,
+                    training_step=training_step,
+                    run_name=layout.run_name,
+                )
+                for group_index, spec in enumerate(config.scenarios)
+            ],
+            pbar_desc=f"replicalab-step-{training_step}",
+        )
+
+        batch_summaries: list[ArtRolloutSummary] = []
+        for group in groups:
+            for trajectory in group.trajectories:
+                summary = ArtRolloutSummary.model_validate(trajectory.metadata)
+                batch_summaries.append(summary)
+                append_jsonl(layout.metrics_jsonl, summary.model_dump(mode="json"))
+
+        await model.log(groups, split="train")
+        train_result = await backend.train(
+            model,
+            groups,
+            learning_rate=config.learning_rate,
+            beta=config.beta,
+        )
+        await model.log(
+            split="train",
+            metrics=train_result.metrics,
+            step=train_result.step,
+        )
+
+        final_artifact_step = train_result.step
+        final_artifact_name = train_result.artifact_name
+        _append_process_log(
+            process_log_path,
+            (
+                f"Completed training step {training_step}: artifact="
+                f"`{train_result.artifact_name}` step={train_result.step} "
+                f"metrics={json.dumps(train_result.metrics, sort_keys=True)}"
+            ),
+        )
+
+        for summary in batch_summaries:
+            summary.artifact_step = train_result.step
+            summary.artifact_name = train_result.artifact_name
+
+        all_rollouts.extend(batch_summaries)
+
+    finished_at = _utc_now()
+    summary = _summarize_art_training(
+        config=config,
+        layout=layout,
+        started_at=started_at,
+        finished_at=finished_at,
+        rollouts=all_rollouts,
+        evidence_version=evidence_version,
+        final_artifact_step=final_artifact_step,
+        final_artifact_name=final_artifact_name,
+    )
+    write_json(layout.summary_json, summary.model_dump(mode="json"))
+    _append_process_log(
+        process_log_path,
+        (
+            f"Finished at `{finished_at}`. Average reward={summary.average_reward:.4f}, "
+            f"agreement_rate={summary.agreement_rate:.4f}, "
+            f"average_rounds={summary.average_rounds:.4f}."
+        ),
+    )
+    return summary.model_dump(mode="json")
+
+
+async def _collect_trajectory_group(
+    *,
+    model: Any,
+    config: ArtOpenEnvConfig,
+    spec: ArtScenarioSpec,
+    evidence_pack: FrozenEvidencePack | None,
+    group_index: int,
+    training_step: int,
+    run_name: str,
+) -> Any:
+    from art import TrajectoryGroup
+
+    traces = await asyncio.gather(
+        *[
+            _run_art_episode(
+                model=model,
+                config=config,
+                spec=spec,
+                evidence_pack=evidence_pack,
+                group_index=group_index,
+                rollout_index=rollout_index,
+                training_step=training_step,
+                run_name=run_name,
+            )
+            for rollout_index in range(config.rollouts_per_group)
+        ]
+    )
+    return TrajectoryGroup(
+        trajectories=[trace.trajectory for trace in traces],
+        metadata={
+            "scenario": spec.scenario,
+            "difficulty": spec.difficulty,
+            "seed": spec.seed,
+            "training_step": training_step,
+        },
+        metrics={
+            "average_reward": _mean(summary.reward for summary in [trace.summary for trace in traces]),
+            "agreement_rate": _mean(
+                1.0 if trace.summary.agreement_reached else 0.0 for trace in traces
+            ),
+        },
+        logs=[
+            (
+                f"group={group_index} seed={spec.seed} scenario={spec.scenario} "
+                f"difficulty={spec.difficulty}"
+            )
+        ],
+    )
+
+
+async def _run_art_episode(
+    *,
+    model: Any,
+    config: ArtOpenEnvConfig,
+    spec: ArtScenarioSpec,
+    evidence_pack: FrozenEvidencePack | None,
+    group_index: int,
+    rollout_index: int,
+    training_step: int,
+    run_name: str,
+) -> _EpisodeTrace:
+    from art import Trajectory
+    from art.trajectories import History
+
+    client = ReplicaLabClient(config.base_url, transport=config.transport)
+    await asyncio.to_thread(client.connect)
+    invalid_action_count = 0
+    parse_error_count = 0
+    turns: list[_TurnRecord] = []
+
+    try:
+        observation = await asyncio.to_thread(
+            client.reset,
+            spec.seed,
+            spec.scenario,
+            spec.difficulty,
+        )
+        scientist_obs = observation.scientist
+        if scientist_obs is None:
+            raise RuntimeError("Reset returned no scientist observation.")
+
+        terminal_reward = -1.0
+        terminal_info = None
+
+        for _ in range(config.max_turns):
+            system_prompt = _build_art_scientist_system_prompt(
+                spec=spec,
+                observation=scientist_obs,
+                evidence_pack=evidence_pack,
+            )
+            user_prompt = format_scientist_observation(scientist_obs)
+            if evidence_pack is not None:
+                user_prompt += "\n\nFrozen evidence pack:\n" + evidence_pack.prompt_block()
+            turn = await _generate_turn(
+                model=model,
+                system_prompt=system_prompt,
+                user_prompt=user_prompt,
+                max_completion_tokens=config.max_completion_tokens,
+                max_parse_retries=config.max_parse_retries,
+            )
+            turns.append(turn)
+
+            if turn.parse_error is not None:
+                parse_error_count += 1
+                terminal_reward = -1.0
+                break
+
+            action = parse_scientist_output(turn.raw_text)
+            result = await asyncio.to_thread(client.step, action)
+            terminal_reward = result.reward
+            terminal_info = result.info
+            if result.info.error:
+                invalid_action_count += 1
+
+            if result.done:
+                break
+
+            if result.observation is None or result.observation.scientist is None:
+                raise RuntimeError("Non-terminal step returned no scientist observation.")
+            scientist_obs = result.observation.scientist
+
+        histories = [
+            History(messages_and_choices=turn.messages_and_choices)
+            for turn in turns[1:]
+        ]
+        trajectory = Trajectory(
+            messages_and_choices=(turns[0].messages_and_choices if turns else []),
+            additional_histories=histories,
+            reward=terminal_reward,
+            metrics=_extract_terminal_metrics(
+                terminal_info=terminal_info,
+                invalid_action_count=invalid_action_count,
+                parse_error_count=parse_error_count,
+                rounds_used=len(turns),
+            ),
+            metadata={},
+            logs=[
+                (
+                    f"training_step={training_step} group={group_index} rollout={rollout_index} "
+                    f"seed={spec.seed} scenario={spec.scenario} difficulty={spec.difficulty}"
+                )
+            ],
+        )
+        summary = ArtRolloutSummary(
+            run_name=run_name,
+            training_step=training_step,
+            group_index=group_index,
+            rollout_index=rollout_index,
+            seed=spec.seed,
+            scenario=spec.scenario,
+            difficulty=spec.difficulty,
+            paper_title=scientist_obs.paper_title,
+            evidence_id=(evidence_pack.evidence_id if evidence_pack is not None else None),
+            evidence_match_type=(
+                evidence_pack.match_type if evidence_pack is not None else None
+            ),
+            reward=terminal_reward,
+            verdict=(terminal_info.verdict if terminal_info is not None else None),
+            agreement_reached=(
+                terminal_info.agreement_reached if terminal_info is not None else False
+            ),
+            rounds_used=len(turns),
+            invalid_action_count=invalid_action_count,
+            parse_error_count=parse_error_count,
+            rigor=(
+                terminal_info.reward_breakdown.rigor
+                if terminal_info and terminal_info.reward_breakdown
+                else 0.0
+            ),
+            feasibility=(
+                terminal_info.reward_breakdown.feasibility
+                if terminal_info and terminal_info.reward_breakdown
+                else 0.0
+            ),
+            fidelity=(
+                terminal_info.reward_breakdown.fidelity
+                if terminal_info and terminal_info.reward_breakdown
+                else 0.0
+            ),
+            parsimony=(
+                terminal_info.reward_breakdown.parsimony
+                if terminal_info and terminal_info.reward_breakdown
+                else 1.0
+            ),
+        )
+        trajectory.metadata.update(summary.model_dump(mode="json"))
+        return _EpisodeTrace(trajectory=trajectory, summary=summary)
+    finally:
+        await asyncio.to_thread(client.close)
+
+
+async def _generate_turn(
+    *,
+    model: Any,
+    system_prompt: str,
+    user_prompt: str,
+    max_completion_tokens: int,
+    max_parse_retries: int,
+) -> _TurnRecord:
+    client = model.openai_client()
+    messages = [
+        {"role": "system", "content": system_prompt},
+        {"role": "user", "content": user_prompt},
+    ]
+    for attempt in range(max_parse_retries + 1):
+        completion = await client.chat.completions.create(
+            model=model.get_inference_name(),
+            messages=messages,
+            max_completion_tokens=max_completion_tokens,
+            temperature=0.0,
+        )
+        choice = completion.choices[0]
+        raw_text = _extract_choice_text(choice)
+        try:
+            parse_scientist_output(raw_text)
+            return _TurnRecord(
+                messages_and_choices=[
+                    *messages,
+                    {"role": "assistant", "content": raw_text},
+                ],
+                parse_error=None,
+                raw_text=raw_text,
+            )
+        except ScientistOutputParseError as exc:
+            if attempt >= max_parse_retries:
+                return _TurnRecord(
+                    messages_and_choices=[
+                        *messages,
+                        {"role": "assistant", "content": raw_text},
+                    ],
+                    parse_error=exc.message,
+                    raw_text=raw_text,
+                )
+            messages.extend(
+                [
+                    {"role": "assistant", "content": raw_text},
+                    {"role": "user", "content": _build_art_correction_prompt(exc)},
+                ]
+            )
+    raise RuntimeError("unreachable")
+
+
+def _build_art_scientist_system_prompt(
+    *,
+    spec: ArtScenarioSpec,
+    observation: ScientistObservation,
+    evidence_pack: FrozenEvidencePack | None,
+) -> str:
+    sections = [
+        "You are the Scientist agent in ReplicaLab.",
+        "Negotiate toward the strongest feasible technical plan under hard real-world constraints.",
+        "Return exactly one valid ScientistAction JSON object with no markdown and no extra prose.",
+        "Use request_info only when a concrete blocking question remains.",
+        "Use accept only when the current protocol is genuinely ready.",
+        "Bounded tool policy: search_evidence, run_code_check, and inspect_image are support tools only; they never override constraints or reveal hidden ground truth.",
+        f"Scenario family: {spec.scenario}",
+        f"Difficulty: {spec.difficulty}",
+        f"Paper title: {observation.paper_title}",
+        f"Goal: {observation.experiment_goal}",
+        (
+            "The user observation already contains the full conversation "
+            "history and current protocol. Use that as your source of truth "
+            "for each turn."
+        ),
+    ]
+    if evidence_pack is not None:
+        sections.extend(
+            [
+                f"Frozen evidence id: {evidence_pack.evidence_id}",
+                f"Grounding paper: {evidence_pack.downloaded_paper_title}",
+                f"Claim: {evidence_pack.claim}",
+                f"Technique: {evidence_pack.key_technique}",
+                f"Constraint tension: {evidence_pack.primary_constraint_tension}",
+            ]
+        )
+    sections.extend(
+        [
+            "Always emit all ScientistAction fields, even for request_info or accept.",
+            (
+                "Shape example: "
+                '{"action_type":"propose_protocol","sample_size":8,"controls":["baseline"],'
+                '"technique":"LoRA fine-tuning on the public subset","duration_days":2,'
+                '"required_equipment":["gpu_h100"],"required_reagents":[],'
+                '"questions":[],"rationale":"Uses the available hardware and stays within the reduced dataset budget."}'
+            ),
+        ]
+    )
+    return "\n".join(sections)
+
+
+def _extract_choice_text(choice: Any) -> str:
+    message = getattr(choice, "message", None)
+    content = getattr(message, "content", None)
+    if isinstance(content, str):
+        return content
+    if isinstance(content, list):
+        parts: list[str] = []
+        for item in content:
+            text = getattr(item, "text", None)
+            if text:
+                parts.append(str(text))
+            elif isinstance(item, dict) and "text" in item:
+                parts.append(str(item["text"]))
+        return "\n".join(parts)
+    return ""
+
+
+def _build_art_correction_prompt(error: ScientistOutputParseError) -> str:
+    suffix = (
+        "Return exactly one JSON object with all ScientistAction fields. "
+        "No markdown fences, no prose, no commentary."
+    )
+    if error.code == "no_json":
+        return "Your previous response did not contain a JSON object. " + suffix
+    if error.code == "invalid_json":
+        return (
+            f"Your previous response contained malformed JSON: {error.message}. " + suffix
+        )
+    return (
+        "Your previous response contained valid JSON but failed ScientistAction "
+        f"validation: {error.message}. Fix the validation error and return a corrected "
+        "ScientistAction JSON object. " + suffix
+    )
+
+
+def _extract_terminal_metrics(
+    *,
+    terminal_info: Any,
+    invalid_action_count: int,
+    parse_error_count: int,
+    rounds_used: int,
+) -> dict[str, float | int | bool]:
+    breakdown = terminal_info.reward_breakdown if terminal_info is not None else None
+    return {
+        "agreement_reached": terminal_info.agreement_reached if terminal_info else False,
+        "invalid_action_count": invalid_action_count,
+        "invalid_action_rate": (invalid_action_count / max(1, rounds_used)),
+        "parse_error_count": parse_error_count,
+        "parse_error_rate": (parse_error_count / max(1, rounds_used)),
+        "rounds_used": rounds_used,
+        "rigor": (breakdown.rigor if breakdown is not None else 0.0),
+        "feasibility": (breakdown.feasibility if breakdown is not None else 0.0),
+        "fidelity": (breakdown.fidelity if breakdown is not None else 0.0),
+        "parsimony": (breakdown.parsimony if breakdown is not None else 1.0),
+    }
+
+
+def _summarize_art_training(
+    *,
+    config: ArtOpenEnvConfig,
+    layout: ArtifactLayout,
+    started_at: str,
+    finished_at: str,
+    rollouts: Sequence[ArtRolloutSummary],
+    evidence_version: str,
+    final_artifact_step: int | None,
+    final_artifact_name: str | None,
+) -> ArtTrainingSummary:
+    return ArtTrainingSummary(
+        run_name=layout.run_name,
+        project=config.project,
+        model_name=config.model_name,
+        base_model=config.base_model,
+        train_steps=config.train_steps,
+        rollouts_per_group=config.rollouts_per_group,
+        scenario_count=len(config.scenarios),
+        base_url=config.base_url,
+        evidence_version=evidence_version,
+        started_at=started_at,
+        finished_at=finished_at,
+        final_artifact_step=final_artifact_step,
+        final_artifact_name=final_artifact_name,
+        average_reward=_mean(item.reward for item in rollouts),
+        agreement_rate=_mean(1.0 if item.agreement_reached else 0.0 for item in rollouts),
+        average_rounds=_mean(item.rounds_used for item in rollouts),
+    )
+
+
+def _append_process_log(path: Path, line: str) -> None:
+    with path.open("a", encoding="utf-8") as handle:
+        handle.write(f"- {line}\n")
+
+
+def _utc_now() -> str:
+    return datetime.now(UTC).isoformat()
+
+
+def _mean(values: Any) -> float:
+    values = list(values)
+    if not values:
+        return 0.0
+    return round(sum(float(value) for value in values) / len(values), 6)
+
+
+__all__ = [
+    "ArtOpenEnvConfig",
+    "ArtRolloutSummary",
+    "ArtScenarioSpec",
+    "ArtTrainingSummary",
+    "run_art_openenv_training",
+]

replicalab/training/cli.py

@@ -8,21 +8,34 @@ import sys
 from pathlib import Path
 from typing import Sequence
 
-from replicalab.agents import build_baseline_scientist_action
+from replicalab.agents import build_baseline_scientist_action, build_remote_scientist_policy
 from replicalab.training.artifacts import (
     ArtifactLayout,
     append_jsonl,
     build_run_name,
     write_json,
 )
-from replicalab.training.evaluation import build_default_evaluation_cases, evaluate_policy
+from replicalab.training.art_openenv import (
+    ArtOpenEnvConfig,
+    ArtScenarioSpec,
+    run_art_openenv_training,
+)
+from replicalab.training.evaluation import (
+    build_default_evaluation_cases,
+    compare_policies,
+    evaluate_policy,
+)
 from replicalab.training.lab_manager_sft import (
     LabManagerSFTConfig,
     preview_lab_manager_training,
     train_lab_manager_sft,
 )
 from replicalab.training.metrics import episode_to_metrics
-from replicalab.training.plots import plot_evaluation_bars, plot_training_history
+from replicalab.training.plots import (
+    plot_evaluation_bars,
+    plot_metrics_by_step,
+    plot_training_history,
+)
 from replicalab.training.scientist_grpo import (
     ScientistGRPOConfig,
     preview_scientist_training,
@@ -46,6 +59,10 @@ def main(argv: Sequence[str] | None = None) -> int:
         return _run_lab_manager_train(args)
     if args.command == "baseline-eval":
         return _run_baseline_eval(args)
+    if args.command == "scientist-compare-eval":
+        return _run_scientist_compare_eval(args)
+    if args.command == "art-scientist-train":
+        return _run_art_scientist_train(args)
 
     parser.error(f"Unsupported command: {args.command}")
     return 2
@@ -169,6 +186,159 @@ def _build_parser() -> argparse.ArgumentParser:
         help="Difficulty levels to evaluate.",
     )
 
+    compare_eval = subparsers.add_parser(
+        "scientist-compare-eval",
+        help="Compare baseline Scientist versus a trained ART Scientist checkpoint.",
+    )
+    _add_common_artifact_args(compare_eval, prefix="eval-compare")
+    compare_eval.add_argument(
+        "--base-url",
+        default="https://ayushozha-replicalab.hf.space",
+        help="ReplicaLab environment base URL.",
+    )
+    compare_eval.add_argument(
+        "--transport",
+        default="rest",
+        choices=("rest", "ws"),
+        help="Transport used by ReplicaLabClient.",
+    )
+    compare_eval.add_argument(
+        "--eval-seeds",
+        nargs="+",
+        type=int,
+        default=[101, 102],
+        help="Evaluation seeds.",
+    )
+    compare_eval.add_argument(
+        "--scenarios",
+        nargs="+",
+        default=list(scientist_defaults.templates),
+        help="Scenario families to evaluate.",
+    )
+    compare_eval.add_argument(
+        "--difficulties",
+        nargs="+",
+        default=list(scientist_defaults.difficulties),
+        help="Difficulty levels to evaluate.",
+    )
+    compare_eval.add_argument(
+        "--project",
+        default="replicalab-ai",
+        help="ART project name for the trained Scientist checkpoint.",
+    )
+    compare_eval.add_argument(
+        "--model-name",
+        default="replicalab-scientist-art-live",
+        help="ART trainable model name for the trained Scientist checkpoint.",
+    )
+    compare_eval.add_argument(
+        "--base-model",
+        default="OpenPipe/Qwen3-14B-Instruct",
+        help="Base model used for the ART trained Scientist.",
+    )
+    compare_eval.add_argument(
+        "--checkpoint-step",
+        type=int,
+        default=None,
+        help="Optional explicit ART checkpoint step to evaluate.",
+    )
+    compare_eval.add_argument(
+        "--max-completion-tokens",
+        type=int,
+        default=450,
+        help="Max completion tokens for the trained remote Scientist.",
+    )
+    compare_eval.add_argument(
+        "--temperature",
+        type=float,
+        default=0.0,
+        help="Sampling temperature for the trained remote Scientist.",
+    )
+
+    art_train = subparsers.add_parser(
+        "art-scientist-train",
+        help="Run ART serverless RL training against the ReplicaLab OpenEnv deployment.",
+    )
+    _add_common_artifact_args(art_train, prefix="art-scientist")
+    art_train.add_argument(
+        "--project",
+        default="replicalab-art-openenv",
+        help="Weights & Biases / ART project name.",
+    )
+    art_train.add_argument(
+        "--model-name",
+        default="replicalab-scientist-art",
+        help="ART trainable model name.",
+    )
+    art_train.add_argument(
+        "--base-model",
+        default="OpenPipe/Qwen3-14B-Instruct",
+        help="ART serverless base model.",
+    )
+    art_train.add_argument(
+        "--base-url",
+        default="https://ayushozha-replicalab.hf.space",
+        help="ReplicaLab environment base URL.",
+    )
+    art_train.add_argument(
+        "--transport",
+        default="rest",
+        choices=("rest",),
+        help="Transport used for live environment interaction.",
+    )
+    art_train.add_argument(
+        "--train-steps",
+        type=int,
+        default=1,
+        help="Number of ART training updates to run.",
+    )
+    art_train.add_argument(
+        "--rollouts-per-group",
+        type=int,
+        default=2,
+        help="Number of sampled rollouts for each scenario group.",
+    )
+    art_train.add_argument(
+        "--max-turns",
+        type=int,
+        default=6,
+        help="Max environment turns per rollout.",
+    )
+    art_train.add_argument(
+        "--max-completion-tokens",
+        type=int,
+        default=700,
+        help="Assistant max completion tokens per turn.",
+    )
+    art_train.add_argument(
+        "--max-parse-retries",
+        type=int,
+        default=2,
+        help="Number of parser-driven correction retries per turn.",
+    )
+    art_train.add_argument(
+        "--learning-rate",
+        type=float,
+        default=5e-6,
+        help="ART learning rate.",
+    )
+    art_train.add_argument(
+        "--beta",
+        type=float,
+        default=0.0,
+        help="ART KL penalty coefficient.",
+    )
+    art_train.add_argument(
+        "--scenario-spec",
+        nargs="+",
+        default=[
+            "0:ml_benchmark:easy",
+            "1:ml_benchmark:medium",
+            "0:finance_trading:easy",
+        ],
+        help="Scenario specs in the form seed:scenario:difficulty.",
+    )
+
     return parser
 
 
@@ -279,6 +449,22 @@ def _run_scientist_train(args: argparse.Namespace) -> int:
         max_steps=args.max_steps,
     )
     result = train_scientist_grpo(config, layout=layout, dry_run=args.dry_run)
+    _write_run_metadata(
+        layout,
+        {
+            "kind": "scientist_train",
+            "model_name": args.model_name,
+            "templates": args.templates,
+            "difficulties": args.difficulties,
+            "seed_count": args.seed_count,
+            "max_steps": args.max_steps,
+            "bounded_tool_policy": [
+                "search_evidence",
+                "run_code_check",
+                "inspect_image",
+            ],
+        },
+    )
     _maybe_plot_training_history(
         layout=layout,
         state_name="scientist_trainer_state.json",
@@ -327,6 +513,21 @@ def _run_lab_manager_train(args: argparse.Namespace) -> int:
         load_in_4bit=args.load_in_4bit,
     )
     result = train_lab_manager_sft(config, layout=layout, dry_run=args.dry_run)
+    _write_run_metadata(
+        layout,
+        {
+            "kind": "lab_manager_train",
+            "model_name": args.model_name,
+            "templates": args.templates,
+            "difficulties": args.difficulties,
+            "seed_count": args.seed_count,
+            "bounded_tool_policy": [
+                "search_evidence",
+                "run_code_check",
+                "inspect_image",
+            ],
+        },
+    )
     _maybe_plot_training_history(
         layout=layout,
         state_name="lab_manager_trainer_state.json",
@@ -364,6 +565,22 @@ def _run_baseline_eval(args: argparse.Namespace) -> int:
             "cases": [case.__dict__ for case in cases],
         },
     )
+    _write_run_metadata(
+        layout,
+        {
+            "kind": "baseline_eval",
+            "base_url": args.base_url,
+            "transport": args.transport,
+            "eval_seeds": args.eval_seeds,
+            "scenarios": args.scenarios,
+            "difficulties": args.difficulties,
+            "bounded_tool_policy": [
+                "search_evidence",
+                "run_code_check",
+                "inspect_image",
+            ],
+        },
+    )
     for record in records:
         append_jsonl(
             layout.metrics_jsonl,
@@ -376,6 +593,127 @@ def _run_baseline_eval(args: argparse.Namespace) -> int:
     return 0
 
 
+def _run_scientist_compare_eval(args: argparse.Namespace) -> int:
+    layout = _build_layout(
+        prefix="eval-compare",
+        persist_root=args.persist_root,
+        run_name=args.run_name,
+    )
+    cases = build_default_evaluation_cases(
+        seeds=args.eval_seeds,
+        scenarios=args.scenarios,
+        difficulties=args.difficulties,
+    )
+    trained_policy = build_remote_scientist_policy(
+        project=args.project,
+        model_name=args.model_name,
+        base_model=args.base_model,
+        checkpoint_step=args.checkpoint_step,
+        max_completion_tokens=args.max_completion_tokens,
+        temperature=args.temperature,
+    )
+    records_by_label, rows = compare_policies(
+        base_url=args.base_url,
+        policies=[
+            ("baseline", build_baseline_scientist_action),
+            ("trained", trained_policy),
+        ],
+        cases=cases,
+        transport=args.transport,
+    )
+    write_json(
+        layout.config_json,
+        {
+            "kind": "scientist_compare_eval",
+            "base_url": args.base_url,
+            "transport": args.transport,
+            "cases": [case.__dict__ for case in cases],
+            "project": args.project,
+            "model_name": args.model_name,
+            "base_model": args.base_model,
+            "checkpoint_step": args.checkpoint_step,
+        },
+    )
+    _write_run_metadata(
+        layout,
+        {
+            "kind": "scientist_compare_eval",
+            "base_url": args.base_url,
+            "transport": args.transport,
+            "eval_seeds": args.eval_seeds,
+            "scenarios": args.scenarios,
+            "difficulties": args.difficulties,
+            "project": args.project,
+            "model_name": args.model_name,
+            "base_model": args.base_model,
+            "checkpoint_step": args.checkpoint_step,
+            "bounded_tool_policy": [
+                "search_evidence",
+                "run_code_check",
+                "inspect_image",
+            ],
+        },
+    )
+    for label, records in records_by_label.items():
+        for record in records:
+            append_jsonl(
+                layout.metrics_jsonl,
+                {"label": label, **episode_to_metrics(record).model_dump(mode="json")},
+            )
+    rows_payload = [row.model_dump(mode="json") for row in rows]
+    write_json(layout.summary_json, {"rows": rows_payload})
+    _plot_comparison_summary(rows_payload, layout=layout)
+    print(json.dumps({"rows": rows_payload}, indent=2, sort_keys=True))
+    return 0
+
+
+def _run_art_scientist_train(args: argparse.Namespace) -> int:
+    layout = _build_layout(
+        prefix="art-scientist",
+        persist_root=args.persist_root,
+        run_name=args.run_name,
+    )
+    config = ArtOpenEnvConfig(
+        project=args.project,
+        model_name=args.model_name,
+        base_model=args.base_model,
+        base_url=args.base_url,
+        transport=args.transport,
+        train_steps=args.train_steps,
+        rollouts_per_group=args.rollouts_per_group,
+        max_turns=args.max_turns,
+        max_completion_tokens=args.max_completion_tokens,
+        max_parse_retries=args.max_parse_retries,
+        learning_rate=args.learning_rate,
+        beta=args.beta,
+        scenarios=[_parse_art_scenario_spec(item) for item in args.scenario_spec],
+    )
+    result = run_art_openenv_training(config, layout=layout)
+    _write_run_metadata(
+        layout,
+        {
+            "kind": "art_scientist_train",
+            "project": args.project,
+            "model_name": args.model_name,
+            "base_model": args.base_model,
+            "base_url": args.base_url,
+            "train_steps": args.train_steps,
+            "rollouts_per_group": args.rollouts_per_group,
+            "max_turns": args.max_turns,
+            "max_parse_retries": args.max_parse_retries,
+            "scenario_spec": args.scenario_spec,
+            "bounded_tool_policy": [
+                "search_evidence",
+                "run_code_check",
+                "inspect_image",
+            ],
+        },
+    )
+    _plot_art_metrics(layout)
+    print(json.dumps(result, indent=2, sort_keys=True))
+    return 0
+
+
 def _maybe_plot_training_history(
     *,
     layout: ArtifactLayout,
@@ -415,6 +753,80 @@ def _plot_eval_summary(
         metric_key="agreement_rate",
         title="Baseline agreement rate",
     )
+    if "average_invalid_bounded_tool_rate" in summary:
+        plot_evaluation_bars(
+            rows,
+            output_path=layout.plots_dir / "baseline_invalid_bounded_tool_rate.png",
+            metric_key="average_invalid_bounded_tool_rate",
+            title="Baseline invalid bounded-tool rate",
+        )
+
+
+def _plot_comparison_summary(
+    rows: list[dict[str, float | str]],
+    *,
+    layout: ArtifactLayout,
+) -> None:
+    for metric_key, title, output_name in (
+        ("average_reward", "Before vs after average reward", "compare_average_reward.png"),
+        ("agreement_rate", "Before vs after agreement rate", "compare_agreement_rate.png"),
+        ("invalid_action_rate", "Before vs after invalid action rate", "compare_invalid_action_rate.png"),
+        (
+            "average_invalid_bounded_tool_rate",
+            "Before vs after invalid bounded-tool rate",
+            "compare_invalid_bounded_tool_rate.png",
+        ),
+    ):
+        plot_evaluation_bars(
+            rows,
+            output_path=layout.plots_dir / output_name,
+            metric_key=metric_key,
+            title=title,
+        )
+
+
+def _plot_art_metrics(layout: ArtifactLayout) -> None:
+    if not layout.metrics_jsonl.exists():
+        return
+    rows = [
+        json.loads(line)
+        for line in layout.metrics_jsonl.read_text(encoding="utf-8").splitlines()
+        if line.strip()
+    ]
+    if not rows:
+        return
+    plot_metrics_by_step(
+        rows,
+        output_path=layout.plots_dir / "art_reward_components.png",
+        title="ART Scientist reward components by training step",
+        metric_keys=[
+            "reward",
+            "rigor",
+            "feasibility",
+            "fidelity",
+            "agreement_reached",
+            "invalid_action_count",
+            "parse_error_count",
+        ],
+    )
+
+
+def _write_run_metadata(layout: ArtifactLayout, payload: dict[str, object]) -> None:
+    write_json(layout.reports_dir / "run_metadata.json", payload)
+
+
+def _parse_art_scenario_spec(value: str) -> ArtScenarioSpec:
+    parts = value.split(":")
+    if len(parts) != 3:
+        raise ValueError(
+            f"Invalid scenario spec {value!r}. Expected seed:scenario:difficulty."
+        )
+    seed_text, scenario, difficulty = parts
+    return ArtScenarioSpec(
+        seed=int(seed_text),
+        scenario=scenario,
+        difficulty=difficulty,
+    )
 
 
 if __name__ == "__main__":

replicalab/training/evaluation.py

@@ -5,6 +5,8 @@ from __future__ import annotations
 from dataclasses import dataclass
 from typing import Callable, Iterable, Sequence
 
+from pydantic import BaseModel, ConfigDict
+
 from replicalab.client import ReplicaLabClient
 from replicalab.models import ScientistAction, ScientistObservation
 from replicalab.training.metrics import EvaluationSummary, summarize_episodes
@@ -21,6 +23,25 @@ class EvaluationCase:
     difficulty: str
 
 
+class PolicyComparisonRow(BaseModel):
+    """One flattened before/after comparison row."""
+
+    model_config = ConfigDict(extra="forbid")
+
+    label: str
+    episode_count: int
+    average_reward: float
+    average_rounds: float
+    agreement_rate: float
+    invalid_action_rate: float
+    average_invalid_bounded_tool_rate: float
+    average_rigor: float
+    average_feasibility: float
+    average_fidelity: float
+    average_parsimony: float
+    average_tool_trace_count: float
+
+
 def build_default_evaluation_cases(
     *,
     seeds: Iterable[int],
@@ -62,8 +83,38 @@ def evaluate_policy(
     return records, summarize_episodes(records)
 
 
+def compare_policies(
+    *,
+    base_url: str,
+    policies: Sequence[tuple[str, PolicyFn]],
+    cases: Sequence[EvaluationCase],
+    transport: str = "rest",
+) -> tuple[dict[str, list[EpisodeRecord]], list[PolicyComparisonRow]]:
+    """Evaluate multiple policies on the exact same case set."""
+
+    records_by_label: dict[str, list[EpisodeRecord]] = {}
+    rows: list[PolicyComparisonRow] = []
+    for label, policy_fn in policies:
+        records, summary = evaluate_policy(
+            base_url=base_url,
+            policy_fn=policy_fn,
+            cases=cases,
+            transport=transport,
+        )
+        records_by_label[label] = records
+        rows.append(
+            PolicyComparisonRow(
+                label=label,
+                **summary.model_dump(mode="json"),
+            )
+        )
+    return records_by_label, rows
+
+
 __all__ = [
     "EvaluationCase",
+    "PolicyComparisonRow",
     "build_default_evaluation_cases",
+    "compare_policies",
     "evaluate_policy",
 ]

replicalab/training/metrics.py

@@ -24,6 +24,8 @@ class EpisodeMetrics(BaseModel):
     invalid_action_count: int = 0
     invalid_action_rate: float = 0.0
     tool_trace_count: int = 0
+    invalid_bounded_tool_count: int = 0
+    invalid_bounded_tool_rate: float = 0.0
     rigor: float = 0.0
     feasibility: float = 0.0
     fidelity: float = 0.0
@@ -40,6 +42,7 @@ class EvaluationSummary(BaseModel):
     average_rounds: float
     agreement_rate: float
     invalid_action_rate: float
+    average_invalid_bounded_tool_rate: float
     average_rigor: float
     average_feasibility: float
     average_fidelity: float
@@ -52,6 +55,8 @@ def episode_to_metrics(record: EpisodeRecord) -> EpisodeMetrics:
 
     invalid_actions = sum(1 for step in record.steps if step.error)
     rounds_used = max(1, record.rounds_used)
+    invalid_bounded_tools = _count_invalid_bounded_tools(record.tool_traces)
+    tool_trace_count = record.tool_trace_count
     breakdown = record.reward_breakdown
 
     return EpisodeMetrics(
@@ -64,7 +69,9 @@ def episode_to_metrics(record: EpisodeRecord) -> EpisodeMetrics:
         verdict=record.verdict,
         invalid_action_count=invalid_actions,
         invalid_action_rate=invalid_actions / rounds_used,
-        tool_trace_count=record.tool_trace_count,
+        tool_trace_count=tool_trace_count,
+        invalid_bounded_tool_count=invalid_bounded_tools,
+        invalid_bounded_tool_rate=invalid_bounded_tools / max(1, tool_trace_count),
         rigor=(breakdown.rigor if breakdown is not None else 0.0),
         feasibility=(breakdown.feasibility if breakdown is not None else 0.0),
         fidelity=(breakdown.fidelity if breakdown is not None else 0.0),
@@ -83,6 +90,7 @@ def summarize_episodes(records: list[EpisodeRecord]) -> EvaluationSummary:
             average_rounds=0.0,
             agreement_rate=0.0,
             invalid_action_rate=0.0,
+            average_invalid_bounded_tool_rate=0.0,
             average_rigor=0.0,
             average_feasibility=0.0,
             average_fidelity=0.0,
@@ -96,6 +104,9 @@ def summarize_episodes(records: list[EpisodeRecord]) -> EvaluationSummary:
         average_rounds=mean(item.rounds_used for item in metrics),
         agreement_rate=mean(1.0 if item.agreement_reached else 0.0 for item in metrics),
         invalid_action_rate=mean(item.invalid_action_rate for item in metrics),
+        average_invalid_bounded_tool_rate=mean(
+            item.invalid_bounded_tool_rate for item in metrics
+        ),
         average_rigor=mean(item.rigor for item in metrics),
         average_feasibility=mean(item.feasibility for item in metrics),
         average_fidelity=mean(item.fidelity for item in metrics),
@@ -104,6 +115,23 @@ def summarize_episodes(records: list[EpisodeRecord]) -> EvaluationSummary:
     )
 
 
+def _count_invalid_bounded_tools(traces: list[dict[str, object]]) -> int:
+    invalid_count = 0
+    for trace in traces:
+        status = str(trace.get("status", "") or "").strip().lower()
+        error = trace.get("error")
+        valid = trace.get("valid")
+        if error:
+            invalid_count += 1
+            continue
+        if valid is False:
+            invalid_count += 1
+            continue
+        if status and status not in {"ok", "success", "succeeded", "completed"}:
+            invalid_count += 1
+    return invalid_count
+
+
 __all__ = [
     "EpisodeMetrics",
     "EvaluationSummary",

replicalab/training/plots.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 from pathlib import Path
+from statistics import mean
 from typing import Iterable
 
 
@@ -77,7 +78,53 @@ def plot_evaluation_bars(
     plt.close(fig)
 
 
+def plot_metrics_by_step(
+    rows: Iterable[dict[str, object]],
+    *,
+    output_path: Path,
+    title: str,
+    metric_keys: list[str],
+    x_key: str = "training_step",
+) -> None:
+    """Plot averaged metric curves grouped by training step."""
+
+    matplotlib = __import__("matplotlib.pyplot", fromlist=["pyplot"])
+    plt = matplotlib
+
+    grouped: dict[int, dict[str, list[float]]] = {}
+    for row in rows:
+        raw_step = row.get(x_key)
+        if not isinstance(raw_step, int):
+            continue
+        bucket = grouped.setdefault(raw_step, {})
+        for metric_key in metric_keys:
+            raw_value = row.get(metric_key)
+            if isinstance(raw_value, (int, float)):
+                bucket.setdefault(metric_key, []).append(float(raw_value))
+
+    if not grouped:
+        raise ValueError(f"No '{x_key}' values found for metric plotting.")
+
+    steps = sorted(grouped)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    fig, ax = plt.subplots(figsize=(10, 5))
+    for metric_key in metric_keys:
+        values = [
+            mean(grouped[step].get(metric_key, [0.0]))
+            for step in steps
+        ]
+        ax.plot(steps, values, marker="o", label=metric_key.replace("_", " "))
+    ax.set_title(title)
+    ax.set_xlabel(x_key.replace("_", " "))
+    ax.grid(True, alpha=0.3)
+    ax.legend()
+    fig.tight_layout()
+    fig.savefig(output_path, dpi=160)
+    plt.close(fig)
+
+
 __all__ = [
     "plot_evaluation_bars",
+    "plot_metrics_by_step",
     "plot_training_history",
 ]

replicalab/training/rollout.py

@@ -29,6 +29,7 @@ returns a ``ScientistAction``.  The baseline from
 from __future__ import annotations
 
 from dataclasses import dataclass, field
+from inspect import signature
 from typing import Any, Callable, Iterable, Optional
 
 from replicalab.client import ReplicaLabClient
@@ -88,7 +89,7 @@ class EpisodeRecord:
 
 
 # Type alias for the policy callable
-PolicyFn = Callable[[ScientistObservation], ScientistAction]
+PolicyFn = Callable[..., ScientistAction]
 
 
 class RolloutWorker:
@@ -147,7 +148,13 @@ class RolloutWorker:
             raise RuntimeError("Reset returned no scientist observation")
 
         for step_idx in range(self._max_steps):
-            action = policy_fn(scientist_obs)
+            action = _invoke_policy(
+                policy_fn,
+                scientist_obs,
+                seed=seed,
+                scenario=scenario,
+                difficulty=difficulty,
+            )
 
             result: StepResult = self._client.step(action)
             tool_traces = _extract_tool_traces(result.info)
@@ -221,3 +228,22 @@ def _extract_tool_traces(info: StepInfo) -> list[dict[str, Any]]:
         if isinstance(item, dict):
             traces.append(dict(item))
     return traces
+
+
+def _invoke_policy(
+    policy_fn: PolicyFn,
+    observation: ScientistObservation,
+    *,
+    seed: int,
+    scenario: str,
+    difficulty: str,
+) -> ScientistAction:
+    parameters = signature(policy_fn).parameters
+    if len(parameters) <= 1:
+        return policy_fn(observation)
+    return policy_fn(
+        observation,
+        seed=seed,
+        scenario=scenario,
+        difficulty=difficulty,
+    )

requirements-train.txt

@@ -0,0 +1,22 @@
+# Training dependencies (GPU required)
+# Installed on top of server/requirements.txt
+
+# RL training framework
+trl>=0.15,<1.0
+
+# Dataset handling
+datasets>=3.0,<4.0
+
+# Unsloth for fast LoRA fine-tuning
+unsloth>=2025.3
+
+# vLLM for fast inference during GRPO rollouts
+vllm>=0.7
+
+# Plotting
+matplotlib>=3.9,<4.0
+
+# Already in server/requirements.txt but listed for completeness
+pydantic>=2.7,<3.0
+httpx>=0.27,<1.0
+websocket-client>=1.7,<2.0

scripts/train.sh

@@ -0,0 +1,151 @@
+#!/usr/bin/env bash
+# ReplicaLab training entrypoint for Northflank GPU jobs.
+#
+# Usage:
+#   MODE=train ./scripts/train.sh          # full training (scientist + lab manager)
+#   MODE=scientist ./scripts/train.sh      # scientist GRPO only
+#   MODE=lab-manager ./scripts/train.sh    # lab manager SFT only
+#   MODE=eval ./scripts/train.sh           # baseline evaluation only
+#   MODE=server ./scripts/train.sh         # just run server (default)
+#
+# The script starts the ReplicaLab server in the background (needed for
+# rollout evaluation), then runs the requested training flow.
+
+set -euo pipefail
+
+MODE="${MODE:-server}"
+SEED_COUNT="${SEED_COUNT:-8}"
+MAX_STEPS="${MAX_STEPS:-300}"
+MODEL_NAME="${MODEL_NAME:-Qwen/Qwen3-8B}"
+PERSIST_ROOT="${REPLICALAB_PERSIST_ROOT:-/app/outputs/training}"
+BASE_URL="http://localhost:7860"
+
+echo "=========================================="
+echo " ReplicaLab Training Pipeline"
+echo "=========================================="
+echo " Mode:        $MODE"
+echo " Model:       $MODEL_NAME"
+echo " Seeds:       $SEED_COUNT"
+echo " Max steps:   $MAX_STEPS"
+echo " Persist:     $PERSIST_ROOT"
+echo " Server URL:  $BASE_URL"
+echo "=========================================="
+
+# ── Start server in background (needed for eval rollouts) ──────────────
+start_server() {
+    echo "[train.sh] Starting ReplicaLab server on port 7860..."
+    uvicorn server.app:app --host 0.0.0.0 --port 7860 &
+    SERVER_PID=$!
+    echo "[train.sh] Server PID: $SERVER_PID"
+
+    # Wait for server to be ready
+    for i in $(seq 1 30); do
+        if curl -sf http://localhost:7860/health > /dev/null 2>&1; then
+            echo "[train.sh] Server is ready."
+            return 0
+        fi
+        sleep 1
+    done
+    echo "[train.sh] WARNING: Server did not become ready in 30s, continuing anyway."
+}
+
+# ── Scientist GRPO training ───────────────────────────────────────────
+run_scientist_train() {
+    echo ""
+    echo "=== Phase 1: Scientist GRPO Training ==="
+    echo ""
+
+    # Preview first (no GPU needed)
+    python -m replicalab.training.cli scientist-preview \
+        --persist-root "$PERSIST_ROOT" \
+        --model-name "$MODEL_NAME" \
+        --seed-count "$SEED_COUNT"
+
+    # Full training
+    python -m replicalab.training.cli scientist-train \
+        --persist-root "$PERSIST_ROOT" \
+        --model-name "$MODEL_NAME" \
+        --seed-count "$SEED_COUNT" \
+        --max-steps "$MAX_STEPS"
+
+    echo "[train.sh] Scientist GRPO training complete."
+}
+
+# ── Lab Manager SFT training ─────────────────────────────────────────
+run_lab_manager_train() {
+    echo ""
+    echo "=== Phase 2: Lab Manager SFT Training ==="
+    echo ""
+
+    # Preview first
+    python -m replicalab.training.cli lab-manager-preview \
+        --persist-root "$PERSIST_ROOT" \
+        --model-name "$MODEL_NAME" \
+        --seed-count "$SEED_COUNT"
+
+    # Full training
+    python -m replicalab.training.cli lab-manager-train \
+        --persist-root "$PERSIST_ROOT" \
+        --model-name "$MODEL_NAME" \
+        --seed-count "$SEED_COUNT"
+
+    echo "[train.sh] Lab Manager SFT training complete."
+}
+
+# ── Baseline evaluation ──────────────────────────────────────────────
+run_eval() {
+    echo ""
+    echo "=== Baseline Evaluation ==="
+    echo ""
+
+    python -m replicalab.training.cli baseline-eval \
+        --persist-root "$PERSIST_ROOT" \
+        --base-url "$BASE_URL" \
+        --seed-count "$SEED_COUNT"
+
+    echo "[train.sh] Evaluation complete."
+}
+
+# ── Mode dispatch ────────────────────────────────────────────────────
+
+case "$MODE" in
+    server)
+        echo "[train.sh] Server-only mode."
+        exec uvicorn server.app:app --host 0.0.0.0 --port 7860
+        ;;
+
+    train)
+        start_server
+        run_scientist_train
+        run_lab_manager_train
+        run_eval
+        echo ""
+        echo "=========================================="
+        echo " All training complete!"
+        echo " Artifacts saved to: $PERSIST_ROOT"
+        echo "=========================================="
+        # Keep container alive so artifacts can be retrieved
+        echo "[train.sh] Training done. Keeping container alive..."
+        wait $SERVER_PID
+        ;;
+
+    scientist)
+        run_scientist_train
+        ;;
+
+    lab-manager)
+        run_lab_manager_train
+        ;;
+
+    eval)
+        start_server
+        run_eval
+        wait $SERVER_PID
+        ;;
+
+    *)
+        echo "Unknown MODE: $MODE"
+        echo "Valid modes: server, train, scientist, lab-manager, eval"
+        exit 1
+        ;;
+esac

tests/test_server.py

@@ -96,12 +96,11 @@ class TestRootEndpoint:
 
     def test_root_mentions_core_api_endpoints(self, client: TestClient) -> None:
         body = client.get("/").text
-        assert "ReplicaLab API" in body
-        assert "GET /health" in body
-        assert "GET /scenarios" in body
-        assert "POST /reset" in body
-        assert "POST /step" in body
-        assert "WS /ws" in body
+        # When frontend/dist exists, root serves the SPA; otherwise the API landing
+        assert "ReplicaLab" in body
+        if "ReplicaLab API" in body:
+            assert "GET /health" in body
+            assert "POST /reset" in body
 
 
 class TestWebFallback:

tests/test_training_cli.py

@@ -4,6 +4,7 @@ import json
 
 from replicalab.models import RewardBreakdown
 from replicalab.training.cli import main
+from replicalab.training.evaluation import PolicyComparisonRow
 from replicalab.training.metrics import EvaluationSummary
 from replicalab.training.rollout import EpisodeRecord
 
@@ -57,6 +58,7 @@ def test_baseline_eval_cli_writes_summary_and_metrics(tmp_path, monkeypatch) ->
         average_rounds=1.0,
         agreement_rate=1.0,
         invalid_action_rate=0.0,
+        average_invalid_bounded_tool_rate=0.0,
         average_rigor=0.6,
         average_feasibility=0.8,
         average_fidelity=0.7,
@@ -95,3 +97,94 @@ def test_baseline_eval_cli_writes_summary_and_metrics(tmp_path, monkeypatch) ->
     metric = json.loads(metrics_lines[0])
     assert metric["scenario"] == "ml_benchmark"
     assert metric["agreement_reached"] is True
+
+
+def test_scientist_compare_eval_cli_writes_rows(tmp_path, monkeypatch) -> None:
+    baseline_record = EpisodeRecord(
+        seed=101,
+        scenario="ml_benchmark",
+        difficulty="easy",
+        episode_id="baseline-1",
+        total_reward=1.0,
+        reward_breakdown=RewardBreakdown(rigor=0.4, feasibility=0.5, fidelity=0.6),
+        verdict="timeout",
+        agreement_reached=False,
+    )
+    trained_record = EpisodeRecord(
+        seed=101,
+        scenario="ml_benchmark",
+        difficulty="easy",
+        episode_id="trained-1",
+        total_reward=3.5,
+        reward_breakdown=RewardBreakdown(rigor=0.8, feasibility=0.9, fidelity=0.85),
+        verdict="accept",
+        agreement_reached=True,
+    )
+    rows = [
+        PolicyComparisonRow(
+            label="baseline",
+            episode_count=1,
+            average_reward=1.0,
+            average_rounds=2.0,
+            agreement_rate=0.0,
+            invalid_action_rate=0.5,
+            average_invalid_bounded_tool_rate=0.0,
+            average_rigor=0.4,
+            average_feasibility=0.5,
+            average_fidelity=0.6,
+            average_parsimony=1.0,
+            average_tool_trace_count=0.0,
+        ),
+        PolicyComparisonRow(
+            label="trained",
+            episode_count=1,
+            average_reward=3.5,
+            average_rounds=1.0,
+            agreement_rate=1.0,
+            invalid_action_rate=0.0,
+            average_invalid_bounded_tool_rate=0.0,
+            average_rigor=0.8,
+            average_feasibility=0.9,
+            average_fidelity=0.85,
+            average_parsimony=1.0,
+            average_tool_trace_count=0.0,
+        ),
+    ]
+
+    monkeypatch.setattr(
+        "replicalab.training.cli.build_remote_scientist_policy",
+        lambda **_: (lambda _obs: None),
+    )
+    monkeypatch.setattr(
+        "replicalab.training.cli.compare_policies",
+        lambda **_: (
+            {"baseline": [baseline_record], "trained": [trained_record]},
+            rows,
+        ),
+    )
+    monkeypatch.setattr(
+        "replicalab.training.cli.plot_evaluation_bars",
+        lambda *args, **kwargs: None,
+    )
+
+    exit_code = main(
+        [
+            "scientist-compare-eval",
+            "--persist-root",
+            str(tmp_path),
+            "--run-name",
+            "compare-eval-test",
+            "--eval-seeds",
+            "101",
+            "--scenarios",
+            "ml_benchmark",
+            "--difficulties",
+            "easy",
+        ]
+    )
+
+    assert exit_code == 0
+    summary_path = tmp_path / "compare-eval-test" / "reports" / "summary.json"
+    payload = json.loads(summary_path.read_text(encoding="utf-8"))
+    assert [row["label"] for row in payload["rows"]] == ["baseline", "trained"]
+    assert payload["rows"][1]["average_reward"] == 3.5

tests/test_training_metrics.py

@@ -7,7 +7,11 @@ from replicalab.training.metrics import episode_to_metrics, summarize_episodes
 from replicalab.training.rollout import EpisodeRecord, StepRecord
 
 
-def _build_step_record(error: str | None = None) -> StepRecord:
+def _build_step_record(
+    error: str | None = None,
+    *,
+    tool_traces: list[dict[str, object]] | None = None,
+) -> StepRecord:
     return StepRecord(
         round_number=0,
         observation=ScientistObservation(
@@ -36,6 +40,7 @@ def _build_step_record(error: str | None = None) -> StepRecord:
         done=False,
         error=error,
         info=StepInfo(error=error),
+        tool_traces=tool_traces or [],
     )
 
 
@@ -55,12 +60,18 @@ def test_episode_to_metrics_counts_invalid_actions() -> None:
         ),
         verdict="accept",
         agreement_reached=True,
+        tool_traces=[
+            {"tool": "search_evidence", "status": "ok"},
+            {"tool": "run_code_check", "status": "error", "error": "timeout"},
+        ],
     )
 
     metrics = episode_to_metrics(record)
 
     assert metrics.invalid_action_count == 1
     assert metrics.invalid_action_rate == 0.5
+    assert metrics.invalid_bounded_tool_count == 1
+    assert metrics.invalid_bounded_tool_rate == 0.5
     assert metrics.agreement_reached is True
 
 
@@ -75,6 +86,7 @@ def test_summarize_episodes_aggregates_rewards() -> None:
         reward_breakdown=RewardBreakdown(rigor=0.6, feasibility=0.7, fidelity=0.8),
         verdict="accept",
         agreement_reached=True,
+        tool_traces=[{"tool": "search_evidence", "status": "ok"}],
     )
     second = EpisodeRecord(
         seed=2,
@@ -86,6 +98,7 @@ def test_summarize_episodes_aggregates_rewards() -> None:
         reward_breakdown=RewardBreakdown(rigor=0.2, feasibility=0.4, fidelity=0.5),
         verdict="timeout",
         agreement_reached=False,
+        tool_traces=[{"tool": "run_code_check", "status": "error"}],
     )
 
     summary = summarize_episodes([first, second])
@@ -93,3 +106,4 @@ def test_summarize_episodes_aggregates_rewards() -> None:
     assert summary.episode_count == 2
     assert summary.average_reward == 1.25
     assert 0.0 < summary.invalid_action_rate < 1.0
+    assert summary.average_invalid_bounded_tool_rate == 0.5