Harbor run viewer: Phase 0 eval, 15 harnesses x 50 tasks, pass@4

.gitignore

@@ -0,0 +1,2 @@
+# Run data lives in the bucket, not in git: that is the whole point of the design.
+data/

CONTRACT.md

@@ -0,0 +1,135 @@
+# Information contract
+
+Everything the viewer shows lives in **one directory**, identical whether it is a local folder or a
+Hugging Face bucket mounted into a Space. A mounted bucket appears as an ordinary filesystem path inside
+the container, so local and hosted use the same code path and there is nothing to keep in sync.
+
+## Hierarchy
+
+```
+project              "data-agent"                          — a body of work
+└── dataset          "eval-v1", "eval-easy50", "dabstep"    — variations within it
+    └── task         one ROW in the table
+        └── cell     one (model, harness) pair
+            └── attempts   k tries → pass@k, each with a trace
+```
+
+Two axes cross at a cell, and **which one is the column is a view choice, not a storage choice** — so
+cells are keyed `"<model>|<harness>"` and the viewer pivots either way. Storing per-axis would force a
+rewrite to flip the table.
+
+```
+<DATA_DIR>/
+└── projects/
+    └── <project_id>/
+        ├── project.json                    # REQUIRED — label, description, source, support tiers
+        ├── datasets/
+        │   └── <dataset_id>/
+        │       ├── dataset.json            # optional label/notes
+        │       ├── summary.json            # tasks[] with cells, by_model, by_harness
+        │       └── traces/<trace_id>.json  # one attempt each; referenced from a cell
+        └── runs/
+            └── <run_id>/                   # training runs belonging to this project
+                ├── run.json
+                └── train/metrics.jsonl
+```
+
+Discovery is by file presence: `project.json` makes a project, `summary.json` a dataset, `run.json` a
+training run. Nothing else is required and the viewer never assumes a file exists.
+
+## `project.json`
+
+```json
+{
+  "project_id": "data-agent",
+  "label": "Data-Agent Bench",
+  "description": "Verified data-analysis tasks over Kaggle datasets, graded exact / numeric / LLM-judge.",
+  "source": {"hf_dataset": "AdithyaSK/data_agent_rl_environment_eval"},
+  "support": {
+    "mini-swe-agent": {"tier": "stable"},
+    "opencode": {"tier": "experimental",
+                 "caveats": ["no step limit; ~90 turns and ~17% of rollouts retry after exit 137"]}
+  }
+}
+```
+
+`support` is per-harness and drives the stable / experimental badge and its warning. A tier with no
+caveat is just a colour, so an experimental harness has to say what does not work.
+
+## `datasets/<id>/summary.json`
+
+```json
+{
+  "k_max": 4,
+  "models": ["Qwen/Qwen3.5-2B"],
+  "harnesses": ["mini-swe-agent", "opencode"],
+  "summary": {"tasks_total": 50, "tasks_any_pass": 31, "attempts_total": 400,
+              "attempts_passed": 74, "n_all_infra": 0},
+  "by_model":   [{"model": "Qwen/Qwen3.5-2B", "cells": 100, "pass@1": 0.17, "pass@4": 0.31,
+                  "n_measured": 100, "mean_turns": 11.7}],
+  "by_harness": [{"harness": "mini-swe-agent", "cells": 100, "pass@1": 0.17, "pass@4": 0.31,
+                  "n_measured": 100, "mean_turns": 11.7}],
+  "tasks": [
+    {"id": "0000_416_416942_qa_3", "index": 7, "difficulty_level": 0, "difficulty": "easy",
+     "question": "What is the median of ...?", "answer": "0.4056", "reward_mode": "numeric",
+     "cells": {
+       "Qwen/Qwen3.5-2B|mini-swe-agent": {
+         "passed_at": 3,
+         "attempts": [{"attempt": 1, "reward": 0.0, "n_turns": 12, "elapsed_sec": 41.2,
+                       "trace": "traces/0000_416_416942_qa_3-2b-mini-1.json"}]
+       }
+     }}
+  ]
+}
+```
+
+Conventions that carry meaning rather than being formatting:
+
+* **`reward: null` means the verifier never ran** — dead sandbox, no answer. Not a zero. Aggregates
+  exclude those attempts and count a task under `n_all_infra` when every attempt is null. Averaging them
+  in as zeros makes infrastructure failure look like a weak model: one run reported `pass@4 0.333` from
+  3 measured tasks out of 8 before that rule existed.
+* **`passed_at`** is the first passing attempt, so `pass@1` and `pass@k` both derive from one structure
+  and cannot disagree.
+* **`trace`** is a path relative to the dataset directory, so opening a cell needs no naming convention
+  the viewer has to guess.
+
+## `traces/<trace_id>.json`
+
+Free-form per attempt; rendered as whatever is present.
+
+```json
+{"task_id": "...", "model": "...", "harness": "...", "attempt": 1,
+ "reward": 0.0, "reward_method": "llm", "n_turns": 12, "elapsed_sec": 41.2,
+ "gold": "0.4056", "predicted": "0.41",
+ "messages": [{"role": "user", "content": "..."},
+              {"role": "assistant", "content": "...", "tool_calls": [...]},
+              {"role": "tool", "content": "..."}]}
+```
+
+`reward_method` is worth carrying: the grader is three-tier (exact → numeric → gpt-4o-mini judge), so
+knowing a 0 came from the judge rather than exact match separates a wrong answer from a formatting
+mismatch.
+
+## `runs/<run_id>/train/metrics.jsonl`
+
+One object per step, append-only so a live run is readable without rewriting.
+
+```json
+{"step": 3, "reward": 0.829, "reward_std": 0.146, "loss": 0.279, "ratio": 0.9999,
+ "kl": 0.032, "entropy": 0.51, "n_rollouts": 8, "n_unscorable": 1, "wall_s": 47.6}
+```
+
+`reward_std` sits next to `reward` because a step whose generations all score the same has zero
+advantage and teaches nothing — a reward curve alone cannot tell learning from a flat run.
+
+## Publishing
+
+```sh
+hf buckets create <ns>/harbor-runs --exist-ok
+HF_HUB_DISABLE_XET=1 hf sync ./data hf://buckets/<ns>/harbor-runs
+hf spaces volumes set <ns>/harbor-viewer -v hf://buckets/<ns>/harbor-runs:/data
+```
+
+`HF_HUB_DISABLE_XET=1` is not decoration: the Xet upload path hangs indefinitely on this cluster while
+the plain HTTP path finishes in seconds.

Dockerfile

@@ -0,0 +1,19 @@
+# Hugging Face Docker Space. The data is NOT baked in: a bucket is mounted at /data, so the image stays
+# tiny and a new run needs no rebuild — only a bucket sync and a click of Refresh.
+FROM python:3.12-slim
+
+# Spaces run as uid 1000.
+RUN useradd -m -u 1000 user
+USER user
+ENV PATH="/home/user/.local/bin:$PATH"
+WORKDIR /app
+
+COPY --chown=user requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+COPY --chown=user . .
+
+# Mount the bucket here: hf spaces volumes set <space> -v hf://buckets/<ns>/<bucket>:/data
+ENV DATA_DIR=/data
+
+# HF routes public traffic to app_port from README.md frontmatter.
+CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7860"]

README.md

@@ -1,13 +1,67 @@
 ---
-title: Dataagent Phase0 Evals
-emoji: 👀
-colorFrom: purple
-colorTo: purple
-sdk: gradio
-sdk_version: 6.26.0
-python_version: '3.13'
-app_file: app.py
+title: Harbor Run Viewer
+emoji: 🔭
+colorFrom: indigo
+colorTo: blue
+sdk: docker
+app_port: 7860
 pinned: false
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Harbor Run Viewer
+
+Train and eval runs from the OpenEnv × Harbor stack. **The data lives in a Hugging Face bucket; this app
+only renders it.**
+
+## The one design decision
+
+A bucket mounted into a Space appears as an ordinary filesystem path inside the container, so the Space
+and a laptop run *the same* code against *the same* layout — `DATA_DIR` is `/data` in one case and
+`./data` in the other. There is no local-vs-remote branch to keep in step, and no data in the image.
+
+```
+laptop:   DATA_DIR=./data          uvicorn app:app
+Space:    DATA_DIR=/data           ← bucket mounted read-only or read-write
+fallback: DATA_BUCKET=ns/bucket    ← read a bucket over HfFileSystem without mounting it
+```
+
+## Run it locally
+
+```sh
+pip install -r requirements.txt
+DATA_DIR=./data uvicorn app:app --port 8000     # http://localhost:8000
+```
+
+## Publish runs to the bucket
+
+```sh
+hf buckets create <ns>/harbor-runs --exist-ok
+hf sync ./data hf://buckets/<ns>/harbor-runs           # add --delete to mirror exactly
+```
+
+## Point the Space at the bucket
+
+```sh
+hf spaces volumes set <ns>/harbor-run-viewer -v hf://buckets/<ns>/harbor-runs:/data
+```
+
+Only buckets support read-write mounts; models, datasets and Spaces are always read-only. A private
+bucket needs `HF_TOKEN` as a Space secret.
+
+Then press **⟳ Refresh** in the UI: it drops every cache and rescans. A mounted bucket already reflects
+new writes, so nothing needs rebuilding or restarting to see a run that finished a minute ago.
+
+## Layout
+
+See [CONTRACT.md](CONTRACT.md). Briefly: one directory per run under `runs/<run_id>/`, identified by a
+`run.json`, with optional `eval/summary.json` and `train/metrics.jsonl`. A run with only training data
+renders as training; one with both shows both; nothing is required beyond `run.json`.
+
+Two conventions in that contract carry real meaning rather than being formatting:
+
+- **`reward: null` is not zero.** It means the verifier never ran — a dead sandbox, no answer. Those
+  attempts are excluded, and a task whose every attempt is null is counted as *unmeasured*. Averaging
+  them in as zeros makes infrastructure failure look like a weak model.
+- **`reward_std` sits next to `reward`.** A step whose generations all score the same has zero advantage
+  and teaches nothing, so the viewer reports *steps with a gradient* — a reward curve alone cannot
+  distinguish learning from a flat run.

app.py

@@ -0,0 +1,278 @@
+"""FastAPI viewer for Harbor train + eval runs. The data lives in a bucket; this only renders it.
+
+ONE READ PATH, TWO DEPLOYMENTS. A Hugging Face bucket mounted into a Space appears as an ordinary
+filesystem path inside the container (`Volume(type="bucket", source=..., mount_path="/data")`), so the
+Space and a laptop run the *same* code against the same layout — `DATA_DIR` is just `/data` in one case
+and `./data` in the other. The alternative, branching between a local reader and an `HfFileSystem`
+reader, means two code paths where only one is ever exercised by whoever is debugging.
+
+`DATA_BUCKET` exists for the case the mount cannot cover: reading a remote bucket from a laptop without
+mounting it. It goes through `HfFileSystem`, which speaks `hf://buckets/<ns>/<name>/<path>`. When both
+are set the local directory wins, because a mount is always fresher and cheaper than the network.
+
+    DATA_DIR     = ./data                 # a local dir, or a mounted bucket at /data
+    DATA_BUCKET  = (unset)                # e.g. AdithyaSK/harbor-runs — remote fallback
+    REFRESH_TTL  = 30                     # seconds a listing is trusted before rescanning
+
+WHY A REFRESH BUTTON AND NOT A WATCHER. A run in progress appends to `metrics.jsonl` and drops new files
+into `traces/`; a mounted bucket reflects that without the app doing anything. But listings are cached
+so that a page load does not restat thousands of files, so the UI needs a way to say "look again now" —
+`POST /api/refresh` drops every cache. Nothing is precomputed, so refresh is the only invalidation
+needed.
+
+HIERARCHY: project -> dataset -> task (row) -> cell (model x harness) -> attempts (pass@k) -> trace.
+Cells are keyed "<model>|<harness>" so the viewer can pivot which axis is the column without refetching;
+that is a view choice, and storing it per-axis would force a rewrite to flip the table.
+
+Endpoints:
+    GET  /                                          → the viewer
+    GET  /api/projects                              → tree: projects, their datasets and runs
+    GET  /api/projects/{pid}/datasets/{did}         → the table (tasks + cells + aggregates)
+    GET  /api/projects/{pid}/datasets/{did}/trace?path=  → one attempt
+    GET  /api/projects/{pid}/runs/{rid}             → a training run's run.json
+    GET  /api/projects/{pid}/runs/{rid}/train       → its metrics.jsonl, parsed
+    POST /api/refresh                               → drop caches; returns what it now sees
+    GET  /healthz                                   → {ok, source, n_projects, ...}
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import time
+from pathlib import PurePosixPath
+from typing import Any
+
+from fastapi import FastAPI, HTTPException, Query
+from fastapi.responses import HTMLResponse, JSONResponse
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+SITE = os.path.join(HERE, "site")
+DATA_DIR = os.getenv("DATA_DIR", os.path.join(HERE, "data"))
+DATA_BUCKET = os.getenv("DATA_BUCKET", "")
+REFRESH_TTL = float(os.getenv("REFRESH_TTL", "30"))
+
+app = FastAPI(title="Harbor run viewer", docs_url="/docs")
+
+_cache: dict[str, Any] = {}
+_cache_at: dict[str, float] = {}
+
+
+# --- storage: a local tree, or a bucket over fsspec -------------------------------------------------
+class _Local:
+    kind = "local"
+
+    def __init__(self, root: str) -> None:
+        self.root = root
+
+    def describe(self) -> str:
+        return f"local:{self.root}"
+
+    def exists(self, rel: str) -> bool:
+        return os.path.exists(os.path.join(self.root, rel))
+
+    def listdir(self, rel: str) -> list[str]:
+        p = os.path.join(self.root, rel)
+        return sorted(os.listdir(p)) if os.path.isdir(p) else []
+
+    def read_text(self, rel: str) -> str:
+        with open(os.path.join(self.root, rel), encoding="utf-8") as f:
+            return f.read()
+
+    def mtime(self, rel: str) -> float:
+        try:
+            return os.path.getmtime(os.path.join(self.root, rel))
+        except OSError:
+            return 0.0
+
+
+class _Bucket:
+    """`hf://buckets/<ns>/<name>/<path>` over HfFileSystem. Used only when nothing is mounted."""
+
+    kind = "bucket"
+
+    def __init__(self, bucket: str) -> None:
+        from huggingface_hub import HfFileSystem
+
+        self.bucket = bucket
+        self.fs = HfFileSystem()
+
+    def _p(self, rel: str) -> str:
+        return f"buckets/{self.bucket}/{rel}".rstrip("/")
+
+    def describe(self) -> str:
+        return f"hf://buckets/{self.bucket}"
+
+    def exists(self, rel: str) -> bool:
+        return bool(self.fs.exists(self._p(rel)))
+
+    def listdir(self, rel: str) -> list[str]:
+        try:
+            return sorted(PurePosixPath(p).name for p in self.fs.ls(self._p(rel), detail=False))
+        except FileNotFoundError:
+            return []
+
+    def read_text(self, rel: str) -> str:
+        with self.fs.open(self._p(rel), "r") as f:
+            return f.read()
+
+    def mtime(self, rel: str) -> float:
+        return 0.0  # not exposed uniformly; the refresh button is the invalidation story
+
+
+def store():
+    """Prefer a real directory: a mounted bucket is fresher and cheaper than the network."""
+    if os.path.isdir(DATA_DIR):
+        return _Local(DATA_DIR)
+    if DATA_BUCKET:
+        return _Bucket(DATA_BUCKET)
+    return _Local(DATA_DIR)  # missing; every read 404s with a path in the message
+
+
+def _cached(key: str, produce):
+    now = time.time()
+    if key in _cache and now - _cache_at.get(key, 0.0) < REFRESH_TTL:
+        return _cache[key]
+    value = produce()
+    _cache[key] = value
+    _cache_at[key] = now
+    return value
+
+
+def _safe_rel(*parts: str) -> str:
+    """Join a relative path and refuse to escape the data directory.
+
+    Trace paths come straight from summary.json, which this app did not write, so a `../` in one of them
+    must not read outside DATA_DIR.
+    """
+    for part in parts:
+        if not part or part.startswith("/") or ".." in PurePosixPath(part).parts:
+            raise HTTPException(400, f"unsafe path component: {part!r}")
+    return str(PurePosixPath(*parts))
+
+
+def _read_json(st, rel: str) -> Any:
+    if not st.exists(rel):
+        raise HTTPException(404, f"missing: {st.describe()}/{rel}")
+    try:
+        return json.loads(st.read_text(rel))
+    except json.JSONDecodeError as exc:
+        raise HTTPException(500, f"{rel} is not valid JSON: {exc}") from exc
+
+
+def _list_projects() -> list[dict]:
+    """A directory with project.json is a project. Datasets and runs are listed alongside it so the
+    sidebar can render the whole tree from one request instead of N+1."""
+    st = store()
+    out = []
+    for pid in st.listdir("projects"):
+        rel = f"projects/{pid}/project.json"
+        if not st.exists(rel):
+            continue
+        try:
+            meta = json.loads(st.read_text(rel))
+        except Exception:  # noqa: BLE001 - one malformed project must not hide the others
+            meta = {"project_id": pid, "error": "project.json is unreadable"}
+        meta.setdefault("project_id", pid)
+        meta["datasets"] = [
+            d for d in st.listdir(f"projects/{pid}/datasets")
+            if st.exists(f"projects/{pid}/datasets/{d}/summary.json")
+        ]
+        meta["runs"] = [
+            r for r in st.listdir(f"projects/{pid}/runs")
+            if st.exists(f"projects/{pid}/runs/{r}/run.json")
+        ]
+        out.append(meta)
+    out.sort(key=lambda p: p.get("label") or p["project_id"])
+    return out
+
+
+@app.get("/", response_class=HTMLResponse)
+def index() -> HTMLResponse:
+    p = os.path.join(SITE, "viewer.html")
+    if not os.path.exists(p):
+        raise HTTPException(503, "site/viewer.html is missing")
+    with open(p, encoding="utf-8") as f:
+        return HTMLResponse(f.read())
+
+
+@app.get("/api/projects")
+def projects() -> JSONResponse:
+    return JSONResponse(_cached("projects", _list_projects))
+
+
+@app.get("/api/projects/{pid}/datasets/{did}")
+def dataset(pid: str, did: str) -> JSONResponse:
+    """The table: tasks as rows, cells keyed '<model>|<harness>'. The viewer pivots which axis is the
+    column, so this is returned once and re-rendered client-side rather than fetched per view."""
+    rel = _safe_rel("projects", pid, "datasets", did, "summary.json")
+    return JSONResponse(_cached(f"ds:{pid}/{did}", lambda: _read_json(store(), rel)))
+
+
+@app.get("/api/projects/{pid}/datasets/{did}/trace")
+def trace(pid: str, did: str, path: str = Query(..., description="path relative to the dataset dir")) -> JSONResponse:
+    """One attempt. `path` comes from summary.json, which this app did not write, so it is validated
+    against escaping the data directory before being opened."""
+    rel = _safe_rel("projects", pid, "datasets", did, *PurePosixPath(path).parts)
+    return JSONResponse(_read_json(store(), rel))
+
+
+@app.get("/api/projects/{pid}/runs/{rid}")
+def run(pid: str, rid: str) -> JSONResponse:
+    return JSONResponse(
+        _cached(f"run:{pid}/{rid}", lambda: _read_json(store(), _safe_rel("projects", pid, "runs", rid, "run.json")))
+    )
+
+
+@app.get("/api/projects/{pid}/runs/{rid}/train")
+def run_train(pid: str, rid: str) -> JSONResponse:
+    """metrics.jsonl -> list. A malformed final line is skipped rather than fatal: a run being appended
+    to right now can have a half-written line, and refusing the file would make live runs unviewable."""
+
+    def produce():
+        st = store()
+        rel = _safe_rel("projects", pid, "runs", rid, "train", "metrics.jsonl")
+        if not st.exists(rel):
+            raise HTTPException(404, f"missing: {st.describe()}/{rel}")
+        rows, skipped = [], 0
+        for line in st.read_text(rel).splitlines():
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                rows.append(json.loads(line))
+            except json.JSONDecodeError:
+                skipped += 1
+        return {"steps": rows, "skipped_lines": skipped}
+
+    return JSONResponse(_cached(f"train:{pid}/{rid}", produce))
+
+
+@app.post("/api/refresh")
+def refresh() -> JSONResponse:
+    """Drop every cache and rescan. A mounted bucket already reflects new writes; this is what makes
+    the UI notice them without a restart."""
+    _cache.clear()
+    _cache_at.clear()
+    listing = _list_projects()
+    return JSONResponse({
+        "refreshed": True,
+        "source": store().describe(),
+        "projects": len(listing),
+        "datasets": sum(len(p.get("datasets", [])) for p in listing),
+        "runs": sum(len(p.get("runs", [])) for p in listing),
+    })
+
+
+@app.get("/healthz")
+def healthz() -> dict:
+    st = store()
+    return {
+        "ok": os.path.exists(os.path.join(SITE, "viewer.html")),
+        "source": st.describe(),
+        "source_kind": st.kind,
+        "data_dir_exists": os.path.isdir(DATA_DIR),
+        "bucket_fallback": DATA_BUCKET or None,
+        "n_projects": len(_list_projects()),
+        "refresh_ttl_s": REFRESH_TTL,
+    }

requirements.txt

@@ -0,0 +1,5 @@
+fastapi>=0.115
+uvicorn[standard]>=0.30
+# Only needed for the DATA_BUCKET fallback (reading a bucket without mounting it). A mounted bucket is
+# an ordinary directory, so the Space itself does not import this.
+huggingface_hub>=1.0

site/viewer.html

@@ -0,0 +1,607 @@
+<!doctype html>
+<meta charset="utf-8">
+<title>Harbor runs</title>
+<style>
+:root{
+  --bg:#0d1117;--panel:#161b22;--panel2:#0e1116;--border:#30363d;--text:#e6edf3;
+  --muted:#8b949e;--gray:#6e7681;--accent:#58a6ff;--pass:#3ec97a;--fail:#e25c5c;--warn:#e6b455;
+}
+*{box-sizing:border-box}
+body{margin:0;background:var(--bg);color:var(--text);font:13px/1.5 ui-sans-serif,system-ui,sans-serif}
+/* ── top bar ─────────────────────────────────────────────────────────── */
+header{display:flex;align-items:center;gap:8px;padding:8px 14px;border-bottom:1px solid var(--border);
+       background:var(--panel);position:sticky;top:0;z-index:30}
+header h1{font-size:13px;margin:0;font-weight:600;letter-spacing:.2px}
+select,button{background:var(--panel2);color:var(--text);border:1px solid var(--border);border-radius:5px;
+              padding:4px 8px;font:inherit;font-size:12px;cursor:pointer}
+select:hover,button:hover{border-color:var(--accent)}
+.view-btn.on{background:rgba(89,166,255,.14);border-color:var(--accent);color:var(--accent)}
+.src{font:10.5px ui-monospace,monospace;color:var(--gray)}
+.pill{font-size:10px;padding:1px 6px;border-radius:3px;border:1px solid var(--border);white-space:nowrap}
+.pill.ok{background:rgba(62,201,122,.15);color:var(--pass);border-color:transparent}
+.pill.exp{background:rgba(230,180,85,.15);color:var(--warn);border-color:transparent}
+/* ── layout ──────────────────────────────────────────────────────────── */
+main{display:grid;grid-template-columns:184px minmax(0,1fr);height:calc(100vh - 41px)}
+.statwrap{border-bottom:1px solid var(--border)}
+.statwrap>summary{cursor:pointer;padding:7px 14px;font-size:11.5px;color:var(--muted);list-style:none}
+.statwrap>summary::-webkit-details-marker{display:none}
+.statwrap>summary::before{content:"▸ ";color:var(--gray)}
+.statwrap[open]>summary::before{content:"▾ "}
+#burger{display:none}
+/* Laptop-first: below 1100px the sidebar narrows, below 820px it becomes an overlay so the grid keeps
+   the full width — a data-dense table is the thing that must not be squeezed. */
+@media(max-width:1100px){
+  main{grid-template-columns:150px minmax(0,1fr)}
+  td.cell{width:52px}
+  table.heatmap th.col-task,table.heatmap td.col-task{min-width:210px;max-width:210px}
+}
+@media(max-width:820px){
+  #burger{display:inline-block}
+  main{grid-template-columns:minmax(0,1fr)}
+  aside{position:fixed;left:0;top:41px;bottom:0;width:210px;z-index:25;transform:translateX(-100%);
+        transition:transform .15s;box-shadow:2px 0 12px rgba(0,0,0,.4)}
+  body.nav-open aside{transform:none}
+  .matrices{flex-direction:column}
+  table.heatmap th.col-task,table.heatmap td.col-task{min-width:150px;max-width:150px}
+}
+aside{border-right:1px solid var(--border);overflow:auto;background:var(--panel2)}
+.pgroup{padding:7px 10px 3px;font-size:10px;text-transform:uppercase;letter-spacing:.6px;color:var(--gray)}
+.kid{padding:5px 10px 5px 16px;font-size:12px;color:var(--muted);cursor:pointer;border-left:2px solid transparent}
+.kid:hover{background:var(--panel);color:var(--text)}
+.kid.sel{background:rgba(89,166,255,.10);border-left-color:var(--accent);color:var(--text);font-weight:600}
+section{overflow:auto;padding:0}
+/* ── stats ───────────────────────────────────────────────────────────── */
+.stats{display:grid;grid-template-columns:repeat(auto-fit,minmax(110px,1fr));gap:8px;padding:12px 14px 4px}
+.stat{background:var(--panel);border:1px solid var(--border);border-radius:6px;padding:8px 12px}
+.stat .num{font-size:17px;font-weight:600}
+.stat .lbl{font-size:11px;color:var(--muted);margin-top:2px}
+/* ── aggregate matrices ──────────────────────────────────────────────── */
+.matrices{display:flex;flex-wrap:wrap;gap:10px;padding:10px 14px}
+.matrix-card{background:var(--panel2);border:1px solid var(--border);border-radius:6px;padding:11px 13px}
+.matrix-card h4{margin:0 0 9px;font-size:12px;color:var(--muted);font-weight:500;display:flex;gap:8px}
+.matrix-card h4 .sub{color:var(--gray);font-size:10.5px}
+.matrix-card table{border-collapse:collapse;font-size:12px}
+.matrix-card th,.matrix-card td{padding:5px 9px;border:1px solid var(--border);text-align:center;white-space:nowrap}
+.matrix-card th{background:var(--panel);color:var(--muted);font-weight:500}
+.matrix-card td.lbl{background:var(--panel);text-align:left;font-size:11.5px}
+.matrix-card td.cell b{font-size:13px}
+.matrix-card td.cell .det{color:rgba(255,255,255,.55);font-size:10px}
+.matrix-card td.empty{color:var(--gray)}
+/* ── the heatmap ───────────────────────────���─────────────────────────── */
+.hwrap{overflow:auto;padding:4px 14px 24px}
+table.heatmap{border-collapse:separate;border-spacing:0;font-size:12px}
+table.heatmap th{background:var(--panel);color:var(--muted);font-weight:500;font-size:11px;
+                 padding:5px 8px;border-bottom:1px solid var(--border);position:sticky;top:0;z-index:6;white-space:nowrap}
+table.heatmap td{border-bottom:1px solid var(--border);padding:0}
+table.heatmap th.col-task,table.heatmap td.col-task{position:sticky;left:0;z-index:5;background:var(--panel2);
+       min-width:280px;max-width:280px;padding:5px 9px;text-align:left;border-right:2px solid var(--border)}
+table.heatmap thead th.col-task{z-index:8;background:var(--panel)}
+td.col-task .tid{font:11px ui-monospace,monospace;color:var(--muted)}
+td.col-task .q{color:var(--text);font-size:11.5px;display:-webkit-box;-webkit-line-clamp:2;
+               -webkit-box-orient:vertical;overflow:hidden}
+.diff{font-size:9.5px;text-transform:uppercase;font-weight:600;padding:1px 5px;border-radius:3px;margin-right:5px}
+.diff.easy{background:rgba(62,201,122,.15);color:var(--pass)}
+.diff.medium{background:rgba(230,180,85,.15);color:var(--warn)}
+.diff.hard{background:rgba(226,92,92,.15);color:var(--fail)}
+td.cell{width:62px;height:26px;cursor:pointer;text-align:center;font:11px ui-monospace,monospace;
+        transition:filter .1s;border-left:1px solid var(--border)}
+td.cell:hover{filter:brightness(1.45);outline:1px solid var(--accent);outline-offset:-1px}
+/* pass@k gradient — darker green = passed on an earlier attempt */
+td.cell.pass-1{background:rgba(62,201,122,.55);color:#fff;font-weight:600}
+td.cell.pass-2{background:rgba(62,201,122,.40);color:#fff;font-weight:600}
+td.cell.pass-3{background:rgba(62,201,122,.28);color:#fff;font-weight:600}
+td.cell.pass-4{background:rgba(62,201,122,.18);color:#fff;font-weight:600}
+td.cell.fail{background:rgba(226,92,92,.25);color:rgba(255,255,255,.85)}
+td.cell.na{background:rgba(230,180,85,.16);color:var(--warn)}
+td.cell.empty{background:transparent;color:var(--gray);cursor:default}
+td.cell.empty:hover{filter:none;outline:0}
+.filters{display:flex;flex-wrap:wrap;gap:6px;align-items:center;padding:8px 14px 2px;border-bottom:1px solid var(--border)}
+.fchip{font-size:11px;padding:2px 7px;border-radius:11px;border:1px solid var(--border);background:var(--panel2);
+       color:var(--muted);cursor:pointer;white-space:nowrap}
+.fchip:hover{border-color:var(--accent);color:var(--text)}
+.fchip.on{background:rgba(89,166,255,.16);border-color:var(--accent);color:var(--accent)}
+.fkey{font-size:10px;text-transform:uppercase;letter-spacing:.5px;color:var(--gray);margin-left:6px}
+.kid input{margin-right:6px;vertical-align:-1px}
+.ctrls{display:flex;flex-wrap:wrap;gap:10px;align-items:center;padding:8px 14px;border-bottom:1px solid var(--border)}
+.ctrls label{font-size:11px;color:var(--muted);display:flex;align-items:center;gap:5px}
+input[type=range]{width:120px;accent-color:var(--accent)}
+.chartgrid{display:grid;grid-template-columns:repeat(auto-fill,minmax(320px,1fr));gap:8px;padding:8px 14px 24px}
+.chartbox{background:var(--panel2);border:1px solid var(--border);border-radius:6px;padding:6px 8px 2px}
+.chartbox h5{margin:0 0 2px;font-size:11px;color:var(--muted);font-weight:500;display:flex;justify-content:space-between}
+.chartbox h5 span{color:var(--gray);font-size:10px}
+.rlegend{display:flex;flex-wrap:wrap;gap:10px;padding:4px 14px;font-size:11px}
+.rlegend i{display:inline-block;width:16px;height:3px;border-radius:2px;margin-right:4px;vertical-align:2px}
+.taskcard{background:var(--panel2);border:1px solid var(--border);border-radius:6px;padding:10px 12px;margin-bottom:10px}
+.metagrid{display:grid;grid-template-columns:repeat(auto-fit,minmax(150px,1fr));gap:6px;margin-top:8px}
+.metagrid div{background:var(--panel);border:1px solid var(--border);border-radius:4px;padding:4px 8px;font-size:11px}
+.metagrid em{display:block;font-style:normal;color:var(--gray);font-size:10px}
+.legend{display:flex;gap:12px;align-items:center;padding:6px 14px;font-size:10.5px;color:var(--muted)}
+.legend i{display:inline-block;width:14px;height:11px;border-radius:2px;margin-right:3px;vertical-align:middle}
+/* ── trace dialog ────────────────────────────────────────────────────── */
+dialog{background:var(--bg);color:var(--text);border:1px solid var(--border);border-radius:9px;
+       width:min(1020px,95vw);max-height:90vh;padding:0}
+dialog::backdrop{background:rgba(0,0,0,.65)}
+.dhead{display:flex;align-items:center;gap:9px;padding:10px 14px;border-bottom:1px solid var(--border);
+       background:var(--panel);position:sticky;top:0}
+.dbody{padding:12px 14px;overflow:auto;max-height:calc(90vh - 48px)}
+.trial-table{width:100%;border-collapse:collapse;font-size:12px;margin:8px 0}
+.trial-table th,.trial-table td{padding:5px 8px;border-bottom:1px solid var(--border);text-align:left}
+.trial-table .reward.pass{color:var(--pass);font-weight:600}
+.trial-table .reward.fail{color:var(--fail)}
+.trial-table .reward.na{color:var(--warn)}
+.msg{border:1px solid var(--border);border-radius:6px;margin-bottom:6px;overflow:hidden}
+.msg-head{background:var(--panel);padding:3px 9px;font:10.5px ui-monospace,monospace;color:var(--muted)}
+.msg-body{margin:0;padding:7px 10px;white-space:pre-wrap;word-break:break-word;
+          font:11px/1.5 ui-monospace,monospace;max-height:280px;overflow:auto}
+.gold{color:var(--pass)}
+.empty-msg{color:var(--gray);padding:34px;text-align:center}
+svg{background:var(--panel2);border:1px solid var(--border);border-radius:6px}
+</style>
+<header>
+  <button id="burger" title="Show/hide runs">☰</button>
+  <select id="proj"></select>
+  <button id="refresh" title="Re-read the bucket / data directory">⟳</button>
+  <span style="flex:1"></span>
+  <span class="src">cols</span>
+  <button id="axH" class="view-btn on">harness</button>
+  <button id="axM" class="view-btn">model</button>
+  <span class="src" id="src"></span>
+</header>
+<main>
+  <aside id="tree"></aside>
+  <section id="body"><div class="empty-msg">select a dataset</div></section>
+</main>
+<dialog id="dlg">
+  <div class="dhead"><b id="dtitle"></b><span id="dpills"></span><span style="flex:1"></span>
+    <button onclick="document.getElementById('dlg').close()">esc</button></div>
+  <div class="dbody" id="dbody"></div>
+</dialog>
+<script>
+const $=s=>document.querySelector(s);
+let TREE=[],AXIS="harness",DS=null,SEL=null,KMAX=4;
+// Active metadata filters: {key: Set(values)}. Metadata is whatever the suite recorded, so nothing is
+// hardcoded — difficulty, package_tier and reward_mode are exactly the fields that differ between
+// suites, and promoting any of them to a column breaks the viewer when a suite renames one.
+let FILT={};
+// Several training runs can be on screen at once, so selection is a set and the charts overlay.
+// Comparing curves across runs is the whole reason to look at them side by side.
+let RUNSEL=new Set(), RUNCACHE={}, SMOOTH=0.6, LOGX=false;
+const RUNCOLORS=["#58a6ff","#3ec97a","#e6b455","#e25c5c","#bc8cff","#39c5cf","#f778ba","#a5d6ff"];
+const api=async(p,o)=>{const r=await fetch(p,o);if(!r.ok)throw new Error(`${p}: ${r.status}`);return r.json()};
+const n=(v,d=3)=>v==null?"–":(typeof v==="number"?(Number.isInteger(v)?v:v.toFixed(d)):v);
+const pct=v=>v==null?"–":(v*100).toFixed(0)+"%";
+const short=m=>String(m).split("/").pop();
+const esc=s=>String(s??"").replace(/[<>&"]/g,c=>({"<":"&lt;",">":"&gt;","&":"&amp;",'"':"&quot;"}[c]));
+
+// Heat for the aggregate matrices: one hue, alpha carries the value. Two hues would imply a midpoint
+// that pass-rate does not have.
+const heatBg=p=>p==null?"transparent":`rgba(62,201,122,${(0.08+0.5*p).toFixed(3)})`;
+// Only fields that can actually partition the rows are offered: a key with one value filters nothing,
+// and one with a value per task (an id, a free-text question) is a list, not a filter.
+function metaFacets(tasks){
+  const counts={};
+  tasks.forEach(t=>Object.entries(t.meta||{}).forEach(([k,v])=>{
+    if(v==null||typeof v==="object")return;
+    counts[k]=counts[k]||{};
+    counts[k][String(v)]=(counts[k][String(v)]||0)+1;
+  }));
+  const out={};
+  for(const [k,vals] of Object.entries(counts)){
+    const nv=Object.keys(vals).length;
+    if(nv>1 && nv<=12 && nv<tasks.length) out[k]=vals;
+  }
+  return out;
+}
+function passesFilters(t){
+  return Object.entries(FILT).every(([k,set])=>
+    !set.size || set.has(String((t.meta||{})[k])));
+}
+const heatOf=(pass,measured)=>measured?pass/measured:null;
+
+async function loadTree(){
+  TREE=await api("/api/projects");
+  const h=await api("/healthz");
+  $("#src").textContent=h.source;
+  $("#proj").innerHTML=TREE.map(p=>`<option value="${p.project_id}">${esc(p.label||p.project_id)}</option>`).join("");
+  renderRuns();
+  const f=document.querySelector(".kid"); if(f&&!SEL) f.click();
+}
+
+// One list, in the sidebar, scoped to the selected project. Two dropdowns made the project choice and
+// the run choice look like peers when the second is nested inside the first.
+function syncBoxes(){
+  document.querySelectorAll(".kid input[data-run]").forEach(cb=>cb.checked=RUNSEL.has(cb.dataset.run));
+}
+
+function renderRuns(){
+  const p=TREE.find(x=>x.project_id===$("#proj").value)||TREE[0]||{};
+  const ds=(p.datasets||[]), rs=(p.runs||[]);
+  $("#tree").innerHTML=`<div class="pgroup">Runs</div>
+    ${ds.length?ds.map(d=>`<div class="kid" data-p="${p.project_id}" data-d="${d}"
+        title="${esc(d)}">▦ ${esc(d)}</div>`).join(""):""}
+    ${rs.length?`<div class="pgroup">Training</div>`+rs.map(r=>`<div class="kid" data-p="${p.project_id}" data-r="${r}"
+        title="${esc(r)}"><input type="checkbox" data-run="${esc(r)}"${RUNSEL.has(r)?" checked":""}>${esc(r)}</div>`).join(""):""}
+    ${!ds.length&&!rs.length?`<div class="empty-msg" style="padding:18px;font-size:11px">nothing here yet</div>`:""}`;
+  document.querySelectorAll(".kid").forEach(el=>el.onclick=(ev)=>{
+    if(ev.target.tagName==="INPUT")return;   // the checkbox drives comparison, not navigation
+    document.querySelectorAll(".kid").forEach(k=>k.classList.remove("sel"));el.classList.add("sel");
+    if(window.matchMedia("(max-width:820px)").matches) document.body.classList.remove("nav-open");
+    if(el.dataset.d){RUNSEL.clear();openDataset(el.dataset.p,el.dataset.d)}
+    else{RUNSEL.clear();RUNSEL.add(el.dataset.r);syncBoxes();openRuns(el.dataset.p)}
+  });
+  document.querySelectorAll(".kid input[data-run]").forEach(cb=>cb.onchange=()=>{
+    cb.checked?RUNSEL.add(cb.dataset.run):RUNSEL.delete(cb.dataset.run);
+    if(RUNSEL.size) openRuns($("#proj").value);
+  });
+}
+$("#proj").onchange=()=>{renderRuns();const f=document.querySelector(".kid");if(f)f.click()};
+$("#burger").onclick=()=>document.body.classList.toggle("nav-open");
+
+async function openDataset(pid,did){
+  SEL={pid,did};$("#body").innerHTML=`<div class="empty-msg">loading ${did}…</div>`;
+  try{DS=await api(`/api/projects/${pid}/datasets/${did}`)}
+  catch(e){$("#body").innerHTML=`<div class="empty-msg">${e.message}</div>`;return}
+  KMAX=DS.k_max||4; $("#proj").value=pid;
+  renderGrid();
+}
+
+function renderGrid(){
+  const d=DS, models=d.models||[], harnesses=d.harnesses||[];
+  const cols = AXIS==="harness"?harnesses:models;
+  const subs = AXIS==="harness"?models:harnesses;
+  const keyFor=(c,s)=>AXIS==="harness"?`${s}|${c}`:`${c}|${s}`;
+  const s=d.summary||{};
+
+  const facets=metaFacets(d.tasks||[]);
+  const shown=(d.tasks||[]).filter(passesFilters);
+  const filterBar=Object.keys(facets).length?`<div class="filters">
+    ${Object.entries(facets).map(([k,vals])=>`<span class="fkey">${esc(k)}</span>` +
+      Object.entries(vals).sort((a,b)=>a[0].localeCompare(b[0],undefined,{numeric:true})).map(([v,c])=>
+        `<span class="fchip ${(FILT[k]&&FILT[k].has(v))?"on":""}" data-fk="${esc(k)}" data-fv="${esc(v)}"
+           >${esc(v)} <span class="src">${c}</span></span>`).join("")).join("")}
+    ${Object.values(FILT).some(x=>x.size)?`<span class="fchip" id="fclear">clear</span>`:""}
+    <span class="src">${shown.length}/${(d.tasks||[]).length} tasks</span></div>`:"";
+
+  const stats=[["tasks",n(s.tasks_total)],["solved ≤k",n(s.tasks_any_pass)],
+    ["attempts",n(s.attempts_total)],["passed",n(s.attempts_passed)],
+    // Beside the pass rates on purpose: an all-infra task is unmeasured, not failed, and folding it in
+    // as a zero would read as a real (bad) measurement.
+    ["unmeasured",n(s.n_all_infra)],["k",KMAX]]
+    .map(([l,v])=>`<div class="stat"><div class="num">${v}</div><div class="lbl">${l}</div></div>`).join("");
+
+  // aggregate matrix: model × harness, cells = pass@k
+  const agg={};
+  shown.forEach(t=>Object.entries(t.cells||{}).forEach(([k,c])=>{
+    const [m,h]=k.split("|"); agg[m]=agg[m]||{}; agg[m][h]=agg[m][h]||{cells:0,pass:0,measured:0};
+    const a=agg[m][h]; a.cells++;
+    const graded=(c.attempts||[]).filter(x=>x.reward!=null);
+    if(graded.length){a.measured++; if(c.passed_at)a.pass++}
+  }));
+  const mxh=`<table><thead><tr><th></th>${harnesses.map(h=>`<th>${esc(h)}</th>`).join("")}</tr></thead><tbody>
+    ${models.map(m=>`<tr><td class="lbl">${esc(short(m))}</td>${harnesses.map(h=>{
+      const a=(agg[m]||{})[h];
+      if(!a||!a.measured) return `<td class="empty">—</td>`;
+      const p=a.pass/a.measured;
+      return `<td class="cell" style="background:${heatBg(p)}"><b>${pct(p)}</b><div class="det">${a.pass}/${a.measured}</div></td>`;
+    }).join("")}</tr>`).join("")}</tbody></table>`;
+
+  // Second matrix pivots on whichever metadata field has the most values — chosen from the data
+  // rather than hardcoded to difficulty, because that field is not guaranteed to exist.
+  const facetKey=Object.entries(facets).sort((a,b)=>Object.keys(b[1]).length-Object.keys(a[1]).length)[0]?.[0];
+  let mxd="";
+  if(facetKey){
+    const vals=[...new Set(shown.map(t=>String((t.meta||{})[facetKey])).filter(v=>v!=="undefined"))]
+      .sort((a,b)=>a.localeCompare(b,undefined,{numeric:true}));
+    const acc={};
+    shown.forEach(t=>Object.entries(t.cells||{}).forEach(([k,c])=>{
+      const h=k.split("|")[1], v=String((t.meta||{})[facetKey]); if(v==="undefined")return;
+      acc[h]=acc[h]||{}; acc[h][v]=acc[h][v]||{measured:0,pass:0};
+      if((c.attempts||[]).some(x=>x.reward!=null)){acc[h][v].measured++; if(c.passed_at)acc[h][v].pass++}
+    }));
+    mxd=`<table><thead><tr><th></th>${vals.map(v=>`<th>${esc(v)}</th>`).join("")}</tr></thead><tbody>
+      ${harnesses.map(h=>`<tr><td class="lbl">${esc(h)}</td>${vals.map(v=>{
+        const a=(acc[h]||{})[v];
+        if(!a||!a.measured)return `<td class="empty">—</td>`;
+        const pr=heatOf(a.pass,a.measured);
+        return `<td class="cell" style="background:${heatBg(pr)}"><b>${pct(pr)}</b><div class="det">${a.pass}/${a.measured}</div></td>`;
+      }).join("")}</tr>`).join("")}</tbody></table>`;
+  }
+
+  // the per-task heatmap
+  const head=`<tr><th class="col-task">task</th>${cols.flatMap(c=>
+    (subs.length?subs:[null]).map(sb=>`<th title="${esc(c)}${sb?" · "+esc(sb):""}">${esc(subs.length>1?short(sb):c)}</th>`)).join("")}</tr>`
+    + (subs.length>1?`<tr><th class="col-task"></th>${cols.flatMap(c=>subs.map(()=>`<th>${esc(c)}</th>`)).join("")}</tr>`:"");
+  const rows=shown.map(t=>{
+    const tds=cols.flatMap(c=>(subs.length?subs:[null]).map(sb=>{
+      const cell=(t.cells||{})[keyFor(c,sb)];
+      if(!cell) return `<td class="cell empty">—</td>`;
+      const graded=(cell.attempts||[]).filter(x=>x.reward!=null);
+      // pass-N shades by WHICH attempt passed: darker = solved sooner, so the grid shows reliability
+      // rather than a flat pass/fail.
+      const cls=cell.passed_at?`pass-${Math.min(cell.passed_at,4)}`:graded.length?"fail":"na";
+      const mark=cell.passed_at?`✓${cell.passed_at}`:graded.length?"✗":"·";
+      return `<td class="cell ${cls}" data-t="${esc(t.id)}" data-k="${esc(keyFor(c,sb))}"
+               title="${graded.length}/${(cell.attempts||[]).length} graded">${mark}</td>`;
+    })).join("");
+    // Whichever facet is active labels the row, so the pill reflects the data instead of assuming a
+    // difficulty scale exists.
+    const lbl=facetKey?String((t.meta||{})[facetKey]??""):"";
+    return `<tr><td class="col-task" data-task="${esc(t.id)}">
+        <div>${lbl?`<span class="diff ${diffLike(facetKey,lbl)}">${esc(lbl)}</span>`:""}<span class="tid">${esc(t.id)}</span></div>
+        <div class="q">${esc((t.question||"").slice(0,150))}</div></td>${tds}</tr>`;
+  }).join("");
+
+  // Collapsed by default: the grid is what you came for, and six counters above it push it below the
+  // fold on a laptop.
+  const statSummary=`${n(s.tasks_total)} tasks · ${n(s.tasks_any_pass)} solved ≤k · ${n(s.attempts_total)} attempts`
+    + (s.n_all_infra?` · ${n(s.n_all_infra)} unmeasured`:"");
+  $("#body").innerHTML=`${filterBar}
+    <details class="statwrap"><summary>${statSummary}</summary><div class="stats">${stats}</div></details>
+    <div class="matrices">
+      <div class="matrix-card"><h4>model × harness <span class="sub">pass@${KMAX}</span></h4>${mxh}</div>
+      ${mxd?`<div class="matrix-card"><h4>harness × ${esc(facetKey)} <span class="sub">pass@${KMAX}</span></h4>${mxd}</div>`:""}
+    </div>
+    <div class="legend">
+      <span><i style="background:rgba(62,201,122,.55)"></i>passed 1st</span>
+      <span><i style="background:rgba(62,201,122,.18)"></i>passed 4th</span>
+      <span><i style="background:rgba(226,92,92,.25)"></i>graded, failed</span>
+      <span><i style="background:rgba(230,180,85,.16)"></i>unmeasured (infra)</span>
+      <span class="src">click a cell for attempts + trace</span>
+    </div>
+    <div class="hwrap"><table class="heatmap"><thead>${head}</thead><tbody>${rows}</tbody></table></div>`;
+  document.querySelectorAll("td.cell[data-t]").forEach(td=>td.onclick=()=>openCell(td.dataset.t,td.dataset.k));
+  document.querySelectorAll("td.col-task[data-task]").forEach(td=>td.onclick=()=>openTask(td.dataset.task));
+  document.querySelectorAll(".fchip[data-fk]").forEach(el=>el.onclick=()=>{
+    const k=el.dataset.fk,v=el.dataset.fv;
+    FILT[k]=FILT[k]||new Set();
+    FILT[k].has(v)?FILT[k].delete(v):FILT[k].add(v);
+    renderGrid();
+  });
+  const fc=$("#fclear"); if(fc) fc.onclick=()=>{FILT={};renderGrid()};
+}
+
+function diffLike(key,val){
+  // Colour a facet value only when it plausibly orders low->high; otherwise leave it neutral rather
+  // than implying a ranking the field may not have.
+  const num=Number(val);
+  if(!/level|difficulty|tier|rank/i.test(key||"")||Number.isNaN(num))return "";
+  return num<=0?"easy":num<=2?"medium":"hard";
+}
+
+function openTask(taskId){
+  const t=(DS.tasks||[]).find(x=>String(x.id)===String(taskId))||{};
+  const cells=Object.entries(t.cells||{});
+  $("#dtitle").textContent=t.id||taskId;
+  $("#dpills").innerHTML=`<span class="pill">${cells.length} cell(s)</span>`;
+  const meta=Object.entries(t.meta||{}).map(([k,v])=>
+    `<div><em>${esc(k)}</em>${esc(typeof v==="object"?JSON.stringify(v):v)}</div>`).join("");
+  const perCell=cells.map(([k,c])=>{
+    const [m,h]=k.split("|");
+    const graded=(c.attempts||[]).filter(a=>a.reward!=null);
+    return `<tr><td>${esc(h)}</td><td>${esc(short(m))}</td>
+      <td class="reward ${c.passed_at?"pass":graded.length?"fail":"na"}">
+        ${c.passed_at?`✓ attempt ${c.passed_at}`:graded.length?"✗ failed":"unmeasured"}</td>
+      <td>${graded.length}/${(c.attempts||[]).length}</td>
+      <td>${c.attempts?.[0]?.n_turns??"–"}</td>
+      <td><button data-cell="${esc(k)}">open</button></td></tr>`;
+  }).join("");
+  $("#dbody").innerHTML=`
+    <div class="taskcard"><div class="msg-head" style="background:none;padding:0 0 4px">question</div>
+      <pre class="msg-body" style="padding:0">${esc(t.question||"(none recorded)")}</pre></div>
+    ${t.answer!=null?`<div class="taskcard"><div class="msg-head" style="background:none;padding:0 0 4px">gold answer</div>
+      <pre class="msg-body gold" style="padding:0">${esc(t.answer)}</pre></div>`:""}
+    ${meta?`<div class="taskcard"><div class="msg-head" style="background:none;padding:0">metadata</div>
+      <div class="metagrid">${meta}</div></div>`:""}
+    <table class="trial-table"><thead><tr><th>harness</th><th>model</th><th>result</th><th>graded</th><th>turns</th><th></th></tr></thead>
+      <tbody>${perCell}</tbody></table>`;
+  $("#dlg").showModal();
+  document.querySelectorAll("#dbody button[data-cell]").forEach(b=>
+    b.onclick=()=>openCell(taskId,b.dataset.cell));
+}
+
+async function openCell(taskId,cellKey){
+  const t=(DS.tasks||[]).find(x=>String(x.id)===String(taskId))||{};
+  const cell=(t.cells||{})[cellKey]||{};
+  const [model,harness]=cellKey.split("|");
+  const graded=(cell.attempts||[]).filter(a=>a.reward!=null);
+  const pass1=graded.length?graded.filter(a=>a.reward>0).length/graded.length:null;
+  $("#dtitle").textContent=`${taskId}`;
+  $("#dpills").innerHTML=`<span class="pill">${esc(harness)}</span> <span class="pill">${esc(short(model))}</span>
+    <span class="pill ${cell.passed_at?"ok":"exp"}">${cell.passed_at?`pass@${KMAX} ✓ (attempt ${cell.passed_at})`:graded.length?"failed":"unmeasured"}</span>`;
+  $("#dbody").innerHTML=`
+    <div class="stats" style="padding:0 0 8px">
+      <div class="stat"><div class="num">${pass1==null?"–":pct(pass1)}</div><div class="lbl">pass@1</div></div>
+      <div class="stat"><div class="num">${graded.length}/${(cell.attempts||[]).length}</div><div class="lbl">graded</div></div>
+      <div class="stat"><div class="num">${cell.passed_at??"–"}</div><div class="lbl">first pass</div></div>
+    </div>
+    ${t.question?`<div class="msg"><div class="msg-head">question</div><pre class="msg-body">${esc(t.question)}</pre></div>`:""}
+    ${t.answer!=null?`<div class="msg"><div class="msg-head">gold</div><pre class="msg-body gold">${esc(t.answer)}</pre></div>`:""}
+    <table class="trial-table"><thead><tr><th>#</th><th>reward</th><th>turns</th><th>elapsed</th><th></th></tr></thead>
+      <tbody>${(cell.attempts||[]).map(a=>`<tr>
+        <td>${a.attempt}</td>
+        <td class="reward ${a.reward==null?"na":a.reward>0?"pass":"fail"}">${a.reward==null?"unmeasured":n(a.reward,2)}</td>
+        <td>${n(a.n_turns)}</td><td>${a.elapsed_sec?n(a.elapsed_sec,1)+"s":"–"}</td>
+        <td>${a.trace?`<button data-tr="${esc(a.trace)}">trace</button>`:`<span class="src">no trace recorded</span>`}</td>
+      </tr>`).join("")}</tbody></table>
+    <div id="tr"></div>`;
+  $("#dlg").showModal();
+  document.querySelectorAll("#dbody button[data-tr]").forEach(b=>b.onclick=()=>showTrace(b.dataset.tr));
+}
+
+async function showTrace(path){
+  $("#tr").innerHTML=`<div class="empty-msg">loading…</div>`;
+  let tr;
+  try{tr=await api(`/api/projects/${SEL.pid}/datasets/${SEL.did}/trace?path=${encodeURIComponent(path)}`)}
+  catch(e){$("#tr").innerHTML=`<div class="empty-msg">${e.message}</div>`;return}
+  // reward_method is surfaced because the grader is exact -> numeric -> LLM judge: a 0 from the judge is
+  // a wrong answer, a 0 from exact match may only be a formatting mismatch.
+  const meta=["reward","reward_method","n_turns","elapsed_sec","gold","predicted"].filter(k=>tr[k]!==undefined)
+    .map(k=>`<div class="stat"><div class="num">${esc(n(tr[k],2))}</div><div class="lbl">${k}</div></div>`).join("");
+  const msgs=(tr.messages||[]).map(m=>{
+    const calls=(m.tool_calls||[]).map(c=>JSON.stringify(c)).join("\n");
+    const body=(typeof m.content==="string"?m.content:JSON.stringify(m.content,null,2))||"";
+    return `<div class="msg"><div class="msg-head">${esc(m.role)}${calls?" · tool_calls":""}</div>
+      <pre class="msg-body">${esc(body)}${calls?"\n\n"+esc(calls):""}</pre></div>`;
+  }).join("");
+  $("#tr").innerHTML=`<div class="stats" style="padding:10px 0 6px">${meta}</div>
+    ${msgs||`<pre class="msg-body">${esc(JSON.stringify(tr,null,2).slice(0,4000))}</pre>`}`;
+}
+
+async function openRuns(pid){
+  const ids=[...RUNSEL];
+  if(!ids.length){$("#body").innerHTML=`<div class="empty-msg">select a training run</div>`;return}
+  SEL={pid,rid:ids[0],runs:ids};
+  $("#body").innerHTML=`<div class="empty-msg">loading ${ids.length} run(s)…</div>`;
+
+  const loaded=[];
+  for(const rid of ids){
+    const key=`${pid}/${rid}`;
+    if(!RUNCACHE[key]){
+      let meta={},d={steps:[]};
+      try{meta=await api(`/api/projects/${pid}/runs/${rid}`)}catch(e){}
+      try{d=await api(`/api/projects/${pid}/runs/${rid}/train`)}
+      catch(e){RUNCACHE[key]={rid,meta,steps:[],error:e.message};loaded.push(RUNCACHE[key]);continue}
+      RUNCACHE[key]={rid,meta,steps:d.steps||[],skipped:d.skipped_lines,rollouts:d.rollout_steps||[]};
+    }
+    loaded.push(RUNCACHE[key]);
+  }
+  renderTrain(loaded);
+}
+
+function renderTrain(runs){
+  // Chart every numeric series the runs actually logged, rather than a fixed list: a trainer that adds
+  // a metric should show it without a code change here, the same reason task metadata is discovered.
+  const keys=[...new Set(runs.flatMap(r=>r.steps.flatMap(s=>
+    Object.entries(s).filter(([k,v])=>typeof v==="number"&&k!=="step").map(([k])=>k))))];
+  const preferred=["reward","reward_std","loss","ratio","kl","entropy","learning_rate"];
+  keys.sort((a,b)=>{
+    const ia=preferred.indexOf(a),ib=preferred.indexOf(b);
+    return (ia<0?99:ia)-(ib<0?99:ib) || a.localeCompare(b);
+  });
+
+  const summary=runs.map((r,i)=>{
+    const g=r.steps.filter(s=>(s.reward_std||0)>0).length;
+    return `<div class="stat" style="border-left:3px solid ${RUNCOLORS[i%RUNCOLORS.length]}">
+      <div class="num">${r.steps.length}<span class="src"> steps</span></div>
+      <div class="lbl">${esc(r.rid)}</div>
+      <div class="lbl">${g}/${r.steps.length} with a gradient${r.error?" · "+esc(r.error):""}</div></div>`;
+  }).join("");
+
+  const legend=runs.map((r,i)=>
+    `<span><i style="background:${RUNCOLORS[i%RUNCOLORS.length]}"></i>${esc(r.rid)}</span>`).join("");
+
+  const charts=keys.map(k=>{
+    const series=runs.map((r,i)=>({
+      pts:r.steps.map((s,j)=>[s.step??j,s[k]]).filter(pt=>typeof pt[1]==="number"),
+      color:RUNCOLORS[i%RUNCOLORS.length],
+    })).filter(x=>x.pts.length);
+    if(!series.length)return "";
+    return `<div class="chartbox"><h5>${esc(k)}<span>${series.map(x=>x.pts.length).join(" / ")} pts</span></h5>
+      ${multiChart(series)}</div>`;
+  }).join("");
+
+  const anyRollouts=runs.some(r=>(r.rollouts||[]).length);
+  $("#body").innerHTML=`
+    <div class="ctrls">
+      <label>smoothing <input type="range" id="sm" min="0" max="0.95" step="0.05" value="${SMOOTH}">
+        <span class="src" id="smv">${SMOOTH.toFixed(2)}</span></label>
+      <label><input type="checkbox" id="logx"${LOGX?" checked":""}> log x</label>
+      <span class="src">${runs.length} run(s) overlaid — tick more in the sidebar to compare</span>
+      ${anyRollouts?`<button id="rollbtn">rollout traces</button>`:`<span class="src">no rollout traces recorded for these runs</span>`}
+    </div>
+    <details class="statwrap" open><summary>${runs.length} run(s)</summary><div class="stats">${summary}</div></details>
+    <div class="rlegend">${legend}</div>
+    <div class="chartgrid">${charts||`<div class="empty-msg">no numeric series logged</div>`}</div>`;
+
+  $("#sm").oninput=(e)=>{SMOOTH=+e.target.value;$("#smv").textContent=SMOOTH.toFixed(2);renderTrain(runs)};
+  $("#logx").onchange=(e)=>{LOGX=e.target.checked;renderTrain(runs)};
+  const rb=$("#rollbtn"); if(rb) rb.onclick=()=>openRollouts(runs);
+}
+
+// EMA, applied left to right. Shown WITH the raw series rather than instead of it: a smoothed curve
+// hides the variance, and on a GRPO run the variance is the signal — a flat reward with high spread
+// means different things from a flat reward with none.
+function ema(pts,alpha){
+  if(alpha<=0)return pts;
+  const out=[];let acc=null;
+  for(const [x,y] of pts){acc = acc===null ? y : alpha*acc+(1-alpha)*y; out.push([x,acc])}
+  return out;
+}
+
+function multiChart(series){
+  const W=330,H=140,P=34,B=20;
+  const all=series.flatMap(s=>s.pts);
+  const xs=all.map(p=>p[0]),ys=all.map(p=>p[1]);
+  let x0=Math.min(...xs),x1=Math.max(...xs);if(x0===x1)x1=x0+1;
+  let y0=Math.min(...ys),y1=Math.max(...ys);if(y0===y1){y0-=.5;y1+=.5}
+  const pad=(y1-y0)*0.08;y0-=pad;y1+=pad;
+  const fx=v=>LOGX?Math.log10(Math.max(v,1)):v;
+  const fx0=fx(x0),fx1=Math.max(fx(x1),fx0+1e-9);
+  const sx=v=>P+(fx(v)-fx0)/(fx1-fx0)*(W-P-8);
+  const sy=v=>H-B-(v-y0)/(y1-y0)*(H-B-14);
+  const grid=[0,.5,1].map(f=>{const y=y0+(y1-y0)*f;
+    return `<line x1="${P}" x2="${W-8}" y1="${sy(y).toFixed(1)}" y2="${sy(y).toFixed(1)}" stroke="#30363d" stroke-width=".5"/>
+            <text x="2" y="${(sy(y)+3).toFixed(1)}" fill="#6e7681" font-size="8.5">${y.toFixed(2)}</text>`}).join("");
+  const paths=series.map(s=>{
+    const raw=s.pts.map((p,i)=>`${i?"L":"M"}${sx(p[0]).toFixed(1)},${sy(p[1]).toFixed(1)}`).join("");
+    const sm=ema(s.pts,SMOOTH).map((p,i)=>`${i?"L":"M"}${sx(p[0]).toFixed(1)},${sy(p[1]).toFixed(1)}`).join("");
+    return `<path d="${raw}" fill="none" stroke="${s.color}" stroke-width="1" opacity="${SMOOTH>0?0.22:1}"/>
+            ${SMOOTH>0?`<path d="${sm}" fill="none" stroke="${s.color}" stroke-width="1.8"/>`:""}`;
+  }).join("");
+  return `<svg viewBox="0 0 ${W} ${H}" style="width:100%;height:auto">${grid}${paths}
+    <text x="${P}" y="${H-4}" fill="#6e7681" font-size="8.5">${x0}</text>
+    <text x="${W-30}" y="${H-4}" fill="#6e7681" font-size="8.5">${x1}</text></svg>`;
+}
+
+async function openRollouts(runs){
+  const withR=runs.filter(r=>(r.rollouts||[]).length);
+  $("#dtitle").textContent="rollout traces";
+  $("#dpills").innerHTML=`<span class="pill">${withR.length} run(s)</span>`;
+  $("#dbody").innerHTML=withR.map(r=>`<div class="taskcard"><b>${esc(r.rid)}</b>
+    <table class="trial-table"><thead><tr><th>step</th><th>rollouts</th><th>rewards</th><th></th></tr></thead>
+    <tbody>${r.rollouts.map(st=>`<tr><td>${st.step}</td><td>${st.n}</td>
+      <td>${(st.rewards||[]).map(x=>x==null?"·":Number(x).toFixed(2)).join(" ")}</td>
+      <td><button data-run="${esc(r.rid)}" data-step="${st.step}">open</button></td></tr>`).join("")}
+    </tbody></table></div>`).join("") || `<div class="empty-msg">no rollout traces recorded</div>`;
+  $("#dlg").showModal();
+  document.querySelectorAll("#dbody button[data-step]").forEach(b=>b.onclick=()=>showRolloutStep(b.dataset.run,b.dataset.step));
+}
+
+async function showRolloutStep(rid,step){
+  const box=$("#dbody");
+  box.insertAdjacentHTML("afterbegin",`<div id="rs" class="empty-msg">loading step ${step}…</div>`);
+  let rows;
+  try{rows=await api(`/api/projects/${SEL.pid}/runs/${rid}/rollouts/${step}`)}
+  catch(e){$("#rs").outerHTML=`<div class="warnbox">${e.message}</div>`;return}
+  $("#rs").outerHTML=`<div class="taskcard"><b>step ${esc(step)} · ${rows.rollouts.length} rollout(s)</b>
+    ${rows.rollouts.map((ro,i)=>`
+      <details><summary class="src">#${i+1} · reward ${ro.reward==null?"unmeasured":n(ro.reward,2)}
+        · ${n(ro.n_turns)} turns · ${esc(ro.task_id||"")}</summary>
+        ${(ro.messages||[]).map(m=>`<div class="msg"><div class="msg-head">${esc(m.role)}</div>
+          <pre class="msg-body">${esc(typeof m.content==="string"?m.content:JSON.stringify(m.content,null,2))}</pre></div>`).join("")
+          || `<div class="src" style="padding:6px">no conversation recorded for this rollout</div>`}
+      </details>`).join("")}</div>`;
+}
+
+function chart(steps,key){
+  const pts=steps.map((s,i)=>[s.step??i,s[key]]).filter(p=>typeof p[1]==="number");
+  if(!pts.length)return"";
+  const W=380,H=112,P=30,xs=pts.map(p=>p[0]),ys=pts.map(p=>p[1]);
+  const x0=Math.min(...xs),x1=Math.max(...xs,x0+1);let y0=Math.min(...ys),y1=Math.max(...ys);
+  if(y0===y1){y0-=.5;y1+=.5}
+  const sx=v=>P+(v-x0)/(x1-x0)*(W-P-8),sy=v=>H-P+4-(v-y0)/(y1-y0)*(H-P-12);
+  return `<svg width="${W}" height="${H}" style="margin:0 8px 8px 0">
+    <text x="6" y="13" fill="#8b949e" font-size="10.5">${key}</text>
+    <text x="6" y="${H-6}" fill="#6e7681" font-size="9">${y0.toFixed(2)}</text>
+    <text x="${W-44}" y="${H-6}" fill="#6e7681" font-size="9">step ${x1}</text>
+    <path d="${pts.map((p,i)=>`${i?"L":"M"}${sx(p[0]).toFixed(1)},${sy(p[1]).toFixed(1)}`).join("")}"
+      fill="none" stroke="#58a6ff" stroke-width="1.5"/></svg>`;
+}
+$("#axH").onclick=()=>{AXIS="harness";$("#axH").classList.add("on");$("#axM").classList.remove("on");if(DS&&SEL?.did)renderGrid()};
+$("#axM").onclick=()=>{AXIS="model";$("#axM").classList.add("on");$("#axH").classList.remove("on");if(DS&&SEL?.did)renderGrid()};
+$("#refresh").onclick=async()=>{$("#refresh").textContent="…";
+  try{await api("/api/refresh",{method:"POST"});RUNCACHE={};const k=SEL;await loadTree();
+      if(k?.did)await openDataset(k.pid,k.did);else if(RUNSEL.size)await openRuns(k.pid)}
+  finally{$("#refresh").textContent="⟳"}};
+loadTree();
+</script>

tools/classify.py

@@ -0,0 +1,147 @@
+"""Derive the stable / experimental tiers from a sweep, instead of asserting them.
+
+The tier is a claim about evidence, so it should be computed from the evidence and carry it. A badge
+that someone typed by hand goes stale the moment a harness improves or regresses, and a tier with no
+stated reason is just a colour.
+
+RULES, in the order they are applied:
+
+  stable        the harness produced graded rollouts for essentially every task AND has a verified
+                training run. Both halves matter: capture working proves the tokens are right, and a
+                completed training step proves the trainer can consume them — they are separate failure
+                modes, and this stack has hit each independently.
+  experimental  anything else, with the specific gap named. Never a bare tier.
+
+WHAT IS NOT A REASON TO DOWNGRADE. A low pass rate. A harness scoring 0.0 on hard tasks is working
+correctly and reporting a real result; treating that as a defect would rank harnesses by how easy their
+tasks were. Only unmeasured rollouts, pauses, and known skew count against a harness here.
+
+Prompt re-render skew IS recorded as a caveat rather than a downgrade on its own: it is harmless for
+eval (nothing is trained) and disqualifying for training, so the caveat says which.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parents[1]
+
+# Measured over 670 turns against the engine's own prompt_token_ids. Eval-safe, training-unsafe.
+KNOWN_SKEW = {
+    "claude-code": "+2 tokens per prompt re-render — harmless for eval, forks every turn when training",
+    "gemini-cli": "+2 tokens per prompt re-render — harmless for eval, forks every turn when training",
+    "kimi-cli": "-10 tokens per tool call — the largest measured skew; unsafe to train on",
+}
+# Reads os.environ inside run(), so concurrency depends on the context-local overlay.
+NEEDS_OVERLAY = {"goose", "claude-code", "gemini-cli"}
+NO_STEP_LIMIT_NOTE = ("no step-limit expression in its seam, so rollouts run to the timeout; "
+                      "the kill surfaces as exit 137 and the rollout is retried")
+
+
+def classify(sweep: dict, trained: set[str], measured_floor: float) -> dict:
+    summary = sweep.get("summary", {})
+    per = summary.get("harnesses", {})
+    paused = summary.get("paused_harnesses", {})
+    k = summary.get("k", 4)
+    out = {}
+
+    for harness, m in sorted(per.items()):
+        caveats, tier = [], "experimental"
+        n_tasks = m.get("n_tasks") or 0
+        measured = m.get("n_measured") or 0
+        coverage = (measured / n_tasks) if n_tasks else 0.0
+
+        if harness in paused:
+            caveats.append(f"PAUSED mid-sweep: {paused[harness]}")
+        elif coverage < measured_floor:
+            caveats.append(
+                f"only {measured}/{n_tasks} tasks produced a graded rollout "
+                f"({coverage:.0%} < {measured_floor:.0%} required)"
+            )
+        elif harness not in trained:
+            caveats.append(
+                "eval measured but no verified training run — capture working does not prove the "
+                "trainer can consume it, which is a separate failure mode"
+            )
+        else:
+            tier = "stable"
+
+        if harness in KNOWN_SKEW:
+            caveats.append(KNOWN_SKEW[harness])
+        if harness in NEEDS_OVERLAY:
+            caveats.append("reads os.environ inside run(); concurrent only via the context-local overlay")
+
+        entry = {
+            "tier": tier,
+            "evidence": (
+                f"pass@{k} {m.get(f'pass@{k}')}, pass@1 {m.get('pass@1')}, "
+                f"{measured}/{n_tasks} tasks measured, mean {m.get('mean_turns')} turns"
+            ),
+        }
+        if caveats:
+            entry["caveats"] = caveats
+        out[harness] = entry
+
+    # A harness that never appeared in the sweep at all is not 'experimental', it is untested — saying
+    # otherwise would imply it was tried.
+    for harness in paused:
+        out.setdefault(harness, {"tier": "experimental", "caveats": [f"PAUSED: {paused[harness]}"]})
+    return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    # Several files, because a sweep can be split across jobs — and it was: one 15-harness job projected
+    # past its time limit, so it became three. Merging here rather than requiring one file means the
+    # split is an operational detail instead of something the classification has to know about.
+    ap.add_argument("--sweep", required=True, nargs="+", help="one or more eval sweep JSONs")
+    ap.add_argument("--project", default="data-agent")
+    ap.add_argument("--trained", default="mini-swe-agent,opencode",
+                    help="harnesses with a verified training run")
+    ap.add_argument("--measured-floor", type=float, default=0.9,
+                    help="fraction of tasks that must produce a graded rollout to be stable")
+    ap.add_argument("--dry-run", action="store_true")
+    args = ap.parse_args()
+
+    merged = {"summary": {"harnesses": {}, "paused_harnesses": {}, "k": None}}
+    for f in args.sweep:
+        one = json.loads(Path(f).read_text())
+        sm = one.get("summary", {})
+        merged["summary"]["k"] = merged["summary"]["k"] or sm.get("k")
+        merged["summary"]["paused_harnesses"].update(sm.get("paused_harnesses") or {})
+        for h, m in (sm.get("harnesses") or {}).items():
+            prev = merged["summary"]["harnesses"].get(h)
+            # A harness can appear in more than one file — opencode ran in the cancelled job AND in the
+            # relaunch. Keep whichever measured more tasks: that is the more complete evidence, and
+            # averaging two partial runs of different sizes would invent a number neither produced.
+            if prev is None or (m.get("n_measured") or 0) > (prev.get("n_measured") or 0):
+                merged["summary"]["harnesses"][h] = m
+    sweep = merged
+    trained = {h.strip() for h in args.trained.split(",") if h.strip()}
+    support = classify(sweep, trained, args.measured_floor)
+
+    for h, e in sorted(support.items(), key=lambda kv: (kv[1]["tier"] != "stable", kv[0])):
+        print(f"  {h:18s} {e['tier']:13s} {e.get('evidence','')}")
+        for c in e.get("caveats", []):
+            print(f"       · {c}")
+
+    if args.dry_run:
+        return 0
+    p = HERE / "data" / "projects" / args.project / "project.json"
+    d = json.loads(p.read_text()) if p.exists() else {"project_id": args.project}
+    d["support"] = support
+    d["tier_rule"] = (
+        "stable = graded rollouts on >=90% of tasks AND a verified training run. experimental = "
+        "anything else, with the gap named. A low pass rate is never a downgrade: a harness scoring 0.0 "
+        "is reporting a real result, and penalising that would rank harnesses by task difficulty."
+    )
+    d["support_source"] = [Path(f).name for f in args.sweep]
+    p.write_text(json.dumps(d, indent=2))
+    print(f"\nwrote {p.relative_to(HERE)}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tools/ingest.py

@@ -0,0 +1,274 @@
+"""Turn what our runs actually produce into the viewer's contract.
+
+The stack already writes two things we do not control the shape of: `eval_pass_at_k.py` dumps a JSON of
+per-harness summaries plus raw rows, and AsyncGRPO logs training metrics to a trackio sqlite. Rather
+than change either — a viewer should not dictate how a trainer logs — this converts them into
+`runs/<run_id>/` as CONTRACT.md describes.
+
+    # an eval sweep
+    python tools/ingest.py eval  --json logs/eval_6harness.json --run-id 2b-6harness --model Qwen/Qwen3.5-2B
+
+    # a training run, straight from the trackio db
+    python tools/ingest.py train --trackio ~/runs/agrpo_harbor/trackio/<project>.db \
+                                 --run-name Qwen3.5-2B-mini-swe-agent-20steps-46552 --run-id 2b-mini-20
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sqlite3
+from collections import defaultdict
+from datetime import datetime, timezone
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parents[1]
+
+
+def _now() -> str:
+    return datetime.now(timezone.utc).isoformat(timespec="seconds")
+
+
+def _write(run_dir: Path, rel: str, payload) -> None:
+    p = run_dir / rel
+    p.parent.mkdir(parents=True, exist_ok=True)
+    p.write_text(json.dumps(payload, indent=2, default=str) if not isinstance(payload, str) else payload)
+    print(f"  wrote {p.relative_to(HERE)}")
+
+
+def _task_metadata(split: str) -> dict[int, dict]:
+    """index -> {id, question, answer, difficulty_level} from the Harbor suite itself."""
+    import sys as _sys
+
+    _sys.path.insert(0, str(HERE.parents[0] / "async_grpo_harbor_data_agent" / "src"))
+    from harbor_tasks import _read_meta, download_suite  # type: ignore
+
+    root = download_suite(split)
+    out = {}
+    for i, toml in enumerate(sorted(root.glob("tasks/*/task.toml"))):
+        m = dict(_read_meta(toml.parent).get("metadata") or {})
+        instr = toml.parent / "instruction.md"
+        # `meta` is whatever the suite records, carried verbatim. Promoting a fixed set of keys to
+        # columns means the viewer breaks the moment a suite adds or renames one — and difficulty,
+        # package_tier and reward_mode are exactly the kind of field that changes between suites. The
+        # viewer discovers the keys instead and offers them as filters.
+        out[i] = {
+            "id": toml.parent.name,
+            "answer": m.pop("gold_answer", None),
+            "question": (instr.read_text()[:4000] if instr.exists() else None),
+            "meta": m,
+        }
+    return out
+
+
+def ds_dir_for(args):
+    return HERE / "data" / "projects" / args.project / "datasets" / args.dataset_id
+
+
+def ingest_eval(args) -> None:
+    raw = json.loads(Path(args.json).read_text())
+    summary_in = raw.get("summary", {})
+    per_harness_in = summary_in.get("harnesses") or {}
+    rows_in = raw.get("rows") or {}
+
+    # Two shapes exist in the wild and both are real output from this stack: the multi-harness sweep
+    # writes {"rows": {harness: [...]}} while the earlier single-harness runs wrote
+    # {"per_task": {index: [...]}} with no harness on the rows. Normalising here rather than rejecting
+    # the older one keeps already-collected results usable — they are the only baseline we have.
+    if not rows_in and raw.get("per_task"):
+        harness = summary_in.get("harness") or args.harness_fallback
+        rows_in = {harness: [r for rows in raw["per_task"].values() for r in rows]}
+        if not per_harness_in:
+            per_harness_in = {harness: {k: v for k, v in summary_in.items() if k.startswith("pass@") or k == "mean_turns"}}
+
+    # pass@k over TASKS, pass@1 over SAMPLES, and `reward is None` excluded rather than scored 0 —
+    # the same rule the eval tool applies, restated here so the published numbers cannot drift from it.
+    by_harness, tasks_acc = [], defaultdict(dict)
+    task_meta: dict[int, dict] = {}
+    if args.tasks_from:
+        # Question, gold answer and difficulty come from the suite, not from the eval output. Without
+        # them a row is an opaque index and a cell cannot be judged by eye.
+        task_meta = _task_metadata(args.tasks_from)
+    totals = {"tasks_total": 0, "tasks_any_pass": 0, "attempts_total": 0, "attempts_passed": 0, "n_all_infra": 0}
+    k = summary_in.get("k", args.k)
+
+    for harness, rows in rows_in.items():
+        by_task = defaultdict(list)
+        for r in rows:
+            by_task[r["index"]].append(r)
+        measured = solved = infra = 0
+        samples, turns = [], []
+        for index, rs in by_task.items():
+            graded = [r for r in rs if r.get("reward") is not None]
+            key = f"{args.model}|{harness}"
+            attempts = []
+            for i, r in enumerate(sorted(rs, key=lambda r: r.get("rep", 0))):
+                att = {"attempt": i + 1, "reward": r.get("reward"), "n_turns": r.get("n_turns")}
+                # A trace is written only when the row carries one. Referencing a file that does not
+                # exist would give the viewer an "open" button that always 404s.
+                att["elapsed_sec"] = r.get("wall_s")
+                if r.get("messages"):
+                    tid = f"{index}-{harness}-{i + 1}.json"
+                    _write(ds_dir_for(args), f"traces/{tid}", {
+                        "task_index": index, "task_id": r.get("task_id"),
+                        "model": args.model, "harness": harness, "attempt": i + 1,
+                        "reward": r.get("reward"), "rewards": r.get("rewards") or {},
+                        "n_turns": r.get("n_turns"), "elapsed_sec": r.get("wall_s"),
+                        "rollout_type": r.get("rollout_type"),
+                        "n_trainable_tokens": r.get("n_trainable_tokens"),
+                        "trial_name": r.get("trial_name"),
+                        "messages": r["messages"],
+                    })
+                    att["trace"] = f"traces/{tid}"
+                attempts.append(att)
+            first_pass = next((a["attempt"] for a in attempts if (a["reward"] or 0) > 0), None)
+            tasks_acc[index][key] = {"passed_at": first_pass, "attempts": attempts}
+            if not graded:
+                infra += 1
+                continue
+            measured += 1
+            if any((r["reward"] or 0) > 0 for r in graded):
+                solved += 1
+            samples.extend(r["reward"] for r in graded)
+            turns.extend(r.get("n_turns") or 0 for r in graded)
+        m = per_harness_in.get(harness, {})
+        by_harness.append({
+            "harness": harness,
+            f"pass@{k}": m.get(f"pass@{k}", round(solved / measured, 4) if measured else None),
+            "pass@1": m.get("pass@1", round(sum(samples) / len(samples), 4) if samples else None),
+            "mean_turns": m.get("mean_turns", round(sum(turns) / len(turns), 2) if turns else None),
+            "n_measured": measured, "cells": len(by_task), "n_all_infra": infra,
+        })
+        totals["attempts_total"] += sum(len(v) for v in by_task.values())
+        totals["attempts_passed"] += sum(1 for s in samples if s > 0)
+        totals["n_all_infra"] += infra
+
+    totals["tasks_total"] = len(tasks_acc)
+    totals["tasks_any_pass"] = sum(
+        1 for cells in tasks_acc.values() if any(c["passed_at"] for c in cells.values())
+    )
+
+    proj = HERE / "data" / "projects" / args.project
+    ds = proj / "datasets" / args.dataset_id
+
+    # by_model mirrors by_harness so the viewer's pivot has aggregates on both axes. With one model in a
+    # sweep it is a single row, which is honest rather than empty.
+    model_samples = [a["reward"] for cells in tasks_acc.values() for c in cells.values()
+                     for a in c["attempts"] if a["reward"] is not None]
+    by_model = [{
+        "model": args.model,
+        "cells": totals["attempts_total"],
+        f"pass@{k}": round(totals["tasks_any_pass"] / totals["tasks_total"], 4) if totals["tasks_total"] else None,
+        "pass@1": round(sum(1 for r in model_samples if r > 0) / len(model_samples), 4) if model_samples else None,
+        "n_measured": totals["tasks_total"] - totals["n_all_infra"],
+        "mean_turns": None,
+    }]
+
+    _write(ds, "summary.json", {
+        "k_max": k,
+        "models": [args.model],
+        "harnesses": sorted(rows_in),
+        "summary": totals,
+        "by_harness": sorted(by_harness, key=lambda h: -(h.get(f"pass@{k}") or 0)),
+        "by_model": by_model,
+        "tasks": [
+            {"id": task_meta.get(i, {}).get("id", str(i)), "index": i,
+             "question": task_meta.get(i, {}).get("question"),
+             "answer": task_meta.get(i, {}).get("answer"),
+             "meta": task_meta.get(i, {}).get("meta", {}),
+             "cells": cells}
+            for i, cells in sorted(tasks_acc.items())
+        ],
+    })
+    _write(ds, "dataset.json", {
+        "dataset_id": args.dataset_id, "label": args.dataset_label or args.dataset_id,
+        "split": "eval", "source": args.dataset, "k": k,
+        "created_at": _now(), "notes": args.notes,
+    })
+    if not (proj / "project.json").exists():
+        _write(proj, "project.json", {
+            "project_id": args.project, "label": args.project_label or args.project,
+            "description": args.project_description,
+            "source": {"hf_dataset": args.dataset},
+            "support": {},
+        })
+
+
+def ingest_train(args) -> None:
+    """Read trackio's sqlite directly: it is the source of truth for a finished run, and re-deriving
+    metrics from stdout would invent numbers the trainer never logged."""
+    con = sqlite3.connect(args.trackio)
+    rows = con.execute(
+        "select step, metrics from metrics where run_name like ? and length(metrics) > 4 order by step",
+        (f"%{args.run_name}%",),
+    ).fetchall()
+    if not rows:
+        raise SystemExit(f"no metric rows matching {args.run_name!r} in {args.trackio}")
+
+    lines = []
+    for step, blob in rows:
+        d = json.loads(blob if isinstance(blob, (str, bytes)) else str(blob))
+        lines.append(json.dumps({
+            "step": step,
+            "reward": d.get("train/reward"), "reward_std": d.get("train/reward_std"),
+            "loss": d.get("train/loss"), "ratio": d.get("train/ratio"),
+            "kl": d.get("train/kl"), "entropy": d.get("train/entropy"),
+            "learning_rate": d.get("train/learning_rate"),
+        }))
+
+    run_dir = HERE / "data" / "projects" / args.project / "runs" / args.run_id
+    _write(run_dir, "train/metrics.jsonl", "\n".join(lines) + "\n")
+    with_grad = sum(1 for line in lines if (json.loads(line).get("reward_std") or 0) > 0)
+    _write(run_dir, "run.json", {
+        "run_id": args.run_id, "kind": "train", "created_at": _now(), "updated_at": _now(),
+        "model": args.model, "harnesses": [args.harness] if args.harness else [],
+        "sandbox": args.sandbox, "dataset": args.dataset, "split": "train",
+        "notes": args.notes or f"{len(lines)} steps, {with_grad} with a non-zero gradient",
+        "config": {"trackio_run": args.run_name},
+    })
+    print(f"  {len(lines)} steps, {with_grad} with reward_std > 0")
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    e = sub.add_parser("eval")
+    e.add_argument("--json", required=True, help="output of tools/eval_pass_at_k.py")
+    e.add_argument("--model", default="Qwen/Qwen3.5-2B")
+    e.add_argument("--sandbox", default="e2b")
+    e.add_argument("--dataset", default="AdithyaSK/data_agent_rl_environment_eval")
+    e.add_argument("--k", type=int, default=4)
+    e.add_argument("--notes", default="")
+    e.add_argument("--project", default="data-agent")
+    e.add_argument("--project-label", default="Data-Agent Bench")
+    e.add_argument("--project-description", default="")
+    e.add_argument("--dataset-id", required=True, help="a variation within the project, e.g. eval-easy50")
+    e.add_argument("--dataset-label", default="")
+    e.add_argument("--tasks-from", default="", help="Harbor suite to pull question/answer/difficulty from")
+    e.add_argument("--harness-fallback", default="mini-swe-agent",
+                   help="harness name for older single-harness JSON that does not record one")
+    e.set_defaults(fn=ingest_eval)
+
+    t = sub.add_parser("train")
+    t.add_argument("--trackio", required=True, help="path to the trackio sqlite db")
+    t.add_argument("--run-name", required=True, help="substring of the trackio run name")
+    t.add_argument("--run-id", required=True)
+    t.add_argument("--model", default="Qwen/Qwen3.5-2B")
+    t.add_argument("--harness", default="mini-swe-agent")
+    t.add_argument("--sandbox", default="e2b")
+    t.add_argument("--dataset", default="AdithyaSK/data_agent_rl_environment_train")
+    t.add_argument("--notes", default="")
+    t.add_argument("--project", default="data-agent")
+    t.set_defaults(fn=ingest_train)
+
+    args = ap.parse_args()
+    where = (f"projects/{args.project}/datasets/{args.dataset_id}" if args.cmd == "eval"
+             else f"projects/{args.project}/runs/{args.run_id}")
+    print(f"ingesting {args.cmd} -> data/{where}/")
+    args.fn(args)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tools/watch_ingest.sh

@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+# Re-ingest a live sweep into the viewer every INTERVAL seconds, so progress is visible while it runs
+# rather than only at the end. The eval tool writes partial summaries atomically, so reading one mid-run
+# is safe; a half-written file would have looked like a corrupt result instead of an in-progress one.
+set -uo pipefail
+VIZ="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+PY="${PY:-/fsx/adithyaskolavi/projects/trl_prod/.venv312/bin/python}"
+SRC="${SRC:?set SRC=/path/to/sweep.json}"
+DSID="${DSID:-phase0}"
+INTERVAL="${INTERVAL:-120}"
+JOB="${JOB:-}"
+LOG="$VIZ/logs/watch_ingest.log"; mkdir -p "$VIZ/logs"
+
+log(){ echo "$(date -Is) $*" >> "$LOG"; }
+log "watching $SRC -> dataset $DSID every ${INTERVAL}s (job ${JOB:-none})"
+
+while true; do
+  if [ -f "$SRC" ]; then
+    if "$PY" "$VIZ/tools/ingest.py" eval --json "$SRC" --dataset-id "$DSID" \
+         --dataset-label "phase 0 · 15 harnesses · 50 tasks · k=4" \
+         --project data-agent --tasks-from AdithyaSK/data_agent_rl_environment_eval \
+         --notes "live sweep" >>"$LOG" 2>&1; then
+      # The viewer caches listings, so tell it to drop them; otherwise a refresh in the browser shows
+      # the same numbers for up to REFRESH_TTL seconds and looks stuck.
+      curl -fs -m 10 -X POST http://127.0.0.1:8090/api/refresh >/dev/null 2>&1 || true
+      log "ingested $(stat -c%s "$SRC") bytes"
+    else
+      log "ingest failed (likely a partial write mid-move); retrying next tick"
+    fi
+  else
+    log "waiting for $SRC to appear"
+  fi
+  # Stop once the job is gone AND one final ingest has run, so the last state is always captured.
+  if [ -n "$JOB" ] && ! squeue -h -j "$JOB" >/dev/null 2>&1; then
+    log "job $JOB finished; final ingest done, exiting"
+    exit 0
+  fi
+  sleep "$INTERVAL"
+done