CONTRACT.md

# Information contract

Everything the viewer shows lives in **one directory**, identical whether it is a local folder or a
Hugging Face bucket mounted into a Space. A mounted bucket appears as an ordinary filesystem path inside
the container, so local and hosted use the same code path and there is nothing to keep in sync.

## Hierarchy

```
project              "data-agent"                          — a body of work
└── dataset          "eval-v1", "eval-easy50", "dabstep"    — variations within it
    └── task         one ROW in the table
        └── cell     one (model, harness) pair
            └── attempts   k tries → pass@k, each with a trace
```

Two axes cross at a cell, and **which one is the column is a view choice, not a storage choice** — so
cells are keyed `"<model>|<harness>"` and the viewer pivots either way. Storing per-axis would force a
rewrite to flip the table.

```
<DATA_DIR>/
└── projects/
    └── <project_id>/
        ├── project.json                    # REQUIRED — label, description, source, support tiers
        ├── datasets/
        │   └── <dataset_id>/
        │       ├── dataset.json            # optional label/notes
        │       ├── summary.json            # tasks[] with cells, by_model, by_harness
        │       └── traces/<trace_id>.json  # one attempt each; referenced from a cell
        └── runs/
            └── <run_id>/                   # training runs belonging to this project
                ├── run.json
                └── train/metrics.jsonl
```

Discovery is by file presence: `project.json` makes a project, `summary.json` a dataset, `run.json` a
training run. Nothing else is required and the viewer never assumes a file exists.

## `project.json`

```json
{
  "project_id": "data-agent",
  "label": "Data-Agent Bench",
  "description": "Verified data-analysis tasks over Kaggle datasets, graded exact / numeric / LLM-judge.",
  "source": {"hf_dataset": "AdithyaSK/data_agent_rl_environment_eval"},
  "support": {
    "mini-swe-agent": {"tier": "stable"},
    "opencode": {"tier": "experimental",
                 "caveats": ["no step limit; ~90 turns and ~17% of rollouts retry after exit 137"]}
  }
}
```

`support` is per-harness and drives the stable / experimental badge and its warning. A tier with no
caveat is just a colour, so an experimental harness has to say what does not work.

## `datasets/<id>/summary.json`

```json
{
  "k_max": 4,
  "models": ["Qwen/Qwen3.5-2B"],
  "harnesses": ["mini-swe-agent", "opencode"],
  "summary": {"tasks_total": 50, "tasks_any_pass": 31, "attempts_total": 400,
              "attempts_passed": 74, "n_all_infra": 0},
  "by_model":   [{"model": "Qwen/Qwen3.5-2B", "cells": 100, "pass@1": 0.17, "pass@4": 0.31,
                  "n_measured": 100, "mean_turns": 11.7}],
  "by_harness": [{"harness": "mini-swe-agent", "cells": 100, "pass@1": 0.17, "pass@4": 0.31,
                  "n_measured": 100, "mean_turns": 11.7}],
  "tasks": [
    {"id": "0000_416_416942_qa_3", "index": 7, "difficulty_level": 0, "difficulty": "easy",
     "question": "What is the median of ...?", "answer": "0.4056", "reward_mode": "numeric",
     "cells": {
       "Qwen/Qwen3.5-2B|mini-swe-agent": {
         "passed_at": 3,
         "attempts": [{"attempt": 1, "reward": 0.0, "n_turns": 12, "elapsed_sec": 41.2,
                       "trace": "traces/0000_416_416942_qa_3-2b-mini-1.json"}]
       }
     }}
  ]
}
```

Conventions that carry meaning rather than being formatting:

* **`reward: null` means the verifier never ran** — dead sandbox, no answer. Not a zero. Aggregates
  exclude those attempts and count a task under `n_all_infra` when every attempt is null. Averaging them
  in as zeros makes infrastructure failure look like a weak model: one run reported `pass@4 0.333` from
  3 measured tasks out of 8 before that rule existed.
* **`passed_at`** is the first passing attempt, so `pass@1` and `pass@k` both derive from one structure
  and cannot disagree.
* **`trace`** is a path relative to the dataset directory, so opening a cell needs no naming convention
  the viewer has to guess.

## `traces/<trace_id>.json`

Free-form per attempt; rendered as whatever is present.

```json
{"task_id": "...", "model": "...", "harness": "...", "attempt": 1,
 "reward": 0.0, "reward_method": "llm", "n_turns": 12, "elapsed_sec": 41.2,
 "gold": "0.4056", "predicted": "0.41",
 "messages": [{"role": "user", "content": "..."},
              {"role": "assistant", "content": "...", "tool_calls": [...]},
              {"role": "tool", "content": "..."}]}
```

`reward_method` is worth carrying: the grader is three-tier (exact → numeric → gpt-4o-mini judge), so
knowing a 0 came from the judge rather than exact match separates a wrong answer from a formatting
mismatch.

## `runs/<run_id>/train/metrics.jsonl`

One object per step, append-only so a live run is readable without rewriting.

```json
{"step": 3, "reward": 0.829, "reward_std": 0.146, "loss": 0.279, "ratio": 0.9999,
 "kl": 0.032, "entropy": 0.51, "n_rollouts": 8, "n_unscorable": 1, "wall_s": 47.6}
```

`reward_std` sits next to `reward` because a step whose generations all score the same has zero
advantage and teaches nothing — a reward curve alone cannot tell learning from a flat run.

## Publishing

```sh
hf buckets create <ns>/harbor-runs --exist-ok
HF_HUB_DISABLE_XET=1 hf sync ./data hf://buckets/<ns>/harbor-runs
hf spaces volumes set <ns>/harbor-viewer -v hf://buckets/<ns>/harbor-runs:/data
```

`HF_HUB_DISABLE_XET=1` is not decoration: the Xet upload path hangs indefinitely on this cluster while
the plain HTTP path finishes in seconds.