CONTRACT.md

Information contract

Everything the viewer shows lives in one directory, identical whether it is a local folder or a Hugging Face bucket mounted into a Space. A mounted bucket appears as an ordinary filesystem path inside the container, so local and hosted use the same code path and there is nothing to keep in sync.

Hierarchy

project              "data-agent"                          — a body of work
└── dataset          "eval-v1", "eval-easy50", "dabstep"    — variations within it
    └── task         one ROW in the table
        └── cell     one (model, harness) pair
            └── attempts   k tries → pass@k, each with a trace

Two axes cross at a cell, and which one is the column is a view choice, not a storage choice — so cells are keyed "<model>|<harness>" and the viewer pivots either way. Storing per-axis would force a rewrite to flip the table.

<DATA_DIR>/
└── projects/
    └── <project_id>/
        ├── project.json                    # REQUIRED — label, description, source, support tiers
        ├── datasets/
        │   └── <dataset_id>/
        │       ├── dataset.json            # optional label/notes
        │       ├── summary.json            # tasks[] with cells, by_model, by_harness
        │       └── traces/<trace_id>.json  # one attempt each; referenced from a cell
        └── runs/
            └── <run_id>/                   # training runs belonging to this project
                ├── run.json
                └── train/metrics.jsonl

Discovery is by file presence: project.json makes a project, summary.json a dataset, run.json a training run. Nothing else is required and the viewer never assumes a file exists.

project.json

{
  "project_id": "data-agent",
  "label": "Data-Agent Bench",
  "description": "Verified data-analysis tasks over Kaggle datasets, graded exact / numeric / LLM-judge.",
  "source": {"hf_dataset": "AdithyaSK/data_agent_rl_environment_eval"},
  "support": {
    "mini-swe-agent": {"tier": "stable"},
    "opencode": {"tier": "experimental",
                 "caveats": ["no step limit; ~90 turns and ~17% of rollouts retry after exit 137"]}
  }
}

support is per-harness and drives the stable / experimental badge and its warning. A tier with no caveat is just a colour, so an experimental harness has to say what does not work.

datasets/<id>/summary.json

{
  "k_max": 4,
  "models": ["Qwen/Qwen3.5-2B"],
  "harnesses": ["mini-swe-agent", "opencode"],
  "summary": {"tasks_total": 50, "tasks_any_pass": 31, "attempts_total": 400,
              "attempts_passed": 74, "n_all_infra": 0},
  "by_model":   [{"model": "Qwen/Qwen3.5-2B", "cells": 100, "pass@1": 0.17, "pass@4": 0.31,
                  "n_measured": 100, "mean_turns": 11.7}],
  "by_harness": [{"harness": "mini-swe-agent", "cells": 100, "pass@1": 0.17, "pass@4": 0.31,
                  "n_measured": 100, "mean_turns": 11.7}],
  "tasks": [
    {"id": "0000_416_416942_qa_3", "index": 7, "difficulty_level": 0, "difficulty": "easy",
     "question": "What is the median of ...?", "answer": "0.4056", "reward_mode": "numeric",
     "cells": {
       "Qwen/Qwen3.5-2B|mini-swe-agent": {
         "passed_at": 3,
         "attempts": [{"attempt": 1, "reward": 0.0, "n_turns": 12, "elapsed_sec": 41.2,
                       "trace": "traces/0000_416_416942_qa_3-2b-mini-1.json"}]
       }
     }}
  ]
}

Conventions that carry meaning rather than being formatting:

• reward: null means the verifier never ran — dead sandbox, no answer. Not a zero. Aggregates exclude those attempts and count a task under n_all_infra when every attempt is null. Averaging them in as zeros makes infrastructure failure look like a weak model: one run reported pass@4 0.333 from 3 measured tasks out of 8 before that rule existed.
• passed_at is the first passing attempt, so pass@1 and pass@k both derive from one structure and cannot disagree.
• trace is a path relative to the dataset directory, so opening a cell needs no naming convention the viewer has to guess.

traces/<trace_id>.json

Free-form per attempt; rendered as whatever is present.

{"task_id": "...", "model": "...", "harness": "...", "attempt": 1,
 "reward": 0.0, "reward_method": "llm", "n_turns": 12, "elapsed_sec": 41.2,
 "gold": "0.4056", "predicted": "0.41",
 "messages": [{"role": "user", "content": "..."},
              {"role": "assistant", "content": "...", "tool_calls": [...]},
              {"role": "tool", "content": "..."}]}

reward_method is worth carrying: the grader is three-tier (exact → numeric → gpt-4o-mini judge), so knowing a 0 came from the judge rather than exact match separates a wrong answer from a formatting mismatch.

runs/<run_id>/train/metrics.jsonl

One object per step, append-only so a live run is readable without rewriting.

{"step": 3, "reward": 0.829, "reward_std": 0.146, "loss": 0.279, "ratio": 0.9999,
 "kl": 0.032, "entropy": 0.51, "n_rollouts": 8, "n_unscorable": 1, "wall_s": 47.6}

reward_std sits next to reward because a step whose generations all score the same has zero advantage and teaches nothing — a reward curve alone cannot tell learning from a flat run.

Publishing

hf buckets create <ns>/harbor-runs --exist-ok
HF_HUB_DISABLE_XET=1 hf sync ./data hf://buckets/<ns>/harbor-runs
hf spaces volumes set <ns>/harbor-viewer -v hf://buckets/<ns>/harbor-runs:/data

HF_HUB_DISABLE_XET=1 is not decoration: the Xet upload path hangs indefinitely on this cluster while the plain HTTP path finishes in seconds.