File size: 5,738 Bytes
3d20eb8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | # Information contract
Everything the viewer shows lives in **one directory**, identical whether it is a local folder or a
Hugging Face bucket mounted into a Space. A mounted bucket appears as an ordinary filesystem path inside
the container, so local and hosted use the same code path and there is nothing to keep in sync.
## Hierarchy
```
project "data-agent" — a body of work
└── dataset "eval-v1", "eval-easy50", "dabstep" — variations within it
└── task one ROW in the table
└── cell one (model, harness) pair
└── attempts k tries → pass@k, each with a trace
```
Two axes cross at a cell, and **which one is the column is a view choice, not a storage choice** — so
cells are keyed `"<model>|<harness>"` and the viewer pivots either way. Storing per-axis would force a
rewrite to flip the table.
```
<DATA_DIR>/
└── projects/
└── <project_id>/
├── project.json # REQUIRED — label, description, source, support tiers
├── datasets/
│ └── <dataset_id>/
│ ├── dataset.json # optional label/notes
│ ├── summary.json # tasks[] with cells, by_model, by_harness
│ └── traces/<trace_id>.json # one attempt each; referenced from a cell
└── runs/
└── <run_id>/ # training runs belonging to this project
├── run.json
└── train/metrics.jsonl
```
Discovery is by file presence: `project.json` makes a project, `summary.json` a dataset, `run.json` a
training run. Nothing else is required and the viewer never assumes a file exists.
## `project.json`
```json
{
"project_id": "data-agent",
"label": "Data-Agent Bench",
"description": "Verified data-analysis tasks over Kaggle datasets, graded exact / numeric / LLM-judge.",
"source": {"hf_dataset": "AdithyaSK/data_agent_rl_environment_eval"},
"support": {
"mini-swe-agent": {"tier": "stable"},
"opencode": {"tier": "experimental",
"caveats": ["no step limit; ~90 turns and ~17% of rollouts retry after exit 137"]}
}
}
```
`support` is per-harness and drives the stable / experimental badge and its warning. A tier with no
caveat is just a colour, so an experimental harness has to say what does not work.
## `datasets/<id>/summary.json`
```json
{
"k_max": 4,
"models": ["Qwen/Qwen3.5-2B"],
"harnesses": ["mini-swe-agent", "opencode"],
"summary": {"tasks_total": 50, "tasks_any_pass": 31, "attempts_total": 400,
"attempts_passed": 74, "n_all_infra": 0},
"by_model": [{"model": "Qwen/Qwen3.5-2B", "cells": 100, "pass@1": 0.17, "pass@4": 0.31,
"n_measured": 100, "mean_turns": 11.7}],
"by_harness": [{"harness": "mini-swe-agent", "cells": 100, "pass@1": 0.17, "pass@4": 0.31,
"n_measured": 100, "mean_turns": 11.7}],
"tasks": [
{"id": "0000_416_416942_qa_3", "index": 7, "difficulty_level": 0, "difficulty": "easy",
"question": "What is the median of ...?", "answer": "0.4056", "reward_mode": "numeric",
"cells": {
"Qwen/Qwen3.5-2B|mini-swe-agent": {
"passed_at": 3,
"attempts": [{"attempt": 1, "reward": 0.0, "n_turns": 12, "elapsed_sec": 41.2,
"trace": "traces/0000_416_416942_qa_3-2b-mini-1.json"}]
}
}}
]
}
```
Conventions that carry meaning rather than being formatting:
* **`reward: null` means the verifier never ran** — dead sandbox, no answer. Not a zero. Aggregates
exclude those attempts and count a task under `n_all_infra` when every attempt is null. Averaging them
in as zeros makes infrastructure failure look like a weak model: one run reported `pass@4 0.333` from
3 measured tasks out of 8 before that rule existed.
* **`passed_at`** is the first passing attempt, so `pass@1` and `pass@k` both derive from one structure
and cannot disagree.
* **`trace`** is a path relative to the dataset directory, so opening a cell needs no naming convention
the viewer has to guess.
## `traces/<trace_id>.json`
Free-form per attempt; rendered as whatever is present.
```json
{"task_id": "...", "model": "...", "harness": "...", "attempt": 1,
"reward": 0.0, "reward_method": "llm", "n_turns": 12, "elapsed_sec": 41.2,
"gold": "0.4056", "predicted": "0.41",
"messages": [{"role": "user", "content": "..."},
{"role": "assistant", "content": "...", "tool_calls": [...]},
{"role": "tool", "content": "..."}]}
```
`reward_method` is worth carrying: the grader is three-tier (exact → numeric → gpt-4o-mini judge), so
knowing a 0 came from the judge rather than exact match separates a wrong answer from a formatting
mismatch.
## `runs/<run_id>/train/metrics.jsonl`
One object per step, append-only so a live run is readable without rewriting.
```json
{"step": 3, "reward": 0.829, "reward_std": 0.146, "loss": 0.279, "ratio": 0.9999,
"kl": 0.032, "entropy": 0.51, "n_rollouts": 8, "n_unscorable": 1, "wall_s": 47.6}
```
`reward_std` sits next to `reward` because a step whose generations all score the same has zero
advantage and teaches nothing — a reward curve alone cannot tell learning from a flat run.
## Publishing
```sh
hf buckets create <ns>/harbor-runs --exist-ok
HF_HUB_DISABLE_XET=1 hf sync ./data hf://buckets/<ns>/harbor-runs
hf spaces volumes set <ns>/harbor-viewer -v hf://buckets/<ns>/harbor-runs:/data
```
`HF_HUB_DISABLE_XET=1` is not decoration: the Xet upload path hangs indefinitely on this cluster while
the plain HTTP path finishes in seconds.
|