README.md

---
title: Harbor Eval Visualizations
emoji: 📊
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
---

# Harbor Eval Visualizations

Interactive viewer for multiple Harbor benchmark sweeps. Use the **dataset
toggle** at the top of the page to switch between:

- **Qwen3.5 Data-Agent v1** — pass@4 of Qwen3.5 (2B / 4B / 9B / 27B / 35B-A3B)
  on 366 Kaggle data-analysis tasks × 4 harnesses.
- **DABstep (Adyen)** — pass@4 of Qwen3.5 (4B / 9B) on 450 financial
  data-analysis tasks from
  [`adyen/DABstep`](https://huggingface.co/datasets/adyen/DABstep) × 4 harnesses
  (72 easy + 378 hard).

For each sweep:

- **Heatmap**: task × harness, color-coded by pass/fail, filterable by difficulty.
- **Click any cell** → full multi-turn agent trajectory + grader verdict
  (loaded on demand).

## Server

FastAPI (`app.py`) serves:

| Endpoint | What |
|---|---|
| `GET /` | the single-page viewer |
| `GET /api/datasets` | list of available dataset keys |
| `GET /api/<ds>/summary` | per-dataset summary (heatmap + per-attempt metadata) |
| `GET /api/<ds>/trace/{tid}` | one trajectory + grader output, lazy-loaded |
| `GET /healthz` | liveness + which dataset trace files are in memory |

Trace files are loaded into memory **lazily** on the first request per dataset,
so cold start stays small even as more sweeps are added.

## Layout

```
site/
├── viewer.html                  # dataset-toggle aware
├── v1/{summary,traces}.json
└── dabstep/{summary,traces}.json
```

To add another sweep, drop a `site/<name>/{summary,traces}.json` pair and
optionally add a pretty label to `DS_LABELS` in `viewer.html`. Generate the
data with `python build_data.py --name <sweep> --suite <suite> --out site/<name>`.