README.md

metadata

title: Harbor Eval Visualizations
emoji: 📊
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false

Harbor Eval Visualizations

Interactive viewer for multiple Harbor benchmark sweeps. Use the dataset toggle at the top of the page to switch between:

• Qwen3.5 Data-Agent v1 — pass@4 of Qwen3.5 (2B / 4B / 9B / 27B / 35B-A3B) on 366 Kaggle data-analysis tasks × 4 harnesses.
• DABstep (Adyen) — pass@4 of Qwen3.5 (4B / 9B) on 450 financial data-analysis tasks from adyen/DABstep × 4 harnesses (72 easy + 378 hard).

For each sweep:

• Heatmap: task × harness, color-coded by pass/fail, filterable by difficulty.
• Click any cell → full multi-turn agent trajectory + grader verdict (loaded on demand).

Server

FastAPI (app.py) serves:

Endpoint	What
GET /	the single-page viewer
GET /api/datasets	list of available dataset keys
GET /api/<ds>/summary	per-dataset summary (heatmap + per-attempt metadata)
GET /api/<ds>/trace/{tid}	one trajectory + grader output, lazy-loaded
GET /healthz	liveness + which dataset trace files are in memory

Trace files are loaded into memory lazily on the first request per dataset, so cold start stays small even as more sweeps are added.

Layout

site/
├── viewer.html                  # dataset-toggle aware
├── v1/{summary,traces}.json
└── dabstep/{summary,traces}.json

To add another sweep, drop a site/<name>/{summary,traces}.json pair and optionally add a pretty label to DS_LABELS in viewer.html. Generate the data with python build_data.py --name <sweep> --suite <suite> --out site/<name>.