README.md

metadata

title: SQL-Eval
emoji: 🩺
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 6.10.0
app_file: app.py
pinned: false
license: apache-2.0
short_description: Human labeling of SQL result correctness

sql-eval-arena

Human-labeling tool for judging whether an agent's SQL query + result (attached to each ANS/SANS turn in results/chat_pp/all_raw_diag.json) actually answers the user's final intent, including anything resolved via clarification.

Setup

1. pip install -r requirements.txt
2. Regenerate data/sessions.json whenever all_raw_diag.json changes: python3 ../results/chat_pp/build_sql_eval_data.py (run from repo root)
3. Create a .env with:

   GITHUB_TOKEN=...
   GITHUB_REPO=your-org/sql-eval-human-labeling
   

   The target repo must already exist; submitted labels are written to its labels/{pid}_{sid}.json files (one per completed session).
4. python3 app.py

How assignment works

Each "new session" click hands out one whole conversation session (all its judgable turns at once) to whoever clicked. A session counts as done the moment its label file is created in the GitHub repo -- from then on it's excluded from everyone's pool, so no two people label the same session. Assignment itself (which session goes to which click) is tracked in-memory per running process, so it only works correctly as a single app instance (don't scale this to multiple replicas without adding shared claim state).