README.md

---
title: SQL-Eval
emoji: 🩺
colorFrom: blue
colorTo: green
sdk: gradio
sdk_version: 6.10.0
app_file: app.py
pinned: false
license: apache-2.0
short_description: 'Human labeling of SQL result correctness'
---

# sql-eval-arena

Human-labeling tool for judging whether an agent's SQL query + result
(attached to each `ANS`/`SANS` turn in `results/chat_pp/all_raw_diag.json`)
actually answers the user's final intent, including anything resolved via
clarification.

## Setup

1. `pip install -r requirements.txt`
2. Regenerate `data/sessions.json` whenever `all_raw_diag.json` changes:
   `python3 ../results/chat_pp/build_sql_eval_data.py` (run from repo root)
3. Create a `.env` with:
   ```
   GITHUB_TOKEN=...
   GITHUB_REPO=your-org/sql-eval-human-labeling
   ```
   The target repo must already exist; submitted labels are written to its
   `labels/{pid}_{sid}.json` files (one per completed session).
4. `python3 app.py`

## How assignment works

Each "new session" click hands out one whole conversation session (all its
judgable turns at once) to whoever clicked. A session counts as done the
moment its label file is created in the GitHub repo -- from then on it's
excluded from everyone's pool, so no two people label the same session.
Assignment itself (which session goes to which click) is tracked in-memory
per running process, so it only works correctly as a single app instance
(don't scale this to multiple replicas without adding shared claim state).