| # Question sets |
|
|
| Each file here is a flat JSON array of question records (the schema |
| `SQLEnvironment._load_questions` enforces): `question_text`, `database_name`, |
| `gold_sql`, `gold_answer`, `answer_type` (`integer | float | string | list | |
| table`), `difficulty`, `tables_involved`, `question_id`, `split`. |
|
|
| | File | Split | What it is | |
| |------|-------|------------| |
| | `questions_train.json` / `questions_eval.json` | train / eval | Spider train + eval splits (RL training + evaluation). | |
| | `eval_n50.json` (+ `eval_n50_ids.json`) | eval | The frozen N=50 success-gate subset (`scripts/freeze_eval_subset.py`). | |
| | `retail_smb.json` | demo | The business-led demo set — owner-framed decision questions over Maria's pet shop (see below). | |
|
|
| ## `retail_smb.json` — the demo dataset (F009) |
| |
| `retail_smb.json` holds ~26 owner-framed **decision** questions over the synthetic |
| retail-SMB database at `data/databases/retail_smb/retail_smb.sqlite` (persona: |
| "Maria — a 3-location pet-supply + grooming shop"). It is the data behind the |
| judged Gradio demo (ADR 0008, business-led showcase). |
|
|
| - **Regenerate the database:** `uv run python scripts/generate_retail_demo.py` |
| (deterministic — single seed, no wall-clock; re-running reproduces a |
| byte-identical DB, so the gold answers below never drift). |
| - **Validate the questions:** every record passes the gold-answer gate with 0 |
| broken / 0 degenerate: |
| ``` |
| uv run python scripts/validate_questions.py --questions data/questions/retail_smb.json |
| ``` |
| - **`orders.status` is a coded INTEGER** (1 = paid, 2 = pending, 3 = refunded), so |
| questions like "what share of my orders were refunded?" are real |
| `WHERE status = 3` queries (the ADR 0007/0009 refund data-card beat). |
|
|
| ### Revenue conventions |
|
|
| Two revenue lenses coexist in this set; the difference is deliberate, so the |
| golds are internally consistent: |
|
|
| - **Order-level revenue counts PAID orders only** (`WHERE status = 1`). Any |
| question phrased around `orders.amount` — "total revenue from paid orders", |
| "average value of a paid order", revenue per store/month — filters to paid. |
| - **Product- / line-item revenue is GROSS sell-through** (`SUM(line_amount)` |
| across *all* order statuses) — it answers "what's moving off the shelf", |
| which for an SMB owner includes pending and later-refunded lines. So |
| "which product/category brings in the most" (`retail_smb_011`, `_012`, |
| `_023`) is gross by design and is left that way. |
| - **Exception — when a question explicitly says "paid"**, the line-level gold |
| also filters to paid orders. `retail_smb_025` ("…from paid lines") joins |
| `order_items → orders` and applies `WHERE o.status = 1`, so its numbers are |
| strictly lower than the gross category total. |
|
|
| ## Bring your own data (Strava — the secondary path) |
|
|
| The retail set ships in the repo; the Strava path is **user-supplied, nothing |
| committed**. A user exports their own `activities.csv` from Strava (see |
| [`docs/guides/export-strava-data.md`](../../docs/guides/export-strava-data.md)) |
| and loads it through the app's **upload button**, which calls the F003 |
| `ingest_csv` helper. That lands the CSV at |
| `data/uploads/<db_id>/<db_id>.sqlite` — the exact same `<db_dir>/<id>/<id>.sqlite` |
| layout `retail_smb` uses — so the agent queries it with zero engine changes. |
| `data/uploads/` is gitignored; no personal activity file is ever committed. |
|
|
