Update model card: improve results detail and add benchmark reproduction guide

README.md

@@ -87,25 +87,37 @@ Conversations were generated from the [Spider 1.0](https://yale-lily.github.io/s
 
 ## Evaluation Results
 
-Evaluated on the **Spider 1.0 dev set** (1,034 questions) using an agentic benchmark pipeline with execution accuracy.
+Evaluated on the **Spider 1.0 dev set** (1,034 questions) using an agentic benchmark pipeline. The agent autonomously explores database schemas via tool calls, writes pipe SQL, and iterates until correct — matching the training workflow.
+
+### Execution Accuracy
 
 | Metric | Value |
 |--------|-------|
 | **Execution Accuracy** | **60.66%** (626 / 1,032) |
 | **Prediction Rate** | 99.7% (1,031 / 1,034) |
+| **Total Questions** | 1,034 |
+| **Gold Errors Excluded** | 2 |
+
+### Detailed Breakdown
 
-### Status Breakdown
+| Status | Count | % of Total | Description |
+|--------|------:|------------|-------------|
+| **Match** | 626 | 60.5% | Predicted SQL produces identical results to gold SQL |
+| **Mismatch** | 209 | 20.2% | SQL executes but results differ from gold |
+| **Execution Error** | 170 | 16.4% | Transpiled SQL fails to execute against SQLite |
+| **Transpile Error** | 24 | 2.3% | Pipe SQL cannot be transpiled to standard SQL |
+| **No Prediction** | 3 | 0.3% | Agent did not produce a pipe SQL query |
+| **Gold Error** | 2 | 0.2% | Reference gold SQL fails (excluded from denominator) |
 
-| Status | Count | Percentage |
-|--------|-------|------------|
-| Match | 626 | 60.5% |
-| Mismatch | 209 | 20.2% |
-| Execution Error | 170 | 16.4% |
-| Transpile Error | 24 | 2.3% |
-| No Prediction | 3 | 0.3% |
-| Gold Error (excluded) | 2 | 0.2% |
+### Evaluation Methodology
 
-> **Note**: This is an **in-distribution** evaluation — the model was trained on Spider training data, and the dev set uses the same 20 databases. Gold errors (2 questions where the reference SQL fails) are excluded from the accuracy denominator.
+1. The TypeScript agent runs each question through a multi-turn tool-calling loop (max 10 turns, 120s timeout)
+2. The agent's final `execute_pipe_sql` call is extracted as the predicted pipe SQL
+3. Predicted pipe SQL is transpiled to standard SQL using `sqlglot.transpile()`
+4. Both predicted and gold SQL are executed against the Spider SQLite databases
+5. Result sets are compared using order-insensitive set comparison with numeric tolerance
+
+> **Note**: This is an **in-distribution** evaluation — the model was trained on Spider training data, and the dev set uses the same 20 databases.
 
 ## Tools
 
@@ -150,12 +162,177 @@ Tables in database 'concert_singer':
 
 For inference with the correct chat template, see the evaluation server code in the [sqlglot repository](https://github.com/nittygritty-zzy/sqlglot/tree/main/evaluation/server).
 
+## Reproducing the Benchmark
+
+### Prerequisites
+
+- **GPU**: NVIDIA GPU with >= 6 GB VRAM (model runs in float16)
+- **Python**: 3.11+ with pip/uv
+- **Node.js**: 18+ with npm
+- **Disk**: ~1 GB for Spider databases, ~3 GB for model weights
+
+### Step 1: Clone the Repository
+
+```bash
+git clone https://github.com/nittygritty-zzy/sqlglot.git
+cd sqlglot
+```
+
+### Step 2: Set Up Python Environment
+
+```bash
+# Create virtual environment
+uv venv .venv --python 3.11
+source .venv/bin/activate       # Linux/macOS
+# source .venv/Scripts/activate # Windows (Git Bash)
+
+# Install sqlglot (editable)
+uv pip install -e .
+
+# Install evaluation server dependencies
+uv pip install fastapi uvicorn pydantic
+
+# Install PyTorch with CUDA support
+uv pip install torch --index-url https://download.pytorch.org/whl/cu126
+
+# Install model loading dependencies
+uv pip install transformers accelerate
+```
+
+Verify CUDA:
+```bash
+python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
+# Expected: True NVIDIA GeForce RTX ...
+```
+
+### Step 3: Download Spider 1.0 Dataset
+
+The benchmark uses Spider 1.0 dev set (1,034 questions across 20 SQLite databases).
+
+```bash
+# Install gdown for Google Drive downloads
+uv pip install gdown
+
+# Download and extract Spider 1.0 (~1 GB)
+bash scripts/setup_data.sh
+```
+
+Verify:
+```bash
+ls data/spider/dev.json           # 1,034 questions
+ls data/spider/database/ | wc -l  # ~166 databases (20 used by dev set)
+```
+
+### Step 4: Download the Model
+
+```bash
+# Option A: Use huggingface_hub (recommended)
+pip install huggingface_hub
+python -c "
+from huggingface_hub import snapshot_download
+snapshot_download('enycin28/pipe-sql-1.5b', local_dir='finetuning_output/merged')
+"
+
+# Option B: Use git-lfs
+git lfs install
+git clone https://huggingface.co/enycin28/pipe-sql-1.5b finetuning_output/merged
+```
+
+### Step 5: Install Node.js Agent Dependencies
+
+```bash
+cd evaluation/agent
+npm install
+cd ../..
+```
+
+### Step 6: Run the Benchmark
+
+#### Option A: Full Pipeline (Recommended)
+
+```bash
+# Run all 1,034 questions (takes ~2 hours on RTX 4080)
+bash evaluation/run_all.sh
+
+# Smoke test with 5 questions first
+bash evaluation/run_all.sh --limit 5
+```
+
+This script:
+1. Starts the Python evaluation server (model inference + tool execution)
+2. Waits for the server to be ready
+3. Runs the TypeScript agent benchmark
+4. Evaluates results and prints execution accuracy
+
+#### Option B: Run Components Separately
+
+**Start the evaluation server:**
+```bash
+# Default: loads model from finetuning_output/merged/
+python -m evaluation.server.app
+
+# Custom model path:
+MODEL_PATH=path/to/model python -m evaluation.server.app
+```
+
+Wait for `Server ready` in the logs, then in a separate terminal:
+
+**Run the agent benchmark:**
+```bash
+cd evaluation/agent
+npx tsx src/main.ts --benchmark           # All 1,034 questions
+npx tsx src/main.ts --benchmark --limit 5 # Smoke test
+```
+
+**Run single question interactively:**
+```bash
+cd evaluation/agent
+npx tsx src/main.ts "How many singers do we have?" concert_singer
+```
+
+**Evaluate results:**
+```bash
+python evaluation/evaluate.py --results evaluation_output/results.json
+```
+
+### Step 7: Review Results
+
+Results are saved to `evaluation_output/`:
+
+| File | Description |
+|------|-------------|
+| `results.json` | Agent predictions with conversation traces |
+| `eval_results.json` | Per-question evaluation details (match/mismatch/error) |
+| `eval_summary.json` | Aggregate metrics |
+
+### Configuration
+
+| Environment Variable | Default | Description |
+|---------------------|---------|-------------|
+| `MODEL_PATH` | `finetuning_output/merged` | Path to merged model directory |
+| `SPIDER_DB_DIR` | `data/spider/database` | Spider database directory |
+| `SPIDER_DIR` | `data/spider` | Spider data directory (contains dev.json) |
+| `PORT` | `8000` | Evaluation server port |
+| `SERVER_URL` | `http://localhost:8000` | Agent to server connection URL |
+| `OUTPUT_DIR` | `evaluation_output` | Agent output directory |
+
+### Troubleshooting
+
+**Server fails to load model**: Ensure `finetuning_output/merged/` contains `config.json`, `model.safetensors`, and `tokenizer.json`. If using a different path, set `MODEL_PATH`.
+
+**CUDA out of memory**: The 1.5B model needs ~3 GB VRAM in float16. Close other GPU processes or use `CUDA_VISIBLE_DEVICES=0` to select a specific GPU.
+
+**Agent produces garbled tool calls**: The 1.5B model sometimes generates garbled special tokens instead of proper `<tool_call>` tags. The inference server includes fallback parsing for bare function calls — this is handled automatically.
+
+**Spider databases not found**: Run `bash scripts/setup_data.sh` to download Spider 1.0. The script downloads from Google Drive via `gdown`.
+
 ## Limitations
 
 - Trained and evaluated only on Spider 1.0 (SQLite databases)
 - Context window limited to 2,048 tokens during training
 - The 1.5B model may generate garbled special tokens instead of proper `<tool_call>` tags — the inference server includes fallback parsing for bare function calls
 - Performance on out-of-distribution databases (different schemas/domains) has not been extensively tested
+- This is an in-distribution evaluation; real-world performance on unseen databases will likely be lower
 
 ## License