Add model card with training details and benchmark results

README.md

@@ -0,0 +1,162 @@
+---
+language:
+- en
+license: apache-2.0
+base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
+tags:
+- text-to-sql
+- pipe-sql
+- sqlglot
+- tool-calling
+- qwen2
+datasets:
+- spider
+pipeline_tag: text-generation
+model-index:
+- name: pipe-sql-1.5b
+  results:
+  - task:
+      type: text-to-sql
+      name: Text-to-SQL
+    dataset:
+      type: spider
+      name: Spider 1.0 Dev
+    metrics:
+    - type: execution_accuracy
+      value: 60.66
+      name: Execution Accuracy
+---
+
+# Pipe SQL 1.5B
+
+A fine-tuned [Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) model for generating **Pipe SQL** through multi-turn tool-calling conversations.
+
+## What is Pipe SQL?
+
+Pipe SQL is a more readable SQL syntax that uses the `|>` (pipe) operator to chain operations in a linear, top-to-bottom flow:
+
+```sql
+FROM employees
+|> WHERE department = 'Engineering'
+|> AGGREGATE AVG(salary) AS avg_salary GROUP BY level
+|> ORDER BY avg_salary DESC
+```
+
+This is transpiled to standard SQL via [sqlglot](https://github.com/tobymao/sqlglot), an open-source SQL parser and transpiler.
+
+## Model Details
+
+| Property | Value |
+|----------|-------|
+| **Base Model** | Qwen2.5-Coder-1.5B-Instruct |
+| **Architecture** | Qwen2ForCausalLM |
+| **Parameters** | 1.5B |
+| **Hidden Size** | 1536 |
+| **Layers** | 28 |
+| **Attention Heads** | 12 (2 KV heads) |
+| **Context Length** | 2048 tokens (training) |
+
+## Training
+
+The model was fine-tuned using **QLoRA** on multi-turn tool-calling conversations for text-to-SQL generation.
+
+### Training Data
+
+Conversations were generated from the [Spider 1.0](https://yale-lily.github.io/spider) training set, where each conversation follows an agentic workflow:
+1. **Explore** the database schema using `list_tables`, `describe_table`, and `sample_data` tools
+2. **Write** pipe SQL queries using `execute_pipe_sql` and `validate_pipe_sql` tools
+3. **Iterate** based on execution results until the query is correct
+
+### Hyperparameters
+
+| Parameter | Value |
+|-----------|-------|
+| **Method** | QLoRA (4-bit NF4) |
+| **LoRA rank** | 16 |
+| **LoRA alpha** | 32 |
+| **LoRA dropout** | 0.05 |
+| **Target modules** | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
+| **Epochs** | 3 |
+| **Learning rate** | 2e-4 |
+| **LR scheduler** | Cosine |
+| **Warmup ratio** | 0.05 |
+| **Batch size** | 2 (per device) |
+| **Gradient accumulation** | 8 steps |
+| **Weight decay** | 0.01 |
+| **Loss** | Assistant-only (tool responses masked) |
+
+## Evaluation Results
+
+Evaluated on the **Spider 1.0 dev set** (1,034 questions) using an agentic benchmark pipeline with execution accuracy.
+
+| Metric | Value |
+|--------|-------|
+| **Execution Accuracy** | **60.66%** (626 / 1,032) |
+| **Prediction Rate** | 99.7% (1,031 / 1,034) |
+
+### Status Breakdown
+
+| Status | Count | Percentage |
+|--------|-------|------------|
+| Match | 626 | 60.5% |
+| Mismatch | 209 | 20.2% |
+| Execution Error | 170 | 16.4% |
+| Transpile Error | 24 | 2.3% |
+| No Prediction | 3 | 0.3% |
+| Gold Error (excluded) | 2 | 0.2% |
+
+> **Note**: This is an **in-distribution** evaluation — the model was trained on Spider training data, and the dev set uses the same 20 databases. Gold errors (2 questions where the reference SQL fails) are excluded from the accuracy denominator.
+
+## Tools
+
+The model was trained to use 5 tools in a multi-turn conversation:
+
+| Tool | Description |
+|------|-------------|
+| `list_tables` | List all tables in a database |
+| `describe_table` | Get column names, types, and constraints for a table |
+| `sample_data` | Retrieve sample rows from a table |
+| `execute_pipe_sql` | Execute a pipe SQL query against the database |
+| `validate_pipe_sql` | Validate pipe SQL syntax without executing |
+
+## Usage
+
+### Chat Template
+
+The model uses a custom chat template with `<tool_call>` tags for tool invocations:
+
+```
+<|im_start|>assistant
+Let me explore the database first.
+<tool_call>
+list_tables({"db_id": "concert_singer"})
+</tool_call><|im_end|>
+```
+
+Tool responses are formatted as:
+
+```
+<|im_start|>user
+<tool_response>
+Tables in database 'concert_singer':
+- stadium
+- singer
+- concert
+- singer_in_concert
+</tool_response><|im_end|>
+```
+
+### Inference
+
+For inference with the correct chat template, see the evaluation server code in the [sqlglot repository](https://github.com/nittygritty-zzy/sqlglot/tree/main/evaluation/server).
+
+## Limitations
+
+- Trained and evaluated only on Spider 1.0 (SQLite databases)
+- Context window limited to 2,048 tokens during training
+- The 1.5B model may generate garbled special tokens instead of proper `<tool_call>` tags — the inference server includes fallback parsing for bare function calls
+- Performance on out-of-distribution databases (different schemas/domains) has not been extensively tested
+
+## License
+
+This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0), consistent with the base Qwen2.5-Coder model license.