README.md

---
base_model: Qwen/Qwen2.5-Coder-3B-Instruct
library_name: peft
pipeline_tag: text-generation
license: apache-2.0
tags:
- lora
- sft
- text-to-sql
- peft
---

# black-swan SQL 3B (LoRA)

Writes DuckDB SQL from a schema and a question. Half of a two-stage text-to-PyRel
pipeline: this adapter emits SQL, and a deterministic transpiler converts it to
[PyRel](https://github.com/maxdemarzi/swan). Trained by the Black Swan project.

`r=32`, `alpha=64`, LoRA over `Qwen/Qwen2.5-Coder-3B-Instruct`.

## The prompt is part of the model

Generation must use `ARCTIC_SYSTEM`, shipped here as `system_prompt.txt`:

> You are a data science expert. Below, you are provided with a database schema and a
> natural language question. Your task is to understand the schema and generate a valid
> SQL query to answer the question.

with the Arctic user template (`engine`, `db_details`, `question`, `output_format`),
`output_format` = the terse fenced-SQL block, **not** the `<think>`/`<answer>`
reasoning scaffold. The rung is `duckdb-values`: evidence, comments and sample
values on, reasoning off.

A different system prompt is not a small degradation. The sibling PyRel adapter
records that dropping its own system message makes the model emit a different
language entirely, which reads as a broken model rather than a missing message.

| | |
|---|---|
| decoding | greedy, `temperature=0` |
| `max_new_tokens` | 4096 at evaluation — median output is 165 characters, but the long tail reached 13,598 |
| extractor | first ```` ```sql ```` fence, then strip the leading `sql` tag |

Because reasoning is off there is exactly one fence, so first and last coincide.
That distinction matters for reasoning models on this same pipeline, where taking
the first fence scores a draft the model went on to correct.

## Measured

BIRD holdout, 428 scoreable questions, paired against the same 3B fine-tuned to
emit PyRel directly:

| arm | correct | | paired vs PyRel |
|---|---|---|---|
| 3B → PyRel direct | 165/428 | 38.6% | — |
| **3B → SQL → transpiler** (this adapter) | **179/428** | **41.8%** | +60/−46, p = 0.21 |
| routed: SQL where it fires, PyRel where it declines | 206/428 | 48.1% | +60/−19, p = 4.2e-06 |

**The head-to-head is not significant.** This adapter is not measurably better
than emitting PyRel directly; the gain is in routing, because the two arms fail on
different questions. Deploy it as one arm of a pair, or accept 41.8% knowing 33%
of questions get no program at all.

Decomposition: SQL accuracy 44.3% × transpiler carry 94.2% = 41.7% against a
measured 41.8%. Zero malformed outputs in 445 generations.

## The transpiler declines a third of the time

Re-running all 445 predictions through `synthesize_ast`:

```
302  admitted   (ast_join 165, ast_single 54, ast_join_n 36, ast_bound 26, +6 patterns)
143  no template
  0  raised
```

`no template` is a transpiler gap, not a model error — the SQL may be perfectly
correct. Each one is a deterministic, permanently fixable Python bug, which is the
improvement path this arm has and a direct-to-PyRel model does not.

**`sqlglot` is a hard dependency and its absence is silent.** `synthesize_ast`
catches the `ImportError` and returns `None`, which is also how it signals a
decline — so a deployment without sqlglot declines 100% of requests and looks like
a transpiler that cannot handle your queries. Verified: it returned 60 of 60
`no template` on a machine that was simply missing the package.

## Provenance

Trained on BIRD train only. Evaluated on the four BIRD dev databases held out from
training (`california_schools`, `debit_card_specializing`, `superhero`,
`thrombosis_prediction`).

The recipe is pinned to an RTX 5090 deliberately: an identical recipe on different
hardware has moved an adapter by 11 questions on this project, which is larger than
the effect being measured here.