maxdemarzi/black-swan-sql3b-lora

black-swan SQL 3B (LoRA)

Writes DuckDB SQL from a schema and a question. Half of a two-stage text-to-PyRel pipeline: this adapter emits SQL, and a deterministic transpiler converts it to PyRel. Trained by the Black Swan project.

r=32, alpha=64, LoRA over Qwen/Qwen2.5-Coder-3B-Instruct.

The prompt is part of the model

Generation must use ARCTIC_SYSTEM, shipped here as system_prompt.txt:

You are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question.

with the Arctic user template (engine, db_details, question, output_format), output_format = the terse fenced-SQL block, not the <think>/<answer> reasoning scaffold. The rung is duckdb-values: evidence, comments and sample values on, reasoning off.

A different system prompt is not a small degradation. The sibling PyRel adapter records that dropping its own system message makes the model emit a different language entirely, which reads as a broken model rather than a missing message.

decoding	greedy, temperature=0
max_new_tokens	4096 at evaluation — median output is 165 characters, but the long tail reached 13,598
extractor	first ```sql fence, then strip the leading sql tag

Because reasoning is off there is exactly one fence, so first and last coincide. That distinction matters for reasoning models on this same pipeline, where taking the first fence scores a draft the model went on to correct.

Measured

BIRD holdout, 428 scoreable questions, paired against the same 3B fine-tuned to emit PyRel directly:

arm	correct		paired vs PyRel
3B → PyRel direct	165/428	38.6%	—
3B → SQL → transpiler (this adapter)	179/428	41.8%	+60/−46, p = 0.21
routed: SQL where it fires, PyRel where it declines	206/428	48.1%	+60/−19, p = 4.2e-06

The head-to-head is not significant. This adapter is not measurably better than emitting PyRel directly; the gain is in routing, because the two arms fail on different questions. Deploy it as one arm of a pair, or accept 41.8% knowing 33% of questions get no program at all.

Decomposition: SQL accuracy 44.3% × transpiler carry 94.2% = 41.7% against a measured 41.8%. Zero malformed outputs in 445 generations.

The transpiler declines a third of the time

Re-running all 445 predictions through synthesize_ast:

302  admitted   (ast_join 165, ast_single 54, ast_join_n 36, ast_bound 26, +6 patterns)
143  no template
  0  raised

no template is a transpiler gap, not a model error — the SQL may be perfectly correct. Each one is a deterministic, permanently fixable Python bug, which is the improvement path this arm has and a direct-to-PyRel model does not.

sqlglot is a hard dependency and its absence is silent. synthesize_ast catches the ImportError and returns None, which is also how it signals a decline — so a deployment without sqlglot declines 100% of requests and looks like a transpiler that cannot handle your queries. Verified: it returned 60 of 60 no template on a machine that was simply missing the package.

Provenance

Trained on BIRD train only. Evaluated on the four BIRD dev databases held out from training (california_schools, debit_card_specializing, superhero, thrombosis_prediction).

The recipe is pinned to an RTX 5090 deliberately: an identical recipe on different hardware has moved an adapter by 11 questions on this project, which is larger than the effect being measured here.