SQL-R1

SQL-R1 trains Qwen3-4B-Base for complex Text-to-SQL through completion-only supervised fine-tuning and GRPO with verifiable SQLite execution feedback.

This repository contains three LoRA adapters:

Subfolder	Stage	Intended use
sft	Completion-only SFT	Supervised baseline
grpo	SFT + execution-feedback GRPO	Recommended main checkpoint
grpo-hard	Hard-sample curriculum GRPO	Ablation and analysis

All three adapters require Qwen/Qwen3-4B-Base.

Results

Execution accuracy (EX) and execution-valid rate were evaluated on the complete public BIRD Dev and Spider Dev splits using a consistent greedy-decoding and SQLite-execution pipeline.

Checkpoint	BIRD Dev EX	BIRD execution-valid	Spider Dev EX	Spider execution-valid
Qwen3-4B-Base	23.21%	55.61%	55.51%	77.66%
sft	42.37%	87.29%	77.27%	96.62%
grpo	42.37%	87.35%	79.59%	97.00%
grpo-hard	42.50%	88.79%	78.92%	97.20%

The grpo checkpoint improves Spider Dev EX by 2.32 percentage points over SFT while preserving BIRD Dev EX. The curriculum ablation improves BIRD execution validity but shows mild Spider forgetting.

These are project-side development-set measurements rather than hidden-test leaderboard submissions. Systems using database-content retrieval, self-consistency, reranking, or larger proprietary models are not directly comparable.

Training overview

SFT

• Training records: 12,976
• Validation records: 608
• Completion-only loss: prompt tokens are masked and loss is computed only on the target SQL
• LoRA rank/alpha/dropout: 32 / 64 / 0.05
• Trainable parameters: 66.1M (1.62%)
• Precision: BF16

GRPO

• RL training records: 12,885
• Optimization steps: 300
• Candidates per prompt: 4
• Approximate generated rollouts: 1,200
• Reward signals: SQL validity, read-only safety, executability, and execution result equivalence
• Parallel read-only SQLite rollout validation

Gold SQL and execution results are available only to the reward function and are never included in the model prompt.

Hard-sample curriculum ablation

• Curriculum records: 2,522
• BIRD moderate/challenging focus records: 1,891
• Stratified replay records: 631
• Optimization steps: 300
• Validation database overlap: 0

Usage

Choose one of sft, grpo, or grpo-hard. The recommended default is grpo.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "Qwen/Qwen3-4B-Base"
repo_id = "j2521402/SQL-R1"
adapter_subfolder = "grpo"

tokenizer = AutoTokenizer.from_pretrained(
    repo_id,
    subfolder=adapter_subfolder,
)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(
    base_model,
    repo_id,
    subfolder=adapter_subfolder,
).eval()

messages = [
    {
        "role": "system",
        "content": (
            "You are a Text-to-SQL assistant. Given a SQLite database schema, "
            "optional evidence, and a question, return exactly one read-only "
            "SQLite query. Do not include explanations or Markdown fences."
        ),
    },
    {
        "role": "user",
        "content": (
            'Database schema:\nTABLE "singer" ("Singer_ID" INT, "Name" TEXT);\n\n'
            "Question:\nHow many singers do we have?"
        ),
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

completion = output[0, inputs["input_ids"].shape[1]:]
print(tokenizer.decode(completion, skip_special_tokens=True).strip())

Each subfolder also contains its tokenizer configuration and detailed model card.

Limitations

• Training and evaluation focus on English Text-to-SQL and SQLite.
• Prompts contain database schemas and optional evidence but do not automatically retrieve database contents.
• Execution equivalence is a strong but imperfect correctness signal.
• Generated SQL may still be wrong or expensive. Execute it only in a read-only sandbox with time and row limits.
• SQL-R1 is a direct Text-to-SQL model, not a general tool-calling agent.

Code

Data processing, SFT, GRPO reward, training, and evaluation code: j2521402/SQL-R1.