README.md

metadata

base_model:
  - griffith-bigdata/Qwen-2.5-Coder-3B-SQL-Writer
license: apache-2.0
language:
  - en
tags:
  - text-to-sql
  - bird
  - grpo
  - finer-sql
  - code
library_name: transformers
pipeline_tag: text-generation

FINER-SQL-3B-BIRD

Trained from griffith-bigdata/Qwen-2.5-Coder-3B-SQL-Writer using GRPO with two dense rewards from the FINER-SQL paper:

🧠 Memory Reward — aligns reasoning with verified traces ⚙️ Atomic Reward — measures operation-level SQL overlap

✅ 67.5% Execution Accuracy on BIRD Dev when training only on BIRD train. Inference runs on a single 12–24 GB GPU.

📄 See other models: https://huggingface.co/collections/griffith-bigdata/finer-sql 📄 GitHub: https://github.com/thanhdath/finer-sql/tree/main

---

Comparison: FINER-SQL-3B-BIRD vs FINER-SQL-3B-Spider

Both models share the same Qwen-2.5-Coder-3B-SQL-Writer base. They differ only in the GRPO fine-tuning dataset (BIRD train vs Spider train).

Model	BIRD Dev (n=30, vav)	Spider Dev (n=30, vav, +agg_hint)	When to use
FINER-SQL-3B-BIRD (this model)	67.54% ✅	83.8%	Production BIRD; cross-domain SQL where train is BIRD-like
FINER-SQL-3B-Spider	63.04%	85.1% ✅	Production Spider / spider-style schemas

Why two checkpoints? BIRD and Spider use different SQL annotation conventions (BIRD: verbose, evidence-based, alias-heavy; Spider: terse, aggregate-first GROUP BY, exact ORDER BY direction). A single model trained on either dataset specialises to its annotations and loses ~1–4 pp on the other benchmark. We tried joint training and inference-time prompt tricks; they bridge most of the gap but the last 1–2 pp on each benchmark requires the dataset-specific checkpoint.

---

Inference

Quick start (vLLM)

from vllm import LLM, SamplingParams

llm = LLM(
    model="griffith-bigdata/FINER-SQL-3B-BIRD",
    dtype="bfloat16",
    max_model_len=4096,
    gpu_memory_utilization=0.85,
)

system_prompt = """You are a meticulous SQL expert. Generate a single, correct SQL query for the user question and the provided database schema.
Follow this exact response format:

Rules:
- Output exactly one SQL statement.
- The SQL must be executable on SQLite.
- Do not include any explanatory text.
- Output one SQL statement only. Do not include any extra text, tags, or code fences."""

# Generate n=30 candidates with t=1.0, then majority-vote with VAV
sampling = SamplingParams(n=30, temperature=1.0, max_tokens=2048)

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": f"Database Schema:\n{schema}\n\nQuestion: {question}\n\nEvidence: {evidence}"},
]
output = llm.chat(messages, sampling)
candidate_sqls = [c.text.split("</think>")[-1].strip() for c in output[0].outputs]
# Then run majority voting (vav) — see the GitHub repo for the selector

Recommended evaluation pipeline

1. Generate n=30 SQL candidates per question with temperature=1.0
2. Execute each candidate against the database, collect result tuples
3. Group candidates by execution result; pick the candidate from the largest non-empty success group that does not match the all-zero or empty pattern (value-aware voting, "vav")
4. Score against gold SQL with the official BIRD evaluator

This pipeline gives 67.54% BIRD Dev EX. See evaluation/ in the repo for the reference implementation.

---

Detailed BIRD Dev results (n=30, vav, V2 prompts)

Difficulty	Count	Execution Accuracy
Simple	925	74.16%
Moderate	464	58.41%
Challenging	145	54.48%
All	1534	67.54%

Recall@30 (any-correct rate among 30 candidates): 82.2%.

---

Cross-benchmark: this model on Spider Dev

If you must use a single model across both benchmarks, this checkpoint plus a one-line system-prompt addition gets close to the Spider specialist:

Setup	Spider Official EX
Default (no prompt change)	83.2%
+ "list aggregates BEFORE grouping columns" rule	83.8%
FINER-SQL-3B-Spider (specialist)	85.1%

To enable the Spider-friendly mode, append to the system prompt:

- When using GROUP BY, list aggregate functions (COUNT, SUM, AVG, MIN, MAX) in the SELECT clause BEFORE the grouping column(s).

---

Training

Parameter	Value
Base model	griffith-bigdata/Qwen-2.5-Coder-3B-SQL-Writer
Algorithm	GRPO
Train data	BIRD train (V2 prompts, top-30 GRAST schema)
Learning rate	8e-6
Num generations per prompt	32
Gradient accumulation	32
Max completion length	2048
Max prompt length	4096
Temperature (rollout)	1.0
Selection during eval	vav (value-aware voting)
Rewards	Execution + Atomic + Memory + Format
GPU	1× A6000 48 GB

---

License

Inherits the base model's license (Apache 2.0). Not for medical, legal, or other safety-critical autonomous decision-making.

---

Citation

@article{finer-sql-2026,
  title  = {FINER-SQL: Fine-grained reasoning rewards for small Text-to-SQL models},
  author = {Thanh Dat and others},
  year   = {2026},
}