Update README.md

README.md

@@ -1,3 +1,270 @@
 ---
+base_model: Qwen/Qwen3.5-2B
+language:
+- en
 license: apache-2.0
+pipeline_tag: text-generation
+library_name: transformers
+tags:
+- text-to-sql
+- sql
+- sqlite
+- bird
+- self-verification
+- distillation
+- chain-of-thought
+model-index:
+- name: flash-sqlite
+  results:
+  - task:
+      type: text-to-sql
+      name: Text-to-SQL
+    dataset:
+      type: bird-bench
+      name: BIRD (hidden test set)
+    metrics:
+    - type: execution_accuracy
+      value: 73.73
+      name: Execution Accuracy (EX)
+    - type: soft_f1
+      value: 74.63
+      name: Soft F1
+    - type: r_ves
+      value: 67.17
+      name: R-VES
 ---
+
+# flash-sqlite
+
+**A 2B text-to-SQL model that writes SQL and checks its own candidates. It scores 73.73 EX on the hidden BIRD test set.**
+
+flash-sqlite is [Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B), fully fine-tuned for SQLite text-to-SQL. One set of weights does two jobs:
+
+- **Generator:** reasons step by step about the schema, then writes the SQL.
+- **Judge:** given the schema, the question, a candidate query and its execution result, answers *Yes* or *No*. The probability of *Yes* is used to pick among the candidates.
+
+Both roles use the same checkpoint. There is no auxiliary model, no embedding model and no external API. Under the BIRD organisers' ruling, this qualifies for the **single-model track**.
+
+**Key facts**
+
+- **No reinforcement learning:**
+  - flash-sqlite is trained with supervised fine-tuning only, on execution-verified traces.
+  - No RL, GRPO, DPO or reward model is involved.
+  - Several strong entries on the leaderboard, such as Arctic-Text2SQL-R1 and Databricks RLVR, are trained with RL.
+- **Reasoning mode disabled:**
+  - The model runs in Qwen3.5's non-thinking mode (`enable_thinking=False`), for generation and for judging.
+  - It writes a short structured analysis followed by the SQL, with no long hidden reasoning chain.
+  - The judge outputs a single token.
+- **One model, one GPU:** the same 2B checkpoint generates and verifies, with no external model or API.
+
+## Official BIRD results (hidden test set)
+
+The BIRD team scored the model on the hidden test set: 32 candidates per question plus self-judge, on a single A100 80 GB.
+
+| metric | simple | moderate | challenging | **total** |
+|---|---|---|---|---|
+| Execution Accuracy (EX) | 80.19 | 71.53 | 56.49 | **73.73** |
+| Soft F1 | | | | **74.63** |
+| R-VES | | | | **67.17** |
+
+### How it compares
+
+**At 2B parameters, flash-sqlite scores higher on the BIRD test set than every frontier LLM baseline on the leaderboard**:
+- GPT-5.5, Claude Opus 4.6, Claude 4.5 Sonnet, Qwen3-Coder-480B, GLM-4.7, DeepSeek-R1 and Kimi-K2-Thinking;
+- and specialised open models up to 32B.
+
+On the single-model leaderboard it is ahead of 50 of the 66 other entries. On the overall leaderboard, which includes multi-model pipelines and proprietary systems, it is ahead of more than 90.
+
+All scores are **BIRD test-set execution accuracy**. Markers show how many candidates each entry samples per question, using the leaderboard's own categories:
+
+- no marker: one candidate;
+- \* **Few:** 2–7 candidates;
+- \*\* **Many:** 8–32 candidates.
+
+**Frontier and large general-purpose LLMs** (single-model leaderboard):
+
+| model | provider | size | test EX |
+|---|---|---|---|
+| **flash-sqlite (this model)** | — | **2B** | **73.73\*\*** |
+| GPT-5.5-xhigh | OpenAI | proprietary | 72.55 |
+| Claude Opus 4.6 | Anthropic | proprietary | 70.15 |
+| Qwen3-Coder-480B-A35B | Alibaba | 480B MoE | 68.14 |
+| Claude 4.5 Sonnet | Anthropic | proprietary | 66.85 |
+| GLM-4.7 | Zhipu AI | large open-weight | 62.94 |
+| DeepSeek-R1 | DeepSeek | 671B MoE | 60.93 |
+| Kimi-K2-Thinking | Moonshot AI | ~1T MoE | 59.87 |
+| DeepSeek (2024 baseline) | DeepSeek | 236B MoE | 56.68 |
+| Mistral (2024 baseline) | Mistral AI | 123B | 55.84 |
+| GPT-4 | OpenAI | proprietary | 54.89 |
+
+**Specialised text-to-SQL models** (single-model leaderboard):
+
+| model | size | test EX |
+|---|---|---|
+| Native mini | 4B | 76.86\*\* |
+| Arctic-Text2SQL-R1-32B (Snowflake) | 32B | 73.84\* |
+| **flash-sqlite (this model)** | **2B** | **73.73\*\*** |
+| Databricks RLVR 32B | 32B | 73.56 |
+| Arctic-Text2SQL-R1-14B (Snowflake) | 14B | 72.22\* |
+| Arctic-Text2SQL-R1-7B (Snowflake) | 7B | 70.43\* |
+| XiYanSQL-QwenCoder-32B (Alibaba) | 32B | 69.03\* |
+
+flash-sqlite samples 32 candidates and selects one with its own judge (see [How it works](#how-it-works)). All figures are from the public leaderboard at [bird-bench.github.io](https://bird-bench.github.io), taken at the time of writing. The earlier version of this model (2B, majority vote over 7 candidates) scored 68.47\*.
+
+## Speed and efficiency
+
+32 candidates plus a judge sounds expensive. With a 2B model and the right serving setup, it is not.
+
+- **One GPU, no API:** the bf16 weights need about 4.5 GB. Everything runs locally on a single GPU: generation, execution and judging. No data leaves the machine.
+- **32 candidates in parallel:**
+  - The 32 samples per question come from one request (`n=32`), and vLLM batches them together.
+  - With **prefix caching**, the long schema prompt is prefilled once and shared by all 32 samples.
+  - The judge prompts of the same question also share their schema prefix.
+- **Throughput:** about **6,400–7,600 output tokens/s** on one 96 GB RTX 6000 MaxQ GPU, which is 8–11 finished candidates per second.
+- **The judge is almost free:**
+  - It reads one token (P(Yes)) per result group instead of generating text.
+  - On BIRD dev, **38% of questions needed no judge call at all**, because all executable candidates agreed.
+  - On average there were **2.45 judge calls per question** (3,763 calls for 949 contested questions).
+  - The whole judge pass over BIRD dev took **about 30 seconds**.
+- **End to end:** all 1,534 BIRD dev questions take **about 95 minutes**. That covers 49,000 candidates (about 85 minutes), plus executing them and judging (about 10 minutes), or about 3.7 s per question.
+- **Greedy mode:** for latency-sensitive use, a single greedy answer still reaches 57.05 EX on dev. Majority voting over 32 candidates without the judge reaches 62.92.
+
+## BIRD dev results
+
+These come from the dev revision `bird-sql-dev-1106`: 1,532 questions with executable gold SQL, scored by execution accuracy with set comparison. All three rows come from the same generation run.
+
+| inference | EX | simple | moderate | challenging |
+|---|---|---|---|---|
+| **32 candidates + self-judge** | **67.49** | 78.11 | 63.66 | 35.22 |
+| 16 candidates + self-judge | 66.91 | 77.65 | 63.21 | 33.91 |
+| majority vote over 32, no judge | 62.92 | 74.27 | 56.88 | 32.17 |
+| greedy, single answer | 57.05 | | | |
+
+On dev, the self-judge adds **4.6 EX** over plain majority voting with the same 32 candidates. Its gain grows with the number of candidates.
+
+## How it works
+
+For each question:
+
+1. **Generate** 32 candidates at temperature 1.0, top_p 0.95, top_k 20, with a 2,048-token limit and non-thinking mode.
+2. **Execute** each candidate read-only against the database, with a 30 s timeout.
+   - Candidates are grouped by their result set, compared as sets with `1 == 1.0`.
+   - Candidates that fail to run are dropped.
+   - A group's votes are the number of candidates in it.
+3. **Self-judge** — only if more than one result group remains.
+   - The same model gets one prompt per group: the schema and question, the group's SQL, and a 5-row preview of its result.
+   - P(Yes) is read from the first generated token, restricted to *Yes* / *No* and renormalised.
+4. **Select** the group with the highest **votes × P(Yes)**.
+
+## Usage
+
+The model expects the OmniSQL-style prompt shown below and answers with its reasoning followed by a fenced SQL block. Always take the **last** ```sql block. Use **non-thinking mode** (`enable_thinking=False`).
+
+### Prompt format
+
+```text
+Task Overview:
+You are a data science expert. Below, you are provided with a database schema and a natural language question. Your task is to understand the schema and generate a valid SQL query to answer the question.
+
+Database Engine:
+SQLite
+
+Database Schema:
+{db_details}
+This schema describes the database's structure, including tables, columns, primary keys, foreign keys, and any relevant relationships or constraints.
+
+Question:
+{question}
+
+Hint: {evidence}
+
+Instructions:
+- Make sure you only output the information that is asked in the question. If the question asks for a specific column, make sure to only include that column in the SELECT clause, nothing more.
+- The generated query should return all of the information asked in the question without any missing or extra information.
+- Before generating the final SQL query, please think through the steps of how to write the query.
+
+Output Format:
+In your answer, please enclose the generated SQL query in a code block:
+```sql
+-- Your SQL query
+```
+
+Take a deep breath and think step by step to find the correct SQL query.
+```
+
+`{db_details}` contains, for each table:
+- the `CREATE TABLE` statement;
+- a comment block of column descriptions (where the database ships them);
+- a comment block with 3 example rows.
+
+### Generation (vLLM)
+
+```python
+import re
+from transformers import AutoTokenizer
+from vllm import LLM, SamplingParams
+
+model_id = "AlioLeuchtmann/flash-sqlite"
+tok = AutoTokenizer.from_pretrained(model_id)
+llm = LLM(model=model_id, dtype="bfloat16")
+
+def chat(prompt):
+    return tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False,
+                                   add_generation_prompt=True, enable_thinking=False)
+
+def last_sql(text):
+    blocks = re.findall(r"```sql\s*(.*?)```", text, re.S)
+    return blocks[-1].strip() if blocks else ""
+
+prompt = ...  # the prompt above, filled with the schema, question and hint
+
+# a single answer
+greedy = llm.generate([chat(prompt)], SamplingParams(temperature=0.0, max_tokens=2048))
+print(last_sql(greedy[0].outputs[0].text))
+
+# 32 candidates for voting + self-judge
+samples = llm.generate([chat(prompt)], SamplingParams(n=32, temperature=1.0, top_p=0.95, top_k=20, max_tokens=2048))
+candidates = [last_sql(o.text) for o in samples[0].outputs]
+```
+
+### Self-judge
+
+Execute the candidates and group them by result set. Then score one representative query per group with the judge prompt below. Use `max_tokens=1`, `allowed_token_ids=[id("Yes"), id("No")]` and `logprobs=2`, and renormalise P(Yes) over the two tokens.
+
+```text
+Task Overview:
+You are a data science expert. Below, you are provided with a database schema, a natural language question, a candidate SQL query and the result of executing it. Your task is to decide whether the candidate SQL query answers the question correctly.
+
+Database Engine:
+SQLite
+
+{the "Database Schema:" ... "Question:" part of the generation prompt}
+
+Candidate SQL:
+```sql
+{sql}
+```
+
+Execution result ({nrows} rows, first 5 shown):
+{preview}
+
+Does the candidate SQL query answer the question correctly, returning exactly the requested information? Answer with one word: Yes or No.
+```
+
+Pick the group with the highest `votes × P(Yes)`. If two groups tie, prefer the one with more votes, then the one that appeared first.
+
+### Tested environment
+
+`vllm==0.25.1` and `transformers==5.14.1`, which support the Qwen3.5 hybrid architecture. A 2B model in bf16 needs about 4.5 GB of GPU memory for the weights. All 1,534 BIRD dev questions (49k samples plus judging) take about 95 minutes on a single 96 GB workstation GPU.
+
+## Intended use and limitations
+
+- **Built for:** SQLite text-to-SQL on analytical questions in the style of BIRD, in English, optionally with an evidence hint. It is not a general chat model.
+- **The judge needs execution:** it scores a query *together with its result*, so the database has to be available at inference time. It has not been calibrated as a standalone classifier. Use it in the votes × P(Yes) combination it was tuned for.
+- **Dialects:** trained and evaluated only on SQLite. Other dialects, such as PostgreSQL or BigQuery, will need adaptation.
+- **Hard questions:** accuracy on the challenging tier (56.5 EX on test) is still well below the simple tier. Multi-step queries remain the main weakness of a 2B model.
+- **Safety:** generated SQL should run read-only. This model was never trained to avoid writing or destructive statements.
+
+## Acknowledgements
+
+Built on Qwen3.5-2B by the Qwen team. Evaluated by the BIRD team ([bird-bench.github.io](https://bird-bench.github.io)). Developed by Alio Leuchtmann.