FINAL-Bench
/

Darwin-180B-NL2SQL

@@ -1,75 +1,86 @@
----
-license: apache-2.0
-language:
-  - en
-tags:
-  - text-to-sql
-  - nl2sql
-  - bird
-  - sqlite
-  - darwin
-pipeline_tag: text-generation
----
-# Darwin-180B-NL2SQL
-<p align="center">
-  <img src="https://img.shields.io/badge/BIRD_Mini--Dev_SQLite-83.80_EX-gold?style=for-the-badge" alt="BIRD Mini-Dev SQLite 83.80">
-  <a href="https://huggingface.co/FINAL-Bench/Darwin-180B-RSI"><img src="https://img.shields.io/badge/Model-Darwin--180B--RSI-blue?style=for-the-badge" alt="Darwin-180B-RSI"></a>
-  <a href="https://huggingface.co/FINAL-Bench"><img src="https://img.shields.io/badge/Org-FINAL--Bench-black?style=for-the-badge" alt="FINAL-Bench"></a>
-</p>
-Darwin-180B-NL2SQL is VIDRAFT's text-to-SQL system built on [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI). It turns a natural-language question into a SQLite query, runs it against the database and revises the query from what the execution shows, then returns one final query per question.
-This repository holds the system's predictions on BIRD Mini-Dev (SQLite) and the script used to score them. The model weights are those of Darwin-180B-RSI; no weights were changed or trained for this result.
-## Result
-**BIRD Mini-Dev, SQLite, 500 questions: 83.80% execution accuracy (419 / 500).**
-| Difficulty | Correct | Accuracy |
-|---|---|---|
-| Challenging | 384 / 460 | 83.5% |
-| Moderate | 29 / 32 | 90.6% |
-| Simple | 6 / 8 | 75.0% |
-| **All** | **419 / 500** | **83.80%** |
-- Data: the current canonical release on Hugging Face, [`birdsql/bird_mini_dev`](https://huggingface.co/datasets/birdsql/bird_mini_dev), release "Mini-Dev 1024", `data/mini_dev_sqlite.jsonl` (sha256 `926af2a0d1b47e5d0d15a7aa8a34d4721007f4550a881555b6a89496b54a2606`), with the SQLite databases from `minidev.zip`.
-- Metric: strict execution accuracy. A prediction is correct when the set of rows it returns equals the set of rows the reference SQL returns. Each predicted query may run for 60 seconds.
-- Measured on 29 October 2026 (KST). The score was computed twice on an idle machine and was identical both times.
-## Evaluation conditions
-| Condition | Setting |
-|---|---|
-| Reference (gold) SQL used at any stage of inference | **No** |
-| Oracle knowledge beyond the question and evidence given | No |
-| Fine-tuning on BIRD | No |
-| Final candidates per question | 1 |
-| Database access during inference | Read-only execution of the system's own queries |
-The question and evidence in each item are the only task inputs. The reference SQL was used only to score the final predictions.
-## Files
-| File | Contents |
-|---|---|
-| `predict_mini_dev_darwin-180b-nl2sql_SQLite.json` | Predictions in the BIRD Mini-Dev submission format (`SQL\t----- bird -----\tdb_id`, keyed by question ID) |
-| `predictions.jsonl` | The same predictions, one JSON object per line |
-| `score_ex.py` | Strict execution-accuracy scorer used for the numbers above |
-To reproduce the score, place `mini_dev_sqlite.jsonl` next to `score_ex.py` and run:
-```bash
-python score_ex.py predictions.jsonl path/to/MINIDEV/dev_databases
-```
-## About the system
-The system pairs Darwin-180B-RSI with VIDRAFT's NL2SQL verification engine, which checks each generated query against the live database before an answer is returned. The engine is VIDRAFT's own implementation and is not released.
-A public demo of VIDRAFT's NL2SQL engine (Korean and English, read-only, synthetic data) is available at [final-bench-nl2sql-demo.hf.space](https://final-bench-nl2sql-demo.hf.space/).
-## Contact
-VIDRAFT · [vidraft.net](https://vidraft.net) · vidraft@mail.vidraft.net

+---
+license: apache-2.0
+language:
+  - en
+tags:
+  - text-to-sql
+  - nl2sql
+  - bird
+  - sqlite
+  - darwin
+pipeline_tag: text-generation
+base_model: FINAL-Bench/Darwin-180B-RSI
+---
+# Darwin-180B-NL2SQL
+<p align="center">
+  <img src="https://img.shields.io/badge/BIRD_Mini--Dev_SQLite-83.80_EX-gold?style=for-the-badge" alt="BIRD Mini-Dev SQLite 83.80">
+  <a href="https://huggingface.co/FINAL-Bench/Darwin-180B-RSI"><img src="https://img.shields.io/badge/Model-Darwin--180B--RSI-blue?style=for-the-badge" alt="Darwin-180B-RSI"></a>
+  <a href="https://huggingface.co/FINAL-Bench"><img src="https://img.shields.io/badge/Org-FINAL--Bench-black?style=for-the-badge" alt="FINAL-Bench"></a>
+</p>
+Darwin-180B-NL2SQL is VIDRAFT's text-to-SQL system built on [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI). It turns a natural-language question into a SQLite query, runs it against the database and revises the query from what the execution shows, then returns one final query per question.
+This repository holds the system's predictions on BIRD Mini-Dev (SQLite) and the script used to score them.
+> **Weights:** the model is [Darwin-180B-RSI](https://huggingface.co/FINAL-Bench/Darwin-180B-RSI), unchanged. Download the weights from that repository; no weights were changed or trained for this result.
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+repo = "FINAL-Bench/Darwin-180B-RSI"  # weights for Darwin-180B-NL2SQL
+tok = AutoTokenizer.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
+```
+## Result
+**BIRD Mini-Dev, SQLite, 500 questions: 83.80% execution accuracy (419 / 500).**
+| Difficulty | Correct | Accuracy |
+|---|---|---|
+| Challenging | 384 / 460 | 83.5% |
+| Moderate | 29 / 32 | 90.6% |
+| Simple | 6 / 8 | 75.0% |
+| **All** | **419 / 500** | **83.80%** |
+- Data: the current canonical release on Hugging Face, [`birdsql/bird_mini_dev`](https://huggingface.co/datasets/birdsql/bird_mini_dev), release "Mini-Dev 1024", `data/mini_dev_sqlite.jsonl` (sha256 `926af2a0d1b47e5d0d15a7aa8a34d4721007f4550a881555b6a89496b54a2606`), with the SQLite databases from `minidev.zip`.
+- Metric: strict execution accuracy. A prediction is correct when the set of rows it returns equals the set of rows the reference SQL returns. Each predicted query may run for 60 seconds.
+- Measured on 29 October 2026 (KST). The score was computed twice on an idle machine and was identical both times.
+## Evaluation conditions
+| Condition | Setting |
+|---|---|
+| Reference (gold) SQL used at any stage of inference | **No** |
+| Oracle knowledge beyond the question and evidence given | No |
+| Fine-tuning on BIRD | No |
+| Final candidates per question | 1 |
+| Database access during inference | Read-only execution of the system's own queries |
+The question and evidence in each item are the only task inputs. The reference SQL was used only to score the final predictions.
+## Files
+| File | Contents |
+|---|---|
+| `predict_mini_dev_darwin-180b-nl2sql_SQLite.json` | Predictions in the BIRD Mini-Dev submission format (`SQL\t----- bird -----\tdb_id`, keyed by question ID) |
+| `predictions.jsonl` | The same predictions, one JSON object per line |
+| `score_ex.py` | Strict execution-accuracy scorer used for the numbers above |
+To reproduce the score, place `mini_dev_sqlite.jsonl` next to `score_ex.py` and run:
+```bash
+python score_ex.py predictions.jsonl path/to/MINIDEV/dev_databases
+```
+## About the system
+The system pairs Darwin-180B-RSI with VIDRAFT's NL2SQL verification engine, which checks each generated query against the live database before an answer is returned. The engine is VIDRAFT's own implementation and is not released.
+A public demo of VIDRAFT's NL2SQL engine (Korean and English, read-only, synthetic data) is available at [final-bench-nl2sql-demo.hf.space](https://final-bench-nl2sql-demo.hf.space/).
+## Contact
+VIDRAFT · [vidraft.net](https://vidraft.net) · vidraft@mail.vidraft.net