Add model card with eval results

README.md

@@ -1,59 +1,133 @@
 ---
 base_model: google/gemma-3-270m-it
+datasets:
+  - chabab/text2sql-oracle-postgres
 library_name: transformers
-model_name: gemma-3-270m-text2sql-oracle-postgres
+license: apache-2.0
+pipeline_tag: text-generation
 tags:
-- generated_from_trainer
-- sft
-- trl
-- hf_jobs
-licence: license
+  - text-to-sql
+  - sql
+  - oracle
+  - postgresql
+  - gemma3
+  - trl
+  - sft
 ---
 
-# Model Card for gemma-3-270m-text2sql-oracle-postgres
+# gemma-3-270m-text2sql-oracle-postgres
 
-This model is a fine-tuned version of [google/gemma-3-270m-it](https://huggingface.co/google/gemma-3-270m-it).
-It has been trained using [TRL](https://github.com/huggingface/trl).
+[`google/gemma-3-270m-it`](https://huggingface.co/google/gemma-3-270m-it) fine-tuned to turn a
+schema + a natural-language question into **one** dialect-correct SQL statement — Oracle or
+PostgreSQL — with no markdown fences and no commentary.
 
-## Quick start
+At 270M parameters it runs on CPU and quantizes to ~290 MB.
+
+## Results
+
+Held-out test split (60 examples), greedy decoding, normalized exact match against gold SQL:
+
+| Slice | Exact match |
+|---|---|
+| **Overall** | **78.3%** (47/60) |
+| Oracle | 86.7% (39/45) |
+| PostgreSQL | 53.3% (8/15) |
+| easy | 93.3% (14/15) |
+| medium | 69.2% (27/39) |
+| hard | 100% (6/6) |
+
+Per-example predictions are in [`eval_results.json`](./eval_results.json).
+
+Caveats worth knowing before you rely on these numbers:
+
+- The test split is skewed 45 Oracle / 15 PostgreSQL, so the PostgreSQL figure rests on 15
+  examples and has a wide error bar.
+- Exact match is strict. Several "failures" are valid SQL that differs from gold — an extra
+  `LIMIT`, a different but equivalent predicate. Real semantic accuracy is higher than 78.3%.
+- The most common genuine error is dialect leakage: emitting `LIKE` where PostgreSQL gold uses
+  `ILIKE`. If case-insensitive matching matters to you, check that specific pattern.
+- Only the 7 schemas in the training set (hr, sales, banking, inventory, tickets, university,
+  logistics) are represented. Generalization to unseen schemas is untested.
+
+## Usage
+
+The model expects the system prompt naming the dialect, then a `Schema:` block and a `Question:`
+block — the same shape as the training data.
 
 ```python
-from transformers import pipeline
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "chabab/gemma-3-270m-text2sql-oracle-postgres"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    dtype=torch.bfloat16,
+    attn_implementation="eager",  # Gemma-3 needs eager attention for correct generation
+    device_map="auto",
+)
+
+messages = [
+    {"role": "system", "content": "You convert natural language into PostgreSQL SQL. Use only tables and columns from the provided schema. Reply with one SQL statement and nothing else. No markdown fences. No commentary."},
+    {"role": "user", "content": """Schema:
+employees(
+  employee_id INTEGER PK,
+  first_name VARCHAR(50),
+  last_name VARCHAR(50),
+  hire_date DATE,
+  salary NUMERIC(12,2),
+  department_id INTEGER FK->departments.department_id
+)
 
-question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
-generator = pipeline("text-generation", model="chabab/gemma-3-270m-text2sql-oracle-postgres", device="cuda")
-output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
-print(output["generated_text"])
+Question:
+Show the five employees with the largest salary. Return only the SQL."""},
+]
+
+ids = tok.apply_chat_template(messages, add_generation_prompt=True,
+                              return_tensors="pt", return_dict=True).to(model.device)
+out = model.generate(**ids, max_new_tokens=256, do_sample=False)
+print(tok.decode(out[0][ids["input_ids"].shape[-1]:], skip_special_tokens=True).strip())
+# SELECT first_name, last_name, salary FROM employees ORDER BY salary DESC LIMIT 5;
 ```
 
-## Training procedure
+Two things matter for output quality:
+
+- **Use greedy decoding** (`do_sample=False`). The task has one right answer; sampling only adds
+  drift.
+- **Use `attn_implementation="eager"`.** Gemma-3 generates degenerate repeated tokens under the
+  default SDPA path in some configurations.
+
+## GGUF / local inference
 
- 
+Quantized builds for Ollama, LM Studio, and llama.cpp:
+[`chabab/gemma-3-270m-text2sql-oracle-postgres-GGUF`](https://huggingface.co/chabab/gemma-3-270m-text2sql-oracle-postgres-GGUF)
 
+## Training
 
-This model was trained with SFT.
+Full-parameter SFT (no LoRA — the model is small enough to tune end to end) with TRL `SFTTrainer`
+on one L4 GPU, about 10 minutes.
 
-### Framework versions
+| | |
+|---|---|
+| Base | `google/gemma-3-270m-it` |
+| Data | `chabab/text2sql-oracle-postgres` — 684 train / 60 validation / 60 test |
+| Epochs | 5 |
+| Effective batch size | 16 (4 × 4 grad accum) |
+| Learning rate | 5e-5, cosine, 10 warmup steps |
+| Max sequence length | 1024 |
+| Precision | bf16 |
 
-- TRL: 0.24.0
-- Transformers: 4.57.1
-- Pytorch: 2.13.0
-- Datasets: 5.0.1
-- Tokenizers: 0.22.2
+Final metrics: train loss **0.0256**, eval loss **0.0683**, eval token accuracy **98.5%**.
+Eval loss fell monotonically through training with no divergence.
 
-## Citations
+## Limitations
 
+Generated SQL is not validated against a live database. The model can produce syntactically valid
+statements that reference the wrong table or misread the intent — two of the observed test
+failures do exactly that. Review output before executing it, and never run generated SQL against
+production with write permissions.
 
+## License
 
-Cite TRL as:
-    
-```bibtex
-@misc{vonwerra2022trl,
-	title        = {{TRL: Transformer Reinforcement Learning}},
-	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
-	year         = 2020,
-	journal      = {GitHub repository},
-	publisher    = {GitHub},
-	howpublished = {\url{https://github.com/huggingface/trl}}
-}
-```
+Apache 2.0, inheriting the [Gemma terms of use](https://ai.google.dev/gemma/terms) from the base
+model.