logbook.md

# Gemma SQL

# Gemma SQL

> Index of experiments. Click one to open its page. Edit this page freely — the table is just Markdown.

## Experiments

| Status | Experiment |
| --- | --- |
| planned | [Data & eval harness — gretelai/synthetic_text_to_sql; heldout = SQLite-executable subset of test split; metrics = execution accuracy (primary) + normalized exact match](#/data-eval-harness-gretelai-synthetic-text-to-sql-heldout-sqlite-executable-subset-of-test-split-metrics-execution-accuracy-primary-normalized-exact-match) |
| planned | [Baseline — zero-shot google/gemma-3-270m(-it) on heldout via HF Jobs; report EX/EM](#/baseline-zero-shot-google-gemma-3-270m-it-on-heldout-via-hf-jobs-report-ex-em) |
| planned | [Fine-tune — TRL SFT on train split via HF Jobs GPU; push model to Hub](#/fine-tune-trl-sft-on-train-split-via-hf-jobs-gpu-push-model-to-hub) |
| planned | [Eval fine-tuned — same harness on heldout; report improvement over baseline](#/eval-fine-tuned-same-harness-on-heldout-report-improvement-over-baseline) |
| done | [Data & eval harness](#/data-eval-harness) |
| done | [Baseline (zero-shot)](#/baseline-zero-shot) |
| running | [Fine-tune (SFT)](#/fine-tune-sft) |
| done | [Eval fine-tuned](#/eval-fine-tuned) |

# Data & eval harness — gretelai/synthetic_text_to_sql; heldout = SQLite-executable subset of test split; metrics = execution accuracy (primary) + normalized exact match

# Baseline — zero-shot google/gemma-3-270m(-it) on heldout via HF Jobs; report EX/EM

# Fine-tune — TRL SFT on train split via HF Jobs GPU; push model to Hub

# Eval fine-tuned — same harness on heldout; report improvement over baseline

# Data & eval harness

---

### Note

`Jul 02, 2026 · 06:57 UTC`

Built heldout test set abidlabs/gemma-sql-heldout: 1000 examples sampled (seed=1234) from the SQLite-executable subset of gretelai/synthetic_text_to_sql test split. 4560/5851 (77.9%) of gold queries execute cleanly in SQLite; the rest use dialect-specific funcs (YEAR/MONTH), multi-statement, or schema-qualified names. Metric validated: gold-vs-gold = 100.0% execution accuracy and 100.0% exact match on the executable subset. Complexity mix: basic 543, aggregation 209, single join 134, subqueries 47, window 33, multi-join 23. Spans 100 domains. Primary metric = execution accuracy (result-set match, order-sensitive only when gold has ORDER BY); secondary = normalized exact match.

# Baseline (zero-shot)

---

### Note

`Jul 02, 2026 · 07:06 UTC`

BASELINE (zero-shot, no fine-tuning) on 1000-example heldout. google/gemma-3-270m-it with chat template + system prompt: execution accuracy = 29.2%, exact match = 5.0%, valid-SQL rate = 80.0%. By complexity: basic SQL 39.4%, aggregation 18.7%, single join 13.4%, multi-join 21.7%, subqueries 23.4%, window fns 12.1%, set ops 9.1%. Harder query classes (joins, windows) are where the tiny model struggles most. This is the number to beat with fine-tuning. Predictions saved to abidlabs/gemma-sql-preds-baseline-it. Job flavor t4-small, ~103s for 1000 examples.

- https://huggingface.co/datasets/abidlabs/gemma-sql-preds-baseline-it

---

### Note

`Jul 02, 2026 · 07:08 UTC`

BASELINE reference: google/gemma-3-270m (base, NOT instruction-tuned), zero-shot with plain prompt (no chat template): execution accuracy = 12.2%, exact match = 0.4%, valid-SQL rate = 61%. Confirms the instruction-tuned checkpoint (-it, 29.2%) is the right starting point for fine-tuning. Predictions: abidlabs/gemma-sql-preds-baseline-base.

# Fine-tune (SFT)

---

### Note

`Jul 02, 2026 · 07:20 UTC`

Fine-tuning launched on HF Jobs (L4 GPU, job 6a460d35). Full SFT of google/gemma-3-270m-it on the 100k gretel train split: 3 epochs, effective batch 32, lr 2e-5 cosine, assistant-only (completion) loss, max_len 1024, bf16, same schema+question->SQL prompt format as eval. Progress at 11%: train loss dropped from ~9 to 0.35, mean-token-accuracy 90.5%, ~1.15 it/s, 9375 total steps, ETA ~2h. Live curves: https://huggingface.co/spaces/abidlabs/gemma-sql-trackio . Model will push to abidlabs/gemma-3-270m-sql.

- https://huggingface.co/spaces/abidlabs/gemma-sql-trackio

---

### Note

`Jul 02, 2026 · 07:54 UTC`

Fine-tune progress: 36% (step 3379/9375, epoch 1.08), train loss 0.267, mean-token-accuracy 92.2%, ~1.12 it/s, ~1.5h remaining. Curves healthy and still decreasing.

---

### Note

`Jul 02, 2026 · 08:25 UTC`

Fine-tune progress: 59% (step 5531/9375, epoch 1.77), train loss ~0.27 (plateaued), token-accuracy 92.1%, ~57 min remaining. Loss flattened after epoch 1, so most learning happened early; epochs 2-3 are refinement.

---

### Note

`Jul 02, 2026 · 08:56 UTC`

Fine-tune nearing completion: ~1h52m elapsed of ~2h19m est. Model repo abidlabs/gemma-3-270m-sql created; weights upload at end (hub_strategy=end). Will eval once upload completes.

# Eval fine-tuned

---

### Note

`Jul 02, 2026 · 09:28 UTC`

FINE-TUNED RESULT (abidlabs/gemma-3-270m-sql) on the same 1000-example heldout: execution accuracy = 58.6% (baseline 29.2%, +29.4 pts, 2.0x), exact match = 30.3% (baseline 5.0%, 6x), valid-SQL rate = 85% (baseline 80%). Per-complexity vs baseline: basic 39.4->72.4, aggregation 18.7->58.9, single join 13.4->31.3, subqueries 23.4->40.4, set ops 9.1->36.4 (all big gains); window fns 12.1->12.1 (flat, n=33); multiple_joins 21.7->4.3 (REGRESSION, but n=23 so noisy). Net: fine-tuning on 100k gretel examples lifts the tiny 270M model from 29% to 59% execution accuracy. Hardest/rarest classes (multi-join, window) remain weak - candidates for targeted data augmentation. Eval on t4-small, 165s. Predictions: abidlabs/gemma-sql-preds-finetuned.

- https://huggingface.co/abidlabs/gemma-3-270m-sql