Spaces:
Running
Gemma SQL
Gemma SQL
Index of experiments. Click one to open its page. Edit this page freely — the table is just Markdown.
Experiments
Data & eval harness — gretelai/synthetic_text_to_sql; heldout = SQLite-executable subset of test split; metrics = execution accuracy (primary) + normalized exact match
Baseline — zero-shot google/gemma-3-270m(-it) on heldout via HF Jobs; report EX/EM
Fine-tune — TRL SFT on train split via HF Jobs GPU; push model to Hub
Eval fine-tuned — same harness on heldout; report improvement over baseline
Data & eval harness
Note
Jul 02, 2026 · 06:57 UTC
Built heldout test set abidlabs/gemma-sql-heldout: 1000 examples sampled (seed=1234) from the SQLite-executable subset of gretelai/synthetic_text_to_sql test split. 4560/5851 (77.9%) of gold queries execute cleanly in SQLite; the rest use dialect-specific funcs (YEAR/MONTH), multi-statement, or schema-qualified names. Metric validated: gold-vs-gold = 100.0% execution accuracy and 100.0% exact match on the executable subset. Complexity mix: basic 543, aggregation 209, single join 134, subqueries 47, window 33, multi-join 23. Spans 100 domains. Primary metric = execution accuracy (result-set match, order-sensitive only when gold has ORDER BY); secondary = normalized exact match.
Baseline (zero-shot)
Note
Jul 02, 2026 · 07:06 UTC
BASELINE (zero-shot, no fine-tuning) on 1000-example heldout. google/gemma-3-270m-it with chat template + system prompt: execution accuracy = 29.2%, exact match = 5.0%, valid-SQL rate = 80.0%. By complexity: basic SQL 39.4%, aggregation 18.7%, single join 13.4%, multi-join 21.7%, subqueries 23.4%, window fns 12.1%, set ops 9.1%. Harder query classes (joins, windows) are where the tiny model struggles most. This is the number to beat with fine-tuning. Predictions saved to abidlabs/gemma-sql-preds-baseline-it. Job flavor t4-small, ~103s for 1000 examples.
Note
Jul 02, 2026 · 07:08 UTC
BASELINE reference: google/gemma-3-270m (base, NOT instruction-tuned), zero-shot with plain prompt (no chat template): execution accuracy = 12.2%, exact match = 0.4%, valid-SQL rate = 61%. Confirms the instruction-tuned checkpoint (-it, 29.2%) is the right starting point for fine-tuning. Predictions: abidlabs/gemma-sql-preds-baseline-base.
Fine-tune (SFT)
Note
Jul 02, 2026 · 07:20 UTC
Fine-tuning launched on HF Jobs (L4 GPU, job 6a460d35). Full SFT of google/gemma-3-270m-it on the 100k gretel train split: 3 epochs, effective batch 32, lr 2e-5 cosine, assistant-only (completion) loss, max_len 1024, bf16, same schema+question->SQL prompt format as eval. Progress at 11%: train loss dropped from ~9 to 0.35, mean-token-accuracy 90.5%, ~1.15 it/s, 9375 total steps, ETA ~2h. Live curves: https://huggingface.co/spaces/abidlabs/gemma-sql-trackio . Model will push to abidlabs/gemma-3-270m-sql.
Note
Jul 02, 2026 · 07:54 UTC
Fine-tune progress: 36% (step 3379/9375, epoch 1.08), train loss 0.267, mean-token-accuracy 92.2%, ~1.12 it/s, ~1.5h remaining. Curves healthy and still decreasing.
Note
Jul 02, 2026 · 08:25 UTC
Fine-tune progress: 59% (step 5531/9375, epoch 1.77), train loss ~0.27 (plateaued), token-accuracy 92.1%, ~57 min remaining. Loss flattened after epoch 1, so most learning happened early; epochs 2-3 are refinement.
Note
Jul 02, 2026 · 08:56 UTC
Fine-tune nearing completion: ~1h52m elapsed of ~2h19m est. Model repo abidlabs/gemma-3-270m-sql created; weights upload at end (hub_strategy=end). Will eval once upload completes.
Eval fine-tuned
Note
Jul 02, 2026 · 09:28 UTC
FINE-TUNED RESULT (abidlabs/gemma-3-270m-sql) on the same 1000-example heldout: execution accuracy = 58.6% (baseline 29.2%, +29.4 pts, 2.0x), exact match = 30.3% (baseline 5.0%, 6x), valid-SQL rate = 85% (baseline 80%). Per-complexity vs baseline: basic 39.4->72.4, aggregation 18.7->58.9, single join 13.4->31.3, subqueries 23.4->40.4, set ops 9.1->36.4 (all big gains); window fns 12.1->12.1 (flat, n=33); multiple_joins 21.7->4.3 (REGRESSION, but n=23 so noisy). Net: fine-tuning on 100k gretel examples lifts the tiny 270M model from 29% to 59% execution accuracy. Hardest/rarest classes (multi-join, window) remain weak - candidates for targeted data augmentation. Eval on t4-small, 165s. Predictions: abidlabs/gemma-sql-preds-finetuned.