text-to-sql-shop-r3

Recipe: recipes/04-train/text-to-sql · Collection: Analyst

One rung of the text-to-SQL hill climb on Qwen3-4B, thinking on, graded by executing the query. Round r3 starts from r2: GRPO 1,000 steps, lr 2e-5, beta 0.01, vLLM generation, 8 prompts per generate. The eval rows for this round are the eval-r3 config of the text-to-sql-shop dataset.

Where it sits in the climb

81 held-out tasks, 4 samples each, temperature 0.7. The row in bold is this adapter.

Round	From	Method	pass@1 (95% CI)	pass^4	has_sql
r1	base	GRPO 100 steps, lr 2e-5, beta 0.04, HF generate	0.58 (0.52..0.64)	0.29	0.89
r2	r1	GRPO 200 steps, lr 5e-5, beta 0.01	0.60 (0.54..0.67)	0.34	0.90
sft-think	base	self-distillation: 199 verified traces, hosted SFT 2 epochs	0.60 (0.54..0.67)	0.39	0.96
r3	r2	GRPO 1,000 steps, lr 2e-5, beta 0.01, vLLM generation, 8 prompts per generate	0.61 (0.56..0.67)	0.29	0.86
r4	r3	GRPO 1,000 more steps, same settings	0.61 (0.54..0.67)	0.32	0.85

Rounds r1 to r4 are within each other's intervals: two thousand GRPO steps did not move pass@1 on this holdout. The move came in r5, from a re-cut prompt band and masked truncation, and is the text-to-sql-shop-r5 repo. The recipe README explains what each round changed and why.

Arms in this repo

The root holds the arm the recipe README's headline number reports. Every other arm is a subfolder named after it. checkpoints/ never ships.

folder	arm
.	round r3 adapter

Load

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B")
model = PeftModel.from_pretrained(base, "while-ai/text-to-sql-shop-r3")  # the headline arm

Reproduce

git clone https://github.com/whilehq/whileai-sdk && cd whileai-sdk/recipes/04-train/text-to-sql
modal run train_grpo_modal.py --run-name t2s-r1

The recipe README pins the seed, the library versions and the GPU, and its Checks table says what the eval verified. Read the Learned section before quoting a number from this card.