docs/exploration/rl-vs-icl-research.md

---
title: RL vs In-Context Learning for Small Model SQL Agents
description: Research synthesis on when GRPO training adds value over pure prompting with in-context learning for sub-2B parameter models
doc_type: exploration
---

# RL vs In-Context Learning for Small Model SQL Agents

Exploration doc for F011 (Prompting Baseline Notebook).

## Context

We train Qwen3-0.6B/1.7B to explore SQL databases using multi-turn tool
calls (describe, query, answer). This doc synthesizes research on when
GRPO training adds value over pure prompting with in-context learning.

## Key Finding: RL wins for small models on multi-turn tool use

For sub-2B models on multi-step tasks, the evidence strongly favors
RL over pure ICL. But ICL can be a strong baseline — and a hybrid
(ICL during RL rollouts) may be optimal.

## When RL (GRPO) beats ICL

### 1. Small models are weak in-context learners

- Fine-tuning advantage over ICL **grows** as model size **shrinks**
  (NeurIPS 2022: "Few-Shot PEFT is Better and Cheaper than ICL")
- Sub-2B models lack pre-training breadth for reliable few-shot extraction
- Context window is precious — few-shot examples consume tokens needed
  for schema descriptions and conversation history

### 2. Multi-turn exploration needs adaptive behavior

- RL teaches error recovery, retry strategies, exploration planning
- Static few-shot examples can't teach "if your query fails, try a
  different approach"
- ToolRL: GRPO gives +17% over base, +15% over SFT on BFCL V3

### 3. Per-query economics favor RL at deployment

| Factor           | ICL              | RL-trained           |
|------------------|------------------|----------------------|
| Training cost    | Zero             | One-time (~2h on L4) |
| Per-query cost   | High (long prompts) | Low (internalized)  |
| Latency          | Higher           | Lower                |

## When ICL is sufficient

- Large models (7B+) on simple/single-turn tool calls
- Prototyping before committing to training infrastructure
- Constrained output space (few tools, fixed schemas)
- When the task is already within the model's pre-training distribution

## The Hybrid Approach (most promising for our case)

### ToolExpander (arXiv:2510.07737)
- Pure GRPO on 1.5B models is unstable and often collapses mid-training
- Fix: **few-shot guided rollouts during RL** — dynamically substitute
  hard samples with few-shot demonstrations
- Eliminated training collapse, reduced hard samples by 15-20%

### ICRL (arXiv:2603.08068)
- Use few-shot prompts **during RL rollouts** but progressively reduce
  them via curriculum learning
- Transition from few-shot to zero-shot over training
- Eliminates need for SFT entirely
- Achieved SOTA on QA and math reasoning

### Implication for SQLEnv
- Start GRPO with 1-2 few-shot examples in the prompt
- As training progresses, remove examples (curriculum)
- The model internalizes the ICL patterns via RL reward signal

## Context Window Considerations

### Qwen3-0.6B/1.7B context limits
- **TODO**: Measure effective context window and performance degradation
- Need to determine: how many few-shot examples fit alongside the
  system prompt, tools, and conversation history?
- "Lost in the Middle" (TACL): even with perfect retrieval, performance
  degrades 13.9%-85% as input length increases
- "Context Length Alone Hurts" (arXiv:2510.05381): degradation is worse
  for smaller models

### Token budget breakdown (estimated)
- System prompt + tools: ~500 tokens
- Question + table hint: ~50 tokens
- Per describe response: ~50 tokens
- Per query response: ~50-200 tokens
- Per few-shot example: ~300-500 tokens
- **Total available**: model context - above overhead

### Research needed for F011
1. Measure Qwen3-0.6B/1.7B effective context window (when does
   performance degrade?)
2. How many few-shot examples fit before hitting context limits?
3. Does the model attend to examples in the middle of the context?
4. What's the minimum ICL example count for reliable tool-calling?

## Training Results Analysis (2026-04-01)

### Qwen3-1.7B, 1 epoch GRPO, 100-example per-turn SFT warmup

**What works:**
- Model learned proper multi-turn tool-calling (describe → query → answer)
- Generates real SQL with JOINs, GROUP BY, ORDER BY, subqueries
- ~30-40% of episodes get correct answers (reward ~1.15)
- GRPO produces gradient signal (advantages range -1.5 to +1.5)

**What doesn't work:**
- Model doesn't stop after answering — keeps calling tools after
  "Episode is over" (wastes step budget)
- SQL quality varies: correct column names sometimes, wrong others
- Answer format mismatches (correct data, wrong format)
- Training loss oscillates near zero (plateau, not improvement)

**Bottleneck hypothesis:** The model can do tool-calling and basic SQL,
but lacks the SQL reasoning to reliably get correct answers. ICL could
help here by showing the reasoning pattern, not just the format.

## Papers Referenced

| Paper | Key finding | Relevance |
|-------|------------|-----------|
| ToolRL (2504.13958) | GRPO +17% over base for tool-calling | Direct comparison |
| ToolExpander (2510.07737) | Few-shot guided GRPO for 1.5B | Stabilizes small model training |
| ICRL (2603.08068) | ICL + RL curriculum, no SFT needed | Hybrid approach |
| RC-GRPO (2602.03025) | SFT creates peaked policy | Explains plateau |
| PEARL (2601.20439) | Plan + explore + RL for multi-hop | Multi-step tool use |
| Bespoke Labs blog | GRPO on Qwen-2.5-7B multi-turn | Industry baseline |
| Lost in the Middle (TACL) | U-shaped context performance | Context window limits |
| Context Length Alone (2510.05381) | Length degrades small models more | ICL overhead |
| Few-Shot PEFT > ICL (NeurIPS 2022) | FT beats ICL for small models | Baseline comparison |
| STAR pipeline (2603.21972) | Smaller models need curriculum | Training design |
| Distil Labs SLM blog | RL helps generative, not structured | Task-dependent |

## Recommendations for F011 (Prompting Baseline Notebook)

### Techniques to test
1. **Zero-shot** — just tools + question, no examples
2. **1-shot** — one complete trajectory example
3. **3-shot** — three diverse examples (different DBs/query patterns)
4. **Chain-of-thought** — add reasoning before tool calls
5. **Context window test** — measure degradation with increasing examples

### Expected results (based on literature)
- Zero-shot on 1.7B: ~5-15% accuracy (model knows tool format from pre-training)
- Few-shot on 1.7B: ~15-25% accuracy (helps with SQL patterns)
- GRPO-trained: ~30-40% accuracy (current results)
- Gap demonstrates RL value proposition

### Metrics to report
- Accuracy per technique
- Average steps used
- Token budget consumed (prompt length)
- SQL quality (valid SQL rate, correct table/column references)