File size: 1,866 Bytes
b0e88cf | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | # LLM-SQL — Column Reordering for Prefix Caching
When rows of a table are serialized into LLM prompts sequentially, consecutive rows that share leading column values can reuse cached prefixes. This task evolves a column-reordering strategy that maximizes prefix-cache hit rates across multiple real-world datasets without altering the underlying data.
## Setup
1. **Download the datasets** (~69 MB total):
```bash
cd benchmarks/ADRS/llm_sql
bash download_dataset.sh
```
This downloads 5 CSV datasets into `datasets/`:
- `movies.csv` — Rotten Tomatoes movie reviews (~9 MB)
- `beer.csv` — Beer review dataset (~2.5 MB)
- `BIRD.csv` — BIRD text-to-SQL dataset (~34 MB)
- `PDMX.csv` — PDMX metadata dataset (~7.4 MB)
- `products.csv` — Amazon product catalog (~16 MB)
2. **Set your API key:**
```bash
export OPENAI_API_KEY=...
```
## Run
From the repo root:
```bash
uv run skydiscover-run \
benchmarks/ADRS/llm_sql/initial_program.py \
benchmarks/ADRS/llm_sql/evaluator.py \
-c benchmarks/ADRS/llm_sql/config.yaml \
-s [your_algorithm] \
-i 100
```
## Scoring
Combined score: `0.95 * average_hit_rate + 0.05 * (12 - min(12, avg_runtime)) / 12`
- **Hit rate** (95% weight): prefix-cache hit count normalized across 5 datasets
- **Runtime** (5% weight): wall-clock seconds for the reordering algorithm
## Files
| File | Description |
|------|-------------|
| `initial_program.py` | Baseline `Evolved` class with `reorder()` method to evolve |
| `evaluator.py` | Scores programs on prefix hit rate and runtime across 5 datasets |
| `config.yaml` | Task-specific config (LLM, evaluator timeout, system prompt) |
| `solver.py` | Base `Algorithm` class and greedy baseline |
| `utils.py` | Prefix hit count evaluation utilities |
| `download_dataset.sh` | Script to download required CSV datasets |
|