MaXoN654 commited on
Commit
ffe3a09
·
verified ·
1 Parent(s): 6cd4a5f

model card: aligned eval set (same 2717 questions EN/RU), RU 58.4%

Browse files
Files changed (1) hide show
  1. README.md +17 -15
README.md CHANGED
@@ -30,42 +30,44 @@ questions** over a database schema with step-by-step reasoning that ends in a fi
30
  ## Performance Evaluation
31
 
32
  Execution accuracy (predicted SQL executed against SQLite, result-set comparison)
33
- on the **full held-out eval split** (greedy decoding):
 
34
 
35
  | Model | Questions | n | EX accuracy |
36
  |---|---|---|---|
37
- | Base (zero-shot) | EN | 3,032 | 16.0% |
38
- | Base (zero-shot) | RU | 2,729 | 13.9% |
39
- | **RUSQL (this model)** | **RU** | **2,729** | **58.4%** |
40
 
41
- Fine-tuning lifts execution accuracy **from 13.9% to 58.4%** — about 4.2× the base
42
  model, and well above its English-question ceiling (16.0%).
43
 
44
  Breakdown by SQL complexity (RUSQL, RU questions):
45
 
46
  | Complexity | n | EX accuracy |
47
  |---|---|---|
48
- | Simple | 259 | 75.7% |
49
- | Moderate | 858 | 70.3% |
50
- | Complex | 973 | 55.1% |
51
- | Highly Complex | 639 | 40.5% |
52
 
53
  Breakdown by question style (RUSQL, RU questions):
54
 
55
  | Style | n | EX accuracy |
56
  |---|---|---|
57
- | Formal | 310 | 62.9% |
58
  | Interrogative | 310 | 62.6% |
59
- | Multi-turn Dialogue | 270 | 59.6% |
60
- | Colloquial | 336 | 59.5% |
61
  | Imperative | 324 | 58.6% |
62
  | Descriptive | 312 | 58.3% |
63
- | Concise | 273 | 57.1% |
64
  | Vague | 310 | 55.5% |
65
  | Metaphorical | 284 | 50.7% |
66
 
67
- *Full-split evaluation (RU n=2,729 / EN n=3,032), greedy decoding, only 13 of 2,729
68
- RUSQL generations failed to emit a SQL block.*
 
69
 
70
  ## Dataset Overview
71
 
 
30
  ## Performance Evaluation
31
 
32
  Execution accuracy (predicted SQL executed against SQLite, result-set comparison)
33
+ on the **held-out eval split** — the same **2,717** questions in every row (EN vs RU
34
+ are the same items translated), greedy decoding:
35
 
36
  | Model | Questions | n | EX accuracy |
37
  |---|---|---|---|
38
+ | Base (zero-shot) | EN | 2,717 | 16.0% |
39
+ | Base (zero-shot) | RU | 2,717 | 14.0% |
40
+ | **RUSQL (this model)** | **RU** | **2,717** | **58.4%** |
41
 
42
+ Fine-tuning lifts execution accuracy **from 14.0% to 58.4%** — about 4.2× the base
43
  model, and well above its English-question ceiling (16.0%).
44
 
45
  Breakdown by SQL complexity (RUSQL, RU questions):
46
 
47
  | Complexity | n | EX accuracy |
48
  |---|---|---|
49
+ | Simple | 257 | 75.9% |
50
+ | Moderate | 856 | 70.2% |
51
+ | Complex | 968 | 55.1% |
52
+ | Highly Complex | 636 | 40.4% |
53
 
54
  Breakdown by question style (RUSQL, RU questions):
55
 
56
  | Style | n | EX accuracy |
57
  |---|---|---|
58
+ | Formal | 306 | 62.8% |
59
  | Interrogative | 310 | 62.6% |
60
+ | Multi-turn Dialogue | 268 | 60.1% |
61
+ | Colloquial | 334 | 59.3% |
62
  | Imperative | 324 | 58.6% |
63
  | Descriptive | 312 | 58.3% |
64
+ | Concise | 269 | 56.9% |
65
  | Vague | 310 | 55.5% |
66
  | Metaphorical | 284 | 50.7% |
67
 
68
+ *The RU eval questions are the QE-filtered translations of the held-out split, and
69
+ the EN/base rows are scored on the exact same items so the three numbers are directly
70
+ comparable. Greedy decoding; only 13 RUSQL generations failed to emit a SQL block.*
71
 
72
  ## Dataset Overview
73