MaXoN654 commited on
Commit
cb896ef
·
verified ·
1 Parent(s): ffe3a09

model card: aligned eval (same 2729 items EN/RU/SFT)

Browse files
Files changed (1) hide show
  1. README.md +18 -18
README.md CHANGED
@@ -30,44 +30,44 @@ questions** over a database schema with step-by-step reasoning that ends in a fi
30
  ## Performance Evaluation
31
 
32
  Execution accuracy (predicted SQL executed against SQLite, result-set comparison)
33
- on the **held-out eval split** — the same **2,717** questions in every row (EN vs RU
34
- are the same items translated), greedy decoding:
35
 
36
  | Model | Questions | n | EX accuracy |
37
  |---|---|---|---|
38
- | Base (zero-shot) | EN | 2,717 | 16.0% |
39
- | Base (zero-shot) | RU | 2,717 | 14.0% |
40
- | **RUSQL (this model)** | **RU** | **2,717** | **58.4%** |
41
 
42
- Fine-tuning lifts execution accuracy **from 14.0% to 58.4%** — about 4.2× the base
43
- model, and well above its English-question ceiling (16.0%).
44
 
45
  Breakdown by SQL complexity (RUSQL, RU questions):
46
 
47
  | Complexity | n | EX accuracy |
48
  |---|---|---|
49
- | Simple | 257 | 75.9% |
50
- | Moderate | 856 | 70.2% |
51
- | Complex | 968 | 55.1% |
52
- | Highly Complex | 636 | 40.4% |
53
 
54
  Breakdown by question style (RUSQL, RU questions):
55
 
56
  | Style | n | EX accuracy |
57
  |---|---|---|
58
- | Formal | 306 | 62.8% |
59
  | Interrogative | 310 | 62.6% |
60
- | Multi-turn Dialogue | 268 | 60.1% |
61
- | Colloquial | 334 | 59.3% |
62
  | Imperative | 324 | 58.6% |
63
  | Descriptive | 312 | 58.3% |
64
- | Concise | 269 | 56.9% |
65
  | Vague | 310 | 55.5% |
66
  | Metaphorical | 284 | 50.7% |
67
 
68
- *The RU eval questions are the QE-filtered translations of the held-out split, and
69
- the EN/base rows are scored on the exact same items so the three numbers are directly
70
- comparable. Greedy decoding; only 13 RUSQL generations failed to emit a SQL block.*
71
 
72
  ## Dataset Overview
73
 
 
30
  ## Performance Evaluation
31
 
32
  Execution accuracy (predicted SQL executed against SQLite, result-set comparison)
33
+ on the **held-out eval split** — the same **2,729** questions in every row (EN vs RU
34
+ are the same items, English question vs its translation), greedy decoding:
35
 
36
  | Model | Questions | n | EX accuracy |
37
  |---|---|---|---|
38
+ | Base (zero-shot) | EN | 2,729 | 16.0% |
39
+ | Base (zero-shot) | RU | 2,729 | 13.9% |
40
+ | **RUSQL (this model)** | **RU** | **2,729** | **58.4%** |
41
 
42
+ Fine-tuning lifts execution accuracy **from 13.9% to 58.4%** — about 4.2× the base
43
+ model, and above its English-question ceiling (16.0%).
44
 
45
  Breakdown by SQL complexity (RUSQL, RU questions):
46
 
47
  | Complexity | n | EX accuracy |
48
  |---|---|---|
49
+ | Simple | 259 | 75.7% |
50
+ | Moderate | 858 | 70.3% |
51
+ | Complex | 973 | 55.1% |
52
+ | Highly Complex | 639 | 40.5% |
53
 
54
  Breakdown by question style (RUSQL, RU questions):
55
 
56
  | Style | n | EX accuracy |
57
  |---|---|---|
58
+ | Formal | 310 | 62.9% |
59
  | Interrogative | 310 | 62.6% |
60
+ | Multi-turn Dialogue | 270 | 59.6% |
61
+ | Colloquial | 336 | 59.5% |
62
  | Imperative | 324 | 58.6% |
63
  | Descriptive | 312 | 58.3% |
64
+ | Concise | 273 | 57.1% |
65
  | Vague | 310 | 55.5% |
66
  | Metaphorical | 284 | 50.7% |
67
 
68
+ *All three rows are scored on the exact same 2,729 items (the QE-filtered held-out
69
+ split), so the numbers are directly comparable. Greedy decoding; only 13 RUSQL
70
+ generations failed to emit a SQL block.*
71
 
72
  ## Dataset Overview
73