liovina commited on
Commit
9a6d5c7
·
verified ·
1 Parent(s): 30d7071

HF Space README frontmatter

Browse files
Files changed (1) hide show
  1. README.md +7 -6
README.md CHANGED
@@ -19,13 +19,13 @@ Portfolio demo для Senior Data Engineer / Data Analyst. Принимает в
19
  **Headline metrics:**
20
  - **Chinook demo workload (n=60): 100% EA — 60/60.** 30 dev + 30 held-out, balanced split, no overfitting. Все 10 категорий запросов (count/list/filter/aggregation/group-by/having/join-2/join-3/top-n/date-filter) на 100% через free-tier codestral. Это реальный analyst workload, как BI tool в проде.
21
  - **BIRD Mini-Dev SQLite (n=200, hard research benchmark):**
22
- - **Hybrid (codestral + Sonnet challenging + multi-provider voting + grounded-critique retry + self-consistency + Sonnet bridge + selective fewshot expansion on residue): 77.5% EA.**
23
- - Lift trace на n=200: 47% baseline (A_full_schema) → 51% (C_dense_cards) → 55.5% (D_dense_fewshot) → 56.5% (G_verify_retry) → 57.0% (Sonnet challenging hybrid) → 65.5% (+ multi-provider Groq voting v1) → 68.0% (+ Groq voting v2) → 72.0% (+ grounded-critique directed retry, 8 rescues / 0 regressions) → 72.5% (+ Mistral self-consistency) → 77.0% (+ Sonnet 4.6 via GraceKelly Perplexity bridge on the remaining 55 fails, 9 rescues / 0 regressions) → **77.5% (+ selective fewshot_top_k=5 with grounded-critique on the 46-fail residue, 1 rescue / 0 regressions)**.
24
- - **+29.7pp над GPT-4 zero-shot (47.8%), $0 external cost.** Above published SOTA с paid GPT-4 (CHESS/Distillery: 73–76%).
25
- - Per-difficulty: simple **89.6%**, moderate **74.7%**, challenging **61.8%**.
26
  - **Безопасность пайплайна:** AST guard (`sqlglot`) + read-only DB connection + row cap + statement timeout. DML/DDL/multi-statement/ATTACH/PRAGMA отбрасываются до execution.
27
 
28
- **Достигнутый потолок на $0 budget без fine-tuning:** 77.5% на BIRD. Выше published SOTA (CHESS/Distillery — 73–76% с GPT-4 API + custom schema linker). Human expert baseline (BIRD paper) — 92.96%. Три главных рычага: **(1)** grounded-critique directed retry — shape feedback инжектится в re-prompt только на frozen-фейлах (+4.5pp без новых моделей); **(2)** Sonnet 4.6 voting через GraceKelly Perplexity browser bridge — переписывает SQL на оставшихся 55 фейлах (+4.5pp); **(3)** selective `fewshot_top_k=5` с grounded-critique на 46-fail residue после Sonnet (+0.5pp, 1 rescue / 0 regressions — validation что глобально вредный лeверь работает targeted).
29
 
30
  **UI (2026-05-13 редизайн):** Streamlit chrome переписан в editorial monochrome — кастомные шрифты (TT Norms Pro Serif для display, AA Stetica для UI), тёплая бумажная палитра без primary-цветов, EN↔RU пер��ключатель языка, без эмодзи и стоковых иконок. Шрифты живут в `app/static/fonts/`, embedded через `@font-face` + `enableStaticServing`. Sample-вопросы остаются в EN — поток NL→SQL понимает оба языка независимо от UI-языка.
31
 
@@ -102,7 +102,8 @@ For the public Streamlit Cloud demo (free, ~5 min setup), see
102
  | Hybrid + voting + grounded-critique retry | — | 72.0% ✅ |
103
  | + Mistral self-consistency | — | 72.5% ✅ |
104
  | Final 2026-05-13 (+ Sonnet 4.6 bridge on all fails) | — | 77.0% ✅ |
105
- | **Final 2026-05-17 (+ selective fewshot_top_k=5 on residue)** | — | **77.5%** ✅ |
 
106
  | GPT-4 zero-shot reference | — | 47.8% |
107
  | Published SOTA (paid API + fine-tuning) | — | 73–76% (CHESS) |
108
  | Human expert baseline (BIRD paper) | — | 92.96% |
 
19
  **Headline metrics:**
20
  - **Chinook demo workload (n=60): 100% EA — 60/60.** 30 dev + 30 held-out, balanced split, no overfitting. Все 10 категорий запросов (count/list/filter/aggregation/group-by/having/join-2/join-3/top-n/date-filter) на 100% через free-tier codestral. Это реальный analyst workload, как BI tool в проде.
21
  - **BIRD Mini-Dev SQLite (n=200, hard research benchmark):**
22
+ - **Hybrid (codestral + Sonnet challenging + multi-provider voting + grounded-critique retry + self-consistency + Sonnet bridge + selective fewshot expansion + cross-Groq vote on residue): 79.0% EA.**
23
+ - Lift trace на n=200: 47% baseline (A_full_schema) → 51% (C_dense_cards) → 55.5% (D_dense_fewshot) → 56.5% (G_verify_retry) → 57.0% (Sonnet challenging hybrid) → 65.5% (+ multi-provider Groq voting v1) → 68.0% (+ Groq voting v2) → 72.0% (+ grounded-critique directed retry, 8 rescues / 0 regressions) → 72.5% (+ Mistral self-consistency) → 77.0% (+ Sonnet 4.6 via GraceKelly Perplexity bridge on the remaining 55 fails, 9 rescues / 0 regressions) → 77.5% (+ selective fewshot_top_k=5 with grounded-critique on the 46-fail residue, 1 rescue / 0 regressions)**79.0% (+ cross-Groq voting on the 45-fail residue: llama-3.3-70b 2 rescues + qwen3-32b 1 rescue, 0 regressions)**.
24
+ - **+31.2pp над GPT-4 zero-shot (47.8%), $0 external cost.** Выше published SOTA paid GPT-4 (CHESS/Distillery: 73–76%).
25
+ - Per-difficulty: simple **91.0%**, moderate **75.8%**, challenging **64.7%**.
26
  - **Безопасность пайплайна:** AST guard (`sqlglot`) + read-only DB connection + row cap + statement timeout. DML/DDL/multi-statement/ATTACH/PRAGMA отбрасываются до execution.
27
 
28
+ **Достигнутый потолок на $0 budget без fine-tuning:** 79.0% на BIRD. Выше published SOTA (CHESS/Distillery — 73–76% с GPT-4 API + custom schema linker). Human expert baseline (BIRD paper) — 92.96%. Четыре главных рычага: **(1)** grounded-critique directed retry — shape feedback инжектится в re-prompt только на frozen-фейлах (+4.5pp без новых моделей); **(2)** Sonnet 4.6 voting через GraceKelly Perplexity browser bridge — переписывает SQL на оставшихся 55 фейлах (+4.5pp); **(3)** selective `fewshot_top_k=5` с grounded-critique на 46-fail residue после Sonnet (+0.5pp, validation что глобально вредный лeверь работает targeted); **(4)** cross-Groq voting на 45-fail residue после fewshot5 — llama-3.3-70b и qwen3-32b с critique дают ортогональные fixes (+1.5pp, 3 rescues / 0 regressions). Same-family Mistral large negative подтвердил что residue структурный unanimous через Mistral models.
29
 
30
  **UI (2026-05-13 редизайн):** Streamlit chrome переписан в editorial monochrome — кастомные шрифты (TT Norms Pro Serif для display, AA Stetica для UI), тёплая бумажная палитра без primary-цветов, EN↔RU пер��ключатель языка, без эмодзи и стоковых иконок. Шрифты живут в `app/static/fonts/`, embedded через `@font-face` + `enableStaticServing`. Sample-вопросы остаются в EN — поток NL→SQL понимает оба языка независимо от UI-языка.
31
 
 
102
  | Hybrid + voting + grounded-critique retry | — | 72.0% ✅ |
103
  | + Mistral self-consistency | — | 72.5% ✅ |
104
  | Final 2026-05-13 (+ Sonnet 4.6 bridge on all fails) | — | 77.0% ✅ |
105
+ | Final 2026-05-17 EOS (+ selective fewshot_top_k=5 on residue) | — | 77.5% ✅ |
106
+ | **Final 2026-05-17 night (+ cross-Groq llama3.3-70b + qwen3-32b voting)** | — | **79.0%** ✅ |
107
  | GPT-4 zero-shot reference | — | 47.8% |
108
  | Published SOTA (paid API + fine-tuning) | — | 73–76% (CHESS) |
109
  | Human expert baseline (BIRD paper) | — | 92.96% |