HF Space README frontmatter
Browse files
README.md
CHANGED
|
@@ -19,13 +19,13 @@ Portfolio demo для Senior Data Engineer / Data Analyst. Принимает в
|
|
| 19 |
**Headline metrics:**
|
| 20 |
- **Chinook demo workload (n=60): 100% EA — 60/60.** 30 dev + 30 held-out, balanced split, no overfitting. Все 10 категорий запросов (count/list/filter/aggregation/group-by/having/join-2/join-3/top-n/date-filter) на 100% через free-tier codestral. Это реальный analyst workload, как BI tool в проде.
|
| 21 |
- **BIRD Mini-Dev SQLite (n=200, hard research benchmark):**
|
| 22 |
-
- **Hybrid (codestral + Sonnet challenging + multi-provider voting + grounded-critique retry + self-consistency + Sonnet bridge + selective fewshot expansion on residue):
|
| 23 |
-
- Lift trace на n=200: 47% baseline (A_full_schema) → 51% (C_dense_cards) → 55.5% (D_dense_fewshot) → 56.5% (G_verify_retry) → 57.0% (Sonnet challenging hybrid) → 65.5% (+ multi-provider Groq voting v1) → 68.0% (+ Groq voting v2) → 72.0% (+ grounded-critique directed retry, 8 rescues / 0 regressions) → 72.5% (+ Mistral self-consistency) → 77.0% (+ Sonnet 4.6 via GraceKelly Perplexity bridge on the remaining 55 fails, 9 rescues / 0 regressions) →
|
| 24 |
-
- **+
|
| 25 |
-
- Per-difficulty: simple **
|
| 26 |
- **Безопасность пайплайна:** AST guard (`sqlglot`) + read-only DB connection + row cap + statement timeout. DML/DDL/multi-statement/ATTACH/PRAGMA отбрасываются до execution.
|
| 27 |
|
| 28 |
-
**Достигнутый потолок на $0 budget без fine-tuning:**
|
| 29 |
|
| 30 |
**UI (2026-05-13 редизайн):** Streamlit chrome переписан в editorial monochrome — кастомные шрифты (TT Norms Pro Serif для display, AA Stetica для UI), тёплая бумажная палитра без primary-цветов, EN↔RU пер��ключатель языка, без эмодзи и стоковых иконок. Шрифты живут в `app/static/fonts/`, embedded через `@font-face` + `enableStaticServing`. Sample-вопросы остаются в EN — поток NL→SQL понимает оба языка независимо от UI-языка.
|
| 31 |
|
|
@@ -102,7 +102,8 @@ For the public Streamlit Cloud demo (free, ~5 min setup), see
|
|
| 102 |
| Hybrid + voting + grounded-critique retry | — | 72.0% ✅ |
|
| 103 |
| + Mistral self-consistency | — | 72.5% ✅ |
|
| 104 |
| Final 2026-05-13 (+ Sonnet 4.6 bridge on all fails) | — | 77.0% ✅ |
|
| 105 |
-
|
|
|
|
|
| 106 |
| GPT-4 zero-shot reference | — | 47.8% |
|
| 107 |
| Published SOTA (paid API + fine-tuning) | — | 73–76% (CHESS) |
|
| 108 |
| Human expert baseline (BIRD paper) | — | 92.96% |
|
|
|
|
| 19 |
**Headline metrics:**
|
| 20 |
- **Chinook demo workload (n=60): 100% EA — 60/60.** 30 dev + 30 held-out, balanced split, no overfitting. Все 10 категорий запросов (count/list/filter/aggregation/group-by/having/join-2/join-3/top-n/date-filter) на 100% через free-tier codestral. Это реальный analyst workload, как BI tool в проде.
|
| 21 |
- **BIRD Mini-Dev SQLite (n=200, hard research benchmark):**
|
| 22 |
+
- **Hybrid (codestral + Sonnet challenging + multi-provider voting + grounded-critique retry + self-consistency + Sonnet bridge + selective fewshot expansion + cross-Groq vote on residue): 79.0% EA.**
|
| 23 |
+
- Lift trace на n=200: 47% baseline (A_full_schema) → 51% (C_dense_cards) → 55.5% (D_dense_fewshot) → 56.5% (G_verify_retry) → 57.0% (Sonnet challenging hybrid) → 65.5% (+ multi-provider Groq voting v1) → 68.0% (+ Groq voting v2) → 72.0% (+ grounded-critique directed retry, 8 rescues / 0 regressions) → 72.5% (+ Mistral self-consistency) → 77.0% (+ Sonnet 4.6 via GraceKelly Perplexity bridge on the remaining 55 fails, 9 rescues / 0 regressions) → 77.5% (+ selective fewshot_top_k=5 with grounded-critique on the 46-fail residue, 1 rescue / 0 regressions) → **79.0% (+ cross-Groq voting on the 45-fail residue: llama-3.3-70b 2 rescues + qwen3-32b 1 rescue, 0 regressions)**.
|
| 24 |
+
- **+31.2pp над GPT-4 zero-shot (47.8%), $0 external cost.** Выше published SOTA paid GPT-4 (CHESS/Distillery: 73–76%).
|
| 25 |
+
- Per-difficulty: simple **91.0%**, moderate **75.8%**, challenging **64.7%**.
|
| 26 |
- **Безопасность пайплайна:** AST guard (`sqlglot`) + read-only DB connection + row cap + statement timeout. DML/DDL/multi-statement/ATTACH/PRAGMA отбрасываются до execution.
|
| 27 |
|
| 28 |
+
**Достигнутый потолок на $0 budget без fine-tuning:** 79.0% на BIRD. Выше published SOTA (CHESS/Distillery — 73–76% с GPT-4 API + custom schema linker). Human expert baseline (BIRD paper) — 92.96%. Четыре главных рычага: **(1)** grounded-critique directed retry — shape feedback инжектится в re-prompt только на frozen-фейлах (+4.5pp без новых моделей); **(2)** Sonnet 4.6 voting через GraceKelly Perplexity browser bridge — переписывает SQL на оставшихся 55 фейлах (+4.5pp); **(3)** selective `fewshot_top_k=5` с grounded-critique на 46-fail residue после Sonnet (+0.5pp, validation что глобально вредный лeверь работает targeted); **(4)** cross-Groq voting на 45-fail residue после fewshot5 — llama-3.3-70b и qwen3-32b с critique дают ортогональные fixes (+1.5pp, 3 rescues / 0 regressions). Same-family Mistral large negative подтвердил что residue структурный unanimous через Mistral models.
|
| 29 |
|
| 30 |
**UI (2026-05-13 редизайн):** Streamlit chrome переписан в editorial monochrome — кастомные шрифты (TT Norms Pro Serif для display, AA Stetica для UI), тёплая бумажная палитра без primary-цветов, EN↔RU пер��ключатель языка, без эмодзи и стоковых иконок. Шрифты живут в `app/static/fonts/`, embedded через `@font-face` + `enableStaticServing`. Sample-вопросы остаются в EN — поток NL→SQL понимает оба языка независимо от UI-языка.
|
| 31 |
|
|
|
|
| 102 |
| Hybrid + voting + grounded-critique retry | — | 72.0% ✅ |
|
| 103 |
| + Mistral self-consistency | — | 72.5% ✅ |
|
| 104 |
| Final 2026-05-13 (+ Sonnet 4.6 bridge on all fails) | — | 77.0% ✅ |
|
| 105 |
+
| Final 2026-05-17 EOS (+ selective fewshot_top_k=5 on residue) | — | 77.5% ✅ |
|
| 106 |
+
| **Final 2026-05-17 night (+ cross-Groq llama3.3-70b + qwen3-32b voting)** | — | **79.0%** ✅ |
|
| 107 |
| GPT-4 zero-shot reference | — | 47.8% |
|
| 108 |
| Published SOTA (paid API + fine-tuning) | — | 73–76% (CHESS) |
|
| 109 |
| Human expert baseline (BIRD paper) | — | 92.96% |
|