Deploy NL_SQL HEAD to HF Space

app/streamlit_app.py

@@ -61,8 +61,8 @@ I18N: dict[str, dict[str, str]] = {
         "metric_percent": "100%",
         "metric_caption": "30 dev + 30 held-out, balanced split, all ten query categories at 100% on the free-tier codestral pipeline.",
         "research_kicker": "BIRD Mini-Dev research benchmark",
-        "research_value": "79.0% / 200",
-        "research_caption": "Hybrid pipeline: codestral + Sonnet on challenging tier + cross-provider voting + grounded-critique directed retry + Sonnet 4.6 bridge on the remaining fails. +29.2pp over the GPT-4 zero-shot reference (47.8%), $0 external cost.",
+        "research_value": "80.0% / 200",
+        "research_caption": "Hybrid pipeline: codestral + Sonnet on challenging tier + cross-provider voting + grounded-critique directed retry + Sonnet 4.6 bridge on the remaining fails. +32.2pp over the GPT-4 zero-shot reference (47.8%), $0 external cost.",
         "settings_header": "Settings",
         "db_label": "Database",
         "db_dialect": "Dialect",
@@ -131,8 +131,8 @@ I18N: dict[str, dict[str, str]] = {
         "metric_percent": "100%",
         "metric_caption": "30 dev + 30 held-out, сбалансированный сплит, все десять категорий запросов на 100% через бесплатный codestral.",
         "research_kicker": "Исследовательский бенчмарк BIRD Mini-Dev",
-        "research_value": "79.0% / 200",
-        "research_caption": "Гибрид: codestral + Sonnet на challenging-тире + кросс-провайдер voting + grounded-critique directed retry + Sonnet 4.6 bridge на оставшихся фейлах. +29.2 п.п. над zero-shot GPT-4 (47.8%), внешние расходы — ноль.",
+        "research_value": "80.0% / 200",
+        "research_caption": "Гибрид: codestral + Sonnet на challenging-тире + кросс-провайдер voting + grounded-critique directed retry + Sonnet 4.6 bridge на оставшихся фейлах. +32.2 п.п. над zero-shot GPT-4 (47.8%), внешние расходы — ноль.",
         "settings_header": "Настройки",
         "db_label": "База данных",
         "db_dialect": "Диалект",

chroma_data/chroma.sqlite3

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d7f72c510d8781191aa4e8173bee8ba4550f99d4f1f5df7562c5191435058aea
+oid sha256:71e30053ed54f40c313c34a77517af48ab8ad100a4a334b8dbd9e9daeafd67c0
 size 18161664

chroma_data/fc9668d3-4384-40d9-aa8d-0010807a5a68/data_level0.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9f7a9f240805365c053e0d463ec92d146c94b4dc22ec2f05297745fc44c5dcd2
+oid sha256:1ed4ebbcc08c0f8d3d33a18e6a88135ffb08bc518c6adfe04f4aee39f3e44ad8
 size 423600

chroma_data/fc9668d3-4384-40d9-aa8d-0010807a5a68/length.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1a7518686c00d3009a5c05e60bca24d2903519877de75996d3df0d945f18bc8b
+oid sha256:d9142387334b53a34226a53a60878980670c9a2a28a56cd9f8ddf2e7d44f3975
 size 400

docs/NEXT_SESSION.md

@@ -3,11 +3,11 @@
 > Один лист, без воды. Берёшь, делаешь, обновляешь `SESSION_HANDOFF.md`,
 > удаляешь этот файл (или переписываешь под следующий sprint).
 
-## Контекст на 2026-05-17 EOS
+## Контекст на 2026-05-17 late-night
 
-- HEAD `e0ea5ad` (после 9370070 + P2.B fewshot5-residue lift + HF Dockerfile fix)
-- BIRD Mini-Dev n=200: **79.0% EA** (158/200), per tier 91.0/75.8/64.7 (v8 = v7 + cross-Groq llama-3.3-70b + qwen3-32b voting на residue, +3 rescues qids 219+352+366)
-- **Live demo:** <https://liovina-nl-sql.hf.space> RUNNING, headline 79.0% / 200
+- HEAD `fcd7ec3` + v9 sprint (см. SESSION_HANDOFF.md)
+- BIRD Mini-Dev n=200: **80.0% EA** (160/200), per tier 91.0/76.8/67.6 (v9 = v8 + gpt-oss-20b voting +2 rescues qids 571 moderate + 1232 challenging)
+- **Live demo:** <https://liovina-nl-sql.hf.space> RUNNING, headline 80.0% / 200
 - 270 pytest pass, ruff + mypy strict clean (55 source files)
 - Streamlit UI editorial monochrome + EN/RU (закрыто 2026-05-13)
 - Portfolio screenshots: `docs/ui-2026-05-17-{en,ru}.png` (local Streamlit) + `docs/ui-live-en.png` (live HF)
@@ -17,14 +17,14 @@
 
 1. ~~Screenshots EN+RU local Streamlit~~ ✓ закрыто 2026-05-17.
 2. **Короткий live-URL ролик** (`D:\AutoReel\` шаблон ИЛИ Playwright video record):
-   - shot A: hero (headline 77.5% + metric block)
+   - shot A: hero (headline 80.0% + metric block)
    - shot B: sample-click → SQL + answer render
    - shot C: EN→RU toggle
    - **Источник: live URL** (`https://liovina-nl-sql.hf.space`), не localhost — memory `feedback_real_product_over_mockup`.
 
-## P2/P3 — quality push past 79.0% ($0 budget)
+## P2/P3 — quality push past 80.0% ($0 budget)
 
-Остаток **42 фейла** (после v8): 22 row_count_off + 13 filter_or_value + 5 order_by_off + 2 errors.
+Остаток **40 фейлов** (после v9). gpt-oss-20b voting (free tier, lightweight) использован на 13/20 ранее-unattempted residue — +2 rescues, TPM 8K режет на 7/20 с long prompts. Residue после v9 ещё содержит ~28 unattempted llama-3.3-70b (TPD cooldown) + 7 gpt-oss-20b TPM-blocked.
 
 | Эксперимент | Статус | Ожидание |
 |---|---|---|

docs/SESSION_HANDOFF.md

@@ -1,12 +1,26 @@
-# NL_SQL — Session Handoff (2026-05-17 EOS: 77.5% BIRD + live HF Space + autonomous deploy/lift sprint)
-
-> **Tl;dr 2026-05-17 EOS:** P0 closed (live demo on HF Spaces, headless API
-> deploy), P2.B closed (+1 selective fewshot rescue → 77.5% n=200). Full
-> gate green: 270 pytest, ruff + mypy strict clean. Mistral-large voting
-> on residue tried → negative (rate-limit + structural agreement with
-> codestral). Open: P2.A (GraceKelly GPT-5.4) and P2.C (Sonnet
-> rephrasing) gated on Chrome profile confirmation; P2.D (custom
-> schema-linker for row_count_off) research-grade.
+# NL_SQL — Session Handoff (2026-05-17 late-night: 80.0% BIRD + gpt-oss-20b v8 rescue + live HF Space)
+
+> **Tl;dr 2026-05-17 late-night:** P0 closed (live demo on HF Spaces),
+> P2.B closed (+1 selective fewshot rescue → 77.5%), P3 cross-Groq closed
+> (+3 rescues → 79.0%), **gpt-oss-20b voting on v8 residue closed
+> (+2 rescues qids 571 moderate / 1232 challenging → 80.0% n=200, 160/200,
+> simple 91.0 / moderate 76.8 / challenging 67.6)**. Live:
+> <https://liovina-nl-sql.hf.space>, headline 80.0%.
+>
+> **Sprint 2026-05-17 late-night results** (HEAD `fcd7ec3` → v9):
+> - openai/gpt-oss-20b: +2 rescues (qids 571 ratio aggregation, 1232 date-arith) — lightweight model добивает то, что Mistral family unanimous провалил
+> - llama-3.3-70b-versatile retry: TPD ещё не сброшен (96.5K/100K, reset 20-108 мин на момент попытки)
+> - qwen/qwen3-32b retry: TPM 6K hard режет промпты 6.6-12K (saturated, не повторять без promp shrink)
+>
+> Cumulative v9 voting bench:
+> - v8 contributors: llama-3.3-70b +2, qwen3-32b +1
+> - v9 contributors: gpt-oss-20b +2 (free tier, lightweight, кэш-friendly)
+> - Negative: mistral-large (TPD-bound + unanimous structural), codestral fewshot=7, gpt-oss-120b (TPM 8K vs critique 10K+), wide-schema (row_count_off ceiling)
+>
+> Open: P3.D (GraceKelly GPT-5.4) и P3.E (Sonnet rephrasing) gated on
+> Chrome profile confirmation; P3.F (custom JOIN-path schema-linker
+> for row_count_off) research-grade; llama-3.3-70b TPD reset retry на
+> ~28 unattempted cases (~24h cooldown).
 >
 > Read `docs/NEXT_SESSION.md` for the action list and historic context
 > in this file below.

eval/reports/2026-05-17/groq-gpt-oss-20b-on-v8-residue.json

@@ -0,0 +1,220 @@
+{
+  "alt_model": "openai/gpt-oss-20b",
+  "summary": {
+    "voted_better": 2,
+    "voted_worse": 0,
+    "voted_same": 11,
+    "groq_input_tokens": 78981,
+    "groq_output_tokens": 13200
+  },
+  "records": [
+    {
+      "question_id": 571,
+      "db_id": "codebase_community",
+      "difficulty": "moderate",
+      "question": "For the user No.24, how many times is the number of his/her posts compared to his/her votes?",
+      "gold_sql": "SELECT CAST(COUNT(DISTINCT T2.Id) AS REAL) / COUNT(DISTINCT T1.Id) FROM votes AS T1 INNER JOIN posts AS T2 ON T1.UserId = T2.OwnerUserId WHERE T1.UserId = 24",
+      "baseline_pred": "SELECT CAST(COUNT(DISTINCT p.Id) AS REAL) / NULLIF(COUNT(DISTINCT v.Id), 0) AS post_to_vote_ratio FROM posts p LEFT JOIN votes v ON p.Id = v.PostId WHERE p.OwnerUserId = 24",
+      "alt_pred": "SELECT CAST(COUNT(*) AS REAL) / (SELECT COUNT(*) FROM votes WHERE UserId = 24) AS ratio FROM posts WHERE OwnerUserId = 24",
+      "alt_confidence": 0.95,
+      "baseline_match": false,
+      "alt_match": true,
+      "vote_match": true,
+      "vote_source": "alt-pick",
+      "agree": false,
+      "elapsed_ms": 2499.6056000018143
+    },
+    {
+      "question_id": 584,
+      "db_id": "codebase_community",
+      "difficulty": "moderate",
+      "question": "Write all the comments left by users who edited the post titled 'Why square the difference instead of taking the absolute value in standard deviation?'",
+      "gold_sql": "SELECT T2.Comment FROM posts AS T1 INNER JOIN postHistory AS T2 ON T1.Id = T2.PostId WHERE T1.Title = 'Why square the difference instead of taking the absolute value in standard deviation?'",
+      "baseline_pred": "SELECT comments.Text FROM comments JOIN posts ON comments.PostId = posts.Id WHERE posts.Title = 'Why square the difference instead of taking the absolute value in standard deviation?'",
+      "alt_pred": "SELECT c.Text FROM comments c JOIN postHistory ph ON c.UserId = ph.UserId JOIN posts p ON ph.PostId = p.Id WHERE p.Title = 'Why square the difference instead of taking the absolute value in standard deviation?'",
+      "alt_confidence": 0.95,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "alt-pick",
+      "agree": false,
+      "elapsed_ms": 47810.004000002664
+    },
+    {
+      "question_id": 595,
+      "db_id": "codebase_community",
+      "difficulty": "moderate",
+      "question": "Which user have only one post history per post and having at least 1000 views?",
+      "gold_sql": "SELECT T2.UserId FROM users AS T1 INNER JOIN postHistory AS T2 ON T1.Id = T2.UserId INNER JOIN posts AS T3 ON T2.PostId = T3.Id WHERE T3.ViewCount >= 1000 GROUP BY T2.UserId HAVING COUNT(DISTINCT T2.PostHistoryTypeId) = 1",
+      "baseline_pred": "SELECT DISTINCT ph.UserId FROM postHistory ph JOIN posts p ON ph.PostId = p.Id WHERE p.ViewCount >= 1000 AND ph.UserId IS NOT NULL GROUP BY ph.UserId, ph.PostId HAVING COUNT(*) = 1",
+      "alt_pred": "",
+      "alt_confidence": 0.0,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "base-fallback",
+      "agree": false,
+      "elapsed_ms": 84282.08260000247
+    },
+    {
+      "question_id": 672,
+      "db_id": "codebase_community",
+      "difficulty": "moderate",
+      "question": "Among the users located in United Kingdom, how many users whose post have a total favorite amount of 4 or more?",
+      "gold_sql": "SELECT COUNT(T1.Id) FROM users AS T1 INNER JOIN posts AS T2 ON T1.Id = T2.OwnerUserId WHERE T1.Location = 'United Kingdom' AND T2.FavoriteCount >= 4",
+      "baseline_pred": "SELECT COUNT(DISTINCT u.Id) FROM users u JOIN posts p ON u.Id = p.OwnerUserId WHERE u.Location = 'United Kingdom' AND p.FavoriteCount >= 4",
+      "alt_pred": "SELECT COUNT(*) FROM (SELECT u.Id FROM users u JOIN posts p ON p.OwnerUserId = u.Id WHERE u.Location = 'United Kingdom' GROUP BY u.Id HAVING SUM(p.FavoriteCount) >= 4) AS t",
+      "alt_confidence": 0.99,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "alt-pick",
+      "agree": false,
+      "elapsed_ms": 43925.09460000292
+    },
+    {
+      "question_id": 694,
+      "db_id": "codebase_community",
+      "difficulty": "moderate",
+      "question": "Provide the text of the latest 10 comments to the post with the title 'Analysing wind data with R' and the display name of the user who left it.",
+      "gold_sql": "SELECT T3.Text, T1.DisplayName FROM users AS T1 INNER JOIN posts AS T2 ON T1.Id = T2.OwnerUserId INNER JOIN comments AS T3 ON T2.Id = T3.PostId WHERE T2.Title = 'Analysing wind data with R' ORDER BY T1.CreationDate DESC LIMIT 10",
+      "baseline_pred": "SELECT comments.Text, comments.UserDisplayName FROM comments JOIN posts ON comments.PostId = posts.Id WHERE posts.Title = 'Analysing wind data with R' ORDER BY comments.CreationDate DESC LIMIT 10",
+      "alt_pred": "SELECT c.Text, c.UserDisplayName FROM comments AS c JOIN posts AS p ON c.PostId = p.Id WHERE p.Title = 'Analysing wind data with R' ORDER BY c.CreationDate DESC LIMIT 10",
+      "alt_confidence": 0.99,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "agree",
+      "agree": true,
+      "elapsed_ms": 43493.225999998685
+    },
+    {
+      "question_id": 894,
+      "db_id": "formula_1",
+      "difficulty": "moderate",
+      "question": "What is the best lap time recorded? List the driver and race with such recorded lap time.",
+      "gold_sql": "SELECT T2.milliseconds, T1.forename, T1.surname, T3.name FROM drivers AS T1 INNER JOIN lapTimes AS T2 ON T1.driverId = T2.driverId INNER JOIN races AS T3 ON T2.raceId = T3.raceId ORDER BY T2.milliseconds ASC LIMIT 1",
+      "baseline_pred": "SELECT drivers.forename, drivers.surname, races.name FROM lapTimes JOIN drivers ON lapTimes.driverId = drivers.driverId JOIN races ON lapTimes.raceId = races.raceId WHERE lapTimes.milliseconds = (SELECT MIN(milliseconds) FROM lapTimes) LIMIT 1",
+      "alt_pred": "SELECT d.forename, d.surname, r.name FROM lapTimes lt JOIN drivers d ON lt.driverId = d.driverId JOIN races r ON lt.raceId = r.raceId WHERE lt.milliseconds = (SELECT MIN(milliseconds) FROM lapTimes)",
+      "alt_confidence": 0.99,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "agree",
+      "agree": true,
+      "elapsed_ms": 13408.637300002738
+    },
+    {
+      "question_id": 896,
+      "db_id": "formula_1",
+      "difficulty": "challenging",
+      "question": "Calculate the percentage whereby Hamilton was not at the 1st track of the the f1 circuit since 2010.",
+      "gold_sql": "SELECT CAST(COUNT(CASE WHEN T2.position <> 1 THEN T2.position END) AS REAL) * 100 / COUNT(T2.driverStandingsId) FROM races AS T1 INNER JOIN driverStandings AS T2 ON T2.raceId = T1.raceId INNER JOIN drivers AS T3 ON T3.driverId = T2.driverId WHERE T3.surname = 'Hamilton' AND T1.year >= 2010",
+      "baseline_pred": "SELECT CAST(SUM(CASE WHEN r.positionOrder > 1 THEN 1 ELSE 0 END) AS REAL) * 100.0 / COUNT(*) FROM results r INNER JOIN drivers d ON r.driverId = d.driverId INNER JOIN races ra ON r.raceId = ra.raceId WHERE d.surname = 'Hamilton' AND ra.year >= 2010",
+      "alt_pred": "SELECT CAST(SUM(CASE WHEN r.position > 1 THEN 1 ELSE 0 END) AS REAL) * 100.0 / COUNT(*) AS percentage FROM results r JOIN drivers d ON r.driverId = d.driverId JOIN races ra ON r.raceId = ra.raceId WHERE d.surname = 'Hamilton' AND ra.year >= 2010",
+      "alt_confidence": 0.99,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "alt-pick",
+      "agree": false,
+      "elapsed_ms": 48147.269699999015
+    },
+    {
+      "question_id": 930,
+      "db_id": "formula_1",
+      "difficulty": "simple",
+      "question": "In which Formula_1 race did Lewis Hamilton rank the highest?",
+      "gold_sql": "SELECT name FROM races WHERE raceId IN ( SELECT raceId FROM results WHERE rank = 1 AND driverId = ( SELECT driverId FROM drivers WHERE forename = 'Lewis' AND surname = 'Hamilton' ) )",
+      "baseline_pred": "SELECT races.name FROM races JOIN results ON races.raceId = results.raceId JOIN drivers ON results.driverId = drivers.driverId WHERE drivers.forename = 'Lewis' AND drivers.surname = 'Hamilton' ORDER BY results.rank ASC LIMIT 1",
+      "alt_pred": "SELECT races.name FROM results JOIN drivers ON results.driverId = drivers.driverId JOIN races ON results.raceId = races.raceId WHERE drivers.forename = 'Lewis' AND drivers.surname = 'Hamilton' ORDER BY results.rank ASC LIMIT 1",
+      "alt_confidence": 0.95,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "agree",
+      "agree": true,
+      "elapsed_ms": 7533.188500001415
+    },
+    {
+      "question_id": 988,
+      "db_id": "formula_1",
+      "difficulty": "challenging",
+      "question": "List down top 3 German drivers who has the shortest average pit stop duration and were born between 1980-1985.",
+      "gold_sql": "SELECT T2.forename, T2.surname FROM pitStops AS T1 INNER JOIN drivers AS T2 on T1.driverId = T2.driverId WHERE T2.nationality = 'German' AND STRFTIME('%Y', T2.dob) BETWEEN '1980' AND '1985' GROUP BY T2.forename, T2.surname ORDER BY AVG(T1.duration) LIMIT 3",
+      "baseline_pred": "SELECT d.forename || ' ' || d.surname AS full_name FROM pitStops p JOIN drivers d ON p.driverId = d.driverId WHERE strftime('%Y', d.dob) BETWEEN '1980' AND '1985' AND d.nationality = 'German' GROUP BY d.driverId, d.forename, d.surname ORDER BY AVG(p.milliseconds) ASC LIMIT 3",
+      "alt_pred": "SELECT d.forename || ' ' || d.surname AS full_name, ROUND(SUM(CAST(ps.duration AS REAL))/COUNT(ps.duration), 3) AS avg_duration FROM drivers d JOIN pitStops ps ON d.driverId = ps.driverId WHERE d.nationality = 'German' AND d.dob BETWEEN '1980-01-01' AND '1985-12-31' GROUP BY d.driverId, d.forename, d.surname ORDER BY avg_duration ASC LIMIT 3",
+      "alt_confidence": 0.99,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "alt-pick",
+      "agree": false,
+      "elapsed_ms": 77698.09810000152
+    },
+    {
+      "question_id": 990,
+      "db_id": "formula_1",
+      "difficulty": "challenging",
+      "question": "What is the constructor reference name of the champion in the 2009 Singapore Grand Prix? Please give its website.",
+      "gold_sql": "SELECT T3.constructorRef, T3.url FROM results AS T1 INNER JOIN races AS T2 on T1.raceId = T2.raceId INNER JOIN constructors AS T3 on T1.constructorId = T3.constructorId WHERE T2.name = 'Singapore Grand Prix' AND T2.year = 2009 AND T1.time LIKE '_:%:__.___'",
+      "baseline_pred": "SELECT constructors.constructorRef, constructors.url FROM results INNER JOIN races ON results.raceId = races.raceId INNER JOIN constructors ON results.constructorId = constructors.constructorId WHERE races.year = 2009 AND races.name = 'Singapore Grand Prix' AND results.positionOrder = 1 ORDER BY results.time DESC LIMIT 1",
+      "alt_pred": "SELECT c.constructorRef, c.url FROM constructors AS c JOIN results AS r ON c.constructorId = r.constructorId JOIN races AS ra ON r.raceId = ra.raceId WHERE ra.year = 2009 AND ra.name = 'Singapore Grand Prix' AND r.position = 1",
+      "alt_confidence": 0.95,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "agree",
+      "agree": true,
+      "elapsed_ms": 55691.28920000003
+    },
+    {
+      "question_id": 1168,
+      "db_id": "thrombosis_prediction",
+      "difficulty": "challenging",
+      "question": "The oldest SJS patient's medical laboratory work was completed on what date, and what age was the patient when they initially arrived at the hospital?",
+      "gold_sql": "SELECT T1.Date, STRFTIME('%Y', T2.`First Date`) - STRFTIME('%Y', T2.Birthday),T2.Birthday FROM Laboratory AS T1 INNER JOIN Patient AS T2 ON T1.ID = T2.ID WHERE T2.Diagnosis = 'SJS' AND T2.Birthday IS NOT NULL ORDER BY T2.Birthday ASC LIMIT 1",
+      "baseline_pred": "SELECT L.Date, (strftime('%Y', P.\"First Date\") - strftime('%Y', P.Birthday)) FROM Patient P INNER JOIN Laboratory L ON P.ID = L.ID WHERE P.Diagnosis = 'SJS' AND P.Birthday = (SELECT MIN(Birthday) FROM Patient WHERE Diagnosis = 'SJS') ORDER BY L.Date ASC LIMIT 1",
+      "alt_pred": "",
+      "alt_confidence": 0.0,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "base-fallback",
+      "agree": false,
+      "elapsed_ms": 67138.51780000186
+    },
+    {
+      "question_id": 1205,
+      "db_id": "thrombosis_prediction",
+      "difficulty": "moderate",
+      "question": "Was the patient with the number 57266's uric acid within a normal range?",
+      "gold_sql": "SELECT CASE WHEN (T1.SEX = 'F' AND T2.UA > 6.5) OR (T1.SEX = 'M' AND T2.UA > 8.0) THEN true ELSE false END FROM Patient AS T1 INNER JOIN Laboratory AS T2 ON T1.ID = T2.ID WHERE T1.ID = 57266",
+      "baseline_pred": "SELECT CASE WHEN (T1.UA > 8.0 AND T2.SEX = 'M') OR (T1.UA > 6.5 AND T2.SEX = 'F') THEN 'Yes' ELSE 'No' END AS is_normal FROM Laboratory AS T1 INNER JOIN Patient AS T2 ON T1.ID = T2.ID WHERE T1.ID = 57266 LIMIT 1",
+      "alt_pred": "SELECT CASE WHEN (L.UA > 8.0 AND P.SEX = 'M') OR (L.UA > 6.5 AND P.SEX = 'F') THEN 'Yes' ELSE 'No' END AS within_normal FROM Laboratory AS L JOIN Patient AS P ON L.ID = P.ID WHERE L.ID = 57266",
+      "alt_confidence": 0.99,
+      "baseline_match": false,
+      "alt_match": false,
+      "vote_match": false,
+      "vote_source": "alt-pick",
+      "agree": false,
+      "elapsed_ms": 35649.98669999841
+    },
+    {
+      "question_id": 1232,
+      "db_id": "thrombosis_prediction",
+      "difficulty": "challenging",
+      "question": "Provide ID, sex and age of patient who has blood glucose (GLU) not within normal range but with total cholesterol(T-CHO) within normal range.",
+      "gold_sql": "SELECT DISTINCT T1.ID, T1.SEX , STRFTIME('%Y', CURRENT_TIMESTAMP) - STRFTIME('%Y', T1.Birthday) FROM Patient AS T1 INNER JOIN Laboratory AS T2 ON T1.ID = T2.ID WHERE T2.GLU >= 180 AND T2.`T-CHO` < 250",
+      "baseline_pred": "SELECT DISTINCT p.ID, p.SEX, (strftime('%Y', date('now')) - strftime('%Y', p.Birthday)) AS age FROM Patient p INNER JOIN Laboratory l ON p.ID = l.ID WHERE l.GLU >= 180 AND l.T-CHO < 250",
+      "alt_pred": "SELECT DISTINCT P.ID, P.SEX, (strftime('%Y','now') - strftime('%Y', P.Birthday)) AS age FROM Patient P JOIN Laboratory L ON P.ID = L.ID WHERE L.GLU >= 180 AND L.`T-CHO` < 250",
+      "alt_confidence": 0.95,
+      "baseline_match": false,
+      "alt_match": true,
+      "vote_match": true,
+      "vote_source": "alt-pick",
+      "agree": false,
+      "elapsed_ms": 39681.59949999972
+    }
+  ]
+}

eval/reports/2026-05-17/groq-llama70b-on-v8-residue.json

@@ -0,0 +1,11 @@
+{
+  "alt_model": "llama-3.3-70b-versatile",
+  "summary": {
+    "voted_better": 0,
+    "voted_worse": 0,
+    "voted_same": 0,
+    "groq_input_tokens": 0,
+    "groq_output_tokens": 0
+  },
+  "records": []
+}

eval/reports/2026-05-17/groq-qwen3-on-v8-residue.json

@@ -0,0 +1,11 @@
+{
+  "alt_model": "qwen/qwen3-32b",
+  "summary": {
+    "voted_better": 0,
+    "voted_worse": 0,
+    "voted_same": 0,
+    "groq_input_tokens": 0,
+    "groq_output_tokens": 0
+  },
+  "records": []
+}