Add 4 MoNaCo-parity tabs: Unified eval, Eval Structures v0, Baseline A/B/C trajectories, Compare

Round 1 (4 tabs):
- 🔧 Unified eval — 500-qid subset from $DATA_ROOT/eval/wiki_opentable/unified/test_with_chunks.unified.jsonl
- 🧬 Eval Structures v0 — typed structure docs from test_with_structures.unified.jsonl
- 🛠 Trajectory — Baseline A (per-qid scaffolds, cell4_agentic_a), F1 78.95
- 🛠 Trajectory (Structure Corpus) — Baseline B (flat structure corpus, cell5_agentic_b_corpus), F1 74.84

Round 2 (2 more tabs):
- 🛠 Trajectory (Rawtext Corpus) — Baseline C (flat rawtext corpus, cell6_agentic_c_rawtext), F1 89.22
- 📊 Compare — per-qid side-by-side grid across all 6 cells with local set-based F1/EM
(no LLM judge for wiki_opentable; the deterministic wiki_opentable_adapter.py
is the source of truth — same scorer the trajectory tabs use)

Per-cell mean F1 / EM in the Compare tab:
closed-book F1 22.03 EM 21.00
with-docs F1 85.82 EM 84.80
with-structures F1 80.90 EM 79.80
agentic-a F1 78.95 EM 77.80
agentic-b-corpus F1 74.84 EM 74.00
agentic-c-rawtext F1 89.22 EM 88.00

Sidebar shows a 6-bar sparkline per qid; a 'split' filter narrows to
rows where the cells disagree.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

README.md

@@ -20,12 +20,39 @@ For each of the **6,602** test questions you can inspect:
 
 ## View modes
 
-The topbar's view-mode toggle switches between two parallel datasets:
+The topbar's view-mode toggle switches between eight parallel views (full
+parity with `monaco-benchmark-viewer` where data exists for `wiki_opentable`):
 
 - **📑 Dataset** (default) — browse all 6,602 test questions with their gold answer, reference SQL, and the three buckets of candidate table chunks (hard-positive / positive / negative).
+- **🔧 Unified eval** — the same 500-qid subset used by every downstream eval, fetched from the cross-dataset *unified* bundle (`$DATA_ROOT/eval/wiki_opentable/unified/test_with_chunks.unified.jsonl`). Shows the gold question + gold answers + every gold supporting doc rendered as markdown (the wiki_opentable docs are already markdown tables).
+- **🧬 Eval Structures v0** — the typed structure docs (e.g. `structures/entity/…`, `structures/_raw/…`) that the structures-based eval consumes, fetched from `test_with_structures.unified.jsonl`. Grouped by subtype in the UI.
+- **🛠 Trajectory** — the standalone `agentic_answer` **Baseline A** run on the 500-qid subset (`outputs/agentic_wiki_opentable/cell4_agentic_a.response.jsonl` — per-qid scaffolds). Side-by-side gold + parsed prediction + canonical per-qid F1 / strict-EM / P / R + the full agent loop (events). Mean F1 ≈ 78.95.
+- **🛠 Trajectory (Structure Corpus)** — the parallel **Baseline B** run (`cell5_agentic_b_corpus.response.jsonl` — flat structure corpus). Mean F1 ≈ 74.84.
+- **🛠 Trajectory (Rawtext Corpus)** — the parallel **Baseline C** run (`cell6_agentic_c_rawtext.response.jsonl` — flat rawtext corpus). Mean F1 ≈ 89.22.
+- **📊 Compare** — per-qid side-by-side grid of all 6 cells (closed-book, with-docs, with-structures, agentic A/B/C). Each card shows the model's answer, the parsed prediction items, and the canonical **local set-based F1 / EM / P / R** (no LLM-judge file exists for `wiki_opentable` — the deterministic `wiki_opentable_adapter.py` is the source of truth). The highest-F1 cell per qid is flagged with 🏆. Sidebar rows show a 6-bar sparkline so you can spot disagreement at a glance; a "split" filter narrows to qids where the cells disagree.
 - **🤖 E2E Trajectories** — for the 500 wikitq-* test questions of the **information-scaffolds e2e pipeline** run on `wiki_opentable`, inspect each agent trajectory side-by-side with the gold answer, the parsed prediction, and **per-qid F1 / strict-EM / precision / recall** (computed with the canonical `wiki_opentable_adapter.py` set-based scoring + normalization). Sidebar rows are colour-coded by F1 bucket; filter pills narrow by `stop_reason` (`end_turn` / `error` / `max_turns`) or by score bucket (EM=1 / 0<F1<1 / F1=0).
 
-Deep-link via the URL hash: `#dataset:wikisql-00116` or `#e2e:wikitq-00030`. Legacy `#wikisql-00116` links resolve to dataset mode.
+The four trajectory tabs share the same sidebar filter pills (stop_reason
++ score bucket); `Dataset`, `Unified eval`, `Eval Structures v0`, and
+`Compare` share the WikiSQL / WikiTQ dataset-origin filter pills. The
+Compare tab adds its own *agreement* filter (`all` / `all EM` / `all F1=0`
+/ `split`) so you can drill into the rows where the cells disagree.
+
+> The Structures v0 / v1 / v2 generator tabs from the MoNaCo viewer are
+> intentionally absent: `wiki_opentable` was never paired with a
+> structures-generation AML response JSONL (only the e2e-pipeline
+> hierarchical structure dirs exist).
+
+Deep-link via the URL hash:
+`#dataset:wikisql-00116` ·
+`#unified:wikitq-00030` ·
+`#eval_structures:wikitq-00030` ·
+`#trajectories:wikitq-00030` ·
+`#trajectories_corpus:wikitq-00030` ·
+`#trajectories_rawtext:wikitq-00030` ·
+`#compare:wikitq-00030` ·
+`#e2e:wikitq-00030`.
+Legacy `#wikisql-00116` links resolve to dataset mode.
 
 ## Local dev
 
@@ -61,3 +88,43 @@ python scripts/build_trajectories_e2e.py \
 
 The script inlines the canonical scoring (`wiki_opentable_adapter.py` + `_parse_exact_answer.py`) so it has no dependency on the `information-scaffolds` repo. It writes `trajectories_e2e/index.json` + `trajectories_e2e/records/<qid>.json`; per-event tool-result content is truncated at 8 KB to keep each shard browser-friendly.
 
+## Rebuilding the other 5 shard collections
+
+```bash
+# 🔧 Unified eval (default --src points at $DATA_ROOT/eval/wiki_opentable/unified/test_with_chunks.unified.jsonl)
+python scripts/build_unified.py
+
+# 🧬 Eval Structures v0 (default --src points at .../test_with_structures.unified.jsonl)
+python scripts/build_eval_structures.py
+
+# 🛠 Trajectory (Baseline A — per-qid scaffolds, cell4)
+python scripts/build_trajectories.py \
+  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell4_agentic_a.response.jsonl \
+  --out         trajectories \
+  --label       "Baseline A — per-qid scaffolds (cell4_agentic_a)"
+
+# 🛠 Trajectory (Structure Corpus) (Baseline B — flat corpus, cell5)
+python scripts/build_trajectories.py \
+  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell5_agentic_b_corpus.response.jsonl \
+  --out         trajectories_corpus \
+  --label       "Baseline B — flat structure corpus (cell5_agentic_b_corpus)"
+
+# 🛠 Trajectory (Rawtext Corpus) (Baseline C — flat rawtext corpus, cell6)
+python scripts/build_trajectories.py \
+  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell6_agentic_c_rawtext.response.jsonl \
+  --out         trajectories_rawtext \
+  --label       "Baseline C — flat rawtext corpus (cell6_agentic_c_rawtext)"
+
+# 📊 Compare (per-qid grid across all 6 cells; local set-based F1/EM)
+python scripts/build_compare.py
+```
+
+Each script writes `<out>/index.json` + `<out>/records/<qid>.json` and
+clears stale shards from a previous build so the index and on-disk shards
+never drift. The three trajectory scripts share the canonical
+semicolon-split parser + set-based F1/EM scoring with
+`build_trajectories_e2e.py` (inlined — no `information-scaffolds`
+dependency). `build_compare.py` reuses the same parser + scorer (also
+inlined) so per-cell numbers in the Compare tab match the trajectory tabs
+and `evaluation/CANONICAL.md` exactly.
+

compare/records/wikitq-00030.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00030", "question": "Which Millbrook First Nation communities have at least 100 residents?", "dataset_origin": "WikiTQ", "original_table_id": "204-251", "sql": "SELECT Community FROM table_204_251 WHERE Population >= '100' ", "gold_answers": ["Cole Harbour 30", "Millbrook 27"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Millbrook First Nation has several reserve communities, but the only one I know with a resident population of at least 100 is Millbrook 27. Smaller associated reserves like Cole Harbour 30 do not meet that threshold.\nExact Answer: Millbrook 27\nConfidence: 67%", "pred_items": ["Millbrook 27"], "metrics": {"precision": 1.0, "recall": 0.5, "f1": 0.6667, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 11031.34, "finish_reason": "stop", "usage": {"prompt_tokens": 423, "completion_tokens": 499, "reasoning_tokens": 430, "total_tokens": 922}}, "with-docs": {"answer": "Explanation: I read the table of Millbrook First Nation reserves and selected the communities whose listed population is at least 100. Those are Cole Harbour 30 with 194 residents and Millbrook 27 with 847 residents.\nExact Answer: Cole Harbour 30; Millbrook 27\nConfidence: 99%", "pred_items": ["Cole Harbour 30", "Millbrook 27"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3142.67, "finish_reason": "stop", "usage": {"prompt_tokens": 757, "completion_tokens": 96, "reasoning_tokens": 23, "total_tokens": 853}}, "with-structures": {"answer": "Explanation: I checked the population values in the provided table and selected the communities with at least 100 residents. The only ones meeting that threshold are Cole Harbour 30 with 194 residents and Millbrook 27 with 847 residents.\nExact Answer: Cole Harbour 30; Millbrook 27\nConfidence: 98%", "pred_items": ["Cole Harbour 30", "Millbrook 27"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4441.82, "finish_reason": "stop", "usage": {"prompt_tokens": 1692, "completion_tokens": 136, "reasoning_tokens": 61, "total_tokens": 1828}}, "agentic-a": {"answer": "Explanation: The answer is grounded in the entity table `structures/structures/entity/wiki_opentable_wikitq_00030__Reserve_master_lookup.entities.csv`, which lists each community’s population; the threshold is also consistent with `structures/structures/_raw/wiki_opentable_wikitq_00030__Rankings_and_extrema.md`.\nExact Answer: Cole Harbour 30; Millbrook 27\nConfidence: 99%", "pred_items": ["Cole Harbour 30", "Millbrook 27"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 62989.71, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 12517, "completion_tokens": 567, "total_tokens": 13084, "reasoning_tokens": 277}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The answer is grounded in `structures/structures/entity/wiki_opentable_wikitq_00030__Reserve_master_lookup.entities.csv`, which lists each Millbrook First Nation reserve/community and its `population_int`, and is corroborated by `structures/structures/_raw/wiki_opentable_wikitq_00030__Rankings_and_extrema.md`.\nExact Answer: Millbrook 27 (847); Cole Harbour 30 (194)\nConfidence: 98%", "pred_items": ["Millbrook 27 (847)", "Cole Harbour 30 (194)"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 276633.34, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 50, "tool_call_counts": {"bash": 8, "python": 2, "read_file": 3}, "tokens": {"prompt_tokens": 30670, "completion_tokens": 1574, "total_tokens": 32244, "reasoning_tokens": 640}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In structures/doc_046405.txt (original id 51762), the “Millbrook First Nation - Reserves” table lists each community and its population; the only communities with populations of at least 100 are Cole Harbour 30 (194) and Millbrook 27 (847).\nExact Answer: Cole Harbour 30; Millbrook 27\nConfidence: 99%", "pred_items": ["Cole Harbour 30", "Millbrook 27"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 874178.91, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 8}, "tokens": {"prompt_tokens": 14591, "completion_tokens": 849, "total_tokens": 15440, "reasoning_tokens": 405}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00031.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00031", "question": "Total First Nation Population of Millbrook ?", "dataset_origin": "WikiTQ", "original_table_id": "204-251", "sql": "SELECT Population FROM table_204_251 WHERE Community = 'Millbrook 27' ", "gold_answers": ["847"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: The question appears to ask for the total registered population of Millbrook First Nation. A commonly cited figure for Millbrook First Nation’s total population is 1,845.\nExact Answer: 1845\nConfidence: 54%", "pred_items": ["1845"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4264.29, "finish_reason": "stop", "usage": {"prompt_tokens": 418, "completion_tokens": 166, "reasoning_tokens": 108, "total_tokens": 584}}, "with-docs": {"answer": "Explanation: The table lists populations for Millbrook First Nation reserves as 23, 194, 847, 15, 0, 0, and 0. Adding them gives 1,079.\nExact Answer: 1079\nConfidence: 98%", "pred_items": ["1079"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3092.98, "finish_reason": "stop", "usage": {"prompt_tokens": 752, "completion_tokens": 113, "reasoning_tokens": 48, "total_tokens": 865}}, "with-structures": {"answer": "Explanation: The table lists Millbrook 27 with a population of 847, which is the total First Nation population for Millbrook in the provided data.\nExact Answer: 847\nConfidence: 99%", "pred_items": ["847"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2046.86, "finish_reason": "stop", "usage": {"prompt_tokens": 1740, "completion_tokens": 65, "reasoning_tokens": 13, "total_tokens": 1805}}, "agentic-a": {"answer": "Explanation: The entity table `wiki_opentable_wikitq_00031__Normalized_reserve_reference_index.entities.csv` lists Millbrook 27 with `population_int = 847`, and the ranking summary in `wiki_opentable_wikitq_00031__Comparative_rankings_and_extrema.md` confirms “Millbrook 27 — 847.”\nExact Answer: 847\nConfidence: 99%", "pred_items": ["847"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 54657.44, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 9060, "completion_tokens": 424, "total_tokens": 9484, "reasoning_tokens": 117}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The answer is grounded in `structures/structures/entity/wiki_opentable_wikitq_00030__Reserve_master_lookup.entities.csv`, where `Millbrook 27` has `population_int` 847, and corroborated by `structures/structures/_raw/wiki_opentable_wikitq_00031__Aggregates_and_derived_metrics.md`, which gives the Truro-related subtotal including Millbrook as 847.\nExact Answer: 847\nConfidence: 96%", "pred_items": ["847"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 212057.01, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 50, "tool_call_counts": {"bash": 7, "read_file": 5}, "tokens": {"prompt_tokens": 21342, "completion_tokens": 1277, "total_tokens": 22619, "reasoning_tokens": 452}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_046405.txt` (original id 51762, “Millbrook First Nation - Reserves”), the listed populations are 23, 194, 847, 15, 0, 0, and 0; these sum to 1079.\nExact Answer: 1079\nConfidence: 99%", "pred_items": ["1079"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1241290.04, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 15, "tool_call_counts": {"bash": 7, "read_file": 1}, "tokens": {"prompt_tokens": 26782, "completion_tokens": 1192, "total_tokens": 27974, "reasoning_tokens": 733}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00032.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00032", "question": "How many Millbrook First Nation communities have zero populations?", "dataset_origin": "WikiTQ", "original_table_id": "204-251", "sql": "SELECT COUNT ( Community ) FROM table_204_251 WHERE Population = 0 ", "gold_answers": ["3"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Millbrook First Nation has multiple reserve communities, and all except Millbrook 27 have reported populations of zero. That makes the number of communities with zero population 3.\nExact Answer: 3\nConfidence: 76%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 13368.62, "finish_reason": "stop", "usage": {"prompt_tokens": 421, "completion_tokens": 574, "reasoning_tokens": 517, "total_tokens": 995}}, "with-docs": {"answer": "Explanation: I read the table of Millbrook First Nation reserves and counted the communities with a population listed as 0. These are Truro 27A, Truro 27B, and Truro 27C, for a total of three.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2987.16, "finish_reason": "stop", "usage": {"prompt_tokens": 755, "completion_tokens": 91, "reasoning_tokens": 19, "total_tokens": 846}}, "with-structures": {"answer": "Explanation: From the table, the communities with population 0 are Truro 27A, Truro 27B, and Truro 27C. That makes 3 Millbrook First Nation communities with zero population.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3550.8, "finish_reason": "stop", "usage": {"prompt_tokens": 1652, "completion_tokens": 107, "reasoning_tokens": 41, "total_tokens": 1759}}, "agentic-a": {"answer": "Explanation: The entity table `structures/structures/entity/wiki_opentable_wikitq_00032__Normalized_reserve_reference.entities.csv` lists populations for Millbrook First Nation communities, and three entries—Truro 27A, Truro 27B, and Truro 27C—have `population_int` of 0; this is also confirmed in the rankings markdown.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 56996.73, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8933, "completion_tokens": 396, "total_tokens": 9329, "reasoning_tokens": 143}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The count comes from `wiki_opentable_wikitq_00030__Reserve_master_lookup.entities.csv`, which lists populations for Millbrook First Nation reserves, and `wiki_opentable_wikitq_00030__Rankings_and_extrema.md`, which identifies Truro 27A, 27B, and 27C as having population 0.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 154300.49, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 5, "read_file": 3}, "tokens": {"prompt_tokens": 14339, "completion_tokens": 817, "total_tokens": 15156, "reasoning_tokens": 252}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_046405.txt` (original id `51762`), the Millbrook First Nation reserves table lists populations for each community; three communities—Truro 27A, Truro 27B, and Truro 27C—have population `0`.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 486477.2, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 3}, "tokens": {"prompt_tokens": 9654, "completion_tokens": 441, "total_tokens": 10095, "reasoning_tokens": 212}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00033.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00033", "question": "Which community has the largest population among the Millbrook First Nation?", "dataset_origin": "WikiTQ", "original_table_id": "204-251", "sql": "SELECT Community FROM table_204_251 ORDER BY Population DESC LIMIT 1 ", "gold_answers": ["Millbrook 27"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Millbrook First Nation’s main community, Millbrook 27 near Truro, is the most populous of its communities. Therefore, the community with the largest population is Millbrook 27.\nExact Answer: Millbrook 27\nConfidence: 78%", "pred_items": ["Millbrook 27"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 6820.66, "finish_reason": "stop", "usage": {"prompt_tokens": 423, "completion_tokens": 302, "reasoning_tokens": 239, "total_tokens": 725}}, "with-docs": {"answer": "Explanation: I compared the population values listed for each Millbrook First Nation community: Beaver Lake 17 (23), Cole Harbour 30 (194), Millbrook 27 (847), Sheet Harbour 36 (15), Truro 27A (0), Truro 27B (0), and Truro 27C (0). The largest population is 847 for Millbrook 27.\nExact Answer: Millbrook 27\nConfidence: 99%", "pred_items": ["Millbrook 27"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3187.26, "finish_reason": "stop", "usage": {"prompt_tokens": 757, "completion_tokens": 117, "reasoning_tokens": 13, "total_tokens": 874}}, "with-structures": {"answer": "Explanation: From the table, the populations are: Beaver Lake 17 (23), Cole Harbour 30 (194), Millbrook 27 (847), Sheet Harbour 36 (15), Truro 27A (0), Truro 27B (0), and Truro 27C (0). The largest population is 847 for Millbrook 27.\nExact Answer: Millbrook 27\nConfidence: 98%", "pred_items": ["Millbrook 27"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3730.03, "finish_reason": "stop", "usage": {"prompt_tokens": 1493, "completion_tokens": 128, "reasoning_tokens": 30, "total_tokens": 1621}}, "agentic-a": {"answer": "Explanation: The entity table in `structures/entity/wiki_opentable_wikitq_00033__Reserve_master_index.entities.csv` lists populations for each community, and the numeric summary in `structures/_raw/wiki_opentable_wikitq_00033__Ranked_comparisons_and_extrema.md` confirms the highest population is 847 for Millbrook 27.\nExact Answer: Millbrook 27\nConfidence: 99%", "pred_items": ["Millbrook 27"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 54837.57, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8840, "completion_tokens": 419, "total_tokens": 9259, "reasoning_tokens": 171}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The answer is directly supported by `structures/structures/entity/wiki_opentable_wikitq_00030__Reserve_master_lookup.entities.csv` and `structures/structures/_raw/wiki_opentable_wikitq_00030__Rankings_and_extrema.md`, which rank the listed Millbrook First Nation communities by population.\nExact Answer: Millbrook 27\nConfidence: 98%", "pred_items": ["Millbrook 27"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 139364.99, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 5, "read_file": 6}, "tokens": {"prompt_tokens": 15847, "completion_tokens": 923, "total_tokens": 16770, "reasoning_tokens": 240}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_046405.txt` (original id `51762`), the “Millbrook First Nation - Reserves” table lists populations for each community; `Millbrook 27` has the highest population at 847, higher than Cole Harbour 30 (194) and the others.\nExact Answer: Millbrook 27\nConfidence: 99%", "pred_items": ["Millbrook 27"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 484308.3, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 2}, "tokens": {"prompt_tokens": 9561, "completion_tokens": 322, "total_tokens": 9883, "reasoning_tokens": 126}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00034.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00034", "question": "What is the total population of Sheet Harbor 36 for Millbrook First Nation?", "dataset_origin": "WikiTQ", "original_table_id": "204-251", "sql": "SELECT Population FROM table_204_251 WHERE Community = 'Sheet Harbour 36' ", "gold_answers": ["15"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Sheet Harbor 36 is a reserve of the Millbrook First Nation, and its listed total population is 0.\nExact Answer: 0\nConfidence: 62%", "pred_items": ["0"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 8005.68, "finish_reason": "stop", "usage": {"prompt_tokens": 426, "completion_tokens": 321, "reasoning_tokens": 275, "total_tokens": 747}}, "with-docs": {"answer": "Explanation: In the table for Millbrook First Nation reserves, the row for Sheet Harbour 36 lists a population of 15. Therefore, the total population is 15.\nExact Answer: 15\nConfidence: 99%", "pred_items": ["15"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2528.63, "finish_reason": "stop", "usage": {"prompt_tokens": 760, "completion_tokens": 74, "reasoning_tokens": 18, "total_tokens": 834}}, "with-structures": {"answer": "Explanation: In the table, Sheet Harbour 36 is listed with a population of 15. The question asks for the total population of Sheet Harbor/Harbour 36, so that value is the answer.\nExact Answer: 15\nConfidence: 97%", "pred_items": ["15"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3386.94, "finish_reason": "stop", "usage": {"prompt_tokens": 1661, "completion_tokens": 106, "reasoning_tokens": 43, "total_tokens": 1767}}, "agentic-a": {"answer": "Explanation: The entity record in `structures/structures/entity/wiki_opentable_wikitq_00034__Canonical_reserve_index.entities.csv` lists `Sheet Harbour 36` with `population_int = 15`; this is corroborated by the population ranking in `structures/structures/_raw/wiki_opentable_wikitq_00034__Rankings_extremes_and_grouped_comparisons.md`.\nExact Answer: 15\nConfidence: 99%", "pred_items": ["15"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 59512.48, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 3, "read_file": 2}, "tokens": {"prompt_tokens": 11941, "completion_tokens": 478, "total_tokens": 12419, "reasoning_tokens": 163}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: In `structures/structures/entity/wiki_opentable_wikitq_00030__Reserve_master_lookup.entities.csv` and `...00031__Normalized_reserve_reference_index.entities.csv`, the row for `Sheet Harbour 36` (Millbrook First Nation reserve) lists `population_int` as 15.\nExact Answer: 15\nConfidence: 98%", "pred_items": ["15"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 228740.71, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 50, "tool_call_counts": {"bash": 11, "read_file": 3}, "tokens": {"prompt_tokens": 25021, "completion_tokens": 1357, "total_tokens": 26378, "reasoning_tokens": 484}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_046405.txt` (original id 51762), the table “Millbrook First Nation - Reserves” lists the row for “Sheet Harbour 36” with a population of 15.\nExact Answer: 15\nConfidence: 99%", "pred_items": ["15"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 465952.46, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 5}, "tokens": {"prompt_tokens": 9770, "completion_tokens": 340, "total_tokens": 10110, "reasoning_tokens": 86}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00067.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00067", "question": "From Germany from the Eurovision Song Contest 1999, how many songs in total are listed in the table?", "dataset_origin": "WikiTQ", "original_table_id": "203-755", "sql": "SELECT COUNT ( Song_title ) FROM table_203_755 ", "gold_answers": ["11"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: The table for Germany’s national selection for the Eurovision Song Contest 1999 lists 12 competing songs in total.  \nExact Answer: 12\nConfidence: 71%", "pred_items": ["12"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 9467.71, "finish_reason": "stop", "usage": {"prompt_tokens": 432, "completion_tokens": 427, "reasoning_tokens": 380, "total_tokens": 859}}, "with-docs": {"answer": "Explanation: The two supporting tables list songs with singing positions 1 through 11. Counting all listed entries gives a total of 11 songs.\nExact Answer: 11\nConfidence: 99%", "pred_items": ["11"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2617.23, "finish_reason": "stop", "usage": {"prompt_tokens": 878, "completion_tokens": 70, "reasoning_tokens": 20, "total_tokens": 948}}, "with-structures": {"answer": "Explanation: The table for Germany's Eurovision Song Contest 1999 selection lists 11 songs total, with final positions running from 1 through 11.\nExact Answer: 11\nConfidence: 99%", "pred_items": ["11"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2714.48, "finish_reason": "stop", "usage": {"prompt_tokens": 2200, "completion_tokens": 88, "reasoning_tokens": 36, "total_tokens": 2288}}, "agentic-a": {"answer": "Explanation: The entity table `wiki_opentable_wikitq_00067__Unified_national_final_entry_index.entities.csv` explicitly lists 11 song rows, and the raw outcome table `wiki_opentable_wikitq_00067__Competitive_outcome_index.md` also runs from final position 1 to 11.\nExact Answer: 11\nConfidence: 99%", "pred_items": ["11"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 55084.89, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 9978, "completion_tokens": 487, "total_tokens": 10465, "reasoning_tokens": 207}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The count comes from `structures/structures/entity/wiki_opentable_wikitq_00069__Consolidated_final_results_and_key_status.entities.csv`, which lists 11 song rows and includes `total_listed_finalists_int = 11` for the Germany 1999 national final; `wiki_opentable_wikitq_00067__Unified_national_final_entry_index.entities.csv` corroborates the same 11 entries.\nExact Answer: 11\nConfidence: 98%", "pred_items": ["11"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 208992.51, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 50, "tool_call_counts": {"bash": 8, "read_file": 4}, "tokens": {"prompt_tokens": 25956, "completion_tokens": 1275, "total_tokens": 27231, "reasoning_tokens": 378}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: The Germany 1999 page is split across `doc_045355.txt` and `doc_045356.txt` (original ids 50817 and 50818). Their table rows run from singing positions 1–8 in `doc_045355.txt` and 9–11 in `doc_045356.txt`, for 11 songs total.\nExact Answer: 11\nConfidence: 99%", "pred_items": ["11"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1084357.14, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 9}, "tokens": {"prompt_tokens": 20283, "completion_tokens": 1070, "total_tokens": 21353, "reasoning_tokens": 459}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00068.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00068", "question": "Which song from Germany did not have more than 15% of the votes at the Eurovision Song Contest 1999 and was not unknown?", "dataset_origin": "WikiTQ", "original_table_id": "203-755", "sql": "SELECT Song_title FROM table_203_755 WHERE Percentage_of_votes <= '15' ", "gold_answers": ["Das tut unheimlich weh"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Germany’s entry at the Eurovision Song Contest 1999 was “Reise nach Jerusalem – Kudüs'e Seyahat” by Sürpriz. It received 140 points, which is under 15% of the total points awarded, and the song is clearly not titled or listed as unknown.  \nExact Answer: Reise nach Jerusalem – Kudüs'e Seyahat\nConfidence: 93%", "pred_items": ["Reise nach Jerusalem – Kudüs'e Seyahat"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 8072.5, "finish_reason": "stop", "usage": {"prompt_tokens": 438, "completion_tokens": 330, "reasoning_tokens": 239, "total_tokens": 768}}, "with-docs": {"answer": "Explanation: I checked the listed German songs and their vote percentages. The songs with a known percentage were: “Das tut unheimlich weh” (12.2%), “Reise nach Jerusalem - Kudüs`e Seyahat” (16.2%), “Hör den Kindern einfach zu” (32.6%), and “Together we`re strong” (15.9%). The only one that was not unknown and did not exceed 15% was “Das tut unheimlich weh.”\nExact Answer: Das tut unheimlich weh\nConfidence: 98%", "pred_items": ["Das tut unheimlich weh"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 6497.03, "finish_reason": "stop", "usage": {"prompt_tokens": 884, "completion_tokens": 262, "reasoning_tokens": 136, "total_tokens": 1146}}, "with-structures": {"answer": "Explanation: From the listed Germany entries with known vote percentages, the songs were: “Das tut unheimlich weh” (12.2%), “Reise nach Jerusalem - Kudüs`e Seyahat” (16.2%), “Hör den Kindern einfach zu” (32.6%), and “Together we`re strong” (15.9%). The only song that was not marked unknown and did not have more than 15% of the votes is “Das tut unheimlich weh.”\nExact Answer: Das tut unheimlich weh\nConfidence: 98%", "pred_items": ["Das tut unheimlich weh"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 5872.56, "finish_reason": "stop", "usage": {"prompt_tokens": 1832, "completion_tokens": 224, "reasoning_tokens": 97, "total_tokens": 2056}}, "agentic-a": {"answer": "Explanation: The entity table `structures/entity/wiki_opentable_wikitq_00068__Unified_candidate_lookup_across_both_split_documents.entities.csv` lists songs with vote percentages, and the raw anomaly notes confirm only four acts had known percentages. Among those, the only song with a known vote share not exceeding 15% is Jeanette Biedermann’s entry at 12.2%.\nExact Answer: Das tut unheimlich weh\nConfidence: 96%", "pred_items": ["Das tut unheimlich weh"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 62145.72, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 9224, "completion_tokens": 662, "total_tokens": 9886, "reasoning_tokens": 353}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The answer is grounded in `structures/structures/entity/wiki_opentable_wikitq_00069__Consolidated_final_results_and_key_status.entities.csv`, which lists each German entry’s `vote_percent`, and `structures/structures/_raw/wiki_opentable_wikitq_00069__Search_lookup_matrix_by_entity_type.md`, which confirms song-act mappings and flags the other low-ranked entries as having unknown vote shares.\nExact Answer: Das tut unheimlich weh\nConfidence: 97%", "pred_items": ["Das tut unheimlich weh"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 254195.45, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 50, "tool_call_counts": {"bash": 10, "read_file": 3}, "tokens": {"prompt_tokens": 38737, "completion_tokens": 1614, "total_tokens": 40351, "reasoning_tokens": 696}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In Germany’s Eurovision Song Contest 1999 national final table in `doc_045355.txt` (original id 50817), the only song with a known vote share not exceeding 15% is **“Das tut unheimlich weh”** at **12.2%**; the other known percentages are 15.9%, 16.2%, and 32.6%.\nExact Answer: Das tut unheimlich weh\nConfidence: 98%", "pred_items": ["Das tut unheimlich weh"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1310897.89, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 15, "tool_call_counts": {"bash": 9}, "tokens": {"prompt_tokens": 41689, "completion_tokens": 1670, "total_tokens": 43359, "reasoning_tokens": 896}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00069.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00069", "question": "How many songs are on the list for the Eurovision Song Contest 1999 in Germany?", "dataset_origin": "WikiTQ", "original_table_id": "203-755", "sql": "SELECT COUNT ( Song_title ) FROM table_203_755 ", "gold_answers": ["11"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: The Eurovision Song Contest 1999 featured one song from each participating country, and there were 23 participating countries in total, including Germany. So the list contained 23 songs.\n\nExact Answer: 23\n\nConfidence: 68%", "pred_items": ["23"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 13539.89, "finish_reason": "stop", "usage": {"prompt_tokens": 428, "completion_tokens": 558, "reasoning_tokens": 499, "total_tokens": 986}}, "with-docs": {"answer": "Explanation: The two supporting tables list entries with singing positions 1 through 11, so there are 11 songs in total on Germany’s Eurovision Song Contest 1999 selection list.\nExact Answer: 11\nConfidence: 99%", "pred_items": ["11"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2351.38, "finish_reason": "stop", "usage": {"prompt_tokens": 874, "completion_tokens": 76, "reasoning_tokens": 18, "total_tokens": 950}}, "with-structures": {"answer": "Explanation: The supporting table lists 11 finalist songs in Germany’s Eurovision Song Contest 1999 national final, and it explicitly states the total number of finalists as 11. Therefore, the number of songs on the list is 11.\nExact Answer: 11\nConfidence: 99%", "pred_items": ["11"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2871.43, "finish_reason": "stop", "usage": {"prompt_tokens": 2715, "completion_tokens": 84, "reasoning_tokens": 15, "total_tokens": 2799}}, "agentic-a": {"answer": "Explanation: The entity table `structures/entity/wiki_opentable_wikitq_00069__Consolidated_final_results_and_key_status.entities.csv` lists 11 song entries, and the claim file `structures/claim/wiki_opentable_wikitq_00069__Outcome_anomalies_and_interpretation_guide.claims.jsonl` explicitly states `number_of_finalists = 11`.\nExact Answer: 11\nConfidence: 99%", "pred_items": ["11"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 60005.16, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 2, "read_file": 2}, "tokens": {"prompt_tokens": 9790, "completion_tokens": 423, "total_tokens": 10213, "reasoning_tokens": 153}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The files `structures/structures/entity/wiki_opentable_wikitq_00069__Consolidated_final_results_and_key_status.entities.csv` and `structures/structures/claim/wiki_opentable_wikitq_00069__Outcome_anomalies_and_interpretation_guide.claims.jsonl` both show `total_listed_finalists_int` / `number_of_finalists` as 11 for Germany in the Eurovision Song Contest 1999 selection list.\nExact Answer: 11\nConfidence: 96%", "pred_items": ["11"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 279729.36, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 50, "tool_call_counts": {"bash": 10, "read_file": 3}, "tokens": {"prompt_tokens": 31676, "completion_tokens": 1696, "total_tokens": 33372, "reasoning_tokens": 643}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In the Germany 1999 Eurovision entry table split across doc_045355.txt (id 50817) and doc_045356.txt (id 50818), the listed songs are numbered 1 through 11, so the list contains 11 songs.\nExact Answer: 11\nConfidence: 99%", "pred_items": ["11"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1656267.34, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 15, "tool_call_counts": {"bash": 10}, "tokens": {"prompt_tokens": 35864, "completion_tokens": 1761, "total_tokens": 37625, "reasoning_tokens": 1157}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00124.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00124", "question": "How many opponents are playing at 4:00 p.m. in the 2009-10 Colgate Raiders women's hockey season?", "dataset_origin": "WikiTQ", "original_table_id": "204-718", "sql": "SELECT COUNT ( Opponent ) FROM table_204_718 WHERE Time_ = '4 p.m.' ", "gold_answers": ["10"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I interpret the question as asking for the count of scheduled opponents/games on the 2009–10 Colgate Raiders women’s hockey schedule that were listed with a 4:00 p.m. start time. Based on that schedule, the count is 12.\nExact Answer: 12\nConfidence: 34%", "pred_items": ["12"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 32849.55, "finish_reason": "stop", "usage": {"prompt_tokens": 437, "completion_tokens": 1668, "reasoning_tokens": 1591, "total_tokens": 2105}}, "with-docs": {"answer": "Explanation: I counted all schedule entries with the time listed as exactly 4 p.m. They are against University of Connecticut, Clarkson University, Princeton University, Yale University, Niagara University, Harvard University, St. Lawrence University, Quinnipiac University, Brown University, and Rensselaer — 10 total opponents/games at 4:00 p.m.\n\nExact Answer: 10\nConfidence: 98%", "pred_items": ["10"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 9660.0, "finish_reason": "stop", "usage": {"prompt_tokens": 1376, "completion_tokens": 424, "reasoning_tokens": 329, "total_tokens": 1800}}, "with-structures": {"answer": "Explanation: The provided references list the opponents and dates but not the game times. Based on the standard formatting and scheduling pattern for this season’s women’s hockey schedule, the 4:00 p.m. games appear to correspond to the Saturday/afternoon entries, yielding a count of 12 opponents/games at 4:00 p.m. This is the best-supported estimate from the available material.\nExact Answer: 12\nConfidence: 34%", "pred_items": ["12"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 26449.31, "finish_reason": "stop", "usage": {"prompt_tokens": 2985, "completion_tokens": 1048, "reasoning_tokens": 946, "total_tokens": 4033}}, "agentic-a": {"answer": "Explanation: I grounded this in the season schedule structures named in `structures/_index.json`, primarily the `time` unit `wiki_opentable_wikitq_00124__Unified_season_schedule_timeline.timeline.json`, with a check against the scored-result exceptions in `wiki_opentable_wikitq_00124__Confirmed_score_results_ledger.claims.jsonl`. Counting the 4:00 p.m. schedule entries gives 14.\nExact Answer: 14\nConfidence: 48%", "pred_items": ["14"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 260010.57, "stop_reason": "end_turn", "n_turns": 12, "max_turns": 35, "tool_call_counts": {"bash": 9, "read_file": 7, "python": 1}, "tokens": {"prompt_tokens": 88721, "completion_tokens": 12381, "total_tokens": 101102, "reasoning_tokens": 11317}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: I used `structures/structures/entity/wiki_opentable_wikitq_00125__Opponent_centric_lookup.entities.csv`, which lists each opponent and its `times_text` for the 2009-10 schedule. The rows with `4 p.m.` are Clarkson University, Princeton University, and University of Connecticut.\nExact Answer: 3\nConfidence: 97%", "pred_items": ["3"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 648183.46, "stop_reason": "end_turn", "n_turns": 20, "max_turns": 50, "tool_call_counts": {"bash": 36, "python": 3, "read_file": 5}, "tokens": {"prompt_tokens": 186891, "completion_tokens": 9548, "total_tokens": 196439, "reasoning_tokens": 6136}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In the schedule tables in `doc_048163.txt` (original id 53344) and `doc_048164.txt` (original id 53345), there are 10 games listed at `4 p.m.`: University of Connecticut, Clarkson University, Princeton University, Yale University, Niagara University, Harvard University, St. Lawrence University, Quinnipiac University, Brown University, and Rensselaer.\nExact Answer: 10\nConfidence: 98%", "pred_items": ["10"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1879721.11, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 15, "tool_call_counts": {"bash": 16, "read_file": 2}, "tokens": {"prompt_tokens": 54366, "completion_tokens": 3409, "total_tokens": 57775, "reasoning_tokens": 2399}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00125.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00125", "question": "How many games will be played at 2:00 p.m. in the 2009-10 Colgate Raiders hockey season?", "dataset_origin": "WikiTQ", "original_table_id": "204-718", "sql": "SELECT COUNT ( * ) FROM table_204_718 WHERE Time_ = '2 p.m.' ", "gold_answers": ["2"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I’m counting the scheduled games on the 2009–10 Colgate Raiders hockey season page that list a start time of 2:00 p.m. The total is 3.\nExact Answer: 3\nConfidence: 33%", "pred_items": ["3"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 22517.94, "finish_reason": "stop", "usage": {"prompt_tokens": 437, "completion_tokens": 1216, "reasoning_tokens": 1155, "total_tokens": 1653}}, "with-docs": {"answer": "Explanation: I checked the schedule and counted the entries listed at exactly 2 p.m. There are two such games: October 24, 2009 vs. Wayne State University and November 28, 2009 vs. Mercyhurst College.\n\nExact Answer: 2\nConfidence: 98%", "pred_items": ["2"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4450.86, "finish_reason": "stop", "usage": {"prompt_tokens": 958, "completion_tokens": 162, "reasoning_tokens": 91, "total_tokens": 1120}}, "with-structures": {"answer": "Explanation: I checked the listed game times in the supporting documents and counted only those scheduled at exactly 2:00 p.m. The games at 2:00 p.m. are against Wayne State University on 2009-10-24 and Mercyhurst College on 2009-11-28, for a total of 2 games.\nExact Answer: 2\nConfidence: 98%", "pred_items": ["2"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4257.16, "finish_reason": "stop", "usage": {"prompt_tokens": 2205, "completion_tokens": 146, "reasoning_tokens": 55, "total_tokens": 2351}}, "agentic-a": {"answer": "Explanation: Using the entity structure file `structures/structures/entity/wiki_opentable_wikitq_00125__Opponent_centric_lookup.entities.csv`, I counted two game entries with `times_text` equal to `2 p.m.` (Wayne State University on 2009-10-24 and Mercyhurst College on 2009-11-28). The time unit `Chronological_game_schedule_index.timeline.json` confirms these are games from the 2009-10 season schedule.\nExact Answer: 2\nConfidence: 98%", "pred_items": ["2"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 68523.56, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4, "python": 1}, "tokens": {"prompt_tokens": 14059, "completion_tokens": 968, "total_tokens": 15027, "reasoning_tokens": 446}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: In `structures/structures/entity/wiki_opentable_wikitq_00125__Opponent_centric_lookup.entities.csv`, the only schedule rows that include “2 p.m.” are Wayne State University (`7 p.m.; 2 p.m.`) and Mercyhurst College (`7 p.m.; 2 p.m.`), so each contributes one 2:00 p.m. game.\nExact Answer: 2\nConfidence: 88%", "pred_items": ["2"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 561603.17, "stop_reason": "end_turn", "n_turns": 17, "max_turns": 50, "tool_call_counts": {"bash": 37, "read_file": 13}, "tokens": {"prompt_tokens": 159422, "completion_tokens": 6247, "total_tokens": 165669, "reasoning_tokens": 3360}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In the schedule for the 2009–10 Colgate Raiders women’s ice hockey season, only two entries are listed at 2 p.m.—10/24/2009 vs. Wayne State University and 11/28/2009 vs. Mercyhurst College—both in `structures/doc_048163.txt` (original id 53344).\nExact Answer: 2\nConfidence: 97%", "pred_items": ["2"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 2421283.42, "stop_reason": "end_turn", "n_turns": 11, "max_turns": 15, "tool_call_counts": {"bash": 25}, "tokens": {"prompt_tokens": 73267, "completion_tokens": 3562, "total_tokens": 76829, "reasoning_tokens": 1941}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00126.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00126", "question": "Regarding the 2009/10 Colgate Raiders women's hockey season, how many games were scheduled from 10/9/2009 to 2/20/2010?", "dataset_origin": "WikiTQ", "original_table_id": "204-718", "sql": "SELECT COUNT ( * ) FROM table_204_718 ", "gold_answers": ["34"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: The date range from 10/9/2009 through 2/20/2010 covers the Colgate Raiders women's hockey team's regular-season schedule for 2009–10. Counting the scheduled games in that span gives 34 total games.\nExact Answer: 34\nConfidence: 63%", "pred_items": ["34"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 15353.38, "finish_reason": "stop", "usage": {"prompt_tokens": 446, "completion_tokens": 621, "reasoning_tokens": 548, "total_tokens": 1067}}, "with-docs": {"answer": "Explanation: I counted all scheduled games listed in the two regular-season schedule tables between 10/9/2009 and 2/20/2010. The first document lists 18 games, and the second lists 16 games, for a total of 34 scheduled games.\n\nExact Answer: 34\nConfidence: 99%", "pred_items": ["34"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 7861.88, "finish_reason": "stop", "usage": {"prompt_tokens": 1385, "completion_tokens": 332, "reasoning_tokens": 254, "total_tokens": 1717}}, "with-structures": {"answer": "Explanation: The schedule lists games from 2009-10-09 through 2010-02-20, and the result tracker explicitly states there were “34 games” in that span. Therefore, the number of scheduled games in that date range is 34.\nExact Answer: 34\nConfidence: 99%", "pred_items": ["34"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3642.8, "finish_reason": "stop", "usage": {"prompt_tokens": 3977, "completion_tokens": 111, "reasoning_tokens": 36, "total_tokens": 4088}}, "agentic-a": {"answer": "Explanation: The time structure `structures/structures/time/wiki_opentable_wikitq_00126__Unified_season_schedule.timeline.json` lists games from 2009-10-09 through 2010-02-20, and the claim structure `structures/structures/claim/wiki_opentable_wikitq_00126__Result_record_signal_tracker.claims.jsonl` explicitly states “34 games from 2009-10-09 through 2010-02-20.”  \nExact Answer: 34  \nConfidence: 99%", "pred_items": ["34"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 51067.1, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 11951, "completion_tokens": 481, "total_tokens": 12432, "reasoning_tokens": 176}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The schedule entries in `structures/structures/time/wiki_opentable_wikitq_00124__Unified_season_schedule_timeline.timeline.json` and `structures/structures/time/wiki_opentable_wikitq_00126__Unified_season_schedule.timeline.json` list one game per date from 2009-10-09 through 2010-02-20. Counting those entries gives 34 scheduled games.\nExact Answer: 34\nConfidence: 96%", "pred_items": ["34"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 474792.25, "stop_reason": "end_turn", "n_turns": 14, "max_turns": 50, "tool_call_counts": {"bash": 31, "read_file": 7}, "tokens": {"prompt_tokens": 103466, "completion_tokens": 5054, "total_tokens": 108520, "reasoning_tokens": 2339}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: The schedule for the 2009–10 Colgate Raiders women's ice hockey season is split across structures/doc_048163.txt (original id 53344) and structures/doc_048164.txt (original id 53345). Counting all listed game dates from 10/9/2009 through 2/20/2010 gives 34 scheduled games.\nExact Answer: 34\nConfidence: 99%", "pred_items": ["34"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 718078.1, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 4}, "tokens": {"prompt_tokens": 10437, "completion_tokens": 637, "total_tokens": 11074, "reasoning_tokens": 344}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00127.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00127", "question": "When is the earliest time this game was played during the season of the Colgate Raiders 2009–10 women's hockey season?", "dataset_origin": "WikiTQ", "original_table_id": "204-718", "sql": "SELECT MIN ( Time_ ) FROM table_204_718 ", "gold_answers": ["2 p.m."], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I interpret the question as asking for the earliest start time of any game listed on the Colgate Raiders 2009–10 women's hockey season schedule. The earliest listed game time on that season schedule is 2:00 PM.\n\nExact Answer: 2:00 PM\nConfidence: 57%", "pred_items": ["2:00 PM"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 12437.8, "finish_reason": "stop", "usage": {"prompt_tokens": 436, "completion_tokens": 533, "reasoning_tokens": 461, "total_tokens": 969}}, "with-docs": {"answer": "Explanation: I scanned the schedule times in both supporting documents and identified the earliest time of day listed for any Colgate Raiders 2009–10 women’s hockey game. The earliest listed start time is 2 p.m. (earlier than 2:30 p.m., 3 p.m., 3:30 p.m., 4 p.m., 5 p.m., and 7 p.m.).\nExact Answer: 2 p.m.\nConfidence: 96%", "pred_items": ["2 p.m."], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 5292.77, "finish_reason": "stop", "usage": {"prompt_tokens": 1375, "completion_tokens": 223, "reasoning_tokens": 116, "total_tokens": 1598}}, "with-structures": {"answer": "Explanation: The schedule’s earliest listed game is against the University of New Hampshire on 2009-10-09, so the earliest time the game was played that season was October 9, 2009.\n\nExact Answer: 2009-10-09\nConfidence: 98%", "pred_items": ["2009-10-09"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4104.44, "finish_reason": "stop", "usage": {"prompt_tokens": 3575, "completion_tokens": 141, "reasoning_tokens": 72, "total_tokens": 3716}}, "agentic-a": {"answer": "Explanation: The time structure `Unified_season_schedule_timeline.timeline.json` lists the first game chronologically on 2009-10-09, and the claim file `Known_results_and_structure_ledger.claims.jsonl` confirms the earliest listed game was on that date.\nExact Answer: 2009-10-09\nConfidence: 98%", "pred_items": ["2009-10-09"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 54665.27, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 9878, "completion_tokens": 850, "total_tokens": 10728, "reasoning_tokens": 563}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: I used `structures/structures/entity/wiki_opentable_wikitq_00125__Opponent_centric_lookup.entities.csv`, which lists the game `times_text` for the Colgate Raiders 2009–10 women’s hockey schedule, and confirmed the season context with `structures/structures/claim/wiki_opentable_wikitq_00127__Known_results_and_structure_ledger.claims.jsonl`. The earliest listed game time in the extracted schedule is 2 p.m.\nExact Answer: 2 p.m.\nConfidence: 74%", "pred_items": ["2 p.m."], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 563459.17, "stop_reason": "end_turn", "n_turns": 17, "max_turns": 50, "tool_call_counts": {"bash": 36, "read_file": 17}, "tokens": {"prompt_tokens": 149168, "completion_tokens": 7485, "total_tokens": 156653, "reasoning_tokens": 4396}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: The schedule for the 2009–10 Colgate Raiders women's hockey season in doc_048163.txt and doc_048164.txt (original ids 53344 and 53345) shows the earliest listed game time is 2 p.m. in the Time column.\nExact Answer: 2 p.m.\nConfidence: 98%", "pred_items": ["2 p.m."], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1371928.54, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 15, "tool_call_counts": {"bash": 8, "read_file": 2}, "tokens": {"prompt_tokens": 24885, "completion_tokens": 1845, "total_tokens": 26730, "reasoning_tokens": 1371}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00128.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00128", "question": "How many entries are there in total for the Best Driver ESPY Award?", "dataset_origin": "WikiTQ", "original_table_id": "203-364", "sql": "SELECT COUNT ( * ) FROM table_203_364 ", "gold_answers": ["21"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: The Best Driver ESPY Award has been awarded annually from 1994 through 2024. Counting inclusively, that gives 31 total entries.\nExact Answer: 31\nConfidence: 58%", "pred_items": ["31"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 19254.08, "finish_reason": "stop", "usage": {"prompt_tokens": 425, "completion_tokens": 1011, "reasoning_tokens": 958, "total_tokens": 1436}}, "with-docs": {"answer": "Explanation: I counted the award entries listed in both supporting tables. Doc 1 has 12 entries (1993–2004), and Doc 2 has 9 entries (2005–2013). Adding them gives 21 total entries.\nExact Answer: 21\nConfidence: 99%", "pred_items": ["21"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3325.58, "finish_reason": "stop", "usage": {"prompt_tokens": 945, "completion_tokens": 116, "reasoning_tokens": 45, "total_tokens": 1061}}, "with-structures": {"answer": "Explanation: I counted the listed Best Driver ESPY Award winners from 1993 through 2013 in the provided timeline. There is one entry for each year, and the span includes 21 years total. The summary tables also confirm this with NASCAR 14 + CART 3 + Formula One 3 + IndyCar Series 1 = 21 awards.\nExact Answer: 21\nConfidence: 99%", "pred_items": ["21"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3015.61, "finish_reason": "stop", "usage": {"prompt_tokens": 2616, "completion_tokens": 109, "reasoning_tokens": 16, "total_tokens": 2725}}, "agentic-a": {"answer": "Explanation: The timeline file `structures/structures/time/wiki_opentable_wikitq_00128__Best_Driver_ESPY_complete_winner_timeline.timeline.json` lists winners from 1993 through 2013, totaling 21 yearly events; this matches the series summary in `structures/structures/_raw/wiki_opentable_wikitq_00128__By_racing_series.md` (14 + 3 + 3 + 1 = 21 awards).\nExact Answer: 21\nConfidence: 99%", "pred_items": ["21"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 82632.78, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 15, "tool_call_counts": {"bash": 3, "read_file": 6}, "tokens": {"prompt_tokens": 19230, "completion_tokens": 803, "total_tokens": 20033, "reasoning_tokens": 275}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The timeline files `structures/structures/time/wiki_opentable_wikitq_00128__Best_Driver_ESPY_complete_winner_timeline.timeline.json` and `structures/structures/time/wiki_opentable_wikitq_00130__Unified_Best_Driver_ESPY_winners_timeline.timeline.json` each list the Best Driver ESPY Award winners from 1993 through 2013, totaling 21 events.\nExact Answer: 21\nConfidence: 99%", "pred_items": ["21"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 84643.6, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 50, "tool_call_counts": {"bash": 1, "read_file": 2}, "tokens": {"prompt_tokens": 10924, "completion_tokens": 446, "total_tokens": 11370, "reasoning_tokens": 155}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: The Best Driver ESPY Award table is split across `structures/doc_043874.txt` (1993–2004) and `structures/doc_043875.txt` (2005–2013). Counting the year rows across both documents gives 21 total entries.\nExact Answer: 21\nConfidence: 99%", "pred_items": ["21"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 618627.89, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 4}, "tokens": {"prompt_tokens": 13628, "completion_tokens": 657, "total_tokens": 14285, "reasoning_tokens": 370}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00129.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00129", "question": "For Best Driver ESPY Awards, how many consecutive Espy Awards has Jimmy Johnson won?", "dataset_origin": "WikiTQ", "original_table_id": "203-364", "sql": "SELECT COUNT ( * ) FROM table_203_364 WHERE Driver = 'Jimmie Johnson' ", "gold_answers": ["4"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Jimmie Johnson won the Best Driver ESPY in a streak of five straight years. Therefore, his total number of consecutive ESPY Awards in that category is 5.\n\nExact Answer: 5\nConfidence: 93%", "pred_items": ["5"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 8121.51, "finish_reason": "stop", "usage": {"prompt_tokens": 428, "completion_tokens": 385, "reasoning_tokens": 327, "total_tokens": 813}}, "with-docs": {"answer": "Explanation: The table shows Jimmie Johnson won the Best Driver ESPY Award in 2008, 2009, 2010, and 2011. These are four wins in a row, so his consecutive total is 4.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3190.67, "finish_reason": "stop", "usage": {"prompt_tokens": 680, "completion_tokens": 101, "reasoning_tokens": 30, "total_tokens": 781}}, "with-structures": {"answer": "Explanation: The documents list Jimmie Johnson as winning in 2008, 2009, 2010, and 2011, which is a streak of 4 consecutive Best Driver ESPY Awards.\nExact Answer: 4\nConfidence: 98%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3312.55, "finish_reason": "stop", "usage": {"prompt_tokens": 1400, "completion_tokens": 107, "reasoning_tokens": 43, "total_tokens": 1507}}, "agentic-a": {"answer": "Explanation: The entity aggregate file `structures/structures/entity/wiki_opentable_wikitq_00129__Driver_aggregates_repeat_wins_streaks.entities.csv` lists Jimmie Johnson with a `longest_consecutive_streak_int` of 4, and the timeline file confirms wins in 2008, 2009, 2010, and 2011.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 78481.25, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 15, "tool_call_counts": {"bash": 3, "read_file": 5}, "tokens": {"prompt_tokens": 16533, "completion_tokens": 643, "total_tokens": 17176, "reasoning_tokens": 181}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The timelines in `structures/structures/time/wiki_opentable_wikitq_00128__Best_Driver_ESPY_complete_winner_timeline.timeline.json` and `...00130__Unified_Best_Driver_ESPY_winners_timeline.timeline.json` show Jimmie Johnson winning in 2008, 2009, 2010, and 2011 before another winner appears in 2012.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 82510.16, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 50, "tool_call_counts": {"bash": 1, "read_file": 2}, "tokens": {"prompt_tokens": 10959, "completion_tokens": 408, "total_tokens": 11367, "reasoning_tokens": 106}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In doc_043875.txt (original id 49486), the Best Driver ESPY Award winners table lists Jimmie Johnson in 2008, 2009, 2010, and 2011, before switching to Tony Stewart in 2012, so the consecutive run is four.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 476081.0, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 3}, "tokens": {"prompt_tokens": 9817, "completion_tokens": 573, "total_tokens": 10390, "reasoning_tokens": 349}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00130.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00130", "question": "How many times has Jeff Gordon won the Best Driver ESPY Award?", "dataset_origin": "WikiTQ", "original_table_id": "203-364", "sql": "SELECT COUNT ( * ) FROM table_203_364 WHERE Driver = 'Jeff Gordon' ", "gold_answers": ["4"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Jeff Gordon won the Best Driver ESPY Award 4 times. He is one of the award’s multiple-time winners and is commonly listed with four victories.\nExact Answer: 4\nConfidence: 83%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 11168.1, "finish_reason": "stop", "usage": {"prompt_tokens": 424, "completion_tokens": 550, "reasoning_tokens": 496, "total_tokens": 974}}, "with-docs": {"answer": "Explanation: I checked the winner lists in the supporting documents and found Jeff Gordon listed as the Best Driver ESPY Award winner in 1996, 1998, 1999, and 2007. That totals 4 wins.\n\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3119.87, "finish_reason": "stop", "usage": {"prompt_tokens": 944, "completion_tokens": 94, "reasoning_tokens": 25, "total_tokens": 1038}}, "with-structures": {"answer": "Explanation: The supporting documents list Jeff Gordon as the Best Driver ESPY Award winner in 1996, 1998, 1999, and 2007, which totals 4 wins.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3172.89, "finish_reason": "stop", "usage": {"prompt_tokens": 2340, "completion_tokens": 85, "reasoning_tokens": 24, "total_tokens": 2425}}, "agentic-a": {"answer": "Explanation: The driver-centered entity file `structures/structures/entity/wiki_opentable_wikitq_00130__Driver_centered_index.entities.csv` lists Jeff Gordon’s winning years as 1996, 1998, 1999, and 2007; the timeline file confirms those four wins.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 79013.36, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 15, "tool_call_counts": {"bash": 3, "read_file": 7}, "tokens": {"prompt_tokens": 17858, "completion_tokens": 623, "total_tokens": 18481, "reasoning_tokens": 137}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The timelines in `structures/structures/time/wiki_opentable_wikitq_00128__Best_Driver_ESPY_complete_winner_timeline.timeline.json` and `structures/structures/time/wiki_opentable_wikitq_00130__Unified_Best_Driver_ESPY_winners_timeline.timeline.json` list Jeff Gordon as winner in 1996, 1998, 1999, and 2007.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 99766.56, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 50, "tool_call_counts": {"bash": 1, "read_file": 2}, "tokens": {"prompt_tokens": 10931, "completion_tokens": 397, "total_tokens": 11328, "reasoning_tokens": 101}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In the two “Best Driver ESPY Award - Winners and nominees” tables, Jeff Gordon appears as the winner in 1996, 1998, 1999, and 2007 in doc_043874.txt (id: 49485) and doc_043875.txt (id: 49486), for a total of four wins.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1370127.11, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 15, "tool_call_counts": {"bash": 10}, "tokens": {"prompt_tokens": 32325, "completion_tokens": 1318, "total_tokens": 33643, "reasoning_tokens": 577}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00131.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00131", "question": "Among the Best Driver ESPY Awards, which racing series has the most winners?", "dataset_origin": "WikiTQ", "original_table_id": "203-364", "sql": "SELECT Racing_series FROM table_203_364 GROUP BY Racing_series ORDER BY COUNT ( * ) DESC LIMIT 1 ", "gold_answers": ["NASCAR"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Looking across Best Driver ESPY Award winners, the largest share come from stock-car drivers competing in NASCAR’s top level. By total winners/award occurrences, NASCAR’s premier series leads the other racing series represented, such as IndyCar and Formula One.\n\nExact Answer: NASCAR Cup Series\nConfidence: 93%", "pred_items": ["NASCAR Cup Series"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 11106.97, "finish_reason": "stop", "usage": {"prompt_tokens": 426, "completion_tokens": 555, "reasoning_tokens": 481, "total_tokens": 981}}, "with-docs": {"answer": "Explanation: I counted the winners by racing series across the listed Best Driver ESPY Award years. NASCAR appears 14 times, more than Formula One (3), CART (3), and IndyCar Series (1). Therefore, NASCAR has the most winners.\n\nExact Answer: NASCAR\nConfidence: 99%", "pred_items": ["NASCAR"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4899.63, "finish_reason": "stop", "usage": {"prompt_tokens": 946, "completion_tokens": 209, "reasoning_tokens": 138, "total_tokens": 1155}}, "with-structures": {"answer": "Explanation: From the listed Best Driver ESPY winners, NASCAR appears most often. It has the largest number of winning drivers and also the most award years represented among the winners shown.\nExact Answer: NASCAR\nConfidence: 97%", "pred_items": ["NASCAR"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 7904.95, "finish_reason": "stop", "usage": {"prompt_tokens": 1873, "completion_tokens": 350, "reasoning_tokens": 294, "total_tokens": 2223}}, "agentic-a": {"answer": "Explanation: The entity structure `structures/structures/entity/wiki_opentable_wikitq_00131__Driver_aggregate_index.entities.csv` lists each winner’s `series_represented`; NASCAR appears for the most winners (Jeff Gordon, Jimmie Johnson, Tony Stewart, Dale Jarrett, Bobby Labonte, and Dale Earnhardt, Jr.), more than CART, Formula One, or IndyCar Series.\nExact Answer: NASCAR\nConfidence: 98%", "pred_items": ["NASCAR"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 104033.45, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 15, "tool_call_counts": {"bash": 2, "read_file": 5}, "tokens": {"prompt_tokens": 20155, "completion_tokens": 1619, "total_tokens": 21774, "reasoning_tokens": 1219}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The table in `structures/structures/_raw/wiki_opentable_wikitq_00128__By_racing_series.md` lists Best Driver ESPY Awards by racing series and shows NASCAR with 14 awards, more than CART and Formula One (3 each) and IndyCar Series (1). The winner timeline in `structures/structures/time/wiki_opentable_wikitq_00128__Best_Driver_ESPY_complete_winner_timeline.timeline.json` is consistent with that total.\nExact Answer: NASCAR\nConfidence: 99%", "pred_items": ["NASCAR"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 171730.59, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 50, "tool_call_counts": {"bash": 3, "read_file": 3}, "tokens": {"prompt_tokens": 20165, "completion_tokens": 892, "total_tokens": 21057, "reasoning_tokens": 413}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: The winner tables in `structures/doc_043874.txt` (id 49485) and `structures/doc_043875.txt` (id 49486) list Best Driver ESPY Award winners by year and racing series; counting those entries shows NASCAR appears most often (14 times), more than Formula One, CART, or IndyCar Series.\nExact Answer: NASCAR\nConfidence: 99%", "pred_items": ["NASCAR"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 833330.13, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 5, "python": 1}, "tokens": {"prompt_tokens": 14008, "completion_tokens": 1079, "total_tokens": 15087, "reasoning_tokens": 587}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00132.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00132", "question": "Best Driver ESPY Award next to Nascar, from which other racing series do ESPY winning drivers come?", "dataset_origin": "WikiTQ", "original_table_id": "203-364", "sql": "SELECT DISTINCT Racing_series FROM table_203_364 WHERE Racing_series != 'NASCAR' ", "gold_answers": ["Formula One", "CART", "IndyCar Series"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Looking at Best Driver ESPY winners, most are NASCAR drivers. The non-NASCAR winners are from American open-wheel racing and Formula One: early winners like Nigel Mansell, Al Unser Jr., and Jimmy Vasser were in CART, later Dario Franchitti was in IndyCar, and drivers like Sebastian Vettel and Lewis Hamilton were in Formula One. So the other racing series represented are CART, IndyCar, and Formula One.\n\nExact Answer: CART; IndyCar Series; Formula One\nConfidence: 77%", "pred_items": ["CART", "IndyCar Series", "Formula One"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 32471.7, "finish_reason": "stop", "usage": {"prompt_tokens": 432, "completion_tokens": 1477, "reasoning_tokens": 1360, "total_tokens": 1909}}, "with-docs": {"answer": "Explanation: I looked at the racing series listed for Best Driver ESPY Award winners and excluded NASCAR. The remaining series appearing in the tables are CART, Formula One, and IndyCar Series.\nExact Answer: CART; Formula One; IndyCar Series\nConfidence: 99%", "pred_items": ["CART", "Formula One", "IndyCar Series"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3566.81, "finish_reason": "stop", "usage": {"prompt_tokens": 952, "completion_tokens": 130, "reasoning_tokens": 65, "total_tokens": 1082}}, "with-structures": {"answer": "Explanation: The winners are listed as coming from several racing series. Excluding NASCAR, the other series represented are Formula One, CART, and the IndyCar Series.\nExact Answer: Formula One; CART; IndyCar Series\nConfidence: 98%", "pred_items": ["Formula One", "CART", "IndyCar Series"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3788.34, "finish_reason": "stop", "usage": {"prompt_tokens": 2502, "completion_tokens": 129, "reasoning_tokens": 69, "total_tokens": 2631}}, "agentic-a": {"answer": "Explanation: The non-NASCAR series are listed in the racing-series structures, especially `structures/structures/relation/wiki_opentable_wikitq_00132__Racing_series_totals.relations.jsonl`, and are also confirmed by drivers’ `series_represented_in_wins` in `structures/structures/entity/wiki_opentable_wikitq_00132__Driver_reverse_index_and_repeat_winner_summary.entities.csv`.\nExact Answer: Formula One; CART; IndyCar Series\nConfidence: 98%", "pred_items": ["Formula One", "CART", "IndyCar Series"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 104107.45, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 15, "tool_call_counts": {"bash": 3, "read_file": 7}, "tokens": {"prompt_tokens": 19392, "completion_tokens": 933, "total_tokens": 20325, "reasoning_tokens": 403}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The table in `structures/structures/_raw/wiki_opentable_wikitq_00128__By_racing_series.md` lists all Best Driver ESPY Award-winning racing series, and the winner timelines in `structures/structures/time/wiki_opentable_wikitq_00128__Best_Driver_ESPY_complete_winner_timeline.timeline.json` confirm the non-NASCAR winners.\nExact Answer: CART; Formula One; IndyCar Series\nConfidence: 99%", "pred_items": ["CART", "Formula One", "IndyCar Series"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 107603.12, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 50, "tool_call_counts": {"bash": 2, "read_file": 4}, "tokens": {"prompt_tokens": 12116, "completion_tokens": 742, "total_tokens": 12858, "reasoning_tokens": 288}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In the winner tables in `structures/doc_043874.txt` (id 49485) and `structures/doc_043875.txt` (id 49486), the `Racing_series` column shows ESPY Best Driver winners from NASCAR as well as three other series. The non-NASCAR winning series listed are Formula One, CART, and IndyCar Series.\nExact Answer: Formula One; CART; IndyCar Series\nConfidence: 98%", "pred_items": ["Formula One", "CART", "IndyCar Series"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 882095.46, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 4}, "tokens": {"prompt_tokens": 14188, "completion_tokens": 883, "total_tokens": 15071, "reasoning_tokens": 546}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00134.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00134", "question": "Which employer has the most employees in Mundelein, Illinois?", "dataset_origin": "WikiTQ", "original_table_id": "203-737", "sql": "SELECT Employer FROM table_203_737 ORDER BY _of_Employees_ DESC LIMIT 1 ", "gold_answers": ["Medline"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Mundelein’s largest listed employer is generally identified as Medline Industries, which has the highest employee count among major employers in the village.\nExact Answer: Medline Industries\nConfidence: 72%", "pred_items": ["Medline Industries"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 6609.08, "finish_reason": "stop", "usage": {"prompt_tokens": 423, "completion_tokens": 250, "reasoning_tokens": 198, "total_tokens": 673}}, "with-docs": {"answer": "Explanation: The table of top employers in Mundelein, Illinois shows Medline with 1200 employees, which is the highest number listed.\nExact Answer: Medline\nConfidence: 99%", "pred_items": ["Medline"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2032.85, "finish_reason": "stop", "usage": {"prompt_tokens": 601, "completion_tokens": 55, "reasoning_tokens": 5, "total_tokens": 656}}, "with-structures": {"answer": "Explanation: From the listed top employers in Mundelein, Illinois, Medline is ranked 1st with 1,200 employees, which is the highest among all employers shown.\nExact Answer: Medline\nConfidence: 99%", "pred_items": ["Medline"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2605.79, "finish_reason": "stop", "usage": {"prompt_tokens": 1195, "completion_tokens": 71, "reasoning_tokens": 13, "total_tokens": 1266}}, "agentic-a": {"answer": "Explanation: The entity table in `structures/structures/entity/wiki_opentable_wikitq_00134__Top_employers_lookup.entities.csv` ranks employers in Mundelein by employee count, and the headcount summary in `structures/structures/_raw/wiki_opentable_wikitq_00134__Headcount_stats_and_comparison_grid.md` confirms the top employer.  \nExact Answer: Medline\nConfidence: 99%", "pred_items": ["Medline"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 31493.56, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 6945, "completion_tokens": 326, "total_tokens": 7271, "reasoning_tokens": 64}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The files `structures/structures/_raw/wiki_opentable_wikitq_00138__Top_employers_master_table.md` and `structures/structures/entity/wiki_opentable_wikitq_00142__Top_employer_lookup_and_ranking.entities.csv` list Mundelein employers by rank and employee count; both show Medline ranked #1 with 1,200 employees.\nExact Answer: Medline\nConfidence: 99%", "pred_items": ["Medline"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 120288.26, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 5, "read_file": 3}, "tokens": {"prompt_tokens": 14412, "completion_tokens": 741, "total_tokens": 15153, "reasoning_tokens": 163}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In structures/doc_045293.txt (original id 50761), the “Mundelein, Illinois - Economy” table lists top employers by number of employees; Medline is ranked #1 with 1200 employees, the highest in the table.\nExact Answer: Medline\nConfidence: 99%", "pred_items": ["Medline"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 937728.75, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 8}, "tokens": {"prompt_tokens": 14555, "completion_tokens": 606, "total_tokens": 15161, "reasoning_tokens": 197}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00135.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00135", "question": "What is the difference between the number of employees in Mundelein Elementary School District 75 and Fremon School District 79 in Mundelein, Illinois?", "dataset_origin": "WikiTQ", "original_table_id": "203-737", "sql": "SELECT ABS ( ( SELECT _of_Employees_ FROM table_203_737 WHERE Employer = 'Mundelein Elementary School District 75' ) - ( SELECT _of_Employees_ FROM table_203_737 WHERE Employer = 'Fremont School District 79' ) ) ", "gold_answers": ["74"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I compared the reported employee counts for the two school districts and subtracted the smaller from the larger to get the difference. Based on the figures I found, the difference is 16 employees.\n\nExact Answer: 16\nConfidence: 22%", "pred_items": ["16"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 17069.38, "finish_reason": "stop", "usage": {"prompt_tokens": 442, "completion_tokens": 812, "reasoning_tokens": 751, "total_tokens": 1254}}, "with-docs": {"answer": "Explanation: The table shows Fremon(t) School District 79 with 287 employees and Mundelein Elementary School District 75 with 213 employees. Subtracting 213 from 287 gives 74.\nExact Answer: 74\nConfidence: 99%", "pred_items": ["74"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2858.45, "finish_reason": "stop", "usage": {"prompt_tokens": 620, "completion_tokens": 94, "reasoning_tokens": 30, "total_tokens": 714}}, "with-structures": {"answer": "Explanation: From the table, Fremont School District 79 has 287 employees and Mundelein Elementary School District 75 has 213 employees. The difference is 287 − 213 = 74.\nExact Answer: 74\nConfidence: 99%", "pred_items": ["74"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3424.53, "finish_reason": "stop", "usage": {"prompt_tokens": 1357, "completion_tokens": 93, "reasoning_tokens": 31, "total_tokens": 1450}}, "agentic-a": {"answer": "Explanation: The employee counts come from the entity table `structures/structures/entity/wiki_opentable_wikitq_00135__Employer_ranking_and_direct_lookup.entities.csv`, which lists Fremont School District 79 with 287 employees and Mundelein Elementary School District 75 with 213 employees. The difference is 74.\nExact Answer: 74\nConfidence: 99%", "pred_items": ["74"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 28114.26, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 2}, "tokens": {"prompt_tokens": 8295, "completion_tokens": 312, "total_tokens": 8607, "reasoning_tokens": 127}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The employee counts are listed in `structures/structures/_raw/wiki_opentable_wikitq_00138__Top_employers_master_table.md` and corroborated by `structures/structures/entity/wiki_opentable_wikitq_00137__Employer_ranking_and_counts.entities.csv`: Fremont School District 79 has 287 employees and Mundelein Elementary School District 75 has 213.\nExact Answer: 74\nConfidence: 99%", "pred_items": ["74"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 116964.61, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 4, "read_file": 3}, "tokens": {"prompt_tokens": 15135, "completion_tokens": 727, "total_tokens": 15862, "reasoning_tokens": 194}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: Based on doc_045293.txt (original id 50761), the Mundelein, Illinois employers table lists Fremont School District 79 with 287 employees and Mundelein Elementary School District 75 with 213 employees, so the difference is 74.\nExact Answer: 74\nConfidence: 99%", "pred_items": ["74"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 613863.75, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 4}, "tokens": {"prompt_tokens": 10220, "completion_tokens": 588, "total_tokens": 10808, "reasoning_tokens": 272}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00136.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00136", "question": "How many employees does mundelein High School in Mundelein, Illinois have?", "dataset_origin": "WikiTQ", "original_table_id": "203-737", "sql": "SELECT _of_Employees_ FROM table_203_737 WHERE Employer = 'Mundelein High School' ", "gold_answers": ["211"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: The phrasing matches a company-profile style employee-size listing for Mundelein High School in Mundelein, Illinois, which is typically given as an employee range rather than an exact headcount. The listed size is 201–500 employees.\n\nExact Answer: 201-500\nConfidence: 58%", "pred_items": ["201-500"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 9613.01, "finish_reason": "stop", "usage": {"prompt_tokens": 427, "completion_tokens": 565, "reasoning_tokens": 491, "total_tokens": 992}}, "with-docs": {"answer": "Explanation: The table in the provided document lists “Mundelein High School” with 211 employees.\nExact Answer: 211\nConfidence: 99%", "pred_items": ["211"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 1895.07, "finish_reason": "stop", "usage": {"prompt_tokens": 605, "completion_tokens": 49, "reasoning_tokens": 6, "total_tokens": 654}}, "with-structures": {"answer": "Explanation: In the employer ranking table, Mundelein High School is listed with 211 employees.\nExact Answer: 211\nConfidence: 99%", "pred_items": ["211"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2390.49, "finish_reason": "stop", "usage": {"prompt_tokens": 1267, "completion_tokens": 56, "reasoning_tokens": 15, "total_tokens": 1323}}, "agentic-a": {"answer": "Explanation: The entity table `structures/structures/entity/wiki_opentable_wikitq_00136__Top_employers_ranked.entities.csv` lists “Mundelein High School” with `employees_int` = 211; the classification table also identifies it as the school in question.\nExact Answer: 211\nConfidence: 99%", "pred_items": ["211"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 34903.22, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 8942, "completion_tokens": 323, "total_tokens": 9265, "reasoning_tokens": 53}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The employee count is stated in `structures/structures/entity/wiki_opentable_wikitq_00137__Employer_ranking_and_counts.entities.csv` and corroborated by `structures/structures/_raw/wiki_opentable_wikitq_00137__Numeric_lookup_index_for_count_rank_questions.md`, both listing Mundelein High School with 211 employees.\nExact Answer: 211\nConfidence: 98%", "pred_items": ["211"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 113613.04, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 5, "read_file": 5}, "tokens": {"prompt_tokens": 15261, "completion_tokens": 792, "total_tokens": 16053, "reasoning_tokens": 160}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_045293.txt` (original id `50761`), the table “Mundelein, Illinois - Economy” lists top employers and shows “Mundelein High School” with `211` employees.\nExact Answer: 211\nConfidence: 99%", "pred_items": ["211"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 936895.84, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 6}, "tokens": {"prompt_tokens": 20109, "completion_tokens": 677, "total_tokens": 20786, "reasoning_tokens": 232}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00137.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00137", "question": "Which employer in Mundelein, Illinois has the most employees?", "dataset_origin": "WikiTQ", "original_table_id": "203-737", "sql": "SELECT Employer FROM table_203_737 ORDER BY _of_Employees_ DESC LIMIT 1 ", "gold_answers": ["Medline"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I identified the likely largest employer in Mundelein by recalling the village’s major-employer listings and the large Medline Industries facility located there. Among employers associated with Mundelein, Medline Industries is generally cited as having the highest employee count.\nExact Answer: Medline Industries\nConfidence: 47%", "pred_items": ["Medline Industries"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 11512.35, "finish_reason": "stop", "usage": {"prompt_tokens": 423, "completion_tokens": 480, "reasoning_tokens": 406, "total_tokens": 903}}, "with-docs": {"answer": "Explanation: The table of top employers in Mundelein, Illinois shows Medline with 1200 employees, which is the highest number listed.\nExact Answer: Medline\nConfidence: 99%", "pred_items": ["Medline"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2247.98, "finish_reason": "stop", "usage": {"prompt_tokens": 601, "completion_tokens": 56, "reasoning_tokens": 6, "total_tokens": 657}}, "with-structures": {"answer": "Explanation: From the provided employer ranking table, Medline is ranked 1st with 1,200 employees, the highest count listed for employers in Mundelein, Illinois.\nExact Answer: Medline\nConfidence: 99%", "pred_items": ["Medline"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3122.39, "finish_reason": "stop", "usage": {"prompt_tokens": 1360, "completion_tokens": 62, "reasoning_tokens": 5, "total_tokens": 1422}}, "agentic-a": {"answer": "Explanation: The entity ranking file `structures/structures/entity/wiki_opentable_wikitq_00137__Employer_ranking_and_counts.entities.csv` shows employers in Mundelein ranked by employee count, and the numeric lookup markdown confirms rank 1. Both identify Medline as the top employer with 1,200 employees.\nExact Answer: Medline\nConfidence: 99%", "pred_items": ["Medline"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 23770.38, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8318, "completion_tokens": 332, "total_tokens": 8650, "reasoning_tokens": 69}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The files `structures/structures/_raw/wiki_opentable_wikitq_00138__Top_employers_master_table.md` and `structures/structures/entity/wiki_opentable_wikitq_00137__Employer_ranking_and_counts.entities.csv` both rank Mundelein employers and show Medline at rank 1 with 1,200 employees.\nExact Answer: Medline\nConfidence: 99%", "pred_items": ["Medline"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 90918.11, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 50, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 11463, "completion_tokens": 424, "total_tokens": 11887, "reasoning_tokens": 89}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_045293.txt` (original id `50761`), the “Mundelein, Illinois - Economy” table of top employers lists Medline with 1200 employees, the highest count in the table.\nExact Answer: Medline\nConfidence: 99%", "pred_items": ["Medline"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 760732.68, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 7}, "tokens": {"prompt_tokens": 14645, "completion_tokens": 691, "total_tokens": 15336, "reasoning_tokens": 328}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00138.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00138", "question": "How many employees do the first three employers in Mundelein, Illinois have combined?", "dataset_origin": "WikiTQ", "original_table_id": "203-737", "sql": "SELECT SUM ( _of_Employees_ ) FROM table_203_737 WHERE _ <= '3' ", "gold_answers": ["1972"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: The commonly listed first three major employers in Mundelein, Illinois are Medline Industries, Sysmex America, and Stepan Company, with employee counts that sum to 2,350. Adding them together gives the combined total.\nExact Answer: 2350\nConfidence: 33%", "pred_items": ["2350"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 36990.79, "finish_reason": "stop", "usage": {"prompt_tokens": 427, "completion_tokens": 1636, "reasoning_tokens": 1566, "total_tokens": 2063}}, "with-docs": {"answer": "Explanation: The first three employers listed are Medline with 1200 employees, Mundelein Park District with 422 employees, and Amcor with 350 employees. Adding them gives 1200 + 422 + 350 = 1972.\nExact Answer: 1972\nConfidence: 99%", "pred_items": ["1972"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2741.03, "finish_reason": "stop", "usage": {"prompt_tokens": 605, "completion_tokens": 86, "reasoning_tokens": 13, "total_tokens": 691}}, "with-structures": {"answer": "Explanation: From the table, the first three employers are Medline (1200), Mundelein Park District (422), and Amcor (350). Adding them gives 1200 + 422 + 350 = 1972.\nExact Answer: 1972\nConfidence: 99%", "pred_items": ["1972"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2856.65, "finish_reason": "stop", "usage": {"prompt_tokens": 1570, "completion_tokens": 77, "reasoning_tokens": 7, "total_tokens": 1647}}, "agentic-a": {"answer": "Explanation: The numeric answer is stated directly in `structures/structures/_raw/wiki_opentable_wikitq_00138__Numeric_facts_and_derived_comparisons.md` and is consistent with the first three rows in `...__Top_employers_master_table.md` (Medline 1200, Mundelein Park District 422, Amcor 350).\nExact Answer: 1972\nConfidence: 99%", "pred_items": ["1972"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 35410.35, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8365, "completion_tokens": 373, "total_tokens": 8738, "reasoning_tokens": 119}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The ranking and employee counts are given in `structures/structures/entity/wiki_opentable_wikitq_00136__Top_employers_ranked.entities.csv` and confirmed by `structures/structures/_raw/wiki_opentable_wikitq_00138__Top_employers_master_table.md`, where the first three employers are Medline (1200), Mundelein Park District (422), and Amcor (350).\nExact Answer: 1972\nConfidence: 99%", "pred_items": ["1972"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 115139.26, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 3, "read_file": 3}, "tokens": {"prompt_tokens": 18980, "completion_tokens": 825, "total_tokens": 19805, "reasoning_tokens": 254}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_045293.txt` (original id `50761`), the “Mundelein, Illinois - Economy” table lists the top three employers as Medline (1200), Mundelein Park District (422), and Amcor (350). Summing these gives 1972.\nExact Answer: 1972\nConfidence: 99%", "pred_items": ["1972"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 526675.27, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 5}, "tokens": {"prompt_tokens": 9937, "completion_tokens": 580, "total_tokens": 10517, "reasoning_tokens": 242}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00139.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00139", "question": "How many employers have 300 or more employees in Mundelein, Illinois?", "dataset_origin": "WikiTQ", "original_table_id": "203-737", "sql": "SELECT COUNT ( Employer ) FROM table_203_737 WHERE _of_Employees_ >= '300' ", "gold_answers": ["3"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I counted the employers in Mundelein’s major-employers listing that have at least 300 employees. There are 7 such employers.\nExact Answer: 7\nConfidence: 58%", "pred_items": ["7"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 10277.93, "finish_reason": "stop", "usage": {"prompt_tokens": 426, "completion_tokens": 404, "reasoning_tokens": 352, "total_tokens": 830}}, "with-docs": {"answer": "Explanation: From the table of top employers in Mundelein, Illinois, the employers with 300 or more employees are Medline (1200), Mundelein Park District (422), and Amcor (350). That makes 3 employers total.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3680.24, "finish_reason": "stop", "usage": {"prompt_tokens": 604, "completion_tokens": 109, "reasoning_tokens": 37, "total_tokens": 713}}, "with-structures": {"answer": "Explanation: From the listed employers in Mundelein, Illinois, those with 300 or more employees are Medline (1200), Mundelein Park District (422), and Amcor (350). That makes 3 employers total.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3589.84, "finish_reason": "stop", "usage": {"prompt_tokens": 1341, "completion_tokens": 97, "reasoning_tokens": 28, "total_tokens": 1438}}, "agentic-a": {"answer": "Explanation: The count comes from the entity table `Top_employers_ranked_index.entities.csv`, confirmed by the numeric summary in `Employment_distribution_and_comparison_sheet.md`, which lists one employer at 1000+ and two in the 300–499 range.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 25250.5, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8719, "completion_tokens": 356, "total_tokens": 9075, "reasoning_tokens": 123}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The answer is given directly in `structures/structures/_raw/wiki_opentable_wikitq_00135__Quantitative_summary_and_filter_thresholds.md`, which lists employers in Mundelein with “at least 300” employees, and it is consistent with the counts in `structures/structures/entity/wiki_opentable_wikitq_00137__Employer_ranking_and_counts.entities.csv`.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 111674.62, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 4, "read_file": 4}, "tokens": {"prompt_tokens": 16951, "completion_tokens": 715, "total_tokens": 17666, "reasoning_tokens": 162}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_045293.txt` (original id 50761), the “Mundelein, Illinois - Economy” table lists employer sizes. The employers with 300 or more employees are Medline (1200), Mundelein Park District (422), and Amcor (350), so there are three.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 526959.27, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 5}, "tokens": {"prompt_tokens": 10117, "completion_tokens": 493, "total_tokens": 10610, "reasoning_tokens": 232}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00140.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00140", "question": "How many employers in Mundelein, Illinois have fewer than 300 employees?", "dataset_origin": "WikiTQ", "original_table_id": "203-737", "sql": "SELECT COUNT ( Employer ) FROM table_203_737 WHERE _of_Employees_ < '300' ", "gold_answers": ["5"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I counted the employers listed for Mundelein, Illinois with employee totals below 300. There are 5 such employers.\nExact Answer: 5\nConfidence: 31%", "pred_items": ["5"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 17136.6, "finish_reason": "stop", "usage": {"prompt_tokens": 426, "completion_tokens": 887, "reasoning_tokens": 839, "total_tokens": 1313}}, "with-docs": {"answer": "Explanation: From the table, the employers with fewer than 300 employees are Fremont School District 79 (287), University of Saint Mary of the Lake (220), Mundelein Elementary School District 75 (213), Mundelein High School (211), and Village of Mundelein (183). That makes 5 employers.\nExact Answer: 5\nConfidence: 99%", "pred_items": ["5"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3155.93, "finish_reason": "stop", "usage": {"prompt_tokens": 604, "completion_tokens": 117, "reasoning_tokens": 29, "total_tokens": 721}}, "with-structures": {"answer": "Explanation: From the listed employers in Mundelein, Illinois, those with fewer than 300 employees are Fremont School District 79 (287), University of Saint Mary of the Lake (220), Mundelein Elementary School District 75 (213), Mundelein High School (211), and Village of Mundelein (183). That makes 5 employers.\nExact Answer: 5\nConfidence: 99%", "pred_items": ["5"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4140.04, "finish_reason": "stop", "usage": {"prompt_tokens": 1351, "completion_tokens": 142, "reasoning_tokens": 48, "total_tokens": 1493}}, "agentic-a": {"answer": "Explanation: Using the employee counts in `structures/structures/entity/wiki_opentable_wikitq_00140__Employer_ranking_and_direct_lookup.entities.csv`, 5 employers have counts below 300; this is also consistent with the threshold summary in `structures/structures/_raw/wiki_opentable_wikitq_00140__Employment_statistics_and_comparison_matrix.md` showing 3 employers with 300+ out of 8 total.\nExact Answer: 5\nConfidence: 99%", "pred_items": ["5"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 28313.6, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8652, "completion_tokens": 438, "total_tokens": 9090, "reasoning_tokens": 171}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The count comes directly from `structures/structures/_raw/wiki_opentable_wikitq_00135__Quantitative_summary_and_filter_thresholds.md`, cross-checked against `structures/structures/_raw/wiki_opentable_wikitq_00138__Top_employers_master_table.md`. These list the eight Mundelein employers and show that five have employee counts under 300.\nExact Answer: 5\nConfidence: 99%", "pred_items": ["5"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 117938.0, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 3, "read_file": 3}, "tokens": {"prompt_tokens": 17949, "completion_tokens": 694, "total_tokens": 18643, "reasoning_tokens": 238}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_045293.txt` (original id 50761, “Mundelein, Illinois - Economy”), the listed employer employee counts are 1200, 422, 350, 287, 220, 213, 211, and 183; five of these are fewer than 300.\nExact Answer: 5\nConfidence: 99%", "pred_items": ["5"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 550782.45, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 3}, "tokens": {"prompt_tokens": 10909, "completion_tokens": 486, "total_tokens": 11395, "reasoning_tokens": 192}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00141.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00141", "question": "Which employers in Mundelein, Illinois have 300 or more employees?", "dataset_origin": "WikiTQ", "original_table_id": "203-737", "sql": "SELECT Employer FROM table_203_737 WHERE _of_Employees_ >= '300' ", "gold_answers": ["Medline", "Mundelein Park District", "Amcor"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I identified the major employers commonly listed for Mundelein, Illinois and filtered to those reported at roughly 300 or more employees. The employers that meet that threshold are the large industrial, distribution, and school-district employers in the village.\nExact Answer: Medline Industries, Inc.; Sysco Food Services of Chicago, Inc.; Stepan Company; Mundelein High School District 120; Mundelein Elementary School District 75; Fremont School District 79; ZF Services, LLC?\nConfidence: 54%", "pred_items": ["Medline Industries, Inc.", "Sysco Food Services of Chicago, Inc.", "Stepan Company", "Mundelein High School District 120", "Mundelein Elementary School District 75", "Fremont School District 79", "ZF Services, LLC"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 14293.46, "finish_reason": "stop", "usage": {"prompt_tokens": 425, "completion_tokens": 737, "reasoning_tokens": 620, "total_tokens": 1162}}, "with-docs": {"answer": "Explanation: From the table of top employers in Mundelein, Illinois, the employers with 300 or more employees are Medline (1200), Mundelein Park District (422), and Amcor (350). All others listed have fewer than 300 employees.\nExact Answer: Medline; Mundelein Park District; Amcor\nConfidence: 99%", "pred_items": ["Medline", "Mundelein Park District", "Amcor"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3255.06, "finish_reason": "stop", "usage": {"prompt_tokens": 603, "completion_tokens": 112, "reasoning_tokens": 28, "total_tokens": 715}}, "with-structures": {"answer": "Explanation: From the employer table for Mundelein, Illinois, the employers with 300 or more employees are those listed with employee counts of 300+ : Medline (1200), Mundelein Park District (422), and Amcor (350).\nExact Answer: Medline; Mundelein Park District; Amcor\nConfidence: 98%", "pred_items": ["Medline", "Mundelein Park District", "Amcor"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3415.83, "finish_reason": "stop", "usage": {"prompt_tokens": 1293, "completion_tokens": 111, "reasoning_tokens": 29, "total_tokens": 1404}}, "agentic-a": {"answer": "Explanation: From the raw markdown structures `wiki_opentable_wikitq_00141__Employer_rank_to_count_index.md` and `wiki_opentable_wikitq_00141__Employment_statistics_and_comparisons.md`, the employers with at least 300 employees are the three listed at 1200, 422, and 350 employees.\nExact Answer: Medline; Mundelein Park District; Amcor\nConfidence: 99%", "pred_items": ["Medline", "Mundelein Park District", "Amcor"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 28442.74, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 8949, "completion_tokens": 392, "total_tokens": 9341, "reasoning_tokens": 99}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The answer is taken from `structures/structures/_raw/wiki_opentable_wikitq_00138__Top_employers_master_table.md` and confirmed by `structures/structures/entity/wiki_opentable_wikitq_00137__Employer_ranking_and_counts.entities.csv`, which list Mundelein employers and employee counts. Filtering for 300 or more employees leaves three employers.\nExact Answer: Medline; Mundelein Park District; Amcor\nConfidence: 98%", "pred_items": ["Medline", "Mundelein Park District", "Amcor"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 90902.91, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 4, "read_file": 3}, "tokens": {"prompt_tokens": 16386, "completion_tokens": 697, "total_tokens": 17083, "reasoning_tokens": 188}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_045293.txt` (original id `50761`), the “Mundelein, Illinois - Economy” table lists top employers and employee counts; the employers with at least 300 employees are Medline (1200), Mundelein Park District (422), and Amcor (350).\nExact Answer: Medline; Mundelein Park District; Amcor\nConfidence: 99%", "pred_items": ["Medline", "Mundelein Park District", "Amcor"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 620835.51, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 6}, "tokens": {"prompt_tokens": 10516, "completion_tokens": 596, "total_tokens": 11112, "reasoning_tokens": 272}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00142.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00142", "question": "How many employers listed in Mundelein, Illinois have fewer than 250 employees?", "dataset_origin": "WikiTQ", "original_table_id": "203-737", "sql": "SELECT COUNT ( Employer ) FROM table_203_737 WHERE _of_Employees_ < '250' ", "gold_answers": ["4"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I counted the employers in the Mundelein, Illinois list whose employee totals are strictly less than 250. That gives a total of 6 employers.\nExact Answer: 6\nConfidence: 33%", "pred_items": ["6"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 16982.46, "finish_reason": "stop", "usage": {"prompt_tokens": 427, "completion_tokens": 966, "reasoning_tokens": 911, "total_tokens": 1393}}, "with-docs": {"answer": "Explanation: From the table, the employers with fewer than 250 employees are University of Saint Mary of the Lake (220), Mundelein Elementary School District 75 (213), Mundelein High School (211), and Village of Mundelein (183). That makes 4 employers.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3645.32, "finish_reason": "stop", "usage": {"prompt_tokens": 605, "completion_tokens": 116, "reasoning_tokens": 36, "total_tokens": 721}}, "with-structures": {"answer": "Explanation: From the listed employers and employee counts, those with fewer than 250 employees are: University of Saint Mary of the Lake (220), Mundelein Elementary School District 75 (213), Mundelein High School (211), and Village of Mundelein (183). That makes 4 employers.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3883.46, "finish_reason": "stop", "usage": {"prompt_tokens": 1162, "completion_tokens": 113, "reasoning_tokens": 29, "total_tokens": 1275}}, "agentic-a": {"answer": "Explanation: The count comes from the entity table `structures/structures/entity/wiki_opentable_wikitq_00142__Top_employer_lookup_and_ranking.entities.csv`, which lists 8 Mundelein employers and their employee totals. Four of them have fewer than 250 employees: 220, 213, 211, and 183.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 24833.32, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8569, "completion_tokens": 392, "total_tokens": 8961, "reasoning_tokens": 140}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The count comes directly from `structures/structures/_raw/wiki_opentable_wikitq_00135__Quantitative_summary_and_filter_thresholds.md`, which lists the employers “Below 250,” and is consistent with the employee counts in `structures/structures/entity/wiki_opentable_wikitq_00137__Employer_ranking_and_counts.entities.csv`.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 114225.26, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 4, "read_file": 4}, "tokens": {"prompt_tokens": 16955, "completion_tokens": 773, "total_tokens": 17728, "reasoning_tokens": 228}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_045293.txt` (original id `50761`), the Mundelein, Illinois “Top employers” table lists employee counts of 1200, 422, 350, 287, 220, 213, 211, and 183. The counts below 250 are 220, 213, 211, and 183, so there are 4 such employers.\nExact Answer: 4\nConfidence: 99%", "pred_items": ["4"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 620971.15, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 3}, "tokens": {"prompt_tokens": 10392, "completion_tokens": 683, "total_tokens": 11075, "reasoning_tokens": 400}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00150.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00150", "question": "Which tournament under Florent Pietrus had the most points per game?", "dataset_origin": "WikiTQ", "original_table_id": "203-527", "sql": "SELECT Tournament FROM table_203_527 ORDER BY Points_per_game DESC LIMIT 1 ", "gold_answers": ["2006 FIBA World Championship"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Florent Piétrus’s highest points-per-game average came in the 2010 FIBA World Championship, where he posted his best scoring rate among his listed tournaments.\n\nExact Answer: 2010 FIBA World Championship\n\nConfidence: 58%", "pred_items": ["2010 FIBA World Championship"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 18769.36, "finish_reason": "stop", "usage": {"prompt_tokens": 424, "completion_tokens": 989, "reasoning_tokens": 927, "total_tokens": 1413}}, "with-docs": {"answer": "Explanation: I compared the points-per-game values listed for each tournament in Florent Piétrus’s national team stats. The highest value is 9.7 points per game, which occurred at the 2006 FIBA World Championship.\nExact Answer: 2006 FIBA World Championship\nConfidence: 99%", "pred_items": ["2006 FIBA World Championship"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3196.35, "finish_reason": "stop", "usage": {"prompt_tokens": 745, "completion_tokens": 96, "reasoning_tokens": 22, "total_tokens": 841}}, "with-structures": {"answer": "Explanation: I checked the ranked comparison of tournaments by points per game. The highest listed scoring rate is 9.7 PPG, which occurred at the 2006 FIBA World Championship.\nExact Answer: 2006 FIBA World Championship\nConfidence: 98%", "pred_items": ["2006 FIBA World Championship"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2649.71, "finish_reason": "stop", "usage": {"prompt_tokens": 1990, "completion_tokens": 85, "reasoning_tokens": 20, "total_tokens": 2075}}, "agentic-a": {"answer": "Explanation: The ranking in `structures/structures/_raw/wiki_opentable_wikitq_00150__Extremes_and_ranked_comparisons.md` lists tournaments by points per game and shows the top entry as 2006 FIBA World Championship at 9.7 PPG; the timeline file confirms the tournament naming.  \nExact Answer: 2006 FIBA World Championship\nConfidence: 99%", "pred_items": ["2006 FIBA World Championship"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 101313.93, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 15, "tool_call_counts": {"bash": 4, "read_file": 7}, "tokens": {"prompt_tokens": 20828, "completion_tokens": 842, "total_tokens": 21670, "reasoning_tokens": 281}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The files `structures/structures/entity/wiki_opentable_wikitq_00153__Tournament_lookup_leaders.entities.csv` and `structures/structures/_raw/wiki_opentable_wikitq_00154__Extremes_rankings_and_comparison_hooks.md` list Florent Pietrus’s tournament PPG values and identify the top one.\nExact Answer: 2006 FIBA World Championship\nConfidence: 99%", "pred_items": ["2006 FIBA World Championship"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 309809.0, "stop_reason": "end_turn", "n_turns": 9, "max_turns": 50, "tool_call_counts": {"bash": 17, "read_file": 4}, "tokens": {"prompt_tokens": 36802, "completion_tokens": 2399, "total_tokens": 39201, "reasoning_tokens": 1024}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_044417.txt` (original id `49974`, “Florent Piétrus - National team career”), the `Points_per_game` column is highest at `9.7` for `2006 FIBA World Championship`.\nExact Answer: 2006 FIBA World Championship\nConfidence: 99%", "pred_items": ["2006 FIBA World Championship"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1127211.75, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 10, "read_file": 1}, "tokens": {"prompt_tokens": 19172, "completion_tokens": 901, "total_tokens": 20073, "reasoning_tokens": 423}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00151.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00151", "question": "In which tournament did Florent Pietrus score the fewest points per game on average?", "dataset_origin": "WikiTQ", "original_table_id": "203-527", "sql": "SELECT Tournament FROM table_203_527 ORDER BY Points_per_game ASC LIMIT 1 ", "gold_answers": ["2013 EuroBasket"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Florent Piétrus was primarily known for defense and rebounding, and among his major senior national-team tournaments, his lowest scoring average came at the 2012 Olympic tournament, where he averaged the fewest points per game.\nExact Answer: 2012 Summer Olympics\nConfidence: 44%", "pred_items": ["2012 Summer Olympics"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 15336.99, "finish_reason": "stop", "usage": {"prompt_tokens": 428, "completion_tokens": 681, "reasoning_tokens": 609, "total_tokens": 1109}}, "with-docs": {"answer": "Explanation: I compared the points-per-game averages listed for each tournament and identified the lowest value. The smallest average is 1.9 points per game, which occurred at the 2013 EuroBasket.\nExact Answer: 2013 EuroBasket\nConfidence: 99%", "pred_items": ["2013 EuroBasket"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3148.79, "finish_reason": "stop", "usage": {"prompt_tokens": 749, "completion_tokens": 85, "reasoning_tokens": 20, "total_tokens": 834}}, "with-structures": {"answer": "Explanation: I looked at the points-per-game values across the listed tournaments and identified the minimum. The lowest average is 1.9 points per game, which occurred in the 2013 EuroBasket.\nExact Answer: 2013 EuroBasket\nConfidence: 99%", "pred_items": ["2013 EuroBasket"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3084.69, "finish_reason": "stop", "usage": {"prompt_tokens": 1848, "completion_tokens": 83, "reasoning_tokens": 18, "total_tokens": 1931}}, "agentic-a": {"answer": "Explanation: The entity table in `structures/structures/entity/wiki_opentable_wikitq_00151__Tournament_stat_lookup.entities.csv` lists Florent Pietrus’s points per game by tournament, and the raw leaderboard file `structures/structures/_raw/wiki_opentable_wikitq_00151__Leaderboards_extrema_and_derived_totals.md` confirms the lowest PPG was 1.9 in 2013 EuroBasket.\nExact Answer: 2013 EuroBasket\nConfidence: 99%", "pred_items": ["2013 EuroBasket"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 18013.14, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 9175, "completion_tokens": 375, "total_tokens": 9550, "reasoning_tokens": 111}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The answer is stated directly in `structures/structures/_raw/wiki_opentable_wikitq_00150__Extremes_and_ranked_comparisons.md` and corroborated by `structures/structures/_raw/wiki_opentable_wikitq_00154__Extremes_rankings_and_comparison_hooks.md`, both listing Florent Pietrus’s lowest points-per-game tournament.\nExact Answer: 2013 EuroBasket\nConfidence: 99%", "pred_items": ["2013 EuroBasket"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 333749.27, "stop_reason": "end_turn", "n_turns": 10, "max_turns": 50, "tool_call_counts": {"bash": 20, "read_file": 5}, "tokens": {"prompt_tokens": 40557, "completion_tokens": 2294, "total_tokens": 42851, "reasoning_tokens": 823}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_044417.txt` (original id `49974`), the table “Florent Piétrus - National team career” lists his points per game by tournament; the lowest value shown is 1.9 in `2013 EuroBasket`.\nExact Answer: 2013 EuroBasket\nConfidence: 99%", "pred_items": ["2013 EuroBasket"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 752524.88, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 4, "read_file": 1}, "tokens": {"prompt_tokens": 14379, "completion_tokens": 529, "total_tokens": 14908, "reasoning_tokens": 235}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00152.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00152", "question": "In which tournament did Florent Pietrus' team score the most points per game?", "dataset_origin": "WikiTQ", "original_table_id": "203-527", "sql": "SELECT Tournament FROM table_203_527 ORDER BY Points_per_game DESC LIMIT 1 ", "gold_answers": ["2006 FIBA World Championship"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Florent Pietrus played for France in multiple major FIBA tournaments. Comparing France’s scoring averages across those events, the highest team points-per-game total came at EuroBasket 2015, when France had its most productive offensive tournament of Pietrus’ international career.\n\nExact Answer: EuroBasket 2015\nConfidence: 58%", "pred_items": ["EuroBasket 2015"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 12977.88, "finish_reason": "stop", "usage": {"prompt_tokens": 427, "completion_tokens": 567, "reasoning_tokens": 488, "total_tokens": 994}}, "with-docs": {"answer": "Explanation: I compared Florent Piétrus' listed points per game across all tournaments. The highest value is 9.7 points per game, which occurred at the 2006 FIBA World Championship.\nExact Answer: 2006 FIBA World Championship\nConfidence: 99%", "pred_items": ["2006 FIBA World Championship"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2874.4, "finish_reason": "stop", "usage": {"prompt_tokens": 748, "completion_tokens": 90, "reasoning_tokens": 22, "total_tokens": 838}}, "with-structures": {"answer": "Explanation: I looked at the points-per-game values across the listed tournaments and identified the highest one. The maximum PPG is 9.7, which occurred in the 2006 FIBA World Championship.\nExact Answer: 2006 FIBA World Championship\nConfidence: 99%", "pred_items": ["2006 FIBA World Championship"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2960.63, "finish_reason": "stop", "usage": {"prompt_tokens": 2127, "completion_tokens": 90, "reasoning_tokens": 22, "total_tokens": 2217}}, "agentic-a": {"answer": "Explanation: The entity table in `structures/structures/entity/wiki_opentable_wikitq_00152__Tournament_stat_lookup.entities.csv` lists Florent Pietrus’s points per game by tournament, and the ranking summary in `structures/structures/_raw/wiki_opentable_wikitq_00152__Extremes_ranks_and_quick_comparisons.md` shows the highest scoring rate was 9.7 PPG in 2006 FIBA World Championship.\nExact Answer: 2006 FIBA World Championship\nConfidence: 96%", "pred_items": ["2006 FIBA World Championship"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 36359.66, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8883, "completion_tokens": 622, "total_tokens": 9505, "reasoning_tokens": 322}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The tournament-by-tournament stats in `structures/structures/entity/wiki_opentable_wikitq_00151__Tournament_stat_lookup.entities.csv` and the ranking summary in `structures/structures/_raw/wiki_opentable_wikitq_00151__Leaderboards_extrema_and_derived_totals.md` both show the highest points-per-game value as 9.7 in the 2006 FIBA World Championship.\nExact Answer: 2006 FIBA World Championship\nConfidence: 96%", "pred_items": ["2006 FIBA World Championship"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 587917.22, "stop_reason": "end_turn", "n_turns": 19, "max_turns": 50, "tool_call_counts": {"bash": 36, "python": 1, "read_file": 9}, "tokens": {"prompt_tokens": 100690, "completion_tokens": 9086, "total_tokens": 109776, "reasoning_tokens": 6132}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In Florent Piétrus’s national team career table in `structures/doc_044417.txt` (original id 49974), the highest `Points_per_game` value is 9.7, which corresponds to the 2006 FIBA World Championship.\nExact Answer: 2006 FIBA World Championship\nConfidence: 97%", "pred_items": ["2006 FIBA World Championship"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1287541.03, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 15, "tool_call_counts": {"bash": 10, "read_file": 1}, "tokens": {"prompt_tokens": 24287, "completion_tokens": 1501, "total_tokens": 25788, "reasoning_tokens": 911}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00153.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00153", "question": "How many international tournament games has Florent Piétrus played so far?", "dataset_origin": "WikiTQ", "original_table_id": "203-527", "sql": "SELECT SUM ( Games_played ) FROM table_203_527 ", "gold_answers": ["69"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Florent Piétrus accumulated a very long France national-team career across the Olympics, FIBA World Cup/World Championship, and EuroBasket. Counting his appearances in those senior international tournaments up through the latter part of his France career gives a total of about 103 games. This is the best-supported total from his tournament participation record.\nExact Answer: 103\nConfidence: 34%", "pred_items": ["103"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 36566.18, "finish_reason": "stop", "usage": {"prompt_tokens": 425, "completion_tokens": 1706, "reasoning_tokens": 1616, "total_tokens": 2131}}, "with-docs": {"answer": "Explanation: I added the Games_played values from each listed international tournament: 6 + 7 + 9 + 7 + 8 + 4 + 11 + 6 + 11 = 69.\nExact Answer: 69\nConfidence: 99%", "pred_items": ["69"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3635.95, "finish_reason": "stop", "usage": {"prompt_tokens": 746, "completion_tokens": 94, "reasoning_tokens": 28, "total_tokens": 840}}, "with-structures": {"answer": "Explanation: The listed international tournaments for Florent Piétrus show games played of 6, 7, 9, 7, 8, 4, 11, 6, and 11. Summing these gives 69 total games.\nExact Answer: 69\nConfidence: 99%", "pred_items": ["69"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2635.45, "finish_reason": "stop", "usage": {"prompt_tokens": 2282, "completion_tokens": 83, "reasoning_tokens": 9, "total_tokens": 2365}}, "agentic-a": {"answer": "Explanation: The raw numeric summary in `structures/_raw/wiki_opentable_wikitq_00153__Derived_aggregates_and_weighted_career_averages.md` explicitly states “Total games played: 69,” and the tournament entity table in `structures/entity/wiki_opentable_wikitq_00153__Tournament_lookup_leaders.entities.csv` sums to the same total.\nExact Answer: 69\nConfidence: 99%", "pred_items": ["69"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 40017.36, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 9986, "completion_tokens": 418, "total_tokens": 10404, "reasoning_tokens": 121}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The files `structures/structures/entity/wiki_opentable_wikitq_00151__Tournament_stat_lookup.entities.csv` and `structures/structures/_raw/wiki_opentable_wikitq_00151__Leaderboards_extrema_and_derived_totals.md` list Florent Piétrus’s tournament-by-tournament international appearances and explicitly give a career total of 69 games across the listed tournaments.\nExact Answer: 69\nConfidence: 84%", "pred_items": ["69"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 662566.92, "stop_reason": "end_turn", "n_turns": 22, "max_turns": 50, "tool_call_counts": {"bash": 43, "read_file": 12, "python": 1}, "tokens": {"prompt_tokens": 228721, "completion_tokens": 8364, "total_tokens": 237085, "reasoning_tokens": 4591}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_044417.txt` (original id `49974`), the “Florent Piétrus - National team career” table lists Games_played by tournament as 6, 7, 9, 7, 8, 4, 11, 6, and 11; summing these gives 69 total international tournament games so far.\nExact Answer: 69\nConfidence: 98%", "pred_items": ["69"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 2118982.91, "stop_reason": "end_turn", "n_turns": 9, "max_turns": 15, "tool_call_counts": {"bash": 8, "read_file": 1}, "tokens": {"prompt_tokens": 61184, "completion_tokens": 1538, "total_tokens": 62722, "reasoning_tokens": 861}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00154.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00154", "question": "From Florent Piétrus' list, what was Florent's average number of assists per game during the 2006 Fiba World Cup?", "dataset_origin": "WikiTQ", "original_table_id": "203-527", "sql": "SELECT Assists_per_game_ FROM table_203_527 WHERE Tournament = '2006 FIBA World Championship' ", "gold_answers": ["0.6"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: The 2006 FIBA World Cup statistics for Florent Piétrus list him at about 1 assist per game, specifically 1.1 assists per game.\nExact Answer: 1.1\nConfidence: 41%", "pred_items": ["1.1"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 13299.11, "finish_reason": "stop", "usage": {"prompt_tokens": 440, "completion_tokens": 596, "reasoning_tokens": 537, "total_tokens": 1036}}, "with-docs": {"answer": "Explanation: In the table for Florent Piétrus' national team career, the row for the 2006 FIBA World Championship lists his assists per game as 0.6.\nExact Answer: 0.6\nConfidence: 99%", "pred_items": ["0.6"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2767.99, "finish_reason": "stop", "usage": {"prompt_tokens": 761, "completion_tokens": 79, "reasoning_tokens": 18, "total_tokens": 840}}, "with-structures": {"answer": "Explanation: In the 2006 FIBA World Championship entry, Florent Piétrus is listed with stats of 9.7 points, 6.7 rebounds, and 0.6 assists per game. So his average assists per game was 0.6.\nExact Answer: 0.6\nConfidence: 98%", "pred_items": ["0.6"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4077.47, "finish_reason": "stop", "usage": {"prompt_tokens": 2539, "completion_tokens": 132, "reasoning_tokens": 53, "total_tokens": 2671}}, "agentic-a": {"answer": "Explanation: The tournament timeline in `structures/structures/time/wiki_opentable_wikitq_00154__Tournament_stat_lookup.timeline.json` lists “2006 FIBA World Championship — 9 GP, 9.7 / 6.7 / 0.6,” and the rankings markdown confirms 2006 FIBA World Championship had 0.6 APG.\nExact Answer: 0.6 assists per game\nConfidence: 99%", "pred_items": ["0.6 assists per game"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 118733.47, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 15, "tool_call_counts": {"bash": 4, "read_file": 7}, "tokens": {"prompt_tokens": 21478, "completion_tokens": 812, "total_tokens": 22290, "reasoning_tokens": 232}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The answer is given in `structures/structures/entity/wiki_opentable_wikitq_00151__Tournament_stat_lookup.entities.csv`, which lists Florent Piétrus’ tournament stats and shows the 2006 FIBA World Championship entry with `apg_float` 0.6. The `_raw/wiki_opentable_wikitq_00150__Extremes_and_ranked_comparisons.md` file confirms 2006 FIBA World Championship at 0.6 assists per game.\nExact Answer: 0.6 assists per game\nConfidence: 98%", "pred_items": ["0.6 assists per game"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 216805.95, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 50, "tool_call_counts": {"bash": 11, "read_file": 4}, "tokens": {"prompt_tokens": 23361, "completion_tokens": 1406, "total_tokens": 24767, "reasoning_tokens": 375}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In Florent Piétrus’ national team career table in `structures/doc_044417.txt` (original id `49974`), the row for the 2006 FIBA World Championship lists `Assists_per_game` as `0.6`.\nExact Answer: 0.6\nConfidence: 98%", "pred_items": ["0.6"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 643818.08, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 5}, "tokens": {"prompt_tokens": 10201, "completion_tokens": 572, "total_tokens": 10773, "reasoning_tokens": 281}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00155.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00155", "question": "For Florent Piétrus, how many assists per game did Eurobasket 2013 have overall?", "dataset_origin": "WikiTQ", "original_table_id": "203-527", "sql": "SELECT Assists_per_game_ FROM table_203_527 WHERE Tournament = '2013 EuroBasket' ", "gold_answers": ["0.2"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I identified the question as asking for Florent Piétrus’s assists per game in the overall EuroBasket 2013 tournament statistics. His overall assists-per-game figure for that event was 0.8.\nExact Answer: 0.8\nConfidence: 58%", "pred_items": ["0.8"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 17491.78, "finish_reason": "stop", "usage": {"prompt_tokens": 431, "completion_tokens": 770, "reasoning_tokens": 703, "total_tokens": 1201}}, "with-docs": {"answer": "Explanation: In the table for Florent Piétrus's national team career, the row for 2013 EuroBasket lists assists per game as 0.2.\nExact Answer: 0.2\nConfidence: 99%", "pred_items": ["0.2"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2458.63, "finish_reason": "stop", "usage": {"prompt_tokens": 752, "completion_tokens": 72, "reasoning_tokens": 15, "total_tokens": 824}}, "with-structures": {"answer": "Explanation: I looked at the row for 2013 EuroBasket in the provided table. The assists per game value (`apg_float`) is 0.2, which is also confirmed by the ranking document listing 2013 EuroBasket as the lowest APG at 0.2.\n\nExact Answer: 0.2\nConfidence: 99%", "pred_items": ["0.2"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3851.97, "finish_reason": "stop", "usage": {"prompt_tokens": 1728, "completion_tokens": 101, "reasoning_tokens": 19, "total_tokens": 1829}}, "agentic-a": {"answer": "Explanation: The entity table in `structures/structures/entity/wiki_opentable_wikitq_00155__Tournament_lookup_with_derived_totals.entities.csv` lists Florent Piétrus’s `apg_float` for `2013_eurobasket` as 0.2; the raw rankings file also confirms 2013 EuroBasket had the lowest APG at 0.2.\nExact Answer: 0.2\nConfidence: 99%", "pred_items": ["0.2"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 56792.16, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 7852, "completion_tokens": 446, "total_tokens": 8298, "reasoning_tokens": 140}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: In `structures/structures/entity/wiki_opentable_wikitq_00151__Tournament_stat_lookup.entities.csv` and `structures/structures/entity/wiki_opentable_wikitq_00155__Tournament_lookup_with_derived_totals.entities.csv`, the row for `2013 EuroBasket` lists `apg_float` as 0.2, consistent with 2.2 total assists over 11 games.\nExact Answer: 0.2 assists per game\nConfidence: 99%", "pred_items": ["0.2 assists per game"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 209845.33, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 50, "tool_call_counts": {"bash": 9, "read_file": 5}, "tokens": {"prompt_tokens": 22978, "completion_tokens": 1252, "total_tokens": 24230, "reasoning_tokens": 343}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_044417.txt` (original id `49974`), the “Florent Piétrus - National team career” table lists his `2013 EuroBasket` row with `Assists_per_game` of `0.2`.\nExact Answer: 0.2\nConfidence: 98%", "pred_items": ["0.2"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 547436.75, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 6}, "tokens": {"prompt_tokens": 10146, "completion_tokens": 500, "total_tokens": 10646, "reasoning_tokens": 209}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00176.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00176", "question": "How many schools are represented in the table of list of technical universities in the United States?", "dataset_origin": "WikiTQ", "original_table_id": "203-721", "sql": "SELECT COUNT ( School ) FROM table_203_721 ", "gold_answers": ["28"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: The question asks for the number of schools listed in the table for technical universities in the United States. Counting the entries in that table gives a total of 35 schools.\nExact Answer: 35\nConfidence: 31%", "pred_items": ["35"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 22222.94, "finish_reason": "stop", "usage": {"prompt_tokens": 428, "completion_tokens": 1054, "reasoning_tokens": 997, "total_tokens": 1482}}, "with-docs": {"answer": "Explanation: I counted the school entries across the three table segments: 9 in Doc 1, 9 in Doc 2, and 10 in Doc 3. Adding them gives 28 schools total.\nExact Answer: 28\nConfidence: 99%", "pred_items": ["28"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 5237.61, "finish_reason": "stop", "usage": {"prompt_tokens": 1329, "completion_tokens": 201, "reasoning_tokens": 137, "total_tokens": 1530}}, "with-structures": {"answer": "Explanation: I counted all institutions listed across the states in the reference table: California (3), Colorado (2), Georgia (2), Massachusetts (2), New Jersey (2), New York (4), and 1 each in Ohio, Florida, Illinois, Michigan, Missouri, Montana, New Mexico, Oregon, Indiana, South Dakota, Texas, Virginia, and Wisconsin. The total is 28 schools.\nExact Answer: 28\nConfidence: 98%", "pred_items": ["28"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 5965.0, "finish_reason": "stop", "usage": {"prompt_tokens": 2277, "completion_tokens": 260, "reasoning_tokens": 157, "total_tokens": 2537}}, "agentic-a": {"answer": "Explanation: Counting the institution entries listed by state in `wiki_opentable_wikitq_00176__Geographic_lookup_state_city_institutions.md` gives 28 schools; I cross-checked that the additional schools beyond the partial entity CSV also appear in `wiki_opentable_wikitq_00176__Extremes_rankings_and_ties.md`.\nExact Answer: 28\nConfidence: 95%", "pred_items": ["28"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 165284.58, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 15, "tool_call_counts": {"bash": 4, "read_file": 4, "python": 2}, "tokens": {"prompt_tokens": 35673, "completion_tokens": 2200, "total_tokens": 37873, "reasoning_tokens": 1364}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The table is represented in `structures/structures/entity/wiki_opentable_wikitq_00178__Master_institution_lookup.entities.csv` and corroborated by `structures/structures/entity/wiki_opentable_wikitq_00180__Institution_master_index.entities.csv`; each file lists 28 unique institutions from the “list of technical universities in the United States.”\nExact Answer: 28\nConfidence: 98%", "pred_items": ["28"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 245523.62, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 50, "tool_call_counts": {"bash": 13, "read_file": 7, "python": 1}, "tokens": {"prompt_tokens": 52439, "completion_tokens": 1911, "total_tokens": 54350, "reasoning_tokens": 646}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: The table titled “List of United States technological universities - Institutions” is split across doc_045257.txt (9 schools), doc_045259.txt (9 schools), and doc_045260.txt (10 schools), for a total of 28 schools.\nExact Answer: 28\nConfidence: 99%", "pred_items": ["28"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1398383.06, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 15, "tool_call_counts": {"bash": 8, "read_file": 3}, "tokens": {"prompt_tokens": 26370, "completion_tokens": 1361, "total_tokens": 27731, "reasoning_tokens": 700}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00178.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00178", "question": "Among the list of technical universities in the United States, which school had the most enrollments?", "dataset_origin": "WikiTQ", "original_table_id": "203-721", "sql": "SELECT School FROM table_203_721 ORDER BY Enrollment DESC LIMIT 1 ", "gold_answers": ["Texas Tech University"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Comparing enrollments among well-known U.S. technical universities, Georgia Institute of Technology has the largest student enrollment, exceeding other technical-focused institutions like RIT, NJIT, and Illinois Tech.\nExact Answer: Georgia Institute of Technology\nConfidence: 93%", "pred_items": ["Georgia Institute of Technology"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 6355.68, "finish_reason": "stop", "usage": {"prompt_tokens": 429, "completion_tokens": 260, "reasoning_tokens": 197, "total_tokens": 689}}, "with-docs": {"answer": "Explanation: I compared the enrollment figures listed for all the schools in the provided tables. The highest enrollment shown is 32,327, which belongs to Texas Tech University.\nExact Answer: Texas Tech University\nConfidence: 98%", "pred_items": ["Texas Tech University"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2867.7, "finish_reason": "stop", "usage": {"prompt_tokens": 1330, "completion_tokens": 95, "reasoning_tokens": 39, "total_tokens": 1425}}, "with-structures": {"answer": "Explanation: I checked the enrollment figures in the provided table. The highest enrollment listed is 32,327, which belongs to Texas Tech University. Therefore, it had the most enrollments among the schools shown.\nExact Answer: Texas Tech University\nConfidence: 99%", "pred_items": ["Texas Tech University"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2759.15, "finish_reason": "stop", "usage": {"prompt_tokens": 3235, "completion_tokens": 77, "reasoning_tokens": 13, "total_tokens": 3312}}, "agentic-a": {"answer": "Explanation: The entity lookup in `structures/entity/wiki_opentable_wikitq_00178__Master_institution_lookup.entities.csv` lists each school’s enrollment, and the numeric ranking in `structures/_raw/wiki_opentable_wikitq_00178__Comparative_matrix_type_control_age_size.md` confirms the top enrollment. Both show Texas Tech University as the largest by enrollment.\nExact Answer: Texas Tech University\nConfidence: 99%", "pred_items": ["Texas Tech University"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 70421.36, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 10011, "completion_tokens": 418, "total_tokens": 10429, "reasoning_tokens": 143}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The enrollment values in `structures/structures/entity/wiki_opentable_wikitq_00178__Master_institution_lookup.entities.csv` and the duplicate-indexed `wiki_opentable_wikitq_00180__Institution_master_index.entities.csv` show that the highest `enrollment_int` among the listed U.S. technical universities is 32,327 for Texas Tech University.\nExact Answer: Texas Tech University\nConfidence: 98%", "pred_items": ["Texas Tech University"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 273667.37, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 50, "tool_call_counts": {"bash": 12, "read_file": 4, "python": 1}, "tokens": {"prompt_tokens": 43392, "completion_tokens": 1738, "total_tokens": 45130, "reasoning_tokens": 553}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In doc_045257.txt (original id 50729, “List of United States technological universities - Institutions”), the enrollment column shows Colorado Technical University with 30,020 students, higher than every other listed school.\nExact Answer: Colorado Technical University\nConfidence: 96%", "pred_items": ["Colorado Technical University"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1701063.43, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 15, "tool_call_counts": {"bash": 12, "read_file": 1}, "tokens": {"prompt_tokens": 34503, "completion_tokens": 1435, "total_tokens": 35938, "reasoning_tokens": 804}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00179.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00179", "question": "Which of the technical universities in the United States was founded first?", "dataset_origin": "WikiTQ", "original_table_id": "203-721", "sql": "SELECT School FROM table_203_721 ORDER BY Founded_ LIMIT '1' ", "gold_answers": ["Rensselaer Polytechnic Institute"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: To find which U.S. technical university was founded first, compare the founding dates of major American technical universities. Rensselaer Polytechnic Institute was founded in 1824, earlier than MIT (1861), Michigan Technological University (1885), and others. Therefore it is the earliest-founded technical university in the United States.\nExact Answer: Rensselaer Polytechnic Institute\nConfidence: 77%", "pred_items": ["Rensselaer Polytechnic Institute"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 10195.57, "finish_reason": "stop", "usage": {"prompt_tokens": 423, "completion_tokens": 402, "reasoning_tokens": 307, "total_tokens": 825}}, "with-docs": {"answer": "Explanation: I compared the founding years of all the listed U.S. technological universities in the supporting documents. The earliest year shown is 1824 for Rensselaer Polytechnic Institute, which is earlier than all the others listed.\nExact Answer: Rensselaer Polytechnic Institute\nConfidence: 99%", "pred_items": ["Rensselaer Polytechnic Institute"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3735.63, "finish_reason": "stop", "usage": {"prompt_tokens": 1324, "completion_tokens": 123, "reasoning_tokens": 50, "total_tokens": 1447}}, "with-structures": {"answer": "Explanation: I looked for the earliest founded institution among the listed U.S. technical universities. In the “Oldest institutions” ranking, the earliest founding year is 1824 for Rensselaer Polytechnic Institute, making it the one founded first.\n\nExact Answer: Rensselaer Polytechnic Institute\nConfidence: 98%", "pred_items": ["Rensselaer Polytechnic Institute"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3893.89, "finish_reason": "stop", "usage": {"prompt_tokens": 3541, "completion_tokens": 124, "reasoning_tokens": 47, "total_tokens": 3665}}, "agentic-a": {"answer": "Explanation: The ranking in `structures/structures/_raw/wiki_opentable_wikitq_00179__Extremes_and_ranked_lists.md` lists the oldest institution by founded year, and the entity records in `structures/structures/entity/wiki_opentable_wikitq_00179__Attribute_filter_matrix.entities.csv` confirm the founding years.  \nExact Answer: Rensselaer Polytechnic Institute\nConfidence: 99%", "pred_items": ["Rensselaer Polytechnic Institute"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 104895.1, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 15, "tool_call_counts": {"bash": 3, "read_file": 7}, "tokens": {"prompt_tokens": 18456, "completion_tokens": 782, "total_tokens": 19238, "reasoning_tokens": 321}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The answer is grounded in `structures/structures/_raw/wiki_opentable_wikitq_00180__Comparative_rankings_and_extrema.md`, which lists the “Oldest founded” technical universities, and corroborated by `structures/structures/entity/wiki_opentable_wikitq_00178__Master_institution_lookup.entities.csv`, which gives each institution’s founding year. Both identify Rensselaer Polytechnic Institute as the earliest, founded in 1824.\nExact Answer: Rensselaer Polytechnic Institute\nConfidence: 98%", "pred_items": ["Rensselaer Polytechnic Institute"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 196174.11, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 50, "tool_call_counts": {"bash": 9, "read_file": 5}, "tokens": {"prompt_tokens": 30409, "completion_tokens": 1368, "total_tokens": 31777, "reasoning_tokens": 424}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_045257.txt` (original id `50729`), the table “List of United States technological universities - Institutions” lists each school with its founding year; the earliest year shown is 1874 for Colorado School of Mines.\nExact Answer: Colorado School of Mines\nConfidence: 98%", "pred_items": ["Colorado School of Mines"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1501446.69, "stop_reason": "end_turn", "n_turns": 7, "max_turns": 15, "tool_call_counts": {"bash": 8, "read_file": 1}, "tokens": {"prompt_tokens": 29934, "completion_tokens": 1304, "total_tokens": 31238, "reasoning_tokens": 769}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00180.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00180", "question": "What is the total number of schools listed in the table below the list of technical universities in the United States?", "dataset_origin": "WikiTQ", "original_table_id": "203-721", "sql": "SELECT COUNT ( School ) FROM table_203_721 ", "gold_answers": ["28"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I interpreted the question as asking for the count of school entries in the table that appears beneath the list of technical universities in the United States. Counting the listed schools gives a total of 35.\nExact Answer: 35\nConfidence: 44%", "pred_items": ["35"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 24641.22, "finish_reason": "stop", "usage": {"prompt_tokens": 432, "completion_tokens": 1363, "reasoning_tokens": 1301, "total_tokens": 1795}}, "with-docs": {"answer": "Explanation: I counted the schools listed in the three table sections: 9 in Doc 1, 9 in Doc 2, and 10 in Doc 3. Adding them gives 28.\nExact Answer: 28\nConfidence: 99%", "pred_items": ["28"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2984.37, "finish_reason": "stop", "usage": {"prompt_tokens": 1333, "completion_tokens": 98, "reasoning_tokens": 36, "total_tokens": 1431}}, "with-structures": {"answer": "Explanation: I counted the schools listed in the provided table of technical universities. There are 28 institutions total. This is also consistent with the control grouping totals: 15 public + 13 private = 28.\n\nExact Answer: 28\nConfidence: 99%", "pred_items": ["28"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 7694.13, "finish_reason": "stop", "usage": {"prompt_tokens": 3660, "completion_tokens": 219, "reasoning_tokens": 155, "total_tokens": 3879}}, "agentic-a": {"answer": "Explanation: The count comes from the entity index in `structures/entity/wiki_opentable_wikitq_00180__Institution_master_index.entities.csv`, which lists each school in the table as one row. Counting those entries gives 28 schools.\nExact Answer: 28\nConfidence: 99%", "pred_items": ["28"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 76159.14, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 2, "read_file": 2}, "tokens": {"prompt_tokens": 12864, "completion_tokens": 548, "total_tokens": 13412, "reasoning_tokens": 303}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: I grounded this in `structures/structures/entity/wiki_opentable_wikitq_00176__Faceted_institution_index.entities.csv`, which lists the schools from the technical universities table; it has 20 data rows (21 lines including the header). The related raw summaries for `wiki_opentable_wikitq_00176` confirm this is the technical universities in the United States table.\nExact Answer: 20\nConfidence: 97%", "pred_items": ["20"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 280276.45, "stop_reason": "end_turn", "n_turns": 9, "max_turns": 50, "tool_call_counts": {"bash": 18, "read_file": 3, "python": 1}, "tokens": {"prompt_tokens": 39556, "completion_tokens": 2357, "total_tokens": 41913, "reasoning_tokens": 1034}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: The institutions table for “List of United States technological universities” is split across `doc_045257.txt` (id 50729), `doc_045259.txt` (id 50730), and `doc_045260.txt` (id 50731). Counting the school rows in those three table fragments gives 9 + 9 + 10 = 28.\nExact Answer: 28\nConfidence: 99%", "pred_items": ["28"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 2570739.44, "stop_reason": "end_turn", "n_turns": 10, "max_turns": 15, "tool_call_counts": {"bash": 17, "read_file": 1}, "tokens": {"prompt_tokens": 49563, "completion_tokens": 2511, "total_tokens": 52074, "reasoning_tokens": 1620}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00181.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00181", "question": "Which party finished last in the 1998 Swedish general election?", "dataset_origin": "WikiTQ", "original_table_id": "202-231", "sql": "SELECT Party FROM table_202_231 ORDER BY __2 LIMIT 1 ", "gold_answers": ["Sweden Democrats"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: In the 1998 Swedish general election, among the parties represented in the Riksdag results, the Green Party received the smallest vote share, finishing last with about 4.5% of the vote.\nExact Answer: Green Party\nConfidence: 96%", "pred_items": ["Green Party"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 5307.07, "finish_reason": "stop", "usage": {"prompt_tokens": 423, "completion_tokens": 199, "reasoning_tokens": 134, "total_tokens": 622}}, "with-docs": {"answer": "Explanation: From the listed parties in the 1998 Swedish general election results, the Sweden Democrats had the fewest votes (19,624, or 0.4%) and no seats, so they finished last among the parties.\nExact Answer: Sweden Democrats\nConfidence: 98%", "pred_items": ["Sweden Democrats"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3368.1, "finish_reason": "stop", "usage": {"prompt_tokens": 733, "completion_tokens": 104, "reasoning_tokens": 36, "total_tokens": 837}}, "with-structures": {"answer": "Explanation: From the results table, “Other parties” is an aggregate category rather than a single party. Among the named parties, the lowest-ranked finisher was the Sweden Democrats, with 19,624 votes (0.4%) and 0 seats.\nExact Answer: Sweden Democrats\nConfidence: 93%", "pred_items": ["Sweden Democrats"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4515.21, "finish_reason": "stop", "usage": {"prompt_tokens": 1448, "completion_tokens": 134, "reasoning_tokens": 61, "total_tokens": 1582}}, "agentic-a": {"answer": "Explanation: The entity structure `structures/structures/entity/wiki_opentable_wikitq_00181__Party_results_leaderboard.entities.csv` ranks parties by votes in the 1998 Swedish general election. The last-ranked entry is `Other parties` with rank 9.\nExact Answer: Other parties\nConfidence: 96%", "pred_items": ["Other parties"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 79678.63, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 2, "read_file": 4}, "tokens": {"prompt_tokens": 13152, "completion_tokens": 764, "total_tokens": 13916, "reasoning_tokens": 432}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: In `structures/structures/entity/wiki_opentable_wikitq_00181__Party_results_leaderboard.entities.csv`, the ranked results place Sweden Democrats last among the named parties (rank 8, 0.4% of the vote); `structures/structures/_raw/wiki_opentable_wikitq_00181__Representation_threshold_index.md` confirms Sweden Democrats had the lowest vote share of the listed parties.\nExact Answer: Sweden Democrats\nConfidence: 91%", "pred_items": ["Sweden Democrats"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 296225.17, "stop_reason": "end_turn", "n_turns": 10, "max_turns": 50, "tool_call_counts": {"bash": 20, "python": 2, "read_file": 2}, "tokens": {"prompt_tokens": 56262, "completion_tokens": 2821, "total_tokens": 59083, "reasoning_tokens": 1265}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In the results table in `structures/doc_042741.txt` (original id 48465, “Swedish general election, 1998 - Results”), the lowest vote total among named parties is for the Sweden Democrats with 19,624 votes (0.4%).\nExact Answer: Sweden Democrats\nConfidence: 98%", "pred_items": ["Sweden Democrats"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1181004.65, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 9}, "tokens": {"prompt_tokens": 18192, "completion_tokens": 695, "total_tokens": 18887, "reasoning_tokens": 292}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00317.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00317", "question": "List of Olympic medalists in snowboarding, who has more silver medals: Karine Ruby or Shaun White?", "dataset_origin": "WikiTQ", "original_table_id": "203-308", "sql": "SELECT Athlete FROM table_203_308 WHERE Athlete IN ( 'Karine Ruby' , 'Shaun White' ) ORDER BY Silver DESC LIMIT 1 ", "gold_answers": ["Karine Ruby"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Karine Ruby won 1 Olympic silver medal in snowboarding, while Shaun White won 0 Olympic silver medals. Therefore, Karine Ruby has more silver medals.\nExact Answer: Karine Ruby\nConfidence: 98%", "pred_items": ["Karine Ruby"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4210.86, "finish_reason": "stop", "usage": {"prompt_tokens": 432, "completion_tokens": 153, "reasoning_tokens": 96, "total_tokens": 585}}, "with-docs": {"answer": "Explanation: In the table, Karine Ruby has 1 silver medal, while Shaun White has 0 silver medals. Therefore, Karine Ruby has more silver medals.\nExact Answer: Karine Ruby\nConfidence: 99%", "pred_items": ["Karine Ruby"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2736.49, "finish_reason": "stop", "usage": {"prompt_tokens": 761, "completion_tokens": 76, "reasoning_tokens": 20, "total_tokens": 837}}, "with-structures": {"answer": "Explanation: From the provided medal table, Karine Ruby has 1 silver medal and Shaun White has 0 silver medals. Therefore, Karine Ruby has more silver medals.\nExact Answer: Karine Ruby\nConfidence: 99%", "pred_items": ["Karine Ruby"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2798.54, "finish_reason": "stop", "usage": {"prompt_tokens": 1646, "completion_tokens": 75, "reasoning_tokens": 18, "total_tokens": 1721}}, "agentic-a": {"answer": "Explanation: The entity table `structures/entity/wiki_opentable_wikitq_00317__Athlete_medal_leaders_lookup.entities.csv` lists silver medals for both athletes: Karine Ruby has 1 silver and Shaun White has 0; the raw field-value index also supports these listed athlete facts.\nExact Answer: Karine Ruby\nConfidence: 99%", "pred_items": ["Karine Ruby"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 56343.61, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8910, "completion_tokens": 353, "total_tokens": 9263, "reasoning_tokens": 116}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The snowboarding Olympic medalist tables in `structures/structures/_raw/wiki_opentable_wikitq_00319__Olympics_span_repeat_medalist_index.md` and `structures/structures/entity/wiki_opentable_wikitq_00323__Athlete_medal_leaderboard_and_medal_breakdown.entities.csv` list Karine Ruby with 1 silver medal and Shaun White with 0 silver medals.\nExact Answer: Karine Ruby\nConfidence: 99%", "pred_items": ["Karine Ruby"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 169160.05, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 5, "read_file": 3}, "tokens": {"prompt_tokens": 18186, "completion_tokens": 887, "total_tokens": 19073, "reasoning_tokens": 285}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_043618.txt` (original id `49254`), the \"Athlete medal leaders\" table shows Karine Ruby with 1 silver medal and Shaun White with 0 silver medals.\nExact Answer: Karine Ruby\nConfidence: 99%", "pred_items": ["Karine Ruby"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 457064.45, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 4}, "tokens": {"prompt_tokens": 9748, "completion_tokens": 431, "total_tokens": 10179, "reasoning_tokens": 204}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00319.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00319", "question": "List of Olympic medalists in snowboarding, who had the fewest gold medals?", "dataset_origin": "WikiTQ", "original_table_id": "203-308", "sql": "SELECT Athlete FROM table_203_308 ORDER BY Gold LIMIT 1 ", "gold_answers": ["Danny Kass"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: The minimum number of Olympic gold medals for a snowboarding Olympic medalist is 0, so the answer is the snowboard Olympic medalists who won silver and/or bronze but never won Olympic gold. Here are the ones I know.\n\nExact Answer: Daniel Franck; Nicolas Huet; Thomas Prugger; Markus Ebner; Heidi Renoth; Brigitte Köck; Shannon Dunn-Downing; Doris Günther; Ueli Kestenholz; Richard Richardsson; Chris Klug; Rosey Fletcher; Dorian Gray; J. J. Thomas; Doresia Krings; Fabienne Reuteler; Simon Schoch; Amelie Kober; Danny Kass; Markku Koski; Gretchen Bleiler; Kjersti Buaas; Dominique Maltais; Hélène Olafsen; Peetu Piiroinen; Scotty Lago; Benjamin Karl; Mathieu Bozzetto; Ekaterina Ilyukhina; Marion Kreiner; Mike Robertson; Tony Ramoin; Déborah Anthonioz; Olivia Nobs; Ståle Sandbech; Mark McMorris; Enni Rukajärvi; Jenny Jones; Ayumu Hirano; Taku Hiraoka; Žan Košir; Tomoka Takeuchi; Alena Zavarzina; Nikolay Olyunin; Alex Deibold; Chloé Trespeuch; Kyle Mack; Billy Morgan; Laurie Blouin; Jiayu Liu; Arielle Gold; Lee Sang-ho; Selina Jörg; Ramona Hofmeister; Jarryd Hughes; Regino Hernández; Julia Pereira de Sousa; Su Yiming; Kokomo Murase; Tess Coady; Julia Marino; Ruka Hirano; Queralt Castellet; Sena Tomita; Tim Mastnak; Stefan Baumeister; Daniela Ulbing; Gloria Kotnik; Eliot Grondin; Meryeta O'Dine; Omar Visintin; Jakob Dusek\nConfidence: 61%", "pred_items": ["Daniel Franck", "Nicolas Huet", "Thomas Prugger", "Markus Ebner", "Heidi Renoth", "Brigitte Köck", "Shannon Dunn-Downing", "Doris Günther", "Ueli Kestenholz", "Richard Richardsson", "Chris Klug", "Rosey Fletcher", "Dorian Gray", "J. J. Thomas", "Doresia Krings", "Fabienne Reuteler", "Simon Schoch", "Amelie Kober", "Danny Kass", "Markku Koski", "Gretchen Bleiler", "Kjersti Buaas", "Dominique Maltais", "Hélène Olafsen", "Peetu Piiroinen", "Scotty Lago", "Benjamin Karl", "Mathieu Bozzetto", "Ekaterina Ilyukhina", "Marion Kreiner", "Mike Robertson", "Tony Ramoin", "Déborah Anthonioz", "Olivia Nobs", "Ståle Sandbech", "Mark McMorris", "Enni Rukajärvi", "Jenny Jones", "Ayumu Hirano", "Taku Hiraoka", "Žan Košir", "Tomoka Takeuchi", "Alena Zavarzina", "Nikolay Olyunin", "Alex Deibold", "Chloé Trespeuch", "Kyle Mack", "Billy Morgan", "Laurie Blouin", "Jiayu Liu", "Arielle Gold", "Lee Sang-ho", "Selina Jörg", "Ramona Hofmeister", "Jarryd Hughes", "Regino Hernández", "Julia Pereira de Sousa", "Su Yiming", "Kokomo Murase", "Tess Coady", "Julia Marino", "Ruka Hirano", "Queralt Castellet", "Sena Tomita", "Tim Mastnak", "Stefan Baumeister", "Daniela Ulbing", "Gloria Kotnik", "Eliot Grondin", "Meryeta O'Dine", "Omar Visintin", "Jakob Dusek"], "metrics": {"precision": 0.0139, "recall": 1.0, "f1": 0.0274, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 58651.68, "finish_reason": "stop", "usage": {"prompt_tokens": 427, "completion_tokens": 2993, "reasoning_tokens": 2577, "total_tokens": 3420}}, "with-docs": {"answer": "Explanation: From the table, the smallest number of gold medals is 0. The only athlete listed with 0 gold medals is Danny Kass.\nExact Answer: Danny Kass\nConfidence: 98%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3097.75, "finish_reason": "stop", "usage": {"prompt_tokens": 756, "completion_tokens": 82, "reasoning_tokens": 32, "total_tokens": 838}}, "with-structures": {"answer": "Explanation: From the listed snowboarding Olympic medalists, the fewest number of gold medals is 0. The only athlete with 0 gold medals is Danny Kass, who has 2 silver medals.\nExact Answer: Danny Kass\nConfidence: 97%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3507.58, "finish_reason": "stop", "usage": {"prompt_tokens": 1382, "completion_tokens": 106, "reasoning_tokens": 44, "total_tokens": 1488}}, "agentic-a": {"answer": "Explanation: Using the entity medal leaderboard in `structures/entity/wiki_opentable_wikitq_00319__Athlete_medal_leaderboard_tie_groups.entities.csv`, I compared the `gold_int` values for all listed snowboard Olympic medalists; the minimum is 0. The same result is corroborated by the medal profile `2S` in the raw time index file.\nExact Answer: Danny Kass\nConfidence: 98%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 58958.96, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8524, "completion_tokens": 446, "total_tokens": 8970, "reasoning_tokens": 166}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: In `structures/structures/entity/wiki_opentable_wikitq_00323__Athlete_medal_leaderboard_and_medal_breakdown.entities.csv` and `.../wiki_opentable_wikitq_00321__Athlete_leaderboard_and_tie_structure.entities.csv`, Danny Kass is the only listed snowboard Olympic medalist with `gold_int = 0`, the lowest value in the table.\nExact Answer: Danny Kass\nConfidence: 99%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 239247.54, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 50, "tool_call_counts": {"bash": 6, "read_file": 9}, "tokens": {"prompt_tokens": 33058, "completion_tokens": 2012, "total_tokens": 35070, "reasoning_tokens": 1028}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In the table in `structures/doc_043618.txt` (`# id: 49254`, “List of Olympic medalists in snowboarding - Statistics”), the smallest value in the Gold column is 0, which appears only for Danny Kass.\nExact Answer: Danny Kass\nConfidence: 98%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1100505.32, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 7, "read_file": 1}, "tokens": {"prompt_tokens": 18206, "completion_tokens": 1061, "total_tokens": 19267, "reasoning_tokens": 615}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00320.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00320", "question": "Among Olympic medalists in snowboarding, who has won more medals - Shaun White or Kelly Clark?", "dataset_origin": "WikiTQ", "original_table_id": "203-308", "sql": "SELECT Athlete FROM table_203_308 WHERE Athlete IN ( 'Shaun White' , 'Kelly Clark' ) ORDER BY Total_ DESC LIMIT 1 ", "gold_answers": ["Kelly Clark"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Shaun White has won 3 Olympic snowboarding medals, all gold. Kelly Clark has also won 3 Olympic snowboarding medals: 1 gold and 2 bronze. Therefore, neither has won more medals; they are tied.\nExact Answer: Neither — they are tied at 3 medals each\nConfidence: 99%", "pred_items": ["Neither — they are tied at 3 medals each"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4357.46, "finish_reason": "stop", "usage": {"prompt_tokens": 430, "completion_tokens": 138, "reasoning_tokens": 61, "total_tokens": 568}}, "with-docs": {"answer": "Explanation: The table shows Shaun White with 2 total Olympic snowboarding medals and Kelly Clark with 3 total medals. Since 3 is greater than 2, Kelly Clark has won more medals.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2820.18, "finish_reason": "stop", "usage": {"prompt_tokens": 759, "completion_tokens": 83, "reasoning_tokens": 22, "total_tokens": 842}}, "with-structures": {"answer": "Explanation: From the medal totals in the provided table, Kelly Clark has 3 Olympic snowboarding medals, while Shaun White has 2. Therefore, Kelly Clark has won more medals.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2608.98, "finish_reason": "stop", "usage": {"prompt_tokens": 1295, "completion_tokens": 75, "reasoning_tokens": 17, "total_tokens": 1370}}, "agentic-a": {"answer": "Explanation: The entity table `structures/structures/entity/wiki_opentable_wikitq_00320__Athlete_medal_leaderboard_and_key_comparisons.entities.csv` lists Kelly Clark with 3 total medals and Shaun White with 2. Therefore, Kelly Clark has won more Olympic snowboarding medals.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 47687.7, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 10931, "completion_tokens": 344, "total_tokens": 11275, "reasoning_tokens": 89}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The entity tables `structures/structures/entity/wiki_opentable_wikitq_00323__Athlete_medal_leaderboard_and_medal_breakdown.entities.csv` and `wiki_opentable_wikitq_00317__Athlete_medal_leaders_lookup.entities.csv` list Kelly Clark with 3 Olympic medals and Shaun White with 2. The raw summary `structures/structures/_raw/wiki_opentable_wikitq_00319__Olympics_span_repeat_medalist_index.md` confirms the medal profiles (Kelly Clark: 1G, 2B; Shaun White: 2G).\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 165503.01, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 50, "tool_call_counts": {"bash": 5, "read_file": 3}, "tokens": {"prompt_tokens": 17508, "completion_tokens": 894, "total_tokens": 18402, "reasoning_tokens": 238}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_043618.txt` (original id `49254`), the “List of Olympic medalists in snowboarding - Statistics” table shows Shaun White with 2 total medals and Kelly Clark with 3 total medals, so Kelly Clark has more.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 668284.88, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 3, "read_file": 1}, "tokens": {"prompt_tokens": 16771, "completion_tokens": 562, "total_tokens": 17333, "reasoning_tokens": 221}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00321.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00321", "question": "For the list of Olympic snowboarding medalists, who had the fewest gold medals but the most silver medals?", "dataset_origin": "WikiTQ", "original_table_id": "203-308", "sql": "SELECT Athlete FROM table_203_308 ORDER BY Gold LIMIT 1 ", "gold_answers": ["Danny Kass"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: To satisfy both conditions, I looked for Olympic snowboarding medalists with the minimum possible number of gold medals among medalists who won silver medals, which is 0 golds, and then identified who among them had the highest number of silver medals. The highest silver total in that group is 2. The medalists matching that are Danny Kass, Peetu Piiroinen, Claudia Riegler, and Amelie Kober.\n\nExact Answer: Danny Kass; Peetu Piiroinen; Claudia Riegler; Amelie Kober\n\nConfidence: 74%", "pred_items": ["Danny Kass", "Peetu Piiroinen", "Claudia Riegler", "Amelie Kober"], "metrics": {"precision": 0.25, "recall": 1.0, "f1": 0.4, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 22323.35, "finish_reason": "stop", "usage": {"prompt_tokens": 433, "completion_tokens": 1074, "reasoning_tokens": 946, "total_tokens": 1507}}, "with-docs": {"answer": "Explanation: From the table, the fewest gold medals among the listed athletes is 0. The athlete with 0 golds and the highest number of silver medals is Danny Kass, with 2 silver medals.\nExact Answer: Danny Kass\nConfidence: 99%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3964.83, "finish_reason": "stop", "usage": {"prompt_tokens": 762, "completion_tokens": 133, "reasoning_tokens": 69, "total_tokens": 895}}, "with-structures": {"answer": "Explanation: From the listed athletes, the fewest gold medals is 0. Danny Kass has 0 gold medals and 2 silver medals, which is also the highest silver total in the list. Therefore he fits both conditions.\nExact Answer: Danny Kass\nConfidence: 98%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4215.48, "finish_reason": "stop", "usage": {"prompt_tokens": 1428, "completion_tokens": 116, "reasoning_tokens": 49, "total_tokens": 1544}}, "agentic-a": {"answer": "Explanation: The athlete medal counts in `structures/structures/entity/wiki_opentable_wikitq_00321__Athlete_leaderboard_and_tie_structure.entities.csv` show that Danny Kass has the fewest gold medals (0) and the most silver medals (2) among the listed Olympic snowboarding medalists.\nExact Answer: Danny Kass\nConfidence: 99%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 53618.71, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8986, "completion_tokens": 444, "total_tokens": 9430, "reasoning_tokens": 197}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: In `structures/structures/entity/wiki_opentable_wikitq_00320__Athlete_medal_leaderboard_and_key_comparisons.entities.csv` and `...00327__Athlete_medal_snapshot.entities.csv`, Danny Kass is listed with 0 gold medals and 2 silver medals, and the file notes he had the most silvers in the list.\nExact Answer: Danny Kass\nConfidence: 98%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 228933.7, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 50, "tool_call_counts": {"bash": 11, "read_file": 6}, "tokens": {"prompt_tokens": 28943, "completion_tokens": 1673, "total_tokens": 30616, "reasoning_tokens": 616}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In the athlete medal leaders table in `doc_043618.txt` (original id 49254, “List of Olympic medalists in snowboarding - Statistics”), Danny Kass is the only listed athlete with 0 gold medals and he also has the highest silver count at 2.\nExact Answer: Danny Kass\nConfidence: 98%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 902707.97, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 9}, "tokens": {"prompt_tokens": 15202, "completion_tokens": 785, "total_tokens": 15987, "reasoning_tokens": 348}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00322.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00322", "question": "From the list of Olympic snowboarding medalists, who has won the most medals?", "dataset_origin": "WikiTQ", "original_table_id": "203-308", "sql": "SELECT Athlete FROM table_203_308 ORDER BY Total_ DESC LIMIT 1 ", "gold_answers": ["Kelly Clark"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Olympic snowboarding medal records are tied at 3 medals. Several snowboarders have reached that total, and I’m not aware of any snowboarder with 4 or more Olympic snowboarding medals.  \nExact Answer: Shaun White; Karine Ruby; Kelly Clark; Torah Bright; Ayumu Hirano; Lindsey Jacobellis; Mark McMorris; Benjamin Karl; Vic Wild\nConfidence: 72%", "pred_items": ["Shaun White", "Karine Ruby", "Kelly Clark", "Torah Bright", "Ayumu Hirano", "Lindsey Jacobellis", "Mark McMorris", "Benjamin Karl", "Vic Wild"], "metrics": {"precision": 0.1111, "recall": 1.0, "f1": 0.2, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 20067.37, "finish_reason": "stop", "usage": {"prompt_tokens": 427, "completion_tokens": 927, "reasoning_tokens": 834, "total_tokens": 1354}}, "with-docs": {"answer": "Explanation: I compared the totals in the provided snowboarding medal leaders table. Kelly Clark has 3 total medals, which is more than any other athlete listed; the others shown have 2 or fewer.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2550.74, "finish_reason": "stop", "usage": {"prompt_tokens": 756, "completion_tokens": 74, "reasoning_tokens": 12, "total_tokens": 830}}, "with-structures": {"answer": "Explanation: I compared the total medal counts in the provided list of Olympic snowboarding medalists. Kelly Clark has 3 total medals, while all others listed have 2. Therefore, she has won the most medals.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2963.45, "finish_reason": "stop", "usage": {"prompt_tokens": 1434, "completion_tokens": 85, "reasoning_tokens": 20, "total_tokens": 1519}}, "agentic-a": {"answer": "Explanation: The entity structure in `structures/structures/entity/wiki_opentable_wikitq_00322__Athlete_leaderboard_and_extrema.entities.csv` lists each athlete’s `total_int` medals; Kelly Clark has 3, the highest total in the table.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 47764.34, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8764, "completion_tokens": 351, "total_tokens": 9115, "reasoning_tokens": 106}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The snowboarding medalist leaderboard in `structures/structures/entity/wiki_opentable_wikitq_00323__Athlete_medal_leaderboard_and_medal_breakdown.entities.csv` shows Kelly Clark with `total_int = 3`, and all other listed athletes with 2. This is reinforced by `wiki_opentable_wikitq_00320__Athlete_medal_leaderboard_and_key_comparisons.entities.csv`, which labels her “Highest total in this list.”\nExact Answer: Kelly Clark\nConfidence: 98%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 228107.8, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 50, "tool_call_counts": {"bash": 9, "read_file": 8}, "tokens": {"prompt_tokens": 38902, "completion_tokens": 1770, "total_tokens": 40672, "reasoning_tokens": 626}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_043618.txt` (original id `49254`), the “Athlete medal leaders” table for Olympic snowboarding medalists shows Kelly Clark with a total of 3 medals, higher than all others listed.\nExact Answer: Kelly Clark\nConfidence: 98%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 564116.14, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 15, "tool_call_counts": {"bash": 5}, "tokens": {"prompt_tokens": 9974, "completion_tokens": 509, "total_tokens": 10483, "reasoning_tokens": 201}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00323.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00323", "question": "Of the Olympic snowboarding medalists, who had the most bronze medals?", "dataset_origin": "WikiTQ", "original_table_id": "203-308", "sql": "SELECT Athlete FROM table_203_308 ORDER BY Bronze DESC LIMIT 1 ", "gold_answers": ["Kelly Clark"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I considered Olympic snowboarding medalists across all snowboarding events and compared how many bronze medals each athlete won. Mark McMorris won three Olympic bronze medals in snowboarding, which is the highest total for bronze medals in the sport.\nExact Answer: Mark McMorris\nConfidence: 93%", "pred_items": ["Mark McMorris"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 8980.35, "finish_reason": "stop", "usage": {"prompt_tokens": 425, "completion_tokens": 381, "reasoning_tokens": 309, "total_tokens": 806}}, "with-docs": {"answer": "Explanation: From the provided snowboarding medal leaders table, Kelly Clark has 2 bronze medals, which is more than any other listed Olympic snowboarding medalist. Others listed have at most 1 bronze or none.\nExact Answer: Kelly Clark\nConfidence: 96%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3020.75, "finish_reason": "stop", "usage": {"prompt_tokens": 754, "completion_tokens": 98, "reasoning_tokens": 34, "total_tokens": 852}}, "with-structures": {"answer": "Explanation: From the medal breakdown table, Kelly Clark has 2 bronze medals, Ross Powers has 1, and all others have 0. Therefore, Kelly Clark had the most bronze medals.\nExact Answer: Kelly Clark\nConfidence: 98%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2840.99, "finish_reason": "stop", "usage": {"prompt_tokens": 1473, "completion_tokens": 83, "reasoning_tokens": 23, "total_tokens": 1556}}, "agentic-a": {"answer": "Explanation: The entity table `structures/structures/entity/wiki_opentable_wikitq_00323__Athlete_medal_leaderboard_and_medal_breakdown.entities.csv` lists each snowboard medalist’s `bronze_int`; Kelly Clark has 2 bronze medals, more than anyone else in the file.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 45499.98, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 2}, "tokens": {"prompt_tokens": 8504, "completion_tokens": 307, "total_tokens": 8811, "reasoning_tokens": 119}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The structure files `structures/structures/_raw/wiki_opentable_wikitq_00317__Query_normalization_and_field_value_index.md` and `structures/structures/entity/wiki_opentable_wikitq_00317__Athlete_medal_leaders_lookup.entities.csv` both show the listed Olympic snowboarding medalists’ medal breakdowns; Kelly Clark has `Bronze = 2`, the highest bronze count.\nExact Answer: Kelly Clark\nConfidence: 95%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 267774.55, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 50, "tool_call_counts": {"bash": 12, "read_file": 3}, "tokens": {"prompt_tokens": 33247, "completion_tokens": 1815, "total_tokens": 35062, "reasoning_tokens": 762}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_043618.txt` (original id 49254), the “List of Olympic medalists in snowboarding - Statistics” table shows bronze-medal counts by athlete; Kelly Clark has 2 bronze medals, more than any other listed athlete.\nExact Answer: Kelly Clark\nConfidence: 97%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 2042221.64, "stop_reason": "end_turn", "n_turns": 9, "max_turns": 15, "tool_call_counts": {"bash": 18, "read_file": 1}, "tokens": {"prompt_tokens": 53912, "completion_tokens": 2038, "total_tokens": 55950, "reasoning_tokens": 1015}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00324.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00324", "question": "Which of the Olympic medalists in snowboarding has won only silver medals?", "dataset_origin": "WikiTQ", "original_table_id": "203-308", "sql": "SELECT Athlete FROM table_203_308 WHERE Gold = 0 AND Silver != 0 AND Bronze = 0 ", "gold_answers": ["Danny Kass"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I considered Olympic snowboard medalists across the Winter Games and selected those whose Olympic medal record consists only of silver medals, excluding anyone who has also won a gold or bronze in snowboarding.  \nExact Answer: Brigitte Köck; Nicolas Huet; Daniel Franck; Doriane Vidal; Danny Kass; Radoslav Židek; Amelie Kober; Peetu Piiroinen; Mike Robertson; Deborah Anthonioz; Tomoka Takeuchi; Lee Sang-ho; Selina Jörg; Liu Jiayu; Jarryd Hughes; Julia Pereira de Sousa; Kyle Mack; Tim Mastnak; Daniela Ulbing; Queralt Castellet; Eliot Grondin; Julia Marino; Kokomo Murase\nConfidence: 77%", "pred_items": ["Brigitte Köck", "Nicolas Huet", "Daniel Franck", "Doriane Vidal", "Danny Kass", "Radoslav Židek", "Amelie Kober", "Peetu Piiroinen", "Mike Robertson", "Deborah Anthonioz", "Tomoka Takeuchi", "Lee Sang-ho", "Selina Jörg", "Liu Jiayu", "Jarryd Hughes", "Julia Pereira de Sousa", "Kyle Mack", "Tim Mastnak", "Daniela Ulbing", "Queralt Castellet", "Eliot Grondin", "Julia Marino", "Kokomo Murase"], "metrics": {"precision": 0.0435, "recall": 1.0, "f1": 0.0833, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 76629.75, "finish_reason": "stop", "usage": {"prompt_tokens": 425, "completion_tokens": 3476, "reasoning_tokens": 3313, "total_tokens": 3901}}, "with-docs": {"answer": "Explanation: From the table, the only snowboard Olympic medalist with 0 gold, 2 silver, and 0 bronze—meaning all of their medals were silver—is Danny Kass.\nExact Answer: Danny Kass\nConfidence: 99%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3136.39, "finish_reason": "stop", "usage": {"prompt_tokens": 754, "completion_tokens": 95, "reasoning_tokens": 37, "total_tokens": 849}}, "with-structures": {"answer": "Explanation: I checked the listed snowboard Olympic medalists and looked for anyone whose medal counts are silver only, with zero gold and zero bronze. Danny Kass has 0 gold, 2 silver, and 0 bronze, so he is the only one who fits.\nExact Answer: Danny Kass\nConfidence: 98%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3825.11, "finish_reason": "stop", "usage": {"prompt_tokens": 1248, "completion_tokens": 124, "reasoning_tokens": 50, "total_tokens": 1372}}, "agentic-a": {"answer": "Explanation: The entity table `structures/entity/wiki_opentable_wikitq_00324__Athlete_medal_leaderboard_and_tie_map.entities.csv` shows each snowboard medalist’s gold, silver, and bronze counts; only Danny Kass has 0 gold, 2 silver, and 0 bronze. The same file’s quick-lookup metadata also states he is “silver only.”\nExact Answer: Danny Kass\nConfidence: 99%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 51379.06, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 10938, "completion_tokens": 437, "total_tokens": 11375, "reasoning_tokens": 167}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The snowboarding medal tables in `structures/structures/entity/wiki_opentable_wikitq_00323__Athlete_medal_leaderboard_and_medal_breakdown.entities.csv` and `structures/structures/entity/wiki_opentable_wikitq_00327__Athlete_medal_snapshot.entities.csv` show each athlete’s gold/silver/bronze totals; only Danny Kass has 0 gold, 2 silver, 0 bronze.\nExact Answer: Danny Kass\nConfidence: 98%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 158409.23, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 50, "tool_call_counts": {"bash": 5, "read_file": 6}, "tokens": {"prompt_tokens": 23473, "completion_tokens": 1176, "total_tokens": 24649, "reasoning_tokens": 450}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_043618.txt` (original id `49254`), the “List of Olympic medalists in snowboarding - Statistics” table shows medal counts by athlete; only Danny Kass has `Gold = 0`, `Silver = 2`, and `Bronze = 0`, meaning he won only silver medals.\nExact Answer: Danny Kass\nConfidence: 98%", "pred_items": ["Danny Kass"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1501067.51, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 15, "tool_call_counts": {"bash": 10, "read_file": 1}, "tokens": {"prompt_tokens": 22472, "completion_tokens": 1239, "total_tokens": 23711, "reasoning_tokens": 638}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00326.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00326", "question": "How many Olympic medalists in snowboarding have won at least 2 gold medals?", "dataset_origin": "WikiTQ", "original_table_id": "203-308", "sql": "SELECT COUNT ( * ) FROM table_203_308 WHERE Gold >= '2' ", "gold_answers": ["3"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: I counted Olympic snowboarding athletes who have won at least 2 gold medals across all Winter Olympics through Beijing 2022. The athletes meeting that threshold are: Philipp Schoch, Shaun White, Seth Wescott, Pierre Vaultier, Vic Wild, Jamie Anderson, Chloe Kim, Anna Gasser, and Lindsey Jacobellis. That makes 9 in total.\n\nExact Answer: 9\nConfidence: 84%", "pred_items": ["9"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 20874.14, "finish_reason": "stop", "usage": {"prompt_tokens": 427, "completion_tokens": 910, "reasoning_tokens": 814, "total_tokens": 1337}}, "with-docs": {"answer": "Explanation: From the table, the athletes with at least 2 gold medals are Philipp Schoch (2), Shaun White (2), and Seth Wescott (2). That makes 3 medalists.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3043.34, "finish_reason": "stop", "usage": {"prompt_tokens": 756, "completion_tokens": 90, "reasoning_tokens": 27, "total_tokens": 846}}, "with-structures": {"answer": "Explanation: From the listed snowboarding Olympic medalists, the athletes with at least 2 gold medals are Philipp Schoch, Shaun White, and Seth Wescott. That makes 3 medalists in total.\nExact Answer: 3\nConfidence: 98%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2800.73, "finish_reason": "stop", "usage": {"prompt_tokens": 1319, "completion_tokens": 90, "reasoning_tokens": 26, "total_tokens": 1409}}, "agentic-a": {"answer": "Explanation: The entity file `structures/structures/entity/wiki_opentable_wikitq_00326__Athlete_leaderboard_and_key_rankings.entities.csv` lists each snowboard Olympic medalist with `gold_int`; counting athletes with `gold_int >= 2` gives Philipp Schoch, Shaun White, and Seth Wescott.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 46617.1, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 3}, "tokens": {"prompt_tokens": 8341, "completion_tokens": 399, "total_tokens": 8740, "reasoning_tokens": 159}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: In `structures/structures/_raw/wiki_opentable_wikitq_00319__Olympics_span_repeat_medalist_index.md`, the medal profiles show only Philipp Schoch, Seth Wescott, and Shaun White with `2G`. The companion file `structures/structures/_raw/wiki_opentable_wikitq_00321__Olympic_span_repeat_medalist_index.md` confirms these are repeat Olympic medalists in snowboarding.\nExact Answer: 3\nConfidence: 95%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 171231.89, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 50, "tool_call_counts": {"bash": 8, "read_file": 3}, "tokens": {"prompt_tokens": 21971, "completion_tokens": 1536, "total_tokens": 23507, "reasoning_tokens": 624}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_043618.txt` (original id 49254, “List of Olympic medalists in snowboarding - Statistics”), the athlete medal leaders table shows exactly three snowboarders with at least 2 gold medals: Philipp Schoch, Shaun White, and Seth Wescott.\nExact Answer: 3\nConfidence: 98%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1300811.61, "stop_reason": "end_turn", "n_turns": 5, "max_turns": 15, "tool_call_counts": {"bash": 4, "read_file": 1}, "tokens": {"prompt_tokens": 19560, "completion_tokens": 1187, "total_tokens": 20747, "reasoning_tokens": 718}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00327.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00327", "question": "From the list of Olympic medalists in snowboarding, who has won the most bronze medals?", "dataset_origin": "WikiTQ", "original_table_id": "203-308", "sql": "SELECT Athlete FROM table_203_308 ORDER BY Bronze DESC LIMIT 1 ", "gold_answers": ["Kelly Clark"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Reviewing Olympic snowboarding medalists, the athlete with the highest number of bronze medals is Mark McMorris, who won bronze in men’s slopestyle (2014), men’s big air (2018), and men’s slopestyle again (2022), for a total of three bronze medals.\nExact Answer: Mark McMorris\nConfidence: 96%", "pred_items": ["Mark McMorris"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 9038.75, "finish_reason": "stop", "usage": {"prompt_tokens": 429, "completion_tokens": 353, "reasoning_tokens": 266, "total_tokens": 782}}, "with-docs": {"answer": "Explanation: I checked the bronze medal counts in the provided snowboarding medal leaders table. The highest bronze total listed is 2, earned by Kelly Clark. Everyone else listed has either 0 or 1 bronze medal.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2732.45, "finish_reason": "stop", "usage": {"prompt_tokens": 758, "completion_tokens": 79, "reasoning_tokens": 14, "total_tokens": 837}}, "with-structures": {"answer": "Explanation: I compared the bronze medal counts in the listed snowboarding Olympic medalists. Kelly Clark has 2 bronze medals, Ross Powers has 1, and all others have 0. Therefore, Kelly Clark has the most bronze medals.\nExact Answer: Kelly Clark\nConfidence: 98%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3294.3, "finish_reason": "stop", "usage": {"prompt_tokens": 1377, "completion_tokens": 93, "reasoning_tokens": 24, "total_tokens": 1470}}, "agentic-a": {"answer": "Explanation: The entity file `structures/entity/wiki_opentable_wikitq_00327__Athlete_medal_snapshot.entities.csv` lists each snowboard medalist’s `bronze_int` total; Kelly Clark has 2 bronze medals, the highest in the file.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 38442.93, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 2}, "tokens": {"prompt_tokens": 7948, "completion_tokens": 279, "total_tokens": 8227, "reasoning_tokens": 112}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The entity tables `structures/structures/entity/wiki_opentable_wikitq_00326__Athlete_leaderboard_and_key_rankings.entities.csv` and `structures/structures/entity/wiki_opentable_wikitq_00321__Athlete_leaderboard_and_tie_structure.entities.csv` list each snowboard Olympic medalist’s gold/silver/bronze counts. In both, Kelly Clark has 2 bronze medals, more than anyone else listed.\nExact Answer: Kelly Clark\nConfidence: 99%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 279402.34, "stop_reason": "end_turn", "n_turns": 11, "max_turns": 50, "tool_call_counts": {"bash": 18, "read_file": 7}, "tokens": {"prompt_tokens": 49903, "completion_tokens": 2326, "total_tokens": 52229, "reasoning_tokens": 663}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_043618.txt` (original id 49254, “List of Olympic medalists in snowboarding - Statistics”), the “Athlete medal leaders” table shows bronze counts, and Kelly Clark has 2 bronze medals, more than any other listed athlete.\nExact Answer: Kelly Clark\nConfidence: 98%", "pred_items": ["Kelly Clark"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 987094.75, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 4, "read_file": 1}, "tokens": {"prompt_tokens": 14390, "completion_tokens": 781, "total_tokens": 15171, "reasoning_tokens": 469}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00349.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00349", "question": "How many of the coastal provinces are smaller than 10,000 km2?", "dataset_origin": "WikiTQ", "original_table_id": "204-778", "sql": "SELECT COUNT ( County ) FROM table_204_778 WHERE Area_km2 < '10000' ", "gold_answers": ["3"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Interpreting this as Sri Lanka’s provinces, the coastal provinces are Northern, Eastern, North Western, Western, and Southern. Each of these has an area under 10,000 km², so the count is 5.\nExact Answer: 5\nConfidence: 58%", "pred_items": ["5"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 19124.94, "finish_reason": "stop", "usage": {"prompt_tokens": 426, "completion_tokens": 886, "reasoning_tokens": 816, "total_tokens": 1312}}, "with-docs": {"answer": "Explanation: From the table, the Coast counties with area under 10,000 km² are Mombasa (212.5), Kwale (8270.3), and Lamu (6497.7). That makes 3 in total.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 4984.24, "finish_reason": "stop", "usage": {"prompt_tokens": 654, "completion_tokens": 164, "reasoning_tokens": 92, "total_tokens": 818}}, "with-structures": {"answer": "Explanation: From the table, the coastal counties with area smaller than 10,000 km2 are Mombasa (212.5), Kwale (8,270.3), and Lamu (6,497.7). That makes 3 in total.\nExact Answer: 3\nConfidence: 99%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 5165.82, "finish_reason": "stop", "usage": {"prompt_tokens": 1474, "completion_tokens": 166, "reasoning_tokens": 91, "total_tokens": 1640}}, "agentic-a": {"answer": "Explanation: The entity table `structures/entity/wiki_opentable_wikitq_00349__Core_county_fact_index.entities.csv` lists the Coast counties and their areas; three have area below 10,000 km²: Mombasa (212.5), Kwale (8,270.3), and Lamu (6,497.7). The same count is corroborated by the area ranking in `structures/_raw/wiki_opentable_wikitq_00349__Comparative_rankings_and_derived_metrics.md`.\nExact Answer: 3\nConfidence: 98%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 65368.78, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 9273, "completion_tokens": 596, "total_tokens": 9869, "reasoning_tokens": 265}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: I used `structures/structures/entity/wiki_opentable_wikitq_00356__County_field_lookup.entities.csv` for the county areas in the former province Coast, and confirmed the area ranking in `structures/structures/_raw/wiki_opentable_wikitq_00356__Rankings_and_extrema_index.md`. The entries under 10,000 km² are Mombasa, Kwale, and Lamu.\nExact Answer: 3\nConfidence: 96%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 188820.73, "stop_reason": "end_turn", "n_turns": 6, "max_turns": 50, "tool_call_counts": {"bash": 6, "read_file": 3}, "tokens": {"prompt_tokens": 23247, "completion_tokens": 1521, "total_tokens": 24768, "reasoning_tokens": 860}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In `structures/doc_048359.txt` (original id `53520`, “Coast Province - Population data”), the areas listed for the Coast entries show Mombasa (212.5 km2), Kwale (8270.3 km2), and Lamu (6497.7 km2) are below 10,000 km2, while Kilifi, Tana River, and Taita-Taveta are above it.\nExact Answer: 3\nConfidence: 98%", "pred_items": ["3"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 2580387.03, "stop_reason": "end_turn", "n_turns": 11, "max_turns": 15, "tool_call_counts": {"bash": 23}, "tokens": {"prompt_tokens": 118527, "completion_tokens": 3483, "total_tokens": 122010, "reasoning_tokens": 1823}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}

compare/records/wikitq-00350.json

@@ -0,0 +1 @@
+{"qid": "wikitq-00350", "question": "Which county has the largest area among the coastal provinces?", "dataset_origin": "WikiTQ", "original_table_id": "204-778", "sql": "SELECT County FROM table_204_778 ORDER BY Area_km2 DESC LIMIT 1 ", "gold_answers": ["Tana River"], "configs_order": ["closed-book", "with-docs", "with-structures", "agentic-a", "agentic-b-corpus", "agentic-c-rawtext"], "configs": {"closed-book": {"answer": "Explanation: Interpreting this as referring to Ireland, its provinces are coastal, and the county with the largest area is County Cork. Therefore, among the coastal provinces, the largest-area county is County Cork.\nExact Answer: County Cork\nConfidence: 61%", "pred_items": ["County Cork"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "closed-book", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 7417.9, "finish_reason": "stop", "usage": {"prompt_tokens": 421, "completion_tokens": 276, "reasoning_tokens": 213, "total_tokens": 697}}, "with-docs": {"answer": "Explanation: I compared the Area_km2 values for the Coast former province counties: Mombasa 212.5, Kwale 8270.3, Kilifi 12245.9, Tana River 35375.8, Lamu 6497.7, and Taita-Taveta 17083.9. The largest area is Tana River.\nExact Answer: Tana River\nConfidence: 99%", "pred_items": ["Tana River"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 3722.43, "finish_reason": "stop", "usage": {"prompt_tokens": 649, "completion_tokens": 116, "reasoning_tokens": 16, "total_tokens": 765}}, "with-structures": {"answer": "Explanation: From the table, Tana River has the largest area at 35,375.8 km², which is the highest among the listed coastal counties.\nExact Answer: Tana River\nConfidence: 99%", "pred_items": ["Tana River"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "with-docs", "system_prompt_file": "prompts/answer_list_aware_long_monaco_semicolon.md", "max_completion_tokens": 8192, "latency_ms": 2847.97, "finish_reason": "stop", "usage": {"prompt_tokens": 1394, "completion_tokens": 66, "reasoning_tokens": 11, "total_tokens": 1460}}, "agentic-a": {"answer": "Explanation: The county areas in `structures/structures/entity/wiki_opentable_wikitq_00350__County_quick_reference_table.entities.csv` show Tana River at 35,375.8 km², and `structures/structures/_raw/wiki_opentable_wikitq_00350__Ranked_statistics_index.md` ranks it first by area among the coastal counties.\nExact Answer: Tana River\nConfidence: 99%", "pred_items": ["Tana River"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt.md", "max_completion_tokens": 50000, "latency_ms": 49634.74, "stop_reason": "end_turn", "n_turns": 4, "max_turns": 15, "tool_call_counts": {"bash": 1, "read_file": 4}, "tokens": {"prompt_tokens": 9038, "completion_tokens": 426, "total_tokens": 9464, "reasoning_tokens": 143}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-b-corpus": {"answer": "Explanation: The Kenya coast-county structures `structures/structures/_raw/wiki_opentable_wikitq_00351__Ranked_county_comparisons.md` and `structures/structures/entity/wiki_opentable_wikitq_00349__Core_county_fact_index.entities.csv` show the former Coast counties and rank area, with Tana River listed as largest at 35,375.8 km².\nExact Answer: Tana River County\nConfidence: 98%", "pred_items": ["Tana River County"], "metrics": {"precision": 0.0, "recall": 0.0, "f1": 0.0, "em": 0}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_corpus.md", "max_completion_tokens": 50000, "latency_ms": 91496.3, "stop_reason": "end_turn", "n_turns": 3, "max_turns": 50, "tool_call_counts": {"bash": 2, "python": 1, "read_file": 2}, "tokens": {"prompt_tokens": 15105, "completion_tokens": 885, "total_tokens": 15990, "reasoning_tokens": 303}, "finish_reasons": ["tool_calls", "tool_calls", "stop"], "timeout_retries": 0}, "agentic-c-rawtext": {"answer": "Explanation: In doc_048359.txt (original id 53520, “Coast Province - Population data”), the coastal counties’ areas are listed, and Tana River has the largest area at 35,375.8 km².\nExact Answer: Tana River\nConfidence: 98%", "pred_items": ["Tana River"], "metrics": {"precision": 1.0, "recall": 1.0, "f1": 1.0, "em": 1}, "model": "prod-gpt-54-reasoning", "mode": "agentic", "system_prompt_file": "aml_qa_jobs/agentic_answer/system_prompt_rawtext_v2_wiki_opentable.md", "max_completion_tokens": 50000, "latency_ms": 1911125.05, "stop_reason": "end_turn", "n_turns": 8, "max_turns": 15, "tool_call_counts": {"bash": 20}, "tokens": {"prompt_tokens": 38008, "completion_tokens": 2224, "total_tokens": 40232, "reasoning_tokens": 1256}, "finish_reasons": ["tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "tool_calls", "stop"], "timeout_retries": 0}}}