README.md

metadata

title: Open-WikiTable Viewer
emoji: 📑
colorFrom: yellow
colorTo: red
sdk: static
pinned: false

Open-WikiTable Viewer

Interactive viewer for the test split of Open-WikiTable (Kweon et al., ACL Findings 2023) — the first open-domain QA dataset that requires complex reasoning over tables, built on WikiSQL + WikiTableQuestions.

For each of the 6,602 test questions you can inspect:

• The decontextualized question + dataset of origin (WikiSQL / WikiTQ)
• The gold answer(s)
• The reference SQL query
• The three buckets of candidate splitted-table chunks: hard-positive (full match), positive (partial match), negative (BM25-similar distractors) — each rendered as a real table with page/section/caption breadcrumb

View modes

The topbar's view-mode toggle switches between nine parallel views (full parity with monaco-benchmark-viewer where data exists for wiki_opentable):

• 📑 Dataset (default) — browse all 6,602 test questions with their gold answer, reference SQL, and the three buckets of candidate table chunks (hard-positive / positive / negative).
• 🔧 Unified eval — the same 500-qid subset used by every downstream eval, fetched from the cross-dataset unified bundle ($DATA_ROOT/eval/wiki_opentable/unified/test_with_chunks.unified.jsonl). Shows the gold question + gold answers + every gold supporting doc rendered as markdown (the wiki_opentable docs are already markdown tables).
• 🧬 Eval Structures v0 — the typed structure docs (e.g. structures/entity/…, structures/_raw/…) that the structures-based eval consumes, fetched from test_with_structures.unified.jsonl. Grouped by subtype in the UI.
• 🏗 E2E Structures — for each of the 500 test qids, the supporting doc(s) + every structure file the information-scaffolds e2e pipeline extracted from each doc on-the-fly (scaffolds_dir/<shape>/<doc_id>.{csv,json,jsonl}). Five possible shapes per doc: 🪪 entity_fact_records · 🕒 chronology_and_timeline_indexes · 💬 claim_and_theme_summaries · ❓ qa_shortcuts_and_templates · 🕸 relation_graphs_and_mappings. Each structure card renders CSV → mini-table and JSONL → keyed mini-cards so the underlying records are immediately legible. Click a question in the sidebar to inspect that qid's supporting doc + structures.
• 🛠 Structure per q (formerly "Baseline A") — the standalone agentic_answer run on the 500-qid subset with per-qid scaffolds (outputs/agentic_wiki_opentable/cell4_agentic_a.response.jsonl). Side-by-side gold + parsed prediction + canonical per-qid F1 / strict-EM / P / R + the full agent loop (events). Mean F1 ≈ 78.95.
• 🛠 Structure per ds (formerly "Baseline B") — the parallel run with a flat per-dataset structure corpus (cell5_agentic_b_corpus.response.jsonl). Mean F1 ≈ 74.84.
• 🛠 DCI (formerly "Baseline C") — the parallel run with a flat per-dataset rawtext corpus (cell6_agentic_c_rawtext.response.jsonl). Mean F1 ≈ 89.22.
• 📊 Compare — per-qid side-by-side grid of all 7 configs (closed-book, with-docs, with-structures, structure per q, structure per ds, DCI, e2e). Each card shows the model's answer, the parsed prediction items, and the canonical local set-based F1 / EM / P / R (no LLM-judge file exists for wiki_opentable — the deterministic wiki_opentable_adapter.py is the source of truth). The highest-F1 cell per qid is flagged with 🏆. The "e2e" cell is recomputed from the trajectories_e2e/records/ shards on disk (same scorer, no external eval file). Sidebar rows show a 7-bar sparkline so you can spot disagreement at a glance; an "all EM / all F1=0 / split" filter narrows to qids by agreement.
• 🤖 E2E Trajectories — for the 500 wikitq-* test questions of the information-scaffolds e2e pipeline run on wiki_opentable, inspect each agent trajectory side-by-side with the gold answer, the parsed prediction, and per-qid F1 / strict-EM / precision / recall (computed with the canonical wiki_opentable_adapter.py set-based scoring + normalization). Sidebar rows are colour-coded by F1 bucket; filter pills narrow by stop_reason (end_turn / error / max_turns) or by score bucket (EM=1 / 0<F1<1 / F1=0).

The four trajectory tabs share the same sidebar filter pills (stop_reason

• score bucket); Dataset, Unified eval, Eval Structures v0, E2E Structures, and Compare share the WikiSQL / WikiTQ dataset-origin filter pills. The Compare tab adds its own agreement filter (all / all EM / all F1=0 / split) so you can drill into the rows where the cells disagree.

The Structures v0 / v1 / v2 generator tabs from the MoNaCo viewer are intentionally absent: wiki_opentable was never paired with a structures-generation AML response JSONL (only the e2e-pipeline hierarchical structure dirs exist, which 🏗 E2E Structures surfaces).

Deep-link via the URL hash: #dataset:wikisql-00116 · #unified:wikitq-00030 · #eval_structures:wikitq-00030 · #e2e_structures:wikitq-01887 · #trajectories:wikitq-00030 · #trajectories_corpus:wikitq-00030 · #trajectories_rawtext:wikitq-00030 · #compare:wikitq-00030 · #e2e:wikitq-00030. Legacy #wikisql-00116 links resolve to dataset mode.

Local dev

python -m http.server 8000   # → http://localhost:8000/

Rebuilding the shards

If the source dataset is re-extracted, re-run:

python scripts/build_dataset.py \
  --in-split  /path/to/Open_WikiTable/data/test.json \
  --in-tables /path/to/Open_WikiTable/data/splitted_tables.json \
  --out-dir   .

This writes index.json + records/<qid>.json shards (no LFS needed — each shard is a few KB).

Rebuilding the E2E trajectory shards

If the e2e pipeline is re-run on wiki_opentable, regenerate the trajectory view with:

python scripts/build_trajectories_e2e.py \
  --predictions   <run>/named-outputs/predictions/predictions \
  --gold          $DATA_ROOT/eval/wiki_opentable/raw/test_with_chunks.jsonl \
  --eval-results  <run>/named-outputs/eval_results/eval_results.json \
  --out-dir       trajectories_e2e \
  --label         "your run label"

The script inlines the canonical scoring (wiki_opentable_adapter.py + _parse_exact_answer.py) so it has no dependency on the information-scaffolds repo. It writes trajectories_e2e/index.json + trajectories_e2e/records/<qid>.json; per-event tool-result content is truncated at 8 KB to keep each shard browser-friendly.

Rebuilding the other 5 shard collections

# 🔧 Unified eval (default --src points at $DATA_ROOT/eval/wiki_opentable/unified/test_with_chunks.unified.jsonl)
python scripts/build_unified.py

# 🧬 Eval Structures v0 (default --src points at .../test_with_structures.unified.jsonl)
python scripts/build_eval_structures.py

# 🏗 E2E Structures (per-qid · supporting doc + every e2e-pipeline structure file from scaffolds_dir/)
# Defaults point at the wiki-opentable-fullcorpus-…-20260622 run; pass
# --scaffolds-dir to point at a different e2e run's named-outputs/scaffolds_dir.
python scripts/build_e2e_structures.py

# 🛠 Trajectory (Baseline A — per-qid scaffolds, cell4)
python scripts/build_trajectories.py \
  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell4_agentic_a.response.jsonl \
  --out         trajectories \
  --label       "Baseline A — per-qid scaffolds (cell4_agentic_a)"

# 🛠 Structure per q (formerly Baseline A — per-qid scaffolds, cell4)
python scripts/build_trajectories.py \
  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell4_agentic_a.response.jsonl \
  --out         trajectories \
  --label       "structure per q (formerly Baseline A · per-qid scaffolds · cell4_agentic_a)"

# 🛠 Structure per ds (formerly Baseline B — flat structure corpus, cell5)
python scripts/build_trajectories.py \
  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell5_agentic_b_corpus.response.jsonl \
  --out         trajectories_corpus \
  --label       "structure per ds (formerly Baseline B · flat structure corpus · cell5_agentic_b_corpus)"

# 🛠 DCI (formerly Baseline C — flat rawtext corpus, cell6)
python scripts/build_trajectories.py \
  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell6_agentic_c_rawtext.response.jsonl \
  --out         trajectories_rawtext \
  --label       "DCI (formerly Baseline C · flat rawtext corpus · cell6_agentic_c_rawtext)"

# 📊 Compare (per-qid grid across all 7 configs; local set-based F1/EM)
# Reads trajectories_e2e/records/ on disk for the "e2e" cell — make sure
# trajectories_e2e/ has been built first (see "Rebuilding the E2E trajectory shards").
python scripts/build_compare.py

Each script writes <out>/index.json + <out>/records/<qid>.json and clears stale shards from a previous build so the index and on-disk shards never drift. The three trajectory scripts share the canonical semicolon-split parser + set-based F1/EM scoring with build_trajectories_e2e.py (inlined — no information-scaffolds dependency). build_compare.py reuses the same parser + scorer (also inlined) so per-cell numbers in the Compare tab match the trajectory tabs and evaluation/CANONICAL.md exactly.