README.md

---
title: Open-WikiTable Viewer
emoji: 📑
colorFrom: yellow
colorTo: red
sdk: static
pinned: false
---

# Open-WikiTable Viewer

Interactive viewer for the **test split** of [Open-WikiTable](https://github.com/sean0042/Open-WikiTable) (Kweon et al., ACL Findings 2023) — the first open-domain QA dataset that requires complex reasoning over tables, built on WikiSQL + WikiTableQuestions.

For each of the **6,602** test questions you can inspect:

- The decontextualized question + dataset of origin (`WikiSQL` / `WikiTQ`)
- The gold answer(s)
- The reference SQL query
- The three buckets of candidate splitted-table chunks: **hard-positive** (full match), **positive** (partial match), **negative** (BM25-similar distractors) — each rendered as a real table with page/section/caption breadcrumb

## View modes

The topbar's view-mode toggle switches between nine parallel views (full
parity with `monaco-benchmark-viewer` where data exists for `wiki_opentable`):

- **📑 Dataset** (default) — browse all 6,602 test questions with their gold answer, reference SQL, and the three buckets of candidate table chunks (hard-positive / positive / negative).
- **🔧 Unified eval** — the same 500-qid subset used by every downstream eval, fetched from the cross-dataset *unified* bundle (`$DATA_ROOT/eval/wiki_opentable/unified/test_with_chunks.unified.jsonl`). Shows the gold question + gold answers + every gold supporting doc rendered as markdown (the wiki_opentable docs are already markdown tables).
- **🧬 Eval Structures v0** — the typed structure docs (e.g. `structures/entity/…`, `structures/_raw/…`) that the structures-based eval consumes, fetched from `test_with_structures.unified.jsonl`. Grouped by subtype in the UI.
- **🏗 E2E Structures** — for each of the 500 test qids, the supporting doc(s) + every structure file the **information-scaffolds e2e pipeline** extracted from each doc on-the-fly (`scaffolds_dir/<shape>/<doc_id>.{csv,json,jsonl}`). Five possible shapes per doc: 🪪 `entity_fact_records` · 🕒 `chronology_and_timeline_indexes` · 💬 `claim_and_theme_summaries` · ❓ `qa_shortcuts_and_templates` · 🕸 `relation_graphs_and_mappings`. Each structure card renders CSV → mini-table and JSONL → keyed mini-cards so the underlying records are immediately legible. Click a question in the sidebar to inspect that qid's supporting doc + structures.
- **🛠 Structure per q** *(formerly "Baseline A")* — the standalone `agentic_answer` run on the 500-qid subset with **per-qid scaffolds** (`outputs/agentic_wiki_opentable/cell4_agentic_a.response.jsonl`). Side-by-side gold + parsed prediction + canonical per-qid F1 / strict-EM / P / R + the full agent loop (events). Mean F1 ≈ 78.95.
- **🛠 Structure per ds** *(formerly "Baseline B")* — the parallel run with a **flat per-dataset structure corpus** (`cell5_agentic_b_corpus.response.jsonl`). Mean F1 ≈ 74.84.
- **🛠 DCI** *(formerly "Baseline C")* — the parallel run with a **flat per-dataset rawtext corpus** (`cell6_agentic_c_rawtext.response.jsonl`). Mean F1 ≈ 89.22.
- **📊 Compare** — per-qid side-by-side grid of all **7 configs** (closed-book, with-docs, with-structures, structure per q, structure per ds, DCI, **e2e**). Each card shows the model's answer, the parsed prediction items, and the canonical **local set-based F1 / EM / P / R** (no LLM-judge file exists for `wiki_opentable` — the deterministic `wiki_opentable_adapter.py` is the source of truth). The highest-F1 cell per qid is flagged with 🏆. The "e2e" cell is recomputed from the `trajectories_e2e/records/` shards on disk (same scorer, no external eval file). Sidebar rows show a 7-bar sparkline so you can spot disagreement at a glance; an "all EM / all F1=0 / split" filter narrows to qids by agreement.
- **🤖 E2E Trajectories** — for the 500 wikitq-* test questions of the **information-scaffolds e2e pipeline** run on `wiki_opentable`, inspect each agent trajectory side-by-side with the gold answer, the parsed prediction, and **per-qid F1 / strict-EM / precision / recall** (computed with the canonical `wiki_opentable_adapter.py` set-based scoring + normalization). Sidebar rows are colour-coded by F1 bucket; filter pills narrow by `stop_reason` (`end_turn` / `error` / `max_turns`) or by score bucket (EM=1 / 0&lt;F1&lt;1 / F1=0).

The four trajectory tabs share the same sidebar filter pills (stop_reason
+ score bucket); `Dataset`, `Unified eval`, `Eval Structures v0`,
`E2E Structures`, and `Compare` share the WikiSQL / WikiTQ dataset-origin
filter pills. The Compare tab adds its own *agreement* filter (`all` /
`all EM` / `all F1=0` / `split`) so you can drill into the rows where the
cells disagree.

> The Structures v0 / v1 / v2 generator tabs from the MoNaCo viewer are
> intentionally absent: `wiki_opentable` was never paired with a
> structures-generation AML response JSONL (only the e2e-pipeline
> hierarchical structure dirs exist, which **🏗 E2E Structures** surfaces).

Deep-link via the URL hash:
`#dataset:wikisql-00116` ·
`#unified:wikitq-00030` ·
`#eval_structures:wikitq-00030` ·
`#e2e_structures:wikitq-01887` ·
`#trajectories:wikitq-00030` ·
`#trajectories_corpus:wikitq-00030` ·
`#trajectories_rawtext:wikitq-00030` ·
`#compare:wikitq-00030` ·
`#e2e:wikitq-00030`.
Legacy `#wikisql-00116` links resolve to dataset mode.

## Local dev

```bash
python -m http.server 8000   # → http://localhost:8000/
```

## Rebuilding the shards

If the source dataset is re-extracted, re-run:

```bash
python scripts/build_dataset.py \
  --in-split  /path/to/Open_WikiTable/data/test.json \
  --in-tables /path/to/Open_WikiTable/data/splitted_tables.json \
  --out-dir   .
```

This writes `index.json` + `records/<qid>.json` shards (no LFS needed — each shard is a few KB).

## Rebuilding the E2E trajectory shards

If the e2e pipeline is re-run on `wiki_opentable`, regenerate the trajectory view with:

```bash
python scripts/build_trajectories_e2e.py \
  --predictions   <run>/named-outputs/predictions/predictions \
  --gold          $DATA_ROOT/eval/wiki_opentable/raw/test_with_chunks.jsonl \
  --eval-results  <run>/named-outputs/eval_results/eval_results.json \
  --out-dir       trajectories_e2e \
  --label         "your run label"
```

The script inlines the canonical scoring (`wiki_opentable_adapter.py` + `_parse_exact_answer.py`) so it has no dependency on the `information-scaffolds` repo. It writes `trajectories_e2e/index.json` + `trajectories_e2e/records/<qid>.json`; per-event tool-result content is truncated at 8 KB to keep each shard browser-friendly.

## Rebuilding the other 5 shard collections

```bash
# 🔧 Unified eval (default --src points at $DATA_ROOT/eval/wiki_opentable/unified/test_with_chunks.unified.jsonl)
python scripts/build_unified.py

# 🧬 Eval Structures v0 (default --src points at .../test_with_structures.unified.jsonl)
python scripts/build_eval_structures.py

# 🏗 E2E Structures (per-qid · supporting doc + every e2e-pipeline structure file from scaffolds_dir/)
# Defaults point at the wiki-opentable-fullcorpus-…-20260622 run; pass
# --scaffolds-dir to point at a different e2e run's named-outputs/scaffolds_dir.
python scripts/build_e2e_structures.py

# 🛠 Trajectory (Baseline A — per-qid scaffolds, cell4)
python scripts/build_trajectories.py \
  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell4_agentic_a.response.jsonl \
  --out         trajectories \
  --label       "Baseline A — per-qid scaffolds (cell4_agentic_a)"

# 🛠 Structure per q (formerly Baseline A — per-qid scaffolds, cell4)
python scripts/build_trajectories.py \
  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell4_agentic_a.response.jsonl \
  --out         trajectories \
  --label       "structure per q (formerly Baseline A · per-qid scaffolds · cell4_agentic_a)"

# 🛠 Structure per ds (formerly Baseline B — flat structure corpus, cell5)
python scripts/build_trajectories.py \
  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell5_agentic_b_corpus.response.jsonl \
  --out         trajectories_corpus \
  --label       "structure per ds (formerly Baseline B · flat structure corpus · cell5_agentic_b_corpus)"

# 🛠 DCI (formerly Baseline C — flat rawtext corpus, cell6)
python scripts/build_trajectories.py \
  --predictions /home/azureuser/projects/information-scaffolds/outputs/agentic_wiki_opentable/cell6_agentic_c_rawtext.response.jsonl \
  --out         trajectories_rawtext \
  --label       "DCI (formerly Baseline C · flat rawtext corpus · cell6_agentic_c_rawtext)"

# 📊 Compare (per-qid grid across all 7 configs; local set-based F1/EM)
# Reads trajectories_e2e/records/ on disk for the "e2e" cell — make sure
# trajectories_e2e/ has been built first (see "Rebuilding the E2E trajectory shards").
python scripts/build_compare.py
```

Each script writes `<out>/index.json` + `<out>/records/<qid>.json` and
clears stale shards from a previous build so the index and on-disk shards
never drift. The three trajectory scripts share the canonical
semicolon-split parser + set-based F1/EM scoring with
`build_trajectories_e2e.py` (inlined — no `information-scaffolds`
dependency). `build_compare.py` reuses the same parser + scorer (also
inlined) so per-cell numbers in the Compare tab match the trajectory tabs
and `evaluation/CANONICAL.md` exactly.