Spaces:
Running
Running
| title: MoNaCo Dev Viewer | |
| emoji: 🧩 | |
| colorFrom: pink | |
| colorTo: indigo | |
| sdk: static | |
| pinned: false | |
| license: apache-2.0 | |
| # 🧩 MoNaCo Dev Viewer | |
| A per-qid web viewer for the **100-question dev subset** of the | |
| [MoNaCo benchmark](https://github.com/tomerwolgithub/monaco) | |
| ([paper](https://arxiv.org/abs/2506.12523)), mirroring the layout of the | |
| sibling [`open-wikitable-smoke-viewer`](https://huggingface.co/spaces/timchen0618/open-wikitable-smoke-viewer). | |
| Live: **https://timchen0618-monaco-dev-viewer.static.hf.space** | |
| ## 12 tabs | |
| | # | tab | contents | mean LLM judge score | | |
| |--:|---|---|--:| | |
| | 1 | 📑 **Dataset** | raw monaco question + gold answer + flat list of supporting docs | — | | |
| | 2 | 🔧 **Unified eval** | cross-dataset bundle: question + gold answers + gold docs (markdown) | — | | |
| | 3 | 🛠 **Structure per q** (was Baseline A) | agentic run over per-qid scaffolds (cell 4) | 90.48 | | |
| | 4 | 🛠 **Structure per ds** (was Baseline B) | agentic run over flat structure corpus (cell 5) | 87.94 | | |
| | 5 | 🛠 **DCI** (was Baseline C) | agentic run over rawtext corpus (cell 6 v2 rerun) | 82.96 | | |
| | 6 | 🤖 **e2e v0** | e2e_pipeline run with on-the-fly per-shape extraction | 82.01 | | |
| | 7 | 🤖 **e2e v2** | e2e_pipeline run v2 | 80.94 | | |
| | 8 | 🤖 **e2e v3** | e2e_pipeline run v3 | 81.32 | | |
| | 9 | 🏗 **e2e structures v0** | for each supporting doc, the 5-shape structures produced by e2e v0 | — | | |
| | 10 | 🏗 **e2e structures v2** | same for the v2 pipeline | — | | |
| | 11 | 🏗 **e2e structures v3** | same for the v3 pipeline | — | | |
| | 12 | 📊 **Compare** | 9 cells side-by-side per qid (all 6 baselines + 3 e2e runs) | — | | |
| Scoring column is the **LLM judge_score** (0.0-1.0) from the AML | |
| `submit_judge_responses_aml.py` protocol with `--deterministic-extract` | |
| auto-enabled for `--dataset monaco_dev` — this is the canonical | |
| scoring protocol for MoNaCo. `correct = 1 iff judge_score ≥ 0.99`. | |
| Mean judge_scores are over **all 100 qids**: qids missing a judge row | |
| (judge dropouts) count as `judge_score = 0`. If you only want the mean | |
| over judged qids, use the numbers in | |
| `information-scaffolds/data_creation/monaco_dev/README.md`. | |
| ## Repo layout | |
| ``` | |
| monaco-dev-viewer/ | |
| ├── index.html # 12-tab topbar + sidebar + main pane | |
| ├── viewer.js # single-file frontend (~1250 lines) | |
| ├── style.css | |
| ├── README.md # this file | |
| ├── index.json # 📑 Dataset — flat array of {qid, question, ...} | |
| ├── records/<qid>.json # 📑 Dataset per-qid raw docs | |
| ├── unified/ # 🔧 Unified eval | |
| │ ├── index.json | |
| │ └── records/<qid>.json | |
| ├── trajectories/ # 🛠 Structure per q (cell 4) | |
| │ ├── index.json (wrapped: {meta, rows}) | |
| │ └── records/<qid>.json | |
| ├── trajectories_corpus/ # 🛠 Structure per ds (cell 5) | |
| ├── trajectories_rawtext/ # 🛠 DCI (cell 6 v2) | |
| ├── trajectories_e2e_v0/ # 🤖 e2e v0 | |
| ├── trajectories_e2e_v2/ # 🤖 e2e v2 | |
| ├── trajectories_e2e_v3/ # 🤖 e2e v3 | |
| ├── e2e_structures_v0/ # 🏗 e2e structures v0 | |
| ├── e2e_structures_v2/ # 🏗 e2e structures v2 | |
| ├── e2e_structures_v3/ # 🏗 e2e structures v3 | |
| ├── compare/ # 📊 Compare | |
| │ ├── index.json (per-qid {cells:[…], question, gold_answers_length}) | |
| │ └── records/<qid>.json | |
| └── scripts/ # builders (see below) | |
| ├── build_dataset.py | |
| ├── build_unified.py | |
| ├── build_trajectories.py | |
| ├── build_e2e_structures.py | |
| └── build_compare.py | |
| ``` | |
| All shards are small (~5–200 KB); the site is served as static files | |
| (no backend). Deep-linkable via URL hash, e.g. | |
| `#trajectories:42`, `#compare:1048`, `#e2e_structures_v0:12`. | |
| ## Reproduction | |
| Run from this directory. Adjust paths if you're mirroring the layout on | |
| a different machine. | |
| ```bash | |
| cd data_visualizer/monaco-dev-viewer | |
| # 1. Dataset — reads $DATA_ROOT/eval/monaco_dev/raw/test_with_chunks.jsonl | |
| python scripts/build_dataset.py | |
| # 2. Unified eval — reads $DATA_ROOT/eval/monaco_dev/unified/test_with_chunks.unified.jsonl | |
| python scripts/build_unified.py | |
| # 3-5. Baseline trajectories (needs pred + judged JSONLs under information-scaffolds/outputs/monaco_dev) | |
| python scripts/build_trajectories.py \ | |
| --predictions ~/projects/information-scaffolds/outputs/monaco_dev/baselines/cell4_agentic_a/named-outputs/response/response \ | |
| --judged ~/projects/information-scaffolds/outputs/monaco_dev/judges/c4_agentic_a/named-outputs/judged/judged \ | |
| --out trajectories \ | |
| --label "structure per q (monaco_dev · Baseline A · cell4_agentic_a)" | |
| python scripts/build_trajectories.py \ | |
| --predictions ~/projects/information-scaffolds/outputs/monaco_dev/baselines/cell5_agentic_b/named-outputs/response/response \ | |
| --judged ~/projects/information-scaffolds/outputs/monaco_dev/judges/c5_agentic_b/named-outputs/judged/judged \ | |
| --out trajectories_corpus \ | |
| --label "structure per ds (monaco_dev · Baseline B · cell5_agentic_b)" | |
| python scripts/build_trajectories.py \ | |
| --predictions ~/projects/information-scaffolds/outputs/cell6_v2_reruns/monaco_dev_cell6_v2/named-outputs/response/response \ | |
| --judged ~/projects/information-scaffolds/outputs/monaco_dev/judges/c6_agentic_c_v2/named-outputs/judged/judged \ | |
| --out trajectories_rawtext \ | |
| --label "DCI (monaco_dev · Baseline C · cell6_agentic_c v2)" | |
| # 6-8. e2e trajectories (predictions + judged live under run_logs/) | |
| for v in v0 v2 v3; do | |
| python scripts/build_trajectories.py \ | |
| --predictions ~/run_logs/e2e-monaco_dev-comparison/outputs/$v/named-outputs/predictions/predictions \ | |
| --judged ~/run_logs/e2e-monaco_dev-comparison/judge_outputs/$v/named-outputs/judged/judged \ | |
| --out trajectories_e2e_$v \ | |
| --label "e2e $v (monaco_dev)" | |
| done | |
| # 9-11. e2e structures (per-qid supporting docs + 5-shape scaffolds) | |
| for v in v0 v2 v3; do | |
| python scripts/build_e2e_structures.py \ | |
| --scaffolds-dir ~/run_logs/e2e-monaco_dev-comparison/outputs/$v/named-outputs/scaffolds_dir \ | |
| --out e2e_structures_$v \ | |
| --label "e2e $v (monaco_dev)" | |
| done | |
| # 12. Compare — reads all 9 (pred, judged) pairs and joins by qid | |
| python scripts/build_compare.py | |
| ``` | |
| ## Local preview | |
| ```bash | |
| python -m http.server 8877 | |
| # then open http://127.0.0.1:8877/ | |
| ``` | |
| ## Deploy | |
| ```bash | |
| HF_TOKEN="$HF_TOKEN" python ~/.copilot/skills/hf-space-push/push_hf_space.py \ | |
| timchen0618/monaco-dev-viewer . \ | |
| --commit-msg "Monaco dev viewer — 12 tabs" | |
| ``` | |
| The Space is `sdk: static`, so it redeploys instantly on push (no build step). | |
| ## Data notes | |
| - **Cell 1 (closed-book)** has no `outputs/monaco_dev/baselines/cell1_closedbook/` | |
| dir; the raw response lives at | |
| `outputs/monaco_dev_smoke/sincere_pepper_d54pl76cjj/named-outputs/response/response`. | |
| The compare builder points there directly. | |
| - **Cell 6 (DCI)** — `outputs/monaco_dev/baselines/cell6_agentic_c/` is the | |
| **broken v1** run (100/100 `missing_structures`). The good v2 rerun is at | |
| `outputs/cell6_v2_reruns/monaco_dev_cell6_v2/`, judged under | |
| `outputs/monaco_dev/judges/c6_agentic_c_v2/`. | |
| - **Judge dropouts**: cell 1 = 4/100 missing · cell 2 = 4 · cell 3 = 7 · | |
| cell 4 = 0 · cell 5 = 1 · cell 6 = 4 · e2e v0/v2/v3 = 0. Missing rows | |
| score 0. | |
| - **Non-agentic cells 1-3** have no `events` in the response.jsonl (single | |
| LLM call). The trajectory-record renderer collapses to just | |
| system prompt + user prompt + model answer + score. | |
| ## See also | |
| - Full-benchmark viewer: `data_visualizer/monaco-benchmark-viewer/` | |
| (~1315 questions, docker-backed). | |
| - Sibling for open-wikitable dev: `data_visualizer/open-wikitable-smoke-viewer/`. | |
| - Authoritative reproduction guide with per-cell judge means: | |
| `information-scaffolds/data_creation/monaco_dev/README.md`. | |