---
title: MoNaCo Dev Viewer
emoji: 🧩
colorFrom: pink
colorTo: indigo
sdk: static
pinned: false
license: apache-2.0
---

# 🧩 MoNaCo Dev Viewer

A per-qid web viewer for the **100-question dev subset** of the
[MoNaCo benchmark](https://github.com/tomerwolgithub/monaco)
([paper](https://arxiv.org/abs/2506.12523)), mirroring the layout of the
sibling [`open-wikitable-smoke-viewer`](https://huggingface.co/spaces/timchen0618/open-wikitable-smoke-viewer).

Live: **https://timchen0618-monaco-dev-viewer.static.hf.space**

## 12 tabs

| # | tab | contents | mean LLM judge score |
|--:|---|---|--:|
| 1 | 📑 **Dataset** | raw monaco question + gold answer + flat list of supporting docs | — |
| 2 | 🔧 **Unified eval** | cross-dataset bundle: question + gold answers + gold docs (markdown) | — |
| 3 | 🛠 **Structure per q** (was Baseline A) | agentic run over per-qid scaffolds (cell 4) | 90.48 |
| 4 | 🛠 **Structure per ds** (was Baseline B) | agentic run over flat structure corpus (cell 5) | 87.94 |
| 5 | 🛠 **DCI** (was Baseline C) | agentic run over rawtext corpus (cell 6 v2 rerun) | 82.96 |
| 6 | 🤖 **e2e v0** | e2e_pipeline run with on-the-fly per-shape extraction | 82.01 |
| 7 | 🤖 **e2e v2** | e2e_pipeline run v2 | 80.94 |
| 8 | 🤖 **e2e v3** | e2e_pipeline run v3 | 81.32 |
| 9 | 🏗 **e2e structures v0** | for each supporting doc, the 5-shape structures produced by e2e v0 | — |
| 10 | 🏗 **e2e structures v2** | same for the v2 pipeline | — |
| 11 | 🏗 **e2e structures v3** | same for the v3 pipeline | — |
| 12 | 📊 **Compare** | 9 cells side-by-side per qid (all 6 baselines + 3 e2e runs) | — |

Scoring column is the **LLM judge_score** (0.0-1.0) from the AML
`submit_judge_responses_aml.py` protocol with `--deterministic-extract`
auto-enabled for `--dataset monaco_dev` — this is the canonical
scoring protocol for MoNaCo. `correct = 1 iff judge_score ≥ 0.99`.

Mean judge_scores are over **all 100 qids**: qids missing a judge row
(judge dropouts) count as `judge_score = 0`. If you only want the mean
over judged qids, use the numbers in
`information-scaffolds/data_creation/monaco_dev/README.md`.

## Repo layout

```
monaco-dev-viewer/
├── index.html                # 12-tab topbar + sidebar + main pane
├── viewer.js                 # single-file frontend (~1250 lines)
├── style.css
├── README.md                 # this file
├── index.json                # 📑 Dataset  — flat array of {qid, question, ...}
├── records/<qid>.json        # 📑 Dataset per-qid raw docs
├── unified/                  # 🔧 Unified eval
│   ├── index.json
│   └── records/<qid>.json
├── trajectories/             # 🛠 Structure per q (cell 4)
│   ├── index.json  (wrapped: {meta, rows})
│   └── records/<qid>.json
├── trajectories_corpus/      # 🛠 Structure per ds (cell 5)
├── trajectories_rawtext/     # 🛠 DCI (cell 6 v2)
├── trajectories_e2e_v0/      # 🤖 e2e v0
├── trajectories_e2e_v2/      # 🤖 e2e v2
├── trajectories_e2e_v3/      # 🤖 e2e v3
├── e2e_structures_v0/        # 🏗 e2e structures v0
├── e2e_structures_v2/        # 🏗 e2e structures v2
├── e2e_structures_v3/        # 🏗 e2e structures v3
├── compare/                  # 📊 Compare
│   ├── index.json  (per-qid {cells:[…], question, gold_answers_length})
│   └── records/<qid>.json
└── scripts/                  # builders (see below)
    ├── build_dataset.py
    ├── build_unified.py
    ├── build_trajectories.py
    ├── build_e2e_structures.py
    └── build_compare.py
```

All shards are small (~5–200 KB); the site is served as static files
(no backend). Deep-linkable via URL hash, e.g.
`#trajectories:42`, `#compare:1048`, `#e2e_structures_v0:12`.

## Reproduction

Run from this directory. Adjust paths if you're mirroring the layout on
a different machine.

```bash
cd data_visualizer/monaco-dev-viewer

# 1. Dataset  — reads $DATA_ROOT/eval/monaco_dev/raw/test_with_chunks.jsonl
python scripts/build_dataset.py

# 2. Unified eval  — reads $DATA_ROOT/eval/monaco_dev/unified/test_with_chunks.unified.jsonl
python scripts/build_unified.py

# 3-5. Baseline trajectories (needs pred + judged JSONLs under information-scaffolds/outputs/monaco_dev)
python scripts/build_trajectories.py \
  --predictions ~/projects/information-scaffolds/outputs/monaco_dev/baselines/cell4_agentic_a/named-outputs/response/response \
  --judged      ~/projects/information-scaffolds/outputs/monaco_dev/judges/c4_agentic_a/named-outputs/judged/judged \
  --out         trajectories \
  --label       "structure per q (monaco_dev · Baseline A · cell4_agentic_a)"

python scripts/build_trajectories.py \
  --predictions ~/projects/information-scaffolds/outputs/monaco_dev/baselines/cell5_agentic_b/named-outputs/response/response \
  --judged      ~/projects/information-scaffolds/outputs/monaco_dev/judges/c5_agentic_b/named-outputs/judged/judged \
  --out         trajectories_corpus \
  --label       "structure per ds (monaco_dev · Baseline B · cell5_agentic_b)"

python scripts/build_trajectories.py \
  --predictions ~/projects/information-scaffolds/outputs/cell6_v2_reruns/monaco_dev_cell6_v2/named-outputs/response/response \
  --judged      ~/projects/information-scaffolds/outputs/monaco_dev/judges/c6_agentic_c_v2/named-outputs/judged/judged \
  --out         trajectories_rawtext \
  --label       "DCI (monaco_dev · Baseline C · cell6_agentic_c v2)"

# 6-8. e2e trajectories (predictions + judged live under run_logs/)
for v in v0 v2 v3; do
  python scripts/build_trajectories.py \
    --predictions ~/run_logs/e2e-monaco_dev-comparison/outputs/$v/named-outputs/predictions/predictions \
    --judged      ~/run_logs/e2e-monaco_dev-comparison/judge_outputs/$v/named-outputs/judged/judged \
    --out         trajectories_e2e_$v \
    --label       "e2e $v (monaco_dev)"
done

# 9-11. e2e structures (per-qid supporting docs + 5-shape scaffolds)
for v in v0 v2 v3; do
  python scripts/build_e2e_structures.py \
    --scaffolds-dir ~/run_logs/e2e-monaco_dev-comparison/outputs/$v/named-outputs/scaffolds_dir \
    --out           e2e_structures_$v \
    --label         "e2e $v (monaco_dev)"
done

# 12. Compare — reads all 9 (pred, judged) pairs and joins by qid
python scripts/build_compare.py
```

## Local preview

```bash
python -m http.server 8877
# then open http://127.0.0.1:8877/
```

## Deploy

```bash
HF_TOKEN="$HF_TOKEN" python ~/.copilot/skills/hf-space-push/push_hf_space.py \
    timchen0618/monaco-dev-viewer . \
    --commit-msg "Monaco dev viewer — 12 tabs"
```

The Space is `sdk: static`, so it redeploys instantly on push (no build step).

## Data notes

- **Cell 1 (closed-book)** has no `outputs/monaco_dev/baselines/cell1_closedbook/`
  dir; the raw response lives at
  `outputs/monaco_dev_smoke/sincere_pepper_d54pl76cjj/named-outputs/response/response`.
  The compare builder points there directly.
- **Cell 6 (DCI)** — `outputs/monaco_dev/baselines/cell6_agentic_c/` is the
  **broken v1** run (100/100 `missing_structures`). The good v2 rerun is at
  `outputs/cell6_v2_reruns/monaco_dev_cell6_v2/`, judged under
  `outputs/monaco_dev/judges/c6_agentic_c_v2/`.
- **Judge dropouts**: cell 1 = 4/100 missing · cell 2 = 4 · cell 3 = 7 ·
  cell 4 = 0 · cell 5 = 1 · cell 6 = 4 · e2e v0/v2/v3 = 0. Missing rows
  score 0.
- **Non-agentic cells 1-3** have no `events` in the response.jsonl (single
  LLM call). The trajectory-record renderer collapses to just
  system prompt + user prompt + model answer + score.

## See also

- Full-benchmark viewer: `data_visualizer/monaco-benchmark-viewer/`
  (~1315 questions, docker-backed).
- Sibling for open-wikitable dev: `data_visualizer/open-wikitable-smoke-viewer/`.
- Authoritative reproduction guide with per-cell judge means:
  `information-scaffolds/data_creation/monaco_dev/README.md`.