Spaces:

timchen0618
/

monaco-dev-viewer

Running

App Files Files Community

monaco-dev-viewer / README.md

timchen0618

Monaco dev viewer — initial 12-tab launch

61b5f50 verified 1 day ago

preview code

Raw

History Blame Contribute Delete

8.02 kB

metadata

title: MoNaCo Dev Viewer
emoji: 🧩
colorFrom: pink
colorTo: indigo
sdk: static
pinned: false
license: apache-2.0

🧩 MoNaCo Dev Viewer

A per-qid web viewer for the 100-question dev subset of the MoNaCo benchmark (paper), mirroring the layout of the sibling open-wikitable-smoke-viewer.

Live: https://timchen0618-monaco-dev-viewer.static.hf.space

12 tabs

#	tab	contents	mean LLM judge score
1	📑 Dataset	raw monaco question + gold answer + flat list of supporting docs	—
2	🔧 Unified eval	cross-dataset bundle: question + gold answers + gold docs (markdown)	—
3	🛠 Structure per q (was Baseline A)	agentic run over per-qid scaffolds (cell 4)	90.48
4	🛠 Structure per ds (was Baseline B)	agentic run over flat structure corpus (cell 5)	87.94
5	🛠 DCI (was Baseline C)	agentic run over rawtext corpus (cell 6 v2 rerun)	82.96
6	🤖 e2e v0	e2e_pipeline run with on-the-fly per-shape extraction	82.01
7	🤖 e2e v2	e2e_pipeline run v2	80.94
8	🤖 e2e v3	e2e_pipeline run v3	81.32
9	🏗 e2e structures v0	for each supporting doc, the 5-shape structures produced by e2e v0	—
10	🏗 e2e structures v2	same for the v2 pipeline	—
11	🏗 e2e structures v3	same for the v3 pipeline	—
12	📊 Compare	9 cells side-by-side per qid (all 6 baselines + 3 e2e runs)	—

Scoring column is the LLM judge_score (0.0-1.0) from the AML submit_judge_responses_aml.py protocol with --deterministic-extract auto-enabled for --dataset monaco_dev — this is the canonical scoring protocol for MoNaCo. correct = 1 iff judge_score ≥ 0.99.

Mean judge_scores are over all 100 qids: qids missing a judge row (judge dropouts) count as judge_score = 0. If you only want the mean over judged qids, use the numbers in information-scaffolds/data_creation/monaco_dev/README.md.

Repo layout

monaco-dev-viewer/
├── index.html                # 12-tab topbar + sidebar + main pane
├── viewer.js                 # single-file frontend (~1250 lines)
├── style.css
├── README.md                 # this file
├── index.json                # 📑 Dataset  — flat array of {qid, question, ...}
├── records/<qid>.json        # 📑 Dataset per-qid raw docs
├── unified/                  # 🔧 Unified eval
│   ├── index.json
│   └── records/<qid>.json
├── trajectories/             # 🛠 Structure per q (cell 4)
│   ├── index.json  (wrapped: {meta, rows})
│   └── records/<qid>.json
├── trajectories_corpus/      # 🛠 Structure per ds (cell 5)
├── trajectories_rawtext/     # 🛠 DCI (cell 6 v2)
├── trajectories_e2e_v0/      # 🤖 e2e v0
├── trajectories_e2e_v2/      # 🤖 e2e v2
├── trajectories_e2e_v3/      # 🤖 e2e v3
├── e2e_structures_v0/        # 🏗 e2e structures v0
├── e2e_structures_v2/        # 🏗 e2e structures v2
├── e2e_structures_v3/        # 🏗 e2e structures v3
├── compare/                  # 📊 Compare
│   ├── index.json  (per-qid {cells:[…], question, gold_answers_length})
│   └── records/<qid>.json
└── scripts/                  # builders (see below)
    ├── build_dataset.py
    ├── build_unified.py
    ├── build_trajectories.py
    ├── build_e2e_structures.py
    └── build_compare.py

All shards are small (~5–200 KB); the site is served as static files (no backend). Deep-linkable via URL hash, e.g. #trajectories:42, #compare:1048, #e2e_structures_v0:12.

Reproduction

Run from this directory. Adjust paths if you're mirroring the layout on a different machine.

cd data_visualizer/monaco-dev-viewer

# 1. Dataset  — reads $DATA_ROOT/eval/monaco_dev/raw/test_with_chunks.jsonl
python scripts/build_dataset.py

# 2. Unified eval  — reads $DATA_ROOT/eval/monaco_dev/unified/test_with_chunks.unified.jsonl
python scripts/build_unified.py

# 3-5. Baseline trajectories (needs pred + judged JSONLs under information-scaffolds/outputs/monaco_dev)
python scripts/build_trajectories.py \
  --predictions ~/projects/information-scaffolds/outputs/monaco_dev/baselines/cell4_agentic_a/named-outputs/response/response \
  --judged      ~/projects/information-scaffolds/outputs/monaco_dev/judges/c4_agentic_a/named-outputs/judged/judged \
  --out         trajectories \
  --label       "structure per q (monaco_dev · Baseline A · cell4_agentic_a)"

python scripts/build_trajectories.py \
  --predictions ~/projects/information-scaffolds/outputs/monaco_dev/baselines/cell5_agentic_b/named-outputs/response/response \
  --judged      ~/projects/information-scaffolds/outputs/monaco_dev/judges/c5_agentic_b/named-outputs/judged/judged \
  --out         trajectories_corpus \
  --label       "structure per ds (monaco_dev · Baseline B · cell5_agentic_b)"

python scripts/build_trajectories.py \
  --predictions ~/projects/information-scaffolds/outputs/cell6_v2_reruns/monaco_dev_cell6_v2/named-outputs/response/response \
  --judged      ~/projects/information-scaffolds/outputs/monaco_dev/judges/c6_agentic_c_v2/named-outputs/judged/judged \
  --out         trajectories_rawtext \
  --label       "DCI (monaco_dev · Baseline C · cell6_agentic_c v2)"

# 6-8. e2e trajectories (predictions + judged live under run_logs/)
for v in v0 v2 v3; do
  python scripts/build_trajectories.py \
    --predictions ~/run_logs/e2e-monaco_dev-comparison/outputs/$v/named-outputs/predictions/predictions \
    --judged      ~/run_logs/e2e-monaco_dev-comparison/judge_outputs/$v/named-outputs/judged/judged \
    --out         trajectories_e2e_$v \
    --label       "e2e $v (monaco_dev)"
done

# 9-11. e2e structures (per-qid supporting docs + 5-shape scaffolds)
for v in v0 v2 v3; do
  python scripts/build_e2e_structures.py \
    --scaffolds-dir ~/run_logs/e2e-monaco_dev-comparison/outputs/$v/named-outputs/scaffolds_dir \
    --out           e2e_structures_$v \
    --label         "e2e $v (monaco_dev)"
done

# 12. Compare — reads all 9 (pred, judged) pairs and joins by qid
python scripts/build_compare.py

Local preview

python -m http.server 8877
# then open http://127.0.0.1:8877/

Deploy

HF_TOKEN="$HF_TOKEN" python ~/.copilot/skills/hf-space-push/push_hf_space.py \
    timchen0618/monaco-dev-viewer . \
    --commit-msg "Monaco dev viewer — 12 tabs"

The Space is sdk: static, so it redeploys instantly on push (no build step).

Data notes

Cell 1 (closed-book) has no outputs/monaco_dev/baselines/cell1_closedbook/ dir; the raw response lives at outputs/monaco_dev_smoke/sincere_pepper_d54pl76cjj/named-outputs/response/response. The compare builder points there directly.
Cell 6 (DCI) — outputs/monaco_dev/baselines/cell6_agentic_c/ is the broken v1 run (100/100 missing_structures). The good v2 rerun is at outputs/cell6_v2_reruns/monaco_dev_cell6_v2/, judged under outputs/monaco_dev/judges/c6_agentic_c_v2/.
Judge dropouts: cell 1 = 4/100 missing · cell 2 = 4 · cell 3 = 7 · cell 4 = 0 · cell 5 = 1 · cell 6 = 4 · e2e v0/v2/v3 = 0. Missing rows score 0.
Non-agentic cells 1-3 have no events in the response.jsonl (single LLM call). The trajectory-record renderer collapses to just system prompt + user prompt + model answer + score.