README.md

---
title: MEBench Viewer
emoji: 🧩
colorFrom: green
colorTo: blue
sdk: static
pinned: false
license: apache-2.0
---

# MEBench Viewer

Static viewer for [**MEBench**](https://github.com/tl2309/MEBench)
([paper](https://arxiv.org/abs/2502.18993), EMNLP 2025;
[HF dataset](https://huggingface.co/datasets/Tim999999/MEBench)) — a
cross-document **multi-entity** QA benchmark. Questions ask a model to retrieve,
consolidate and reason over facts scattered across many entities that belong to
a shared *topic* (e.g. the members of the Ivy League), across 8 question types
in 3 reasoning categories.

Two tabs:

- **📚 Corpus** — the entity documents. A dropdown selects a document **by
  title** (filter by title text and/or topic, or use prev/next). Each shows the
  entity's topic membership(s), a link to Wikipedia, and its intro text.
- **❓ Eval** — the questions, with a **split selector** (train / test /
  single_test). A dropdown selects a **question** (filter by topic and/or
  question type, or search). Each shows the question, the gold answer
  (train answers are SQL-result tables; `∅` marks empty/null results), the
  **supporting documents** (clickable entity chips that jump to the Corpus tab;
  train questions range over a whole topic, so they link to *all* documents of
  that topic), and metadata (qid, reasoning category, type, topic, relation
  edge, extracted properties, hops/class, and the ground-truth SQL for train).

## Splits

| Split | Questions | Notes |
|---|---|---|
| `train` | 5,764 | includes the ground-truth `SQL`; answers are SQL-result tables |
| `test` | 1,373 | per-entity questions (single/multi-hop), NL answers |
| `single_test` | 5,940 | single-entity questions, NL answers |

All 7 topics are university associations: Ivy League, Group of Eight, Russell
Group, C9 League, Universitas 21, Association of American Universities, and the
European University Association.

## The corpus

MEBench ships **questions only** — in the benchmark, each "document" is an
entity's Wikipedia intro paragraph, generated live by the pipeline and not
included in the dataset files. This viewer therefore **reconstructs** the corpus
from the entities that actually appear in the dataset (the `Entity` field of the
`test` + `single_test` splits, which together cover all 7 topics): **172 unique
entities**, each fetched from Wikipedia (`extracts`, intro only) and tagged with
the topic(s) it appears under. 3 / 172 entities had no English-Wikipedia intro
and show a placeholder. It is a representative subset of each topic's full
membership, not the exhaustive corpus.

## Data layout

| File | Contents |
|---|---|
| `sets.json` | split manifest (per-split question count, topics, type histogram) |
| `corpus.json` | entity **index**: `{title, topics, size, wiki_url, file}` (shared across splits) |
| `corpus/*.txt` | one Wikipedia-intro **shard** per entity (lazy-loaded) |
| `eval_<split>.json` | questions: `{qid, topic, type, category, question, answer, edge, properties, supporting_titles, topic_level, sql?, class?, hops?, entity?}` |

## Rebuilding the data

```bash
python scripts/build_data.py \
    --data-dir /mnt/ramdisk/blobstore/timchen0618/data/mebench
```

Reads `train/test/single_test.jsonl`, (re)fetches Wikipedia intros (cached to
`<data-dir>/wiki_cache.jsonl`), and rewrites `corpus.json` + `corpus/` +
`eval_<split>.json` + `sets.json`. The raw MEBench JSONLs (gated on Hugging
Face) are **not** committed here.

## Local dev

```bash
python -m http.server 8000   # then open http://localhost:8000/
```

Serves statically — no build step. Push to the HF Space remote to deploy.

Dataset license: apache-2.0.