institutional-proceedings.md

---
name: institutional-proceedings
description: "Use the Institutional Proceedings family of Hugging Face datasets from the Institutional Data Initiative — each instance covers one institution's governing-body proceedings as enriched, structured data. Currently covers the Proceedings of the Board of Regents of the University of Michigan: layout-aware meeting text, per-meeting metadata, extracted governance events, reasoning-derived meeting/year/decade summaries, and human-readable EPUBs (one per meeting). Use when a task involves this corpus: reading or analyzing historical meeting text, tracking governance events (leadership, buildings, degree programs, fundraising, org units, strategic plans), summarizing meetings across time, or producing a readable EPUB of a meeting."
---

# Institutional Proceedings — datasets

Institutional Proceedings is a growing family of Hugging Face datasets from the [Institutional Data Initiative](https://institutional.org/): each instance takes one institution's governing-body proceedings, re-OCR's them with a layout-aware model, segments them into individual meetings, and enriches them. New institutions are added over time — browse the [Institutional Proceedings collection](https://huggingface.co/collections/institutional/institutional-proceedings) to see what's available; a given instance may publish a different set of repos.

**This skill covers the instance published so far — the Proceedings of the Board of Regents of the University of Michigan:**

| Repo | Type | Contents |
| --- | --- | --- |
| `institutional/institutional-proceedings-um` | tabular | configs `core` (1,755 meetings) and `summaries` (3,916 rows) |
| `institutional/institutional-proceedings-um-epubs` | files | one EPUB 3 per meeting at `epubs/<meeting_id>.epub` |

Both are **gated** (see [Auth](#auth--gating)).

**Two ideas that drive every decision:**

1. **`meeting_id_gen` is the join key.** One meeting = one id (e.g. `march_1858`). It ties `core` rows, `summaries` rows, and the EPUB filename together. Relate them with a join or a filename lookup — **not** a re-download.
2. **`summaries` are precomputed reasoning outputs — there is no runnable model here.** Both repos ship *data only*. To pick a "model," read the set you want: filter `summaries` by `model_tag_gen` (`qwen35` vs `gptoss20b`). Read `summary_gen` / `reasoning_gen`; don't try to regenerate them.

## Pick the right asset

| I need to… | Use |
| --- | --- |
| Full meeting text + metadata (date, location, attendees) | `-um`, config `core` → `text_gen` + `*_gen` fields |
| A specific governance event (leadership, buildings, funding…) | `core` → `events_gen.<category>` |
| A reasoning-derived summary of a meeting / year / decade | config `summaries` → filter `summary_level_gen`, pick model via `model_tag_gen` |
| The model's reasoning / chain-of-thought for a summary | `summaries` → `reasoning_gen`, `cot_trace_gen` (meeting rows only) |
| Cheaply skip very long meetings before loading text | `core` → `n_tokens_gen` (o200k_base token count) |
| A human-readable EPUB of a meeting | `-um-epubs` → `epubs/<meeting_id>.epub` |
| Browse / search / filter rows without downloading | Dataset Viewer HTTP API (see [below](#lightweight-no-download-browsing)) |

## Datasets

Load the tabular repo with `datasets`; it is gated, so pass `token=True` (see [Auth](#auth--gating)).

### config `core` — one row per meeting
Key fields: `meeting_id_gen` (join key), `volume_id_src`, `volume_src` (TEI bibliographic struct), `year_gen`/`month_gen`/`date_gen`, `location_gen`, `presiding_officer_gen`, `attendees_gen`/`absent_gen` (list of `{name, role, presiding, is_regent}`), `text_gen` (layout-aware text; Markdown markup, tables as HTML), `elements_gen` (dots.ocr layout elements), `n_tokens_gen`, and `events_gen` — a struct with one list per governance-event category: `degree_programs`, `leadership_transitions`, `org_units`, `fundraising`, `buildings`, `strategic_plans`.

```python
from datasets import load_dataset

core = load_dataset("institutional/institutional-proceedings-um", "core", split="train", token=True)
row = core[0]
print(row["meeting_id_gen"], row["date_gen"], row["n_tokens_gen"])
print(row["text_gen"][:500])
print(row["events_gen"]["leadership_transitions"])
```

### config `summaries` — reasoning-derived, long form
Key fields: `summary_level_gen` (`meeting` | `year` | `decade`), `model_tag_gen` (`qwen35` | `gptoss20b`), `meeting_id_gen` (set on `meeting` rows; joins to `core`), `year_gen`/`month_gen`/`decade_gen`, `summary_gen`, `reasoning_gen`, `cot_trace_gen` (meeting rows only), `source_meeting_ids_gen` / `source_years_gen` (roll-up provenance for year/decade rows), `model_gen` (full HF model id).

```python
summaries = load_dataset("institutional/institutional-proceedings-um", "summaries", split="train", token=True)
# one model's meeting-level summaries:
meeting_summaries = summaries.filter(
    lambda r: r["summary_level_gen"] == "meeting" and r["model_tag_gen"] == "qwen35"
)
```

### `-um-epubs` — one EPUB per meeting
Files live at `epubs/<meeting_id>.epub`, where `<meeting_id>` == `core.meeting_id_gen`. No dataset viewer; fetch files directly.

```python
from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="institutional/institutional-proceedings-um-epubs",
    filename="epubs/march_1858.epub",
    repo_type="dataset",
    token=True,
)
```

## How they connect

**Field suffixes** tell you a value's provenance:

| Suffix | Meaning |
| --- | --- |
| `_src` | From the source collection (original bibliographic / OCR data) |
| `_gen` | Generated by the IDI pipeline |
| `_exp` | Experimental / exploratory generation |

**Join `core` ↔ `summaries` on `meeting_id_gen`, then map straight to the EPUB filename** — the id is the same everywhere:

```python
core = load_dataset("institutional/institutional-proceedings-um", "core", split="train", token=True)
summaries = load_dataset("institutional/institutional-proceedings-um", "summaries", split="train", token=True)

mid = core[0]["meeting_id_gen"]                      # e.g. "adjourned_august_1852"

# its summaries (one per model_tag_gen):
its_summaries = [r for r in summaries
                 if r["summary_level_gen"] == "meeting" and r["meeting_id_gen"] == mid]

# its readable EPUB — same id, no lookup table needed:
epub = hf_hub_download(
    repo_id="institutional/institutional-proceedings-um-epubs",
    filename=f"epubs/{mid}.epub", repo_type="dataset", token=True,
)
```

For year/decade summaries, follow `source_meeting_ids_gen` / `source_years_gen` back to the meetings (and thus `core` rows / EPUBs) that fed the synthesis.

## Lightweight, no-download browsing

To page, search, or filter the tabular repo without loading it, use the Dataset Viewer HTTP API (`config` is `core` or `summaries`; send your token). Handy for locating a `meeting_id_gen` before pulling text or an EPUB.

```bash
curl -H "Authorization: Bearer $HF_TOKEN" \
  "https://datasets-server.huggingface.co/search?dataset=institutional/institutional-proceedings-um&config=core&split=train&query=medical%20school&length=10"
```

## Auth & gating

The datasets are gated, so you **always need an `HF_TOKEN`** whose account has clicked **"Agree and access"** on each dataset page — otherwise loads return **401**.

```bash
export HF_TOKEN="hf_..."
```

Or load it from a local `.env` (this repo already depends on `python-dotenv`):

```python
from dotenv import load_dotenv
load_dotenv()  # sets HF_TOKEN in the environment
```

Then pass `token=True` to `load_dataset` / `hf_hub_download` (reads `HF_TOKEN`), or `token=os.environ["HF_TOKEN"]`.

## Dependencies

```bash
pip install datasets huggingface_hub python-dotenv
```