institutional-proceedings.md

metadata

name: institutional-proceedings
description: >-
  Use the Institutional Proceedings family of Hugging Face datasets from the
  Institutional Data Initiative — each instance covers one institution's
  governing-body proceedings as enriched, structured data. Currently covers the
  Proceedings of the Board of Regents of the University of Michigan:
  layout-aware meeting text, per-meeting metadata, extracted governance events,
  reasoning-derived meeting/year/decade summaries, and human-readable EPUBs (one
  per meeting). Use when a task involves this corpus: reading or analyzing
  historical meeting text, tracking governance events (leadership, buildings,
  degree programs, fundraising, org units, strategic plans), summarizing
  meetings across time, or producing a readable EPUB of a meeting.

Institutional Proceedings — datasets

Institutional Proceedings is a growing family of Hugging Face datasets from the Institutional Data Initiative: each instance takes one institution's governing-body proceedings, re-OCR's them with a layout-aware model, segments them into individual meetings, and enriches them. New institutions are added over time — browse the Institutional Proceedings collection to see what's available; a given instance may publish a different set of repos.

This skill covers the instance published so far — the Proceedings of the Board of Regents of the University of Michigan:

Repo	Type	Contents
institutional/institutional-proceedings-um	tabular	configs core (1,755 meetings) and summaries (3,916 rows)
institutional/institutional-proceedings-um-epubs	files	one EPUB 3 per meeting at epubs/<meeting_id>.epub

Both are gated (see Auth).

Two ideas that drive every decision:

1. meeting_id_gen is the join key. One meeting = one id (e.g. march_1858). It ties core rows, summaries rows, and the EPUB filename together. Relate them with a join or a filename lookup — not a re-download.
2. summaries are precomputed reasoning outputs — there is no runnable model here. Both repos ship data only. To pick a "model," read the set you want: filter summaries by model_tag_gen (qwen35 vs gptoss20b). Read summary_gen / reasoning_gen; don't try to regenerate them.

Pick the right asset

I need to…	Use
Full meeting text + metadata (date, location, attendees)	-um, config core → text_gen + *_gen fields
A specific governance event (leadership, buildings, funding…)	core → events_gen.<category>
A reasoning-derived summary of a meeting / year / decade	config summaries → filter summary_level_gen, pick model via model_tag_gen
The model's reasoning / chain-of-thought for a summary	summaries → reasoning_gen, cot_trace_gen (meeting rows only)
Cheaply skip very long meetings before loading text	core → n_tokens_gen (o200k_base token count)
A human-readable EPUB of a meeting	-um-epubs → epubs/<meeting_id>.epub
Browse / search / filter rows without downloading	Dataset Viewer HTTP API (see below)

Datasets

Load the tabular repo with datasets; it is gated, so pass token=True (see Auth).

config core — one row per meeting

Key fields: meeting_id_gen (join key), volume_id_src, volume_src (TEI bibliographic struct), year_gen/month_gen/date_gen, location_gen, presiding_officer_gen, attendees_gen/absent_gen (list of {name, role, presiding, is_regent}), text_gen (layout-aware text; Markdown markup, tables as HTML), elements_gen (dots.ocr layout elements), n_tokens_gen, and events_gen — a struct with one list per governance-event category: degree_programs, leadership_transitions, org_units, fundraising, buildings, strategic_plans.

from datasets import load_dataset

core = load_dataset("institutional/institutional-proceedings-um", "core", split="train", token=True)
row = core[0]
print(row["meeting_id_gen"], row["date_gen"], row["n_tokens_gen"])
print(row["text_gen"][:500])
print(row["events_gen"]["leadership_transitions"])

config summaries — reasoning-derived, long form

Key fields: summary_level_gen (meeting | year | decade), model_tag_gen (qwen35 | gptoss20b), meeting_id_gen (set on meeting rows; joins to core), year_gen/month_gen/decade_gen, summary_gen, reasoning_gen, cot_trace_gen (meeting rows only), source_meeting_ids_gen / source_years_gen (roll-up provenance for year/decade rows), model_gen (full HF model id).

summaries = load_dataset("institutional/institutional-proceedings-um", "summaries", split="train", token=True)
# one model's meeting-level summaries:
meeting_summaries = summaries.filter(
    lambda r: r["summary_level_gen"] == "meeting" and r["model_tag_gen"] == "qwen35"
)

-um-epubs — one EPUB per meeting

Files live at epubs/<meeting_id>.epub, where <meeting_id> == core.meeting_id_gen. No dataset viewer; fetch files directly.

from huggingface_hub import hf_hub_download

path = hf_hub_download(
    repo_id="institutional/institutional-proceedings-um-epubs",
    filename="epubs/march_1858.epub",
    repo_type="dataset",
    token=True,
)

How they connect

Field suffixes tell you a value's provenance:

Suffix	Meaning
_src	From the source collection (original bibliographic / OCR data)
_gen	Generated by the IDI pipeline
_exp	Experimental / exploratory generation

Join core ↔ summaries on meeting_id_gen, then map straight to the EPUB filename — the id is the same everywhere:

core = load_dataset("institutional/institutional-proceedings-um", "core", split="train", token=True)
summaries = load_dataset("institutional/institutional-proceedings-um", "summaries", split="train", token=True)

mid = core[0]["meeting_id_gen"]                      # e.g. "adjourned_august_1852"

# its summaries (one per model_tag_gen):
its_summaries = [r for r in summaries
                 if r["summary_level_gen"] == "meeting" and r["meeting_id_gen"] == mid]

# its readable EPUB — same id, no lookup table needed:
epub = hf_hub_download(
    repo_id="institutional/institutional-proceedings-um-epubs",
    filename=f"epubs/{mid}.epub", repo_type="dataset", token=True,
)

For year/decade summaries, follow source_meeting_ids_gen / source_years_gen back to the meetings (and thus core rows / EPUBs) that fed the synthesis.

Lightweight, no-download browsing

To page, search, or filter the tabular repo without loading it, use the Dataset Viewer HTTP API (config is core or summaries; send your token). Handy for locating a meeting_id_gen before pulling text or an EPUB.

curl -H "Authorization: Bearer $HF_TOKEN" \
  "https://datasets-server.huggingface.co/search?dataset=institutional/institutional-proceedings-um&config=core&split=train&query=medical%20school&length=10"

Auth & gating

The datasets are gated, so you always need an HF_TOKEN whose account has clicked "Agree and access" on each dataset page — otherwise loads return 401.

export HF_TOKEN="hf_..."

Or load it from a local .env (this repo already depends on python-dotenv):

from dotenv import load_dotenv
load_dotenv()  # sets HF_TOKEN in the environment

Then pass token=True to load_dataset / hf_hub_download (reads HF_TOKEN), or token=os.environ["HF_TOKEN"].

Dependencies

pip install datasets huggingface_hub python-dotenv