web_docs/docs/guides/loading-meeting-data.md

metadata

sidebar_position: 8

Loading Meeting Data for Priority States

Guide to loading existing LocalView meeting data and updating with recent scraping for AL, GA, IN, MA, WA, WI.

📊 Current Meeting Data Status

What You Have Now

OpenStates Legislative Events (Already loaded):

• Alabama (AL): Legislative committee hearings
• Georgia (GA): Legislative events
• Indiana (IN): Legislative events
• Massachusetts (MA): Legislative events
• Washington (WA): Legislative events
• Wisconsin (WI): Legislative events

Location: data/gold/states/{STATE}/events_events.parquet

What's Missing: Municipal/local government meetings (city council, county boards, school boards)

🎯 Three-Step Process

Step 1: Check What LocalView Data Exists

LocalView is a Harvard dataset with 153K+ municipal meeting transcripts from YouTube.

# Check if you have LocalView data downloaded
ls -lh data/cache/localview/

# If you only see municipality_channels.csv with demo data, 
# you need to download the actual LocalView dataset

Step 2: Download LocalView Dataset (Optional)

If you want the full historical dataset (2006-2023):

Manual Download from Harvard Dataverse:

1. Visit: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/NJTBEM
2. Download these files to data/cache/localview/:
   • municipalities.csv (list of 1000+ jurisdictions)
   • meetings.csv (153K meeting metadata)
   • transcripts.csv (video transcripts with captions)
3. Run ingestion script:

python scripts/datasources/localview/localview_ingestion.py

What This Gives You:

• 153,452 meeting transcripts (2006-2023)
• 1,000+ municipalities across all states
• Full text search of meeting discussions
• Speaker identification and attendance tracking

Step 3: Scrape Recent Meetings (2024-2026)

The LocalView dataset ends in 2023. To get 2024-2026 data, scrape YouTube directly.

A. Find Municipal YouTube Channels

First, you need to discover YouTube channels for municipalities in your 6 states.

Option 1: Use Discovery Scripts

# Discover government URLs for priority states
python scripts/discovery/discover_jurisdictions.py \
    --states AL,GA,IN,MA,WA,WI \
    --types city,county \
    --output data/discovered_urls.json

# Extract YouTube channels from discovered sites
python scripts/localview/update_municipality_list.py \
    --states AL,GA,IN,MA,WA,WI

Option 2: Manual Research

For major cities in each state:

1. Go to city's official website
2. Look for "Meetings", "City Council", "Agendas"
3. Check if they have YouTube channel link
4. Add to data/cache/localview/municipality_channels.csv

Example format:

municipality,channel_id,state,population,added_date
Birmingham AL,UCxxxxxxxxxxxxxxxxxx,AL,200000,2026-05-03
Atlanta GA,UCyyyyyyyyyyyyyyyyyy,GA,500000,2026-05-03
Indianapolis IN,UCzzzzzzzzzzzzzzzzzz,IN,880000,2026-05-03

B. Run YouTube Scraper

Setup:

# Get YouTube API key from Google Cloud Console
# https://console.cloud.google.com/apis/credentials

# Add to .env
echo "YOUTUBE_API_KEY=your_key_here" >> .env

Scrape by State:

# Scrape all 6 priority states
python scripts/localview/scrape_youtube_channels.py \
    --states AL,GA,IN,MA,WA,WI \
    --since 2024-01-01

# Or scrape specific channels
python scripts/localview/scrape_youtube_channels.py \
    --channels "UCxxxxx,UCyyyyy" \
    --max-videos 100

What This Does:

1. Gets recent videos from municipal YouTube channels
2. Downloads auto-generated captions/transcripts
3. Extracts meeting metadata (date, title, description)
4. Saves to data/cache/localview/videos_{STATE}_{date}.parquet

C. Process Transcripts

Extract speaker names and meeting details:

# Extract transcripts from videos
python scripts/localview/extract_transcripts.py \
    --states AL,GA,IN,MA,WA,WI \
    --output data/gold/meetings/

# Extract contact information from transcripts
python scripts/manage_contacts.py extract \
    --states AL,GA,IN,MA,WA,WI \
    --batch-size 1000

📁 Final Data Structure

After completing all steps:

data/gold/
├── states/
│   ├── AL/
│   │   ├── events_events.parquet          # OpenStates legislative events
│   │   ├── events_participants.parquet
│   │   ├── meetings_local.parquet         # LocalView municipal meetings
│   │   └── contact_official.parquet     # Legislators + local officials
│   ├── GA/
│   │   ├── events_events.parquet
│   │   ├── meetings_local.parquet
│   │   └── contact_official.parquet
│   └── ... (IN, MA, WA, WI)
│
└── meetings/                              # Cross-state meeting data
    ├── meetings_transcripts.parquet       # All 153K transcripts
    ├── contacts_local_officials.parquet   # Aggregated local officials
    └── contacts_meeting_attendance.parquet # Attendance records

🔍 Querying Meeting Data

Find Meetings by State

import polars as pl

# Load Alabama meetings
al_meetings = pl.read_parquet('data/gold/states/AL/meetings_local.parquet')

# Filter by date range
recent = al_meetings.filter(
    pl.col('meeting_date') >= '2024-01-01'
)

print(f"Found {len(recent)} Alabama meetings since 2024")

Search Meeting Transcripts

# Load all transcripts
transcripts = pl.read_parquet('data/gold/meetings/meetings_transcripts.parquet')

# Search for oral health mentions
oral_health = transcripts.filter(
    pl.col('caption_text').str.contains('(?i)dental|teeth|oral health|fluoride')
)

# Group by state
by_state = oral_health.group_by('state').count()
print(by_state)

Find Officials Attending Meetings

# Load attendance records
attendance = pl.read_parquet('data/gold/meetings/contacts_meeting_attendance.parquet')

# Find all meetings for a specific official
officials_meetings = attendance.filter(
    pl.col('name') == 'Stephanie Briggs'
)

print(f"Stephanie Briggs attended {len(officials_meetings)} meetings")

🚀 Quick Start for Development

If you just want to test with sample data:

# 1. Use existing OpenStates events (already loaded)
python -c "
import polars as pl
df = pl.read_parquet('data/gold/states/AL/events_events.parquet')
print(f'Alabama legislative events: {len(df)}')
print(df.head())
"

# 2. Scrape a single city for testing
python scripts/localview/scrape_youtube_channels.py \
    --channels "UCMFAKdxL6sATpkRqLdJyKUg" \
    --max-videos 10

# 3. Process and view results
python scripts/localview/extract_transcripts.py --latest

📚 Related Documentation

• LocalView Integration Guide
• Contacts & Meetings Workflow
• Data Sources

⚠️ Important Notes

API Quotas

YouTube API has daily quotas:

• 10,000 units/day (free tier)
• Fetching 1 video = ~3 units
• Can scrape ~3,000 videos/day

Strategy: Prioritize high-population cities, scrape incrementally

Storage Requirements

• Full LocalView dataset: ~3 GB compressed
• Transcripts with embeddings: ~10 GB
• Videos (if downloading): ~500 GB (not recommended)

Recommendation: Store transcripts only, not videos

Data Freshness

• OpenStates: Updated weekly (legislative sessions)
• YouTube scraping: Run weekly/monthly for new meetings
• LocalView dataset: Historical data only (2006-2023)

🐛 Troubleshooting

"YOUTUBE_API_KEY not found"

# Get API key from Google Cloud Console
# Enable YouTube Data API v3
# Create credentials -> API Key
echo "YOUTUBE_API_KEY=your_key_here" >> .env

"No videos found for channel"

• Verify channel ID is correct (starts with "UC")
• Check channel has public videos
• Ensure videos have captions enabled

"Transcript not available"

• Only videos with auto-captions or manual captions work
• Some cities disable captions (can't extract text)
• Fallback: Use video metadata (title, description)

"Out of API quota"

# Check quota usage at:
# https://console.cloud.google.com/apis/api/youtube.googleapis.com/quotas

# Solutions:
# 1. Wait 24 hours for reset
# 2. Request quota increase
# 3. Use multiple API keys (rotate daily)