# ML Pipeline — Research Document Archive

## Overview
Machine learning pipeline for analyzing 234K declassified government documents across 7 collections. Extracts dates, correlates documents with historical crises, classifies topics, detects redactions, and builds entity networks.

## Prerequisites
- Python venv: `/opt/epstein_env/` (torch, spacy, transformers, sentence-transformers)
- PostgreSQL: `epstein_research` database
- RunPod: GPU-heavy tasks (zero-shot classification, BERTopic)

## Pipeline Scripts

| Script | Purpose | Runs On | Status |
|--------|---------|---------|--------|
| `01_extract_dates.py` | Extract dates from filenames, congress sessions, NER | Hetzner CPU | Done |
| `02_seed_events.py` | Seed 20 historical events | Hetzner CPU | Done |
| `03_correlate_crises.py` | Multi-signal crisis correlation | Hetzner CPU | Done |
| `04_export_for_topics.py` | Export JSONL for GPU classification | Hetzner CPU | Done |
| `05_import_topics.py` | Import topic results from RunPod | Hetzner CPU | Ready |
| `06_extract_keywords.py` | TF-IDF keyword extraction | Hetzner CPU | TODO |
| `07_detect_redactions.py` | OpenCV redaction detection | Hetzner CPU | TODO |
| `08_find_duplicates.py` | Embedding-based dedup | Hetzner SQL | TODO |
| `09_entity_networks.py` | Co-occurrence & graph analysis | Hetzner CPU | TODO |

## Running

### Full pipeline (stages 1-3)
```bash
tmux new-session -s ml-pipeline "/var/www/research/ml/run_pipeline.sh"
```

### Individual scripts
```bash
cd /var/www/research/ml
/opt/epstein_env/bin/python3 01_extract_dates.py
```

### GPU tasks (RunPod)
Export data, transfer to RunPod, run, transfer results back:
```bash
/opt/epstein_env/bin/python3 04_export_for_topics.py
# scp topic_export.jsonl to RunPod
# Run classify_fast.py on RunPod
# scp topic_results.jsonl back
/opt/epstein_env/bin/python3 05_import_topics.py
```

## Configuration
- `config.py` — DB credentials, topic labels, historical events, congress dates
- `db.py` — Database connection helpers
- `schema.sql` — Table definitions for ML pipeline

## Results (as of 2026-03-06)
- 234K documents dated (181K from congress, 36K from folders, 9K from NER)
- 20 historical events seeded
- 296K document-event correlations
- Topic classification: processing on RunPod