app/ingest/README.md

Public S3 ingestion

OpenAlex literature comes exclusively from the anonymous public bucket s3://openalex/data/parquet/works/. No OpenAlex API key, AWS account or signed URL is used. ClinicalTrials.gov still uses its separate, keyless API. The snapshot contains bibliographic metadata, citation references and available abstracts; it is not a collection of full-text PDFs. The normalizer keeps each work's PubMed Central ID when present; the API uses it at query time to read open-access full text from Europe PMC (see fulltext.py). No full text is stored by ingestion.

Run these commands from backend/ on the batch host, such as the sponsor EC2 machine. Python reads selected Parquet columns over HTTPS, retains matching rows there, computes embeddings there, and writes to Elasticsearch. Serving FastAPI there also puts query embedding inference there. OpenAI handles PICO parsing and bounded evidence extraction through the existing API integration. The browser only sends queries and receives results; it never scans S3.

Scope and planning

The default hypertension-kidney profile includes the union of those fields: hypertension topics, the Nephrology subfield, additional renal topics, and matching terms in titles, reconstructed abstracts and topic names. Exact rules and the registry query are versioned in scope.py. All years and work types are retained, including records without abstracts. Nonprimary works are retained for discovery but excluded from primary-study statistical pools. Missing abstracts do not imply missing results.

The opt-in hypertension-kidney-pubmed profile is the same topical scope restricted to PubMed-indexed works (a PMID in ids, or pubmed in indexed_in). Update-date partitions differ enormously in quality: in a 2026-09-19 check, a 2025-10 part had 0.0% PMIDs and 4.9% DOIs, a 2026-01 part 11.5% and 99.6%, and the identifier-poor parts contain many records whose title and abstract belong to different works. Registry linking matches on PMID and full text needs a PMCID, so a corpus built from such parts cannot link. --largest-first does not avoid them; it selects by bytes. The profile is recorded in checkpoints and provenance as its own scope. The Parquet snapshot carries no pmcid in ids; the normalizer recovers it from PMC location URLs.

Completion means all matches under those rules in the supplied snapshot. Incomplete metadata, terminology and source coverage prevent a promise of every relevant study worldwide. The snapshot is a dated release, not a live feed.

.venv/bin/python -m app.bootstrap --plan

Planning is the default when neither --plan nor --run is supplied. It reads only the manifest and prints its hash, date, part count, physical byte budget and target index. On 2026-09-19 the public works manifest described 2,446 parts, 724,970,323,127 bytes, dated 2026-06-26.

Partitions are update dates, not topics. Broad topic coverage requires scanning all parts. Column projection can reduce traffic; the physical size budget is conservative, not a measurement of transferred bytes or required output disk space. The 5 GB default refuses a full scan instead of silently importing a partial corpus.

Full build on remote compute

Install dependencies using the root setup guide. Configure Elasticsearch and models in the remote host's backend/.env. For an auditable fresh corpus, choose a dedicated index such as ELASTIC_INDEX=studies-hypertension-kidney-s3. Existing indexes are preserved and updated by ID; importing does not delete unrelated older records.

Choose an existing persistent mount with room for the retained JSONL and index. Inspect storage first: the sponsor machine's root disk may be small. The code does not format or mount disks. After inspecting the plan, an example run on an adequately provisioned batch host is:

.venv/bin/python -m app.bootstrap --run \
  --data-dir /mnt/research/nullmap/2026-06-26 \
  --max-snapshot-bytes 750000000000

Replace that directory with the actual persistent mount. Set the budget from the printed manifest and host capacity. Repeat the same command and directory to resume. There is no default year or record cap. The registry condition query includes all statuses and study types and exhausts its cursor. Snapshot records are normalized, weakly classified, embedded, linked across batches and indexed. Bootstrap performs no paid LLM labeling or extraction.

For a bounded smoke test:

.venv/bin/python -m app.bootstrap --plan --max-files 1
.venv/bin/python -m app.bootstrap --run --max-files 1 --max-records 100 \
  --registry-limit 20 --data-dir data/s3-smoke

If the selected part exceeds 5 GB, choose an explicit budget after inspecting its size. A part may contain no scope matches. --max-files, --max-records and --registry-limit are explicit partial runs. Use a separate directory for the full run: changing selected files invalidates the scan checkpoint.

To continue after a --max-files N run without rescanning, start a new directory with --skip-files N (optionally with another --max-files to work in chunks). Provenance keeps the fingerprint of the whole release. A run that skips parts never sets complete_snapshot_scope=true, even when every part has been covered across runs; coverage is then the union of the per-directory checkpoints.

.venv/bin/python -m app.bootstrap --run --skip-registry --skip-files 550 --max-files 30 \
  --max-snapshot-bytes 40000000000 --data-dir data/research-s3-continue/p0550-0579

Checkpoints and coverage

Keep each output and its sidecars together:

Artifact	Purpose
manifest.json	Pinned release metadata and part sizes
openalex.jsonl	Matched source records and snapshot provenance
openalex.jsonl.checkpoint.json	Committed bytes, part, within-part progress and coverage
ctgov.jsonl and checkpoint	Registry records, query signature and paging cursor
*.materialize.json	Indexed offset, counts, model and target-index signature
corpus-summary.json	Stage, declared scope, source completion and index health

Writes flush before checkpoints advance. Restart rolls back uncommitted JSONL bytes and repeats uncommitted index batches with idempotent IDs. Source, scope, model and target-index signatures prevent incompatible resumes. Capped source files may grow; materialization resumes at its committed offset. A new snapshot release requires a new directory. A remote manifest change detected at scan completion invalidates the completion claim.

complete_snapshot_scope=true means every part in the supplied manifest was exhausted. A subset manifest establishes coverage only of that subset. The corpus summary is complete only when the snapshot and registry query both finish. Limits, interruptions and skipped sources remain explicit. Low disk space stops the scan with its checkpoint preserved; it does not provision storage.

The importer uses bounded batches and an index-backed linker. It neither loads the whole corpus into RAM nor duplicates all vectors into JSONL. Registry facts override linked paper numbers; canonical links prevent double counting. Metadata-only records are embedded using their available title/text and marked as abstract unavailable in search results.

Reference expansion without API calls

Search expands review citations using already indexed works and their stored vectors, reporting missing references. There are no live OpenAlex lookups. For missing references, collect IDs and run a separate S3 pass into the same Elasticsearch index using a separate output directory:

.venv/bin/python -m app.ingest reference-ids \
  --input /mnt/research/nullmap/2026-06-26/openalex.jsonl \
  --output /mnt/research/nullmap/missing-references.txt
.venv/bin/python -m app.bootstrap --run --skip-registry \
  --work-ids /mnt/research/nullmap/missing-references.txt \
  --data-dir /mnt/research/nullmap/reference-backfill \
  --max-snapshot-bytes 750000000000

Skip the backfill if the ID file is empty. --work-ids overrides topic filtering so differently worded references can be recovered. This requires another scan: S3 has no work-ID search endpoint. Only works in the snapshot can be recovered. Backfills keep separate provenance and do not claim completion of the main build.

Individual stages and model choices

python -m app.ingest --help exposes snapshot scans, registry fetching, normalization, classification, embedding, indexing, reference IDs, TEI full-text extraction, labeling, classifier training and the pagerank pass, which recomputes within-corpus citation PageRank (references outside the index are dropped) and bulk-updates the pagerank/authority rank-feature fields used by retrieval. For example:

.venv/bin/python -m app.ingest snapshot --plan
.venv/bin/python -m app.ingest snapshot --input /data/sample.parquet \
  --profile hypertension-kidney --output data/sample.jsonl --resume
.venv/bin/python -m app.ingest fetch ctgov --query 'kidney' --filter '' \
  --all --output data/ctgov.jsonl --resume
.venv/bin/python -m app.ingest tei --input /data/tei/ --output data/tei.jsonl

Manifests use { "files": [{ "url": "…", "size_bytes": 123 }] }; the public meta.content_length form is supported. Local fixtures are supported for tests. Remote paths accept only unsigned public OpenAlex works URLs. DuckDB installs its httpfs extension under the output directory on first use.

The tei stage reads GROBID-style TEI XML full text (files or directories of *.xml) and extracts null-hypothesis evidence with the deterministic, auditable heuristic in tei.py: a dictionary of statistical terms (confidence interval, p-value, correlation, effect size, sample size) locates candidate sentences; the surrounding paragraph is scanned for null confirmations ("no significant difference", non-significant p-values, and confidence intervals that straddle the null value of 0 for differences/correlations or 1 for ratios); and the associated numbers are parsed from the body text, tables and appendix. Each row carries the study fields consumed by statistics.py and classifier.py (estimate, ci_low/ci_high, p_value, effect_type, n, result_label, null_score, evidence_span) plus a tei_findings list and a null_findings subset, each keeping the exact source sentence. Scores are heuristic, never calibrated probabilities. OpenAlex itself ships bibliographic Parquet, not TEI; point this stage at a full-text TEI corpus keyed by the same work identifiers.

The legacy standalone link command loads its input into memory and is suitable for small slices. Bootstrap uses the bounded index-backed path. For lexical-only operation use --skip-embeddings and serve with EMBEDDINGS_ENABLED=false. Keep indexing and query embedding models identical.

The default classifier remains the auditable phrase heuristic. The earlier small LLM-labeled pilot failed evaluation and was not promoted. Labeling and model training remain explicit experiments; see the historical validation record and registry fixture provenance.

Sources: OpenAlex snapshot documentation and the sponsor's OpenAlex README.