Blame for app/ingest/README.md

# Public S3 ingestion

OpenAlex literature comes exclusively from the anonymous public bucket
`s3://openalex/data/parquet/works/`. No OpenAlex API key, AWS account or signed URL
is used. ClinicalTrials.gov still uses its separate, keyless API. The snapshot
contains bibliographic metadata, citation references and available abstracts;
it is not a collection of full-text PDFs. The normalizer keeps each work's PubMed
Central ID when present; the API uses it at query time to read open-access full text
from Europe PMC (see [fulltext.py](../fulltext.py)). No full text is stored by ingestion.

Run these commands from `backend/` **on the batch host**, such as the sponsor
EC2 machine. Python reads selected Parquet columns over HTTPS, retains matching
rows there, computes embeddings there, and writes to Elasticsearch. Serving
FastAPI there also puts query embedding inference there. OpenAI handles PICO
parsing and bounded evidence extraction through the existing API integration.
The browser only sends queries and receives results; it never scans S3.

## Scope and planning

The default `hypertension-kidney` profile includes the **union** of those fields:
hypertension topics, the Nephrology subfield, additional renal topics, and
matching terms in titles, reconstructed abstracts and topic names. Exact rules
and the registry query are versioned in [scope.py](scope.py). All years and work
types are retained, including records without abstracts. Nonprimary works are
retained for discovery but excluded from primary-study statistical pools.
Missing abstracts do not imply missing results.

The opt-in `hypertension-kidney-pubmed` profile is the same topical scope restricted
to PubMed-indexed works (a PMID in `ids`, or `pubmed` in `indexed_in`). Update-date
partitions differ enormously in quality: in a 2026-09-19 check, a `2025-10` part had
0.0% PMIDs and 4.9% DOIs, a `2026-01` part 11.5% and 99.6%, and the identifier-poor
parts contain many records whose title and abstract belong to different works.
Registry linking matches on PMID and full text needs a PMCID, so a corpus built from
such parts cannot link. `--largest-first` does not avoid them; it selects by bytes.
The profile is recorded in checkpoints and provenance as its own scope. The Parquet
snapshot carries no `pmcid` in `ids`; the normalizer recovers it from PMC location URLs.

Completion means all matches under those rules in the supplied snapshot.
Incomplete metadata, terminology and source coverage prevent a promise of every
relevant study worldwide. The snapshot is a dated release, not a live feed.

```sh
.venv/bin/python -m app.bootstrap --plan
```

Planning is the default when neither `--plan` nor `--run` is supplied. It reads
only the manifest and prints its hash, date, part count, physical byte budget
and target index. On 2026-09-19 the public works manifest described **2,446 parts,
724,970,323,127 bytes, dated 2026-06-26**.

Partitions are update dates, **not topics**. Broad topic coverage requires
scanning all parts. Column projection can reduce traffic; the physical size
budget is conservative, not a measurement of transferred bytes or required
output disk space. The 5 GB default refuses a full scan instead of silently
importing a partial corpus.

## Full build on remote compute

Install dependencies using the root [setup guide](../../../README.md). Configure
Elasticsearch and models in the remote host's `backend/.env`. For an auditable
fresh corpus, choose a dedicated index such as
`ELASTIC_INDEX=studies-hypertension-kidney-s3`. Existing indexes are preserved and
updated by ID; importing does not delete unrelated older records.

Choose an existing persistent mount with room for the retained JSONL and index.
Inspect storage first: the sponsor machine's root disk may be small. The code
does not format or mount disks. After inspecting the plan, an example run on an
adequately provisioned batch host is:

```sh
.venv/bin/python -m app.bootstrap --run \
  --data-dir /mnt/research/nullmap/2026-06-26 \
  --max-snapshot-bytes 750000000000
```

Replace that directory with the actual persistent mount. Set the budget from
the printed manifest and host capacity. Repeat the same command and directory
to resume. There is no default year or record cap. The registry condition query
includes all statuses and study types and exhausts its cursor. Snapshot records
are normalized, weakly classified, embedded, linked across batches and indexed.
Bootstrap performs no paid LLM labeling or extraction.

For a bounded smoke test:

```sh
.venv/bin/python -m app.bootstrap --plan --max-files 1
.venv/bin/python -m app.bootstrap --run --max-files 1 --max-records 100 \
  --registry-limit 20 --data-dir data/s3-smoke
```

If the selected part exceeds 5 GB, choose an explicit budget after inspecting
its size. A part may contain no scope matches. `--max-files`, `--max-records` and
`--registry-limit` are explicit partial runs. Use a separate directory for the
full run: changing selected files invalidates the scan checkpoint.

To continue after a `--max-files N` run without rescanning, start a new directory
with `--skip-files N` (optionally with another `--max-files` to work in chunks).
Provenance keeps the fingerprint of the whole release. A run that skips parts
never sets `complete_snapshot_scope=true`, even when every part has been covered
across runs; coverage is then the union of the per-directory checkpoints.

```sh
.venv/bin/python -m app.bootstrap --run --skip-registry --skip-files 550 --max-files 30 \
  --max-snapshot-bytes 40000000000 --data-dir data/research-s3-continue/p0550-0579
```

## Checkpoints and coverage

Keep each output and its sidecars together:

| Artifact | Purpose |
| --- | --- |
| `manifest.json` | Pinned release metadata and part sizes |
| `openalex.jsonl` | Matched source records and snapshot provenance |
| `openalex.jsonl.checkpoint.json` | Committed bytes, part, within-part progress and coverage |
| `ctgov.jsonl` and checkpoint | Registry records, query signature and paging cursor |
| `*.materialize.json` | Indexed offset, counts, model and target-index signature |
| `corpus-summary.json` | Stage, declared scope, source completion and index health |

Writes flush before checkpoints advance. Restart rolls back uncommitted JSONL
bytes and repeats uncommitted index batches with idempotent IDs. Source, scope,
model and target-index signatures prevent incompatible resumes. Capped source
files may grow; materialization resumes at its committed offset. A new snapshot
release requires a new directory. A remote manifest change detected at scan
completion invalidates the completion claim.

`complete_snapshot_scope=true` means every part in the supplied manifest was
exhausted. A subset manifest establishes coverage only of that subset. The corpus
summary is complete only when the snapshot and registry query both finish.
Limits, interruptions and skipped sources remain explicit. Low disk space stops
the scan with its checkpoint preserved; it does not provision storage.

The importer uses bounded batches and an index-backed linker. It neither loads
the whole corpus into RAM nor duplicates all vectors into JSONL. Registry facts
override linked paper numbers; canonical links prevent double counting.
Metadata-only records are embedded using their available title/text and marked
as abstract unavailable in search results.

## Reference expansion without API calls

Search expands review citations using **already indexed** works and their stored
vectors, reporting missing references. There are no live OpenAlex lookups.
For missing references, collect IDs and run a separate S3 pass into the same
Elasticsearch index using a separate output directory:

```sh
.venv/bin/python -m app.ingest reference-ids \
  --input /mnt/research/nullmap/2026-06-26/openalex.jsonl \
  --output /mnt/research/nullmap/missing-references.txt
.venv/bin/python -m app.bootstrap --run --skip-registry \
  --work-ids /mnt/research/nullmap/missing-references.txt \
  --data-dir /mnt/research/nullmap/reference-backfill \
  --max-snapshot-bytes 750000000000
```

Skip the backfill if the ID file is empty. `--work-ids` overrides topic filtering
so differently worded references can be recovered. This requires another scan:
S3 has no work-ID search endpoint. Only works in the snapshot can be recovered.
Backfills keep separate provenance and do not claim completion of the main build.

## Individual stages and model choices

`python -m app.ingest --help` exposes snapshot scans, registry fetching,
normalization, classification, embedding, indexing, reference IDs, TEI full-text
extraction, labeling, classifier training and the `pagerank` pass, which
recomputes within-corpus citation PageRank (references outside the index are
dropped) and bulk-updates the `pagerank`/`authority` rank-feature fields used
by retrieval. For example:

```sh
.venv/bin/python -m app.ingest snapshot --plan
.venv/bin/python -m app.ingest snapshot --input /data/sample.parquet \
  --profile hypertension-kidney --output data/sample.jsonl --resume
.venv/bin/python -m app.ingest fetch ctgov --query 'kidney' --filter '' \
  --all --output data/ctgov.jsonl --resume
.venv/bin/python -m app.ingest tei --input /data/tei/ --output data/tei.jsonl
```

Manifests use `{ "files": [{ "url": "…", "size_bytes": 123 }] }`; the public
`meta.content_length` form is supported. Local fixtures are supported for tests.
Remote paths accept only unsigned public OpenAlex works URLs. DuckDB installs
its `httpfs` extension under the output directory on first use.

The `tei` stage reads GROBID-style TEI XML full text (files or directories of
`*.xml`) and extracts null-hypothesis evidence with the deterministic, auditable
heuristic in [tei.py](tei.py): a dictionary of statistical terms (confidence
interval, p-value, correlation, effect size, sample size) locates candidate
sentences; the surrounding paragraph is scanned for null confirmations
("no significant difference", non-significant p-values, and confidence intervals
that straddle the null value of 0 for differences/correlations or 1 for
ratios); and the associated numbers are parsed from the body text, tables and
appendix. Each row carries the study fields consumed by `statistics.py` and
`classifier.py` (`estimate`, `ci_low`/`ci_high`, `p_value`, `effect_type`, `n`,
`result_label`, `null_score`, `evidence_span`) plus a `tei_findings` list and a
`null_findings` subset, each keeping the exact source sentence. Scores are
heuristic, never calibrated probabilities. OpenAlex itself ships bibliographic
Parquet, not TEI; point this stage at a full-text TEI corpus keyed by the same
work identifiers.

The legacy standalone `link` command loads its input into memory and is suitable
for small slices. Bootstrap uses the bounded index-backed path. For lexical-only
operation use `--skip-embeddings` and serve with `EMBEDDINGS_ENABLED=false`.
Keep indexing and query embedding models identical.

The default classifier remains the auditable phrase heuristic. The earlier small
LLM-labeled pilot failed evaluation and was not promoted. Labeling and model
training remain explicit experiments; see the historical
[validation record](../../../docs/VALIDATION.md) and
[registry fixture provenance](../../tests/fixtures/README.md).

Sources: [OpenAlex snapshot documentation](https://help.openalex.org/access/snapshot/)
and the [sponsor's OpenAlex README](https://voloridge-hack-mit-2026.s3.us-east-1.amazonaws.com/src/openalex/README.md).