app/extraction/README.md

Extraction Design

Purpose

The extraction directory owns PDF-to-structure conversion through MinerU.

MinerU should extract structured markdown, headings, LaTeX math, tables, images, page metadata, and layout metadata. The output of extraction is an ordered list of chunks ready for database insertion.

Files

pipeline.py

Coordinates the full extraction workflow: receive PDF path, run MinerU, normalize output, extract assets, create ordered chunks, persist through ingestion services, and trigger embedding jobs.

mineru_client.py

Wraps local MinerU execution, captures process output, locates generated artifacts, and reports structured extraction errors.

chunker.py

Converts normalized MinerU output into structural chunks. It must preserve physical document order and assign monotonically increasing sequence_id values.

normalizer.py

Cleans MinerU output into stable internal structures. It normalizes markdown, math blocks, image references, whitespace, and plain text for embeddings.

assets.py

Moves extracted images and layout artifacts into local storage and returns metadata that can be persisted by the asset repository.

jobs.py

Defines ingestion job state names and transition expectations. Expected states are queued, extracting, chunking, embedding, complete, and failed.

Chunking Rules

• Preserve the physical order of the PDF.
• Assign sequence IDs starting at 1 per document.
• Keep math blocks intact.
• Attach images to nearby structural chunks.
• Preserve heading paths, page spans, and bounding boxes where available.

Data Dependencies

extraction depends on core.paths for storage, services.ingestion for persistence, and embeddings.service after chunks are stored.

The sequence ID is created by extraction.chunker and persisted transactionally by services.ingestion into PostgreSQL.

Embedding payloads are generated after chunk persistence by embeddings.service and stored in PostgreSQL through database.repositories.embeddings and database.pgvector.