app/workers/README.md

Workers Design

Purpose

The workers directory contains the Celery task definitions and thin helpers for background PDF ingestion (MinerU + chunking + assets) and embedding generation.

Active Files

tasks.py

Contains the real Celery @celery_app.task definitions:

• process_ingestion — full MinerU extraction pipeline (sync DB session)
• embed_document — batch embedding for a document
• generate_section_summaries — high-quality hierarchical section + paper-level summarization (runs after embeddings; can take many minutes; quality-first personal feature)

These are what the API actually calls via .delay().

ingestion_worker.py

Thin async wrapper around the pipeline (primarily used by tests and as documentation of the old design).

Historical Note

The original design used FastAPI BackgroundTasks + an in-memory asyncio.Queue + runner.py / embedding_worker.py. This was replaced by Celery + Redis for better isolation and durability.

Data Dependencies

workers depends on extraction, embeddings, services, and database (both async and sync session layers).