docs/INGESTION_SPEC.md

Ingestion Pipeline Specification

Summary

• Purpose: build the ingestion pipeline that extracts text from PDFs, PPTX, TXT, and URLs; chunks text; computes embeddings; and stores embeddings + metadata in a vector database with per-user and per-notebook isolation for RAG with citations.

Goals & Scope

• Support file and URL ingestion for per-user notebooks.
• Produce chunked, embedded documents with provenance metadata (source, page/slide, offsets).
• Provide an API/worker flow with status tracking and retries.

High-level Architecture

• Ingestion API: Accept uploads/URLs, validate, create source record, enqueue ingestion job.
• Worker: Extract -> Preprocess -> Chunk -> Embed -> Upsert to vector DB -> Persist metadata.
• Storage: raw files in data/raw/{user_id}/{notebook_id}/, metadata in a lightweight DB (SQLite/JSON), embeddings in a vector DB (Chroma/FAISS).

Components

• Upload Handler: validates file type/size and writes raw file.
• Extractors: modular functions for each type (PDF, PPTX, TXT, URL).
• Preprocessing: normalize whitespace, remove boilerplate, preserve page/slide indices and char offsets.
• Chunker: token-aware sliding window with overlap; store chunk-level metadata.
• Embedder: pluggable adapter (local sentence-transformers or API-based embeddings).
• Vectorstore Adapter: pluggable (default Chroma). Upserts include chunk text + metadata.
• Metadata Store: tracks notebooks, sources, ingestion status, timestamps, and source enable/disable flags.
• Job Orchestrator: simple background queue with retries (Redis/RQ or asyncio-based worker).

Extractors (recommended implementations)

• PDF: PyMuPDF (fitz) or pdfminer.six; extract per-page text and char offsets.
• PPTX: python-pptx extract slide text + speaker notes.
• TXT: read with encoding detection (chardet / charset-normalizer).
• URL: requests + BeautifulSoup with readability extraction; sanitize HTML.

Chunking

• Default: 500 tokens per chunk with 50 token overlap (configurable via CHUNK_TOKENS, CHUNK_OVERLAP).
• Alternative simple default: 2000 characters with 400 char overlap for token-agnostic environments.
• Store for each chunk: chunk_id, source_id, char_start, char_end, page, text_preview.

Embeddings

• Adapter interface supports local models (sentence-transformers/all-MiniLM) or API-based embeddings (OpenAI/HF).
• Configurable batch size (default 64) and rate limiting.

Vector DB

• Default: Chroma for HF Spaces compatibility; can swap to FAISS or Weaviate later.
• Use namespacing by user_id + notebook_id or include those fields in metadata for isolation.

Metadata Schema (suggested)

• Notebook: { notebook_id, user_id, name, created_at, updated_at }
• Source: { source_id, notebook_id, user_id, filename, url?, status, pages, size_bytes, created_at, error? }
• Chunk: { chunk_id, source_id, notebook_id, user_id, char_start, char_end, page, text_preview }

API Contract (ingest endpoints)

• POST /ingest/upload — multipart file, user_id, notebook_id → returns { source_id, job_id }.
• POST /ingest/url — body { url, user_id, notebook_id } → returns { source_id, job_id }.
• GET /ingest/status?job_id=... → returns status and error if failed.
• POST /ingest/enable_source — enable/disable source for retrieval.

Citation Strategy

• Store source_name, file_path or url, and page/slide in each chunk's metadata.
• Retrieval returns top-k chunks with metadata; present citations like [SourceName — page 3].

Job Orchestration

• Small files: synchronous ingestion (fast path).
• Large files: enqueue to background worker with retry/backoff and status updates.

Security & Operational Considerations

• Sanitize HTML from URLs; validate file types and size limits (configurable, e.g., 100MB).
• Rate-limit embedding calls to control cost.
• Log ingestion events and basic metrics.

File Layout (suggested)

• ingest/
  • api.py — upload + status endpoints
  • worker.py — ingestion worker and job runner
  • extractors.py — file/URL extraction functions
  • chunker.py — token-aware chunking utilities
  • embedder.py — embedding adapter
  • vectorstore.py — vector DB adapter
  • metadata.py — metadata store (SQLite/TinyDB)
• data/raw/ and data/meta/

Config / Environment Variables (examples)

• EMBEDDING_PROVIDER=local|openai|hf
• CHUNK_TOKENS=500
• CHUNK_OVERLAP=50
• EMBED_BATCH_SIZE=64
• VECTORSTORE=chroma|faiss|weaviate

Acceptance Criteria

• API accepts PDF/PPTX/TXT/URL and returns source_id + job_id.
• Worker extracts text with page/slide metadata and chunks according to config.
• Embeddings stored in vector DB with metadata linking back to source and offsets.
• Retrieval returns chunk text + metadata for citation.
• Per-user and per-notebook isolation enforced.

Testing

• Unit tests for extractors (use small sample files), chunker determinism, embedder adapter (mocked), and vectorstore upsert/retrieve (in-memory).

CI / Deployment

• Add GitHub Actions to run tests and push to Hugging Face Space on main branch updates.

Next Steps (immediate)

1. Scaffold ingest/ module files and a small runner script.
2. Add unit tests and sample files for extractors and chunker.
3. Implement Chroma adapter and local sentence-transformers embedding support.

---

If you'd like, I can now scaffold the ingest/ module files and tests. Also confirm preferred default embedding provider: sentence-transformers (local, free) or openai/hf (API-based).