INGEST_API_BACKEND.md

Ingest API — Backend (Python) Quickstart

Purpose

• FastAPI service that ingests a URL or PDF, creates embeddings (Mistral), and upserts vectors to Pinecone.

Key files

• ingest_api.py — main FastAPI service (endpoints, job tracking, OCR fallback).
• pdf-to-pinecone.py, 3-embedding-pinecone.py — original CLI scripts (reference).

Environment variables (required)

• PINECONE_API_KEY — Pinecone API key
• PINECONE_HOST — Pinecone host URL
• PINECONE_INDEX — Pinecone index name
• MISTRAL_API_KEY — Mistral embeddings API key
• ADMIN_API_KEY or ADMIN_API_KEYS — comma-separated admin keys for x-api-key

Optional / OCR

• TESSERACT_PATH — path to Tesseract executable (default: C:\Program Files\Tesseract-OCR\tesseract.exe)
• POPPLER_PATH — poppler bin path for pdf2image (default: C:\poppler\Library\bin)
• OCR_LANG — tesseract languages (e.g. ben+eng)

Install & run

1. Create venv and install deps:

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
pip install pdf2image pytesseract

2. Start service:

uvicorn ingest_api:app --host 0.0.0.0 --port 8000

API endpoints

• POST /ingest/url

  • Headers: x-api-key: <ADMIN_API_KEY>
  • Body JSON: { "url":"https://...", "namespace":"workspace", "webhook_url":"https://..." }
  • Response: 202 { "status":"accepted","job_id":"..." }

• POST /ingest/pdf

  • Headers: x-api-key: <ADMIN_API_KEY>
  • Multipart form: file (PDF), optional namespace, optional webhook_url
  • Response: 202 { "status":"accepted","job_id":"..." }

• GET /jobs/{job_id}

  • Headers: x-api-key: <ADMIN_API_KEY>
  • Returns job status and result JSON

Webhook

• If webhook_url provided in request, backend POSTs job result { job_id, status, result, error } when finished (best-effort).

OCR notes

• service tries pdfplumber first; if pages appear scanned/non-text it falls back to pdf2image + pytesseract.
• Install Tesseract + Poppler on server for OCR.

Production recommendations

• Do not expose ADMIN_API_KEY to browsers. Use a server-side proxy (Next.js, Supabase function, etc.).
• Replace in-memory JOBS with persistent store (Supabase/Redis/Postgres) to survive restarts.
• Use a task queue (Celery/RQ) for retries/long-running jobs in production.

Example curl

curl -X POST http://localhost:8000/ingest/url \
  -H "Content-Type: application/json" \
  -H "x-api-key: supersecret" \
  -d '{"url":"https://example.com/article","namespace":"workspace"}'

curl -X POST http://localhost:8000/ingest/pdf \
  -H "x-api-key: supersecret" \
  -F "file=@/path/to/file.pdf" \
  -F "namespace=workspace"

See also: INGEST_API.md for general integration notes and Next.js example in admin-nextjs-example/.