INGEST_API.md

Ingest API — Usage & Integration

Overview

• Small Python FastAPI service that ingests URLs and PDFs, creates embeddings (Mistral), and upserts vectors to Pinecone.
• Endpoints return a job_id; jobs are processed in background and can optionally POST results to a webhook_url.

APIs

• POST /ingest/url

  • Auth: header x-api-key
  • Body (JSON): { "url": "https://...", "namespace": "workspace", "webhook_url": "https://..." }
  • Response: 202 Accepted, { "status": "accepted", "job_id": "..." }

• POST /ingest/pdf

  • Auth: header x-api-key
  • Multipart form fields: file (PDF), namespace (optional), webhook_url (optional)
  • Response: 202 Accepted, { "status": "accepted", "job_id": "..." }

• GET /jobs/{job_id}

  • Auth: header x-api-key
  • Returns job object with keys: id, type, status, created_at, started_at, finished_at, result, error.

Webhooks

• Provide webhook_url in request body/form. When job finishes the service will POST JSON { job_id, status, result, error } to that URL (best-effort; failures are ignored).

Environment variables (Python ingest service)

• PINECONE_API_KEY — Pinecone API key
• PINECONE_HOST — Pinecone host (e.g. https://xyz.pinecone.io)
• PINECONE_INDEX — Pinecone index name
• MISTRAL_API_KEY — Mistral embeddings API key
• ADMIN_API_KEY or ADMIN_API_KEYS — comma-separated admin API key(s) required by x-api-key
• TESSERACT_PATH — (optional) path to tesseract executable for OCR fallback; default: C:\Program Files\Tesseract-OCR\tesseract.exe
• POPPLER_PATH — (optional) path to poppler binaries for pdf2image; default: C:\poppler\Library\bin
• OCR_LANG — languages for tesseract (example: ben+eng)

Run locally (Python)

• Create venv and install deps (ensure tesseract + poppler installed for OCR):

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt
pip install pdf2image pytesseract

• Set required env vars (example PowerShell):

$env:PINECONE_API_KEY="your_key"
$env:PINECONE_HOST="https://your-index-host.pinecone.io"
$env:PINECONE_INDEX="pregnancy-knowledge"
$env:MISTRAL_API_KEY="your_mistral_key"
$env:ADMIN_API_KEY="supersecret"
# Optional OCR
$env:TESSERACT_PATH="C:\Program Files\Tesseract-OCR\tesseract.exe"
$env:POPPLER_PATH="C:\poppler\Library\bin"
$env:OCR_LANG="ben+eng"
uvicorn ingest_api:app --host 0.0.0.0 --port 8000

Next.js / Frontend integration

• Two approaches:
  1. Server-proxy (recommended: keep admin key server-side): create Next.js API route that calls Python service using INGEST_API_URL and INGEST_API_KEY.
     • Env (server): INGEST_API_URL, INGEST_API_KEY.
     • Example file provided: admin-nextjs-example/pages/api/proxy/ingest-url.js.
  2. Direct client upload (simpler for quick setups) — client sends x-api-key and form data directly to Python service.
     • If using client-facing key, set NEXT_PUBLIC_INGEST_API_KEY and NEXT_PUBLIC_INGEST_API_URL (NOT recommended for production).

Example curl

• Ingest URL (server or client with valid API key):

curl -X POST http://localhost:8000/ingest/url \
  -H "Content-Type: application/json" \
  -H "x-api-key: supersecret" \
  -d '{"url":"https://example.com/article","namespace":"workspace"}'

• Upload PDF:

curl -X POST http://localhost:8000/ingest/pdf \
  -H "x-api-key: supersecret" \
  -F "file=@/path/to/file.pdf" \
  -F "namespace=workspace"

Admin UI examples in this repo

• admin_dashboard.html — static demo (opens in browser, uses same-origin requests to Python service).
• admin-nextjs-example/ — Next.js example: pages/admin.js (client UI); pages/api/proxy/ingest-url.js (server proxy example); pages/api/proxy/ingest-pdf.js (placeholder explaining multipart server proxying).

Notes & production recommendations

• Current job store is in-memory (JOBS) — process restart will lose state. For production use Supabase/Redis/Postgres to persist jobs.
• For reliable background processing and retries, replace BackgroundTasks with a worker system (Celery, RQ, or a serverless job queue).
• Do not expose ADMIN_API_KEY in browser. Use a server-side proxy with authentication.

Questions or next steps

• I can implement server-side PDF proxy parsing in the Next.js example (using formidable) or wire Supabase auth to the admin UI. Which should I do next?