OPTIMIZATION_SUMMARY.md

# Latency optimization summary (Phase 1–2)

## Phase 1 — Async pipeline, retrieval, concurrency

| Feature | Env flags | Notes |
|---------|-----------|--------|
| Async LLM | `ENABLE_ASYNC_PIPELINE=true` | `AsyncLLMAdapter` + throttled OpenAI calls |
| Qdrant vector store | `VECTORSTORE_BACKEND=qdrant`, `QDRANT_URL` | Re-ingest after migration: `python scripts/reingest_qdrant.py --tenant-id …` |
| Hybrid BM25 + RRF | `ENABLE_HYBRID_RETRIEVAL=true` | Qdrant only |
| Semantic retrieval cache | `SEMANTIC_CACHE_ENABLED=true` | Qdrant collection `QDRANT_CACHE_COLLECTION` |
| Parallel multi-section | Same as async + Postgres or `ALLOW_SQLITE_PARALLEL_SECTIONS=true` | REST + agentic use `multi_section_parallel_enabled()` |
| LLM throttle | `MAX_CONCURRENT_LLM_CALLS` | Global semaphore + retry on 429 |
| Stale job recovery | `GENERATION_TIMEOUT_SECONDS`, `GENERATION_STALE_SWEEP_SECONDS` | Uses `reports.generation_started_at` |

## Phase 2 — Speculation, prompt cache, Temporal

| Feature | Env flags | Notes |
|---------|-----------|--------|
| Speculative tools | `ENABLE_SPECULATIVE_EXECUTOR=true` | Requires async pipeline; inspector loop only |
| Prompt cache | `ENABLE_PROMPT_CACHING=true` | System prefix pad + `prompt_cache_key` (includes tenant) |
| Temporal workflows | `ENABLE_TEMPORAL_WORKFLOW=true` | `docker compose --profile temporal up`; `python worker.py` |
| Workflow ID | Response field `workflow_id` on `POST /generate` when Temporal starts |

## Inspector vs standard generate

- Default `NOTES_ONLY_GENERATION=true` uses the **standard** RAG pipeline on `POST /generate` (no inspector speculation).
- Set `AGENTIC_INSPECTOR_WHEN_NOTES_ONLY=true` or `NOTES_ONLY_GENERATION=false` to run the inspector on main generate.
- `POST /reports/{id}/agentic/generate` persists sections and report status like REST generate.

## Phase 3 — Redis coordination (scale-out)

| Feature | Env flags | Notes |
|---------|-----------|--------|
| Distributed rate limits | `REDIS_URL` | Shared sliding window across API replicas (auto when URL set) |
| Generation job queue | `ENABLE_JOB_QUEUE=true` + `REDIS_URL` | API enqueues; run `python jobs_worker.py` or `docker compose --profile jobs` |
| Job concurrency | `JOB_QUEUE_MAX_CONCURRENT` | Per worker process |

Priority for `POST /generate`: Temporal (if enabled) → Redis job queue → in-process asyncio task.

## Health

`GET /health` returns `optimization_warnings`, `redis`, `job_queue`, and `rate_limit_backend` when Phase 3 is configured.