OPTIMIZATION_SUMMARY.md

Latency optimization summary (Phase 1–2)

Phase 1 — Async pipeline, retrieval, concurrency

Feature	Env flags	Notes
Async LLM	ENABLE_ASYNC_PIPELINE=true	AsyncLLMAdapter + throttled OpenAI calls
Qdrant vector store	VECTORSTORE_BACKEND=qdrant, QDRANT_URL	Re-ingest after migration: python scripts/reingest_qdrant.py --tenant-id …
Hybrid BM25 + RRF	ENABLE_HYBRID_RETRIEVAL=true	Qdrant only
Semantic retrieval cache	SEMANTIC_CACHE_ENABLED=true	Qdrant collection QDRANT_CACHE_COLLECTION
Parallel multi-section	Same as async + Postgres or ALLOW_SQLITE_PARALLEL_SECTIONS=true	REST + agentic use multi_section_parallel_enabled()
LLM throttle	MAX_CONCURRENT_LLM_CALLS	Global semaphore + retry on 429
Stale job recovery	GENERATION_TIMEOUT_SECONDS, GENERATION_STALE_SWEEP_SECONDS	Uses reports.generation_started_at

Phase 2 — Speculation, prompt cache, Temporal

Feature	Env flags	Notes
Speculative tools	ENABLE_SPECULATIVE_EXECUTOR=true	Requires async pipeline; inspector loop only
Prompt cache	ENABLE_PROMPT_CACHING=true	System prefix pad + prompt_cache_key (includes tenant)
Temporal workflows	ENABLE_TEMPORAL_WORKFLOW=true	docker compose --profile temporal up; python worker.py
Workflow ID	Response field workflow_id on POST /generate when Temporal starts

Inspector vs standard generate

• Default NOTES_ONLY_GENERATION=true uses the standard RAG pipeline on POST /generate (no inspector speculation).
• Set AGENTIC_INSPECTOR_WHEN_NOTES_ONLY=true or NOTES_ONLY_GENERATION=false to run the inspector on main generate.
• POST /reports/{id}/agentic/generate persists sections and report status like REST generate.

Phase 3 — Redis coordination (scale-out)

Feature	Env flags	Notes
Distributed rate limits	REDIS_URL	Shared sliding window across API replicas (auto when URL set)
Generation job queue	ENABLE_JOB_QUEUE=true + REDIS_URL	API enqueues; run python jobs_worker.py or docker compose --profile jobs
Job concurrency	JOB_QUEUE_MAX_CONCURRENT	Per worker process

Priority for POST /generate: Temporal (if enabled) → Redis job queue → in-process asyncio task.

Health

GET /health returns optimization_warnings, redis, job_queue, and rate_limit_backend when Phase 3 is configured.