Speed up generation and ingestion; add live report progress on /status.

feat: async pipeline, job queue, generation hardening, and docs

feat: enhance embedding model configuration and retrieval logic