Hugging Face's logo

Spaces:
StormShadow308
/
RandomZ
Sleeping

App Files Community
RandomZ / docs /PRODUCTION_REPORT.md
StormShadow308's picture
StormShadow308
feat: async pipeline, job queue, generation hardening, and docs
732b14f
preview code
|
Raw
History Blame Contribute Delete
15.3 kB

Production Report — RICS Report Genius AI

Document purpose: Single reference for production deployment, scale, AI operations, and ownership.
Scope: Current codebase behaviour (FastAPI, SQLite default, FAISS, LangChain, OpenAI), target production on AWS with OpenAI retained.
Audience: Engineering leads, DevOps, AI engineers, backend engineers, frontend engineers, security/compliance.

Table of contents

  1. Executive summary
  2. System architecture (as implemented)
  3. Data & state
  4. AI & LLM pipeline
  5. APIs, concurrency & failure modes
  6. Security & tenancy
  7. Performance & scale
  8. AWS production stack (OpenAI retained)
  9. Observability & operations
  10. Role responsibilities & deliverables
  11. Pre-launch checklist
  12. Post-launch & continuous improvement

1. Executive summary

The application is a FastAPI service serving a static frontend, with per-tenant RAG (FAISS + LangChain embeddings), OpenAI for generation and several post-processing passes, and async SQLite by default. It is fit for pilot or single-tenant–style deployments out of the box. For thousands of concurrent users, production requires:

Pillar Gap in default / local shape Production requirement
Data SQLite + local files PostgreSQL (RDS) + S3 for uploads
Coordination In-process rate limits, per-pod FAISS/cache Redis (ElastiCache) + SQS workers + clear vector strategy
Tenancy X-Tenant-ID header only Authenticated identity mapped to tenant
AI reliability OpenAI rate limits, retries, cost Queues, concurrency caps, timeouts, observability
Ops Ephemeral disk on some hosts Backups, alarms, stuck-job handling

OpenAI remains the LLM provider; AWS provides compute, data, queue, cache, secrets, and logs — not a replacement for OpenAI’s API limits or model behaviour.

2. System architecture (as implemented)

2.1 Runtime

  • Framework: FastAPI (app/main.py).
  • Entry: Uvicorn; Docker image targets port 7860 (HF Spaces) or 8000 (local/docker-compose) per Dockerfile / env.
  • Startup: DB init, vector store pre-warm, embedding client pre-warm; optional KB ingest in background; production guard requires OPENAI_API_KEY when DEV_MODE=false.

2.2 Major components

Component Location / behaviour
HTTP API app/api/* — reports, generate, upload, status, export, photos, etc.
Generation orchestration app/services/generation.py — RAG, style, LLM, verify, retries, persist.
Vector store app/vectorstore/factory.py — FAISS singleton, LangChain-backed.
Embeddings app/embeddings/factory.py — default HuggingFace local; optional OpenAI via config.
Prompts app/generator/prompts.py — LCEL vars, interference levels, token budgets.
LLM adapter app/generator/adapter.py — OpenAI / mock.
Post-process app/generator/postprocess.py — async_enforce_verify, etc.
Section cache app/cache/section_cache.py — filesystem JSON keyed by hash of inputs.
Rate limiting app/api/rate_limit.py — in-process sliding window per tenant_id from header.

2.3 Generation modes (API)

  • generate — Full pipeline (default product path; notes_only_generation forces standard pipeline in current code paths).
  • proofread / enhance — Separate pipelines reusing parts of generation.

2.4 AI interference (product)

  • Qualitative levels: minimum | medium | maximum (interference_level on GenerateRequest; maps to internal ai_percent bands in app/models/schemas.py).
  • Prompts branch on interference level in app/generator/prompts.py.

3. Data & state

3.1 Database

  • Default: sqlite+aiosqlite:///./dev.db (app/config.py).
  • Production risk: SQLite single-writer semantics, file locking, no horizontal write scale — unsuitable for high concurrent write load.
  • Target: Amazon RDS PostgreSQL (or Aurora PostgreSQL) with connection pooling (app uses pool_size / max_overflow for non-SQLite in app/db/database.py).

3.2 Uploads & artifacts

  • Default: Local upload_dir on instance filesystem.
  • Production risk: Ephemeral disks lose data on redeploy/restart; multi-replica inconsistent views of files.
  • Target: Amazon S3 + DB rows storing object keys; presigned URLs for client download where needed.

3.3 Vector index (FAISS)

  • Current: In-process singleton; persisted under app-configured paths (e.g. user home / cache dirs).
  • Production risk: Not shared across replicas; each pod has its own index unless you mount shared storage (generally discouraged for FAISS at scale).
  • Target options:
    • RDS pgvector — vectors in Postgres; one operational system.
    • Amazon OpenSearch kNN — separate service; strong scale/filter story.
    • Single replica only — acceptable only for early internal launch, not “thousands concurrent.”

3.4 Section result cache

  • Current: File-based cache in settings.cache_dir; atomic write via temp + rename.
  • Production risk: Per-node only; no cross-instance deduplication.
  • Target: Redis for hot cache keys (optional) or accept cache miss on each pod.

4. AI & LLM pipeline

4.1 OpenAI usage (representative chain per section)

Depending on settings and branch, a generate path may include:

  • Notes expansion — LLM when OpenAI key present and expansion not skipped (app/generator/notes_expander.py).
  • Main section generation — LangChain LCEL + ChatOpenAI (app/generator/adapter.py).
  • async_enforce_verify — Additional LLM pass when key present (app/generator/postprocess.py).
  • Retries — Verbatim ratio, identity consistency, survey-tier hints — can trigger extra generate_section calls.
  • Optional LLM section validator — llm_section_validator_enabled in app/config.py (default false); when true, extra validate + possible regenerate.
  • Photo / vision — When photos exist and policy triggers (app/services/photo_vision.py).

Implication: Wall-clock and cost scale with number of LLM round-trips, not only “one GPT call per section.”

4.2 Embeddings

  • Default: prefer_local_embeddings=True → HuggingFace sentence-transformers on CPU per process (app/embeddings/factory.py).
  • Production trade-off: Each replica loads model into RAM; cold start and memory multiply with replica count.
  • Mitigation: Set prefer_local_embeddings=false to use OpenAI embeddings API (cost vs RAM); or keep HF on GPU workers dedicated to ingest (heavier ops design).

4.3 Quality & guardrails (behavioural, not infra)

  • Notes-only mode (notes_only_generation, default true): bullets are factual authority; uploads are reference-only in that design; provenance suppressed accordingly.
  • Interference levels encode how much drafting vs mapping is allowed (see app/generator/prompts.py and schemas).
  • Production AI engineering: golden-set evaluation, pinned models, documented limits of each guard (what verify does / does not guarantee).

4.4 Cost & limits (OpenAI)

  • Account-level rate limits and token pricing apply regardless of AWS scale.
  • Mitigation: SQS-backed workers with global + per-tenant concurrency caps, exponential backoff on 429, timeouts, circuit breaker patterns.

5. APIs, concurrency & failure modes

5.1 Report generation locking

  • POST /reports/{report_id}/generate uses a compare-and-swap so Report.status is only flipped to generating if not already generating (app/api/generate.py).
  • Second concurrent generate for same report → 409 until the first completes or fails.
  • Worker crash after CAS but before completion can leave status generating until the stale-generation sweeper marks the report failed (generation_stale_sweep_seconds, generation_started_at on reports). Runs on the API process and on the Temporal worker when enabled.

5.2 Background execution

  • Generation runs as asyncio.create_task from the API process; errors in run_generation set report failed and error_message (app/services/generation.py).
  • Risk: Process kill loses in-flight task; see stuck state above.

5.3 Rate limiting

  • Generate: default 20 RPM per tenant id (header); Read: 120 RPM (app/api/rate_limit.py, env RATE_LIMIT_GENERATE_RPM, RATE_LIMIT_READ_RPM).
  • Limitation: In-process — incorrect under multiple workers/pods (each has its own counter). Production should use Redis or API Gateway throttling.

5.4 HTTP status summary (client expectations)

Code Typical cause
401 Missing / empty X-Tenant-ID on non-exempt paths (app/api/middleware.py).
404 Report or document not found / wrong tenant.
409 Report already generating; or business rule conflicts elsewhere.
413 Payload too large (e.g. section text PATCH limit).
422 Validation (e.g. invalid interference_level).
429 Rate limit exceeded.

6. Security & tenancy

6.1 Current tenant model

  • TenantAuthMiddleware: requires non-empty X-Tenant-ID header; no cryptographic proof of ownership (app/api/middleware.py).

6.2 Production requirement

  • Authenticate users (Cognito, Auth0, custom JWT, API keys behind API Gateway, etc.) and derive tenant_id from trusted claims — do not treat the raw header as the security boundary.

6.3 Secrets

  • OPENAI_API_KEY: required in production when DEV_MODE=false (app/main.py).
  • TENANT_SECRET_KEY: defaults trigger warnings; should be non-sentinel before any future signing use (app/main.py).

6.4 CORS

  • allowed_origins default ["*"] with credentials false in app/main.py — tighten for production web origins.

7. Performance & scale

7.1 Dominant bottlenecks

  1. Sequential LLM calls per section (expand → generate → verify → retries).
  2. OpenAI latency + rate limits.
  3. Database under concurrent writes if SQLite remains.
  4. Per-replica FAISS + HF embeddings — memory and duplicate ingest work.

7.2 Application-level levers (documented in config)

  • Retrieval pool sizes (hierarchical_k_*, retrieval_top_k, rerank_top_n in app/config.py).
  • Optional LLM section validator (off by default).
  • Interference level — Maximum mode increases output tokens and time.

7.3 Architecture-level levers

  • Queue + workers (SQS) for generate jobs.
  • Bounded concurrency per tenant and globally toward OpenAI.
  • Shared vector store (pgvector or OpenSearch) + Postgres OLTP.
  • Redis for rate limits and optional response cache.

8. AWS production stack (OpenAI retained)

Minimal recommended set (OpenAI unchanged):

Service Role
Amazon RDS PostgreSQL Primary OLTP; optional pgvector for embeddings index.
Amazon S3 Uploads, exports, large artifacts.
Amazon ElastiCache Redis Distributed rate limits, locks, optional cache.
Amazon SQS (+ DLQ) Async generation jobs, backoff, worker scaling.
Amazon ECS on Fargate (or EKS) API service + worker service.
Application Load Balancer Routing, health checks.
AWS Secrets Manager OpenAI key, DB credentials.
Amazon CloudWatch Logs, metrics, alarms.

Defer until needed: WAF, Step Functions, X-Ray, OpenSearch (if pgvector suffices), EventBridge (can add for stuck-report sweeper).

9. Observability & operations

9.1 Minimum viable

  • Structured logs (JSON): request_id, tenant_id, report_id, section_code, latency, OpenAI error codes.
  • Alarms: 5xx rate, p95 latency, queue depth, DB connections, OpenAI 429 spikes, stuck generating %.

9.2 Backups & DR

  • RDS automated backups + restore drill.
  • S3 versioning / lifecycle as per compliance.
  • RPO/RTO targets documented.

9.3 Runbooks

  • OpenAI key rotation.
  • Queue poison message (DLQ replay).
  • Stuck report reset procedure until automated sweeper exists.

10. Role responsibilities & deliverables

Role Owns Key deliverables
DevOps / SRE AWS accounts, networking, secrets, CI/CD, scaling, backups RDS, S3, Redis, SQS, ECS/Fargate, ALB, Secrets Manager, CloudWatch alarms, TLS
Backend Application integration with AWS; job lifecycle; vector adapter Postgres schema, S3 ingest, SQS consumer, Redis limiter, pgvector/OpenSearch VectorStore, stuck-job handling, real auth → tenant
AI engineering Models, prompts, evals, cost/latency policy Golden sets, pinned models, retry policy, document guardrail limits, Bedrock not required if staying on OpenAI
Frontend UX for auth, long jobs, errors Bearer/session auth, 429/409 handling, progress against real job status

11. Pre-launch checklist

  • RDS Postgres live; app DATABASE_URL pointed; migrations applied.
  • S3 for uploads; no critical path depends on local disk alone.
  • Redis rate limiter (or gateway limits) — verified under multi-task load.
  • SQS (or equivalent) — worker processes tested; DLQ monitored.
  • Vector strategy chosen — pgvector or OpenSearch or explicit single-replica limitation documented.
  • Auth — tenant cannot be spoofed via header alone.
  • Secrets — OpenAI + DB in Secrets Manager; no keys in images.
  • Stuck generating — sweeper or manual runbook tested once.
  • Load test — representative: upload → ingest → N sections generate; measure p95 and error rate.
  • AI eval — regression set run on candidate prod config.
  • CORS — restricted to real frontends.

12. Post-launch & continuous improvement

  • Review OpenAI spend and 429 dashboards weekly after launch.
  • Tune worker concurrency vs OpenAI tier.
  • Expand golden sets with production-redacted examples.
  • Revisit retrieval k and interference defaults based on measured quality vs latency.

Document control

Field Value
Repository path docs/PRODUCTION_REPORT.md
Intent Living reference — update when architecture or AWS choices change.

End of report.