Disable LLM RAG sanitisation by default on HF Spaces to fix slow ingestion.

One OpenAI call per PDF page was taking 330s+ per document on CPU Spaces; style_corpus uploads still force scrubbing.

Co-authored-by: Cursor <cursoragent@cursor.com>

.env.example

@@ -61,6 +61,8 @@ OPENAI_API_KEY=sk-replace-me
 
 # ── RAG upload sanitisation (strip PII before vector indexing; on-disk files unchanged) ─
 # ENABLE_RAG_UPLOAD_SANITISATION=true
+# On Hugging Face Spaces this defaults to false (one OpenAI call per PDF page → ~5+ min/doc).
+# style_corpus uploads still force sanitisation when enabled in code.
 # RAG_SANITISATION_SKIP_KB_TENANT=true
 # RAG_SANITISATION_CHUNK_CHARS=12000
 # RAG_SANITISATION_MAX_OUTPUT_TOKENS=4096

DEPLOY_HF_SPACES.md

@@ -96,6 +96,8 @@ Optional — override Dockerfile defaults if you want different behaviour:
 | Type | Name | Default | Notes |
 |---|---|---|---|
 | Variable | `KNOWLEDGE_BASE_ENABLED` | `false` | Set `true` *only* if you also bake KB PDFs into the image. The startup guard logs a `WARNING` when this is `false` in production — that's expected on the HF free tier (KB source PDFs are gitignored), so the warning is informational. |
+| Variable | `ENABLE_RAG_UPLOAD_SANITISATION` | `false` | **Keep `false` on HF.** When `true`, every PDF page gets an OpenAI scrub call before indexing (~5+ min per building survey). `style_corpus` uploads still sanitise even when this is `false`. |
+| Variable | `MAX_CONCURRENT_INGESTS` | `2` | Lower than local dev (4) to reduce CPU/RAM pressure on the free tier. |
 | Variable | `INSPECTOR_TOOL_AGENT` | `true` | Disable to force the legacy fixed pipeline. |
 | Variable | `UVICORN_WORKERS` | `1` | Correct for the 1-vCPU HF Spaces free tier. On bigger hosts, set to `(2 * CPU cores) + 1`. |
 | Variable | `RATE_LIMIT_GENERATE_RPM` | `20` | Per-tenant /generate rate limit. |

Dockerfile

@@ -101,6 +101,8 @@ ENV PORT=7860 \
     UVICORN_WORKERS=1 \
     DEV_MODE=false \
     KNOWLEDGE_BASE_ENABLED=false \
+    ENABLE_RAG_UPLOAD_SANITISATION=false \
+    MAX_CONCURRENT_INGESTS=2 \
     DATABASE_URL=sqlite+aiosqlite:////home/user/.report_genius/dev.db \
     FAISS_INDEX_PATH=/home/user/.report_genius/faiss_index \
     UPLOAD_DIR=/home/user/.report_genius/uploads \

app/config.py

@@ -672,12 +672,20 @@ class Settings(BaseSettings):
 
     @model_validator(mode="after")
     def _sync_openai_key_from_process_env(self) -> Self:
-        """HF Space secrets inject OPENAI_API_KEY at runtime (not in the image)."""
+        """HF Space runtime tweaks (secrets + sensible free-tier defaults)."""
         if not os.environ.get("SPACE_ID"):
             return self
         env_key = (os.environ.get("OPENAI_API_KEY") or "").strip()
         if env_key:
             self.openai_api_key = env_key
+        # RAG upload sanitisation runs one LLM call per PDF page before indexing.
+        # On HF CPU Spaces that turns a ~30s ingest into 5+ minutes per file.
+        # Opt in explicitly via ENABLE_RAG_UPLOAD_SANITISATION=true; style_corpus
+        # uploads still force sanitisation regardless of this flag.
+        if "ENABLE_RAG_UPLOAD_SANITISATION" not in os.environ:
+            self.enable_rag_upload_sanitisation = False
+        if "MAX_CONCURRENT_INGESTS" not in os.environ:
+            self.max_concurrent_ingests = min(int(self.max_concurrent_ingests), 2)
         return self
 
 

app/main.py

@@ -259,6 +259,14 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
         except Exception as exc:  # noqa: BLE001
             logger.warning("Redis configured but not reachable at startup: %s", exc)
 
+    if os.environ.get("SPACE_ID"):
+        logger.info(
+            "HF Space ingest policy: rag_upload_sanitisation=%s max_concurrent_ingests=%d "
+            "(set ENABLE_RAG_UPLOAD_SANITISATION=true to opt into LLM scrub per PDF page)",
+            settings.enable_rag_upload_sanitisation,
+            int(settings.max_concurrent_ingests),
+        )
+
     logger.info("Startup complete.")
     yield
     from app.redis_client import close_redis, redis_configured as _redis_cfg