Spaces:
Sleeping
Sleeping
| # Data Security & Privacy — RICS Report Generator | |
| **Audience:** Internal engineering (DevOps + backend), founder for client conversations and webinars. | |
| **Status:** Working document. Decisions outstanding for the Friday meeting are flagged with **[DECIDE]**. | |
| **Last reviewed:** 14 May 2026 | |
| ### How to read this document | |
| | Term | Meaning here | | |
| |---|---| | |
| | **This repository / this app** | The FastAPI Python service under `app/` in this git repo—not the customer’s VPN, cloud control plane, or desktop browsers unless we say so. | | |
| | **Customer / firm** | The organisation that deploys the software and owns the disk and database. | | |
| | **Tenant** | Logical partition in our data model (`tenant_id`). Isolation between tenants is enforced in API and FAISS code paths; **per-user ACL inside one tenant is not implemented** (see §9). | | |
| | **Tier A** | Direct OpenAI API with standard commercial logging/retention as published by OpenAI at the time. **Matches shipped code today** when `OPENAI_API_KEY` is set. | | |
| | **Tier B** | Tier A plus **Zero Data Retention** on the OpenAI **organisation** (OpenAI programme + endpoint rules). **Not a separate code path yet** beyond what OpenAI enforces; optional `zdr_enforced`-style guardrails are listed in §10 and are **not** present in code as of last review. | | |
| | **Tier C** | Customer inference against **Azure OpenAI** in the customer’s Microsoft tenant (e.g. UK South). **Roadmap: not implemented** in this repo (no Azure client or settings fields). | | |
| | **Tier D** | Inference against a **self-hosted** OpenAI-compatible HTTP API (e.g. Ollama, vLLM). **Roadmap: not implemented** in this repo. | | |
| | **Vendor claims** | Anything about training, retention, regions, or subprocessors is **only** as accurate as the linked page on the day you read it—this file is not legal advice. | | |
| --- | |
| ## 1. Two-paragraph summary you can use in any conversation | |
| Surveyors trust us with property addresses, owner names, valuations, and condition findings — every report contains personal data and commercially sensitive information. Our job is to give them a clear, defensible answer to one question: **where does this data go?** | |
| **Customer content (uploads, database, FAISS index) stays on infrastructure the customer controls when they self-host or run in their own cloud account.** **OpenAI’s HTTPS API** is used for LLM features (and **optionally for embeddings** under the exact rules in **§2.2**). Under OpenAI’s published terms, **API data is not used to train models by default**, and **retention is time-limited** (see **§3** — always confirm the live policy pages). **Other HTTPS traffic** can still occur from **dependencies** (for example, downloading HuggingFace model weights on first use); our `app/` package does not add separate `requests`/`httpx` integrations for analytics or third-party vector DBs (**§2.1** search proof). For stronger guarantees we describe **roadmap tiers**: Zero Data Retention (OpenAI programme), Azure OpenAI (not implemented in this repo yet), and self-hosted models (not implemented yet)—each needs explicit engineering and operations work beyond flipping a single flag. | |
| --- | |
| ## 2. Where data goes in this repository (code-backed) | |
| This section answers: **which bytes leave the host because our application asked them to**, and **where we store data locally**. It is based on reading `app/` and `app/embeddings/factory.py` at **last review (14 May 2026)**. It **excludes** infrastructure you add yourself (cloud logging, APM, reverse proxies, SIEM, Windows Update, Docker registry pulls, etc.). | |
| ### 2.1 Data this app keeps on the host (no OpenAI client for these artefacts) | |
| These features read and write customer data **only** on the deployment host using local disk + local SQL + in-process Python. They do **not** import `openai.OpenAI`, `langchain_openai.ChatOpenAI`, or `OpenAIEmbeddings` for the listed job. | |
| | Component | Where in code | Default on-disk / DB location (from `app/config.py` unless env overrides) | | |
| |---|---|---| | |
| | Uploaded reports (PDF/DOCX) | `app/api/upload.py`, `app/ingest/*` | `upload_dir` → `~/.report_genius/uploads` | | |
| | Inspector notes | `GenerateRequest.bullets` and related ORM models | Same database as other report fields | | |
| | Vector index | `app/vectorstore/faiss_wrapper.py` | `faiss_index_path` → `~/.report_genius/faiss_index` | | |
| | SQL database | All persistence through SQLAlchemy / settings | `database_url` → `sqlite+aiosqlite:///./dev.db`; production usually overrides with Postgres via `DATABASE_URL` | | |
| | Ingestion, chunking, FAISS search, retrieval orchestration | `app/ingest/*`, `app/retrieval/*`, vectorstore | Same host; **no** `requests` / `httpx` / `urllib.request` usage in `app/**/*.py` for these code paths | | |
| **What we proved by search:** `rg "requests\\.(get|post)|httpx\\.|urllib\\.request" app/` → **zero matches**. So this repo does **not** ship first-party calls to random analytics hosts, ad networks, or hosted vector DBs. | |
| **What that search does *not* prove:** the Python process cannot make HTTPS calls. Dependencies (`openai`, `langchain_openai`, `langchain_huggingface`, `sentence-transformers`, etc.) use their own TLS stacks. | |
| ### 2.2 Embeddings — exact decision table (`app/embeddings/factory.py`) | |
| `get_embedding_client()` builds **one** global `Embeddings` instance using `Settings.prefer_local_embeddings` and `Settings.openai_api_key`: | |
| | `prefer_local_embeddings` | Order tried | OpenAI receives text chunks for embedding when | | |
| |---|---|---| | |
| | `True` (default) | ① `HuggingFaceEmbeddings(model_name=local_embedding_model)` — default model **`all-MiniLM-L6-v2`**. ② If ① returns `None` (missing deps / import failure): `OpenAIEmbeddings(model=embedding_model)` **if** `openai_api_key` is non-empty. ③ If still nothing: `FakeEmbeddings` (test / broken install). | Only when ① failed **and** ② succeeded. | | |
| | `False` (legacy) | ① `OpenAIEmbeddings` if key present. ② else HuggingFace. ③ else Fake. | Whenever ① succeeds with a key. | | |
| When OpenAI embeddings run, the model name is `Settings.embedding_model`, default **`text-embedding-3-small`**. | |
| **First-time HuggingFace use:** downloading `all-MiniLM-L6-v2` weights is **HTTPS to Hugging Face Hub (or a mirror)**. That traffic is **public model weights**, not a copy of the customer’s report archive sent as “training data” to OpenAI. | |
| ### 2.3 OpenAI chat, vision, and tools — complete production module list | |
| Every **non-test** file under `app/` that constructs `openai.OpenAI` or `langchain_openai.ChatOpenAI` for real inference: | |
| | File | When it runs | What OpenAI sees (plain English) | | |
| |---|---|---| | |
| | `app/generator/adapter.py` | `openai_api_key` non-empty and caller uses generate / proofread / enhance / weave | Prompts: section skeleton, bullets, RAG standard passages, instructions from `app/generator/prompts.py` | | |
| | `app/generator/notes_expander.py` | Key set; LLM expansion path chosen | Raw inspector bullets | | |
| | `app/generator/style_analyzer.py` | Key set; style profile requested | Paragraph samples from uploaded firm documents | | |
| | `app/generator/vision_analyzer.py` | Key set; vision path not skipped | **Pixels** (encoded image) + vision instructions | | |
| | `app/generator/postprocess.py` | Key set; LLM grounding branch taken | Draft section text + grounding snippets | | |
| | `app/services/photo_vision.py` | Key set; batch photo analysis runs | **Pixels** + instructions | | |
| | `app/services/canonical_rollout.py` | Key set; LLM merge helper runs | Two competing text bodies for merge | | |
| | `app/agentic/inspector_loop.py` | Key set; agentic pipeline enabled | Full inspector loop: notes, seeds, tool JSON, partial completions | | |
| | `app/agentic/agents.py` | Key set; JSON risk helper used | Bullets / short context block passed into that helper | | |
| `app/services/generation.py` **orchestrates** the above; it does not construct its own parallel OpenAI client for those features. | |
| **Model names in config (`app/config.py`):** `chat_model` default **`gpt-4o-mini`** (main adapter). `inspector_body_model` default **`gpt-4o`** (inspector loop body drafting). | |
| **Azure:** There is **no** `AzureOpenAI` client and **no** Azure endpoint settings in this repository today. Tier C is documentation only until we ship that code. | |
| ### 2.4 Network egress diagram (application-designed paths only) | |
| ``` | |
| ┌─────────────────────────────────────────────────────────────┐ | |
| │ Surveyor firm's environment │ | |
| │ │ | |
| │ ┌─ Web UI │ | |
| │ │ │ | |
| │ ▼ │ | |
| │ FastAPI ──► SQLite/Postgres │ | |
| │ │ │ | |
| │ ├──► File storage (PDFs, DOCX) │ | |
| │ │ │ | |
| │ ├──► FAISS index (local disk) │ | |
| │ │ │ | |
| │ ├──► HuggingFace MiniLM-L6-v2 (local, on CPU/GPU) │ | |
| │ │ (first run may download weights — HTTPS) │ | |
| │ │ │ | |
| │ └──► OpenAI Python client ───────┐ TLS 1.2+ │ | |
| │ │ (primary LLM API) │ | |
| └──────────────────────────────────────────│──────────────────┘ | |
| │ | |
| ▼ | |
| api.openai.com (OpenAI-hosted) | |
| ``` | |
| **Primary egress for customer *content* designed by this app:** TLS to **OpenAI** for chat/vision/tools when `OPENAI_API_KEY` is set and those code paths run; plus **OpenAI** for embeddings **only** in the §2.2 cases. **Other TLS the app indirectly causes:** Hugging Face Hub (or a mirror) when sentence-transformers downloads `local_embedding_model` weights. | |
| **Explicitly out of this diagram:** cloud provider metadata APIs, log shippers, antivirus updates, DNS, NTP, container registries—**your** platform team owns that surface. | |
| --- | |
| ## 3. OpenAI's published data policy (vendor — re-read before contracts) | |
| The bullets and quotes below are **summaries or verbatim excerpts** from OpenAI’s public pages linked in **§11**, current as of **May 2026** when this file was written. **They are not legal advice** and they **will** drift—before a customer signature, open each URL and read the live text. | |
| **Relationship to our code:** Training/retention rules are enforced by **OpenAI’s platform and your contract**, not by anything we can “prove” inside `app/`. | |
| ### 3.1 Training | |
| > "By default, OpenAI does not use data from ChatGPT Enterprise, ChatGPT Business, ChatGPT Edu, ChatGPT for Healthcare, ChatGPT for Teachers, or their API platform — including inputs or outputs — for training or improving their models." | |
| > — [openai.com/enterprise-privacy](https://openai.com/enterprise-privacy/) | |
| OpenAI states this default applied from **1 March 2023** onward (API platform included in the quoted list). **Practical meaning for us:** any customer using our integration against the **OpenAI API** should verify their **exact** OpenAI product (Business vs consumer ChatGPT vs API) on the live page—this document treats **API / Business** as the relevant class for this codebase. | |
| ### 3.2 Retention | |
| > "After 30 days, API inputs and outputs are removed from OpenAI logs, unless legally required to retain them." | |
| > — [openai.com/enterprise-privacy](https://openai.com/enterprise-privacy/) | |
| OpenAI describes this window as tied to **abuse monitoring** and trust-and-safety workflows. **Operational detail:** assume prompts/responses **may be processed or sampled** by automated systems and, where their policy allows, humans during the stated retention—read their current abuse-monitoring explanation rather than assuming “nobody ever sees it.” | |
| ### 3.3 Zero Data Retention (ZDR) | |
| > "If you are a business customer that uses OpenAI's Zero Data Retention (ZDR) API, OpenAI never retains the prompts you send or the answers returned." | |
| > — [platform.openai.com/docs/guides/your-data](https://platform.openai.com/docs/guides/your-data) | |
| **Operational caveats (read OpenAI’s ZDR page before promising anything):** | |
| 1. **Enrolment:** ZDR is **not** automatic for every API key—it is a **programme** you apply for on the OpenAI org. | |
| 2. **Endpoint compatibility:** Some platform features (e.g. extended prompt caching, background mode, Assistants API with threads, certain file/Batch flows) are called out by OpenAI as **incompatible** or restricted with ZDR—our engineering checklist is in **§10**. | |
| 3. **Legal overrides:** Court orders, litigation holds, or government demands can still force retention **outside** normal commercial deletion timelines. ZDR narrows **contractual** logging; it does not create immunity from law. | |
| **Why we still want ZDR:** when eligible and correctly configured, it removes the **routine 30-day log retention** described for standard API processing—again, verify the **current** ZDR description on OpenAI’s site. | |
| ### 3.4 Data Processing Addendum | |
| The DPA text is published by OpenAI and is **legally binding only after** your organisation completes whatever acceptance / order flow OpenAI requires for **Business / API** use. This document does **not** record your signature status—legal owns that record. | |
| The published DPA covers (high level; read the PDF): | |
| - GDPR Article 28 processor obligations (OpenAI = Processor, our firm clients = Controller) | |
| - Standard Contractual Clauses (SCCs) for EU transfers | |
| - UK IDTA Addendum for UK transfers | |
| - Sub-processor general authorisation with change-notification | |
| - Breach notification, data-subject-request assistance | |
| - HIPAA BAA available on Enterprise/Scale tiers (not relevant to RICS but indicates depth) | |
| Source: [openai.com/policies/data-processing-addendum](https://openai.com/policies/data-processing-addendum/), PDF v.010126 (effective 1 Jan 2026). | |
| ### 3.5 Sub-processors (where data physically lives) | |
| OpenAI uses Microsoft Azure as its primary infrastructure, with Google Cloud, Oracle Cloud, and CoreWeave for additional compute. Other sub-processors handle support, payments, and trust-and-safety review. The authoritative list is [openai.com/policies/sub-processor-list](https://openai.com/policies/sub-processor-list/) — we should reference it as a URL rather than copying it (the list changes). | |
| **Material point for UK surveyors (direct OpenAI API — Tier A):** OpenAI’s **sub-processor list** shows US and other jurisdictions. A UK controller using the **public** OpenAI API therefore typically relies on **SCCs + UK IDTA** (or equivalent) for **international transfers** to OpenAI US operations—exact mechanism is in OpenAI’s DPA. | |
| **What Tier C changes (once we ship it):** inference runs in the **customer’s Microsoft Azure subscription** in a region the customer selects (e.g. **UK South** for London). Data still goes to **Microsoft** as processor; it does **not** remove the need for a DPA—it **changes** the processor and the **geographic / network boundary** compared with calling `api.openai.com` from Tier A. Always validate region, private link, and key management against the **current** Azure OpenAI documentation before promising a map pin. | |
| ### 3.6 Encryption in transit and at rest (OpenAI direct API — Tier A) | |
| - **Transit:** TLS 1.2+ between our Python process and OpenAI’s API hosts (per OpenAI’s security documentation). | |
| - **At rest (on OpenAI’s side):** OpenAI documents **AES-256** for stored artefacts **during whatever retention window applies** to your account tier (standard vs ZDR). **Customer-managed keys for data at rest inside OpenAI’s direct API** are **not** a documented feature today—if a customer demands CMK for inference payloads, that is a **Tier C (Azure)** conversation. | |
| --- | |
| ## 4. The four-tier privacy ladder | |
| | Tier | Shipped in this repository today? | What changes vs previous tier | | |
| |---|---|---| | |
| | **A** | **Yes** — direct OpenAI HTTPS calls when `OPENAI_API_KEY` is set | Baseline commercial API. | | |
| | **B** | **Partially** — still Tier A code paths; ZDR is an **OpenAI org configuration** + legal paperwork, not a different Python SDK | Same HTTPS destination; **contractual** logging/retention rules change per OpenAI ZDR programme. Optional app guardrails in §10 are **not implemented** yet. | | |
| | **C** | **No** — requires new settings + client factory + refactors listed in §10 | HTTPS goes to **customer’s Azure OpenAI endpoint** instead of `api.openai.com`. | | |
| | **D** | **No** — requires local OpenAI-compatible HTTP endpoint + same factory refactors | HTTPS goes to **customer-controlled** inference server (or stays on loopback). | | |
| The narrative below (pros/cons/Q&A) is for **sales and architecture planning**. **Engineering truth** is the table above. | |
| ### Tier A — Standard OpenAI API (where we are today) | |
| **Posture (plain English):** Our Python service sends prompts to **OpenAI’s API** under whatever **OpenAI Business / API terms** you have signed. OpenAI’s public pages state that **API inputs/outputs are not used to train models by default** and describe **time-bounded retention** for logs—see **§3** for quotes and links. | |
| **Pros:** No Azure or GPU capex for LLM inference; swap `chat_model` / `inspector_body_model` in `app/config.py` when OpenAI releases newer chat models; billing is usage-based. | |
| **Cons:** Prompts and completions leave the firm’s network boundary and are processed in OpenAI’s multi-tenant cloud. OpenAI’s documentation and sub-processor list describe **US and other** processing locations for parts of the service—**not** “UK-only” on Tier A. | |
| **What a surveyor's compliance team will ask (short answers — verify on live OpenAI pages):** | |
| - Is it used to train models? → **OpenAI says “not by default” for the API platform** (see §3.1 quote). | |
| - How long is it stored? → **OpenAI publishes ~30-day log retention for standard API processing** with carve-outs for legal holds (see §3.2 quote). | |
| - Is there a DPA? → **OpenAI publishes a DPA for business use** — your legal team must confirm it is **executed for your org** (§3.4). | |
| - Where physically? → **See OpenAI’s current sub-processor / infrastructure disclosures** — not a single static country list in this file. | |
| - Can humans see prompts? → **During any retention window OpenAI defines for abuse monitoring, assume automated and (where their policy permits) human review is possible** — read their abuse-monitoring pages; do not promise “no human ever.” | |
| **Cost:** Order-of-magnitude **$0.001–0.005 per section generation** at public `gpt-4o-mini` list pricing—recalculate from OpenAI’s **current** pricing page before quoting a customer. | |
| ### Tier B — Standard OpenAI API + Zero Data Retention + signed DPA | |
| **Posture:** Identical **Python code paths** as Tier A. The difference is **contract + OpenAI org settings**: you must (a) have a **signed** Business DPA on file, (b) be **accepted into OpenAI’s ZDR programme** for that org, and (c) restrict product features to **ZDR-allowed** endpoints only. | |
| **Pros:** If OpenAI’s ZDR terms apply to your org as configured, **routine long-lived log retention of prompts/completions is removed** per OpenAI’s ZDR description. Tokens are still billed the same way as Tier A. | |
| **Cons:** You must maintain eligibility (endpoint restrictions). Legal/compliance must read OpenAI’s **current** ZDR rules—this file is not the contract. | |
| **Implementation work (split by owner):** | |
| | Owner | Task | | |
| |---|---| | |
| | **Legal / admin** | Apply for ZDR in the OpenAI org dashboard; store executed DPA + ZDR confirmation. | | |
| | **Engineering (recommended, not done yet)** | Add `openai_zdr_enforced` (name TBD) boolean + startup validation that refuses known ZDR-incompatible SDK features (checklist §10). | | |
| | **Customer success** | Publish which OpenAI org ID / region / model names the deployment uses so customers can mirror it in their vendor reviews. | | |
| **Technical note:** ZDR enforcement happens **inside OpenAI’s platform** once enabled. Our repository **does not yet contain** the guardrail flag described in §10—today we rely on OpenAI’s org settings plus careful endpoint choice. | |
| **Recommendation:** **Pursue ZDR for every production OpenAI org** that serves paying surveyors. Cost at inference time is **$0 extra**; benefit is **narrower retained footprint on OpenAI’s side** per their ZDR materials. | |
| ### Tier C — Azure OpenAI Service (roadmap — not in this repository) | |
| **What Tier C is:** After we implement it, the Python service will call **`{customer}.openai.azure.com`** (or regional equivalent) with the customer’s **own** API key / managed identity, instead of calling **`api.openai.com`**. The **customer’s** Microsoft Customer Agreement / Online Services Terms + Microsoft’s Azure OpenAI privacy pages govern inference—not OpenAI’s **direct** consumer API terms. | |
| **Relationship to Microsoft 365 Copilot:** Copilot is a **separate Microsoft 365 product** with its own privacy statement. Microsoft also offers **Azure OpenAI** as a platform service. **Marketing must not claim “we are Copilot.”** Accurate line: *“Same **platform family** Microsoft uses for enterprise AI in Azure; Copilot-specific data flows are documented on Microsoft’s Copilot privacy pages.”* | |
| **Microsoft-published commitments (Azure OpenAI — read the live page; excerpt below can drift):** | |
| > "Your prompts (inputs) and completions (outputs), your embeddings, and your training data: | |
| > - are NOT available to other customers. | |
| > - are NOT available to OpenAI or other Azure Direct Model providers. | |
| > - are NOT used by Azure Direct Model providers to improve their models or services. | |
| > - are NOT used to train any generative AI foundation models without your permission or instruction." | |
| > — [Azure OpenAI — Data, privacy, and security](https://learn.microsoft.com/en-us/azure/ai-foundry/responsible-ai/openai/data-privacy) | |
| **Infrastructure fact (still read Microsoft docs):** Model weights for Azure OpenAI deployments are served from **Microsoft-operated** Azure regions. Microsoft documents that this path does **not** send your prompts to the **public ChatGPT service** or to **OpenAI’s consumer endpoints** as part of that product architecture—verify wording on the page above. | |
| **Regions (examples Microsoft lists for deployments — confirm in Azure portal before sales):** UK South (London), West Europe, North Europe, Sweden Central, France Central, Germany West Central, Switzerland North, etc. **Strict “UK-only processing”** means: create the Azure OpenAI resource in **UK South**, pick a **regional** (non-global) deployment SKU per Microsoft’s current guidance, and review networking diagrams with the customer’s cloud architect—**we do not guarantee routing** in this markdown file. | |
| **Compliance Q&A (each answer must be checked against the linked Microsoft page the week you pitch it):** | |
| | Question | Where to read the authoritative answer | Short pointer (not a substitute for the doc) | | |
| |---|---|---| | |
| | Training use of customer content? | Same “Data, privacy, and security” page (quoted block above) | Microsoft states **no training** on customer prompts/completions without permission, for the described Azure OpenAI / Azure Direct Models configuration. | | |
| | Physical region? | Azure region picker + Microsoft region docs | Customer chooses region (e.g. **UK South**). | | |
| | OpenAI employees viewing prompts? | Same privacy page + OpenAI’s role statement there | Microsoft states prompts/completions are **not** made available to OpenAI for the Azure OpenAI service model described. | | |
| | Abuse monitoring / human review? | [Abuse monitoring for Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/abuse-monitoring) | Microsoft may perform automated abuse detection; **Modified Abuse Monitoring** is the documented programme to reduce certain reviews—eligibility is Microsoft-gated. | | |
| | Encryption at rest + CMK? | [Encrypt Azure OpenAI data at rest](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/encrypt-data-at-rest) | Microsoft documents **AES-256** by default and **customer-managed keys** via Key Vault for supported scenarios—confirm feature flags for the exact SKU. | | |
| | Private network? | [Network isolation / private endpoint docs](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/on-your-data-configuration) | Microsoft supports **private endpoints** and disabling public access **when configured**—not automatic. | | |
| | Contract? | Customer’s existing Microsoft agreements | Often the **Microsoft Products and Services Data Protection Addendum (DPA)** applies—legal must confirm which agreement is in force. | | |
| **Pros (why customers ask for Tier C):** Data processing moves to **Microsoft Azure** under customer-controlled regions and optional private networking/CMK—reduces reliance on **OpenAI direct US API** subprocessors from §3.5. | |
| **Cons:** ~**20–30% higher per-token list price** than direct OpenAI (historical ballpark—re-quote from Azure pricing). Customer must own Azure subscription + quota approvals. We must maintain a **second HTTP client** (Azure OpenAI) and regression-test every call site in §2.3. | |
| **Implementation work (engineering backlog — not started):** | |
| 1. Add `AzureOpenAI` client module + Pydantic settings (`azure_openai_endpoint`, `azure_openai_api_version`, `azure_openai_deployment_name`, optional `azure_openai_api_key` or managed-identity flags—**exact field names to be chosen at implementation time**). | |
| 2. Central factory returns either `OpenAI` or `AzureOpenAI` compatible chat/embeddings clients. | |
| 3. Refactor every file listed in **§2.3** to consume the factory. | |
| 4. Write customer runbook: resource group, UK South OpenAI resource, private endpoint, Key Vault CMK, RBAC, logging exclusions. | |
| **Cost model (illustrative only — recompute before contracts):** Same rough order of magnitude as earlier draft (~$25/mo inference + ~$15/mo private endpoint/Key Vault for a small tenant) — **treat numbers as Fermi estimates**, not quotes. | |
| **This is the architecture to recommend** when a prospect’s written requirement is **“model calls must land in our Microsoft tenant / UK Azure region.”** It is **not** something they can buy from us **today** out of this repository without custom integration work. | |
| ### Tier D — Self-hosted open-weight model inside the firm's perimeter | |
| This is what many **on-premises LLM** pitches advocate: **customer report content** is not sent to OpenAI or another cloud LLM API when inference runs entirely on infrastructure the customer controls. | |
| **Models to consider** (as of May 2026): | |
| | Model | Params | Notes | | |
| |---|---|---| | |
| | Google Gemma 3 27B / 4 26B (when GA) | 27B / 26B | Strong general reasoning, instruction-tuned, permissive Gemma licence | | |
| | Meta Llama 3.3 70B | 70B | Excellent quality, Llama Community licence (free for <700M MAU) | | |
| | Mistral Large 2 / Codestral | varies | EU-headquartered vendor, Mistral Research Licence | | |
| | Qwen 2.5 72B / Qwen 3 | 72B / varies | Strong Chinese-vendor model, Apache 2.0 for most variants | | |
| | Phi-3 / Phi-4 (small) | 3.8B / 14B | Microsoft, MIT licence, fast | | |
| For RICS report generation specifically, **the model needs to handle ~20-30k tokens of context** (uploaded report excerpts + notes + schema). Gemma 3 27B, Llama 3.3 70B, and Qwen 2.5 72B all handle this comfortably with 128k context windows. | |
| **Target posture:** No cloud **LLM vendor** receives customer prompts or completions. Model weights are local; inference is a local process. **Operational reality:** OS updates, package mirrors, NTP, monitoring agents, or downloading weights can still create network traffic unless you engineer an air-gapped image—that is **deployment** work, not something this application disables by itself. | |
| **What a surveyor's compliance team will ask:** | |
| - Where does inference run? → **On VMs or bare-metal the customer pays for**, inside their network boundary (or a colo they contract with). | |
| - Is it used to train anything? → **Not in the SaaS sense** — open-weight files are static; **no** gradient updates to the base model happen during normal chat inference. (If you fine-tune on customer data, that is a **separate** data decision.) | |
| - Who can see prompts? → **Anyone with OS / hypervisor / backup access to the host** can, in principle, read RAM or disk—same as any sensitive workload. **No** separate OpenAI/Microsoft LLM API receives the text **when** you truly keep calls on-loopback to your own server. | |
| - Encryption? → **Full-disk encryption, TLS inside the cluster, and key management** are the customer’s controls—same as running Postgres or HR systems on-prem. | |
| **Pros:** | |
| - **Sales-accurate headline:** Customer report text is **not** sent to OpenAI’s public API **for inference** when every LLM call stays on an endpoint the customer controls. | |
| - **Cost shape:** Predictable GPU hire or amortised hardware vs per-token cloud bills—magnitude depends on concurrency; **do not** treat viral “10× cheaper” posts as benchmarks without your own measurement run. | |
| - **Vendor surface:** No OpenAI/Azure LLM **subscription** for inference; you still owe diligence on the model **licence**, security patches, and supply chain of the weight files. | |
| **Cons:** | |
| - Significant infrastructure investment: a single NVIDIA H100 or two A100s per tenant for a 27B model at INT4 quantisation. ~£25k–£50k capex per host, or ~£3–6/hour on cloud GPU rental. | |
| - We become the MLOps team — model patching, scaling, monitoring, drift detection, evaluation runs. | |
| - Quality risk: frontier cloud models still win on **some** long reasoning and edge-case refusal tests. For **RICS section drafting with heavy RAG**, the practical gap must be measured with **your** prompts—run a blinded QA review before promising parity. | |
| - Cold-start onboarding: each firm needs hardware procurement or a managed deployment. | |
| **Hybrid Tier D variant — managed-but-isolated (commercial concept, not shipped):** A hosting provider runs **dedicated** GPU + DB + vector index **per customer** so prompts never cross tenant boundaries. Vendors such as RunPod, Together, or Modal sell variations of this **isolated tenancy** idea. **This repository does not include** Kubernetes manifests, per-tenant GPU schedulers, or billing for that model—it is listed here only so sales knows the category exists. | |
| **Implementation work:** | |
| 1. Add a **`LocalLLM` client path** (thin wrapper around an OpenAI-compatible HTTP API such as **Ollama** or **vLLM**). | |
| 2. Add config: `local_llm_endpoint`, `local_llm_model_name`, `local_llm_api_key` (if required). | |
| 3. Use the same **factory refactor** described for Tier C so all LLM calls can target local base URLs. | |
| 4. Re-tune prompts for smaller models; run evaluation against the current GPT-4o-mini baseline before promising parity. | |
| 5. Document the per-tenant deployment topology (hardware spec, network policy, model patching). | |
| **Cost (order-of-magnitude planning numbers — re-price before any quote):** | |
| - **Capex path:** A single high-memory GPU server suitable for a 27B-class INT4 model is commonly quoted in the **low tens of thousands of GBP** capital range before power and datacentre fit-out. | |
| - **Rent path:** Cloud GPU **spot / on-demand** pricing fluctuates daily; budget **single-digit GBP per GPU-hour** for H100-class cards as a Fermi check, then multiply by expected surveyor concurrency and session length. | |
| - **Throughput rule of thumb:** One H100-class GPU can comfortably serve **roughly single-digit concurrent interactive surveyors** for ~20–30k-token contexts **if** batching is tuned—run your own load test; do not treat this as an SLA. | |
| **When Tier D wins on total cost:** Usually when **many** tenants or **very high** monthly token volume amortises fixed GPU cost. Below that crossover, Tier A/B stays cheaper because you are not paying for idle silicon. | |
| --- | |
| ## 5. Recommended path (decision tree for sales) | |
| Use the ASCII tree below as a **conversation map**, not a promise of SKU availability: | |
| ``` | |
| "Where does my data go?" | |
| │ | |
| ┌──────────────────┼──────────────────┐ | |
| ▼ ▼ ▼ | |
| "We need it gone "We use Microsoft "Standard SaaS, | |
| from the 365 / Azure already. cost-effective, | |
| internet." Compliance team will compliance team | |
| prefer that boundary." is comfortable | |
| with OpenAI." | |
| │ │ │ | |
| ▼ ▼ ▼ | |
| TIER D TIER C TIER A + ZDR | |
| Self-hosted Azure OpenAI (Tier B) | |
| open-weight (UK South) | |
| (Gemma/Llama) + CMK + private | |
| endpoint | |
| ``` | |
| **Default commercial posture to aim for:** **Tier B (Standard OpenAI + ZDR)** once OpenAI approves ZDR for your org and you have audited endpoint compatibility. Until then, describe Tier B as the **target** baseline, not something already switched on in every deployment. | |
| **Premium offering (roadmap): Tier C.** Pitch this as the **Microsoft/Azure boundary** pattern for firms already on M365/Azure—**after** the Azure OpenAI integration exists in product. | |
| **Enterprise / on-premises offering (roadmap): Tier D.** For firms who insist on no public-cloud LLM. Higher setup cost; quote per deployment. | |
| --- | |
| ## 6. Microsoft 365 Copilot vs Azure OpenAI vs this product | |
| **Three different things:** | |
| | Name | What it is | Relationship to this repo | | |
| |---|---|---| | |
| | **Microsoft 365 Copilot** | A **Microsoft 365** end-user product (Word/Outlook/etc. integrations) with its own privacy statement. | **We are not Copilot.** We do not ship Office add-ins or Microsoft Graph connectors for Copilot in this repository. | | |
| | **Azure OpenAI Service** | A **Microsoft Azure** platform API for GPT-family models inside a customer subscription. | **Roadmap integration (Tier C).** When shipped, our FastAPI service would call the customer’s Azure endpoint instead of `api.openai.com`. | | |
| | **This RICS report generator** | A **Python FastAPI** application in this git repo. | **Shipped today:** Tier A paths to OpenAI when a key is set (see **§2.3**). **Not shipped:** Azure adapter, local LLM adapter, ZDR guardrail flag. | | |
| **Approved explanation for non-technical buyers:** “Microsoft publishes **separate** privacy documentation for **Copilot** and for **Azure OpenAI**. If your procurement rule is *‘model calls must stay inside our Azure tenant,’* that maps to **Azure OpenAI (Tier C)**—an integration we have **specified** in §4 but **not coded** yet. If your rule is *‘must match Copilot,’* read the **Copilot** privacy page literally; it is a different product.” | |
| **Approved explanation once Tier C exists:** “We call **your** Azure OpenAI deployment in **your** chosen region (for example UK South), optionally with private endpoints and customer-managed keys exactly as described in Microsoft’s Azure OpenAI documentation linked in §4.” | |
| --- | |
| ## 7. Friday meeting — concrete decisions to make | |
| Bring printed copies of this document. Walk through the four tiers. Ask the engineers: | |
| 1. **[DECIDE]** Do we apply for OpenAI Zero Data Retention this week? (Tier B). Estimated effort: 30 minutes of paperwork + a one-day audit to confirm we use no ZDR-incompatible endpoints. **Recommendation: yes, no downside.** | |
| 2. **[DECIDE]** Do we build **Azure OpenAI support** (client factory + config) and start offering Tier C as a paid premium tier? Estimated effort: 1 engineer × ~1–2 weeks for the factory + call-site refactor + docs + Azure subscription cost during testing. **Recommendation: yes, but commit only after we have 1 prospect who has asked for it.** | |
| 3. **[DECIDE]** Do we prototype Tier D with Ollama + a Gemma or Llama model? Estimated effort: 1 engineer × 2 weeks for the router + 2 weeks of evaluation runs to confirm quality is acceptable. **Recommendation: prototype now, productise later — start with a sandbox H100 instance from RunPod (~£200/month) so the work is reversible.** | |
| 4. **[DECIDE]** Do we add a configuration flag (`llm_provider: openai | azure | local`) at the global / per-tenant level? Estimated effort: 2 days, mostly tests. **Recommendation: yes, regardless — even if we only ship Tier B in the first version, this flag is the foundation for everything else.** | |
| 5. **[DECIDE]** Do we publish a public-facing "Data Security" page that quotes from this document? Estimated effort: 1 day. **Recommendation: yes, by end of next sprint. It pre-empts the question we get in every webinar.** | |
| --- | |
| ## 8. Webinar talking points | |
| Five sentences that match **§2 (code)** + **§3 (OpenAI vendor text)** as of last review. **Replace** sentences 3–4 after Azure/local LLM code ships. | |
| 1. *"Your uploaded reports, database, and FAISS search index live on the infrastructure that runs our app—typically your server or your cloud account. **By default** we compute embeddings locally with a Hugging Face model; we can optionally call **OpenAI embeddings** depending on configuration. For drafting, proofreading, and optional photo analysis, we call **OpenAI’s API** over HTTPS with the prompts and context each feature needs."* | |
| 2. *"OpenAI’s **public** API and enterprise privacy pages describe that API inputs and outputs are **not used to train** their models by default, and describe **limited log retention** for safety and abuse review. For the strongest contractual minimisation of retention, customers should ask us about **Zero Data Retention** once we have completed OpenAI’s programme enrolment—always read the **live** vendor page before signing."* | |
| 3. *"For firms that need Microsoft’s UK boundary, **Azure OpenAI** is the industry pattern Microsoft documents for regulated workloads—it is on our **roadmap** as an integration option, not a switch that exists in this repository yet."* | |
| 4. *"For firms that require **no** public-cloud LLM, we can target a **self-hosted** open-weight model behind an OpenAI-compatible endpoint—that is also **roadmap** engineering, not the default install today."* | |
| 5. *"Between surveying firms, we enforce **tenant isolation** in the API and in vector search so one tenant’s chunks are not returned to another. **Inside** one firm, finer-grained per-user document ACLs are still a roadmap item—your IAM and VPN policies still matter."* | |
| --- | |
| ## 9. Risk register | |
| Residual risks after you have handled **LLM vendor choice**. Each row states **what can go wrong**, **severity**, and **what we already do in code** (if anything): | |
| | Risk | Severity | Mitigation | | |
| |---|---|---| | |
| | **Indirect prompt injection** via uploaded reports — instructions hidden in a PDF/DOCX | Medium | Defence in depth: grounded generation, optional **LLM grounding** in `app/generator/postprocess.py`, tenant-scoped retrieval. No separate “Pebblo” integration in this repo—evaluate third-party input filters if needed. | | |
| | **RAG access control** — benign question returns chunks the caller should not see | High | **Between tenants:** FAISS filters on `tenant_id` (`app/vectorstore/faiss_wrapper.py`). **Within one tenant:** per-user ACL is **not** implemented—any user who can act as that tenant can access that tenant’s indexed documents. Treat as roadmap. | | |
| | **Sub-processor change** at OpenAI / Azure | Low | Subscribe to vendor change notifications; update this doc. | | |
| | **Model updates** breaking output quality | Medium | `chat_model` in settings pins a model name; re-run regression tests before changing. | | |
| | **OpenAI court orders / litigation holds** affecting retention | Medium (Tier A) | ZDR and legal review if a customer needs stricter handling than standard API terms. | | |
| | **Token logging in our own infrastructure** — bodies in load-balancer or app logs | Medium | Audit deployment logging; confirm prompts/responses are not written at INFO in production. | | |
| | **Photo data** to OpenAI vision | Medium | `app/services/photo_vision.py` and `app/generator/vision_analyzer.py` when enabled. | | |
| | **Backup / replica copies** of SQLite/Postgres | Low–Medium | Deployer responsibility; document encryption at rest for backups. | | |
| --- | |
| ## 10. Implementation checklist per tier | |
| ### Tier A → Tier B (Zero Data Retention) | |
| - [ ] Apply for ZDR on the OpenAI org dashboard | |
| - [ ] Audit code for ZDR-incompatible endpoints (extended caching, background mode, Assistants API, Batch API). Confirm against **actual** call sites in this repository. | |
| - [ ] Add config flag `openai_zdr_enforced: bool` and a startup check that refuses to fall back | |
| - [ ] Add a `Data-Handling-Tier: zdr` HTTP response header (so compliance teams can verify) | |
| - [ ] Update website / public-facing data security page | |
| ### Tier C (Azure OpenAI) | |
| - [ ] Build **Azure OpenAI** client support (factory + settings). | |
| - [ ] Add config: `llm_provider`, `azure_openai_endpoint`, `azure_openai_api_version`, `azure_openai_deployment_name`, `azure_use_managed_identity` | |
| - [ ] Refactor OpenAI call sites (adapter, inspector loop, notes expander, style analyser, postprocess, vision, photo vision, canonical rollout, agents risk helper, generation orchestration) to use the factory | |
| - [ ] Document tenant Azure resource provisioning (Azure OpenAI resource + private endpoint + Key Vault + RBAC) | |
| - [ ] Document the "use your own Azure subscription" model vs "we provision one per tenant" | |
| - [ ] Apply for **Modified Abuse Monitoring** when a customer requires Microsoft’s documented reduction of certain abuse-monitoring reviews — confirm scope in [Azure OpenAI abuse monitoring](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/abuse-monitoring) (not a blanket “no humans” guarantee). | |
| - [ ] Add tests: connection works, falls back gracefully if Azure auth fails | |
| - [ ] Update website | |
| ### Tier D (Self-hosted open-weight) | |
| - [ ] Build **local LLM** path (OpenAI-compatible HTTP API — Ollama is the simplest backend). | |
| - [ ] Add config: `local_llm_endpoint`, `local_llm_model_name`, `local_llm_concurrent_requests` | |
| - [ ] Stand up a sandbox: RunPod H100 or A100 instance with Ollama serving Gemma 3 27B or Llama 3.3 70B (INT4 quantised) | |
| - [ ] Run evaluation against the local model. Compare to GPT-4o-mini baseline. Tune prompts if quality gap is unacceptable. | |
| - [ ] Document the per-tenant deployment topology (hardware spec, network policy, model patching) | |
| - [ ] Add a `vLLM` deployment as an alternative for higher-concurrency tenants | |
| - [ ] Document the "managed-but-isolated" middle option (we operate the model on RunPod or Modal per tenant) | |
| - [ ] Vision model: stand up LLaVA or Pixtral on the same host | |
| --- | |
| ## 11. References (vendor URLs — open before relying on any claim above) | |
| These URLs are the **authoritative** sources for OpenAI/Microsoft training, retention, regions, abuse monitoring, and subprocessors. If a sentence in §3–§4 disagrees with the live page, **the live page wins**. | |
| - [Enterprise privacy at OpenAI](https://openai.com/enterprise-privacy/) | |
| - [Business data privacy, security, and compliance | OpenAI](https://openai.com/business-data/) | |
| - [Data controls in the OpenAI platform](https://platform.openai.com/docs/guides/your-data) | |
| - [OpenAI Data Processing Addendum](https://openai.com/policies/data-processing-addendum/) | |
| - [OpenAI Sub-processor List](https://openai.com/policies/sub-processor-list/) | |
| - [OpenAI Response to NYT data demands](https://openai.com/index/response-to-nyt-data-demands/) | |
| - [Azure OpenAI / Azure Direct Models: Data, privacy, and security](https://learn.microsoft.com/en-us/azure/ai-foundry/responsible-ai/openai/data-privacy) | |
| - [Azure OpenAI abuse monitoring](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/abuse-monitoring) | |
| - [Azure OpenAI encryption of data at rest (CMK / BYOK)](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/encrypt-data-at-rest) | |
| - [Azure OpenAI network & access configuration](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/on-your-data-configuration) | |
| - [Foundry Models sold directly by Azure — including UK South](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/concepts/models-sold-directly-by-azure) | |
| - [Microsoft 365 Copilot — Data, Privacy, and Security](https://learn.microsoft.com/en-us/copilot/microsoft-365/microsoft-365-copilot-privacy) | |
| --- | |
| *Circulate internally and reuse text for customer-facing pages only after legal review. When OpenAI or Microsoft changes policy, re-read **§11** and update **§3–§4**; when our code changes egress, re-audit **§2** and update **§10**.* | |