README.md

---
title: RICS Report Genius
emoji: 🏠
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
pinned: false
license: mit
short_description: RICS v2 report generator from past reports + notes.
---
# Report Genius AI

**Active baseline:** `e561e67` + **standard RAG/LLM generate** (see [CODEBASE_BASELINE.md](CODEBASE_BASELINE.md)).
Confirm with `GET /health` → `codebase_baseline` and `primary_generate_pipeline: "standard"`.

A production-quality **Retrieval-Augmented Generation (RAG)** system for producing
RICS-style property survey reports. Each user's uploaded documents are indexed in a
single pooled vector store with strict `tenant_id` isolation — your data never leaks
into another user's results.

---

## Architecture

```
Upload (.docx/.pdf)
       │
       ▼
  Parser → Normaliser → Chunker
       │
       ▼
  Embeddings (OpenAI text-embedding-3-small)
       │
       ▼
  Vector Store (FAISS, on-disk) ← metadata: tenant_id, doc_id, chunk_id
       │
       ▼ (query + tenant_id filter)
  Retriever (top-10) → Reranker (top-3) → ≤400 token context
       │
       ▼
  LLM Adapter (gpt-4o-mini, edit/adapt prompt)
       │
       ▼
  Post-processor ([VERIFY] enforcement)
       │
       ▼
  Section Cache (SHA-256 keyed) → Response JSON
```

---

## Quick start

### 1. Prerequisites

- Python 3.11+
- Docker & Docker Compose (for full stack)
- An OpenAI API key

### 2. Clone and configure

```bash
git clone https://github.com/My-Report-AI/Report-genius-ai.git
cd Report-genius-ai
cp .env.example .env
# Edit .env and set OPENAI_API_KEY=sk-...
```

### 3a. Run locally (without Docker)

```bash
python -m venv .venv
# Windows:
.venv\Scripts\activate
# macOS/Linux:
source .venv/bin/activate

pip install -e ".[dev]"

# FAISS runs inside the API process (index under ~/.report_genius/faiss_index by default).
uvicorn app.main:app --reload --port 8000
```

### 3b. Run with Docker Compose

```bash
docker-compose up --build
```

API available at `http://localhost:8000`.  
Interactive docs at `http://localhost:8000/docs`.

---

## API endpoints

| Method | Path | Description |
|--------|------|-------------|
| `POST` | `/upload` | Upload a `.docx` or `.pdf`; triggers ingestion |
| `POST` | `/reports/{report_id}/generate` | Run retrieval + adapt pipeline |
| `GET` | `/reports/{report_id}/sections` | Fetch generated sections |
| `GET` | `/reports/{report_id}/status` | Poll ingestion / generation status |
| `POST` | `/test/ingest` | **Dev only** — ingest a file from `/mnt/data/samples/` |

### Example: upload a file

```bash
curl -X POST http://localhost:8000/upload \
  -H "X-Tenant-ID: tenant_abc" \
  -F "file=@/path/to/survey.docx" \
  -F "tenant_id=tenant_abc"
```

### Example: generate a section

```bash
curl -X POST http://localhost:8000/reports/{report_id}/generate \
  -H "X-Tenant-ID: tenant_abc" \
  -H "Content-Type: application/json" \
  -d '{
    "template_id": "B1",
    "bullets": [
      "Property is a semi-detached house built circa 1965",
      "Floor area approx 95 sqm",
      "Located in NW3 postcode"
    ]
  }'
```

### Similar content, deduplication, and refreshing the knowledge library

When surveyors add or revise inspection notes, the app can **surface similar material** already in the tenant’s workspace:

1. **Indexed uploads** (`.docx` / `.pdf` ingested into the FAISS index) — semantic (embedding) similarity.
2. **Other RICS sections’ draft notes** (sent from the browser) — lexical overlap (token Jaccard) to catch duplicate lines across sections.

**API**

```bash
curl -X POST http://localhost:8000/content/similar \
  -H "X-Tenant-ID: tenant_abc" \
  -H "Content-Type: application/json" \
  -d '{
    "text": "DPC minimum 150mm above ground per current guidance",
    "section_code": "E4",
    "peer_sections": { "I1": "…other section bullets…" },
    "exclude_document_ids": [],
    "limit": 8,
    "min_relevance_percent": 28
  }'
```

- **`exclude_document_ids`**: omit UUIDs of uploads you do not want in the match list (the UI passes the **current survey file** so hits focus on separate reference/guidance documents).
- **`DELETE /documents/{document_id}`**: removes that file from disk, the database, and the vector index. **HTTP 409** if a report still references the file — start a new report or keep the file for traceability.

**UI**: On the configure step, each section’s **Inspection Notes** toolbar has **Check similar**. The modal supports keeping notes, merging library or peer wording, replacing notes, or **removing an outdated upload** from the library.

This is **RAG corpus hygiene**, not LLM fine-tuning: updating or removing indexed documents changes what retrieval sees on the next generation.

---

## Environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `OPENAI_API_KEY` | *(required)* | OpenAI secret key |
| `FAISS_INDEX_PATH` | `~/.report_genius/faiss_index` | Directory for the persisted FAISS index |
| `DATABASE_URL` | `sqlite+aiosqlite:///./dev.db` | SQLAlchemy async DB URL |
| `UPLOAD_DIR` | `/mnt/data/uploads` | Where uploaded files are stored |
| `CACHE_DIR` | `/tmp/section_cache` | Section output cache directory |
| `MAX_CONTEXT_TOKENS` | `400` | Max tokens fed to LLM as context |
| `MAX_OUTPUT_TOKENS` | `300` | Max tokens in LLM response |
| `RETRIEVAL_TOP_K` | `10` | Vector search top-k |
| `RERANK_TOP_N` | `3` | Snippets kept after reranking |
| `CHUNK_SIZE` | `500` | Target chunk size in tokens |
| `CHUNK_OVERLAP` | `75` | Overlap between consecutive chunks |
| `DEV_MODE` | `false` | Enables SQL echo and dev endpoint |
| `TENANT_SECRET_KEY` | *(required in prod)* | Used to sign tenant tokens |

---

## Running tests

```bash
# All tests with coverage
pytest

# Specific acceptance tests
pytest app/tests/test_isolation.py -v   # tenant isolation
pytest app/tests/test_generator.py -v  # non-invention + token limit
pytest app/tests/test_cache.py -v      # cache prevents duplicate LLM calls
pytest app/tests/test_content_similar_api.py -v  # similar-content API
```

Tests run in < 2 minutes (all LLM and embedding calls are mocked).

---

## Linting & type-checks

```bash
black --check .
ruff check .
mypy app/
```

---

## Incremental milestones

| Milestone | Description |
|-----------|-------------|
| M1 | Upload + DOCX/PDF parser → chunks saved to disk |
| M2 | Chunker + FAISS index + mock embeddings |
| M3 | Retriever + reranker + generator (mock OpenAI) |
| M4 | Real OpenAI calls + section cache |
| M5 | CI, security middleware, acceptance tests |

---

## Non-invention guarantee

The LLM is always prompted with:

> "Do not invent facts. Use only facts present in the bullets or retrieved_snippets.
>  If a fact is missing or unverifiable, mark it **[VERIFY]**."

A post-processor additionally scans the output for any numeric fact or named entity
not present in the source bullets or retrieved snippets, and automatically wraps it
with `[VERIFY]`. These items are flagged for human review before the report is
finalised.

### Inline section editing

The results grid renders each generated section in a `.result-text`
`contenteditable="plaintext-only"` block. Edits are persisted server-side via
`PATCH /reports/{report_id}/sections/{section_code}` (body
`{ "text": "..." }`, max 200 KB UTF-8, empty string allowed). Saves are
debounced **700 ms** after the last keystroke, fire immediately on **blur**,
and on **Cmd/Ctrl+S** (the browser's save-page dialog is suppressed). An
in-flight request is aborted if the user resumes typing before it completes.

The card header carries a small status pill that walks through:
`✎ Editable → ⏳ Saving… → ✓ Saved` (or `⚠ Save failed` on a non-2xx response).

When a save succeeds the backend stamps `meta.user_edited = true` inside the
section's persisted `provenance` JSON; existing `provenance.sources`, `mode`,
`confidence`, `ai_level`, `ai_percent`, `ai_transparency`, `style_profile`,
`pipeline`, `fallback_used`, and `inspector` metadata are preserved. The
client also updates `state.results[code].text` in place, so a subsequent
**Re-Generate** of the same section sends the edited body as
`draft_paragraph` rather than the original LLM output.

---

## Personalisation guarantees (per user / tenant)

This system personalises output **per tenant** and **per report request**.

### 1) User isolation (who the model learns from)

- Retrieval is always filtered by `tenant_id`.
- Writing-style analysis samples only that tenant's indexed content.
- Style profile cache is keyed by tenant, so one user's style never leaks to another.

### 2) Style personalisation (how it writes)

For generation, the pipeline detects and applies a `WritingStyleProfile`:

- tone
- formality level
- sentence complexity
- vocabulary level
- common phrases
- structural patterns

These fields are injected into prompts so sections match the user's own writing voice.

### 3) Preferences and choices (how much AI is allowed)

The frontend AI scale (`ai_level` 1..5) controls behaviour:

- **1 (RAG only)**: minimal rewrite, no notes-expansion, no style-profile injection.
- **2..4 (light to strong)**: increasing rewrite strength and style matching.
- **5 (full AI)**: strongest style adaptation and paraphrasing.

`ai_level` is applied in all three modes:

- `generate`: controls temperature + creativity hint + expansion behaviour.
- `proofread`: controls edit intensity (conservative at low levels, stronger at high).
- `enhance`: controls technical-expansion intensity and prose freedom.

### 4) Priorities (what the model should preserve first)

Prompt and post-processing priorities are fixed in this order:

1. Preserve user facts (especially numbers/measurements/dates/addresses).
2. Use retrieved evidence from the same tenant.
3. Match user style/profile.
4. Mark uncertain/unsupported claims with `[VERIFY]`.

This means user-provided facts and evidence are prioritised above stylistic flourish.

### 5) Current scope

Current personalisation is driven by:

- uploaded reference documents (style + evidence),
- section bullets / extracted notes (user intent),
- selected mode (`generate`, `proofread`, `enhance`),
- selected AI scale (`ai_level`).

There is currently no separate long-term "user preference profile" store beyond
tenant style analysis and request-level controls above.

---

## Licence

MIT — see `LICENSE`.