EXPERIMENT_NOTE.md

# Experiment Lab — LangChain RAG Implementation

This branch rewrites the core RAG pipeline using **LangChain** and its
supporting frameworks, replacing the custom implementations in `master`.

## What Changed

| Component | master branch | experiment-lab branch |
|---|---|---|
| **DOCX loading** | `python-docx` + manual XML block extraction | `langchain_community` `Docx2txtLoader` |
| **PDF loading** | `pymupdf` + font-size heading heuristics | `langchain_community` `PyMuPDFLoader` |
| **Text splitting** | Custom recursive token splitter | `langchain_text_splitters` `RecursiveCharacterTextSplitter` (tiktoken) |
| **Embeddings** | Custom `EmbeddingClient` ABC + manual OpenAI/SentenceTransformer wrappers | `langchain_openai.OpenAIEmbeddings` / `langchain_huggingface.HuggingFaceEmbeddings` |
| **Vector store** | Manual `chromadb` + FAISS with custom metadata filtering | `langchain_community.vectorstores.FAISS` only — local, no Chroma |
| **Retrieval** | Manual query embedding → `vs.search(vector, tenant_id)` | `vs.search(text, tenant_id)` — store embeds internally |
| **LLM generation** | Manual `openai.OpenAI().chat.completions.create()` | **LCEL chain**: `RICS_PROMPT \| ChatOpenAI \| StrOutputParser()` |
| **Retry logic** | None | Built-in via `ChatOpenAI(max_retries=3)` |

## What Stayed the Same

These components contain business logic specific to this system and were
**not** replaced with LangChain equivalents:

- `app/api/` — FastAPI endpoints, tenant middleware, upload/generate/status
- `app/db/` — SQLAlchemy ORM models and session management
- `app/cache/section_cache.py` — SHA-256 keyed JSON section cache
- `app/generator/postprocess.py` — `[VERIFY]` non-invention enforcement
- `app/generator/reranker.py` — Custom Jaccard + numeric overlap reranker
- `app/templates/rics_templates.py` — RICS section skeleton templates
- `app/models/schemas.py` — Pydantic API schemas
- `app/config.py` — Settings management

## New Dependencies

```toml
langchain>=0.3.0
langchain-core>=0.3.0
langchain-openai>=0.2.0
langchain-community>=0.3.0
faiss-cpu>=1.8.0
langchain-huggingface>=0.1.0
langchain-text-splitters>=0.3.0
docx2txt>=0.8
```

## Key Design Decisions

### Why keep a custom reranker?
LangChain's built-in rerankers (`FlashrankRerank`, `CrossEncoderReranker`)
require additional model downloads or paid APIs. The custom Jaccard+numeric
reranker is lightweight, free, and tailored to RICS numeric fact verification.

### Why keep `[VERIFY]` post-processing?
This is a domain-specific safety feature with no LangChain equivalent.
It enforces the non-invention guarantee that is core to RICS report integrity.

### LCEL Chain Structure
```python
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI
from langchain_core.output_parsers import StrOutputParser

chain = RICS_PROMPT | ChatOpenAI(model="gpt-4o-mini", temperature=0.2) | StrOutputParser()
text = chain.invoke({"skeleton": ..., "bullets": ..., "examples": ...})
```

### Vectorstore Pipeline
```python
# Ingestion — store handles embedding internally
vectorstore.add_documents(langchain_docs)

# Retrieval — store embeds query internally
results = vectorstore.search(query_text, tenant_id=tenant_id, k=10)
```

## Merged product features (AI transparency + tenant RAG)

The following were merged from the main-line feature work and are **in addition**
to the LangChain stack above:

- **`retrieve_for_report()`** — tenant-scoped retrieval with the report’s source
  document prioritised.
- **Provenance enrichment** — filenames, snippet previews, and section hints for
  citations in the API and UI.
- **`compute_ai_transparency()`** — heuristic AI-involvement estimates stored in
  section metadata.
- **Frontend** — aggregate AI involvement bar, per-section transparency,
  citation chips, and fixed proofread/enhance re-run handling.
- **Generate mode** uses **LCEL** (`RICS_PROMPT | ChatOpenAI | StrOutputParser`)
  with the same rich `build_user_prompt` / style-profile content as before
  (single `user_content` variable in the chat template).

Proofread and enhance modes still use the OpenAI Chat Completions client for
stable, predictable behaviour.

## Running This Branch

```bash
git checkout experiment-lab
pip install -e ".[dev]"
cp .env.example .env  # add OPENAI_API_KEY if available
uvicorn app.main:app --reload
```

Open http://localhost:8000 — the same UI as master.

## Trade-offs vs Master Branch

| Aspect | Master (custom) | Experiment-lab (LangChain) |
|---|---|---|
| Code volume | ~800 lines of RAG pipeline code | ~400 lines (LangChain handles the rest) |
| Structural fidelity | Heading/list/table block types preserved | Plain text only (heading levels lost) |
| Debuggability | Full control, easy to step through | LangChain abstractions can obscure flow |
| Observability | Manual logging | LangSmith tracing available |
| Upgradability | Manual updates to each component | Single `pip upgrade langchain` |
| Vendor lock-in | None | LangChain API surface |