README_RAG.md

# 🗼 RAG Chat API — Gustave Eiffel Hackathon 2026

A complete **Retrieval-Augmented Generation (RAG)** system deployed as a Hugging Face Space, with a `/query` API endpoint designed for the RAG evaluation system.

---
---

## Overview

This application demonstrates how to build a production-ready RAG system within the Hugging Face ecosystem. It covers:

| Requirement | Solution |
|---|---|
| LLM API calls | Azure OpenAI (`gpt-5` via REST) |
| Text → Embeddings | Azure OpenAI (`text-embedding-3-small` via REST) |
| Vector Store | ChromaDB (persistent, runs in-process) |
| API Endpoint | FastAPI with `POST /query` |
| UI | Gradio Blocks (chat + document ingestion) |

---

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Hugging Face Space                         │
│                                                              │
│  ┌──────────┐     ┌──────────────┐     ┌───────────────┐   │
│  │  Gradio  │     │   FastAPI    │     │   ChromaDB    │   │
│  │   UI     │────▶│  /query      │────▶│  Vector Store │   │
│  │          │     │  /ingest     │     │  (persistent) │   │
│  └──────────┘     └──────┬───────┘     └───────────────┘   │
│                          │                      ▲            │
│                          ▼                      │            │
│               ┌──────────────────┐    ┌─────────────────┐   │
│               │  Azure OpenAI    │    │  Azure OpenAI   │   │
│               │  GPT-5 (LLM)    │    │  text-embedding │   │
│               │                  │    │  -3-small       │   │
│               └──────────────────┘    └─────────────────┘   │
└─────────────────────────────────────────────────────────────┘
```

---

## Step-by-Step Explanation

### Step 1: Document Ingestion & Chunking

Before we can answer questions, we need to prepare our knowledge base.

1. **Load documents** — Read text files from `sample_documents/` directory
2. **Chunk text** — Split documents into smaller overlapping chunks (512 tokens, 50 token overlap) using `RecursiveCharacterTextSplitter`. This ensures each chunk fits within the embedding model's context window while maintaining semantic coherence.

```python
splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document_text)
```

### Step 2: Generate Embeddings

Convert text chunks into dense vector representations that capture semantic meaning.

1. **Call Azure OpenAI** — We use the `text-embedding-3-small` model via the Azure OpenAI embeddings endpoint
2. **Encode text** — Each chunk is transformed into a fixed-size vector where semantically similar texts are closer together in vector space

```python
import requests as http_requests

headers = {"api-key": AZURE_API_KEY, "Content-Type": "application/json"}
payload = {"input": ["chunk 1 text", "chunk 2 text"], "model": "text-embedding-3-small"}
resp = http_requests.post(EMBEDDING_ENDPOINT_URL, headers=headers, json=payload)
embeddings = [item["embedding"] for item in resp.json()["data"]]
```

### Step 3: Store in Vector Database (ChromaDB)

Persist embeddings in a vector store optimized for similarity search.

1. **Initialize ChromaDB** — Create a persistent client that stores data on disk (survives Space restarts)
2. **Create collection** — A named collection with cosine similarity metric
3. **Add documents** — Store embeddings alongside the original text and metadata

```python
import chromadb

client = chromadb.PersistentClient(path="./data/chroma_db")
collection = client.get_or_create_collection(
    name="rag_documents",
    metadata={"hnsw:space": "cosine"},
)
collection.add(
    ids=["doc_0", "doc_1"],
    embeddings=embeddings.tolist(),
    documents=["chunk 1 text", "chunk 2 text"],
    metadatas=[{"source": "file.txt"}, {"source": "file.txt"}],
)
```

### Step 4: Query & Retrieval

When a user asks a question, find the most relevant context.

1. **Embed the query** — Use the same Azure OpenAI embedding model to convert the question to a vector
2. **Similarity search** — Find the top-K nearest vectors in ChromaDB (cosine similarity)
3. **Return context** — Extract the original text chunks for the closest matches

```python
query_embedding = generate_embeddings(["What is the Eiffel Tower?"])[0]
results = collection.query(
    query_embeddings=[query_embedding],
    n_results=3,
)
```

### Step 5: LLM Generation (Augmented Response)

Combine retrieved context with the user's question and generate an answer.

1. **Build prompt** — Load the template from [`prompts/rag_prompt.txt`](prompts/rag_prompt.txt), inject retrieved context and the user's question
2. **Call Azure OpenAI** — Send the prompt to the Azure OpenAI chat/completions endpoint (`gpt-5`)
3. **Return response** — The LLM generates an answer grounded in the provided context

The prompt template (`prompts/rag_prompt.txt`):

```
You are a helpful assistant. Answer the user's question based ONLY on the provided context.
If the context does not contain enough information to answer, say "I don't have enough information to answer this question."
Always be concise and factual.

Context:
{context}

Question: {question}
```

The template is loaded once at startup and sent as the user message to the chat endpoint:

```python
RAG_PROMPT_TEMPLATE = Path("prompts/rag_prompt.txt").read_text(encoding="utf-8")

# At query time:
prompt = RAG_PROMPT_TEMPLATE.format(context=context_text, question=user_query)
headers = {"api-key": AZURE_API_KEY, "Content-Type": "application/json"}
payload = {
    "model": "gpt-5",
    "messages": [{"role": "user", "content": prompt}],
    "max_completion_tokens": 512,
    "temperature": 0.7,
    "top_p": 0.95,
}
resp = requests.post(LLM_ENDPOINT_URL, headers=headers, json=payload)
answer = resp.json()["choices"][0]["message"]["content"]
```

> **Tip:** Edit `prompts/rag_prompt.txt` to tune the model's behaviour (tone, language, output format) without touching application code.

### Step 6: API Endpoint (`/query`)

The FastAPI endpoint ties everything together for the evaluation system.

```python
@app.post("/query")
async def query_endpoint(request: QueryRequest):
    # 1. Retrieve relevant context
    # 2. Build augmented prompt
    # 3. Generate LLM response
    # 4. Return answer + sources
    result = rag_query(request.query, top_k=request.top_k)
    return JSONResponse(content=result)
```

---

## API Endpoints

### `POST /query`

The primary endpoint for the RAG evaluation system.

**Request:**
```json
{
    "query": "What materials is the Eiffel Tower made of?",
    "top_k": 3
}
```

**Response:**
```json
{
    "answer": "The Eiffel Tower is made of wrought iron (puddled iron)...",
    "sources": [
        {"source": "eiffel_tower.txt", "score": 0.87},
        {"source": "paris_landmarks.txt", "score": 0.72}
    ],
    "query": "What materials is the Eiffel Tower made of?"
}
```

### `POST /ingest`

Add new documents to the knowledge base.

**Request:**
```json
{
    "text": "The Eiffel Tower was built in 1889...",
    "source": "my_document.txt"
}
```

**Response:**
```json
{
    "status": "success",
    "chunks_added": 5,
    "total_chunks": 42
}
```

### `GET /health`

System health check.

**Response:**
```json
{
    "status": "healthy",
    "documents_in_store": 42,
    "embedding_model": "text-embedding-3-small",
    "llm_model": "gpt-5"
}
```

---