README_RAG.md

🗼 RAG Chat API — Gustave Eiffel Hackathon 2026

A complete Retrieval-Augmented Generation (RAG) system deployed as a Hugging Face Space, with a /query API endpoint designed for the RAG evaluation system.

---

---

Overview

This application demonstrates how to build a production-ready RAG system within the Hugging Face ecosystem. It covers:

Requirement	Solution
LLM API calls	Azure OpenAI (gpt-5 via REST)
Text → Embeddings	Azure OpenAI (text-embedding-3-small via REST)
Vector Store	ChromaDB (persistent, runs in-process)
API Endpoint	FastAPI with POST /query
UI	Gradio Blocks (chat + document ingestion)

---

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Hugging Face Space                         │
│                                                              │
│  ┌──────────┐     ┌──────────────┐     ┌───────────────┐   │
│  │  Gradio  │     │   FastAPI    │     │   ChromaDB    │   │
│  │   UI     │────▶│  /query      │────▶│  Vector Store │   │
│  │          │     │  /ingest     │     │  (persistent) │   │
│  └──────────┘     └──────┬───────┘     └───────────────┘   │
│                          │                      ▲            │
│                          ▼                      │            │
│               ┌──────────────────┐    ┌─────────────────┐   │
│               │  Azure OpenAI    │    │  Azure OpenAI   │   │
│               │  GPT-5 (LLM)    │    │  text-embedding │   │
│               │                  │    │  -3-small       │   │
│               └──────────────────┘    └─────────────────┘   │
└─────────────────────────────────────────────────────────────┘

---

Step-by-Step Explanation

Step 1: Document Ingestion & Chunking

Before we can answer questions, we need to prepare our knowledge base.

1. Load documents — Read text files from sample_documents/ directory
2. Chunk text — Split documents into smaller overlapping chunks (512 tokens, 50 token overlap) using RecursiveCharacterTextSplitter. This ensures each chunk fits within the embedding model's context window while maintaining semantic coherence.

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document_text)

Step 2: Generate Embeddings

Convert text chunks into dense vector representations that capture semantic meaning.

1. Call Azure OpenAI — We use the text-embedding-3-small model via the Azure OpenAI embeddings endpoint
2. Encode text — Each chunk is transformed into a fixed-size vector where semantically similar texts are closer together in vector space

import requests as http_requests

headers = {"api-key": AZURE_API_KEY, "Content-Type": "application/json"}
payload = {"input": ["chunk 1 text", "chunk 2 text"], "model": "text-embedding-3-small"}
resp = http_requests.post(EMBEDDING_ENDPOINT_URL, headers=headers, json=payload)
embeddings = [item["embedding"] for item in resp.json()["data"]]

Step 3: Store in Vector Database (ChromaDB)

Persist embeddings in a vector store optimized for similarity search.

1. Initialize ChromaDB — Create a persistent client that stores data on disk (survives Space restarts)
2. Create collection — A named collection with cosine similarity metric
3. Add documents — Store embeddings alongside the original text and metadata

import chromadb

client = chromadb.PersistentClient(path="./data/chroma_db")
collection = client.get_or_create_collection(
    name="rag_documents",
    metadata={"hnsw:space": "cosine"},
)
collection.add(
    ids=["doc_0", "doc_1"],
    embeddings=embeddings.tolist(),
    documents=["chunk 1 text", "chunk 2 text"],
    metadatas=[{"source": "file.txt"}, {"source": "file.txt"}],
)

Step 4: Query & Retrieval

When a user asks a question, find the most relevant context.

1. Embed the query — Use the same Azure OpenAI embedding model to convert the question to a vector
2. Similarity search — Find the top-K nearest vectors in ChromaDB (cosine similarity)
3. Return context — Extract the original text chunks for the closest matches

query_embedding = generate_embeddings(["What is the Eiffel Tower?"])[0]
results = collection.query(
    query_embeddings=[query_embedding],
    n_results=3,
)

Step 5: LLM Generation (Augmented Response)

Combine retrieved context with the user's question and generate an answer.

1. Build prompt — Load the template from prompts/rag_prompt.txt, inject retrieved context and the user's question
2. Call Azure OpenAI — Send the prompt to the Azure OpenAI chat/completions endpoint (gpt-5)
3. Return response — The LLM generates an answer grounded in the provided context

The prompt template (prompts/rag_prompt.txt):

You are a helpful assistant. Answer the user's question based ONLY on the provided context.
If the context does not contain enough information to answer, say "I don't have enough information to answer this question."
Always be concise and factual.

Context:
{context}

Question: {question}

The template is loaded once at startup and sent as the user message to the chat endpoint:

RAG_PROMPT_TEMPLATE = Path("prompts/rag_prompt.txt").read_text(encoding="utf-8")

# At query time:
prompt = RAG_PROMPT_TEMPLATE.format(context=context_text, question=user_query)
headers = {"api-key": AZURE_API_KEY, "Content-Type": "application/json"}
payload = {
    "model": "gpt-5",
    "messages": [{"role": "user", "content": prompt}],
    "max_completion_tokens": 512,
    "temperature": 0.7,
    "top_p": 0.95,
}
resp = requests.post(LLM_ENDPOINT_URL, headers=headers, json=payload)
answer = resp.json()["choices"][0]["message"]["content"]

Tip: Edit prompts/rag_prompt.txt to tune the model's behaviour (tone, language, output format) without touching application code.

Step 6: API Endpoint (/query)

The FastAPI endpoint ties everything together for the evaluation system.

@app.post("/query")
async def query_endpoint(request: QueryRequest):
    # 1. Retrieve relevant context
    # 2. Build augmented prompt
    # 3. Generate LLM response
    # 4. Return answer + sources
    result = rag_query(request.query, top_k=request.top_k)
    return JSONResponse(content=result)

---

API Endpoints

POST /query

The primary endpoint for the RAG evaluation system.

Request:

{
    "query": "What materials is the Eiffel Tower made of?",
    "top_k": 3
}

Response:

{
    "answer": "The Eiffel Tower is made of wrought iron (puddled iron)...",
    "sources": [
        {"source": "eiffel_tower.txt", "score": 0.87},
        {"source": "paris_landmarks.txt", "score": 0.72}
    ],
    "query": "What materials is the Eiffel Tower made of?"
}

POST /ingest

Add new documents to the knowledge base.

Request:

{
    "text": "The Eiffel Tower was built in 1889...",
    "source": "my_document.txt"
}

Response:

{
    "status": "success",
    "chunks_added": 5,
    "total_chunks": 42
}

GET /health

System health check.

Response:

{
    "status": "healthy",
    "documents_in_store": 42,
    "embedding_model": "text-embedding-3-small",
    "llm_model": "gpt-5"
}

---