better documentations

README.md

@@ -21,12 +21,10 @@ A complete **Retrieval-Augmented Generation (RAG)** system deployed as a Hugging
 
 1. [Overview](#overview)
 2. [Architecture](#architecture)
-3. [Step-by-Step Explanation](#step-by-step-explanation)
-4. [API Endpoints](#api-endpoints)
-5. [Setup & Deployment](#setup--deployment)
-6. [Adding Binary Files to the HF Space](#adding-binary-files-to-the-hf-space)
-7. [Configuration](#configuration)
-8. [How It Works (Detailed)](#how-it-works-detailed)
+3. [Setup & Deployment](#setup--deployment)
+4. [Adding Binary Files to the HF Space](#adding-binary-files-to-the-hf-space)
+5. [Configuration](#configuration)
+6. [How It Works (Detailed)](#how-it-works-detailed)
 
 ---
 
@@ -67,201 +65,6 @@ This application demonstrates how to build a production-ready RAG system within
 
 ---
 
-## Step-by-Step Explanation
-
-### Step 1: Document Ingestion & Chunking
-
-Before we can answer questions, we need to prepare our knowledge base.
-
-1. **Load documents** — Read text files from `sample_documents/` directory
-2. **Chunk text** — Split documents into smaller overlapping chunks (512 tokens, 50 token overlap) using `RecursiveCharacterTextSplitter`. This ensures each chunk fits within the embedding model's context window while maintaining semantic coherence.
-
-```python
-splitter = RecursiveCharacterTextSplitter(
-    chunk_size=512,
-    chunk_overlap=50,
-    separators=["\n\n", "\n", ". ", " ", ""],
-)
-chunks = splitter.split_text(document_text)
-```
-
-### Step 2: Generate Embeddings
-
-Convert text chunks into dense vector representations that capture semantic meaning.
-
-1. **Call Azure OpenAI** — We use the `text-embedding-3-small` model via the Azure OpenAI embeddings endpoint
-2. **Encode text** — Each chunk is transformed into a fixed-size vector where semantically similar texts are closer together in vector space
-
-```python
-import requests as http_requests
-
-headers = {"api-key": AZURE_API_KEY, "Content-Type": "application/json"}
-payload = {"input": ["chunk 1 text", "chunk 2 text"], "model": "text-embedding-3-small"}
-resp = http_requests.post(EMBEDDING_ENDPOINT_URL, headers=headers, json=payload)
-embeddings = [item["embedding"] for item in resp.json()["data"]]
-```
-
-### Step 3: Store in Vector Database (ChromaDB)
-
-Persist embeddings in a vector store optimized for similarity search.
-
-1. **Initialize ChromaDB** — Create a persistent client that stores data on disk (survives Space restarts)
-2. **Create collection** — A named collection with cosine similarity metric
-3. **Add documents** — Store embeddings alongside the original text and metadata
-
-```python
-import chromadb
-
-client = chromadb.PersistentClient(path="./data/chroma_db")
-collection = client.get_or_create_collection(
-    name="rag_documents",
-    metadata={"hnsw:space": "cosine"},
-)
-collection.add(
-    ids=["doc_0", "doc_1"],
-    embeddings=embeddings.tolist(),
-    documents=["chunk 1 text", "chunk 2 text"],
-    metadatas=[{"source": "file.txt"}, {"source": "file.txt"}],
-)
-```
-
-### Step 4: Query & Retrieval
-
-When a user asks a question, find the most relevant context.
-
-1. **Embed the query** — Use the same Azure OpenAI embedding model to convert the question to a vector
-2. **Similarity search** — Find the top-K nearest vectors in ChromaDB (cosine similarity)
-3. **Return context** — Extract the original text chunks for the closest matches
-
-```python
-query_embedding = generate_embeddings(["What is the Eiffel Tower?"])[0]
-results = collection.query(
-    query_embeddings=[query_embedding],
-    n_results=3,
-)
-```
-
-### Step 5: LLM Generation (Augmented Response)
-
-Combine retrieved context with the user's question and generate an answer.
-
-1. **Build prompt** — Load the template from [`prompts/rag_prompt.txt`](prompts/rag_prompt.txt), inject retrieved context and the user's question
-2. **Call Azure OpenAI** — Send the prompt to the Azure OpenAI chat/completions endpoint (`gpt-5`)
-3. **Return response** — The LLM generates an answer grounded in the provided context
-
-The prompt template (`prompts/rag_prompt.txt`):
-
-```
-You are a helpful assistant. Answer the user's question based ONLY on the provided context.
-If the context does not contain enough information to answer, say "I don't have enough information to answer this question."
-Always be concise and factual.
-
-Context:
-{context}
-
-Question: {question}
-```
-
-The template is loaded once at startup and sent as the user message to the chat endpoint:
-
-```python
-RAG_PROMPT_TEMPLATE = Path("prompts/rag_prompt.txt").read_text(encoding="utf-8")
-
-# At query time:
-prompt = RAG_PROMPT_TEMPLATE.format(context=context_text, question=user_query)
-headers = {"api-key": AZURE_API_KEY, "Content-Type": "application/json"}
-payload = {
-    "model": "gpt-5",
-    "messages": [{"role": "user", "content": prompt}],
-    "max_completion_tokens": 512,
-    "temperature": 0.7,
-    "top_p": 0.95,
-}
-resp = requests.post(LLM_ENDPOINT_URL, headers=headers, json=payload)
-answer = resp.json()["choices"][0]["message"]["content"]
-```
-
-> **Tip:** Edit `prompts/rag_prompt.txt` to tune the model's behaviour (tone, language, output format) without touching application code.
-
-### Step 6: API Endpoint (`/query`)
-
-The FastAPI endpoint ties everything together for the evaluation system.
-
-```python
-@app.post("/query")
-async def query_endpoint(request: QueryRequest):
-    # 1. Retrieve relevant context
-    # 2. Build augmented prompt
-    # 3. Generate LLM response
-    # 4. Return answer + sources
-    result = rag_query(request.query, top_k=request.top_k)
-    return JSONResponse(content=result)
-```
-
----
-
-## API Endpoints
-
-### `POST /query`
-
-The primary endpoint for the RAG evaluation system.
-
-**Request:**
-```json
-{
-    "query": "What materials is the Eiffel Tower made of?",
-    "top_k": 3
-}
-```
-
-**Response:**
-```json
-{
-    "answer": "The Eiffel Tower is made of wrought iron (puddled iron)...",
-    "sources": [
-        {"source": "eiffel_tower.txt", "score": 0.87},
-        {"source": "paris_landmarks.txt", "score": 0.72}
-    ],
-    "query": "What materials is the Eiffel Tower made of?"
-}
-```
-
-### `POST /ingest`
-
-Add new documents to the knowledge base.
-
-**Request:**
-```json
-{
-    "text": "The Eiffel Tower was built in 1889...",
-    "source": "my_document.txt"
-}
-```
-
-**Response:**
-```json
-{
-    "status": "success",
-    "chunks_added": 5,
-    "total_chunks": 42
-}
-```
-
-### `GET /health`
-
-System health check.
-
-**Response:**
-```json
-{
-    "status": "healthy",
-    "documents_in_store": 42,
-    "embedding_model": "text-embedding-3-small",
-    "llm_model": "gpt-5"
-}
-```
-
----
 
 ## Setup & Deployment
 
@@ -293,6 +96,7 @@ source ~/.venv/hackathon-eiffel/bin/activate
 ~\.venv\hackathon-eiffel\Scripts\Activate.ps1
 # Windows (Command Prompt)
 ~\.venv\hackathon-eiffel\Scripts\activate.bat
+
 ```
 
 Once your virtual environment is active, install dependencies and run the app:
@@ -310,6 +114,8 @@ $env:AZURE_API_KEY = "your_azure_api_key_here"
 # Run the application
 python app.py
 # Server starts at http://localhost:7860
+
+# To make it work, you first need to create embeddings , you can learn about it from README_RAG.md
 ```
 
 > **Tip:** To deactivate the virtual environment when you are done, run `deactivate` (venv) or `conda deactivate` (conda).

README_HF.md

@@ -0,0 +1,181 @@
+# How to Push Code to a New HuggingFace Space
+
+## Prerequisites
+
+- [Git](https://git-scm.com/) installed
+- A [HuggingFace account](https://huggingface.co/join)
+- A HuggingFace Access Token (create one at [Settings > Tokens](https://huggingface.co/settings/tokens) with **Write** permission)
+
+---
+
+## Steps
+
+### 1. Create a New Space on HuggingFace
+
+1. Go to [huggingface.co/new-space](https://huggingface.co/new-space)
+2. Choose a **Space name** (e.g., `My_App`)
+3. Select the **SDK** (Gradio, Streamlit, Docker, or Static)
+4. Choose visibility (Public or Private)
+5. Click **Create Space**
+
+Your Space URL will be: `https://huggingface.co/spaces/<YOUR_USERNAME>/<SPACE_NAME>`
+
+---
+
+### 2. Clone the Empty Space Locally
+
+```bash
+git clone https://huggingface.co/spaces/<YOUR_USERNAME>/<SPACE_NAME>
+cd <SPACE_NAME>
+```
+
+When prompted for credentials:
+- **Username:** Your HuggingFace username
+- **Password:** Your HuggingFace Access Token (NOT your account password)
+
+---
+
+### 3. Add Your Code
+
+You have **two options** to get code into your new Space:
+
+#### Option A: Pull from an Existing Repo into the Space
+
+If the code you want already lives in another git repo (e.g., a teammate's HF Space or a GitHub repo), you can pull it in:
+
+```bash
+# Inside your cloned Space folder:
+cd <SPACE_NAME>
+
+# Add the source repo as a second remote
+git remote add source https://huggingface.co/spaces/<SOURCE_OWNER>/<SOURCE_REPO>
+# or from GitHub:
+# git remote add source https://github.com/<OWNER>/<REPO>.git
+
+# Fetch all branches from the source
+git fetch source
+
+# Merge the source's main branch into your Space
+git merge source/main --allow-unrelated-histories -m "Pull code from source repo"
+```
+
+> If there are merge conflicts, resolve them, then `git add -A` and `git commit`.
+
+#### Option B: Copy Files Manually
+
+Simply copy/paste your project files into the cloned Space folder.
+
+---
+
+**Either way**, make sure you have a `.gitignore` to exclude unnecessary files:
+
+```
+.venv
+__pycache__/
+**/__pycache__/
+*.sqlite3
+chroma_db/
+.env
+```
+
+---
+
+### 4. Commit and Push
+
+```bash
+git add -A
+git commit -m "Initial commit"
+git push origin main
+git remote remove source
+```
+
+---
+
+## Alternative (OPTIONAL): Push an Existing Local Project (with Full History)
+
+If you already have a local project with commits and want to push everything (all history) to a new HF Space:
+
+### 1. Add the Space as a Remote (OPTIONAL)
+
+```bash
+cd /path/to/your/project
+git remote add hfspace https://<YOUR_USERNAME>:<HF_TOKEN>@huggingface.co/spaces/<YOUR_USERNAME>/<SPACE_NAME>
+```
+
+> **Tip:** Embedding the token in the URL avoids repeated password prompts.
+
+### 2. Make Sure Binary Files Are NOT Tracked
+
+HuggingFace rejects any push containing binary files (`.sqlite3`, `.pkl`, `.bin`, etc.).  
+Before pushing, ensure they are in `.gitignore` **and** removed from the entire git history.
+
+```bash
+# Add binary paths to .gitignore first, then:
+git rm -r --cached path/to/binary/files
+git add -A
+git commit -m "Remove binary files from tracking"
+```
+
+If binaries exist in **older commits**, you must rewrite history (see Troubleshooting below).
+
+### 3. Push the Current Branch with All Commits (OPTIONAL)
+
+```bash
+git push hfspace HEAD:main --force
+```
+
+- `HEAD` = your current branch (whatever it's called)
+- `HEAD:main` = push it to the `main` branch on the Space
+- `--force` = overwrite the Space's existing initial commit
+
+This preserves your full commit history on the Space.
+
+---
+
+## Troubleshooting
+
+### Binary File Rejection
+
+HuggingFace rejects pushes containing binary files (e.g., `.sqlite3`, `.pkl`, `.bin`).
+
+**Fix:**
+
+1. Add the binary files to `.gitignore`
+2. Remove them from git tracking:
+   ```bash
+   git rm -r --cached path/to/binary/file
+   ```
+3. Squash history to purge them completely:
+   ```bash
+   git add -A
+   git reset --soft $(git rev-list --max-parents=0 HEAD)
+   git commit -m "Clean initial commit"
+   git push hfspace main --force
+   ```
+
+### Authentication Failed
+
+- HuggingFace does **not** accept account passwords for git. Use an **Access Token**.
+- Make sure the token has **Write** permission.
+- You can embed the token in the remote URL:
+  ```bash
+  git remote set-url hfspace https://<USERNAME>:<TOKEN>@huggingface.co/spaces/<USERNAME>/<SPACE_NAME>
+  ```
+
+### Wrong Branch Name
+
+Some repos use `master` instead of `main`. Check with:
+
+```bash
+git branch
+```
+
+Push to whichever branch your Space expects (usually `main`).
+
+---
+
+## Security Reminder
+
+- **Never** commit your HF token or API keys to the repo.
+- If a token is accidentally exposed, revoke it immediately at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) and generate a new one.
+- Use environment variables or HuggingFace Space **Secrets** (Settings > Variables and secrets) for sensitive values.

README_RAG.md

@@ -0,0 +1,239 @@
+# 🗼 RAG Chat API — Gustave Eiffel Hackathon 2026
+
+A complete **Retrieval-Augmented Generation (RAG)** system deployed as a Hugging Face Space, with a `/query` API endpoint designed for the RAG evaluation system.
+
+---
+---
+
+## Overview
+
+This application demonstrates how to build a production-ready RAG system within the Hugging Face ecosystem. It covers:
+
+| Requirement | Solution |
+|---|---|
+| LLM API calls | Azure OpenAI (`gpt-5` via REST) |
+| Text → Embeddings | Azure OpenAI (`text-embedding-3-small` via REST) |
+| Vector Store | ChromaDB (persistent, runs in-process) |
+| API Endpoint | FastAPI with `POST /query` |
+| UI | Gradio Blocks (chat + document ingestion) |
+
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    Hugging Face Space                         │
+│                                                              │
+│  ┌──────────┐     ┌──────────────┐     ┌───────────────┐   │
+│  │  Gradio  │     │   FastAPI    │     │   ChromaDB    │   │
+│  │   UI     │────▶│  /query      │────▶│  Vector Store │   │
+│  │          │     │  /ingest     │     │  (persistent) │   │
+│  └──────────┘     └──────┬───────┘     └───────────────┘   │
+│                          │                      ▲            │
+│                          ▼                      │            │
+│               ┌──────────────────┐    ┌─────────────────┐   │
+│               │  Azure OpenAI    │    │  Azure OpenAI   │   │
+│               │  GPT-5 (LLM)    │    │  text-embedding │   │
+│               │                  │    │  -3-small       │   │
+│               └──────────────────┘    └─────────────────┘   │
+└─────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## Step-by-Step Explanation
+
+### Step 1: Document Ingestion & Chunking
+
+Before we can answer questions, we need to prepare our knowledge base.
+
+1. **Load documents** — Read text files from `sample_documents/` directory
+2. **Chunk text** — Split documents into smaller overlapping chunks (512 tokens, 50 token overlap) using `RecursiveCharacterTextSplitter`. This ensures each chunk fits within the embedding model's context window while maintaining semantic coherence.
+
+```python
+splitter = RecursiveCharacterTextSplitter(
+    chunk_size=512,
+    chunk_overlap=50,
+    separators=["\n\n", "\n", ". ", " ", ""],
+)
+chunks = splitter.split_text(document_text)
+```
+
+### Step 2: Generate Embeddings
+
+Convert text chunks into dense vector representations that capture semantic meaning.
+
+1. **Call Azure OpenAI** — We use the `text-embedding-3-small` model via the Azure OpenAI embeddings endpoint
+2. **Encode text** — Each chunk is transformed into a fixed-size vector where semantically similar texts are closer together in vector space
+
+```python
+import requests as http_requests
+
+headers = {"api-key": AZURE_API_KEY, "Content-Type": "application/json"}
+payload = {"input": ["chunk 1 text", "chunk 2 text"], "model": "text-embedding-3-small"}
+resp = http_requests.post(EMBEDDING_ENDPOINT_URL, headers=headers, json=payload)
+embeddings = [item["embedding"] for item in resp.json()["data"]]
+```
+
+### Step 3: Store in Vector Database (ChromaDB)
+
+Persist embeddings in a vector store optimized for similarity search.
+
+1. **Initialize ChromaDB** — Create a persistent client that stores data on disk (survives Space restarts)
+2. **Create collection** — A named collection with cosine similarity metric
+3. **Add documents** — Store embeddings alongside the original text and metadata
+
+```python
+import chromadb
+
+client = chromadb.PersistentClient(path="./data/chroma_db")
+collection = client.get_or_create_collection(
+    name="rag_documents",
+    metadata={"hnsw:space": "cosine"},
+)
+collection.add(
+    ids=["doc_0", "doc_1"],
+    embeddings=embeddings.tolist(),
+    documents=["chunk 1 text", "chunk 2 text"],
+    metadatas=[{"source": "file.txt"}, {"source": "file.txt"}],
+)
+```
+
+### Step 4: Query & Retrieval
+
+When a user asks a question, find the most relevant context.
+
+1. **Embed the query** — Use the same Azure OpenAI embedding model to convert the question to a vector
+2. **Similarity search** — Find the top-K nearest vectors in ChromaDB (cosine similarity)
+3. **Return context** — Extract the original text chunks for the closest matches
+
+```python
+query_embedding = generate_embeddings(["What is the Eiffel Tower?"])[0]
+results = collection.query(
+    query_embeddings=[query_embedding],
+    n_results=3,
+)
+```
+
+### Step 5: LLM Generation (Augmented Response)
+
+Combine retrieved context with the user's question and generate an answer.
+
+1. **Build prompt** — Load the template from [`prompts/rag_prompt.txt`](prompts/rag_prompt.txt), inject retrieved context and the user's question
+2. **Call Azure OpenAI** — Send the prompt to the Azure OpenAI chat/completions endpoint (`gpt-5`)
+3. **Return response** — The LLM generates an answer grounded in the provided context
+
+The prompt template (`prompts/rag_prompt.txt`):
+
+```
+You are a helpful assistant. Answer the user's question based ONLY on the provided context.
+If the context does not contain enough information to answer, say "I don't have enough information to answer this question."
+Always be concise and factual.
+
+Context:
+{context}
+
+Question: {question}
+```
+
+The template is loaded once at startup and sent as the user message to the chat endpoint:
+
+```python
+RAG_PROMPT_TEMPLATE = Path("prompts/rag_prompt.txt").read_text(encoding="utf-8")
+
+# At query time:
+prompt = RAG_PROMPT_TEMPLATE.format(context=context_text, question=user_query)
+headers = {"api-key": AZURE_API_KEY, "Content-Type": "application/json"}
+payload = {
+    "model": "gpt-5",
+    "messages": [{"role": "user", "content": prompt}],
+    "max_completion_tokens": 512,
+    "temperature": 0.7,
+    "top_p": 0.95,
+}
+resp = requests.post(LLM_ENDPOINT_URL, headers=headers, json=payload)
+answer = resp.json()["choices"][0]["message"]["content"]
+```
+
+> **Tip:** Edit `prompts/rag_prompt.txt` to tune the model's behaviour (tone, language, output format) without touching application code.
+
+### Step 6: API Endpoint (`/query`)
+
+The FastAPI endpoint ties everything together for the evaluation system.
+
+```python
+@app.post("/query")
+async def query_endpoint(request: QueryRequest):
+    # 1. Retrieve relevant context
+    # 2. Build augmented prompt
+    # 3. Generate LLM response
+    # 4. Return answer + sources
+    result = rag_query(request.query, top_k=request.top_k)
+    return JSONResponse(content=result)
+```
+
+---
+
+## API Endpoints
+
+### `POST /query`
+
+The primary endpoint for the RAG evaluation system.
+
+**Request:**
+```json
+{
+    "query": "What materials is the Eiffel Tower made of?",
+    "top_k": 3
+}
+```
+
+**Response:**
+```json
+{
+    "answer": "The Eiffel Tower is made of wrought iron (puddled iron)...",
+    "sources": [
+        {"source": "eiffel_tower.txt", "score": 0.87},
+        {"source": "paris_landmarks.txt", "score": 0.72}
+    ],
+    "query": "What materials is the Eiffel Tower made of?"
+}
+```
+
+### `POST /ingest`
+
+Add new documents to the knowledge base.
+
+**Request:**
+```json
+{
+    "text": "The Eiffel Tower was built in 1889...",
+    "source": "my_document.txt"
+}
+```
+
+**Response:**
+```json
+{
+    "status": "success",
+    "chunks_added": 5,
+    "total_chunks": 42
+}
+```
+
+### `GET /health`
+
+System health check.
+
+**Response:**
+```json
+{
+    "status": "healthy",
+    "documents_in_store": 42,
+    "embedding_model": "text-embedding-3-small",
+    "llm_model": "gpt-5"
+}
+```
+
+---

requirements.txt

@@ -1,14 +1,15 @@
 fastapi==0.115.0
 uvicorn==0.30.0
 gradio==4.44.1
-chromadb==0.5.0
-huggingface-hub==0.25.0
-langchain==0.3.0
-langchain-community==0.3.0
-langchain-huggingface==0.1.0
-langchain-chroma==0.2.0
-langchain-text-splitters==0.3.0
-pypdf==4.3.0
-python-multipart==0.0.9
-pydantic==2.9.0
+chromadb>=1.0.0
+sentence-transformers>=3.0.0
+huggingface-hub==0.36.2
+langchain>=0.3.0
+langchain-community>=0.3.0
+langchain-huggingface>=0.1.0
+langchain-chroma>=0.2.2
+langchain-text-splitters>=0.3.0
+pypdf>=4.3.0
+python-multipart>=0.0.9
+pydantic>=2.9.0
 requests>=2.31.0