live evaluations

.dockerignore

@@ -0,0 +1,12 @@
+venv/
+.venv/
+__pycache__/
+*.pyc
+.git/
+.gitignore
+.env
+*.log
+evaluation_report.csv
+.DS_Store
+tests/
+assets/

Dockerfile

@@ -1,23 +1,29 @@
-# Use an official Python runtime as a parent image
+# Docker image for HuggingFace Spaces (SDK: docker, app_port: 7860)
 FROM python:3.10-slim
 
-# Set the working directory in the container
-WORKDIR /app
+# HF Spaces run containers as a non-root user; create one with a writable home
+RUN useradd -m -u 1000 user
+USER user
+
+ENV HOME=/home/user \
+    PATH=/home/user/.local/bin:$PATH \
+    HF_HOME=/home/user/.cache/huggingface \
+    PYTHONUNBUFFERED=1 \
+    TOKENIZERS_PARALLELISM=false
 
-# Copy the requirements file into the container
-COPY requirements.txt .
+WORKDIR /app
 
-# Install any needed packages specified in requirements.txt
-RUN pip install --no-cache-dir -r requirements.txt
+# Install Python dependencies
+COPY --chown=user:user requirements.txt .
+RUN pip install --no-cache-dir --user -r requirements.txt
 
-# Copy the current directory contents into the container at /app
-COPY . .
+# Copy the application code
+COPY --chown=user:user . .
 
-# Expose the port FastAPI runs on
-EXPOSE 8000
+# Bake the embedding + NLI models into the image (fast, offline cold starts)
+RUN python -m src.prefetch_models
 
-# Define environment variable for unbuffered logs 
-ENV PYTHONUNBUFFERED=1
+# Gradio UI (public) runs on 7860; FastAPI backend runs internally on 8000
+EXPOSE 7860
 
-# Run the application
-CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]
+CMD ["bash", "start.sh"]

README.md

@@ -1,7 +1,14 @@
+
 # Agentic RAG Knowledge Search
 
 An Autonomous AI Microservice with Hybrid Retrieval & Self-Evaluation. Built with FastAPI, LangChain, Docker, and Google Gemini.
 
+> **Deploying on HuggingFace Spaces:** create a **Docker** Space and push this repo. No API-key
+> secret is required — the app uses **Bring Your Own Key (BYOK)**: each visitor enters their own
+> Google Gemini key in the UI, so the public demo never spends your quota. The container runs the
+> FastAPI backend (internal, port 8000) and the Gradio UI (public, port 7860) together via
+> `start.sh`. Models are baked into the image at build time, so cold starts are fast.
+
 ## Overview
 
 This project is a Production-Grade AI Microservice designed to solve the "Knowledge Silo" problem. Unlike traditional RAG systems that only look at internal documents, this Agentic System intelligently decides where to find the answer.
@@ -33,6 +40,35 @@ The system follows a Hybrid RAG architecture. The Agent acts as the central brai
   
 - REST API: Fully documented API using FastAPI.
 
+- User Document Upload: Users can upload their own files (PDF, DOCX, TXT, MD, CSV) at runtime. Uploaded documents are indexed instantly and searched first; if nothing is uploaded, the agent falls back to the built-in legal/policy document.
+
+- Live, No-Cost Evaluation Metrics: Every answer is scored in real time using local models only (no extra API calls) — see [Live Evaluation Metrics](#live-evaluation-metrics) below.
+
+- Single-Image Deployment: The FastAPI backend and Gradio UI run together from one Docker image (`start.sh`), ready for HuggingFace Spaces.
+
+## User Document Upload
+
+The Gradio UI includes an upload panel. Users can drag in one or more files and click **Process & Index Files**.
+
+- Supported formats: `.pdf`, `.docx`, `.txt`, `.md`, `.csv`
+- Multiple files can be uploaded; new uploads are merged into the existing index.
+- **Clear Uploaded Documents** resets the index back to the built-in default.
+- If no files are uploaded, the agent uses the bundled `data/policy.pdf` as the default knowledge base.
+
+## Live Evaluation Metrics
+
+After every response, three metrics are computed **locally — no extra LLM/API calls** — and shown in the UI. This keeps hallucination/quality monitoring free and fast, even on CPU-only HuggingFace Spaces.
+
+| Metric | Always shown? | How it works | What it catches |
+|---|---|---|---|
+| **Faithfulness** | Yes | NLI entailment (`cross-encoder/nli-deberta-v3-small`): each answer sentence is checked for *entailment* against the best-matching source passage it actually used (documents **or** web results). | Hallucinations and contradictions — not just topic drift. A claim that contradicts the source scores near 0. |
+| **Answer Relevance** | Yes | Cosine similarity between the question and the answer (`all-MiniLM-L6-v2`). Needs no reference. | Off-topic or evasive answers. |
+| **Accuracy** | Only with a reference | ROUGE-L F1 between the answer and a user-supplied reference answer. | Drift from a known-correct answer. |
+
+> **Why NLI instead of plain similarity?** Cosine similarity measures *topical* overlap, so "the treaty can be terminated" and "the treaty cannot be terminated" score nearly identically despite opposite meaning. The NLI model checks logical *entailment*, so it correctly flags contradictions as unfaithful.
+
+The offline `tests/evaluate.py` pipeline additionally uses an LLM-as-a-Judge for a second opinion against a golden dataset (see [Running Evaluations](#running-evaluations)).
+
 ## Demo & Outputs
 
 1. Interactive API (Swagger UI)
@@ -58,7 +94,13 @@ A generated CSV report scoring the agent's performance against ground truth data
 - src/rag_engine.py: Handles PDF ingestion and Vector Database (FAISS).
 - src/agent.py: Defines the Agent, Tools, and LangChain logic.
 - src/main.py: The FastAPI server entry point.
-- data/: Place your PDF documents here.
+- src/file_processor.py: Indexes user-uploaded files (PDF/DOCX/TXT/MD/CSV) at runtime.
+- src/embeddings.py: Shared, single-load embedding model reused by RAG and the evaluator.
+- src/evaluator.py: Local evaluation metrics (NLI faithfulness, relevance, accuracy).
+- src/prefetch_models.py: Downloads models at image-build time for fast cold starts.
+- app.py: The Gradio user interface (chat, file upload, live metrics).
+- start.sh: Launches the FastAPI backend and Gradio UI together (used by Docker).
+- data/: Place your default PDF documents here. (used when user didn't upload any files)
 
 ### Prerequisites
 
@@ -73,8 +115,10 @@ A generated CSV report scoring the agent's performance against ground truth data
 
 2. Configure Environment
 
-    Create a .env file in the root directory:
-    
+    The app uses **BYOK** — you enter your Gemini key directly in the UI, so no `.env` is needed to chat.
+
+    A `.env` file is only required to run the **offline evaluation** suite (`tests/evaluate.py`):
+
     ```GOOGLE_API_KEY=your_actual_api_key_here```
 
 3. Add Data
@@ -109,6 +153,40 @@ A generated CSV report scoring the agent's performance against ground truth data
     
     This isolates the application and ensures it runs consistently on any machine.
 
+6. Option C: Run the full app (UI + API together)
+
+    To run the Gradio UI and FastAPI backend together exactly as they run in the container:
+
+    ```bash start.sh```
+
+    Then open the UI at http://localhost:7860 (the API stays internal on port 8000).
+    To run them separately during development, use two terminals: `python -m src.main` and `python app.py`.
+
+## Deploying on HuggingFace Spaces
+
+This repo is configured as a **Docker** Space (see the YAML frontmatter at the top of this file).
+
+1. Create a new Space → choose **Docker** as the SDK.
+2. Push this repository to the Space.
+3. HuggingFace builds the image and serves the Gradio UI at the Space URL.
+
+No API-key secret is needed: the app uses **Bring Your Own Key (BYOK)** — each visitor enters their own Google Gemini key in the UI (see below).
+
+The container runs the FastAPI backend (internal, port 8000) and the Gradio UI (public, port 7860) together via `start.sh`. The embedding and NLI models are baked into the image at build time, so cold starts are fast and require no network access for models.
+
+## API Key Strategy for Public Deployment
+
+The Gemini free tier allows roughly **1,500 requests/day**, and each user question costs **2 calls** (one to route, one to answer) — about **750 questions/day total, shared across everyone**. If you publish a Space using your personal key, public traffic will exhaust it quickly and run on *your* quota.
+
+**This app uses Bring Your Own Key (BYOK).** Each visitor enters their own Google Gemini API key in the UI:
+
+- The key is sent only with that visitor's requests and is **never stored** on the server.
+- A new free key takes seconds to create at [Google AI Studio](https://aistudio.google.com/apikey).
+- The public Space therefore costs **you nothing** and can never exhaust your personal quota.
+- A question cannot be submitted until a key is provided; an invalid key returns a clear error.
+
+This is the standard pattern for public LLM demos. (Alternatives, if you ever want them: keep your own key as a Space secret with rate limiting, make the Space private, or upgrade to a paid Gemini plan.)
+
 ## Running Evaluations
 
 This project prioritizes reliability. You can run the evaluation suite to test the agent against a "Golden Dataset" of questions and ground truths.

app.py

@@ -1,13 +1,9 @@
 import os
-import logging
 import gradio as gr
 import requests
 
 os.environ["TOKENIZERS_PARALLELISM"] = "false"
 
-logging.basicConfig(level=logging.INFO)
-logger = logging.getLogger(__name__)
-
 FASTAPI_URL = os.getenv("FASTAPI_URL", "http://127.0.0.1:8000")
 CHAT_ENDPOINT = f"{FASTAPI_URL}/chat"
 UPLOAD_ENDPOINT = f"{FASTAPI_URL}/upload"
@@ -15,6 +11,18 @@ RESET_ENDPOINT = f"{FASTAPI_URL}/reset"
 
 SUPPORTED_TYPES = [".pdf", ".docx", ".txt", ".md", ".csv"]
 
+CSS = """
+#submit-btn { margin-top: auto; margin-bottom: auto; height: 80px; }
+#input-row  { align-items: center; }
+"""
+
+SOURCE_LABELS = {
+    "rag":     "Uploaded Documents (RAG)",
+    "web":     "Web Search",
+    "rag+web": "Documents + Web Search",
+    "unknown": "Unknown",
+}
+
 
 def upload_files(files) -> str:
     if not files:
@@ -23,10 +31,8 @@ def upload_files(files) -> str:
         multipart = []
         for f in files:
             path = f if isinstance(f, str) else f.name
-            filename = os.path.basename(path)
             with open(path, "rb") as fp:
-                multipart.append(("files", (filename, fp.read(), "application/octet-stream")))
-
+                multipart.append(("files", (os.path.basename(path), fp.read(), "application/octet-stream")))
         resp = requests.post(UPLOAD_ENDPOINT, files=multipart, timeout=120)
         resp.raise_for_status()
         return resp.json().get("status", "Files processed.")
@@ -43,101 +49,200 @@ def reset_documents() -> str:
         return f"Reset failed: {e}"
 
 
-def determine_source(text: str) -> str:
-    lower = text.lower()
-    if any(k in lower for k in ["search", "web", "duckduckgo", "internet", "online"]):
-        return "Web Search"
-    if any(k in lower for k in ["uploaded", "file:", "page", "document", "policy", "pdf"]):
-        return "Uploaded Documents (RAG)"
-    return "RAG + Web Search"
+def _bar(value: float, width: int = 12) -> str:
+    filled = round(value * width)
+    return "█" * filled + "░" * (width - filled)
+
+
+def _fmt(label: str, value: float, hint: str) -> str:
+    return f"**{label}**  \n`{_bar(value)}  {value:.2f} / 1.0`  \n_{hint}_\n"
+
+
+def _faith_hint(score: float, source: str) -> str:
+    src = SOURCE_LABELS.get(source, source)
+    what = "web results" if source == "web" else "retrieved documents"
+    if score >= 0.75:
+        return f"Answer is well-grounded in the {what}"
+    if score >= 0.50:
+        return f"Answer is mostly grounded in the {what}, minor unsupported details possible"
+    return f"Low grounding — answer may contain content not present in the {what}"
+
+
+def _relevance_hint(score: float) -> str:
+    if score >= 0.70:
+        return "Answer directly addresses the question"
+    if score >= 0.45:
+        return "Answer is related but may not fully address the question"
+    return "Answer may be off-topic or incomplete"
+
 
+def _accuracy_hint(score: float) -> str:
+    if score >= 0.75:
+        return "Strong match with your reference answer"
+    if score >= 0.40:
+        return "Partial match — some key points differ from your reference"
+    return "Low overlap with your reference answer"
 
-def process_query(message: str, chat_history: list) -> tuple[list, str]:
+
+def _format_metrics(source: str, faithfulness, answer_relevance, accuracy) -> str:
+    if faithfulness is None and answer_relevance is None:
+        return ""
+
+    src_label = SOURCE_LABELS.get(source, source)
+    lines = [f"**Answer source: {src_label}**", "---"]
+
+    if faithfulness is not None:
+        faith_label = "Faithfulness — grounded in documents" if source != "web" else "Faithfulness — grounded in web results"
+        lines.append(_fmt(faith_label, faithfulness, _faith_hint(faithfulness, source)))
+
+    if answer_relevance is not None:
+        lines.append(_fmt("Answer Relevance — addresses the question", answer_relevance, _relevance_hint(answer_relevance)))
+
+    if accuracy is not None:
+        lines.append(_fmt("Accuracy — matches your reference", accuracy, _accuracy_hint(accuracy)))
+    else:
+        lines.append("_Accuracy: provide a reference answer above to see this score._")
+
+    return "\n".join(lines)
+
+
+def process_query(message: str, api_key: str, reference: str, chat_history: list) -> tuple[list, str, str]:
+    if not api_key.strip():
+        return chat_history, "Enter your Google Gemini API key above to start.", ""
     if not message.strip():
-        return chat_history, "Please enter a question."
+        return chat_history, "Please enter a question.", ""
     try:
-        resp = requests.post(CHAT_ENDPOINT, json={"query": message}, timeout=120)
-        resp.raise_for_status()
-        response_text = resp.json().get("response", "")
+        payload = {"query": message, "api_key": api_key.strip()}
+        if reference.strip():
+            payload["reference"] = reference.strip()
+
+        resp = requests.post(CHAT_ENDPOINT, json=payload, timeout=180)
+
+        if resp.status_code in (400, 401, 429):
+            detail = resp.json().get("detail", "Request could not be completed.")
+            chat_history.append({"role": "user", "content": message})
+            chat_history.append({"role": "assistant", "content": f"Sorry, I can't respond right now:\n\n{detail}"})
+            return chat_history, detail, ""
 
-        source = determine_source(response_text)
-        full_response = f"{response_text}\n\n--- Source: {source} ---"
+        resp.raise_for_status()
+        data = resp.json()
+        response_text = data.get("response", "")
+        source = data.get("source", "unknown")
 
+        src_label = SOURCE_LABELS.get(source, source)
         chat_history.append({"role": "user", "content": message})
-        chat_history.append({"role": "assistant", "content": full_response})
-        return chat_history, f"Done — answered via {source}"
+        chat_history.append({"role": "assistant", "content": f"{response_text}\n\n--- Source: {src_label} ---"})
+
+        metrics_md = _format_metrics(
+            source,
+            data.get("faithfulness"),
+            data.get("answer_relevance"),
+            data.get("accuracy"),
+        )
+        return chat_history, f"Done — answered via {src_label}", metrics_md
+
     except requests.exceptions.RequestException as e:
         error = f"Request failed: {e}"
         chat_history.append({"role": "user", "content": message})
         chat_history.append({"role": "assistant", "content": f"ERROR: {error}"})
-        return chat_history, error
+        return chat_history, error, ""
 
 
-def clear_chat() -> tuple[list, str]:
-    return [], ""
+def clear_chat() -> tuple[list, str, str]:
+    return [], "", ""
 
 
-# --- UI ---
 with gr.Blocks(title="Agentic RAG Knowledge Search") as demo:
     gr.Markdown("# Agentic RAG Knowledge Search")
     gr.Markdown(
-        "Upload your documents, then ask questions. "
-        "The agent searches your files first, then the web if needed."
+        "Upload your own documents and ask questions. "
+        "The agent searches your files via RAG first, then falls back to web search if needed. "
+        "**No files uploaded?** The agent uses a built-in legal/policy document as the default knowledge base."
     )
 
-    # File upload panel
     with gr.Group():
-        gr.Markdown("### Upload Documents")
-        gr.Markdown(f"Supported: {', '.join(SUPPORTED_TYPES)}")
-        with gr.Row():
-            file_input = gr.File(
-                label="Select Files",
-                file_count="multiple",
-                file_types=SUPPORTED_TYPES,
-            )
+        gr.Markdown(
+            "### Your Gemini API Key (required)\n"
+            "This app uses **your own** Google Gemini key — it is sent only with your requests and never stored. "
+            "Get a free key at [Google AI Studio](https://aistudio.google.com/apikey)."
+        )
+        api_key_input = gr.Textbox(
+            label="Google Gemini API Key",
+            placeholder="Paste your API key here (starts with AIza...)",
+            type="password",
+            lines=1,
+        )
+
+    with gr.Group():
+        gr.Markdown(
+            f"### Upload Documents\n"
+            f"Supported: **{', '.join(SUPPORTED_TYPES)}** — multiple files allowed, new uploads add to the index."
+        )
+        file_input = gr.File(label="Select Files", file_count="multiple", file_types=SUPPORTED_TYPES)
         with gr.Row():
-            upload_btn = gr.Button("Process Files", variant="primary")
+            upload_btn = gr.Button("Process & Index Files", variant="primary")
             reset_btn = gr.Button("Clear Uploaded Documents", variant="secondary")
-        upload_status = gr.Textbox(label="Upload Status", interactive=False, lines=2)
+        upload_status = gr.Textbox(label="Upload Status", interactive=False, lines=2,
+                                   placeholder="Upload status will appear here...")
 
     gr.Markdown("---")
 
-    # Chat panel
     with gr.Group():
-        gr.Markdown("### Ask a Question")
-        chatbot = gr.Chatbot(
-            label="Conversation",
-            height=420,
+        gr.Markdown(
+            "### Ask a Question\n"
+            "> Press **Enter** or click **Submit**. The agent decides whether to search your documents, the web, or both."
         )
-        with gr.Row():
+        chatbot = gr.Chatbot(label="Conversation", height=420)
+
+        with gr.Row(elem_id="input-row"):
             user_input = gr.Textbox(
                 placeholder="Ask anything about your documents or the web...",
-                label="Your Question",
-                lines=2,
-                scale=4,
+                lines=2, scale=5, show_label=False, container=False,
             )
-            submit_btn = gr.Button("Submit", variant="primary", scale=1)
+            submit_btn = gr.Button("Submit", variant="primary", scale=1, elem_id="submit-btn")
+
+        reference_input = gr.Textbox(
+            label="Reference Answer (optional)",
+            placeholder="Paste an expected correct answer here to also see an Accuracy score...",
+            lines=2,
+        )
+
         with gr.Row():
-            clear_btn = gr.Button("Clear Chat", variant="secondary")
-        status_output = gr.Textbox(label="Status", interactive=False, lines=1)
+            clear_btn = gr.Button("Clear Chat", variant="secondary", scale=1)
+
+        status_output = gr.Textbox(label="Status", interactive=False, lines=1,
+                                   placeholder="Status will appear here...")
+
+    gr.Markdown("---")
+
+    with gr.Group():
+        gr.Markdown(
+            "### Evaluation Metrics\n"
+            "Computed automatically after every response — **no extra API calls, no reference needed** for the first two.\n\n"
+            "| Metric | Always shown? | What it measures |\n"
+            "|---|---|---|\n"
+            "| **Faithfulness** | Yes | Is the answer grounded in the source it used? (docs or web results) |\n"
+            "| **Answer Relevance** | Yes | Does the answer actually address your question? |\n"
+            "| **Accuracy** | Only with reference | Does the answer match an expected correct answer? |"
+        )
+        metrics_output = gr.Markdown()
 
-    # Wiring
     upload_btn.click(fn=upload_files, inputs=[file_input], outputs=[upload_status])
     reset_btn.click(fn=reset_documents, inputs=[], outputs=[upload_status])
 
     submit_btn.click(
         fn=process_query,
-        inputs=[user_input, chatbot],
-        outputs=[chatbot, status_output],
+        inputs=[user_input, api_key_input, reference_input, chatbot],
+        outputs=[chatbot, status_output, metrics_output],
     ).then(fn=lambda: "", inputs=[], outputs=[user_input])
 
     user_input.submit(
         fn=process_query,
-        inputs=[user_input, chatbot],
-        outputs=[chatbot, status_output],
+        inputs=[user_input, api_key_input, reference_input, chatbot],
+        outputs=[chatbot, status_output, metrics_output],
     ).then(fn=lambda: "", inputs=[], outputs=[user_input])
 
-    clear_btn.click(fn=clear_chat, inputs=[], outputs=[chatbot, status_output])
+    clear_btn.click(fn=clear_chat, inputs=[], outputs=[chatbot, status_output, metrics_output])
 
 if __name__ == "__main__":
-    demo.launch(server_name="0.0.0.0", server_port=7860, share=False, theme=gr.themes.Soft())
+    demo.launch(server_name="0.0.0.0", server_port=7860, share=False, theme=gr.themes.Soft(), css=CSS)

requirements.txt

@@ -1,4 +1,3 @@
-
 fastapi>=0.115.0
 uvicorn>=0.30.0
 python-dotenv>=1.0.1
@@ -7,18 +6,15 @@ langchain-community>=0.3.7
 langchain-core>=0.3.15
 langchain-text-splitters>=0.3.2
 langchain-google-genai>=2.0.4
-
-google-generativeai==0.8.2
-
 langgraph>=0.2.53
 langchain-huggingface>=0.1.2
 sentence-transformers>=3.2.0
+sentencepiece>=0.2.0
 faiss-cpu>=1.7.4
 pypdf>=5.1.0
 docx2txt>=0.8
-duckduckgo-search==5.3.1
+rouge-score>=0.1.2
+ddgs
 pydantic>=2.9.0
 requests>=2.32.0
-datasets>=3.1.0
-ddgs
 gradio>=4.44.0

src/agent.py

@@ -1,4 +1,5 @@
 import os
+from functools import lru_cache
 from langchain_google_genai import ChatGoogleGenerativeAI
 from langchain_core.tools import tool
 from langchain_community.tools import DuckDuckGoSearchRun
@@ -9,19 +10,16 @@ from src.file_processor import FileProcessor
 
 load_dotenv()
 
-# --- Shared state ---
-# file_processor is imported and mutated by main.py's /upload endpoint
 file_processor = FileProcessor()
 
-# Optional fallback KB (original policy.pdf); silently skipped if missing
-_PDF_PATH = os.path.join("data", "policy.pdf")
-_fallback_kb = KnowledgeBase(pdf_path=_PDF_PATH)
+_fallback_kb = KnowledgeBase(pdf_path=os.path.join("data", "policy.pdf"))
 try:
     _fallback_kb.load_and_index()
 except Exception as e:
     print(f"Fallback KB skipped: {e}")
 
-# --- Tools ---
+_search_tool = DuckDuckGoSearchRun()
+
 
 @tool
 def lookup_documents(query: str) -> str:
@@ -31,25 +29,30 @@ def lookup_documents(query: str) -> str:
         result = file_processor.retrieve(query)
         if result:
             return result
-    # Fall back to original KB when no uploads exist
     return _fallback_kb.retrieve(query)
 
-search_tool = DuckDuckGoSearchRun()
 
 @tool
 def search_web(query: str) -> str:
     """Search the web for current events, news, or general knowledge not in uploaded documents."""
     try:
-        return search_tool.run(query)
+        return _search_tool.run(query)
     except Exception as e:
         return f"Search failed: {e}"
 
-# --- Agent factory ---
-
-def get_agent_executor():
-    if not os.getenv("GOOGLE_API_KEY"):
-        raise ValueError("GOOGLE_API_KEY not found in .env file")
 
-    print("Initializing Gemini Agent (Model: gemini-2.5-flash-lite)...")
-    llm = ChatGoogleGenerativeAI(model="gemini-2.5-flash-lite", temperature=0)
+@lru_cache(maxsize=32)
+def get_agent_executor(api_key: str):
+    """Build (and cache) a LangGraph agent for the given Gemini API key.
+
+    Each visitor supplies their own key (BYOK), so the LLM is created per key.
+    The shared tools, embeddings, and RAG index are module-level and reused.
+    Cached by key so repeat requests from the same user don't rebuild the graph."""
+    if not api_key or not api_key.strip():
+        raise ValueError("A Google Gemini API key is required.")
+    llm = ChatGoogleGenerativeAI(
+        model="gemini-2.0-flash",
+        temperature=0,
+        google_api_key=api_key.strip(),
+    )
     return create_react_agent(llm, [lookup_documents, search_web])

src/embeddings.py

@@ -0,0 +1,33 @@
+"""
+Shared embedding model — loaded once and reused everywhere.
+
+The same all-MiniLM-L6-v2 model is needed by:
+  - the RAG vector stores (via langchain's HuggingFaceEmbeddings)
+  - the evaluator's cosine-similarity metrics (via the raw SentenceTransformer)
+
+Loading it a single time keeps memory and cold-start cost low on HF Spaces.
+"""
+
+from langchain_huggingface import HuggingFaceEmbeddings
+
+_MODEL_NAME = "all-MiniLM-L6-v2"
+_embeddings = None
+
+
+def get_embeddings() -> HuggingFaceEmbeddings:
+    """Shared langchain embeddings object for FAISS vector stores."""
+    global _embeddings
+    if _embeddings is None:
+        _embeddings = HuggingFaceEmbeddings(model_name=_MODEL_NAME)
+    return _embeddings
+
+
+def get_sentence_transformer():
+    """The underlying SentenceTransformer, reused by the evaluator.
+    Falls back to a direct load if langchain's internal attribute changes."""
+    embeddings = get_embeddings()
+    model = getattr(embeddings, "client", None)
+    if model is None:
+        from sentence_transformers import SentenceTransformer
+        model = SentenceTransformer(_MODEL_NAME)
+    return model

src/evaluator.py

@@ -0,0 +1,105 @@
+"""
+Local evaluation metrics — no LLM API calls, CPU-friendly for HF Spaces.
+
+faithfulness   : NLI entailment of each answer sentence against the best-matching
+                 source passage (docs or web results it actually used).
+                 Detects contradictions / unsupported claims, not just topic overlap.
+                 Falls back to cosine similarity if the NLI model can't load.
+answer_relevance: cosine similarity between question and answer — does the answer
+                 address the question? Needs no reference.
+accuracy       : ROUGE-L F1 vs a user-supplied reference answer. Only when provided.
+"""
+
+import re
+import logging
+import numpy as np
+from rouge_score import rouge_scorer
+from sentence_transformers import util
+from src.embeddings import get_sentence_transformer
+
+logger = logging.getLogger(__name__)
+
+_NLI_MODEL_NAME = "cross-encoder/nli-deberta-v3-small"
+# Label index for "entailment" in this model's output (0=contradiction, 1=entailment, 2=neutral)
+_ENTAILMENT_IDX = 1
+
+_nli_model = None
+_rouge = rouge_scorer.RougeScorer(["rougeL"], use_stemmer=True)
+
+
+# --- helpers ---------------------------------------------------------------
+
+def _nli():
+    global _nli_model
+    if _nli_model is None:
+        from sentence_transformers import CrossEncoder
+        _nli_model = CrossEncoder(_NLI_MODEL_NAME)
+    return _nli_model
+
+
+def _softmax(logits: np.ndarray) -> np.ndarray:
+    logits = np.atleast_2d(logits)
+    exp = np.exp(logits - logits.max(axis=1, keepdims=True))
+    return exp / exp.sum(axis=1, keepdims=True)
+
+
+def _cosine(text_a: str, text_b: str) -> float:
+    if not text_a.strip() or not text_b.strip():
+        return 0.0
+    m = get_sentence_transformer()
+    return round(float(util.cos_sim(
+        m.encode(text_a, convert_to_tensor=True),
+        m.encode(text_b, convert_to_tensor=True),
+    )), 3)
+
+
+def _split_sentences(text: str) -> list[str]:
+    parts = re.split(r"(?<=[.!?])\s+", text.strip())
+    return [p.strip() for p in parts if len(p.strip()) > 15]
+
+
+def _split_passages(context: str) -> list[str]:
+    # retrieve() joins passages with blank lines
+    parts = [p.strip() for p in context.split("\n\n") if p.strip()]
+    return parts or [context.strip()]
+
+
+# --- metrics ---------------------------------------------------------------
+
+def faithfulness_score(answer: str, source_context: str) -> float:
+    """Mean NLI entailment of answer claims given the source they were drawn from.
+
+    For each answer sentence we pick the most similar source passage (so the NLI
+    input stays short and on-point), then ask whether that passage entails the
+    sentence. Returns the mean entailment probability across sentences (0–1)."""
+    if not answer.strip() or not source_context.strip():
+        return 0.0
+
+    sentences = _split_sentences(answer) or [answer.strip()]
+    passages = _split_passages(source_context)
+
+    try:
+        model = get_sentence_transformer()
+        pas_emb = model.encode(passages, convert_to_tensor=True)
+        sen_emb = model.encode(sentences, convert_to_tensor=True)
+        best_idx = util.cos_sim(sen_emb, pas_emb).argmax(dim=1).tolist()
+        pairs = [(passages[best_idx[i]], sentences[i]) for i in range(len(sentences))]
+
+        logits = _nli().predict(pairs)
+        entail = _softmax(np.asarray(logits))[:, _ENTAILMENT_IDX]
+        return round(float(entail.mean()), 3)
+    except Exception as e:
+        logger.warning(f"NLI faithfulness unavailable ({e}); falling back to cosine.")
+        return _cosine(answer, source_context)
+
+
+def answer_relevance_score(question: str, answer: str) -> float:
+    """Does the answer address the question? Cosine similarity (0–1)."""
+    return _cosine(question, answer)
+
+
+def accuracy_score(answer: str, reference: str) -> float:
+    """ROUGE-L F1 vs a user-supplied reference answer (0–1)."""
+    if not answer.strip() or not reference.strip():
+        return 0.0
+    return round(_rouge.score(reference, answer)["rougeL"].fmeasure, 3)

src/file_processor.py

@@ -1,87 +1,66 @@
 import logging
 import os
-from langchain_community.document_loaders import (
-    PyPDFLoader,
-    TextLoader,
-    CSVLoader,
-    Docx2txtLoader,
-)
+from langchain_community.document_loaders import PyPDFLoader, TextLoader, CSVLoader, Docx2txtLoader
 from langchain_text_splitters import RecursiveCharacterTextSplitter
 from langchain_community.vectorstores import FAISS
-from langchain_huggingface import HuggingFaceEmbeddings
+from src.embeddings import get_embeddings
 
 logger = logging.getLogger(__name__)
 
 SUPPORTED_EXTENSIONS = {".pdf", ".txt", ".md", ".csv", ".docx"}
 
 
-def _loader_for(file_path: str):
-    ext = os.path.splitext(file_path)[1].lower()
+def _loader_for(path: str):
+    ext = os.path.splitext(path)[1].lower()
     if ext == ".pdf":
-        return PyPDFLoader(file_path)
+        return PyPDFLoader(path)
     if ext in (".txt", ".md"):
-        return TextLoader(file_path, encoding="utf-8")
+        return TextLoader(path, encoding="utf-8")
     if ext == ".csv":
-        return CSVLoader(file_path)
+        return CSVLoader(path)
     if ext == ".docx":
-        return Docx2txtLoader(file_path)
+        return Docx2txtLoader(path)
     return None
 
 
 class FileProcessor:
-    def __init__(self, embedding_model: str = "all-MiniLM-L6-v2"):
-        self.embeddings = HuggingFaceEmbeddings(model_name=embedding_model)
+    def __init__(self):
+        self.embeddings = get_embeddings()
         self.vector_store = None
         self.status = "No files processed"
-        self._splitter = RecursiveCharacterTextSplitter(
-            chunk_size=1000, chunk_overlap=200
-        )
+        self._splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
 
     def process_files(self, file_paths: list[str]) -> str:
-        """Index a list of file paths into the FAISS vector store."""
         if not file_paths:
             return "No files provided."
 
-        all_docs = []
-        skipped = []
-
+        all_docs, skipped = [], []
         for path in file_paths:
-            ext = os.path.splitext(path)[1].lower()
-            if ext not in SUPPORTED_EXTENSIONS:
+            if os.path.splitext(path)[1].lower() not in SUPPORTED_EXTENSIONS:
                 skipped.append(os.path.basename(path))
                 continue
             try:
-                loader = _loader_for(path)
-                docs = loader.load()
+                docs = _loader_for(path).load()
                 for d in docs:
                     d.metadata["source_file"] = os.path.basename(path)
                 all_docs.extend(docs)
-                logger.info(f"Loaded {len(docs)} page(s) from {os.path.basename(path)}")
             except Exception as e:
                 logger.error(f"Failed to load {path}: {e}")
                 skipped.append(os.path.basename(path))
 
         if not all_docs:
             self.status = "No supported files could be loaded"
-            return (
-                f"Could not load any files. "
-                f"Supported types: {', '.join(sorted(SUPPORTED_EXTENSIONS))}. "
-                + (f"Skipped: {', '.join(skipped)}" if skipped else "")
-            )
+            return f"Could not load any files. Supported: {', '.join(sorted(SUPPORTED_EXTENSIONS))}"
 
         chunks = self._splitter.split_documents(all_docs)
-
         if self.vector_store is None:
             self.vector_store = FAISS.from_documents(chunks, self.embeddings)
         else:
-            # Merge into existing index so previous uploads are retained
-            new_store = FAISS.from_documents(chunks, self.embeddings)
-            self.vector_store.merge_from(new_store)
+            self.vector_store.merge_from(FAISS.from_documents(chunks, self.embeddings))
 
         file_count = len(file_paths) - len(skipped)
         self.status = f"Indexed {len(chunks)} chunks from {file_count} file(s)"
         note = f" (skipped: {', '.join(skipped)})" if skipped else ""
-        logger.info(self.status)
         return f"Done: {self.status}{note}"
 
     def retrieve(self, query: str, k: int = 4) -> str:
@@ -89,11 +68,8 @@ class FileProcessor:
             return ""
         try:
             docs = self.vector_store.similarity_search(query, k=k)
-            if not docs:
-                return ""
             return "\n\n".join(
-                f"[File: {d.metadata.get('source_file', '?')}] {d.page_content}"
-                for d in docs
+                f"[File: {d.metadata.get('source_file', '?')}] {d.page_content}" for d in docs
             )
         except Exception as e:
             logger.error(f"Retrieval error: {e}")
@@ -108,4 +84,3 @@ class FileProcessor:
     def reset(self) -> None:
         self.vector_store = None
         self.status = "No files processed"
-        logger.info("FileProcessor reset")

src/main.py

@@ -1,50 +1,86 @@
 import os
+import re
+import time
 import shutil
 import tempfile
 import logging
+from typing import List, Optional
 from fastapi import FastAPI, HTTPException, UploadFile, File
 from fastapi.responses import JSONResponse
 from pydantic import BaseModel
-from typing import List
 from src.agent import get_agent_executor, file_processor
+from src.evaluator import faithfulness_score, answer_relevance_score, accuracy_score
 
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
 
-app = FastAPI(
-    title="Agentic RAG Service",
-    description="AI microservice that routes between uploaded documents and web search.",
-    version="3.0",
-)
-
-try:
-    agent_executor = get_agent_executor()
-except Exception as e:
-    logger.error(f"Failed to initialize agent: {e}")
-    agent_executor = None
+app = FastAPI(title="Agentic RAG Service", version="3.0")
 
 
 class QueryRequest(BaseModel):
     query: str
+    api_key: Optional[str] = None        # BYOK: each visitor supplies their own key
+    reference: Optional[str] = None
 
 
 class QueryResponse(BaseModel):
     response: str
+    source: str                          # "rag" | "web" | "rag+web"
+    faithfulness: Optional[float] = None
+    answer_relevance: Optional[float] = None
+    accuracy: Optional[float] = None
+
+
+def _retry_delay(error_str: str) -> float:
+    match = re.search(r"retryDelay.*?(\d+\.?\d*)s", error_str)
+    return float(match.group(1)) if match else 0.0
+
+
+def _extract_content(message) -> str:
+    content = message.content
+    if isinstance(content, list):
+        content = " ".join(
+            block["text"] if isinstance(block, dict) else str(block)
+            for block in content
+            if not isinstance(block, dict) or block.get("type") == "text"
+        )
+    return str(content)
+
+
+def _parse_tool_results(messages: list) -> tuple[str, str]:
+    """Return (source_type, combined_tool_output) from the agent message chain.
+
+    source_type is 'rag', 'web', or 'rag+web'.
+    combined_tool_output is the actual text the agent received from its tools,
+    which is what faithfulness should be measured against.
+    """
+    rag_parts, web_parts = [], []
+    for msg in messages:
+        # ToolMessage objects carry a .name attribute
+        name = getattr(msg, "name", None)
+        content = getattr(msg, "content", "") or ""
+        if name == "lookup_documents":
+            rag_parts.append(content)
+        elif name == "search_web":
+            web_parts.append(content)
+
+    if rag_parts and web_parts:
+        return "rag+web", " ".join(rag_parts + web_parts)
+    if rag_parts:
+        return "rag", " ".join(rag_parts)
+    if web_parts:
+        return "web", " ".join(web_parts)
+    return "unknown", ""
 
 
 @app.get("/")
 async def root():
-    return {
-        "status": "active",
-        "service": "Agentic Knowledge Search",
-        "docs_url": "/docs",
-        "uploaded_docs": file_processor.get_status(),
-    }
+    return {"status": "active", "service": "Agentic Knowledge Search",
+            "docs_url": "/docs", "uploaded_docs": file_processor.get_status()}
 
 
 @app.post("/upload")
 async def upload_files(files: List[UploadFile] = File(...)):
-    """Accept uploaded files, index them for RAG, return a status message."""
     tmp_dir = tempfile.mkdtemp()
     try:
         saved_paths = []
@@ -53,13 +89,9 @@ async def upload_files(files: List[UploadFile] = File(...)):
             with open(dest, "wb") as f:
                 shutil.copyfileobj(upload.file, f)
             saved_paths.append(dest)
-            logger.info(f"Saved upload: {upload.filename}")
-
         status = file_processor.process_files(saved_paths)
-        logger.info(f"Processing result: {status}")
         return JSONResponse({"status": status})
     except Exception as e:
-        logger.error(f"Upload error: {e}")
         raise HTTPException(status_code=500, detail=str(e))
     finally:
         shutil.rmtree(tmp_dir, ignore_errors=True)
@@ -67,33 +99,73 @@ async def upload_files(files: List[UploadFile] = File(...)):
 
 @app.post("/reset")
 async def reset_documents():
-    """Clear all uploaded documents from the index."""
     file_processor.reset()
     return {"status": "Uploaded documents cleared."}
 
 
+def _is_invalid_key(error_str: str) -> bool:
+    markers = ("API_KEY_INVALID", "API key not valid", "PERMISSION_DENIED", "API key expired")
+    return any(m in error_str for m in markers)
+
+
 @app.post("/chat", response_model=QueryResponse)
 async def chat(request: QueryRequest):
-    if not agent_executor:
-        raise HTTPException(status_code=500, detail="Agent not initialized (check API key)")
+    # BYOK: a key must be supplied with every request
+    if not request.api_key or not request.api_key.strip():
+        raise HTTPException(
+            status_code=400,
+            detail="Please enter your Google Gemini API key to ask a question.",
+        )
 
     try:
-        logger.info(f"Query: {request.query}")
-        result = agent_executor.invoke({"messages": [("user", request.query)]})
-        last_message = result["messages"][-1]
-
-        content = last_message.content
-        if isinstance(content, list):
-            content = " ".join(
-                block["text"] if isinstance(block, dict) else str(block)
-                for block in content
-                if not isinstance(block, dict) or block.get("type") == "text"
+        agent = get_agent_executor(request.api_key.strip())
+    except Exception as e:
+        raise HTTPException(status_code=400, detail=str(e))
+
+    for attempt in range(3):
+        try:
+            logger.info(f"Query (attempt {attempt + 1}): {request.query}")
+            result = agent.invoke({"messages": [("user", request.query)]})
+            answer = _extract_content(result["messages"][-1])
+
+            source, tool_output = _parse_tool_results(result["messages"])
+
+            # Faithfulness: answer vs the actual source content the agent used
+            faith = faithfulness_score(answer, tool_output) if tool_output else None
+
+            # Answer relevance: always computed, no reference needed
+            relevance = answer_relevance_score(request.query, answer)
+
+            # Accuracy: only when user provides a reference
+            acc = accuracy_score(answer, request.reference) if request.reference else None
+
+            return QueryResponse(
+                response=answer,
+                source=source,
+                faithfulness=faith,
+                answer_relevance=relevance,
+                accuracy=acc,
             )
 
-        return QueryResponse(response=str(content))
-    except Exception as e:
-        logger.error(f"Chat error: {e}")
-        raise HTTPException(status_code=500, detail=str(e))
+        except Exception as e:
+            error_str = str(e)
+            if _is_invalid_key(error_str):
+                raise HTTPException(
+                    status_code=401,
+                    detail="Invalid or unauthorized API key. Please check your Google Gemini API key.",
+                )
+            if "RESOURCE_EXHAUSTED" not in error_str:
+                logger.error(f"Chat error: {error_str}")
+                raise HTTPException(status_code=500, detail=error_str)
+            delay = _retry_delay(error_str)
+            if delay and delay <= 120 and attempt < 2:
+                logger.warning(f"Rate limited — retrying in {delay:.0f}s...")
+                time.sleep(delay + 1)
+                continue
+            raise HTTPException(
+                status_code=429,
+                detail="Daily API quota exhausted. Please wait until tomorrow or upgrade your Gemini API plan.",
+            )
 
 
 if __name__ == "__main__":

src/prefetch_models.py

@@ -0,0 +1,14 @@
+"""
+Download the embedding + NLI models at image-build time so the running
+container starts fast and never needs network access for models at runtime.
+"""
+
+from src.embeddings import get_sentence_transformer
+from src.evaluator import _nli
+
+if __name__ == "__main__":
+    print("Prefetching embedding model (all-MiniLM-L6-v2)...")
+    get_sentence_transformer()
+    print("Prefetching NLI model (cross-encoder/nli-deberta-v3-small)...")
+    _nli()
+    print("Models cached.")

src/rag_engine.py

@@ -1,64 +1,46 @@
 import os
+import logging
 from langchain_community.document_loaders import PyPDFLoader
 from langchain_text_splitters import RecursiveCharacterTextSplitter
 from langchain_community.vectorstores import FAISS
-from langchain_huggingface import HuggingFaceEmbeddings
+from src.embeddings import get_embeddings
 from dotenv import load_dotenv
 
 load_dotenv()
 
+logger = logging.getLogger(__name__)
+
+
 class KnowledgeBase:
     def __init__(self, pdf_path: str):
         self.pdf_path = pdf_path
         self.vector_store = None
-        # Initialize Local Embeddings
-        print("Initializing Local Embeddings (all-MiniLM-L6-v2)...")
-        self.embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
         self.index_path = "faiss_index"
-        
+        self.embeddings = get_embeddings()
+
     def load_and_index(self):
-        # 1. Try to load existing index
         if os.path.exists(self.index_path):
-            print("Found cached index on disk. Loading...")
             try:
                 self.vector_store = FAISS.load_local(
-                    self.index_path, 
-                    self.embeddings, 
-                    allow_dangerous_deserialization=True
+                    self.index_path, self.embeddings, allow_dangerous_deserialization=True
                 )
-                print("Loaded index from disk.")
+                logger.info("Loaded FAISS index from disk.")
                 return
             except Exception as e:
-                print(f"Could not load cache: {e}. Re-indexing...")
+                logger.warning(f"Could not load cached index: {e}. Re-indexing...")
 
-        # 2. Check PDF
         if not os.path.exists(self.pdf_path):
-            print(f"Warning: File not found at {self.pdf_path}")
+            logger.warning(f"PDF not found: {self.pdf_path}")
             return
 
-        # 3. Load and Split
-        print(f"Loading PDF from: {self.pdf_path}...")
-        loader = PyPDFLoader(self.pdf_path)
-        docs = loader.load()
-        
-        print("Splitting text into chunks...")
-        text_splitter = RecursiveCharacterTextSplitter(
-            chunk_size=1000, 
-            chunk_overlap=200
-        )
-        chunks = text_splitter.split_documents(docs)
-        
-        # 4. Create Index (Local)
-        print(f"Indexing {len(chunks)} chunks locally...")
+        docs = PyPDFLoader(self.pdf_path).load()
+        chunks = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200).split_documents(docs)
         self.vector_store = FAISS.from_documents(chunks, self.embeddings)
-        
-        # 5. Save
         self.vector_store.save_local(self.index_path)
-        print("Indexing Complete and Saved.")
+        logger.info(f"Indexed {len(chunks)} chunks and saved to disk.")
 
     def retrieve(self, query: str, k: int = 4) -> str:
         if not self.vector_store:
             return "No internal documents have been indexed."
-            
         docs = self.vector_store.similarity_search(query, k=k)
-        return "\n\n".join([f"[Source: Page {d.metadata.get('page', '?')}] {d.page_content}" for d in docs])
+        return "\n\n".join(f"[Source: Page {d.metadata.get('page', '?')}] {d.page_content}" for d in docs)

start.sh

@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+set -e
+
+# Start the FastAPI backend (internal, not exposed publicly)
+uvicorn src.main:app --host 0.0.0.0 --port 8000 &
+
+# Wait until the backend is ready before launching the UI,
+# so the first query never races against model loading.
+echo "Waiting for FastAPI backend on :8000 ..."
+python - <<'PY'
+import time, urllib.request
+for _ in range(120):
+    try:
+        urllib.request.urlopen("http://127.0.0.1:8000/", timeout=2)
+        print("Backend is up.")
+        break
+    except Exception:
+        time.sleep(2)
+else:
+    print("Backend did not become ready in time; starting UI anyway.")
+PY
+
+# Start the Gradio UI on the public HF Spaces port
+exec python app.py

tests/evaluate.py

@@ -1,104 +1,232 @@
+"""
+Evaluation pipeline — two metrics per question:
+  1. Faithfulness  : is the answer grounded in the retrieved context? (hallucination check)
+  2. Accuracy      : does the answer correctly match the ground truth?
+Run from the project root:  python -m tests.evaluate
+"""
+
 import sys
 import os
+import re
 import pandas as pd
 from langchain_google_genai import ChatGoogleGenerativeAI
-from langchain_core.prompts import PromptTemplate
+from dotenv import load_dotenv
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+load_dotenv()
 
-# Add 'src' to path
-sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 try:
-    from src.agent import get_agent_executor
+    from src.agent import get_agent_executor, file_processor, _fallback_kb
 except ImportError:
-    print("Could not import agent. Make sure you are running this from the root folder!")
+    print("Run this from the project root: python -m tests.evaluate")
     sys.exit(1)
 
-def run_evaluation():
-    print("Starting Custom Evaluation Pipeline (LLM-as-a-Judge)...")
-    
-    # 1. Setup the Judge (Gemini)
-    eval_llm = ChatGoogleGenerativeAI(
-        model="gemini-2.5-flash",
-        temperature=0
-    )
-
-    # 2. Define the Grading Logic
-    grading_prompt = PromptTemplate.from_template(
-        """
-        You are a strict teacher grading a student's answer.
-        
-        Question: {question}
-        Ground Truth: {ground_truth}
-        Student Answer: {answer}
-        
-        On a scale of 1-10, how accurate is the student's answer compared to the ground truth?
-        
-        IMPORTANT: 
-        - If the Student Answer matches the meaning of the Ground Truth, give a high score (8-10).
-        - If the Student Answer adds extra correct details from the document, that is GOOD.
-        - Only penalize if the information is factually wrong or completely unrelated.
-        
-        Format:
-        Score: [1-10]
-        Reason: [Text]
-        """
-    )
-
-    # 3. Test Data 
-    questions = [
-        "What are the reporting requirements for State Parties?",
-        "What happens if a State Party denounces the Protocol?",
-    ]
-    
-    ground_truths = [
-        "State Parties must submit a comprehensive report initially, followed by further information included in reports to the Committee on the Rights of the Child. Other State Parties need to submit reports every five years.", 
-        "Denunciation does not affect acts or situations occurring before the denunciation becomes effective. It also does not prejudice the continued consideration of matters already under consideration."
-    ]
-    
-    results = []
 
-    # 4. Run the Agent & Grade it
-    print("Generating and Grading answers...")
+# ---------------------------------------------------------------------------
+# Judge prompts
+# ---------------------------------------------------------------------------
+
+FAITHFULNESS_PROMPT = """\
+You are evaluating whether an AI answer is grounded in the provided source context.
+
+Question: {question}
+Retrieved Context: {context}
+AI Answer: {answer}
+
+Does the answer contain claims NOT supported by the retrieved context?
+A faithful answer only uses information present in the context.
+A hallucinated answer invents facts or adds information not in the context.
+
+Score 1-10 where:
+  9-10 = fully grounded, no unsupported claims
+  6-8  = mostly grounded, minor additions
+  3-5  = several unsupported claims
+  1-2  = mostly fabricated
+
+Format:
+Score: [1-10]
+Reason: [one sentence]"""
+
+ACCURACY_PROMPT = """\
+You are a strict teacher grading a student's answer.
+
+Question: {question}
+Ground Truth: {ground_truth}
+Student Answer: {answer}
+
+On a scale of 1-10, how accurate and complete is the student's answer compared to the ground truth?
+- 9-10: matches ground truth meaning, may add correct extra details
+- 6-8:  partially correct, missing some key points
+- 3-5:  relevant but significantly incomplete or partially wrong
+- 1-2:  incorrect or completely off-topic
+
+Format:
+Score: [1-10]
+Reason: [one sentence]"""
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+def extract_content(message) -> str:
+    content = message.content
+    if isinstance(content, list):
+        content = " ".join(
+            block["text"] if isinstance(block, dict) else str(block)
+            for block in content
+            if not isinstance(block, dict) or block.get("type") == "text"
+        )
+    return str(content)
+
+
+def parse_score(text: str) -> int:
+    match = re.search(r"Score:\s*(\d+)", text)
+    return int(match.group(1)) if match else 0
+
+
+def get_context(question: str) -> str:
+    """Retrieve the same context the agent would use for RAG questions."""
+    if file_processor.has_documents():
+        ctx = file_processor.retrieve(question)
+        if ctx:
+            return ctx
+    return _fallback_kb.retrieve(question)
+
+
+# ---------------------------------------------------------------------------
+# Test cases
+# ---------------------------------------------------------------------------
+# Add your own Q&A pairs here. For web-search questions, leave ground_truth
+# as None — only faithfulness will be skipped (no ground truth to compare).
+# ---------------------------------------------------------------------------
+
+TEST_CASES = [
+    {
+        "question": "What are the reporting requirements for State Parties?",
+        "ground_truth": (
+            "State Parties must submit a comprehensive report initially, followed by further "
+            "information included in reports to the Committee on the Rights of the Child. "
+            "Other State Parties need to submit reports every five years."
+        ),
+        "source": "rag",
+    },
+    {
+        "question": "What happens if a State Party denounces the Protocol?",
+        "ground_truth": (
+            "Denunciation does not affect acts or situations occurring before the denunciation "
+            "becomes effective. It also does not prejudice the continued consideration of matters "
+            "already under consideration."
+        ),
+        "source": "rag",
+    },
+]
+
+
+# ---------------------------------------------------------------------------
+# Main
+# ---------------------------------------------------------------------------
+
+def run_evaluation(test_cases: list = None):
+    cases = test_cases or TEST_CASES
+    print(f"Starting evaluation — {len(cases)} test case(s)\n")
+
+    api_key = os.getenv("GOOGLE_API_KEY")
+    if not api_key:
+        print("Set GOOGLE_API_KEY in your .env to run the offline evaluation.")
+        return
+
+    judge = ChatGoogleGenerativeAI(model="gemini-2.0-flash", temperature=0, google_api_key=api_key)
+
     try:
-        agent = get_agent_executor()
+        agent = get_agent_executor(api_key)
     except Exception as e:
-        print(f"Error initializing agent: {e}")
+        print(f"Could not initialize agent: {e}")
         return
 
-    for i, q in enumerate(questions):
-        print(f"\nTest Case {i+1}: {q}")
-        try:
-            # A. Get Agent Answer
-            inputs = {"messages": [("user", q)]}
-            result = agent.invoke(inputs)
-            agent_answer = result["messages"][-1].content
-            print(f"   Agent Answer: {agent_answer[:100]}...")
-
-            # B. Grade the Answer
-            grade_request = grading_prompt.format(
-                question=q,
-                ground_truth=ground_truths[i],
-                answer=agent_answer
-            )
-            grading_result = eval_llm.invoke(grade_request).content
-            
-            # C. Store Result
-            results.append({
-                "Question": q,
-                "Agent Answer": agent_answer,
-                "Ground Truth": ground_truths[i],
-                "Grading": grading_result
-            })
-            print(f"   Judge: {grading_result.splitlines()[0]}")
+    results = []
+
+    for i, case in enumerate(cases, 1):
+        question = case["question"]
+        ground_truth = case.get("ground_truth")
+        source = case.get("source", "rag")
 
+        print(f"[{i}/{len(cases)}] {question}")
+
+        # 1. Get agent answer
+        try:
+            result = agent.invoke({"messages": [("user", question)]})
+            answer = extract_content(result["messages"][-1])
         except Exception as e:
-            print(f"Failed: {e}")
+            print(f"  Agent error: {e}\n")
+            results.append({"Question": question, "Answer": f"ERROR: {e}",
+                            "Ground Truth": ground_truth,
+                            "Faithfulness Score": "-", "Faithfulness Reason": str(e),
+                            "Accuracy Score": "-", "Accuracy Reason": str(e)})
+            continue
+
+        print(f"  Answer: {answer[:120]}...")
+
+        # 2. Faithfulness check (hallucination detection) — RAG questions only
+        faithfulness_score, faithfulness_reason = "-", "N/A (web search question)"
+        if source == "rag":
+            try:
+                context = get_context(question)
+                response = judge.invoke(
+                    FAITHFULNESS_PROMPT.format(question=question, context=context, answer=answer)
+                )
+                faith_text = extract_content(response)
+                faithfulness_score = parse_score(faith_text)
+                faithfulness_reason = faith_text.split("Reason:")[-1].strip()
+                print(f"  Faithfulness: {faithfulness_score}/10")
+            except Exception as e:
+                faithfulness_reason = str(e)
+                print(f"  Faithfulness check failed: {e}")
+
+        # 3. Accuracy check — only if ground truth is provided
+        accuracy_score, accuracy_reason = "-", "N/A (no ground truth)"
+        if ground_truth:
+            try:
+                response = judge.invoke(
+                    ACCURACY_PROMPT.format(question=question, ground_truth=ground_truth, answer=answer)
+                )
+                acc_text = extract_content(response)
+                accuracy_score = parse_score(acc_text)
+                accuracy_reason = acc_text.split("Reason:")[-1].strip()
+                print(f"  Accuracy:     {accuracy_score}/10")
+            except Exception as e:
+                accuracy_reason = str(e)
+                print(f"  Accuracy check failed: {e}")
+
+        results.append({
+            "Question": question,
+            "Answer": answer,
+            "Ground Truth": ground_truth or "",
+            "Faithfulness Score": faithfulness_score,
+            "Faithfulness Reason": faithfulness_reason,
+            "Accuracy Score": accuracy_score,
+            "Accuracy Reason": accuracy_reason,
+        })
+        print()
+
+    # Save report
+    if not results:
+        print("No results to save.")
+        return
+
+    df = pd.DataFrame(results)
+    df.to_csv("evaluation_report.csv", index=False)
+
+    # Print summary
+    print("=" * 50)
+    numeric_faith = [r["Faithfulness Score"] for r in results if isinstance(r["Faithfulness Score"], int)]
+    numeric_acc   = [r["Accuracy Score"] for r in results if isinstance(r["Accuracy Score"], int)]
+    if numeric_faith:
+        print(f"Avg Faithfulness (hallucination): {sum(numeric_faith)/len(numeric_faith):.1f}/10")
+    if numeric_acc:
+        print(f"Avg Accuracy:                     {sum(numeric_acc)/len(numeric_acc):.1f}/10")
+    print(f"\nReport saved to evaluation_report.csv")
 
-    # 5. Save Report
-    if results:
-        df = pd.DataFrame(results)
-        df.to_csv("evaluation_report.csv", index=False)
-        print("\nEvaluation Complete! Report saved to 'evaluation_report.csv'")
-        print(df[["Question", "Grading"]])
 
 if __name__ == "__main__":
-    run_evaluation()
+    run_evaluation()