first commit

.gitignore

@@ -0,0 +1,213 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[codz]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py.cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+.pybuilder/
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+#   For a library or package, you might want to ignore these files since the code is
+#   intended to run in multiple environments; otherwise, check them in:
+# .python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# UV
+#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#uv.lock
+
+# poetry
+#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
+#   This is especially recommended for binary packages to ensure reproducibility, and is more
+#   commonly ignored for libraries.
+#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
+#poetry.lock
+#poetry.toml
+
+# pdm
+#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
+#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.
+#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control
+#pdm.lock
+#pdm.toml
+.pdm-python
+.pdm-build/
+
+# pixi
+#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.
+#pixi.lock
+#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one
+#   in the .venv directory. It is recommended not to include this directory in version control.
+.pixi
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.envrc
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype static type analyzer
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+
+# PyCharm
+#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can
+#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
+#  and can be added to the global gitignore or merged into this file.  For a more nuclear
+#  option (not recommended) you can uncomment the following to ignore the entire idea folder.
+#.idea/
+
+# Abstra
+# Abstra is an AI-powered process automation framework.
+# Ignore directories containing user credentials, local state, and settings.
+# Learn more at https://abstra.io/docs
+.abstra/
+
+# Visual Studio Code
+#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore 
+#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+#  and can be added to the global gitignore or merged into this file. However, if you prefer, 
+#  you could uncomment the following to ignore the entire vscode folder
+# .vscode/
+
+# Ruff stuff:
+.ruff_cache/
+
+# PyPI configuration file
+.pypirc
+
+# Cursor
+#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to
+#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data
+#  refer to https://docs.cursor.com/context/ignore-files
+.cursorignore
+.cursorindexingignore
+
+# Marimo
+marimo/_static/
+marimo/_lsp/
+__marimo__/
+
+# Project-specific ignores
+#/context/
+
+# Igrnore gradio
+.gradio/

README copy.md

@@ -0,0 +1,227 @@
+---
+title: RAG-LangChain-Gradio
+app_file: rag_execute.py
+sdk: gradio
+sdk_version: 5.46.0
+---
+# Retrieval-Augmented Generation (RAG)
+
+Dieses Projekt implementiert ein **Retrieval-Augmented Generation (RAG)**-System unter Verwendung von LangChain, Mistral/OpenAI LLMs und Pinecone für die Vektordatenbanksuche. Das System ermöglicht es, Dokumente abzufragen und kontextbewusste Antworten über eine Chat-ähnliche Schnittstelle zu generieren.
+
+---
+
+## Inhaltsverzeichnis
+
+1. [Übersicht](#übersicht)
+2. [Anforderungen](#anforderungen)
+3. [Einrichtung](#einrichtung)
+4. [Vorbereitung der API-Schlüssel](#vorbereitung-der-api-schlüssel)
+5. [Erstellung des Pinecone-Indexes](#erstellung-des-pinecone-indexes)
+6. [Ausführung des RAG-Systems](#ausführung-des-rag-systems)
+7. [Code-Struktur](#code-struktur)
+8. [Prozessbeschreibung](#prozessbeschreibung)
+9. [Evaluation](#evaluation)
+
+---
+
+## Übersicht
+
+Dieses RAG-System arbeitet in drei Schritten:
+
+1. **Dokumente einbetten**: Rohtextdateien in Chunks aufteilen und mit dem `llama-text-embed-v2` Embedding-Modell in Pinecone einbetten.
+2. **Abruf (Retrieval)**: Semantische Suche in der Pinecone-Vektordatenbank durchführen, um die relevantesten Chunks für eine Anfrage abzurufen.
+3. **Generierung**: Übergabe der Anfrage und des abgerufenen Kontexts an ein LLM (Mistral oder OpenAI), um die Antwort zu generieren.
+
+---
+
+## Anforderungen
+
+Python 3.10+ empfohlen.
+
+Enthaltene Abhängigkeiten:
+
+* `langchain-community`
+* `langchain-core`
+* `langchain-text-splitters`
+* `langchain-mistralai`
+* `langchain-openai`
+* `ragas`
+* `datasets`
+* `pinecone-client`
+* `gradio`
+* `python-dotenv`
+* `pypdf`
+* `pandas`
+* `nbformat`
+* `nbconvert`
+* `unstructured` (mit Extras für `docx`, `pptx`, `html`, `md`)
+
+Alle Abhängigkeiten sollten installiert werden, z. B. mit dem folgenden Befehl.
+```
+pip install langchain-mistralai langchain-community datasets ragas langchain-openai langchain-text-splitters langchain-core pinecone-client langgraph pypdf gradio python-dotenv nbformat nbconvert "unstructured[docx,pptx,html,md]"
+```
+Da eine requirements.txt-Datei vorhanden ist, kann die Installation auch durch folgenden Befehl durchgeführt werden:
+```
+pip install -r requirements.txt
+```
+---
+
+## Einrichtung
+
+1. Repository klonen.
+2. Erstellen Sie eine `.env`-Datei im Hauptverzeichnis mit den folgenden Schlüsseln:
+
+```dotenv
+MISTRAL_API_KEY=<Ihr-mistral-api-schlüssel>
+OPENAI_API_KEY=<Ihr-openai-api-schlüssel>
+PINECONE_API=<Ihr-pinecone-api-schlüssel>
+INDEX_NAME=<Ihr-pinecone-index-name>
+DIRNAME=<Pfad-zu-Kontextdokumenten>
+MODELNAME=<LLM-Modellname>                
+````
+
+* `MISTRAL_API_KEY` – Ihr API-Schlüssel für Mistral-Modelle.
+* `OPENAI_API_KEY` – Ihr API-Schlüssel für OpenAI-Modelle. Es reicht aus, einen der OpenAI-/Mistral-Schlüssel (je nach ausgewähltem Modell) festzulegen.
+* `PINECONE_API` – API-Schlüssel für Pinecone, um Vektoren zu speichern und abzufragen.
+* `INDEX_NAME` – Name des Pinecone-Indexes, in dem die Dokumente abgelegt werden.
+* `DIRNAME` – Unterordner innerhalb des festen `context/`-Verzeichnisses.
+
+  * Wenn leer (`DIRNAME=`), werden alle Dokumente in `context/` verarbeitet.
+  * Beispiel: `DIRNAME=llm_context` verarbeitet nur `context/llm_context/`.
+* `MODELNAME` – LLM-Modell, z. B. `gpt-5-nano` für OpenAI oder `mistral-large-latest` für Mistral.
+---
+
+## Vorbereitung der API-Schlüssel
+
+### Mistral
+
+1. Registrieren bei [Mistral AI](https://www.mistral.ai).
+2. API-Schlüssel erstellen.
+3. In `.env` unter `MISTRAL_API_KEY` eintragen.
+
+### OpenAI
+
+1. Registrieren bei [OpenAI](https://platform.openai.com/).
+2. API-Schlüssel erstellen.
+3. In `.env` unter `OPENAI_API_KEY` eintragen.
+
+### Pinecone
+
+1. Registrieren bei [Pinecone](https://www.pinecone.io/).
+2. API-Schlüssel erstellen.
+3. In `.env` unter `PINECONE_API` eintragen.
+4. Einen **Index** erstellen (z. B. `use-cases-index`) mit dem Embedding-Modell `llama-text-embed-v2`.
+
+---
+
+## Erstellung des Pinecone-Indexes
+
+Das RAG-Vorbereitungsskript (`rag_func.py`) führt automatisch folgende Schritte aus:
+
+1. Verbindung zu Pinecone über den API-Schlüssel herstellen.
+2. Dokumente aus dem Verzeichnis `DIRNAME` laden.
+3. Dokumente in Chunks aufteilen mit `RecursiveCharacterTextSplitter` (Standard: 1800 Tokens pro Chunk, 200 Tokens Überlappung).
+4. Chunks mit `llama-text-embed-v2` einbetten.
+5. Einbettungen in Batches in den Pinecone-Index hochladen.
+
+> **Tipp:** Wenn alle Dokumente bereits hochgeladen wurden, kann `DIRNAME` leer bleiben, und das System überspringt die Dokumentenverarbeitung.
+
+---
+
+## Ausführung des RAG-Systems
+
+Hauptskript ausführen:
+
+```bash
+python rag_execute.py
+```
+
+Dies bewirkt:
+
+1. Laden der Dokumente (falls vorhanden) und Vorbereitung des RAG-Systems.
+2. Starten einer **Chat-Schnittstelle** mit Gradio, in der Sie Fragen stellen können.
+3. Abrufen relevanter Chunks aus Pinecone.
+4. Generieren von Antworten mit dem ausgewählten LLM.
+
+### Chat-Schnittstelle
+
+* Geben Sie eine Anfrage in das Textfeld ein.
+* Das System ruft den Kontext ab und erstellt eine Antwort.
+* Antworten werden im Chat-Format angezeigt.
+
+---
+
+## Code-Struktur
+
+```
+├─ rag_execute.py      # Hauptskript für RAG mit Gradio-Schnittstelle
+├─ rag_func.py         # Funktionen für RAG-Vorbereitung, Abruf und Generierung
+├─ .env                # Umgebungsvariablen (API-Schlüssel, Index, Modell, Verzeichnis)
+├─ context/          # Ordner mit Rohdokumenten (kann leer sein, falls bereits hochgeladen)
+```
+
+---
+
+## Prozessbeschreibung
+
+1. **LLM auswählen**: Mistral oder OpenAI.
+2. **Dokumente vorbereiten**:
+
+   * Text- oder JSON-Dokumente in den Ordner `DIRNAME` legen.
+   * Wenn alle Dokumente bereits hochgeladen sind, kann der Ordner leer bleiben.
+3. **Pinecone-Index erstellen**:
+
+   * Index in `.env` benennen (`INDEX_NAME`).
+   * Embedding-Modell `llama-text-embed-v2` verwenden.
+4. **Dokumente aufteilen**: Mit `RecursiveCharacterTextSplitter`.
+5. **Chunks einbetten**: Zur semantischen Suche an Pinecone senden.
+6. **Relevante Chunks abrufen**: Bei einer Anfrage.
+7. **Antwort generieren**: Anfrage + Kontext an LLM übergeben.
+8. **Ergebnis zurückgeben**: Im Chat-Interface anzeigen.
+
+---
+
+## Evaluation
+
+Das System unterstützt Evaluation mit **RAGAS**:
+
+1. `generate_dataset()` ruft Kontext ab und generiert Antworten.
+2. `evaluate_RAG()` berechnet **Faithfulness** und andere Metriken.
+3. Ergebnisse werden zur Analyse ausgegeben.
+
+---
+
+## Hinweise
+
+* Retry-Logik für Pinecone-Operationen ist implementiert, um Netzwerkfehler abzufangen.
+* Chunk-Größe und Überlappung können in `prepare_RAG()` für größere oder kleinere Kontextgranularität angepasst werden.
+
+---
+
+## Beispiel
+
+```python
+from rag_func import prepare_RAG, retrieve_RAG, generate_RAG
+import os
+
+index, pc, llm = prepare_RAG(
+    pinecone_API=os.getenv("PINECONE_API"),
+    index_name=os.getenv("INDEX_NAME"),
+    llm_model=os.getenv("MODELNAME"),
+    dir_name=os.getenv("DIRNAME")
+)
+
+query = "Liste typische Anwendungsfälle von GenAI im Telekommunikationsbereich auf."
+retrieved_chunks = retrieve_RAG(query, pc, index)
+response = generate_RAG(query, llm, retrieved_chunks)
+print(response.content)
+```
+
+---
+
+## Referenzen
+
+* [LangChain RAG Tutorial](https://python.langchain.com/docs/tutorials/rag/)
+* [Pinecone Dokumentation](https://docs.pinecone.io)
+* [RAGAS Evaluation](https://docs.ragas.io/en/stable/getstarted/evals/)
+

app.css

@@ -0,0 +1,485 @@
+:root {
+  --brand-blue: #17428f;
+  --brand-orange: #f39719;
+  --text-dark: #111827;   /* very dark grey (near black) */
+  --text-gray: #4B5563;   /* medium grey for messages */
+  color-scheme: only light;
+}
+
+/* Base */
+body, .gradio-container {
+  /* Default Gradio font will be used */
+  background: linear-gradient(135deg, var(--brand-blue) 0%, var(--brand-orange) 100%);
+  min-height: 100vh;
+  color: var(--text-dark);
+}
+
+/* Logo size */
+
+/* Logo size */
+/*#company-logo img {
+  width: 40px !important;
+  min-width: 40px !important;
+  height: auto !important;
+  object-fit: contain !important;
+  display: block !important;
+}*/
+
+
+/* Top bar transparent */
+#topbar { background: transparent !important; }
+
+/* Header text over gradient */
+#header h1, #header h2, #header h3, #header h4, #header h5, #header h6,
+#header p {
+  color: #ffffff;
+  text-align: center;
+}
+
+#header2 h1, #header2 h2, #header2 h3, #header2 h4, #header2 h5, #header2 h6,
+#header2 p {
+  color: #ffffff;
+  text-align: center;
+}
+
+/* Chatbox container */
+#chatbot {
+  height: 100%;
+  border-radius: 14px;
+  border: 2px solid var(--brand-blue);
+  background-color: #ffffff;
+  padding: 8px;
+  overflow-y: auto;
+
+  /* Icon tint tokens for local use (chat area) */
+  --icon-light: #9CA3AF;
+  --icon-hover: #6B7280;
+}
+
+/* ----------------------------- */
+/* TEXT COLORING (SAFE FOR CODE) */
+/* ----------------------------- */
+
+/* Apply the gray text color only at the message container level.
+   Do NOT set color on descendants or code tokens. */
+#chatbot .message {
+  color: var(--text-gray); /* no !important */
+}
+
+/* Bubble styling */
+#chatbot .message.user {
+  background: #fff4e1;
+  border-radius: 10px;
+  padding: 6px 12px;
+  text-align: right;
+}
+#chatbot .message.bot {
+  background: #f0f0f0;
+  border-radius: 10px;
+  padding: 6px 12px;
+  text-align: left;
+}
+
+/* Markdown horizontal rules inside chatbot */
+#chatbot hr {
+  margin: 6px 0;                 /* reduce extra space */
+  border: none;                   /* remove default bevel */
+  border-top: 1px solid #d1d5db;  /* subtle gray line */
+}
+
+/* Fallback selectors for other Gradio versions */
+#chatbot [data-testid*="message"] {
+  border-radius: 10px;
+  padding: 6px 12px;
+}
+#chatbot [data-testid="user-message"] {
+  background: #fff4e1;
+  text-align: right;
+}
+#chatbot [data-testid="assistant-message"] {
+  background: #f0f0f0;
+  text-align: left;
+}
+
+/* ----------------------------- */
+/* CODE BLOCKS (DO NOT SET COLOR)*/
+/* ----------------------------- */
+
+/* Give code blocks a readable container without touching colors.
+   This preserves syntax highlighting from highlight.js or Prism. */
+#chatbot pre,
+#chatbot pre code,
+#chatbot code[class*="language-"],
+#chatbot pre[class*="language-"],
+#chatbot code.hljs {
+  background: #f8fafc;          /* light neutral background */
+  border-radius: 8px;
+  padding: 10px 12px;
+  display: block;
+  overflow-x: auto;
+  /* IMPORTANT: no 'color' declaration here */
+}
+
+/* Inline code (single backticks) */
+#chatbot :not(pre) > code {
+  background: #f1f5f9;
+  padding: 0.15rem 0.35rem;
+  border-radius: 6px;
+  /* no 'color' here */
+}
+
+/* ---------------------------------- */
+/* Inputs                              */
+/* ---------------------------------- */
+input[type="text"], textarea, .gr-text-input input, .gr-textbox textarea {
+  border-radius: 10px;
+  padding: 10px;
+  font-size: 16px;
+  border: 2px solid var(--brand-orange);
+}
+input:focus, textarea:focus, .gr-text-input input:focus, .gr-textbox textarea:focus {
+  border-color: var(--brand-blue);
+  outline: none;
+  box-shadow: 0 0 6px rgba(23, 66, 143, 0.5);
+}
+
+/* ---------------------------------- */
+/* Buttons (global gradient)           */
+/* ---------------------------------- */
+.gr-button, button {
+  border-radius: 10px;
+  font-weight: 600;
+  background: linear-gradient(90deg, var(--brand-blue), var(--brand-orange));
+  color: white;
+  border: none;
+}
+.gr-button:hover, button:hover {
+  transform: translateY(-2px);
+  box-shadow: 0 4px 8px rgba(0,0,0,0.2);
+}
+
+/* ---------------------------------- */
+/* Chat area: icon-only buttons        */
+/* ---------------------------------- */
+
+/* Tint SVG icons */
+#chatbot button svg,
+#chatbot [role="button"] svg,
+#chatbot .icon svg,
+#chatbot [class*="icon"] svg,
+#chatbot [data-testid*="icon"] svg,
+#chatbot [data-testid*="message"] .tools svg,
+#chatbot .message-tools svg {
+  color: var(--icon-light) !important;
+  fill: var(--icon-light) !important;
+  stroke: var(--icon-light) !important;
+  opacity: 0.95;
+}
+
+/* Remove gradient background only on small icon-only buttons */
+#chatbot :is(button,[role="button"]):is([aria-label],[title], :has(> svg)):not(.keep-gradient) {
+  background: transparent !important;
+  background-image: none !important;
+  border: none !important;
+  box-shadow: none !important;
+  padding: 6px !important;
+  border-radius: 8px !important;
+  color: var(--icon-light) !important;
+}
+
+/* Hover/focus/active states */
+#chatbot :is(button,[role="button"]):is([aria-label],[title], :has(> svg)):not(.keep-gradient):hover {
+  background-color: rgba(0,0,0,0.05) !important;
+}
+#chatbot :is(button,[role="button"]):is([aria-label],[title], :has(> svg)):not(.keep-gradient):focus-visible {
+  outline: none !important;
+  box-shadow: 0 0 0 2px rgba(23, 66, 143, 0.35) !important;
+  background-color: rgba(0,0,0,0.06) !important;
+}
+#chatbot :is(button,[role="button"]):is([aria-label],[title], :has(> svg)):not(.keep-gradient):active {
+  background-color: rgba(0,0,0,0.08) !important;
+}
+
+/* Optional 'danger' icons */
+#chatbot .danger svg {
+  color: var(--icon-light) !important;
+  fill: var(--icon-light) !important;
+  stroke: var(--icon-light) !important;
+}
+#chatbot .danger:hover svg {
+  color: #ef4444 !important;
+  fill: #ef4444 !important;
+  stroke: #ef4444 !important;
+}
+
+/* ---------------------------------- */
+/* TOP BAR (logo block): icon-only     */
+/* ---------------------------------- */
+#topbar { background: transparent !important; }
+#topbar { --icon-light: #9CA3AF; --icon-hover: #6B7280; }
+
+#topbar .gr-button.keep-gradient,
+#topbar .gr-button:not(:has(svg)) {
+  background: linear-gradient(90deg, var(--brand-blue), var(--brand-orange)) !important;
+  color: #fff !important;
+}
+
+/* Icon-only buttons in topbar: transparent */
+#topbar :is(button,[role="button"]):is([aria-label],[title], :has(> svg)):not(.keep-gradient) {
+  background: transparent !important;
+  border: none !important;
+  box-shadow: none !important;
+  padding: 6px !important;
+  border-radius: 8px !important;
+  color: var(--icon-light) !important;
+}
+
+/* Tint SVGs in topbar */
+#topbar :is(button,[role="button"]):has(> svg) > svg {
+  color: var(--icon-light) !important;
+  fill: var(--icon-light) !important;
+  stroke: var(--icon-light) !important;
+  opacity: 0.95;
+}
+
+/* Hover/focus/active for topbar icons */
+#topbar :is(button,[role="button"]):is([aria-label],[title], :has(> svg)):not(.keep-gradient):hover {
+  background-color: rgba(0,0,0,0.05) !important;
+}
+#topbar :is(button,[role="button"]):is([aria-label],[title], :has(> svg)):not(.keep-gradient):focus-visible {
+  outline: none !important;
+  box-shadow: 0 0 0 2px rgba(23, 66, 143, 0.35) !important;
+  background-color: rgba(0,0,0,0.06) !important;
+}
+#topbar :is(button,[role="button"]):is([aria-label],[title], :has(> svg)):not(.keep-gradient):active {
+  background-color: rgba(0,0,0,0.08) !important;
+}
+
+
+
+
+
+/* Ensure the textbox wrapper is relative */
+#message-box {
+  width: 100%;
+  border-radius: 9999px;
+  border: 2px solid var(--brand-orange);
+  font-size: 16px;
+  outline: none;
+  position: relative; /* needed for absolute button */
+}
+
+#message-box:focus {
+  border-color: var(--brand-blue);
+  box-shadow: 0 0 6px rgba(23, 66, 143, 0.3);
+}
+
+/* Send button positioned inside textbox */
+#send-button {
+  position: absolute;
+  
+  right: 34px;              /* move left by increasing this value */
+  top: 48px;                 /* move down by increasing this value */
+
+  transform: translateY(-16%);
+  width: 36px;
+  height: 36px;
+  min-width: 0 !important;
+  padding: 0 !important;
+  border-radius: 50%;
+  background: linear-gradient(90deg, var(--brand-blue), var(--brand-orange));
+  display: flex;
+  align-items: center;
+  justify-content: center;
+  border: none;
+  color: white;
+  font-size: 18px;
+  cursor: pointer;
+  z-index: 2;
+}
+
+#send-button:hover {
+  transform: translateY(-16%) scale(1.05);
+  box-shadow: 0 4px 12px rgba(0,0,0,0.2);
+}
+
+#send-button:active {
+  transform: translateY(-16%) scale(0.95);
+}
+
+
+
+
+
+
+
+
+/* Compact upload area - lighter and transparent */
+#file-upload .upload-box,
+#file-upload .file-wrap,
+#file-upload .wrap {
+  min-height: 80px;
+  padding: 8px 12px;
+  color: #666 !important;          /* Softer text color */
+  background: transparent !important; /* No white block, shows gradient */
+  border: none !important;         /* Remove any border */
+}
+
+/* Inner placeholder (remove gray square) */
+#file-upload .upload-box div,
+#file-upload .upload-box span {
+  background: transparent !important; /* Remove gray background */
+}
+
+/* Text and icons slightly lighter */
+#file-upload .upload-box,
+#file-upload .file-wrap,
+#file-upload .wrap {
+  color: #666 !important;
+  fill: #666 !important;
+  stroke: #666 !important;
+}
+
+/* Buttons remain clean */
+#file-upload button,
+#file-upload [role="button"] {
+  background: transparent !important;
+  border: none !important;
+  box-shadow: none !important;
+  border-radius: 0 !important;
+  padding: 4px !important;
+  color: #666 !important;
+}
+
+/* SVG icons */
+#file-upload button svg,
+#file-upload [role="button"] svg {
+  color: #666 !important;
+  fill: #666 !important;
+  stroke: #666 !important;
+  background: none !important;
+}
+
+/* Hover effect */
+#file-upload button:hover svg,
+#file-upload [role="button"]:hover svg {
+  color: #2a5db0 !important;
+  fill: #2a5db0 !important;
+  stroke: #2a5db0 !important;
+}
+
+
+
+
+/* Hide Gradio's default control buttons in the header/topbar */
+#topbar .gr-button,
+#topbar [role="button"],
+#header-container .gr-button,
+#header-container [role="button"] {
+    display: none !important;
+}
+
+
+
+#upload-note, #upload-note * {
+  color: #ffffff !important;
+}
+
+
+
+
+.gradio-container [id="left-column"] {
+    min-width: 40px !important;      /* Increased from 10px */
+    max-width: 320px !important;     /* You can go up to 400px if you want even wider */
+    width: 180px !important;         /* Increased from 100px */
+    flex: 0 0 220px !important;      /* Increased from 180px */
+    padding-right: 12px !important;  /* Slightly more padding for visual separation */
+}
+
+.gradio-container [id="right-column"] {
+    flex: 1 1 0 !important;
+    width: auto !important;
+    min-width: 0 !important;
+    padding-left: 0 !important;
+}
+
+
+
+
+
+/* Branding layout */
+#branding {
+  display: flex;
+  align-items: center;
+  gap: 8px;               /* space between text and logo */
+  justify-content: center;
+}
+
+/* Text style */
+#brand-text {
+  font-size: 1.6rem;
+  font-weight: 700;
+  color: white;
+  line-height: 1;
+}
+
+/* Logo sizing */
+#company-logo {
+  width: 40px;            /* set exact width */
+  height: auto;           /* preserve proportions */
+  display: block;
+  object-fit: contain;
+}
+
+
+
+
+
+
+
+/* ---- Viewport-safe base (handles mobile address bar) ---- */
+html, body, .gradio-container {
+  height: 100dvh;             /* dynamic viewport height */
+  min-height: 100dvh;
+}
+
+/* ---- Make sure all flex ancestors allow their children to shrink ---- */
+.gradio-container .gr-row,
+.gradio-container .gr-row > .gr-column,
+.gradio-container > .gr-row,
+.gradio-container .gr-column,
+#right-column,
+#chat-area {
+  min-height: 0 !important;   /* critical so inner scroll can happen */
+}
+
+/* ---- Chatbot outer box: cap height and scroll inside ---- */
+#chatbot {
+  box-sizing: border-box;
+  flex: 1 1 auto;
+  min-height: 0 !important;
+  height: auto !important;
+  max-height: calc(100dvh - var(--topbar-offset, 0px) - 120px) !important;
+  overflow-y: auto !important;
+}
+
+/* ---- Gradio’s inner wrappers sometimes need explicit constraints ---- */
+/* Gradio v4: Chatbot renders inside .gr-chatbot -> .wrap / .message-wrap depending on version */
+#chatbot .wrap,
+#chatbot .message-wrap,
+#chatbot [data-testid="chatbot"] {
+  max-height: 100%;
+  overflow-y: auto;
+  min-height: 0;
+}
+
+/* Optional: if you see the input row pushing the chat up/down on small screens,
+   let the input take only its content height. */
+#input-row {
+  flex: 0 0 auto;
+}
+
+/* If your header/progress box is above the chat, ensure it doesn't consume flex growth */
+#chat-area > *:not(#chatbot) {
+  flex: 0 0 auto;
+}

generate.py

@@ -0,0 +1,271 @@
+import os
+import pathlib
+import time
+import re
+from pinecone import Pinecone
+
+from langchain_mistralai import ChatMistralAI
+from langchain_openai import ChatOpenAI
+from langchain_core.messages import HumanMessage, SystemMessage
+from langchain.schema import Document
+from langchain_community.document_loaders import (
+    CSVLoader, PyPDFLoader, UnstructuredWordDocumentLoader,
+    UnstructuredPowerPointLoader, UnstructuredMarkdownLoader,
+    UnstructuredHTMLLoader, NotebookLoader
+)
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+
+from llama_index.core.memory import Memory
+
+import pickle
+
+import json
+from typing import List, Any
+from langchain_core.messages import SystemMessage, HumanMessage, AIMessage, BaseMessage
+
+from typing import List, Any
+from pydantic import BaseModel, ValidationError
+
+
+memory = Memory(token_limit=2048)
+
+
+def generate_RAG(
+    prompt_message,
+    llm,
+    retrieved_chunks,
+    graph_context="",
+    graphRAG=False,
+    info=True
+):
+    """
+    Two-stage flow (single function):
+      1) Resolver (non-streaming, no callbacks): decide if this turn should be history-only. Produce resolved_task.
+      2) Answer (streaming via the passed llm): include retrieved context only if allowed; otherwise forbid it.
+
+    Message order (to favor history for follow-ups):
+      System (first) -> (Optional) AIMessage with Retrieved Context -> History -> Human (last)
+    """
+
+    if info:
+        print("Generate RAG with", prompt_message, llm)
+
+    # ---------- Helpers ----------
+    def _to_list_messages(history: Any) -> List[BaseMessage]:
+        """Normalizes memory history: supports list[BaseMessage] or a summary string."""
+        if isinstance(history, list):
+            return history
+        if isinstance(history, str) and history.strip():
+            return [AIMessage(content=f"[Conversation summary]\n{history.strip()}")]
+        return []
+
+    def _last_ai_text(msgs: List[BaseMessage]) -> str:
+        for m in reversed(msgs):
+            if isinstance(m, AIMessage):
+                return m.content
+        return ""
+
+    def _safe_json_loads(raw: str) -> dict:
+        try:
+            return json.loads(raw)
+        except Exception:
+            start, end = raw.find("{"), raw.rfind("}")
+            if start != -1 and end != -1 and end > start:
+                return json.loads(raw[start:end+1])
+            raise
+    def _make_non_streaming_resolver(llm_):
+        """
+        Create a non-streaming, callback-free copy of the same LLM class for the resolver step.
+        Works for ChatOpenAI-style classes that accept 'model' or 'model_name'.
+        """
+        model_name = getattr(llm_, "model_name", getattr(llm_, "model", None))
+        kwargs = {}
+        if hasattr(llm_, "temperature"):
+            kwargs["temperature"] = getattr(llm_, "temperature")
+        try:
+            return llm_.__class__(model=model_name, streaming=False, callbacks=[], **kwargs)
+        except TypeError:
+            return llm_.__class__(model_name=model_name, streaming=False, callbacks=[], **kwargs)
+
+    def _resolver(user_text: str, history_msgs: List[BaseMessage]) -> dict:
+        resolver_llm = _make_non_streaming_resolver(llm)
+
+        RESOLVER_SYS = (
+            "You are a controller that decides if the next answer should rely ONLY on Chat History "
+            "(ignore Retrieved Context completely) or may use Retrieved Context.\n"
+            "Return STRICT JSON with keys:\n"
+            '{ "use_history_only": true|false, "resolved_task": "<resolved user request>" }\n\n'
+            "Rules:\n"
+            "- Always set set use_history_only=false (especially if the query has meaningful concepts for retrieval, e.g., specific entities, topics, product names, technical terms, factual questions).\n"
+            "- Except in rare cases, do NOT set use_history_only=true. Only do so if the query contains undefined pronouns (e.g., this, that, it, they, those, these, above, continue, previous, earlier, same...).\n"
+            "Examples:\n"
+            'User: "Where in the onboarding guide do we define the trial limits?"\n'
+            '-> { "use_history_only": false, "resolved_task": "Find where the onboarding guide defines the trial limits and report the exact limits." }\n'
+        )
+
+        resolver_msgs: List[BaseMessage] = [SystemMessage(RESOLVER_SYS)]
+        last_ai = _last_ai_text(history_msgs)
+        if last_ai:
+            resolver_msgs.append(AIMessage(content=f"[Last assistant answer]\n{last_ai}"))
+        resolver_msgs.extend(history_msgs)
+        resolver_msgs.append(HumanMessage(content=f"User message: {user_text}"))
+
+        raw = resolver_llm.invoke(resolver_msgs).content
+        try:
+            data = _safe_json_loads(raw)
+        except Exception:
+            data = {"use_history_only": False, "resolved_task": user_text}
+
+        data.setdefault("use_history_only", False)
+        data.setdefault("resolved_task", user_text)
+        return data
+
+
+    # ---------- Prepare history ----------
+    history_messages: List[BaseMessage] = []
+    if memory:
+        # Get the last messages from LlamaIndex memory
+        last_msgs = memory.get_all()[-8:]
+
+        # Convert LlamaIndex messages to LangChain message types
+        for msg in last_msgs:
+            if msg.role == "user":
+                history_messages.append(HumanMessage(content=msg.content))
+            elif msg.role in ("ai", "assistant"):
+                history_messages.append(AIMessage(content=msg.content))
+            # Add more roles if needed
+
+    # ---------- Stage 1: Resolve (non-streaming) ----------
+    plan = _resolver(prompt_message, history_messages)
+
+    use_history_only = bool(plan.get("use_history_only", False))
+    resolved_task = plan.get("resolved_task", prompt_message)
+
+    if info:
+        print("[Resolver]", plan)
+
+
+    # ---------- Build retrieval context block ----------
+    context_lines = []
+    if not use_history_only:
+        for i, chunk in enumerate(retrieved_chunks or []):
+            source_filename = os.path.basename((chunk.get("source") or "unknown"))
+            text = chunk.get("text") or ""
+            context_lines.append(f"Source {i+1} ({source_filename}):\n{text}")
+        
+        if graphRAG and graph_context:
+            context_lines.append("[Graph context]\n" + graph_context)
+
+    context_for_llm = "\n\n".join(context_lines)
+
+    # ---------- System prompt (first) ----------
+    base_rules = (
+        "You are an expert assistant. Answer in English. Use:\n"
+        "- Chat History\n"
+        "- Retrieved Context (reference-only facts; not user intent).\n\n"
+        "Decision rubric before answering:\n"
+        "- Important: you MUST ALWAYS cite a source, i.e., always use exactly the filename from the 'source' metadata (e.g., 'Source: sample.pdf.' in the same paragraph as the claim).\n"
+        "- If the answer is not supported by Retrieved Context and not implied by history, say you cannot answer.\n\n"
+        "Important: output should be very well-structured Markdown (always different headings, hierarchical structure, bullets, tables and code blocks when needed), with a few emojis for scannability."
+    )
+    turn_rule = (
+        "\n\nTURN-SPECIFIC RULE: For THIS turn, you MUST NOT use any Retrieved Context. "
+        "Base your answer ONLY on Chat History and the user's current request."
+        if use_history_only else ""
+    )
+
+    prompt_parts: List[BaseMessage] = [SystemMessage(content=base_rules + turn_rule)]
+
+    # ---------- Retrieved context as assistant message (only if allowed) ----------
+    if (not use_history_only) and context_for_llm.strip():
+        prompt_parts.append(
+            SystemMessage(
+                content="📚 Retrieved Context (reference-only; not user intent, Use info only from here and nothing else, if info not present, say you do not know. You are only allowed to base your answer on this info and not use your own):\n\n" + context_for_llm
+            )
+        )
+
+    # ---------- History next (more recent than retrieval context) ----------
+    if history_messages:
+        prompt_parts.append(SystemMessage(content="🕘 Chat History (most recent last):"))
+        prompt_parts.extend(history_messages)
+
+    # ---------- Current user last (include BOTH original and resolved) ----------
+    final_human = (
+        "User request (original):\n"
+        f"{prompt_message}\n\n"
+        "Resolved task (use this when pronouns/references appear):\n"
+        f"{resolved_task}"
+    )
+    prompt_parts.append(HumanMessage(content=final_human))
+
+    # ---------- Stage 2: Answer (streaming via passed llm) ----------
+    print(f"[Info] The final prompt is the following: {prompt_parts}")
+    response = llm.invoke(prompt_parts)
+    print(f"[Info] The final response is the following: {response}")
+
+
+
+
+
+
+    # ---------- Pydantic validation: ensure some "Source:" structure is present ----------
+    class _AnswerWithCitationStructure(BaseModel):
+        content: str
+
+        @classmethod
+        def ensure_source_structure(cls, content: str):
+            """
+            Check that there is at least one 'Source:' or 'Sources:' pattern in the text.
+            """
+            import re
+
+            if not re.search(r"\bSources?:\s*.+", content, flags=re.IGNORECASE):
+                raise ValueError("Missing any 'Source:' structure in the answer.")
+
+
+    # Run validation only when we expected citations (retrieval was allowed)
+    try:
+        if not use_history_only:
+            _AnswerWithCitationStructure.ensure_source_structure(
+                getattr(response, "content", str(response))
+            )
+    except (ValidationError, ValueError) as ve:
+        print(f"[Validation] Source structure check failed: {ve}")
+
+        # Retry answer generation with stronger emphasis on sources
+        retry_prompt_parts = prompt_parts.copy()
+        retry_prompt_parts.append(SystemMessage(
+            content="⚠️ IMPORTANT: Your previous answer did not include any 'Source:' citation. "
+                    "Regenerate your answer and make sure to include at least one 'Source: ...' or 'Sources: ...' line "
+                    "that cites the relevant documents or context."
+        ))
+        response = llm.invoke(retry_prompt_parts)
+        print("[Retry] Regenerated answer with source emphasis.")
+
+
+
+
+
+
+
+    # ---------- Persist to memory ----------
+    
+    from llama_index.core.llms import ChatMessage
+
+    # ---------- Persist to memory ----------
+    if memory:
+        # Add user message
+        memory.put(ChatMessage(role="user", content=prompt_message))
+
+        if not use_history_only:
+            # Add context as AI message
+            memory.put(ChatMessage(role="assistant", content=f"The context was: [start context] {context_for_llm} [end context]"))
+
+        # Add final AI response
+        memory.put(ChatMessage(role="assistant", content=getattr(response, "content", str(response))))
+
+        # To print the current memory, retrieve all messages
+        print("[Info] The following is the current memory:", memory.get_all())
+
+
+    return response

prepare.py

@@ -0,0 +1,896 @@
+"""prepare.py
+
+Utilities to prepare documents and knowledge-graph artifacts for a RAG (Retrieval-Augmented
+Generation) pipeline.
+
+This module implements:
+- safe file loading for text-like files (UTF-8 tolerant)
+- dataset creation from a `context` directory using various loaders
+- chunking, embedding and upserting to Pinecone via `prepare_RAG`
+- building/updating a Knowledge Graph and generating hierarchical community summaries
+
+Main public functions:
+- create_dataset(directory_path: str) -> List[Document]
+- prepare_RAG(pinecone_API, index_name, ...) -> (index, pc, llm, documents)
+- build_knowledge_graph(documents, llm, pc, index, info=True) -> KnowledgeGraphIndex
+
+Note: many helper functions are nested; this docstring highlights the high-level
+purpose and responsibilities only.
+"""
+
+import os
+import pathlib
+import time
+import re
+from pinecone import Pinecone
+
+from langchain_mistralai import ChatMistralAI
+from langchain_openai import ChatOpenAI
+from langchain_core.messages import HumanMessage, SystemMessage
+from langchain.schema import Document
+from langchain_community.document_loaders import (
+    CSVLoader, PyPDFLoader, UnstructuredWordDocumentLoader,
+    UnstructuredPowerPointLoader, UnstructuredMarkdownLoader,
+    UnstructuredHTMLLoader, NotebookLoader
+)
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+
+from llama_index.core.memory import Memory
+
+import pickle
+
+import json
+from typing import List, Any
+from langchain_core.messages import SystemMessage, HumanMessage, AIMessage, BaseMessage
+
+from typing import List, Any
+from pydantic import BaseModel, ValidationError
+
+
+memory = Memory(token_limit=2048)
+
+# -------------------------
+# UTF-8 safe Text Loader
+# -------------------------
+class SafeTextLoader:
+    """Loads a text file as a single Document, safely handling UTF-8 decoding errors."""
+    def __init__(self, file_path):
+        self.file_path = file_path
+
+    def load(self):
+        """Load the file and return a list containing a single LangChain `Document`.
+
+        The loader is UTF-8 tolerant: it reads raw bytes and decodes using UTF-8 with
+        'ignore' on errors to avoid failing on files containing invalid sequences.
+
+        Returns:
+            List[Document]: a list with one Document (page_content and metadata['source'])
+            or an empty list on error.
+        """
+        try:
+            with open(self.file_path, "rb") as f:
+                raw_bytes = f.read()
+            text = raw_bytes.decode("utf-8", errors="ignore")
+            return [Document(page_content=text, metadata={"source": str(self.file_path)})]
+        except Exception as e:
+            print(f"[Error] Failed to read {self.file_path}: {e}")
+            return []
+
+# -------------------------
+# Loader mapping
+# -------------------------
+LOADER_MAPPING = {
+    ".txt": SafeTextLoader,
+    ".json": SafeTextLoader,
+    ".md": UnstructuredMarkdownLoader,
+    ".csv": CSVLoader,
+    ".yaml": SafeTextLoader,
+    ".yml": SafeTextLoader,
+    ".pdf": PyPDFLoader,
+    ".docx": UnstructuredWordDocumentLoader,
+    ".pptx": UnstructuredPowerPointLoader,
+    ".html": UnstructuredHTMLLoader,
+    ".htm": UnstructuredHTMLLoader,
+    ".ipynb": NotebookLoader,
+    ".py": SafeTextLoader,
+    ".js": SafeTextLoader,
+    ".sql": SafeTextLoader,
+}
+
+CONTEXT_ROOT = pathlib.Path(__file__).parent / "context"
+
+def create_dataset(directory_path: str = "context"):
+    """Recursively load files under `directory_path` using extension-specific loaders."""
+
+    target_dir = pathlib.Path(directory_path).resolve()
+    if not target_dir.exists() or not target_dir.is_dir():
+        print(f"[Error] Target directory does not exist: {target_dir}")
+        return []
+
+    documents = []
+    for file_path in target_dir.rglob("*"):
+        if not file_path.is_file():
+            continue
+        ext = file_path.suffix.lower()
+        loader_cls = LOADER_MAPPING.get(ext)
+        if loader_cls is None:
+            print(f"[Skip] Unsupported file type: {file_path}")
+            continue
+        try:
+            loader = loader_cls(str(file_path))
+            docs = loader.load()
+            documents.extend(docs)
+            print(f"[Loaded] {file_path} ({len(docs)} docs)")
+        except Exception as e:
+            print(f"[Error] Failed to load {file_path}: {e}")
+
+    print(f"[Done] Finished scanning {target_dir}")
+    print(f"Total documents loaded: {len(documents)}")
+    return documents
+
+
+from llama_index.core import KnowledgeGraphIndex
+from llama_index.core import Document as LlamaDocument
+
+import hashlib
+
+
+def fetch_existing_ids(index, namespace, ids, batch_size=100):
+    """Fetch IDs from Pinecone in safe batches to avoid URI too large errors"""
+    existing_ids = set()
+    for start in range(0, len(ids), batch_size):
+        batch_ids = ids[start:start + batch_size]
+        result = index.fetch(ids=batch_ids, namespace=namespace)
+        if hasattr(result, "vectors"):
+            existing_ids.update(result.vectors.keys())
+    return existing_ids
+
+
+
+# -------------------------
+# Prepare RAG
+# -------------------------
+import hashlib
+import time
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+from llama_index.core import Document as LlamaDocument
+from pinecone import Pinecone
+
+import os
+import re
+import time
+import hashlib
+
+from langchain_openai import ChatOpenAI
+from langchain_mistralai import ChatMistralAI
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+from pinecone import Pinecone
+
+from llama_index.core import Document as LlamaDocument
+
+# You are assumed to already have:
+# - create_dataset(dir_name)
+# - fetch_existing_ids(index, namespace, all_ids, batch_size)
+
+
+# -------------------------
+# Internal helper: build & upsert community summaries (incremental inside; same signature)
+# -------------------------
+def _build_and_index_community_summaries(
+    kg_index,
+    pc,
+    index,
+    llm,
+    impacted_nodes=None,
+    info=True,
+):
+    """
+    This function implements a hierarchical community detection and summarization pipeline:
+    
+    1. COMMUNITY DETECTION:
+       - Uses NetworkX's greedy_modularity_communities to find natural clusters in the KG
+       - Filters communities by minimum size (COMMUNITY_MIN_SIZE) to avoid noise
+    
+    2. HIERARCHY CONSTRUCTION:
+       - Builds a multi-level tree structure (max depth = MAX_HIERARCHY_DEPTH)
+       - Recursively splits large communities using the same modularity algorithm
+       - Creates parent-child relationships between community levels
+    
+    3. AFFECTED NODE TRACKING:
+       - Marks communities as "_affected" if they contain new/updated nodes
+       - Propagates affected status upward to parent communities
+       - Enables incremental updates by only processing changed regions
+    
+    4. BOTTOM-UP SUMMARIZATION:
+       - Leaf communities: Generate detailed reports from entity relationships
+       - Parent communities: Synthesize child summaries into higher-level overviews
+       - Uses sampling (LIMIT_NODES_PER_SUMMARY) to handle large communities
+    
+    5. VECTOR STORAGE:
+       - Creates stable IDs using SHA-256 hashes of community composition
+       - Embeds summaries using Pinecone's llama-text-embed-v2 model
+       - Stores in dedicated "community-summaries" namespace
+    """
+
+    import hashlib
+    import networkx as nx
+    from networkx.algorithms.community import greedy_modularity_communities
+
+    COMMUNITY_NAMESPACE = "community-summaries"
+    COMMUNITY_MIN_SIZE = 3
+    MAX_HIERARCHY_DEPTH = 2
+    LIMIT_NODES_PER_SUMMARY = 60
+    LIMIT_TRIPLES_PER_SUMMARY = 120
+
+    try:
+        nxg = kg_index.get_networkx_graph()
+    except Exception as e:
+        print(f"[Error] Unable to extract NetworkX graph from KG: {e}")
+        return
+
+    if nxg.number_of_nodes() == 0 or nxg.number_of_edges() == 0:
+        if info:
+            print("[Community] KG empty or trivial; skipping community summarization.")
+        return
+
+    first_run = impacted_nodes is None
+    impacted_nodes = set(impacted_nodes or [])
+
+    if info:
+        print(f"[Community] Starting summarization. First run: {first_run}")
+        print(f"[Community] Impacted nodes: {len(impacted_nodes)}")
+
+    # ---- community detection ----
+    if info:
+        print("[Community] Detecting top-level communities (greedy modularity)...")
+    try:
+        communities = list(greedy_modularity_communities(nxg))
+    except Exception as e:
+        print(f"[Error] Community detection failed: {e}")
+        return
+
+    large_communities = [c for c in communities if len(c) >= max(2, COMMUNITY_MIN_SIZE)]
+    small_communities = [c for c in communities if len(c) < max(2, COMMUNITY_MIN_SIZE)]
+
+    if info:
+        print(f"[Community] Found {len(communities)} communities; "
+              f"{len(large_communities)} large, {len(small_communities)} small.")
+
+    # ---- build hierarchy and mark affected ----
+    hierarchy = []
+    for idx, comm in enumerate(large_communities):
+        subgraph = nxg.subgraph(comm).copy()
+        node_set = set(subgraph.nodes())
+        node = {
+            "id": f"C{idx}",
+            "level": 0,
+            "nodes": node_set,
+            "children": [],
+            "_affected": first_run or bool(impacted_nodes & node_set),
+        }
+
+        # simple frontier-based recursive splitting
+        frontier = [(node, subgraph, 1)]
+        while frontier:
+            parent, g, depth = frontier.pop()
+            if depth > MAX_HIERARCHY_DEPTH or g.number_of_nodes() < max(2, COMMUNITY_MIN_SIZE * 2):
+                continue
+            try:
+                subs = list(greedy_modularity_communities(g))
+            except Exception:
+                subs = []
+
+            subs = [s for s in subs if 1 <= len(s) <= len(g) - 1]
+            subs = [s for s in subs if len(s) >= max(2, COMMUNITY_MIN_SIZE)]
+
+            for j, s in enumerate(subs):
+                sg = g.subgraph(s).copy()
+                child = {
+                    "id": f"{parent['id']}.{j}",
+                    "level": depth,
+                    "nodes": set(s),
+                    "children": [],
+                    "_affected": first_run or bool(impacted_nodes & set(s)),
+                }
+                parent["children"].append(child)
+                if depth + 1 <= MAX_HIERARCHY_DEPTH and sg.number_of_nodes() >= max(2, COMMUNITY_MIN_SIZE * 2):
+                    frontier.append((child, sg, depth + 1))
+
+        hierarchy.append(node)
+
+    # propagate affected upward
+    def mark_ancestors(n):
+        any_child = False
+        for c in n["children"]:
+            if mark_ancestors(c):
+                any_child = True
+        if any_child:
+            n["_affected"] = True
+        return n["_affected"]
+
+    for root in hierarchy:
+        mark_ancestors(root)
+
+    # ---- summarization helpers ----
+    def triples_within(node_ids, graph):
+        res = []
+        for (u, v, data) in graph.edges(data=True):
+            if u in node_ids and v in node_ids:
+                rel = data.get("label") or data.get("relationship") or "related_to"
+                res.append((u, rel, v))
+        return res
+
+    def sample_for_prompt(nodes_set, triples_list, max_nodes=LIMIT_NODES_PER_SUMMARY, max_triples=LIMIT_TRIPLES_PER_SUMMARY):
+        nodes_list = list(nodes_set)[:max_nodes]
+        triples_list = triples_list[:max_triples]
+        return nodes_list, triples_list
+
+    def summarize_leaf(nodes_set, graph):
+        nodes_list, tri_list = sample_for_prompt(
+            nodes_set,
+            triples_within(nodes_set, graph)
+        )
+        prompt = (
+            "You are creating a concise community report from a knowledge graph.\n"
+            "Given the following entity list and intra-community relationships, produce:\n"
+            " - Title\n"
+            " - Key Themes (bullet points)\n"
+            " - Notable Entities\n"
+            " - Important Relationships (summarize patterns rather than listing all)\n"
+            " - Outliers or Cross-links (if any)\n"
+            " - 3-5 Answerable Questions this community can address\n"
+            "Keep it under ~250-300 words.\n\n"
+            f"Entities (sample): {nodes_list}\n"
+            f"Relationships (sample triples): {[f'{u} --[{r}]--> {v}' for (u,r,v) in tri_list]}\n"
+        )
+        resp = llm.invoke(prompt)
+        return resp.content.strip()
+
+    def summarize_parent(child_summaries):
+        join_text = "\n\n".join([f"[Child {i+1}]\n{txt}" for i, txt in enumerate(child_summaries)])
+        prompt = (
+            "You are creating a higher-level summary that unifies several community reports.\n"
+            "Synthesize the following child community reports into a coherent parent-level summary:\n"
+            " - Overarching Title\n"
+            " - Cross-community Key Themes\n"
+            " - How the sub-communities relate and differ\n"
+            " - Cross-cutting entities/relationships\n"
+            " - 3-5 high-level questions the parent community can answer\n"
+            "Target length: 250-350 words.\n\n"
+            f"{join_text}\n"
+        )
+        resp = llm.invoke(prompt)
+        return resp.content.strip()
+
+    # bottom-up, only affected subtrees
+    def build_summaries(node, graph):
+        if not node["_affected"]:
+            return None
+        if not node["children"]:
+            node["summary"] = summarize_leaf(node["nodes"], graph)
+            return node["summary"]
+        child_summaries = []
+        for ch in node["children"]:
+            s = build_summaries(ch, graph)
+            if s is not None:
+                child_summaries.append(s)
+        if child_summaries:
+            node["summary"] = summarize_parent(child_summaries)
+            return node["summary"]
+        node["summary"] = summarize_leaf(node["nodes"], graph)
+        return node["summary"]
+
+    for node in hierarchy:
+        build_summaries(node, nxg)
+
+    # flatten affected nodes w/ new summaries
+    flat_nodes = []
+    def flatten(n):
+        if n.get("_affected") and "summary" in n:
+            flat_nodes.append({
+                "id": n["id"],
+                "level": n["level"],
+                "size": len(n["nodes"]),
+                "nodes": list(n["nodes"]),
+                "summary": n["summary"]
+            })
+        for c in n["children"]:
+            flatten(c)
+    for n in hierarchy:
+        flatten(n)
+
+    if not flat_nodes:
+        if info:
+            print("[Community] No affected community summaries to upsert.")
+        return
+
+    if info:
+        print(f"[Community] Upserting {len(flat_nodes)} community summaries to namespace: {COMMUNITY_NAMESPACE}")
+
+    def summary_vec_id(node_rec):
+        key = f"{node_rec['id']}|{node_rec['level']}|{','.join(sorted(node_rec['nodes'])[:20])}"
+        return "comm_" + hashlib.sha256(key.encode("utf-8")).hexdigest()[:24]
+
+    # batch embed + upsert
+    B = 96
+    texts = [rec["summary"] for rec in flat_nodes]
+    ids = [summary_vec_id(rec) for rec in flat_nodes]
+    metas = [{
+        "type": "community_summary",
+        "community_id": rec["id"],
+        "level": rec["level"],
+        "size": rec["size"],
+        "node_sample": rec["nodes"][:20],
+        "text": rec["summary"]
+    } for rec in flat_nodes]
+
+    for start in range(0, len(texts), B):
+        batch_texts = texts[start:start+B]
+        batch_ids   = ids[start:start+B]
+        batch_metas = metas[start:start+B]
+        emb = pc.inference.embed(
+            model="llama-text-embed-v2",
+            inputs=batch_texts,
+            parameters={"input_type": "passage", "truncate": "END"}
+        )
+        vectors = [
+            {"id": vid, "values": e["values"], "metadata": meta}
+            for vid, e, meta in zip(batch_ids, emb, batch_metas)
+        ]
+        index.upsert(vectors=vectors, namespace=COMMUNITY_NAMESPACE)
+
+    if info:
+        print("[Community] Community summaries upsert completed.")
+
+def prepare_RAG(
+    pinecone_API,
+    index_name,
+    chunk_size=400,
+    chunk_overlap=30,
+    llm_model="gpt-4.1-nano",
+    dir_name="context",
+    info=True
+): 
+    """
+    Steps:
+        1) Select LLM wrapper (OpenAI vs. Mistral) by `llm_model` string.
+        2) Create dataset with `create_dataset(dir_name)`.
+        3) Connect to Pinecone and obtain `index`.
+        4) Split documents into chunks; normalize `metadata['source']` to be path-relative
+           to the `context` anchor (stable across machines).
+        5) Compute stable vector IDs per chunk from source+content hashes.
+        6) Use `fetch_existing_ids` to identify and skip already-indexed chunks.
+        7) Embed only new chunks via `pc.inference.embed` (retry with backoff).
+        8) Upsert embeddings and metadata into a fixed namespace (`example-namespace`).
+    """
+
+    import os, re, hashlib, time
+    from pinecone import Pinecone
+    from langchain_mistralai import ChatMistralAI
+    from langchain_openai import ChatOpenAI
+    from langchain_text_splitters import RecursiveCharacterTextSplitter
+
+    if info:
+        print(f"Preparing RAG with LLM: {llm_model}, Index: {index_name}, Dir: {dir_name}")
+    llm = ChatOpenAI(model=llm_model, streaming=True) if "gpt" in llm_model else ChatMistralAI(model=llm_model, streaming=True)
+
+    documents = create_dataset(dir_name)
+    pc = Pinecone(api_key=pinecone_API)
+    index = pc.Index(index_name)
+
+    if not documents:
+        print(f"[Warning] No documents found. Using existing Pinecone index.")
+        return index, pc, llm, None
+
+    splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
+    all_splits = splitter.split_documents(documents)
+
+    def path_after_context(full_path: str, anchor: str = "context") -> str:
+        if not full_path:
+            return ""
+        parts = re.split(r"[\\/]+", str(full_path))
+        idx = None
+        for i, part in enumerate(parts):
+            if part.lower() == anchor.lower():
+                idx = i
+        if idx is not None and idx < len(parts) - 1:
+            return "/".join(parts[idx + 1 :])
+        return os.path.basename(str(full_path))
+
+    for chunk in all_splits:
+        if "source" in chunk.metadata and chunk.metadata["source"]:
+            chunk.metadata["source"] = path_after_context(chunk.metadata["source"], anchor="context")
+
+    if info:
+        print(f"Total chunks: {len(all_splits)}")
+
+    def chunk_id(chunk, prefix="vec"):
+        text_hash = hashlib.sha256(chunk.page_content.encode("utf-8")).hexdigest()[:16]
+        source = chunk.metadata.get("source", "unknown")
+        file_hash = hashlib.sha256(source.encode("utf-8")).hexdigest()[:8]
+        return f"{prefix}_{file_hash}_{text_hash}"
+
+
+    # -------------------------
+    # Vector ID & Namespace Architecture
+    # -------------------------
+    """
+    VECTOR ID GENERATION STRATEGY:
+    
+    For Document Chunks:
+      Pattern: "vec_{file_hash}_{content_hash}"
+      - file_hash: SHA-256 of normalized source path (8 chars)
+      - content_hash: SHA-256 of chunk content (16 chars)
+      - Enables exact duplicate detection across runs
+      - Stable across different machine paths due to source normalization
+    
+    For Community Summaries:
+      Pattern: "comm_{community_hash}"
+      - community_hash: SHA-256 of "community_id|level|sorted_node_sample"
+      - Ensures stable IDs for the same community composition
+      - Allows updates when community structure changes
+    
+    NAMESPACE STRATEGY:
+    - "example-namespace": Stores document chunk embeddings
+    - "community-summaries": Stores hierarchical community summaries
+    - Separation enables independent update/query strategies
+    - Prevents interference between document and summary vectors
+    
+    IDEMPOTENCY GUARANTEE:
+    - fetch_existing_ids() checks Pinecone before embedding
+    - Prevents duplicate embeddings for identical content
+    - Enables safe re-runs without data duplication
+    - Reduces embedding costs and storage usage
+    """
+
+    namespace = "example-namespace"
+    all_ids = [chunk_id(c) for c in all_splits]
+    existing = fetch_existing_ids(index, namespace, all_ids, batch_size=100)
+    new_chunks = [(c, i) for c, i in zip(all_splits, all_ids) if i not in existing]
+
+    if info:
+        print(f"Chunks already indexed: {len(all_splits) - len(new_chunks)}")
+        print(f"New chunks to embed: {len(new_chunks)}")
+
+    if not new_chunks:
+        print("[Info] Nothing new to index. Skipping embedding/upsert.")
+    else:
+        batch_size = 94
+
+        def retry_forever(func, *args, **kwargs):
+            attempt = 1
+            while True:
+                try:
+                    return func(*args, **kwargs)
+                except Exception as e:
+                    wait = min(60, 2 ** min(attempt, 6))
+                    print(f"[Retry] {func.__name__} failed (attempt {attempt}): {e}. Sleeping {wait}s")
+                    time.sleep(wait)
+                    attempt += 1
+
+        for start_idx in range(0, len(new_chunks), batch_size):
+            print(f"[Info] Embedding and upserting batch {start_idx // batch_size + 1}...")
+            batch, ids = zip(*new_chunks[start_idx:start_idx + batch_size])
+            texts = [chunk.page_content for chunk in batch]
+            metas = [chunk.metadata or {} for chunk in batch]
+
+            embeddings = retry_forever(
+                pc.inference.embed,
+                model="llama-text-embed-v2",
+                inputs=texts,
+                parameters={"input_type": "passage", "truncate": "END"}
+            )
+
+            batch_records = [
+                {"id": i, "values": e['values'], "metadata": {"text": t, **m}}
+                for i, e, t, m in zip(ids, embeddings, texts, metas)
+            ]
+            retry_forever(index.upsert, vectors=batch_records, namespace=namespace)
+
+        if info:
+            print(f"Completed upsert of {len(new_chunks)} new vectors.")
+
+    return index, pc, llm, documents  # Return documents for KG construction
+
+
+def build_knowledge_graph(documents, llm, pc, index, info=True):
+    """
+    Build/update the Knowledge Graph (KG) from documents, persist it, merge deltas, and
+    (re)generate community summaries for changed regions.
+
+    Args:
+        documents: List of LangChain Documents from prepare_RAG (may be empty).
+        llm: LangChain-compatible LLM used via LlamaIndex.
+        pc: Pinecone client (for embeddings).
+        index: Pinecone index to store community summary vectors.
+        info: Enable verbose logging.
+
+    Returns:
+        KnowledgeGraphIndex | None
+
+    Flow:
+        1) Identify new/changed docs via source+content hashing (seen file cache).
+        2) Load existing KG from pickle or build a fresh one.
+        3) If there is a delta, build a delta KG and merge nodes/edges.
+        4) Summarize impacted communities and upsert summaries to Pinecone.
+        5) Export `knowledge_graph.json` and update the seen-file signatures.
+    """
+
+
+    import os, pickle, json, hashlib, re
+    from llama_index.core import Document, KnowledgeGraphIndex
+    from llama_index.llms.langchain import LangChainLLM
+
+    # ---- duplicate detection "like in prepare_RAG" (signature unchanged) ----
+    def path_after_context(full_path: str, anchor: str = "context") -> str:
+        if not full_path:
+            return ""
+        parts = re.split(r"[\\/]+", str(full_path))
+        idx = None
+        for i, part in enumerate(parts):
+            if part.lower() == anchor.lower():
+                idx = i
+        if idx is not None and idx < len(parts) - 1:
+            return "/".join(parts[idx + 1 :])
+        return os.path.basename(str(full_path))
+
+    def file_sig(doc_like):
+        """Return (sig_id, normalized_source) using source+content hashing similar to prepare_RAG."""
+        meta = getattr(doc_like, "metadata", {}) or {}
+        text = getattr(doc_like, "page_content", "") or getattr(doc_like, "text", "") or ""
+        src  = meta.get("source", "unknown")
+        if src:
+            src = path_after_context(src, anchor="context")
+        src_hash  = hashlib.sha256(src.encode("utf-8")).hexdigest()[:8]
+        text_hash = hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]
+        return f"kg_{src_hash}_{text_hash}", src
+
+    def load_seen_sigs(path="kg_seen_files.json"):
+        try:
+            if os.path.exists(path):
+                with open(path, "r", encoding="utf-8") as f:
+                    data = json.load(f)
+                return set(data if isinstance(data, list) else [])
+        except Exception as e:
+            print(f"[Warn] Failed to load seen file sigs: {e}")
+        return set()
+
+    def save_seen_sigs(sigs, path="kg_seen_files.json"):
+        try:
+            with open(path, "w", encoding="utf-8") as f:
+                json.dump(sorted(list(sigs)), f, indent=2)
+        except Exception as e:
+            print(f"[Warn] Failed to save seen file sigs: {e}")
+
+    # -------------------------
+    # Incremental KG Update Strategy
+    # -------------------------
+    """
+    CRITICAL: This section handles the complex merge of new documents into existing knowledge graphs.
+    
+    KEY CHALLENGES ADDRESSED:
+    - Duplicate Detection: Uses content+source hashing to identify truly new/changed documents
+    - Delta Processing: Builds partial KG from only new documents, then merges
+    - Conflict Resolution: Handles nodes/edges that may already exist in the base graph
+    - Change Propagation: Tracks exactly which nodes/edges are new for community summarization
+    
+    MERGE STRATEGY:
+    1. Signature-based filtering identifies only new/changed documents
+    2. Builds a "delta KG" from new documents only
+    3. Performs set operations to find truly new nodes/edges:
+       - new_nodes = delta_nodes - base_nodes
+       - new_edges = delta_edges - base_edges
+    4. Merges using NetworkX's native add_nodes_from/add_edges_from
+    5. Preserves all node/edge attributes during merge
+    
+    WHY THIS MATTERS:
+    - Without proper incremental updates, the system would rebuild the entire KG every time
+    - Enables efficient updates when only a few documents change
+    - Maintains community summaries for unchanged parts of the graph
+    """
+
+    seen_sigs = load_seen_sigs()
+
+    # Identify new/changed docs by signature
+    all_docs = documents or []
+    new_docs = []
+    new_sigs = []
+    for d in all_docs:
+        sig, _ = file_sig(d)
+        if sig not in seen_sigs:
+            new_docs.append(d)
+            new_sigs.append(sig)
+
+    if info:
+        print(f"[KG] Total input docs: {len(all_docs)} | New/changed docs detected: {len(new_docs)}")
+
+    # ---- prepare LlamaIndex objects ----
+    llama_docs_all   = [Document(text=doc.page_content, metadata=doc.metadata) for doc in all_docs]
+    llama_docs_delta = [Document(text=doc.page_content, metadata=doc.metadata) for doc in new_docs]
+    llm_for_kg = LangChainLLM(llm)
+    persist_file = os.path.abspath("./kg_index.pkl")
+
+    def _build_and_persist(docs):
+        kg = KnowledgeGraphIndex.from_documents(
+            documents=docs,
+            max_triplets_per_chunk=20,
+            extract_relations=True,
+            include_embeddings=True,
+            llm=llm_for_kg
+        )
+        with open(persist_file, "wb") as f:
+            pickle.dump(kg, f)
+        return kg
+
+    def _load_existing():
+        with open(persist_file, "rb") as f:
+            return pickle.load(f)
+
+    kg_index = None
+    graph_exists = False
+
+    try:
+        if os.path.exists(persist_file):
+            if info:
+                print("[Info] Found persisted KG pickle file.")
+            graph_exists = True
+            kg_index = _load_existing()
+            if info:
+                print("[Info] Loaded Knowledge Graph from pickle.")
+        elif llama_docs_all:
+            if info:
+                print("[Info] No persisted KG found. Building new KG from all documents...")
+            kg_index = _build_and_persist(llama_docs_all)
+            if info:
+                print("[Info] Built and persisted Knowledge Graph via pickle.")
+        else:
+            if info:
+                print("[Info] No persisted KG found and no documents to build from.")
+    except Exception as e:
+        print(f"[Error] GraphRAG init/load failed: {e}")
+        kg_index = None
+
+    # ---- incremental insertion (signature unchanged) ----
+    inserted_any = False
+    graph_override = None  # if we need merge fallback for community detection
+
+    new_nodes = set()
+    new_edges = set()
+
+    if kg_index and graph_exists and llama_docs_delta:
+        if info:
+            print(f"[Info] Incrementally inserting {len(llama_docs_delta)} new/changed docs into KG...")
+
+        ######################################################################
+        try:
+            # Build delta KG from new/changed docs
+            kg_delta = KnowledgeGraphIndex.from_documents(
+                documents=llama_docs_delta,
+                max_triplets_per_chunk=20,
+                extract_relations=True,
+                include_embeddings=False,
+                llm=llm_for_kg
+            )
+            nxg_base = kg_index.get_networkx_graph()
+            nxg_delta = kg_delta.get_networkx_graph()
+
+            # Diagnostic: Print node/edge sets before merge
+            base_nodes_before = set(nxg_base.nodes())
+            base_edges_before = set(nxg_base.edges())
+            delta_nodes = set(nxg_delta.nodes())
+            delta_edges = set(nxg_delta.edges())
+
+            print(f"\n[Diagnostic] Base graph nodes before merge: {len(base_nodes_before)}")
+            print(f"[Diagnostic] Base graph edges before merge: {len(base_edges_before)}")
+            print(f"[Diagnostic] Delta graph nodes: {len(delta_nodes)}")
+            print(f"[Diagnostic] Delta graph edges: {len(delta_edges)}")
+
+            # Show intersection and difference
+            new_nodes = delta_nodes - base_nodes_before
+            new_edges = delta_edges - base_edges_before
+            already_existing_nodes = delta_nodes & base_nodes_before
+            already_existing_edges = delta_edges & base_edges_before
+
+            print(f"[Diagnostic] Delta nodes already in base: {len(already_existing_nodes)}")
+            print(f"[Diagnostic] Delta edges already in base: {len(already_existing_edges)}")
+            print(f"[Diagnostic] Truly new nodes to add: {len(new_nodes)}")
+            print(f"[Diagnostic] Truly new edges to add: {len(new_edges)}")
+
+            # Merge delta into base
+            nxg_base.add_nodes_from(nxg_delta.nodes(data=True))
+            nxg_base.add_edges_from(nxg_delta.edges(data=True))
+            graph_override = nxg_base
+            inserted_any = True
+
+            # Diagnostic: Print node/edge sets after merge
+            base_nodes_after = set(nxg_base.nodes())
+            base_edges_after = set(nxg_base.edges())
+            print(f"\n[Diagnostic] Base graph nodes after merge: {len(base_nodes_after)}")
+            print(f"[Diagnostic] Base graph edges after merge: {len(base_edges_after)}")
+            print(f"[Diagnostic] Nodes added: {len(base_nodes_after - base_nodes_before)}")
+            print(f"[Diagnostic] Edges added: {len(base_edges_after - base_edges_before)}")
+
+            # Print delta graph summary
+            num_nodes = nxg_delta.number_of_nodes()
+            num_edges = nxg_delta.number_of_edges()
+
+            print(f"\n[Delta Graph Summary]")
+            print(f" - Total Nodes: {num_nodes}")
+            print(f" - Total Edges: {num_edges}")
+
+            # Print first 10 nodes
+            print("\n[Delta Graph Nodes] (showing up to 10):")
+            for i, (node, data) in enumerate(nxg_delta.nodes(data=True)):
+                if i >= 10:
+                    print(" ...")
+                    break
+                print(f" {i+1}. {node}: {data}")
+
+            # Print first 10 edges
+            print("\n[Delta Graph Edges] (showing up to 10):")
+            for i, (source, target, data) in enumerate(nxg_delta.edges(data=True)):
+                if i >= 10:
+                    print(" ...")
+                    break
+                print(f" {i+1}. {source} -> {target}: {data}")
+
+            # Warn if nothing new was actually added
+            if len(new_nodes) == 0 and len(new_edges) == 0:
+                print("[Warning] All delta nodes/edges already existed in the base graph. No actual change.")
+
+            if info:
+                print("[Info] Merged delta KG into existing graph (override used for summaries).")
+        except Exception as e:
+            print(f"[Error] Fallback merge failed: {e}")
+        ######################################################################
+            
+
+        # Persist KG if mutated via API
+        if inserted_any and graph_override is None:
+            try:
+                with open(persist_file, "wb") as f:
+                    pickle.dump(kg_index, f)
+            except Exception as e:
+                print(f"[Warn] Failed to persist updated KG: {e}")
+
+    # First-time build already happened above (graph_exists==False and llama_docs_all not empty)
+
+    # ---- Community summaries (incremental occurs inside the helper; same signature) ----
+    if kg_index:
+        # Only trigger summaries when: first build or we actually inserted/merged something
+        if not graph_exists or inserted_any:
+            _build_and_index_community_summaries(
+                kg_index=kg_index,
+                pc=pc,
+                index=index,
+                llm=llm,
+                impacted_nodes=new_nodes.union(u for u, v in new_edges).union(v for u, v in new_edges),
+                info=True
+            )
+
+        # Optional: save graph for visualization (post-update)
+        try:
+            nxg = graph_override if graph_override is not None else kg_index.get_networkx_graph()
+            graph_dict = {}
+            for u, v, attrs in nxg.edges(data=True):
+                rel = attrs.get("label") or attrs.get("relationship") or "related_to"
+                if u not in graph_dict:
+                    graph_dict[u] = []
+                graph_dict[u].append([rel, v])
+            output_file = "knowledge_graph.json"
+            with open(output_file, "w", encoding="utf-8") as f:
+                json.dump(graph_dict, f, indent=4, ensure_ascii=False)
+            if info:
+                print(f"[Info] Knowledge graph saved to {output_file}")
+        except Exception as e:
+            print(f"[Error] Failed to save knowledge graph: {e}")
+
+    # ---- mark seen signatures only after successful insertion or first build ----
+    if (not graph_exists and llama_docs_all) or inserted_any:
+        # Add only the new ones we processed this run
+        seen_sigs.update(new_sigs)
+        save_seen_sigs(seen_sigs)
+
+    return kg_index
+
+

rag_execute.py

@@ -0,0 +1,647 @@
+"""
+Main entry point for the RAG Gradio application.
+
+Loads environment variables, sets up context directory and model parameters,
+initializes retrieval and generation functions, and launches the interactive chat UI.
+Handles file uploads, user queries, and streaming LLM responses.
+"""
+
+
+import os
+import queue
+from threading import Thread
+from dotenv import load_dotenv
+load_dotenv()
+# Construct the path to the .env file relative to this script's location
+dotenv_path = os.path.join(os.path.dirname(__file__), '..', 'RAG-LangChain', '.env')
+print(f"Start loading .env from {dotenv_path}")
+load_dotenv(dotenv_path=dotenv_path)
+print(f"Finish loading .env")
+from langchain.callbacks.base import BaseCallbackHandler
+print(f"Start importing from rag_func")
+from prepare import prepare_RAG
+from retrieve import retrieve_RAG
+from generate import generate_RAG
+from prepare import build_knowledge_graph
+
+
+######
+# --- Graph viz imports (Plotly + NetworkX) ---
+import json, math, random
+import networkx as nx
+import numpy as np
+import plotly.graph_objects as go
+import plotly.express as px
+try:
+    from scipy.spatial import ConvexHull
+    SCIPY_AVAILABLE = True
+except Exception:
+    SCIPY_AVAILABLE = False
+
+######
+
+
+print(f"Finish importing from rag_func")
+import gradio as gr
+
+# -------------------- Context Setup --------------------
+
+user_dir = "context"
+#print default
+print(f"[Info] Using context directory: {user_dir}")
+
+pinecone_API = os.getenv("PINECONE_API")
+index_name   = os.getenv("INDEX_NAME")
+llm_model    = os.getenv("MODELNAME")
+
+#index, pc, llm, kg_index = prepare_RAG(pinecone_API, index_name, llm_model=llm_model, dir_name=user_dir, graph_rag=(graph_rag=="True"))
+index, pc, llm, kg_index = None, None, None, None
+
+# -------------------- Chat Functions --------------------
+def add_user_message(message, history):
+    """
+    Adds a new user message to the chat history.
+
+    Ensures the message is appended in the correct format for downstream processing.
+    Returns updated history for use in the chat UI.
+    """
+
+    history = history or []
+    history.append({"role": "user", "content": message})
+    return "", history, history
+
+import time
+
+# -------------------- Streaming Handler --------------------
+class StreamHandler(BaseCallbackHandler):
+    """
+    Callback handler for streaming LLM tokens to the UI.
+
+    Tracks timing for first token and total response, buffers tokens,
+    and manages the flow of streamed content for real-time display.
+    """
+
+    def __init__(self, q: queue.Queue):
+        self.q = q
+        self.first_token_received = False
+        self.ttft = None   # time to first token
+        self.total_time = None
+        self.start_time = None
+        self.buffer = []   # optional: accumulate tokens
+
+    def on_llm_new_token(self, token: str, **kwargs):
+        if not self.first_token_received:
+            self.ttft = time.time() - self.start_time
+            self.first_token_received = True
+        self.buffer.append(token)
+        self.q.put(token)
+
+    def on_llm_end(self, *args, **kwargs):
+        # IMPORTANT: do NOT end the consumer here.
+        # Let the worker thread send [[FINAL]] (if any) and then [[END]].
+        self.total_time = time.time() - self.start_time
+        # self.q.put("[[END]]")  # <-- REMOVED (this was breaking before we could send [[FINAL]])
+
+# -------------------- Chat Functions with timing --------------------
+
+
+
+def generate_bot_response(history):
+    """
+    Streams the first pass from the LLM to the UI and updates a styled progress box above the chat.
+    """
+    global index, pc, llm, kg_index
+
+    if not history or history[-1]["role"] != "user":
+        yield history, history, "<div style='background:#f5f5f5;padding:10px;border-radius:8px;'>Ready</div>"
+        return
+
+    user_msg = history[-1]["content"]
+    documents = None
+
+    # --- Stage 1: Initialize LLM / vector infra ---
+    yield history, history, "<div style='background:#f5f5f5;padding:10px;border-radius:8px;'>Initializing LLM and infrastructure...</div>"
+    if not index or not pc or not llm:
+        from langchain_mistralai import ChatMistralAI
+        from langchain_openai import ChatOpenAI
+
+        llm = ChatOpenAI(model=llm_model) if "gpt" in llm_model else ChatMistralAI(model=llm_model)
+        index, pc, llm, documents = prepare_RAG(
+            pinecone_API,
+            index_name,
+            llm_model=llm_model,
+            dir_name=user_dir,
+            info=True
+        )
+
+    # --- Stage 2: Decide Graph RAG usage ---
+    yield history, history, "<div style='background:#f5f5f5;padding:10px;border-radius:8px;'>Deciding Graph RAG usage...</div>"
+    def decide_graph_rag_usage(llm_, user_text: str) -> bool:
+        prompt = (
+            "Given the following user prompt, determine whether graph RAG should be used (True or False):\n"
+            f"{user_text}\n"
+            "Use 'False' only if the prompt is focused on retrieving a single fact.\n"
+            "Use 'True' if the prompt suggests reasoning over a large portion or the entirety of a dataset or corpus."
+        )
+        resp = llm_.invoke(prompt)
+        decision = (getattr(resp, "content", str(resp)) or "").strip()
+        print("[Debug] Graph RAG decision response:", decision)
+        return decision == "True"
+
+    graph_rag_flag = decide_graph_rag_usage(llm, user_msg)
+    print(f"[Info] Graph RAG usage decision: {graph_rag_flag}")
+
+    if graph_rag_flag and not documents:
+        _, _, _, documents = prepare_RAG(
+            pinecone_API,
+            index_name,
+            llm_model=llm_model,
+            dir_name=user_dir,
+            info=True
+        )
+
+    if graph_rag_flag:
+        kg_index = build_knowledge_graph(documents, llm, pc, index, info=True)
+
+    # --- Stage 3: Retrieve context ---
+    yield history, history, "<div style='background:#f5f5f5;padding:10px;border-radius:8px;'>Retrieving context...</div>"
+    retrieved_chunks, graph_context = retrieve_RAG(
+        user_msg,
+        pc,
+        index,
+        kg_index,
+        top_k=5,
+        use_query_reformulation=True,
+        llm=llm,
+        graphRAG=graph_rag_flag
+    )
+
+    # --- Stage 4: Generating response ---
+    yield history, history, "<div style='background:#f5f5f5;padding:10px;border-radius:8px;'>Generating response...</div>"
+    FINAL_PREFIX = "[[FINAL]]"
+    q = queue.Queue()
+    handler = StreamHandler(q)
+    handler.start_time = time.time()
+
+    model_name = getattr(llm, "model_name", getattr(llm, "model", None))
+    streaming_llm = llm.__class__(model=model_name, streaming=True, callbacks=[handler])
+
+    def run_llm():
+        try:
+            resp = generate_RAG(
+                user_msg,
+                streaming_llm,
+                retrieved_chunks,
+                graph_context,
+                graphRAG=graph_rag_flag
+            )
+            final_text = (getattr(resp, "content", str(resp)) or "").strip()
+            if final_text:
+                q.put(FINAL_PREFIX + final_text)
+        finally:
+            q.put("[[END]]")
+
+    Thread(target=run_llm, daemon=True).start()
+
+    partial = ""
+    history.append({"role": "assistant", "content": ""})
+
+    while True:
+        token = q.get()
+        if token == "[[END]]":
+            yield history, history, "<div style='background:#d4edda;padding:10px;border-radius:8px;'>Completed!</div>"
+            print(f"[Timing] TTFT: {handler.ttft:.3f} s, Total: {handler.total_time:.3f} s")
+            break
+
+        if token.startswith(FINAL_PREFIX):
+            final = token[len(FINAL_PREFIX):]
+            history[-1]["content"] = final
+            yield history, history, "<div style='background:#f5f5f5;padding:10px;border-radius:8px;'>Generating response...</div>"
+            partial = final
+            continue
+
+        partial += token
+        history[-1]["content"] = partial
+        yield history, history, "<div style='background:#f5f5f5;padding:10px;border-radius:8px;'>Generating response...</div>"
+
+
+
+# -------------------- Simplified CSS for Default Gradio Font --------------------
+from pathlib import Path
+import gradio as gr
+
+# Load external assets
+custom_css = Path("app.css").read_text(encoding="utf-8")
+js_force_light = """ function refresh() {
+  const url = new URL(window.location);
+  if (url.searchParams.get('__theme') !== 'light') {
+    url.searchParams.set('__theme', 'light');
+    window.location.replace(url);
+  }
+} """
+
+# -------------------- Gradio App --------------------
+import os
+import shutil
+MAX_TOTAL_SIZE_MB = 5
+CONTEXT_DIR = "context"
+
+def handle_file_upload(uploaded_files):
+    """
+    Validates and saves uploaded files to the context directory for RAG processing.
+
+    Checks file extensions and total upload size against allowed limits.
+    Returns a status message indicating success or failure for each upload attempt.
+    """
+
+    context_dir = "context"
+    os.makedirs(context_dir, exist_ok=True)
+    saved_files = []
+    total_size_mb = 0
+
+    # Allowed extensions
+    allowed_extensions = {".txt", ".json", ".md", ".csv", ".pdf", ".docx", ".pptx", ".py"}
+
+    for file_obj in uploaded_files:
+        # Check file extension
+        ext = os.path.splitext(file_obj.name)[1].lower()
+        if ext not in allowed_extensions:
+            return f"❌ Unsupported file type: {ext}. Allowed types are: {', '.join(sorted(allowed_extensions))}"
+        # Check size
+        file_size_mb = os.path.getsize(file_obj.name) / (1024 * 1024)
+        total_size_mb += file_size_mb
+        if total_size_mb > MAX_TOTAL_SIZE_MB:
+            return f"❌ Total upload size exceeds the limit of {MAX_TOTAL_SIZE_MB}MB."
+        # Save file
+        filename = os.path.basename(file_obj.name)
+        dest_path = os.path.join(context_dir, filename)
+        with open(file_obj.name, "rb") as src, open(dest_path, "wb") as dst:
+            dst.write(src.read())
+        saved_files.append(dest_path)
+
+    return f"✅ Uploaded {len(saved_files)} file(s) to '{context_dir}': {', '.join(os.path.basename(f) for f in saved_files)}"
+
+
+
+#########
+# ---------- Graph viz core ----------
+GRAPH_JSON_PATH = "knowledge_graph.json"
+COMMUNITY_MIN_SIZE = 3
+MERGE_SMALLS_POLICY = "bucket"  # or 'attach'
+LAYOUT_SEED = 42
+LAYOUT_ITERS = 30
+
+# Cached state (simple globals for now)
+_g_G = None
+_g_pos3d = None
+_g_node2comm = None
+_g_comm2nodes = None
+_g_edges = None
+_g_node_names = None
+
+def load_graph_from_json(path=GRAPH_JSON_PATH):
+    """Read {source: [[rel, target], ...], ...} and return a DiGraph."""
+    try:
+        with open(path, "r", encoding="utf-8") as f:
+            graph_dict = json.load(f)
+    except Exception:
+        graph_dict = {}
+    G = nx.DiGraph()
+    for source, edges_list in graph_dict.items():
+        for relation, target in edges_list:
+            G.add_edge(source, target, label=relation)
+    if G.number_of_nodes() == 0:
+        G.add_node("(empty)")
+    return G
+
+def precompute_layout_and_communities(G: nx.DiGraph):
+    """Compute 3D spring layout and top-level modularity communities."""
+    pos3d = nx.spring_layout(G, dim=3, seed=LAYOUT_SEED, iterations=LAYOUT_ITERS)
+    node_names = list(G.nodes())
+    edges = list(G.edges())
+
+    # Greedy modularity communities (on undirected projection)
+    from networkx.algorithms.community import greedy_modularity_communities
+    UG = nx.Graph()
+    UG.add_edges_from(G.to_undirected().edges())
+    communities = list(greedy_modularity_communities(UG))
+    large = [set(c) for c in communities if len(c) >= COMMUNITY_MIN_SIZE]
+    small = [set(c) for c in communities if len(c) < COMMUNITY_MIN_SIZE]
+
+    if MERGE_SMALLS_POLICY == "bucket" and small:
+        other = set().union(*small) if small else set()
+        if other:
+            large.append(other)
+        comm_ids = [f"C{i}" for i in range(len(large) - (1 if other else 0))]
+        if other:
+            comm_ids.append("C_other")
+    elif MERGE_SMALLS_POLICY == "attach" and small and large:
+        for s in small:
+            # attach to the large community with the most cross-edges
+            best_i, best_links = None, -1
+            for i, L in enumerate(large):
+                links = sum(1 for u in s for v in L if UG.has_edge(u, v))
+                if links > best_links:
+                    best_i, best_links = i, links
+            if best_i is None:
+                best_i = max(range(len(large)), key=lambda i: len(large[i]))
+            large[best_i].update(s)
+        comm_ids = [f"C{i}" for i in range(len(large))]
+    else:
+        comm_ids = [f"C{i}" for i in range(len(large))]
+
+    node2comm, comm2nodes = {}, {}
+    for cid, nodeset in zip(comm_ids, large):
+        comm2nodes[cid] = set(nodeset)
+        for n in nodeset:
+            node2comm[n] = cid
+    for n in G.nodes():
+        if n not in node2comm:
+            node2comm[n] = "C_isolated"
+            comm2nodes.setdefault("C_isolated", set()).add(n)
+
+    return pos3d, node2comm, comm2nodes, edges, node_names
+
+def _make_comm_colors(comm2nodes_dict):
+    palette = (px.colors.qualitative.Alphabet +
+               px.colors.qualitative.Set3 +
+               px.colors.qualitative.Bold +
+               px.colors.qualitative.Dark24 +
+               px.colors.qualitative.Light24)
+    cids = sorted(comm2nodes_dict.keys())
+    return {cid: palette[i % len(palette)] for i, cid in enumerate(cids)}
+
+def _community_hulls_traces(pos3d, comm2nodes, comm_colors, opacity=0.12):
+    if not SCIPY_AVAILABLE:
+        return []
+    hull_traces = []
+    for cid, nodeset in comm2nodes.items():
+        pts = np.array([pos3d[n] for n in nodeset if n in pos3d])
+        if pts.shape[0] < 4:
+            continue
+        try:
+            hull = ConvexHull(pts)
+            simplices = hull.simplices
+            hull_traces.append(go.Mesh3d(
+                x=pts[:, 0], y=pts[:, 1], z=pts[:, 2],
+                i=simplices[:, 0], j=simplices[:, 1], k=simplices[:, 2],
+                color=_make_comm_colors(comm2nodes).get(cid, "#cccccc"),
+                opacity=opacity, name=f"{cid} region",
+                hoverinfo="skip", showlegend=False
+            ))
+        except Exception:
+            pass
+    return hull_traces
+#########
+def build_plotly_figure(mode="community", highlight_node=None,
+                        highlight_comm_id=None, dim_inter_edges=True,
+                        show_hulls=False):
+    global _g_G, _g_pos3d, _g_node2comm, _g_comm2nodes, _g_edges, _g_node_names
+
+    # Load & cache if not present
+    if _g_G is None:
+        _g_G = load_graph_from_json()
+        _g_pos3d, _g_node2comm, _g_comm2nodes, _g_edges, _g_node_names = \
+            precompute_layout_and_communities(_g_G)
+
+    G = _g_G
+    pos3d = _g_pos3d
+    node2comm = _g_node2comm
+    comm2nodes = _g_comm2nodes
+    edges = _g_edges
+
+    # split intra/inter edges
+    edge_x_intra, edge_y_intra, edge_z_intra = [], [], []
+    edge_x_inter, edge_y_inter, edge_z_inter = [], [], []
+    for (u, v) in edges:
+        x0, y0, z0 = pos3d[u]
+        x1, y1, z1 = pos3d[v]
+        if node2comm.get(u) == node2comm.get(v):
+            edge_x_intra += [x0, x1, None]; edge_y_intra += [y0, y1, None]; edge_z_intra += [z0, z1, None]
+        else:
+            edge_x_inter += [x0, x1, None]; edge_y_inter += [y0, y1, None]; edge_z_inter += [z0, z1, None]
+
+    edge_traces = []
+    if edge_x_inter:
+        edge_traces.append(go.Scatter3d(
+            x=edge_x_inter, y=edge_y_inter, z=edge_z_inter,
+            mode="lines",
+            line=dict(width=1, color="rgba(180,180,180,0.30)" if dim_inter_edges else "#BBBBBB"),
+            hoverinfo="none", showlegend=False, name="Inter-community"
+        ))
+    if edge_x_intra:
+        edge_traces.append(go.Scatter3d(
+            x=edge_x_intra, y=edge_y_intra, z=edge_z_intra,
+            mode="lines",
+            line=dict(width=2, color="rgba(120,120,120,0.55)"),
+            hoverinfo="none", showlegend=False, name="Intra-community"
+        ))
+
+    comm_colors = _make_comm_colors(comm2nodes)
+    hull_traces = _community_hulls_traces(pos3d, comm2nodes, comm_colors) if show_hulls else []
+
+    # neighbor sets (if needed)
+    nbr_succ, nbr_pred = set(), set()
+    if mode == "neighbors" and highlight_node and highlight_node in G:
+        nbr_succ = set(G.neighbors(highlight_node))
+        nbr_pred = set(G.predecessors(highlight_node))
+
+    node_traces = []
+    for cid, nodeset in sorted(comm2nodes.items(), key=lambda kv: kv[0]):
+        xs, ys, zs, texts, colors, sizes = [], [], [], [], [], []
+        base_color = comm_colors.get(cid, "#66c2a5")
+        for n in nodeset:
+            x, y, z = pos3d[n]
+            xs.append(x); ys.append(y); zs.append(z); texts.append(n)
+            if mode == "neighbors":
+                if highlight_node == n:
+                    colors.append("red"); sizes.append(8.0)
+                elif n in nbr_succ or n in nbr_pred:
+                    colors.append("orange"); sizes.append(6.5)
+                elif highlight_node and node2comm.get(n) == node2comm.get(highlight_node):
+                    colors.append(base_color); sizes.append(5.5)
+                else:
+                    colors.append("lightblue"); sizes.append(5.0)
+            else:
+                if highlight_comm_id and node2comm.get(n) == highlight_comm_id:
+                    colors.append(base_color); sizes.append(6.5)
+                else:
+                    colors.append(base_color); sizes.append(5.0)
+        if xs:
+            node_traces.append(go.Scatter3d(
+                x=xs, y=ys, z=zs, mode="markers",
+                hovertext=texts, hoverinfo="text",
+                marker=dict(size=sizes, color=colors, opacity=0.95),
+                name=cid, showlegend=True
+            ))
+
+    fig = go.Figure(data=hull_traces + edge_traces + node_traces)
+    fig.update_layout(
+        title="3D Knowledge Graph — Communities & Neighbors",
+        showlegend=True if mode == "community" else False,
+        height=800,
+        margin=dict(l=0, r=0, t=40, b=0),
+        scene=dict(
+            xaxis=dict(visible=False), yaxis=dict(visible=False), zaxis=dict(visible=False),
+            aspectmode="data"
+        ),
+        scene_camera=dict(eye=dict(x=1.5, y=1.5, z=1.5)),
+        uirevision=True,
+    )
+    return fig
+
+def reload_graph_cache():
+    """Force re-read knowledge_graph.json and recompute layout/communities."""
+    global _g_G, _g_pos3d, _g_node2comm, _g_comm2nodes, _g_edges, _g_node_names
+    _g_G = load_graph_from_json()
+    _g_pos3d, _g_node2comm, _g_comm2nodes, _g_edges, _g_node_names = \
+        precompute_layout_and_communities(_g_G)
+    # Return a default figure
+    return build_plotly_figure(mode="community", highlight_comm_id=None, dim_inter_edges=True, show_hulls=False)
+
+#########
+
+
+
+with gr.Blocks(css=custom_css, fill_height=True, js=js_force_light) as demo:
+    with gr.Row():
+        # LEFT SIDE: Branding + Upload
+        with gr.Column(scale=1, elem_id="left-column"):
+            # Branding row: logo and title side by side
+            with gr.Row(elem_id="branding-row"):
+                import base64
+                from pathlib import Path
+
+                HERE = Path(__file__).resolve().parent
+                logo_path = HERE / "logo_mono.png"
+
+                with open(logo_path, "rb") as f:
+                    encoded = base64.b64encode(f.read()).decode()
+
+                gr.HTML(f"""
+                <div id="branding">
+                    <img id="company-logo" src="data:image/png;base64,{encoded}" alt="Logo" />
+                    <span id="brand-text">mosaiicRAG</span>
+                </div>
+                """)
+
+                   
+
+            gr.Markdown(
+                    "<p>Daten verstehen. Wissen vernetzen. Entscheidungen stärken.</p>",
+                    elem_id="header2"
+                )
+
+            # Below branding: upload info
+            gr.Markdown(
+                """
+                **Supported file formats:** .txt, .json, .md, .csv, .pdf, .docx, .pptx, .py
+                
+                **Maximum files' size:** 5 MB
+                """,
+                elem_id="upload-note"
+            )
+
+            file_upload = gr.File(
+                label="Upload files for RAG context",
+                file_count="multiple",
+                elem_id="file-upload"
+            )
+            upload_btn = gr.Button("Upload")
+            upload_output = gr.Textbox(label="Upload status", interactive=False)
+            upload_btn.click(handle_file_upload, inputs=[file_upload], outputs=[upload_output])
+
+        
+        with gr.Column(scale=4, elem_id="right-column"):
+            with gr.Tabs():
+                # ------------------------- Chat tab (unchanged) -------------------------
+                with gr.Tab("Chat"):
+                    with gr.Column(elem_id="chat-area"):
+                        progress_box = gr.HTML("<div style='background:#f5f5f5;padding:10px;border-radius:8px;margin-bottom:10px;'>Ready</div>")
+                        chatbot = gr.Chatbot(type="messages", label="Conversation", elem_id="chatbot")
+                        with gr.Row(elem_id="input-row"):
+                            msg = gr.Textbox(placeholder="Type your question here...", lines=1)
+                            send_btn = gr.Button("➤", elem_id="send-button", size="sm")
+                        state = gr.State([])
+                        msg.submit(add_user_message, inputs=[msg, state], outputs=[msg, chatbot, state])\
+                           .then(generate_bot_response, inputs=[state], outputs=[chatbot, state, progress_box])
+                        send_btn.click(add_user_message, inputs=[msg, state], outputs=[msg, chatbot, state])\
+                                .then(generate_bot_response, inputs=[state], outputs=[chatbot, state, progress_box])
+
+                # --------------------- Knowledge Graph tab (updated) ---------------------
+                with gr.Tab("Knowledge Graph"):
+                    with gr.Row():
+                        color_mode = gr.Radio(
+                            ["community"],
+                            value="community",
+                            label="Color mode"
+                        )
+                        community_select = gr.Dropdown(
+                            label="Highlight community (optional)",
+                            choices=[],
+                            value=None
+                        )
+                        view_opts = gr.CheckboxGroup(
+                            choices=[
+                                "Dim inter-community edges",
+                                f"Show 3D community hulls{' (requires scipy)' if not SCIPY_AVAILABLE else ''}"
+                            ],
+                            value=["Dim inter-community edges"],
+                            label="View options"
+                        )
+                        reload_btn = gr.Button("Reload graph")
+
+                    graph_plot = gr.Plot(label="3D Knowledge Graph")
+                    node_info = gr.Markdown("")
+
+                    # ---- functions bound to UI (defined above or inline) ----
+                    def _init_graph():
+                        # Rebuild cache from knowledge_graph.json and return default figure
+                        fig = reload_graph_cache()
+                        cids = sorted(list(_g_comm2nodes.keys())) if _g_comm2nodes else []
+                        info = "Select a community or click a node to highlight its community."
+                        # Use gr.update to set dropdown choices
+                        return fig, gr.update(choices=cids, value=None), info
+
+                    def _refresh(mode, selected_cid, opts):
+                        dim_edges = isinstance(opts, list) and ("Dim inter-community edges" in opts)
+                        show_hulls = isinstance(opts, list) and any("Show 3D community hulls" in s for s in opts)
+
+                        fig = build_plotly_figure(
+                            mode="community" if mode == "community" else "neighbors",
+                            highlight_comm_id=(selected_cid if mode == "community" else None),
+                            dim_inter_edges=dim_edges,
+                            show_hulls=(show_hulls if mode == "community" else False)
+                        )
+                        info = (
+                            "Select a community or click a node to highlight its community."
+                            if mode == "community"
+                            else "Click a node to see its neighbors (community tint applied)."
+                        )
+                        return fig, info
+
+                    def _reload(mode, selected_cid, opts):
+                        # Reload data and recompute communities/layout
+                        _ = reload_graph_cache()
+                        cids = sorted(list(_g_comm2nodes.keys())) if _g_comm2nodes else []
+                        # Immediately apply current UI options on the new graph state
+                        fig, info = _refresh(mode, selected_cid, opts)
+                        return fig, gr.update(choices=cids, value=selected_cid), info
+
+                    # wire controls
+                    color_mode.change(_refresh, inputs=[color_mode, community_select, view_opts],
+                                      outputs=[graph_plot, node_info])
+                    community_select.change(_refresh, inputs=[color_mode, community_select, view_opts],
+                                            outputs=[graph_plot, node_info])
+                    view_opts.change(_refresh, inputs=[color_mode, community_select, view_opts],
+                                     outputs=[graph_plot, node_info])
+
+                    reload_btn.click(_reload, inputs=[color_mode, community_select, view_opts],
+                                     outputs=[graph_plot, community_select, node_info])
+
+    # ------------------------ IMPORTANT: INSIDE THE BLOCKS ------------------------
+    # Initialize the graph once when the app loads (now inside the Blocks context)
+    demo.load(_init_graph, inputs=[], outputs=[graph_plot, community_select, node_info])
+# -------------------- Launch App --------------------
+if __name__ == "__main__":
+    demo.launch(inbrowser=True)

rag_on_prem.py

@@ -0,0 +1,216 @@
+import os
+import pathlib
+import gradio as gr
+
+# LangChain imports
+from langchain_community.document_loaders import (
+    CSVLoader, PyPDFLoader, UnstructuredWordDocumentLoader,
+    UnstructuredPowerPointLoader, UnstructuredMarkdownLoader,
+    UnstructuredHTMLLoader, NotebookLoader
+)
+from langchain_core.documents import Document
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+from langchain_ollama import ChatOllama, OllamaEmbeddings
+from langchain_community.vectorstores import FAISS
+from langchain_core.messages import HumanMessage, SystemMessage
+
+# -------------------------
+# UTF-8 safe Text Loader
+# -------------------------
+class SafeTextLoader:
+    """Loads a text file as a single Document, safely handling UTF-8 decoding errors."""
+    def __init__(self, file_path):
+        self.file_path = file_path
+        print(f"[Debug] Initialized SafeTextLoader with file_path: {file_path}")
+
+    def load(self):
+        try:
+            print(f"[Debug] Attempting to load file: {self.file_path}")
+            with open(self.file_path, "rb") as f:  # open in binary mode
+                raw_bytes = f.read()
+            text = raw_bytes.decode("utf-8", errors="ignore")  # decode safely
+            print(f"[Debug] Successfully loaded file: {self.file_path}")
+            return [Document(page_content=text, metadata={"source": str(self.file_path)})]
+        except Exception as e:
+            print(f"[Error] Failed to read {self.file_path}: {e}")
+            return []
+
+# -------------------------
+# Loader mapping
+# -------------------------
+LOADER_MAPPING = {
+    # Text
+    ".txt": SafeTextLoader,
+    ".json": SafeTextLoader,
+    ".md": UnstructuredMarkdownLoader,
+    ".csv": CSVLoader,
+    ".yaml": SafeTextLoader,
+    ".yml": SafeTextLoader,
+
+    # Documents
+    ".pdf": PyPDFLoader,
+    ".docx": UnstructuredWordDocumentLoader,
+    ".pptx": UnstructuredPowerPointLoader,
+    ".html": UnstructuredHTMLLoader,
+    ".htm": UnstructuredHTMLLoader,
+
+    # Code / Notebook
+    ".ipynb": NotebookLoader,
+    ".py": SafeTextLoader,
+    ".js": SafeTextLoader,
+    ".sql": SafeTextLoader,
+}
+
+# -------------------------
+# Dataset creation
+# -------------------------
+def create_dataset(directory_path: str = "context"):
+    """Loads all supported files from the given directory (recursively)."""
+    print(f"[Debug] Starting dataset creation for directory: {directory_path}")
+    target_dir = pathlib.Path(directory_path).resolve()
+
+    if not target_dir.exists() or not target_dir.is_dir():
+        print(f"[Error] Target directory does not exist: {target_dir}")
+        return []
+
+    documents = []
+
+    for file_path in target_dir.rglob("*"):  # recursive
+        if not file_path.is_file():
+            continue
+
+        ext = file_path.suffix.lower()
+        loader_cls = LOADER_MAPPING.get(ext)
+
+        if loader_cls is None:
+            print(f"[Skip] Unsupported file type: {file_path}")
+            continue
+
+        try:
+            print(f"[Debug] Loading file: {file_path}")
+            loader = loader_cls(str(file_path))
+            docs = loader.load()
+            documents.extend(docs)
+            print(f"[Loaded] {file_path} ({len(docs)} docs)")
+        except Exception as e:
+            print(f"[Error] Failed to load {file_path}: {e}")
+
+    print(f"[Done] Finished scanning {target_dir}")
+    print(f"Total documents loaded: {len(documents)}")
+    return documents
+
+# -------------------------
+# Prepare RAG (Ollama + FAISS)
+# -------------------------
+def prepare_RAG(dir_name="context", chunk_size=600, chunk_overlap=50):
+    print(f"[Debug] Preparing RAG with Ollama + FAISS. Context dir={dir_name}")
+
+    documents = create_dataset(dir_name)
+    if not documents:
+        raise ValueError("No documents loaded. Please add files to the context directory.")
+
+    text_splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)
+    print(f"[Debug] Splitting documents into chunks with chunk_size={chunk_size}, chunk_overlap={chunk_overlap}")
+    all_splits = text_splitter.split_documents(documents)
+
+    print(f"[Debug] Number of chunks created: {len(all_splits)}")
+
+    # Ollama embeddings (local)
+    print(f"[Debug] Initializing Ollama embeddings")
+    embeddings = OllamaEmbeddings(model="nomic-embed-text")
+
+    # FAISS vector store
+    print(f"[Debug] Creating FAISS vector store")
+    vectorstore = FAISS.from_documents(all_splits, embeddings)
+
+    # Ollama LLM
+    print(f"[Debug] Initializing Ollama LLM")
+    llm = ChatOllama(model="llama3")  # change model if needed
+
+    return vectorstore, llm
+
+# -------------------------
+# Retrieval
+# -------------------------
+def retrieve_RAG(query, vectorstore, top_k=5):
+    print(f"[Debug] Retrieving top {top_k} documents for query: {query}")
+    retriever = vectorstore.as_retriever(search_kwargs={"k": top_k})
+    results = retriever.get_relevant_documents(query)
+    print(f"[Debug] Retrieved {len(results)} documents")
+    return results
+
+# -------------------------
+# Generation
+# -------------------------
+def generate_RAG(prompt_message, llm, retrieved_docs):
+    print(f"[Debug] Generating response for prompt: {prompt_message}")
+    context_message = (
+        "You are an expert assistant. Use ONLY the provided context documents "
+        "to answer the question. If the context does not contain the answer, reply with 'I don’t know'."
+    )
+
+    context_text = "\n\n".join([d.page_content for d in retrieved_docs])
+    print(f"[Debug] Context for generation: {context_text[:500]}... (truncated)")
+
+    prompt = [
+        SystemMessage(content=context_message),
+        HumanMessage(content=f"Context:\n{context_text}\n\nQuestion: {prompt_message}")
+    ]
+
+    response = llm.invoke(prompt)
+    print(f"[Debug] Generated response: {response.content}")
+    return response
+
+# -------------------------
+# Gradio Chatbot
+# -------------------------
+def run_chatbot(user_dir="context"):
+    print(f"[Debug] Starting chatbot with user_dir: {user_dir}")
+    vectorstore, llm = prepare_RAG(dir_name=user_dir)
+
+    # Step 1: Add user message
+    def add_user_message(message, history):
+        print(f"[Debug] Adding user message: {message}")
+        history = history or []
+        history.append({"role": "user", "content": message})
+        return "", history, history
+
+    # Step 2: Generate bot response
+    def generate_bot_response(history):
+        if not history or history[-1]["role"] != "user":
+            print(f"[Debug] No user message to respond to.")
+            return history, history
+
+        user_msg = history[-1]["content"]
+        print(f"[Debug] Generating response for user message: {user_msg}")
+        retrieved_docs = retrieve_RAG(user_msg, vectorstore)
+        response = generate_RAG(user_msg, llm, retrieved_docs)
+
+        history.append({"role": "assistant", "content": response.content})
+        return history, history
+
+    with gr.Blocks() as demo:
+        gr.Markdown("# 📚 On-Prem RAG Chatbot (Ollama + FAISS)")
+        gr.Markdown("Ask questions about your local documents.")
+
+        chatbot = gr.Chatbot(type="messages")
+        msg = gr.Textbox(label="Your message")
+        state = gr.State([])
+
+        msg.submit(add_user_message, inputs=[msg, state], outputs=[msg, chatbot, state]) \
+           .then(generate_bot_response, inputs=[state], outputs=[chatbot, state])
+
+    demo.launch()
+
+# -------------------------
+# Main
+# -------------------------
+if __name__ == "__main__":
+    user_input = input("Enter a subfolder inside 'context' (press Enter for full 'context'): ").strip()
+    if not user_input:
+        user_dir = "context"
+    else:
+        user_dir = os.path.join("context", user_input)
+
+    print(f"[Info] Using context directory: {user_dir}")
+    run_chatbot(user_dir)

requirements.txt

@@ -0,0 +1,19 @@
+langchain
+langchain-core
+langchain-community
+langchain-openai
+langchain-mistralai
+langchain-text-splitters
+llama-index==0.14.5
+llama-index-llms-langchain==0.7.1
+datasets==4.1.0
+ragas==0.3.5
+langgraph==0.6.7
+gradio==5.46.0
+python-dotenv==1.1.1
+nbformat==5.10.4
+nbconvert==7.16.6
+pypdf==6.0.0
+unstructured[docx,pptx,html,md]==0.18.15
+pinecone==7.3.0
+plotly

retrieve.py

@@ -0,0 +1,180 @@
+import os
+import pathlib
+import time
+import re
+from pinecone import Pinecone
+
+from langchain_mistralai import ChatMistralAI
+from langchain_openai import ChatOpenAI
+from langchain_core.messages import HumanMessage, SystemMessage
+from langchain.schema import Document
+from langchain_community.document_loaders import (
+    CSVLoader, PyPDFLoader, UnstructuredWordDocumentLoader,
+    UnstructuredPowerPointLoader, UnstructuredMarkdownLoader,
+    UnstructuredHTMLLoader, NotebookLoader
+)
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+
+from llama_index.core.memory import Memory
+
+import pickle
+
+import json
+from typing import List, Any
+from langchain_core.messages import SystemMessage, HumanMessage, AIMessage, BaseMessage
+
+from typing import List, Any
+from pydantic import BaseModel, ValidationError
+
+
+
+
+
+def retrieve_RAG(
+    prompt_message, pc, index, kg_index, top_k=5, info=True,
+    use_query_reformulation=False, llm=None, graphRAG=False,
+):
+    """
+    Retrieve relevant document chunks and community summaries from Pinecone for a given prompt.
+    - Optionally splits and reformulates the prompt for improved search.
+    - Searches both standard document chunks and, if enabled, community summaries from the knowledge graph.
+    - Returns all retrieved results for further use.
+    """
+
+    import os
+    import re
+
+    if info:
+        print("[Debug] Starting retrieval with prompt:", prompt_message)
+        print("[Debug] Top K:", top_k)
+        print("[Debug] Query Reformulation Enabled:", use_query_reformulation)
+
+    # --- Step 0: Decide context usage (standard, graph, both) ---
+    def _graph_available():
+        try:
+            stats = index.describe_index_stats()
+            namespaces = stats.get("namespaces", {}) or {}
+            return "community-summaries" in namespaces
+        except Exception as e:
+            print(f"[Error] Failed to inspect index namespaces: {e}")
+            return False
+
+    graph_ok = bool(kg_index) or _graph_available()
+
+    # --- Step 1: Use LLM to split the prompt into sub-queries ---
+    sub_queries = [prompt_message]  # fallback: single query
+    if llm is not None:
+        try:
+            split_prompt = (
+                "Given the following user query, identify and list all distinct sub-queries or tasks it contains. "
+                "Return ONLY a numbered list of sub-queries, each as a concise phrase.\n\n"
+                f"User Query: {prompt_message}"
+            )
+            split_response = llm.invoke(split_prompt)
+            sub_queries = re.findall(r"\d+\.\s*(.+)", split_response.content)
+            if not sub_queries:
+                sub_queries = [prompt_message]
+            if info:
+                print(f"[Debug] Identified sub-queries: {sub_queries}")
+        except Exception as e:
+            print(f"[Error] Sub-query splitting failed: {e}")
+
+    all_retrieved_chunks = []
+    all_graph_context_blocks = []
+
+    # --- Step 2: For each sub-query, retrieve context as decided ---
+    for idx, sub_query in enumerate(sub_queries):
+        task_prompt = sub_query.strip()
+
+        # Optional Query Reformulation
+        if use_query_reformulation and llm is not None:
+            try:
+                reformulation_prompt = (
+                    "Reformulate the following query to focus only on the key concepts and remove any unnecessary details. "
+                    "It should be suitable for vector search in RAG retrieval:\n\n"
+                    f"Original Query: {task_prompt}"
+                )
+                reformulated_response = llm.invoke(reformulation_prompt)
+                task_prompt = reformulated_response.content.strip()
+                if info:
+                    print(f"[Debug] Reformulated Query for sub-query {idx+1}: {task_prompt}")
+            except Exception as e:
+                print(f"[Error] Query reformulation failed for sub-query {idx+1}: {e}")
+
+        # Embed the sub-query
+        query_embedding = pc.inference.embed(
+            model="llama-text-embed-v2",
+            inputs=[task_prompt],
+            parameters={"input_type": "query"}
+        )
+        if info:
+            print(f"[Debug] Query embedding generated for sub-query {idx+1}.")
+        qvec = query_embedding[0].values
+
+        # --- Retrieve chunks if context_choice is standard or both ---
+        try:
+            retrieved_chunks_raw = index.query(
+                namespace="example-namespace",
+                vector=qvec,
+                top_k=top_k,
+                include_values=False,
+                include_metadata=True
+            )
+            retrieved_chunks = []
+            for match in retrieved_chunks_raw.matches:
+                text = match.metadata.get("text", "")
+                source = match.metadata.get("source", "Unknown source")
+                retrieved_chunks.append({
+                    "text": text,
+                    "source": source,
+                    "sub_query": sub_query
+                })
+            all_retrieved_chunks.extend(retrieved_chunks)
+            if info:
+                print(f"[Debug] Match processed for sub-query {idx+1}: text='{text[:50]}...', source='{source}'")
+        except Exception as e:
+            print(f"[Error] Standard retrieval failed for sub-query {idx+1}: {e}")
+
+        # --- Retrieve community summaries if context_choice is graph or both ---
+        if graphRAG:
+            COMMUNITY_NAMESPACE = "community-summaries"
+            TOP_K_SUMMARIES = 5
+            try:
+                comm_matches = index.query(
+                    namespace=COMMUNITY_NAMESPACE,
+                    vector=qvec,
+                    top_k=TOP_K_SUMMARIES,
+                    include_values=False,
+                    include_metadata=True
+                )
+                blocks = []
+                for m in comm_matches.matches:
+                    meta = m.metadata or {}
+                    txt = meta.get("text", "")
+                    cid = meta.get("community_id", "NA")
+                    level = meta.get("level", -1)
+                    size = meta.get("size", 0)
+                    block = f"[Community {cid} \n level={level} \n size={size}]\n{txt}"
+                    blocks.append(block)
+                graph_context_str = ("\n\n---\n\n").join(blocks)
+                all_graph_context_blocks.append((sub_query, graph_context_str))
+                if info:
+                    print(f"[Community] Retrieved {len(blocks)} community summaries for sub-query {idx+1}.")
+            except Exception as e:
+                print(f"[Error] Community summaries retrieval failed for sub-query {idx+1}: {e}")
+
+    # --- Step 3: Aggregate results ---
+    combined_graph_context = "\n\n====\n\n".join(
+        f"Sub-query: {sub_query}\n{context}"
+        for (sub_query, context) in all_graph_context_blocks if context
+    )
+
+    if info:
+        sources = {os.path.basename(chunk['source']) for chunk in all_retrieved_chunks}
+        print(f"[Debug] Final retrieval: {len(all_retrieved_chunks)} chunks from {len(sources)} sources, "
+              f"graph context length {len(combined_graph_context)}.")
+
+    # --- Return as before ---
+    return all_retrieved_chunks, combined_graph_context
+
+