deploy: automated sync from github main

.gitattributes

@@ -0,0 +1,5 @@
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.sqlite3 filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text

.github/workflows/deploy.yml

@@ -0,0 +1,43 @@
+name: Sync to Hugging Face Space
+
+on:
+  push:
+    branches: [ main ]
+  workflow_dispatch:
+
+jobs:
+  sync-to-huggingface:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Push to Hugging Face Space
+        env:
+          HF_TOKEN: ${{ secrets.HF_TOKEN }}
+        run: |
+          git config --global user.name "GitHub Actions Bot"
+          git config --global user.email "actions@github.com"
+          git checkout --orphan hf-deploy-temp
+          git rm -rf --cached . 2>/dev/null || true
+          git add -A
+          git reset -- Images/ db/ *.png *.jpg *.jpeg 2>/dev/null || true
+          git commit -m "deploy: automated sync from github main" || true
+          git push --force https://neural-arun:${HF_TOKEN}@huggingface.co/spaces/neural-arun/ArunCore hf-deploy-temp:main
+
+      - name: Restart Hugging Face Space
+        env:
+          HF_TOKEN: ${{ secrets.HF_TOKEN }}
+        run: |
+          echo "Triggering Space restart to apply new code..."
+          STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
+            -X POST "https://huggingface.co/api/spaces/neural-arun/ArunCore/restart?factory=true" \
+            -H "Authorization: Bearer ${HF_TOKEN}")
+          echo "HF Restart API response: $STATUS"
+          if [ "$STATUS" = "200" ] || [ "$STATUS" = "204" ]; then
+            echo "✅ Space restart triggered successfully!"
+          else
+            echo "⚠️ Restart returned status $STATUS — Space may need a manual restart."
+          fi

.gitignore

@@ -0,0 +1,36 @@
+# API Keys and Secrets
+.env
+
+# Python Environment
+venv/
+.venv/
+env/
+*.py[cod]
+__pycache__/
+.pytest_cache/
+
+
+
+# Evaluation Logs and Debugging
+evaluation_debug/
+temp_error.txt
+error.txt
+data/test_set/evaluation_report.json
+
+# IDE Settings
+.vscode/
+.idea/
+
+# OS Junk
+.DS_Store
+Thumbs.db
+
+.vercel
+
+# Database Files
+db/
+
+# Image Artifacts
+*.png
+*.jpg
+*.jpeg

.opencode/opencode-loop/ses_04f15b806ffeVMcZMQ3SvTElIa.json

@@ -0,0 +1,4 @@
+{
+  "version": 4,
+  "jobs": []
+}

.opencode/opencode-loop/ses_04f15c7c7ffeptX8ZkC4eI4nwV.json

@@ -0,0 +1,4 @@
+{
+  "version": 4,
+  "jobs": []
+}

.opencode/opencode-loop/ses_06a20babdffeQ3bykyB3CydzUC.json

@@ -0,0 +1,4 @@
+{
+  "version": 4,
+  "jobs": []
+}

.opencode/opencode-loop/ses_06a20e31effezqv3Ypf6NlGToc.json

@@ -0,0 +1,4 @@
+{
+  "version": 4,
+  "jobs": []
+}

.python-version

@@ -0,0 +1 @@
+3.11

AGENTS.md

@@ -0,0 +1,128 @@
+# 🤖 AGENTS.md — Master Protocol & Execution Rules for AI Coding Agents
+
+> **IMPORTANT**: Any coding agent or LLM-assisted development tool (Antigravity, Claude, Cursor, Copilot, etc.) working on this repository **MUST** read this file and follow every rule strictly.
+> **ARCHITECTURE VERSION**: ❄️ **v1.0 FROZEN** (Official Release Specification)
+
+---
+
+## 🎯 Project Mission
+ArunCore is a domain-generic, multi-tenant enterprise AI platform.
+Your primary objective is **NOT** to maximize code generation.
+Your primary objective is to **preserve architectural integrity by default while implementing the smallest correct change**.
+If a requested feature requires architectural changes, explain the tradeoffs and ask for user confirmation before proceeding.
+
+---
+
+## 🔝 Priority Order (Highest → Lowest)
+When resolving conflicts, follow this exact hierarchy:
+1. 👤 **User explicit instructions**
+2. 📄 **AGENTS.md** (Master Protocol & Rules)
+3. 📐 **aruncore_master_blueprint.md** (Architecture Specification)
+4. 📂 **understanding_each_folder/** (Technical Folder Guides)
+5. ⚙️ **Existing Codebase**
+
+---
+
+## 📜 File Reading Priority
+Always inspect files in this exact hierarchy before modifying code:
+1. 📄 Master Architecture Spec: [`aruncore_master_blueprint.md`](file:///home/arun/projects/profile/aruncore_master_blueprint.md)
+2. 📂 Folder Technical Guides: [`understanding_each_folder/README.md`](file:///home/arun/projects/profile/understanding_each_folder/README.md)
+3. 📋 Validation Schemas: `backend/app/schemas/` (`tenant.py`, `chat.py`)
+4. ⚙️ Source Code: `backend/app/` or `frontend/`
+5. 🧪 Test Suite: `tests/`
+
+---
+
+## 🏛️ The 11 Golden Architecture Principles (NON-NEGOTIABLE)
+
+1. 🥇 **Preserve Current Frontend 100%**: Absolutely DO NOT replace or alter the existing Next.js frontend UI (`frontend/app/page.tsx`, `frontend/components/ChatPanel.tsx`, `ProjectsView.tsx`, `ManifestoView.tsx`, `Header.tsx`). The existing luxury UI, design tokens, glowing accent borders, light/dark themes, hero assistant card, and tab views stay 100% intact!
+2. 🥈 **Zero Functionality Loss (Keep & Enhance)**: DO NOT remove a single working feature from the current app (Hybrid RAG, Telegram active learning alerts, 3-way live human chat presence, TTS neural voice, admin mode, session history). The goal is to keep 100% of existing functionality and upgrade it to production grade!
+3. 🥉 **Production-Grade Hybrid RAG Engine**: Maintain and enhance the hybrid RAG architecture (dense ChromaDB vector search + sparse BM25 keyword search + Cohere/LLM reranking) for ultra-fast, high-precision knowledge retrieval per tenant.
+4. 🏅 **Ultra-Simple Demo System**: Demo mode operates by passing a URL query parameter (`?tutor=ed_donner` or `?client=ed_donner`). The frontend fetches metadata from `/api/v1/config?tutor=ed_donner` (or `/config?tutor=ed_donner`), which dynamically populates the existing hero card title, role subtitle, avatar, welcome text, suggested questions, and AI system prompt. Zero complex slot engines required!
+5. 🏅 **Explicit `tenants/` Directory**: All raw PDFs, markdown files, avatars, logos, and vector databases live in `./tenants/` outside the Git code repo. Keep code repos lightweight (~10MB).
+6. 🏅 **Split Targeted Config**: Tenant configs in `tenants/<id>/config/` are split into 6 targeted JSON files (`brand.json`, `agent.json`, `chat.json`, `voice.json`, `seo.json`, `social.json`). Easy debugging, zero merge conflicts, Pydantic validated.
+7. 🏅 **ZERO Backend Client `if` Statements**: Absolutely NO `if client == "ed":` or `if tenant_id == "hitesh":` statements anywhere in Python code! All tenant logic, prompts, tools, and branding MUST be resolved dynamically by `TenantService`.
+8. 🏅 **Config-Driven Tool Registry**: Client tools are controlled via the `enabled_tools` array in `agent.json` (`["search_courses", "book_calendar", "faq_lookup"]`). `ToolExecutor` dynamically binds enabled tools into the LLM execution loop—zero Python code edits required!
+9. 🏅 **Separation of Tenant Assets**: Static brand images (`tenants/<id>/assets/avatars/`, `tenants/<id>/assets/logos/`) are kept separate from code logic.
+10. 🏅 **Decoupled Single-Responsibility Services**: Maintain small, focused backend services (`PromptBuilder`, `MemoryManager`, `ToolExecutor`, `AgentRunner`, `RAGService`, `NotificationService`, `ActiveLearningService`). Do NOT create monster monolithic service files.
+11. 🏅 **Interface Abstractions**: Use abstract base classes for adapters (`VectorStore`, `StateStore`, `NotificationProvider`). Never hardcode direct dependencies in business logic.
+
+---
+
+## ⚙️ Required Execution Workflow
+
+1. **Understand**: Read the task and relevant architecture docs.
+2. **Locate**: Identify the exact affected files.
+3. **Plan**: If the change affects >5 files, public APIs, schemas, or architecture -> create a plan first. Do NOT write code immediately.
+4. **Minimal Change**: Apply the smallest possible correct change. Prefer modifying existing code over spawning new modules.
+5. **Verify**: Run automated tests (`python3 -m unittest discover -s tests`).
+6. **Update Docs**: Immediately update both `aruncore_master_blueprint.md` AND `understanding_each_folder/` guides.
+7. **Self-Review**: Review your diff against the pre-completion checklist.
+
+---
+
+## ❓ Ask Before Proceeding
+
+Always ask for user confirmation before:
+- Deleting files or modules.
+- Renaming large directories.
+- Changing public API contracts or endpoints.
+- Modifying validation schemas or database structures.
+- Changing authentication, security, or permissions logic.
+- Modifying package dependencies or build tooling.
+
+---
+
+## 🛑 When NOT to Code (Escalation Triggers)
+
+Stop coding immediately, explain the conflict, and ask for clarification when:
+- Architecture or requirements are unclear.
+- API contracts or schemas are missing.
+- The request contradicts any Golden Principle or blueprint specification.
+
+---
+
+## ❌ "Never Do These" (Strict Ban List)
+
+Never:
+- Alter or replace the existing Next.js frontend UI layout, headers, tabs, or design system.
+- Remove or degrade existing working features (RAG, Telegram alerts, TTS, active learning loop, admin mode).
+- Invent APIs, schemas, environment variables, endpoints, or tenant configs without authoritative source definitions.
+- Write hardcoded `if client == "..."` logic anywhere in Python code.
+- Wrap failing logic in silent `try/except: pass` or return dummy empty fallbacks.
+- Skip verification commands or leave TODO implementations.
+- Rename large directories, reformat unrelated files, or modify package lockfiles unless explicitly requested.
+- Hardcode secrets, log tokens/API keys, or disable validation/authentication.
+- Introduce N+1 database queries, blocking I/O on main loops, or redundant vector searches.
+
+---
+
+## 🔒 Minimal Change Principle
+- Prefer modifying existing code over creating new abstractions.
+- Only create new modules or files when strictly necessary. Avoid spawning unnecessary files.
+
+---
+
+## ✅ Definition of "DONE"
+
+A task is complete ONLY if:
+- [ ] Code compiles and builds cleanly without warnings.
+- [ ] Existing Next.js frontend UI is 100% preserved and functional.
+- [ ] 100% of existing capabilities (Hybrid RAG, Telegram active learning, TTS, 3-way chat) remain fully intact.
+- [ ] Verification tests pass (`python3 -m unittest discover -s tests`).
+- [ ] Both `aruncore_master_blueprint.md` AND `understanding_each_folder/` guides are fully updated.
+- [ ] No Golden Architecture Principles are violated.
+- [ ] No dead code, commented-out blocks, or unused imports remain.
+
+---
+
+## 📋 Pre-Completion Review Checklist
+
+Before marking any task complete, verify:
+- [ ] **Frontend Preserved**: Existing UI layout, tabs (`Chat`, `Projects`, `Manifesto`), design tokens, and components are 100% intact.
+- [ ] **Features Retained**: Zero feature removals (Hybrid RAG, Telegram alerts, TTS, active learning loop preserved).
+- [ ] **Architecture preserved**: Zero `if client == ...` checks, clean backend decoupling.
+- [ ] **Minimal diff**: No unrelated files edited or reformatted.
+- [ ] **Type safety**: Pydantic validation used in backend; explicit TypeScript interfaces used in frontend.
+- [ ] **Documentation synced**: Blueprint and folder guides match the diff.
+- [ ] **Tests verified**: Test suite passed with zero errors.

Dockerfile

@@ -0,0 +1,28 @@
+FROM python:3.11-slim
+
+# Set working directory
+WORKDIR /app
+
+# Install system dependencies & Node.js for frontend static export
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    curl \
+    && curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
+    && apt-get install -y nodejs \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements and install python dependencies
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy the entire project
+COPY . .
+
+# Build frontend static export
+RUN cd frontend && npm install && npm run build
+
+# Expose standard container port
+EXPOSE 8000
+
+# Run FastAPI server (which serves the mounted frontend and API endpoints)
+CMD ["sh", "-c", "if [ \"$RUN_TELEGRAM_PUBLIC_BOT\" = \"true\" ]; then python -m backend.app.core.bot & fi; uvicorn backend.app.main:app --host 0.0.0.0 --port ${PORT:-8000}"]

GOAL.MD

@@ -0,0 +1,112 @@
+# 🎯 GOAL.MD — ArunCore Production Upgrade Roadmap & Master Plan
+
+> **STATUS**: 📜 **MASTER EXECUTION SPECIFICATION**
+> **PROJECT**: **ArunCore Multi-Tenant Enterprise AI Platform**
+
+---
+
+## 📋 Executive Objective & Mission Statement
+
+The primary goal of this project is to upgrade **ArunCore** into a production-grade, domain-generic, multi-tenant enterprise AI platform while adhering strictly to three non-negotiable directives:
+
+1. 🥇 **100% Frontend Preservation**: The existing Next.js frontend UI layout, components, header, tabs (`ChatPanel`, `ProjectsView`, `ManifestoView`), glowing accent borders, light/dark themes, hero assistant card, and design tokens stay **100% INTACT**. No layout rewrites or UI structural replacements will take place.
+2. 🥈 **Zero Functionality Loss (Keep & Enhance Everything)**: Absolutely **NO** working features will be removed from the current app. All existing capabilities—including the Hybrid RAG engine, Telegram active learning alerts, 3-way live human chat presence notice, TTS neural voice endpoints, admin mode, and session history—will be preserved 100% and upgraded for production-grade scaling!
+3. 🥉 **Production-Grade Hybrid RAG Engine**: The RAG retrieval pipeline (dense ChromaDB vector search + sparse BM25 keyword search + Cohere/LLM reranking) will be enhanced to deliver hyper-fast, accurate, context-aware answers per tenant.
+4. 🏅 **Ultra-Simple Demo Mode (`?tutor=ed_donner` / `?client=ed_donner`)**: Demo mode operates by passing a URL query parameter (`?tutor=ed_donner`). The frontend fetches metadata from `/api/v1/config?tutor=ed_donner` (or `/config?tutor=ed_donner`), which dynamically populates the existing hero card title, role subtitle, avatar, welcome text, suggested questions, and AI system prompt.
+
+---
+
+## 🏛️ The 11 Golden Architecture Principles (NON-NEGOTIABLE)
+
+1. 🥇 **Preserve Current Frontend 100%**: Absolutely DO NOT replace or alter the existing Next.js frontend UI (`frontend/app/page.tsx`, `frontend/components/ChatPanel.tsx`, `ProjectsView.tsx`, `ManifestoView.tsx`, `Header.tsx`). The existing luxury UI, design tokens, glowing accent borders, light/dark themes, hero assistant card, and tab views stay 100% intact!
+2. 🥈 **Zero Functionality Loss (Keep & Enhance)**: DO NOT remove a single working feature from the current app (Hybrid RAG, Telegram active learning alerts, 3-way live human chat presence, TTS neural voice, admin mode, session history). The goal is to keep 100% of existing functionality and upgrade it to production grade!
+3. 🥉 **Production-Grade Hybrid RAG Engine**: Maintain and enhance the hybrid RAG architecture (dense ChromaDB vector search + sparse BM25 keyword search + Cohere/LLM reranking) for ultra-fast, high-precision knowledge retrieval per tenant.
+4. 🏅 **Ultra-Simple Demo System**: Demo mode operates by passing a URL query parameter (`?tutor=ed_donner` or `?client=ed_donner`). The frontend fetches metadata from `/api/v1/config?tutor=ed_donner` (or `/config?tutor=ed_donner`), which dynamically populates the existing hero card title, role subtitle, avatar, welcome text, suggested questions, and AI system prompt. Zero complex slot engines required!
+5. 🏅 **Explicit `tenants/` Directory**: All raw PDFs, markdown files, avatars, logos, and vector databases live in `./tenants/` outside the Git code repo. Keep code repos lightweight (~10MB).
+6. 🏅 **Split Targeted Config**: Tenant configs in `tenants/<id>/config/` are split into 6 targeted JSON files (`brand.json`, `agent.json`, `chat.json`, `voice.json`, `seo.json`, `social.json`). Easy debugging, zero merge conflicts, Pydantic validated.
+7. 🏅 **ZERO Backend Client `if` Statements**: Absolutely NO `if client == "ed":` or `if tenant_id == "hitesh":` statements anywhere in Python code! All tenant logic, prompts, tools, and branding MUST be resolved dynamically by `TenantService`.
+8. 🏅 **Config-Driven Tool Registry**: Client tools are controlled via the `enabled_tools` array in `agent.json` (`["search_courses", "book_calendar", "faq_lookup"]`). `ToolExecutor` dynamically binds enabled tools into the LLM execution loop—zero Python code edits required!
+9. 🏅 **Separation of Tenant Assets**: Static brand images (`tenants/<id>/assets/avatars/`, `tenants/<id>/assets/logos/`) are kept separate from code logic.
+10. 🏅 **Decoupled Single-Responsibility Services**: Maintain small, focused backend services (`PromptBuilder`, `MemoryManager`, `ToolExecutor`, `AgentRunner`, `RAGService`, `NotificationService`, `ActiveLearningService`). Do NOT create monster monolithic service files.
+11. 🏅 **Interface Abstractions**: Use abstract base classes for adapters (`VectorStore`, `StateStore`, `NotificationProvider`). Never hardcode direct dependencies in business logic.
+
+---
+
+## 🛠️ Step-by-Step Execution Plan (Detailed Roadmap)
+
+```
+┌─────────────────────────────────────────────────────────────────────────────┐
+│                       Detailed Step-by-Step Process                         │
+├──────────────┬──────────────┬──────────────┬──────────────┬─────────────────┤
+│ STEP 1       │ STEP 2       │ STEP 3       │ STEP 4       │ STEP 5          │
+│ tenants/     │ Pydantic     │ Decoupled    │ Versioned    │ Testing &       │
+│ Directory &  │ Validation   │ Services     │ API Routers  │ Verification    │
+│ Split Configs│ Schemas      │ Architecture │ (/api/v1/)   │                 │
+└──────────────┴──────────────┴──────────────┴──────────────┴─────────────────┘
+```
+
+### 📍 Step 1: Explicit `tenants/` Directory & Split Targeted Configs
+- Create `./tenants/` root directory structure outside Git.
+- Build `tenants/tenant_starter/` and `tenants/ed_donner/` with 6 split JSON configs:
+  - `brand.json`: Hero title, role subtitle, avatar path, accent color.
+  - `agent.json`: System prompt, guardrails, `enabled_tools` array.
+  - `chat.json`: Welcome message, suggested question chips.
+  - `voice.json`: TTS alloy voice specs.
+  - `seo.json`: Meta title & description.
+  - `social.json`: LinkedIn, X, Udemy links.
+
+### 📍 Step 2: Pydantic Validation Schemas (`backend/app/schemas/`)
+- Create Pydantic models:
+  - `tenant.py`: Validates all 6 tenant JSON configs on load.
+  - `chat.py`: Validates `ChatRequest`, streaming NDJSON chunks, session history.
+  - `webhook.py`: Validates Telegram/WhatsApp active learning replies.
+  - `voice.py`: Validates `TTSRequest` and STT response schemas.
+
+### 📍 Step 3: Decoupled Single-Responsibility Services (`backend/app/services/`)
+- Build decoupled Python services preserving 100% of existing functionality:
+  - `tenant_service.py`: Dynamic loader for `tenants/<id>/config/` with **ZERO backend `if client == ...` statements**.
+  - `prompt_builder.py`: Synthesizes dynamic system prompts, guardrails, bio rules, and 3-way live human chat presence notices.
+  - `memory_manager.py`: Manages rolling chat session window context.
+  - `tool_executor.py`: Dynamic sandbox for tools specified in `agent.json` (`enabled_tools`).
+  - `agent_runner.py`: Handles 100% AI agent control and token-by-token streaming response generation.
+  - `rag_service.py`: Hybrid ChromaDB vector search + BM25 keyword search coordinator with reranking.
+  - `notification_service.py`: Async Telegram & WhatsApp alert dispatcher.
+  - `active_learning_service.py`: Real-time Q&A vector re-ingestion service.
+
+### 📍 Step 4: Interface Adapters (`backend/app/db/`)
+- Implement abstract base classes and adapters:
+  - `interfaces.py`: Abstract classes (`VectorStore`, `StateStore`, `NotificationProvider`).
+  - `vectorstore.py`: ChromaDB + BM25 hybrid implementation.
+  - `state_store.py`: Thread-safe session memory store implementation.
+
+### 📍 Step 5: Versioned API Routers (`backend/app/api/v1/`)
+- Implement FastAPI versioned routers:
+  - `router.py`: Master v1 aggregator.
+  - `chat.py`: Streaming `/api/v1/chat`, `/chat/history`, `/chat/human-messages`.
+  - `config.py`: Dynamic metadata endpoint `/api/v1/config` (powers `?tutor=ed_donner`).
+  - `webhook.py`: Telegram & WhatsApp active learning reply webhooks.
+  - `voice.py`: `/api/v1/voice/tts` neural audio generation.
+
+### 📍 Step 6: Frontend Metadata Wiring & Demo Mode Integration
+- Connect Next.js frontend (`frontend/app/page.tsx`) to fetch metadata from `/api/v1/config?tutor=ed_donner`.
+- Dynamically update hero title, role subtitle, avatar, welcome text, suggested questions, and AI prompt while **preserving 100% of the existing luxury UI layout, tabs, and components**.
+
+### 📍 Step 7: Automated Test Suite & Verification (`tests/`)
+- Create test suite in `tests/`:
+  - `test_schemas.py`: Verifies Pydantic schema validation.
+  - `test_tenant_service.py`: Verifies dynamic tenant loading.
+  - `test_chat_api.py`: Verifies streaming responses and endpoint contracts.
+- Gather concrete empirical runtime verification.
+
+---
+
+## 🎯 Verification Criteria for Success (Definition of "DONE")
+
+- [ ] Existing Next.js frontend UI layout, components, tabs, and design system are 100% preserved.
+- [ ] 100% of current functionality (Hybrid RAG, Telegram active learning alerts, TTS, 3-way live human chat, admin mode) remains fully functional.
+- [ ] Hybrid RAG retrieval (ChromaDB + BM25 + Cohere reranking) is enhanced and production-ready per tenant.
+- [ ] Demo mode (`?tutor=ed_donner`) dynamically populates hero card metadata and AI system prompt.
+- [ ] Backend contains ZERO `if client == "..."` checks anywhere in Python code.
+- [ ] `tenants/` storage structure holds isolated client data outside Git code repo.
+- [ ] All automated tests pass (`python3 -m unittest discover -s tests`).
+- [ ] `aruncore_master_blueprint.md` and `AGENTS.md` match the diff.

README.md

@@ -0,0 +1,149 @@
+---
+title: ArunCore Enterprise AI Platform & Digital Twin System
+emoji: 🧠
+colorFrom: green
+colorTo: blue
+sdk: docker
+app_port: 7860
+pinned: false
+---
+
+# 🧠 ArunCore — Domain-Generic Multi-Tenant Enterprise AI Platform & Digital Twin
+
+> **Architecture Spec Version:** ❄️ **v1.0 FROZEN**  
+> **Engine:** `gpt-4.1-nano` | **RAG:** ChromaDB (Dense) + BM25 (Sparse) + Cohere V3 (Reranker)  
+> **Frontend:** Next.js 16 (App Router) Luxury Dark/Light UI  
+> **Backend:** FastAPI Decoupled Microservices  
+> **Evaluation:** Automated 30-Question Multi-Turn ReAct Stress-Test Suite
+
+---
+
+## 🎯 Platform Overview
+
+**ArunCore** is a domain-generic, multi-tenant enterprise AI platform and stateful personal digital twin built for **Arun Yadav** (AI Systems Architect specializing in Healthcare & Education).
+
+The platform serves dual core functions:
+1. **Personal AI Digital Twin**: A 24/7 interactive representative of Arun Yadav, providing deep technical answers about his AI architectures, healthcare/education projects, GitHub repositories, and LinkedIn publications, backed by instant lead capture and dual Telegram alert capabilities.
+2. **Multi-Tenant Enterprise JSON Engine (`?tutor=<tutor_id>`)**: A zero-code multi-tenant advisor system capable of instantly spinning up bespoke 24/7 AI Course Advisors and Sales Representatives for any client or instructor (e.g. Ed Donner) via a single 238-key enterprise JSON schema dictionary.
+
+---
+
+## 🏛️ Core Architecture Principles (Golden Rules)
+
+1. **Preserve Frontend 100%**: Next.js luxury UI (`app/page.tsx`, `ChatPanel.tsx`, `ProjectsView.tsx`, `ManifestoView.tsx`, `Header.tsx`) with glowing accent borders, light/dark themes, hero card, and tab views stays 100% intact.
+2. **Zero Functionality Loss**: Retains Hybrid RAG, active Telegram alerts, 3-way live human chat takeover, TTS neural voice studio, and evaluation harness.
+3. **Production-Grade Hybrid RAG Engine**: ChromaDB dense vector search + BM25 sparse keyword search + Cohere English V3 reranking for high-precision, zero-hallucination retrieval.
+4. **Dynamic JSON Config Engine**: Zero hardcoded `if client == "ed"` logic in Python or TypeScript. 100% of tenant branding, prompts, tools, headers, and catalogs resolve dynamically from JSON dictionaries (`demos/<tutor_id>_enterprise_dictionary.json`).
+5. **Config-Driven Tool Registry**: Client tools are controlled via `enabled_tools` array in config (`["search_arun_knowledge", "get_github_live_data", "notify_arun"]`).
+6. **Decoupled Backend Services**: Monoliths are banned; small single-responsibility services (`PromptBuilder`, `MemoryManager`, `ToolExecutor`, `RAGService`, `TenantService`, `VoiceService`, `ActiveLearningService`).
+
+---
+
+## ⚡ Key Technical Capabilities
+
+### 1. 🤖 7-Iteration Recursive Agentic Loop (`gpt-4.1-nano`)
+- Powered by `gpt-4.1-nano` with up to **7 recursive tool execution turns** (`search_arun_knowledge`, `get_github_live_data`, `notify_arun`) per turn.
+- Executes multi-step technical comparisons, live GitHub commit fetches, and knowledge base retrievals before generating answers.
+
+### 2. 📚 2-Step GitHub & Knowledge Base Retrieval Pipeline
+- Automatically fetches the 3 most recently updated repositories and live commit logs from GitHub API (`get_github_live_data`).
+- Automatically queries local README architectures (`search_arun_knowledge`) for each returned repository name to provide deep technical details and direct clickable URLs.
+
+### 3. 📊 Markdown Table & Direct Project Link Mandate
+- Presents all multi-system comparisons (e.g. MedCoach vs NEET Bot vs Legal RAG) in clean **Markdown Tables**.
+- Embeds direct, clickable GitHub URLs (`https://github.com/neural-arun/<repo>`) directly inside table headers and project descriptions.
+
+### 4. 💼 LinkedIn Insights & Social Engagement CTA
+- Integrates scraped public LinkedIn posts (`data/linkedin/posts.md`) covering AI workforce trends (2026–2036), NEET CBT practice ecosystem, FastAPI Todo API, and BPSC Rank 5 updates.
+- Appends direct clickable LinkedIn post URLs and natural social engagement calls-to-action inviting visitors to like, comment, or share their thoughts.
+
+### 5. 🚨 Dual-Channel Telegram Alerts & 3-Way Live Human Chat Takeover
+- **Active Learning Logger**: Silently logs all chats to `@ai_twin_alert_bot`.
+- **Urgent Lead Alert Bot**: Instantly alerts Arun's phone for hiring leads, unknown questions, or urgent inquiries with a **1-Click Magic Join Link** for real-time 3-way human chat takeover.
+
+---
+
+## 📁 Repository Directory Map
+
+```text
+profile/
+├── backend/                  # Python FastAPI Backend Architecture
+│   └── app/
+│       ├── api/v1/           # API Endpoints (/config, /chat, /voice, /webhook)
+│       ├── core/             # Core Orchestration (agent.py, api.py, bot.py, ingest.py)
+│       ├── schemas/          # Pydantic Schemas (tenant.py, chat.py, voice.py)
+│       ├── services/         # Decoupled Business Logic Services
+│       └��─ main.py           # FastAPI Application Entrypoint
+├── frontend/                 # Next.js 16 (App Router) Luxury UI
+│   ├── app/                  # Page Routes & Global Styles
+│   ├── components/           # UI Components (ChatPanel, Header, ProjectsView, etc.)
+│   └── public/               # Static Avatars & Logos
+├── data/                     # Data Stores & Knowledge Assets (Excluded from Git code bloat)
+│   ├── github/               # Readme files for all 21 public GitHub repos
+│   ├── linkedin/             # Scraped LinkedIn posts (posts.md)
+│   ├── raw/                  # Personal background & unknown questions DB
+│   └── static/               # Public profile, rules of engagement, voice persona
+├── db/                       # ChromaDB Vector Database & Ingestion State
+├── demos/                    # Enterprise 238-Key Monolithic JSON Schemas
+│   ├── ed_donner_enterprise_dictionary.json
+│   ├── general.json          # Master 238-key JSON Template
+│   └── master_enterprise_dictionary.json
+├── scripts/                  # Automated Maintenance & Evaluation Suite
+│   ├── evaluate.py           # 30-Question Multi-Turn ReAct Test Harness
+│   ├── evaluation_questions.md # 30 Test Questions List
+│   ├── evaluation_results.md  # Generated Output & Execution Traces
+│   ├── ingest.py             # ChromaDB Re-Ingestion Script
+│   ├── sync_github.py        # GitHub API Auto-Sync Script
+│   ├── sync_linkedin.py      # LinkedIn Posts Sync Script
+│   └── sync_all.py           # 1-Click Master Data Sync Runner
+├── tenants/                  # Multi-Tenant 6-File JSON Configurations
+├── tests/                    # Automated Unit & Integration Test Suite
+├── AGENTS.md                 # Master Protocol & Execution Rules for AI Coding Agents
+└── aruncore_master_blueprint.md # Complete Architecture Specification Doc
+```
+
+---
+
+## 🚀 Quick Start & Local Execution
+
+### 1. Environment Setup
+Ensure `.env` exists in root with valid API keys:
+```env
+OPENAI_API_KEY=sk-proj-...
+COHERE_API_KEY=...
+GITHUB_USERNAME=neural-arun
+GITHUB_TOKEN=github_pat_...
+```
+
+### 2. Run Backend Server (FastAPI on Port 8000)
+```bash
+source .venv/bin/activate
+python3 -m uvicorn backend.app.main:app --reload --port 8000
+```
+
+### 3. Run Frontend Dev Server (Next.js on Port 3000)
+```bash
+cd frontend
+npm run dev
+```
+
+### 4. Run Automated Evaluation Test Suite
+```bash
+python3 scripts/evaluate.py
+```
+
+---
+
+## 🧪 Verification & Automated Testing
+
+Execute the complete backend unit test suite:
+```bash
+python3 -m unittest discover -s tests
+```
+*Expected Output:* `Ran 9 tests in 0.028s OK`
+
+---
+
+## 📜 License & Copyright
+
+© 2026 **Arun Yadav** ([neural.arun.dev@gmail.com](mailto:neural.arun.dev@gmail.com)). All rights reserved.

aruncore_master_blueprint.md

@@ -0,0 +1,257 @@
+# 🏛️ ArunCore — Master Enterprise AI System Blueprint & Architecture Specification (v1.0 Frozen)
+
+> **STATUS**: ❄️ **ARCHITECTURE v1.0 FROZEN** (Official Release Specification)
+
+## 📋 Executive Overview & System Philosophy
+
+**ArunCore** is a production-grade, stateful, agentic portfolio and multi-tenant enterprise AI engine platform created by **Arun Yadav** (AI Systems Architect specializing in Healthcare & Education).
+
+**ArunCore** operates on a **Domain-Generic Engine Architecture** with **100% Preservation of Arun's Existing Next.js Frontend UI** and **Zero Loss of Existing Capabilities**. The **AI Agent stays in control 100% of the time**. Instead of complex live web page human takeovers, **ArunCore** relies on a high-precision **Hybrid RAG Engine** and an **Active Learning & Direct Messaging Loop (Telegram / WhatsApp)**:
+- **Hybrid RAG Retrieval**: Combines dense vector search (ChromaDB), sparse keyword search (BM25), and Cohere/LLM reranking to deliver hyper-accurate, context-aware answers per tenant.
+- **Active Learning Loop**: When a visitor asks an unknown, personal, weird, or low-confidence question, the AI Twin responds politely and alerts the owner on Telegram/WhatsApp with a 1-Click reply link.
+- **Real-Time Vector Ingestion**: The owner replies directly in Telegram/WhatsApp, automatically ingesting the answer into ChromaDB in real time to train the AI Twin instantly for all future visitors!
+- **3-Way Live Chat Presence**: When the owner joins a live session via Telegram, the AI system prompt dynamically injects a 3-way chat notice acknowledging the human instructor's live presence alongside the AI assistant.
+
+### The Business & Product Model:
+1. **Ultra-Simple Demo Mode (`?tutor=ed_donner` / `?client=ed_donner`)**: Instant data-driven client demos for 2-minute sales pitch videos to potential buyers. Passing `?tutor=ed_donner` in the URL fetches `/api/v1/config?tutor=ed_donner` (or `/config?tutor=ed_donner`), which dynamically updates the hero card title, role subtitle, avatar, welcome text, suggested questions, and AI system prompt on Arun's existing stunning UI!
+2. **Paid Client Onboarding ($300 – $1,000+ Tier)**: Rapid 5-minute onboarding of custom AI twins for tutors, instructors, doctors, lawyers, or consultants, with zero code duplication, zero backend `if client == ...` statements, zero frontend UI changes, and 100% isolated tenant data in `tenants/`.
+
+---
+
+## 🎯 The 11 Senior Systems Architect Principles
+
+```
+  🥇 1. Preserve Current Frontend 🥈 2. Zero Functionality Loss  🥉 3. Production Hybrid RAG   🏅 4. Ultra-Simple Demos
+  (Keep page.tsx, ChatPanel,     (Keep RAG, Telegram alerts,    (ChromaDB + BM25 + Cohere     (?tutor=ed_donner dynamically
+   ProjectsView & Manifesto)      TTS, 3-way live chat notice)   dense/sparse hybrid engine)    updates hero card & prompt)
+
+  🏅 5. Explicit tenants/ Storage 🏅 6. Split Targeted Config   🏅 7. Zero Client Ifs         🏅 8. Config-Driven Tools
+  (PDFs, docs & configs in      (brand, agent, chat, voice,    (100% TenantService            (enabled_tools array in
+   tenants/ outside Git)        seo, social sub-configs)       dynamic resolution)            agent.json controls tools)
+
+  🏅 9. Separate Tenant Assets  🏅 10. Decoupled Backend       🏅 11. Direct Active Learning
+  (tenants/<id>/assets/ logos  (PromptBuilder, ToolExec,      (Owner replies via Telegram,
+   & avatars isolated)          MemoryManager, AgentRunner)    auto-ingesting into RAG DB)
+```
+
+1. **Preserve Current Frontend 100%**: Absolutely DO NOT replace or alter the existing Next.js frontend UI (`frontend/app/page.tsx`, `frontend/components/ChatPanel.tsx`, `ProjectsView.tsx`, `ManifestoView.tsx`, `Header.tsx`). The existing luxury UI, design tokens, glowing accent borders, light/dark themes, hero assistant card, and tab views stay 100% intact!
+2. **Zero Functionality Loss (Keep & Enhance)**: DO NOT remove a single working feature from the current app (Hybrid RAG, Telegram active learning alerts, 3-way live human chat presence, TTS neural voice, admin mode, session history). The goal is to keep 100% of existing functionality and upgrade it to production grade!
+3. **Production-Grade Hybrid RAG Engine**: Maintain and enhance the hybrid RAG architecture (dense ChromaDB vector search + sparse BM25 keyword search + Cohere/LLM reranking) for ultra-fast, high-precision knowledge retrieval per tenant.
+4. **Ultra-Simple Demo System**: Demo mode operates by passing a URL query parameter (`?tutor=ed_donner` or `?client=ed_donner`). The frontend fetches metadata from `/api/v1/config?tutor=ed_donner` (or `/config?tutor=ed_donner`), which dynamically populates the existing hero card title, role subtitle, avatar, welcome text, suggested questions, and AI system prompt. Zero complex slot engines required!
+5. **Explicit `tenants/` Storage**: All raw PDFs, markdown files, avatars, logos, and vector databases live in `./tenants/` outside the Git code repo. Keeps `git clone` lightweight (~10MB), enables fast deployments, self-documents tenant data isolation, and allows seamless S3/Cloudflare R2 sync.
+6. **Split Targeted Config**: Instead of one monolithic 238-key `config.json`, tenant configuration is divided into targeted JSON files (`brand.json`, `agent.json`, `chat.json`, `voice.json`, `seo.json`, `social.json`). Easy debugging, zero merge conflicts, Pydantic validated.
+7. **ZERO Backend Client Logic**: No `if client == "ed":` statements anywhere in Python code! All tenant logic is resolved dynamically by `TenantService`.
+8. **Config-Driven Tool Registry**: Client tool activation is controlled entirely via `enabled_tools` in `agent.json` (`["search_courses", "book_calendar", "faq_lookup"]`). `ToolExecutor` dynamically registers only enabled tools into the LLM execution loop—zero Python code edits required!
+9. **Separation of Tenant Assets & Engine**: Static brand assets (`tenants/<id>/assets/avatars/`, `tenants/<id>/assets/logos/`) are kept separate from code logic.
+10. **Decoupled Single-Responsibility Backend Services**: `AgentService` is split into clean micro-services: `PromptBuilder`, `MemoryManager`, `ToolExecutor`, `AgentRunner`, `RAGService`, `NotificationService`, `ActiveLearningService`.
+11. **Interface Abstractions**: Abstract base classes for persistence and notification providers (`VectorStore`, `StateStore`, `NotificationProvider`). Swapping ChromaDB to Qdrant, Redis to Memory, or Telegram to WhatsApp takes 1 line of config!
+
+---
+
+## 🏗️ End-to-End System Architecture
+
+```mermaid
+graph TD
+    subgraph Client & UI Layer (frontend/ - 100% Preserved UI)
+        ArunVisitor["Arun Portfolio Visitor<br/>(www.neuralarun.in)"]
+        DemoVisitor["Demo / Client Visitor<br/>(?tutor=ed_donner)"]
+    end
+
+    subgraph API Gateway Layer (backend/app/api/v1/)
+        APIRouter["API Router Aggregator (router.py)"]
+        ChatAPI["/api/v1/chat"]
+        ConfigAPI["/api/v1/config"]
+        VoiceAPI["/api/v1/voice"]
+        WebhookAPI["/api/v1/webhook (Telegram / WhatsApp)"]
+    end
+
+    subgraph Business Logic Services (backend/app/services/)
+        TenantService["Tenant & Config Service"]
+        PromptBuilder["Prompt Builder Service"]
+        MemoryManager["Rolling Memory Manager (MemoryManager =<br/>RollingMemory alias)"]
+        ToolExecutor["Dynamic Config-Driven Tool Executor"]
+        AgentRunner["Agent Runner Engine (100% Agent Control)"]
+        KnowledgeService["Knowledge Retrieval Service (GitHub +<br/>READMEs + LinkedIn + static + Q&A)"]
+        RAGService["Hybrid RAG Coordinator (ChromaDB + BM25)"]
+        NotificationService["Telegram / Alert Dispatcher + Background Queue"]
+        SessionService["Thread-Safe Session & Memory Store"]
+        AuthService["Admin Token Auth (3-Way Live Takeover)"]
+        BackgroundQueue["Shared Background Task Queue"]
+        ActiveLearningService["Active Learning (owner-answer ingestion)"]
+    end
+
+    subgraph Interface Adapters (backend/app/db/ & core/)
+        VectorStore["VectorStore Interface<br/>(ChromaDB + BM25 + Cohere)"]
+        StateStore["StateStore Interface<br/>(Redis / Thread-Safe Memory)"]
+        NotificationProvider["NotificationProvider Interface<br/>(Telegram / WhatsApp API)"]
+        CompositionRoot["core/agent.py (thin composition<br/>root: init_agent + facade re-exports)"]
+        HTTPLayer["core/api.py (thin FastAPI wiring only)"]
+    end
+
+    subgraph Tenant Storage Layer (tenants/)
+        TenantConfigs["tenants/<id>/config/<br/>(brand, agent, chat, voice, seo, social)"]
+        TenantAssets["tenants/<id>/assets/<br/>(avatars, logos, graphics)"]
+        TenantKnowledge["tenants/<id>/knowledge/<br/>(markdown, PDFs, raw data, active_learning.json)"]
+    end
+
+    ArunVisitor --> ConfigAPI
+    ArunVisitor --> ChatAPI
+    DemoVisitor --> ConfigAPI
+    DemoVisitor --> ChatAPI
+    
+    APIRouter --> ChatAPI
+    APIRouter --> ConfigAPI
+    APIRouter --> VoiceAPI
+    APIRouter --> WebhookAPI
+
+    ChatAPI --> AgentRunner
+    ConfigAPI --> TenantService
+    WebhookAPI --> ActiveLearningService
+
+    AgentRunner --> PromptBuilder
+    AgentRunner --> MemoryManager
+    AgentRunner --> ToolExecutor
+    AgentRunner --> KnowledgeService
+    AgentRunner --> RAGService
+    AgentRunner --> NotificationService
+    AgentRunner --> SessionService
+    AgentRunner --> CompositionRoot
+
+    ToolExecutor --> KnowledgeService
+    ToolExecutor --> NotificationService
+
+    NotificationService --> BackgroundQueue
+    NotificationService --> AuthService
+    HTTPLayer --> AgentRunner
+    HTTPLayer --> SessionService
+    HTTPLayer --> AuthService
+    HTTPLayer --> NotificationService
+    
+    TenantService --> TenantConfigs
+    ToolExecutor --> TenantConfigs
+    RAGService --> KnowledgeService
+    KnowledgeService --> VectorStore
+    VectorStore --> TenantKnowledge
+    ActiveLearningService --> VectorStore
+    ActiveLearningService --> RAGService
+    NotificationService --> NotificationProvider
+```
+
+---
+
+## 📁 Repository Directory & Tenant Storage Structure
+
+```
+aruncore/
+├── backend/                      # ⚙️ FASTAPI BACKEND SERVICE
+│   ├── app/
+│   │   ├── api/                  # 🌐 VERSIONED API ROUTERS (v1)
+│   │   │   ├── v1/
+│   │   │   │   ├── router.py     # Master v1 API Router Aggregator
+│   │   │   │   ├── config.py     # /api/v1/config multi-tenant metadata resolution
+│   │   │   │   ├── webhook.py    # Telegram & WhatsApp reply webhook (Active Learning)
+│   │   │   │   └── voice.py      # /api/v1/voice/tts neural TTS audio
+│   │   │
+│   │   ├── schemas/              # 📋 PYDANTIC VALIDATION SCHEMAS
+│   │   │   ├── tenant.py         # Sub-config schemas (Brand, Agent, Chat, Voice, SEO, Social)
+│   │   │   ├── chat.py           # ChatRequest, ChatHistoryResponse, NDJSONStreamChunk schemas
+│   │   │   ├── webhook.py        # Telegram/WhatsApp incoming reply webhook schema
+│   │   │   └── voice.py          # TTSRequest schema
+│   │   │
+│   │   ├── services/             # 🧠 SINGLE-RESPONSIBILITY SERVICES (DECOUPLED)
+│   │   │   ├── background.py     # Shared background task queue + worker thread
+│   │   │   ├── tenant_service.py # Dynamic tenant + legacy demos config resolver
+│   │   │   ├── prompt_builder.py # System prompt, static context & avatar assembly
+│   │   │   ├── memory_manager.py # RollingMemory summary-compression + MemoryManager alias
+│   │   │   ├── tool_executor.py  # Real tool registry (search / github / notify) + placeholders
+│   │   │   ├── agent_runner.py   # Streaming + sync agent loop (7-iters, 3-way live notice)
+│   │   │   ├── knowledge_service.py # Reads/writes ALL knowledge data (GitHub, LinkedIn, static, Q&A)
+│   │   │   ├── rag_service.py    # Hybrid RAG coordinator + active-learning persistence
+│   │   │   ├── notification_service.py # Telegram send/alert/queue/logging + alert dedup
+│   │   │   ├── auth_service.py   # Admin token generation + verification
+│   │   │   ├── session_store.py  # Thread-safe session / human-control / memory store
+│   │   │   ├── active_learning_service.py # owner-reply -> RAG ingestion
+│   │   │   └── voice_service.py  # Speech synthesis (TTS)
+│   │   │
+│   │   ├── db/                   # 🔌 INTERFACE ADAPTERS
+│   │   │   ├── interfaces.py     # Abstract Base Classes (VectorStore, StateStore, NotificationProvider)
+│   │   │   └── state_store.py    # Thread-Safe In-Memory StateStore Implementation
+│   │   │
+│   │   ├── core/                 # 🔧 COMPOSITION ROOT, HTTP LAYER & CHANNELS
+│   │   │   ├── agent.py          # init_agent factory + IPv4 patch + backward-compat facade
+│   │   │   ├── api.py            # Thin FastAPI wiring (/chat, /config, /tts, admin, health)
+│   │   │   ├── bot.py            # Telegram bot (uses shared AgentRunner.sync_reply)
+│   │   │   └── ingest.py         # ChromaDB knowledge ingestion (data/ -> db/)
+│   │   │
+│   │   └── main.py               # FastAPI entrypoint (mounts v1 routers)
+│   │
+│   ├── pyproject.toml
+│   └── Dockerfile
+│
+├── frontend/                     # 🎨 NEXT.JS 16 FRONTEND WEB APP (100% PRESERVED)
+│   ├── app/
+│   │   ├── page.tsx              # Main App Router (Arun's Portfolio + Dynamic Tenant Metadata Loader)
+│   │   ├── layout.tsx            # Global layout provider
+│   │   └── globals.css           # Global design tokens & CSS root variables
+│   │
+│   ├── components/
+│   │   ├── ChatPanel.tsx         # Primary Chat Panel & Hero Assistant Card (100% Preserved)
+│   │   ├── ProjectsView.tsx      # Projects Portfolio View (100% Preserved)
+│   │   ├── ManifestoView.tsx     # Manifesto View (100% Preserved)
+│   │   ├── Header.tsx            # Header bar & navigation (100% Preserved)
+│   │   └── Sidebar.tsx           # Sidebar navigation (100% Preserved)
+│   │
+│   ├── hooks/                    # Custom React Hooks
+│   ├── lib/                      # Types & API helpers
+│   ├── package.json
+│   └── tsconfig.json
+│
+├── tenants/                      # 📦 EXPLICIT TENANT DATA (OUTSIDE GIT REPO / S3 BUCKET)
+│   ├── ed_donner/                # Ed Donner Tenant Package
+│   │   ├── config/               # 📄 SPLIT CONFIG FILES
+│   │   │   ├── brand.json        # Hero title, subtitle, colors, avatar path
+│   │   │   ├── agent.json        # System prompt, guardrails, enabled_tools array
+│   │   │   ├── chat.json         # Quick questions, welcome message
+│   │   │   ├── voice.json        # TTS alloy voice specs
+│   │   │   ├── seo.json          # Meta title & description
+│   │   │   └── social.json       # Udemy, LinkedIn, X, website links
+│   │   ├── assets/               # 🖼️ STATIC ASSETS (avatars, logos, graphics)
+│   │   │   ├── avatar.png
+│   │   │   └── logo.png
+│   │   └── knowledge/            # 📚 RAW KNOWLEDGE (markdown, PDFs, active_learning.json)
+│   │       ├── courses.md
+│   │       └── active_learning.json
+│   │
+│   ├── tenant_starter/           # Quick starter template for instant 1-min onboarding
+│   └── vector_db/                # Persistent ChromaDB sqlite & vector indexes
+│
+├── data/                         # 📹 DEMO DATA & GLOBAL PROFILES
+│   └── static/                   # Arun's static profile & rules of engagement
+│
+├── docs/                         # Operational SOPs & Architecture Playbooks
+├── scripts/                      # Vector ingestion (`ingest_knowledge.py`), evaluation scripts
+├── tests/                        # Automated unit & integration test suites
+├── docker-compose.yml            # Multi-container orchestration (Backend + Redis)
+├── Makefile                      # Developer workflow commands
+└── README.md                     # Master Repository Overview
+```
+
+---
+
+## 🎯 Final Master Checklist
+
+- [x] **100% Frontend Preservation**: Existing Next.js frontend UI (`page.tsx`, `ChatPanel.tsx`, `ProjectsView.tsx`, `ManifestoView.tsx`, `Header.tsx`) is preserved 100% with zero layout changes!
+- [x] **Zero Functionality Loss**: 100% of existing features (Hybrid RAG, Telegram active learning, TTS, 3-way live human chat notice, admin mode) are retained and upgraded.
+- [x] **Production-Grade Hybrid RAG Engine**: ChromaDB dense vector + BM25 sparse keyword search + Cohere/LLM reranking.
+- [x] **Ultra-Simple Demo Mode**: Query param `?tutor=ed_donner` dynamically populates hero card title, role subtitle, avatar, welcome text, suggested questions, and AI system prompt.
+- [x] **100% AI Agent Control**: Web chat is always handled by the AI Twin. Zero complex live web page human takeovers needed.
+- [x] **Direct Active Learning Loop**: Owner replies directly in Telegram or WhatsApp to train their AI Twin in real time.
+- [x] **Split Targeted Config**: 6 targeted JSON files (`brand`, `agent`, `chat`, `voice`, `seo`, `social`).
+- [x] **Explicit `tenants/` Directory**: External tenant data (`tenants/<id>/config`, `assets`, `knowledge`) isolated outside Git code repo.
+- [x] **ZERO Client `if` Statements**: 100% dynamic `TenantService` resolution in backend.
+- [x] **Config-Driven Tool Registry**: `agent.json` controls `enabled_tools` array.
+- [x] **Separation of Tenant Assets**: `tenants/<id>/assets/` separate from code.
+- [x] **Single-Responsibility Services**: Decoupled `PromptBuilder`, `MemoryManager`, `ToolExecutor`, `AgentRunner`.
+- [x] **Interface Abstractions**: Abstract classes for `VectorStore`, `StateStore`, and `NotificationProvider`.
+- [x] **Domain-Generic Backend**: Backend engine only understands `Tenant`, `Conversation`, `Knowledge`, `Tool`, `Active Learning`, `Voice`.
+- [x] **Self-Documenting Naming**: Explicit module and directory names throughout.

backend/README.md

@@ -0,0 +1,64 @@
+# ⚙️ ArunCore Backend Architecture (`backend/`)
+
+> **Architecture Spec:** Decoupled Single-Responsibility Services  
+> **Framework:** FastAPI / Uvicorn  
+> **LLM Engine:** `gpt-4.1-nano` (via LangChain `ChatOpenAI`)  
+> **RAG Engine:** ChromaDB + BM25 + Cohere English V3 Reranker  
+
+---
+
+## 📁 Backend Directory Map
+
+```text
+backend/app/
+├── api/v1/                   # REST API Endpoints & Webhooks
+│   ├── config.py             # GET /api/v1/config (238-key enterprise JSON resolver)
+│   ├── router.py             # Main API Router
+│   ├── voice.py              # POST /api/v1/voice/speak & audio STT endpoint
+│   └── webhook.py            # POST /api/v1/webhook (Telegram live takeover webhook)
+├── core/                     # Agent Orchestration Core
+│   ├── agent.py              # LLM init, search_arun_knowledge, get_github_live_data, notify_arun
+│   ├── api.py                # Legacy API handler compatibility layer
+│   ├── bot.py                # Telegram bot delivery helpers
+│   ├── evaluate.py           # Core evaluation functions
+│   └── ingest.py             # Local vector store ingestion module
+├── schemas/                  # Pydantic Schemas & Data Contracts
+│   ├── chat.py               # ChatRequest, ChatResponse
+│   ├── tenant.py             # TenantConfig schema
+│   ├── voice.py              # VoiceSpeakRequest, VoiceSpeakResponse
+│   └── webhook.py            # TelegramWebhookPayload
+├── services/                 # Decoupled Business Logic Services
+│   ├── active_learning_service.py # Unknown question logger & memory saver
+│   ├── memory_manager.py     # RollingMemory summary compression engine
+│   ├── notification_service.py # Telegram alert queue & delivery service
+│   ├── prompt_builder.py     # Dynamic System Prompt assembler
+│   ├── rag_service.py        # Hybrid Vector Search + BM25 + Reranker pipeline
+│   ├── tenant_service.py     # Dynamic JSON config loader (Zero backend `if` checks)
+│   ├── tool_executor.py      # Config-driven tool binder & execution loop
+│   └── voice_service.py      # OpenAI tts-1 audio synthesizer
+└── main.py                   # FastAPI Application Entrypoint
+```
+
+---
+
+## 🔌 Primary API Endpoints
+
+### 1. `GET /api/v1/config?tutor=<tutor_id>`
+- Resolves tenant configuration dynamically from `demos/<tutor_id>_enterprise_dictionary.json`.
+- Returns full 238-key JSON payload including `theme_design_system`, `frontend_ui_dictionary`, `courses`, and `custom_system_prompt`.
+
+### 2. `POST /chat` / `POST /api/v1/chat`
+- Accepts user messages and executes the 7-iteration ReAct tool loop (`search_arun_knowledge`, `get_github_live_data`, `notify_arun`).
+- Streams or returns the final response along with tool execution traces.
+
+### 3. `POST /api/v1/voice/speak`
+- Synthesizes text into natural studio neural speech audio using OpenAI `tts-1` (`alloy` voice).
+
+---
+
+## 🧪 Testing Backend Services
+
+Run all backend test cases:
+```bash
+python3 -m unittest discover -s tests
+```

backend/app/api/v1/config.py

@@ -0,0 +1,13 @@
+from typing import Optional
+from fastapi import APIRouter
+from backend.app.services.tenant_service import tenant_service
+
+router = APIRouter()
+
+
+@router.get("/config")
+@router.get("/api/v1/config")
+async def get_tutor_config_endpoint(tutor: Optional[str] = None):
+    """Dynamic multi-tenant metadata resolution endpoint powering 1-click demos (?tutor=ed_donner)."""
+    full_cfg = tenant_service.load_tenant_config(tutor)
+    return full_cfg.to_legacy_dict()

backend/app/api/v1/router.py

@@ -0,0 +1,9 @@
+from fastapi import APIRouter
+from backend.app.api.v1.config import router as config_router
+from backend.app.api.v1.voice import router as voice_router
+from backend.app.api.v1.webhook import router as webhook_router
+
+api_v1_router = APIRouter()
+api_v1_router.include_router(config_router)
+api_v1_router.include_router(voice_router)
+api_v1_router.include_router(webhook_router)

backend/app/api/v1/voice.py

@@ -0,0 +1,20 @@
+import io
+from fastapi import APIRouter, HTTPException
+from fastapi.responses import StreamingResponse
+from backend.app.schemas.voice import TTSRequest
+from backend.app.services.voice_service import VoiceService
+
+router = APIRouter()
+
+
+@router.post("/tts")
+@router.post("/api/v1/voice/tts")
+async def tts_endpoint(req: TTSRequest):
+    if not req.text.strip():
+        raise HTTPException(status_code=400, detail="Text snippet cannot be empty.")
+
+    try:
+        audio_bytes = VoiceService.generate_tts_audio(req.text, voice=req.voice or "alloy")
+        return StreamingResponse(io.BytesIO(audio_bytes), media_type="audio/mpeg")
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))

backend/app/api/v1/webhook.py

@@ -0,0 +1,20 @@
+from fastapi import APIRouter, HTTPException
+from backend.app.schemas.webhook import ActiveLearningWebhookPayload
+from backend.app.services.active_learning_service import ActiveLearningService
+
+router = APIRouter()
+
+
+@router.post("/webhook/telegram")
+@router.post("/api/v1/webhook")
+async def telegram_webhook_endpoint(payload: ActiveLearningWebhookPayload):
+    try:
+        active_learning = ActiveLearningService(tenant_id=payload.tutor_id or "arun")
+        success = active_learning.process_incoming_owner_reply(
+            session_id=payload.session_id,
+            question=payload.user_question,
+            answer=payload.owner_answer,
+        )
+        return {"status": "success" if success else "failed", "session_id": payload.session_id}
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))

backend/app/core/README.md

@@ -0,0 +1,52 @@
+# ⚙️ Core Engine (`/core/`)
+
+This folder contains the **Python backend engine** for ArunCore. It handles the FastAPI API server, AI reasoning loop (`gpt-4.1-nano`), vector database search, OpenAI studio voice synthesis, real-time GitHub data fetching, **100% automated Telegram notifications**, a **1-Click Magic Link 3-Way Real Human Takeover Engine**, a **Deterministic Human Control State Machine**, and the **Active Learning Memory Loop**.
+
+---
+
+## 📁 Files Overview & Descriptions
+
+### 1. 🌐 `core/api.py` — FastAPI Web Server & 3-Way Live Chat Engine
+- Runs the async HTTP server (`port 8000` locally, `port 7860` on Hugging Face Spaces).
+- **Key Endpoints**:
+  - `POST /chat`: Streams real-time AI responses token-by-token (NDJSON) with execution trace thoughts. Unconditionally queues automated Telegram chat alerts for EVERY visitor message.
+  - `GET /chat/history`: Returns full central session chat transcript (`SESSION_CHAT_STORE`) so both visitor and Real Arun see the exact same 3-way conversation history in real time.
+  - `POST /chat/human-message`: Receives live messages sent by Real Arun from the Admin Reply Bar or Telegram commands (`/answer`, `/release`). Automatically pairs Real Arun's answer with the visitor's question, appends to `data/raw/unknown_questions.json`, and triggers background ChromaDB vector DB re-ingestion!
+  - `GET /chat/verify-admin-token`: Validates secure HMAC admin tokens generated for 1-click Telegram magic join links.
+  - `POST /tts`: Converts AI text into HD neural studio voice using OpenAI `tts-1` (`alloy` voice).
+  - `GET /health`: Health check endpoint showing active sessions, Telegram log statuses, and system uptime.
+
+---
+
+### 2. 🧠 `core/agent.py` — AI Persona, Agentic Loop & Tools
+- Configures the main LLM (`gpt-4.1-nano`), persona system prompt, and tool functions.
+- **Dynamic 100% Language Matching**: Responds in 100% articulate English for English queries with zero Hindi/Hinglish slang leaks. Responds naturally in Hinglish/Hindi when the user speaks Hindi.
+- **3-Way Human Presence Injection**: When Real Arun is active in a session, dynamically injects a system notice containing Real Arun's messages into the prompt context so the AI Twin recognizes Real Arun's presence and co-pilots seamlessly.
+- **Tools**:
+  - `search_arun_knowledge`: Alias-aware project search (resolves `MedCoach` → `med_coach/README.md`), always includes `public_profile.md` and checks `data/raw/unknown_questions.json` for Arun's verified human answers.
+  - `get_github_live_data`: Fetches real-time public repositories and recent commit activity from GitHub (`api.github.com/users/neural-arun/repos`).
+  - `notify_arun`: Sends instant Telegram alerts (`@ai_twin_alert_bot`) for leads or urgent queries.
+- **1-Click Magic Link Generation**: `generate_admin_token(session_id)` & `verify_admin_token(session_id, token)` create tamper-proof magic links (`https://aruncore.vercel.app/?session_id=...&admin_token=...`).
+- **Active Learning**: `save_unknown_question_answer(question, answer)` appends verified Q&A pairs to `data/raw/unknown_questions.json` and triggers background re-ingestion.
+
+---
+
+### 3. 📚 `core/ingest.py` — Vector Database Compiler
+- Scans all markdown and JSON files inside `data/`, chunks them semantically, generates vector embeddings (`text-embedding-3-small`), and compiles into ChromaDB (`db/`).
+- **Smart JSON Parsing**: Specially parses `data/raw/unknown_questions.json` — each Q&A pair becomes its own searchable vector chunk.
+
+---
+
+### 4. 🤖 `core/bot.py` — Public Telegram Bot Service & Active Learning Handler
+- Runs a standalone Telegram bot allowing visitors to interact with Arun's AI Assistant directly in Telegram.
+- **Active Learning Reply Handler**: When Arun replies to an alert message in Telegram:
+  1. Extracts original user question from alert message.
+  2. Pairs with Arun's reply as verified answer.
+  3. Saves to `data/raw/unknown_questions.json`.
+  4. Triggers background ChromaDB re-ingestion.
+  5. Sends Arun confirmation: *"✅ Answer Saved & Ingested into AI Memory!"*
+
+---
+
+### 5. 🧪 `core/evaluate.py` — Evaluation & Benchmarking Engine
+- Stress-tests benchmark evaluations against test sets to measure retrieval precision and generation quality.

backend/app/core/agent.py

@@ -0,0 +1,136 @@
+"""Agent composition root + backwards-compatible facade.
+
+The heavy lifting previously in this file now lives in single-responsibility
+services (notification, knowledge, memory, prompt, tools, tenant). This module
+only: (1) applies the global IPv4-only DNS patch once, (2) exposes `init_agent`
+as the small factory that wires an LLM + bound tools + chat prompt + memory,
+and (3) re-exports every public symbol the API, Telegram bot, eval scripts,
+and tests still import from `backend.app.core.agent`.
+"""
+import os
+import socket
+from typing import Any, Dict, Optional, Tuple
+
+# Force IPv4-only DNS resolution (Hugging Face Spaces / dual-stack hosts can
+# stall IPv6 lookups). Applied once at import time, mirroring the legacy boot.
+try:
+    _orig_getaddrinfo = socket.getaddrinfo
+
+    def _ipv4_only_getaddrinfo(*args, **kwargs):
+        res = _orig_getaddrinfo(*args, **kwargs)
+        ipv4_res = [r for r in res if r[0] == socket.AF_INET]
+        return ipv4_res or res
+
+    socket.getaddrinfo = _ipv4_only_getaddrinfo
+except Exception:
+    pass
+
+from dotenv import load_dotenv
+from langchain_openai import ChatOpenAI
+
+from backend.app.services.memory_manager import RollingMemory, MemoryManager
+from backend.app.services.prompt_builder import PromptBuilder
+from backend.app.services.tool_executor import (
+    ToolExecutor,
+    search_arun_knowledge,
+    get_github_live_data,
+    notify_arun,
+)
+from backend.app.services.tenant_service import tenant_service, TenantService
+from backend.app.services.knowledge_service import knowledge_service
+from backend.app.services.notification_service import (
+    queue_debug_event,
+    queue_maybe_notify_arun,
+    queue_chat_history_to_telegram,
+    queue_automated_chat_alert,
+    send_automated_chat_alert,
+    send_chat_history_to_telegram,
+    send_debug_event_to_telegram,
+    schedule_notify_arun,
+    TELEGRAM_DELIVERY_LOGS,
+    _deliver_notify_arun,
+)
+from backend.app.services.auth_service import generate_admin_token, verify_admin_token
+from backend.app.services.agent_runner import AgentRunner, agent_runner, run_pre_escalation
+
+load_dotenv()
+
+__all__ = [
+    "init_agent",
+    "RollingMemory",
+    "MemoryManager",
+    "load_static_context",
+    "load_tutor_config",
+    "save_unknown_question_answer",
+    "queue_debug_event",
+    "queue_maybe_notify_arun",
+    "run_pre_escalation",
+    "queue_chat_history_to_telegram",
+    "queue_automated_chat_alert",
+    "send_automated_chat_alert",
+    "send_chat_history_to_telegram",
+    "send_debug_event_to_telegram",
+    "schedule_notify_arun",
+    "generate_admin_token",
+    "verify_admin_token",
+    "TELEGRAM_DELIVERY_LOGS",
+    "search_arun_knowledge",
+    "get_github_live_data",
+    "notify_arun",
+    "ToolExecutor",
+    "AgentRunner",
+    "agent_runner",
+]
+
+
+def init_agent(
+    temperature: float = 0.4,
+    model_name: Optional[str] = None,
+    tutor_id: Optional[str] = None,
+):
+    """Build a ready-to-run ArunCore agent.
+
+    Returns the same 4-tuple as the legacy factory:
+    (main_llm, chat_prompt, memory, tools).
+    """
+    openai_key = os.getenv("OPENAI_API_KEY")
+    if not openai_key:
+        raise ValueError("OPENAI_API_KEY is not set in environment.")
+
+    resolved_model = model_name or os.getenv("OPENAI_MODEL", "gpt-4.1-nano")
+
+    tools = ToolExecutor.get_enabled_tools(ToolExecutor.DEFAULT_TOOLS)
+    main_llm = ChatOpenAI(
+        temperature=temperature,
+        model=resolved_model,
+        api_key=openai_key,
+    ).bind_tools(tools)
+
+    system_prompt = PromptBuilder().build_system_prompt(tutor_id=tutor_id)
+    prompt = PromptBuilder.build_chat_prompt(system_prompt)
+
+    summary_llm = ChatOpenAI(
+        temperature=0.0,
+        model="gpt-4o-mini",
+        api_key=openai_key,
+    )
+    memory = RollingMemory(summary_llm=summary_llm)
+
+    return main_llm, prompt, memory, tools
+
+
+def load_tutor_config(tutor_id: Optional[str]) -> Optional[Dict[str, Any]]:
+    """Legacy demos-dictionary loader (kept for external consumers)."""
+    return tenant_service.load_legacy_tutor_config(tutor_id)
+
+
+def save_unknown_question_answer(question: str, answer: str) -> str:
+    """Persists a verified Q&A pair into the active-learning store."""
+    return knowledge_service.save_verified_answer(question, answer)
+
+
+def load_static_context():
+    """Canonical 5-tuple static context reader (system, guardrails, handoff, profile, rules)."""
+    from backend.app.services.prompt_builder import load_static_context as _load_5
+
+    return _load_5()

backend/app/core/api.py

@@ -0,0 +1,207 @@
+"""ArunCore FastAPI HTTP layer.
+
+This file only wires HTTP: request validation, routing, CORS, and static file
+serving. Every piece of business logic lives in single-responsibility services
+and is delegated here (agent runner loops, session store, notification, voice,
+tenant config, auth, knowledge persistence).
+"""
+import os
+import json
+import io
+from typing import Optional
+
+from dotenv import load_dotenv
+from fastapi import FastAPI, HTTPException
+from pydantic import BaseModel
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import StreamingResponse
+
+from backend.app.core.agent import init_agent, verify_admin_token
+from backend.app.services.session_store import session_store
+from backend.app.services.agent_runner import agent_runner
+from backend.app.services.knowledge_service import knowledge_service
+from backend.app.services.tenant_service import tenant_service
+from backend.app.services.notification_service import TELEGRAM_DELIVERY_LOGS
+
+load_dotenv()
+
+app = FastAPI(title="ArunCore API", description="Stateful Agentic Backend for Arun Yadav's Digital Twin.")
+
+# Enable CORS for external frontends (Vercel, custom domains, local dev)
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_origin_regex=r"https://.*\.vercel\.app",
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# Initialize the ArunCore engine once (mirrors legacy boot).
+try:
+    print("Initializing ArunCore API Backend...")
+    _, _, _, tools = init_agent()
+    global_tool_map = {t.name: t for t in tools}
+    print("API Backend Initialized Successfully.")
+except Exception as e:
+    print(f"Failed to initialize backend: {e}")
+    raise e
+
+
+class ChatRequest(BaseModel):
+    session_id: str
+    message: str
+    tutor_id: Optional[str] = None
+
+
+class TTSRequest(BaseModel):
+    text: str
+    voice: Optional[str] = "alloy"
+
+
+class HumanMessageRequest(BaseModel):
+    session_id: str
+    admin_token: str
+    message: str
+
+
+@app.get("/api/config")
+@app.get("/config")
+async def get_tutor_config_endpoint(tutor: Optional[str] = None):
+    cfg = tenant_service.load_legacy_tutor_config(tutor)
+    if not cfg:
+        return {"tutor_id": tutor or "arun", "config": None}
+    cfg["tutor_id"] = tutor or "arun"
+    return cfg
+
+
+@app.post("/chat")
+async def chat_endpoint(req: ChatRequest):
+    if not req.message.strip():
+        raise HTTPException(status_code=400, detail="Message cannot be empty.")
+
+    async def generate_response():
+        async for chunk in agent_runner.stream_chat(
+            session_id=req.session_id,
+            message=req.message,
+            tutor_id=req.tutor_id,
+            tool_map=global_tool_map,
+        ):
+            yield json.dumps(chunk) + "\n"
+
+    return StreamingResponse(generate_response(), media_type="application/x-ndjson")
+
+
+@app.post("/tts")
+async def tts_endpoint(req: TTSRequest):
+    from backend.app.services.voice_service import VoiceService
+
+    if not req.text.strip():
+        raise HTTPException(status_code=400, detail="Text cannot be empty.")
+
+    try:
+        audio_bytes = VoiceService.generate_tts_audio(req.text, voice=req.voice or "alloy")
+        return StreamingResponse(io.BytesIO(audio_bytes), media_type="audio/mpeg")
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+
+
+@app.get("/health")
+def health_check():
+    alert_tok = os.getenv("TELEGRAM_ALERT_BOT_TOKEN", "")
+    bot_tok = os.getenv("TELEGRAM_BOT_TOKEN", "")
+    alert_cid = os.getenv("TELEGRAM_ALERT_CHAT_ID", "")
+    bot_cid = os.getenv("TELEGRAM_CHAT_ID")
+
+    return {
+        "status": "ok",
+        "active_sessions": session_store.liveness,
+        "telegram_alert_bot_preview": f"{alert_tok[:6]}...{alert_tok[-4:]}" if alert_tok else "MISSING",
+        "telegram_alert_chat_id": alert_cid or "MISSING",
+        "telegram_bot_preview": f"{bot_tok[:6]}...{bot_tok[-4:]}" if bot_tok else "MISSING",
+        "telegram_chat_id": bot_cid or "MISSING",
+        "telegram_logs": TELEGRAM_DELIVERY_LOGS[-10:],
+    }
+
+
+@app.post("/chat/human-message")
+async def post_human_message(req: HumanMessageRequest):
+    if not verify_admin_token(req.session_id, req.admin_token):
+        raise HTTPException(status_code=403, detail="Invalid admin token.")
+
+    clean_text = (req.message or "").strip()
+    if not clean_text:
+        raise HTTPException(status_code=400, detail="Message cannot be empty.")
+
+    lowered = clean_text.lower().strip()
+
+    # Command: /release or /resume -> Hand control back to AI Twin
+    if lowered in ("/release", "/resume"):
+        session_store.set_human_control(req.session_id, False)
+        rel_entry = session_store.record_message(
+            req.session_id, "human_arun",
+            "[Handed back auto-response control to AI Twin]",
+            name="Arun Yadav",
+        )
+        return {"status": "success", "entry": rel_entry, "human_control": False}
+
+    # Command: /answer -> Trigger AI Twin to answer reading the 3-way transcript
+    if lowered.startswith("/answer"):
+        session_store.set_human_control(req.session_id, True)
+        extra_prompt = clean_text[7:].strip()
+        cmd_entry = session_store.record_message(
+            req.session_id, "human_arun", f"/answer {extra_prompt}".strip(), name="Arun Yadav"
+        )
+        ai_reply = await agent_runner.trigger_ai_answer(req.session_id, extra_prompt)
+        return {"status": "success", "entry": cmd_entry, "ai_reply": ai_reply, "human_control": True}
+
+    # Real Arun's first human message activates Human Control Mode
+    session_store.set_human_control(req.session_id, True)
+    entry = session_store.record_message(req.session_id, "human_arun", clean_text, name="Arun Yadav")
+    session_store.append_human_message(req.session_id, entry)
+
+    last_user_msg = session_store.get_last_user_message(req.session_id)
+
+    if last_user_msg:
+        try:
+            knowledge_service.save_verified_answer(last_user_msg, clean_text)
+            print(f"[RAG AUTO-UPDATE] Saved Q&A pair to unknown_questions.json & triggered ChromaDB re-ingestion.")
+        except Exception as e:
+            print(f"[RAG AUTO-UPDATE ERROR] Failed to auto-ingest admin answer: {e}")
+
+    memory = session_store.get_or_create_memory(req.session_id)
+    memory.add_interaction(f"[REAL ARUN JOINED LIVE]: {clean_text}", "Acknowledged real Arun input.")
+
+    return {"status": "success", "entry": entry, "human_control": True}
+
+
+@app.get("/chat/history")
+async def get_chat_history(session_id: str):
+    msgs = session_store.get_history(session_id)
+    return {"session_id": session_id, "messages": msgs}
+
+
+@app.get("/chat/human-messages")
+async def get_human_messages(session_id: str):
+    msgs = session_store.get_human_messages(session_id)
+    return {"session_id": session_id, "messages": msgs}
+
+
+@app.get("/chat/verify-admin-token")
+async def verify_admin(session_id: str, admin_token: str):
+    valid = verify_admin_token(session_id, admin_token)
+    return {"valid": valid, "session_id": session_id}
+
+
+# Mount static frontend export if built (for Hugging Face Spaces production deployment)
+frontend_out = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "frontend", "out")
+if os.path.exists(frontend_out):
+    from fastapi.staticfiles import StaticFiles
+
+    app.mount("/", StaticFiles(directory=frontend_out, html=True), name="static_frontend")
+
+if __name__ == "__main__":
+    import uvicorn
+
+    port = int(os.getenv("PORT", "8000"))
+    uvicorn.run("backend.app.main:app", host="0.0.0.0", port=port, reload=False)

backend/app/core/bot.py

@@ -0,0 +1,134 @@
+import asyncio
+import os
+import re
+from dotenv import load_dotenv
+from langchain_openai import ChatOpenAI
+from telegram import Update
+from telegram.ext import Application, CommandHandler, MessageHandler, filters, ContextTypes
+
+# Import the ArunCore engine (composition root + decoupled services)
+from backend.app.core.agent import init_agent
+from backend.app.services.memory_manager import RollingMemory
+from backend.app.services.agent_runner import agent_runner
+from backend.app.services.knowledge_service import knowledge_service
+
+load_dotenv()
+
+# === In-Memory Session Store (telegram chat_id -> RollingMemory) ===
+sessions: dict[int, RollingMemory] = {}
+
+# Initialize the engine once at startup
+print("Initializing ArunCore Telegram Bot...")
+main_llm, prompt, _, tools = init_agent()
+tool_map = {t.name: t for t in tools}
+print("Bot engine ready.")
+
+
+def get_or_create_memory(chat_id: int) -> RollingMemory:
+    """Returns existing memory for this user, or creates a new one."""
+    if chat_id not in sessions:
+        summary_llm = ChatOpenAI(
+            temperature=0.0,
+            model="gpt-4o-mini",
+            api_key=os.getenv("OPENAI_API_KEY"),
+        )
+        sessions[chat_id] = RollingMemory(summary_llm=summary_llm)
+    return sessions[chat_id]
+
+
+def run_agent(chat_id: int, user_message: str) -> str:
+    """Runs the full stateful agent loop via the shared AgentRunner."""
+    memory = get_or_create_memory(chat_id)
+    return agent_runner.sync_reply(
+        session_id=str(chat_id),
+        user_input=user_message,
+        llm=main_llm,
+        prompt=prompt,
+        memory=memory,
+        tool_map=tool_map,
+        user_metadata={"channel": "telegram", "chat_id": chat_id},
+        max_iterations=3,
+    )
+
+
+# === Telegram Handlers ===
+
+async def start_handler(update: Update, context: ContextTypes.DEFAULT_TYPE):
+    welcome = (
+        "Hi! I'm *ArunCore*, the AI digital twin of *Arun Yadav*.\n\n"
+        "Ask me anything about his projects, skills, or background in AI engineering. "
+        "I'm here to give you the real picture."
+    )
+    await update.message.reply_text(welcome, parse_mode="Markdown")
+
+
+def format_for_telegram(text: str) -> str:
+    """Converts LLM Markdown into Telegram-safe HTML."""
+    text = text.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
+    text = re.sub(r'\*\*(.*?)\*\*', r'<b>\1</b>', text)
+    text = re.sub(r'^###?\s+(.+)$', r'\n<b>\1</b>', text, flags=re.MULTILINE)
+    text = re.sub(r'```(?:[a-zA-Z]+)?\n?(.*?)\n?```', r'<pre>\1</pre>', text, flags=re.DOTALL)
+    text = re.sub(r'`([^`]+)`', r'<code>\1</code>', text)
+    text = re.sub(r'^[*-]\s+', '• ', text, flags=re.MULTILINE)
+
+    def link_repl(match):
+        label, url = match.groups()
+        return f'<a href="{url}">{label}</a>'
+
+    text = re.sub(r'\[([^\]]+)\]\(([^)]+)\)', link_repl, text)
+    return text.strip()
+
+
+async def message_handler(update: Update, context: ContextTypes.DEFAULT_TYPE):
+    user_text = update.message.text
+    chat_id = update.effective_chat.id
+
+    # Check if this is a Telegram reply to an Alert message
+    if update.message.reply_to_message:
+        reply_to_text = update.message.reply_to_message.text or update.message.reply_to_message.caption or ""
+
+        extracted_question = ""
+        if "User Query / Details:" in reply_to_text:
+            parts = reply_to_text.split("User Query / Details:")
+            if len(parts) > 1:
+                extracted_question = parts[1].split("Category:")[0].split("Contact:")[0].split("Chat ID:")[0].strip()
+        elif "User Message:" in reply_to_text:
+            parts = reply_to_text.split("User Message:")
+            if len(parts) > 1:
+                extracted_question = parts[1].split("Category:")[0].split("Contact:")[0].split("Chat ID:")[0].strip()
+
+        if not extracted_question and len(reply_to_text) > 5:
+            extracted_question = reply_to_text.split("\n\n")[0].strip()
+
+        if extracted_question:
+            res = knowledge_service.save_verified_answer(extracted_question, user_text)
+            confirmation = (
+                f"<b>✅ Answer Saved & Ingested into AI Memory!</b>\n\n"
+                f"<b>Question:</b> <code>{extracted_question}</code>\n"
+                f"<b>Your Verified Answer:</b>\n{user_text}\n\n"
+                f"<i>Result: {res}</i>"
+            )
+            await update.message.reply_text(confirmation, parse_mode="HTML")
+            return
+
+    await update.message.chat.send_action("typing")
+    reply = await asyncio.to_thread(run_agent, chat_id, user_text)
+
+    html_reply = format_for_telegram(reply)
+    try:
+        await update.message.reply_text(html_reply, parse_mode="HTML")
+    except Exception:
+        await update.message.reply_text(reply)
+
+
+if __name__ == "__main__":
+    token = os.getenv("TELEGRAM_PUBLIC_BOT_TOKEN")
+    if not token:
+        raise ValueError("TELEGRAM_PUBLIC_BOT_TOKEN not set in .env")
+
+    application = Application.builder().token(token).build()
+    application.add_handler(CommandHandler("start", start_handler))
+    application.add_handler(MessageHandler(filters.TEXT & ~filters.COMMAND, message_handler))
+
+    print("ArunCore Telegram Bot is running...")
+    application.run_polling()

backend/app/core/evaluate.py

@@ -0,0 +1,171 @@
+import json
+import os
+import time
+from pathlib import Path
+from dotenv import load_dotenv
+
+# Import the agent logic from core.agent
+from core.agent import init_agent, answer_query
+
+# Load environment variables
+load_dotenv()
+
+BASE_DIR = Path(__file__).resolve().parent.parent
+EVAL_SET_PATH = BASE_DIR / "data" / "test_set" / "eval_set.json"
+REPORT_PATH = BASE_DIR / "data" / "test_set" / "evaluation_report.json"
+DEBUG_DIR = BASE_DIR / "evaluation_debug"
+
+def fuzzy_match(topic, answer):
+    """
+    Check if a topic sounds like it's in the answer.
+    More lenient than strict substring.
+    """
+    topic_clean = topic.lower().strip()
+    answer_clean = answer.lower().strip()
+    
+    # 1. Direct match
+    if topic_clean in answer_clean:
+        return True
+    
+    # 2. Key word subset check (if all significant words of a topic are in the answer)
+    # This helps catch "RAG Pipelines" vs "AI pipelines for RAG"
+    stop_words = {"and", "the", "a", "an", "is", "for", "vs", "to", "of", "with"}
+    words = [w for w in topic_clean.split() if w not in stop_words]
+    
+    if not words: return False
+    
+    matches = sum(1 for w in words if w in answer_clean)
+    # If 75% of the important words are there, count it as a pass
+    if (matches / len(words)) >= 0.75:
+        return True
+        
+    return False
+
+def save_detailed_log(qid, question, answer, chunks, retrieval_pass, missing_topics):
+    """Save a clean markdown file for manual human inspection of this specific interaction."""
+    os.makedirs(DEBUG_DIR, exist_ok=True)
+    filepath = DEBUG_DIR / f"{qid}.md"
+    
+    with open(filepath, "w", encoding="utf-8") as f:
+        f.write(f"# Evaluation Log: {qid}\n\n")
+        f.write(f"## Question\n{question}\n\n")
+        f.write(f"## Status\n")
+        f.write(f"- **Retrieval Mode:** {'PASS' if retrieval_pass else 'FAIL'}\n")
+        f.write(f"- **Generation Mode:** {'PASS' if not missing_topics else 'FAIL'}\n")
+        if missing_topics:
+            f.write(f"- **Missing Topics:** {', '.join(missing_topics)}\n")
+        f.write(f"\n## ArunCore Answer\n{answer}\n\n")
+        f.write(f"## Retrieved Chunks (Final Top 5)\n")
+        for i, doc in enumerate(chunks):
+            f.write(f"### Chunk {i+1} | Source: {doc.metadata.get('source')}\n")
+            f.write(f"```text\n{doc.page_content}\n```\n\n")
+
+def run_evaluation():
+    print("--- ArunCore Dual-Evaluation Pipeline (Fuzzy Match + Rate Limit Handling) ---")
+    
+    # 1. Initialize Agent
+    print("Initializing Agent...")
+    try:
+        vectorstore, bm25_retriever, compressor, llm, prompt = init_agent()
+    except Exception as e:
+        print(f"Failed to initialize agent: {e}")
+        return
+
+    # 2. Load Eval Set
+    if not EVAL_SET_PATH.exists():
+        print(f"Eval set not found at {EVAL_SET_PATH}")
+        return
+        
+    with open(EVAL_SET_PATH, "r", encoding="utf-8") as f:
+        eval_set = json.load(f)
+
+    results = []
+    passed_retrieval = 0
+    passed_generation = 0
+    total = len(eval_set)
+
+    print(f"Starting evaluation of {total} questions...\n")
+
+    for i, test in enumerate(eval_set):
+        qid = test.get("id", f"Q{i}")
+        question = test.get("question")
+        expected_source = test.get("expected_source")
+        expected_topics = test.get("expected_topics", [])
+
+        print(f"[{i+1}/{total}] Evaluating {qid}: {question[:60]}...")
+
+        # Execute Agent
+        try:
+            # We add a delay to satisfy the 10/min Cohere Trial Limit
+            if i > 0:
+                print(f"  (Rate limit cool-down: 6.5s)")
+                time.sleep(6.5)
+            
+            response = answer_query(question, vectorstore, bm25_retriever, compressor, llm, prompt)
+            answer = response["answer"]
+            chunks = response["retrieved_chunks"]
+        except Exception as e:
+            print(f"  Error Querying Agent: {e}")
+            results.append({
+                "id": qid,
+                "status": "ERROR",
+                "error": str(e)
+            })
+            continue
+
+        # --- Layer 1: Retrieval Check ---
+        retrieval_pass = False
+        if expected_source.startswith("static/"):
+            retrieval_pass = True 
+        else:
+            for doc in chunks:
+                source_meta = doc.metadata.get("source", "").lower()
+                if expected_source.lower() in source_meta:
+                    retrieval_pass = True
+                    break
+        
+        if retrieval_pass: passed_retrieval += 1
+
+        # --- Layer 2: Generation Check ---
+        # Fuzzy match for topics
+        missing_topics = []
+        for topic in expected_topics:
+            if not fuzzy_match(topic, answer):
+                missing_topics.append(topic)
+        
+        generation_pass = len(missing_topics) == 0
+        if generation_pass: passed_generation += 1
+
+        # Log detailed human-readable file
+        save_detailed_log(qid, question, answer, chunks, retrieval_pass, missing_topics)
+
+        # Store result in summary list
+        results.append({
+            "id": qid,
+            "retrieval": "PASS" if retrieval_pass else "FAIL",
+            "generation": "PASS" if generation_pass else "FAIL",
+            "missing": missing_topics
+        })
+
+    # 3. Final Report
+    report = {
+        "summary": {
+            "total_questions": total,
+            "retrieval_accuracy": f"{(passed_retrieval/total)*100:.2f}%",
+            "generation_accuracy": f"{(passed_generation/total)*100:.2f}%",
+        },
+        "details": results
+    }
+
+    with open(REPORT_PATH, "w", encoding="utf-8") as f:
+        json.dump(report, f, indent=4)
+
+    print("\n" + "="*40)
+    print("EVALUATION COMPLETE")
+    print(f"Retrieval Accuracy: {report['summary']['retrieval_accuracy']}")
+    print(f"Generation Accuracy: {report['summary']['generation_accuracy']}")
+    print(f"Detailed logs saved to: {DEBUG_DIR}")
+    print("="*40)
+
+if __name__ == "__main__":
+    run_evaluation()

backend/app/core/ingest.py

@@ -0,0 +1,279 @@
+import os
+import json
+import hashlib
+from pathlib import Path
+from typing import Dict, List
+
+from dotenv import load_dotenv
+from langchain_openai import OpenAIEmbeddings
+from langchain_community.vectorstores import Chroma
+from langchain_text_splitters import (
+    MarkdownHeaderTextSplitter,
+    RecursiveCharacterTextSplitter,
+)
+from langchain_core.documents import Document
+
+# Load environment variables from .env file if it exists
+load_dotenv()
+
+BASE_DIR = Path(__file__).resolve().parent.parent
+DATA_DIR = BASE_DIR / "data"
+DB_DIR = BASE_DIR / "db"
+STATE_FILE = DB_DIR / "ingestion_state.json"
+
+# Folders to parse for the Vector DB
+INGEST_DIRS = ["github", "raw", "linkedin", "static"]
+
+
+def get_file_hash(filepath: Path) -> str:
+    with open(filepath, "r", encoding="utf-8") as f:
+        return hashlib.md5(f.read().encode("utf-8")).hexdigest()
+
+
+def load_state() -> Dict[str, Dict]:
+    """
+    Returns state in this normalized format:
+    {
+        "data/static/public_profile.md": {
+            "hash": "...",
+            "chunk_ids": ["static_N/A_public_profile_chunk_0", ...]
+        }
+    }
+
+    Supports old state format too:
+    {
+        "data/static/public_profile.md": "old_md5_hash"
+    }
+    """
+    if not STATE_FILE.exists():
+        return {}
+
+    with open(STATE_FILE, "r", encoding="utf-8") as f:
+        raw_state = json.load(f)
+
+    if isinstance(raw_state, dict) and "files" in raw_state and isinstance(raw_state["files"], dict):
+        normalized = {}
+        for rel_path, entry in raw_state["files"].items():
+            if isinstance(entry, dict):
+                normalized[rel_path] = {
+                    "hash": entry.get("hash", ""),
+                    "chunk_ids": entry.get("chunk_ids", []),
+                }
+        return normalized
+
+    normalized = {}
+    if isinstance(raw_state, dict):
+        for rel_path, value in raw_state.items():
+            if isinstance(value, str):
+                normalized[rel_path] = {
+                    "hash": value,
+                    "chunk_ids": [],
+                }
+            elif isinstance(value, dict):
+                normalized[rel_path] = {
+                    "hash": value.get("hash", ""),
+                    "chunk_ids": value.get("chunk_ids", []),
+                }
+    return normalized
+
+
+def save_state(state: Dict[str, Dict]) -> None:
+    os.makedirs(DB_DIR, exist_ok=True)
+    payload = {
+        "version": 2,
+        "files": state,
+    }
+    with open(STATE_FILE, "w", encoding="utf-8") as f:
+        json.dump(payload, f, indent=4)
+
+
+def process_markdown_file(filepath: Path, base_folder: str, rel_path: str) -> List[Document]:
+    with open(filepath, "r", encoding="utf-8") as f:
+        text = f.read()
+
+    headers_to_split_on = [
+        ("#", "Header 1"),
+        ("##", "Header 2"),
+        ("###", "Header 3"),
+    ]
+    markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
+    md_header_splits = markdown_splitter.split_text(text)
+
+    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1200, chunk_overlap=150)
+    splits = text_splitter.split_documents(md_header_splits)
+
+    safe_name = filepath.stem.replace(" ", "_").replace(".", "_")
+
+    project_name = "N/A"
+    parts = list(filepath.relative_to(DATA_DIR).parts)
+    if "github" in parts and len(parts) > 1:
+        project_name = parts[1]
+
+    for i, split in enumerate(splits):
+        split.metadata["source"] = rel_path
+        split.metadata["folder"] = base_folder
+        split.metadata["project"] = project_name
+        split.metadata["chunk_id"] = f"{base_folder}_{project_name}_{safe_name}_chunk_{i}"
+
+    return splits
+
+
+def process_json_file(filepath: Path, base_folder: str, rel_path: str) -> List[Document]:
+    with open(filepath, "r", encoding="utf-8") as f:
+        try:
+            data = json.load(f)
+        except Exception:
+            data = None
+
+    if filepath.name == "unknown_questions.json" and isinstance(data, list):
+        docs = []
+        for i, item in enumerate(data):
+            if isinstance(item, dict) and "question" in item and "answer" in item:
+                q = item.get("question", "").strip()
+                a = item.get("answer", "").strip()
+                content = f"### Verified Question & Answer\n**Question:** {q}\n**Answer:** {a}"
+                docs.append(
+                    Document(
+                        page_content=content,
+                        metadata={
+                            "source": rel_path,
+                            "folder": base_folder,
+                            "project": "unknown_questions",
+                            "chunk_id": f"unknown_q_{i}",
+                        },
+                    )
+                )
+        if docs:
+            return docs
+
+    with open(filepath, "r", encoding="utf-8") as f:
+        content = f.read()
+
+    project_name = "N/A"
+    parts = list(filepath.relative_to(DATA_DIR).parts)
+    if "github" in parts and len(parts) > 1:
+        project_name = parts[1]
+
+    safe_name = filepath.stem.replace(" ", "_").replace(".", "_")
+    chunk_id = f"{base_folder}_{project_name}_{safe_name}_chunk_0"
+
+    doc = Document(
+        page_content=content,
+        metadata={
+            "source": rel_path,
+            "folder": base_folder,
+            "project": project_name,
+            "chunk_id": chunk_id,
+        },
+    )
+    return [doc]
+
+
+def process_file(filepath: Path, base_folder: str, rel_path: str) -> List[Document]:
+    if filepath.suffix == ".md":
+        return process_markdown_file(filepath, base_folder, rel_path)
+    if filepath.suffix == ".json":
+        return process_json_file(filepath, base_folder, rel_path)
+    return []
+
+
+def main():
+    if not os.getenv("OPENAI_API_KEY"):
+        print("\n[ERROR] OPENAI_API_KEY not found in the environment.")
+        print("Please set it or create a .env file containing OPENAI_API_KEY=your_key_here")
+        return
+
+    print("Initializing ChromaDB and OpenAI Embeddings...")
+    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
+
+    vectorstore = Chroma(
+        collection_name="aruncore_knowledge",
+        embedding_function=embeddings,
+        persist_directory=str(DB_DIR),
+    )
+
+    previous_state = load_state()
+    new_state: Dict[str, Dict] = {}
+
+    docs_to_upsert: List[Document] = []
+    ids_to_upsert: List[str] = []
+    stale_ids_to_delete = set()
+    current_rel_paths = set()
+
+    folders_to_scan = [folder for folder in INGEST_DIRS if (DATA_DIR / folder).is_dir()]
+
+    for folder in folders_to_scan:
+        target_dir = DATA_DIR / folder
+
+        for ext in ["*.md", "*.json"]:
+            for filepath in target_dir.rglob(ext):
+                if "test_set" in filepath.parts:
+                    continue
+
+                rel_path = filepath.relative_to(BASE_DIR).as_posix()
+                current_rel_paths.add(rel_path)
+
+                file_hash = get_file_hash(filepath)
+                previous_entry = previous_state.get(rel_path, {})
+                previous_hash = previous_entry.get("hash")
+                previous_chunk_ids = previous_entry.get("chunk_ids", [])
+
+                is_unchanged = (
+                    previous_hash == file_hash
+                    and isinstance(previous_chunk_ids, list)
+                    and len(previous_chunk_ids) > 0
+                )
+
+                if is_unchanged:
+                    new_state[rel_path] = {
+                        "hash": previous_hash,
+                        "chunk_ids": previous_chunk_ids,
+                    }
+                    continue
+
+                print(f"File changed/new -> Processing: {rel_path}")
+
+                splits = process_file(filepath, folder, rel_path)
+                new_chunk_ids = [doc.metadata["chunk_id"] for doc in splits]
+
+                docs_to_upsert.extend(splits)
+                ids_to_upsert.extend(new_chunk_ids)
+
+                old_ids_set = set(previous_chunk_ids)
+                new_ids_set = set(new_chunk_ids)
+
+                stale_ids_to_delete.update(old_ids_set - new_ids_set)
+
+                new_state[rel_path] = {
+                    "hash": file_hash,
+                    "chunk_ids": new_chunk_ids,
+                }
+
+    deleted_files = set(previous_state.keys()) - current_rel_paths
+    for rel_path in deleted_files:
+        old_chunk_ids = previous_state.get(rel_path, {}).get("chunk_ids", [])
+        if old_chunk_ids:
+            print(f"File deleted -> Removing old chunks: {rel_path}")
+            stale_ids_to_delete.update(old_chunk_ids)
+
+    if docs_to_upsert:
+        print(f"\nUpserting {len(docs_to_upsert)} new/modified chunks into Vector DB...")
+        vectorstore.add_documents(documents=docs_to_upsert, ids=ids_to_upsert)
+        print("Upsert complete.")
+    else:
+        print("\nNo new or modified files to upsert.")
+
+    if stale_ids_to_delete:
+        stale_ids_list = sorted(stale_ids_to_delete)
+        print(f"Deleting {len(stale_ids_list)} stale chunks from Vector DB...")
+        vectorstore.delete(ids=stale_ids_list)
+        print("Stale chunk cleanup complete.")
+    else:
+        print("No stale chunks to delete.")
+
+    save_state(new_state)
+    print("Ingestion sequence complete.")
+
+
+if __name__ == "__main__":
+    main()

backend/app/main.py

@@ -0,0 +1,15 @@
+import os
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from backend.app.api.v1.router import api_v1_router
+from backend.app.core.api import app as legacy_app
+
+app = legacy_app
+
+# Mount versioned API routes
+app.include_router(api_v1_router)
+
+if __name__ == "__main__":
+    import uvicorn
+    port = int(os.getenv("PORT", "8000"))
+    uvicorn.run("backend.app.main:app", host="0.0.0.0", port=port, reload=True)

backend/app/schemas/chat.py

@@ -0,0 +1,30 @@
+from typing import List, Optional, Dict, Any
+from pydantic import BaseModel, Field
+
+
+class ChatRequest(BaseModel):
+    message: str = Field(..., min_length=1, description="User input message text")
+    session_id: str = Field(..., description="Unique session identifier")
+    tutor_id: Optional[str] = Field("arun", description="Optional tenant or tutor ID")
+
+
+class ChatMessageEntry(BaseModel):
+    id: str
+    sender: str
+    name: str
+    text: str
+    timestamp: str
+    thoughts: Optional[List[str]] = None
+
+
+class ChatHistoryResponse(BaseModel):
+    session_id: str
+    messages: List[ChatMessageEntry] = Field(default_factory=list)
+
+
+class NDJSONStreamChunk(BaseModel):
+    type: str = Field(..., description="'status', 'token', 'final', or 'error'")
+    content: Optional[str] = None
+    reply: Optional[str] = None
+    thoughts: Optional[List[str]] = None
+    session_id: Optional[str] = None

backend/app/schemas/tenant.py

@@ -0,0 +1,122 @@
+from typing import List, Optional, Dict, Any
+from pydantic import BaseModel, Field
+
+
+class BrandConfig(BaseModel):
+    tutor_id: str = Field(..., description="Tenant ID matching directory name")
+    name: str = Field("Arun Yadav", description="Tenant full name")
+    title: str = Field("Arun's AI Assistant", description="Hero assistant title")
+    role: str = Field("AI Systems Architect • Healthcare & Education", description="Hero role subtitle")
+    subtitle: Optional[str] = None
+    avatar_url: str = Field("/profile_photo.png", description="Avatar image URL")
+    logo_url: str = Field("/logo.jpg", description="Brand logo URL")
+    primary_color: str = Field("#6366f1", description="Primary brand hex color")
+    accent_color: str = Field("#818cf8", description="Accent brand hex color")
+    theme: str = Field("dark", description="Light or dark theme preference")
+    cta_text: str = Field("Consult Arun", description="CTA button label")
+
+
+class AgentConfig(BaseModel):
+    tutor_id: str
+    system_prompt: str = Field(..., description="Base system prompt")
+    temperature: float = Field(0.3, ge=0.0, le=1.0)
+    model: str = Field("gpt-4o", description="LLM model identifier")
+    enabled_tools: List[str] = Field(default_factory=list, description="Array of enabled tool names")
+    guardrails: List[str] = Field(default_factory=list, description="Safety guardrails")
+
+
+class ChatConfig(BaseModel):
+    tutor_id: str
+    welcome_message: str = Field(..., description="Hero card welcome text")
+    suggested_questions: List[str] = Field(default_factory=list, description="Chips suggested questions")
+
+
+class VoiceConfig(BaseModel):
+    tutor_id: str
+    voice_id: str = Field("alloy", description="TTS voice identifier")
+    model: str = Field("tts-1", description="TTS model")
+    speed: float = Field(1.0, ge=0.25, le=4.0)
+
+
+class SEOConfig(BaseModel):
+    tutor_id: str
+    meta_title: str
+    meta_description: str
+    keywords: List[str] = Field(default_factory=list)
+
+
+class SocialConfig(BaseModel):
+    tutor_id: str
+    website: Optional[str] = None
+    linkedin: Optional[str] = None
+    udemy: Optional[str] = None
+    github: Optional[str] = None
+    twitter: Optional[str] = None
+
+
+class TenantFullConfig(BaseModel):
+    tutor_id: str
+    brand: BrandConfig
+    agent: AgentConfig
+    chat: ChatConfig
+    voice: VoiceConfig
+    seo: SEOConfig
+    social: SocialConfig
+
+    def to_legacy_dict(self) -> Dict[str, Any]:
+        """Maps split JSON configs back to existing frontend expected dictionary structure (100% backward compatible!)."""
+        return {
+            "tutor_id": self.tutor_id,
+            "title": self.brand.title,
+            "subtitle": self.brand.subtitle or self.brand.role,
+            "role": self.brand.role,
+            "name": self.brand.name,
+            "avatar": self.brand.avatar_url,
+            "logo": self.brand.logo_url,
+            "welcome_message": self.chat.welcome_message,
+            "suggested_questions": self.chat.suggested_questions,
+            "cta_text": self.brand.cta_text,
+            "primary_color": self.brand.primary_color,
+            "accent_color": self.brand.accent_color,
+            "system_prompt": self.agent.system_prompt,
+            "enabled_tools": self.agent.enabled_tools,
+            "frontend_ui_dictionary": {
+                "header": {
+                    "profile_name": self.brand.name,
+                    "profile_badge": self.brand.role,
+                    "cta_button": {
+                        "text": self.brand.cta_text,
+                    },
+                },
+                "sidebar": {
+                    "profile_card": {
+                        "name": self.brand.name,
+                        "title": self.brand.role,
+                        "sub_badge": f"{self.brand.name}'s AI Twin",
+                    },
+                    "contact_button": {
+                        "title": self.brand.cta_text,
+                        "subtitle": f"Connect directly with {self.brand.name}",
+                    },
+                },
+                "chat_panel": {
+                    "hero_card": {
+                        "assistant_title": self.brand.title,
+                        "role_subtitle": self.brand.role,
+                        "welcome_paragraph": self.chat.welcome_message,
+                        "cta_button_text": self.brand.cta_text,
+                    },
+                    "input_bar": {
+                        "placeholder": f"Ask {self.brand.name}'s AI Assistant...",
+                    },
+                    "suggested_questions_section": {
+                        "chips": [{"query": q} for q in self.chat.suggested_questions],
+                    },
+                }
+            },
+            "client_metadata": {
+                "tutor_id": self.tutor_id,
+                "avatar_url": self.brand.avatar_url,
+                "logo_url": self.brand.logo_url,
+            }
+        }

backend/app/schemas/voice.py

@@ -0,0 +1,8 @@
+from typing import Optional
+from pydantic import BaseModel, Field
+
+
+class TTSRequest(BaseModel):
+    text: str = Field(..., min_length=1, max_length=2000, description="Text snippet to synthesize into audio")
+    voice: Optional[str] = Field("alloy", description="TTS voice identifier")
+    tutor_id: Optional[str] = Field("arun", description="Optional tenant identifier")

backend/app/schemas/webhook.py

@@ -0,0 +1,15 @@
+from typing import Optional, Dict, Any
+from pydantic import BaseModel, Field
+
+
+class TelegramUpdate(BaseModel):
+    update_id: int
+    message: Optional[Dict[str, Any]] = None
+    callback_query: Optional[Dict[str, Any]] = None
+
+
+class ActiveLearningWebhookPayload(BaseModel):
+    session_id: str
+    user_question: str
+    owner_answer: str
+    tutor_id: Optional[str] = "arun"

backend/app/services/active_learning_service.py

@@ -0,0 +1,18 @@
+import os
+import json
+from typing import Optional, Dict, Any
+from backend.app.services.rag_service import RAGService
+
+
+class ActiveLearningService:
+    def __init__(self, tenant_id: str = "arun"):
+        self.tenant_id = tenant_id
+        self.rag_service = RAGService(tenant_id=tenant_id)
+
+    def process_incoming_owner_reply(self, session_id: str, question: str, answer: str) -> bool:
+        """Processes owner's Telegram/WhatsApp answer, ingests into active_learning.json & ChromaDB."""
+        print(f"[ACTIVE_LEARNING] Processing owner answer for session '{session_id}': Q: '{question}' ➔ A: '{answer}'")
+        
+        # Ingest into vector store
+        success = self.rag_service.add_knowledge_entry(question, answer)
+        return success

backend/app/services/agent_runner.py

@@ -0,0 +1,428 @@
+"""Agent execution loop (the ReAct harness).
+
+Runs the recursive tool-call loop for the ArunCore digital twin: request ->
+optional escalation -> up to N tool iterations -> token-by-token streaming of
+the final synthesis. Also owns the 3-way live human takeover answer trigger.
+This is where the previous `/chat` monolith body now lives, fully reusable by
+the FastAPI layer, the Telegram bot vector, and future channels.
+"""
+import asyncio
+import json
+from typing import Any, AsyncGenerator, Dict, List, Optional
+
+from langchain_core.messages import SystemMessage
+
+from backend.app.services.session_store import SessionStore, session_store
+from backend.app.services.prompt_builder import PromptBuilder
+
+
+def run_pre_escalation(user_input: str, tool_map: dict, user_metadata: Optional[dict] = None, fast: bool = False) -> Dict[str, Any]:
+    """Checks for direct-contact intent that should immediately ping Arun."""
+    lowered = (user_input or "").lower()
+    urgent_keywords = [
+        "hire", "contact", "talk to arun", "call arun", "meet arun",
+        "whatsapp", "urgent", "consult", "project inquiry", "work together",
+    ]
+
+    for kw in urgent_keywords:
+        if kw in lowered:
+            return {"escalate": True, "reason": f"Urgent contact keyword detected: '{kw}'"}
+
+    return {"escalate": False, "reason": ""}
+
+
+class AgentRunner:
+    """Orchestrates agentic streaming responses for a single chat request."""
+
+    def __init__(
+        self,
+        store: Optional[SessionStore] = None,
+        agent_factory: Any = None,
+        max_iterations: int = 7,
+        max_search_limit: int = 7,
+    ):
+        self.store = store or session_store
+        self.agent_factory = agent_factory
+        self.max_iterations = max_iterations
+        self.max_search_limit = max_search_limit
+
+    async def stream_chat(
+        self,
+        session_id: str,
+        message: str,
+        tutor_id: Optional[str] = None,
+        tool_map: Optional[Dict[str, Any]] = None,
+    ) -> AsyncGenerator[Dict[str, Any], None]:
+        """Yields NDJSON-able chunks: {'type': status|token|final|error}."""
+        thoughts: List[str] = []
+        try:
+            from backend.app.services.notification_service import (
+                queue_debug_event,
+                queue_maybe_notify_arun,
+                queue_chat_history_to_telegram,
+                queue_automated_chat_alert,
+            )
+
+            if self.agent_factory is None:
+                from backend.app.core.agent import init_agent
+
+                self.agent_factory = init_agent
+
+            self.store.record_message(session_id, "user", message)
+
+            queue_debug_event(
+                "user_message",
+                message,
+                {"channel": "api", "session_id": session_id, "tutor_id": tutor_id},
+            )
+
+            # 3-way live chat takeover: real Arun is in control, AI Twin pauses.
+            if self.store.is_human_control(session_id):
+                queue_automated_chat_alert(
+                    session_id=session_id,
+                    user_input=message,
+                    assistant_response="[Real Arun is currently in live control of this chat session. AI Twin paused.]",
+                )
+                yield {"type": "status", "content": "🟢 Real Arun is in control of this session. AI Twin paused. Waiting for Real Arun or /answer command..."}
+                yield {
+                    "type": "final",
+                    "reply": "",
+                    "thoughts": ["Real Arun in live control. AI Twin paused."],
+                    "session_id": session_id,
+                }
+                return
+
+            yield {"type": "status", "content": "Analyzing request & retrieving context..."}
+            thoughts.append("Analyzing request & retrieving context...")
+
+            pre_result = run_pre_escalation(message, tool_map or {})
+            if pre_result.get("escalate"):
+                yield {"type": "status", "content": "Triggering instant Telegram alert..."}
+                thoughts.append("Triggering instant Telegram alert...")
+                queue_maybe_notify_arun(
+                    user_input=message,
+                    reason=pre_result.get("reason"),
+                    channel="api",
+                    session_id=session_id,
+                )
+
+            session_llm, session_prompt, _, _ = self.agent_factory(tutor_id=tutor_id)
+            memory = self.store.get_or_create_memory(session_id)
+
+            scratchpad: List[Any] = []
+            iterations = 0
+            search_count = 0
+            final_response = ""
+            executed_tools: List[str] = []
+            retrieved_chunks: List[str] = []
+            github_data: List[str] = []
+
+            while iterations < self.max_iterations:
+                messages = session_prompt.format_messages(
+                    running_summary=memory.running_summary,
+                    chat_history=memory.get_messages(),
+                    input=message,
+                    agent_scratchpad=scratchpad,
+                )
+
+                # Inject dynamic Real Human Presence notice if the real human is active.
+                arun_human_msgs = self.store.get_human_entries(session_id)
+                if arun_human_msgs:
+                    messages = PromptBuilder.inject_live_human_notice(messages, arun_human_msgs)
+
+                ai_msg = await asyncio.to_thread(session_llm.invoke, messages)
+
+                if ai_msg.tool_calls:
+                    scratchpad.append(ai_msg)
+                    for tc in ai_msg.tool_calls:
+                        tool_name = tc["name"]
+                        tool_args = tc.get("args", {})
+
+                        status_msg = (
+                            "Searching Arun's knowledge base..." if tool_name == "search_arun_knowledge"
+                            else "Sending notification to Arun..." if tool_name == "notify_arun"
+                            else f"Executing {tool_name}..."
+                        )
+
+                        yield {"type": "status", "content": status_msg}
+                        thoughts.append(status_msg)
+                        executed_tools.append(f"{tool_name}({tool_args})")
+
+                        if tool_name == "search_arun_knowledge":
+                            search_count += 1
+
+                        if search_count > self.max_search_limit:
+                            tool_result = f"Search limit reached ({self.max_search_limit}). Finalizing based on existing context."
+                        else:
+                            tool_func = (tool_map or {}).get(tool_name)
+                            tool_result = await asyncio.to_thread(tool_func.invoke, tool_args) if tool_func else f"Unknown tool: {tool_name}"
+
+                        if tool_name == "search_arun_knowledge" and tool_result:
+                            retrieved_chunks.append(str(tool_result)[:1000])
+                        elif tool_name == "get_github_live_data" and tool_result:
+                            github_data.append(str(tool_result)[:1000])
+
+                        scratchpad.append({
+                            "role": "tool",
+                            "name": tool_name,
+                            "tool_call_id": tc["id"],
+                            "content": str(tool_result)[:15000],
+                        })
+                    iterations += 1
+                else:
+                    yield {"type": "status", "content": "Synthesizing final response..."}
+                    thoughts.append("Synthesizing final response...")
+
+                    full_reply = ""
+                    async for chunk in self._stream_tokens(session_llm, messages):
+                        yield chunk
+                        if isinstance(chunk, dict) and chunk.get("type") == "token":
+                            full_reply += chunk["content"]
+
+                    final_response = full_reply
+                    break
+
+            if not final_response:
+                yield {"type": "status", "content": "Synthesizing final response..."}
+                thoughts.append("Synthesizing final response...")
+                messages_fallback = session_prompt.format_messages(
+                    running_summary=memory.running_summary,
+                    chat_history=memory.get_messages(),
+                    input=message,
+                    agent_scratchpad=scratchpad,
+                )
+                full_reply = ""
+                async for chunk in self._stream_tokens(session_llm, messages_fallback):
+                    yield chunk
+                    if isinstance(chunk, dict) and chunk.get("type") == "token":
+                        full_reply += chunk["content"]
+                final_response = full_reply
+
+            memory.add_interaction(message, final_response)
+            self.store.record_message(session_id, "twin", final_response, thoughts=thoughts)
+
+            queue_chat_history_to_telegram(
+                session_id=session_id,
+                user_input=message,
+                assistant_response=final_response,
+                thoughts=thoughts,
+                tool_calls=executed_tools,
+                retrieved_chunks=retrieved_chunks,
+                github_data=github_data,
+            )
+
+            # Unconditionally queue 100% automated chat alert for EVERY chat.
+            queue_automated_chat_alert(
+                session_id=session_id,
+                user_input=message,
+                assistant_response=final_response,
+            )
+
+            yield {
+                "type": "final",
+                "reply": final_response,
+                "thoughts": thoughts,
+                "session_id": session_id,
+            }
+
+        except Exception as err:
+            err_msg = f"API Error: {str(err)}"
+            yield {"type": "error", "content": err_msg}
+
+    async def trigger_ai_answer(self, session_id: str, extra_prompt: str = "") -> str:
+        """Synthesize an AI answer after the /answer command from the real human."""
+        from backend.app.core.agent import init_agent
+
+        memory = self.store.get_or_create_memory(session_id)
+        main_llm, prompt, _, _ = init_agent()
+
+        session_msgs = self.store.get_history(session_id)
+        transcript_lines = []
+        last_user_input = ""
+        for m in session_msgs:
+            sender_label = (
+                "Visitor" if m.get("sender") == "user"
+                else "Real Arun Yadav (👨‍💻)" if m.get("sender") == "human_arun"
+                else "Arun's AI Assistant"
+            )
+            transcript_lines.append(f"{sender_label}: {m.get('text')}")
+            if m.get("sender") == "user":
+                last_user_input = m.get("text")
+
+        full_transcript = "\n".join(transcript_lines)
+
+        system_notice = SystemMessage(content=(
+            f"🟢 3-WAY CHAT INSTRUCTION FOR AI TWIN:\n"
+            f"Real Arun Yadav issued the /answer command for you to respond to the visitor's question.\n"
+            f"Read the full 3-party conversation transcript below (Visitor, Real Arun Yadav, and AI Twin):\n\n"
+            f"--- FULL 3-WAY CHAT TRANSCRIPT ---\n{full_transcript}\n\n"
+            f"MANDATORY INSTRUCTIONS FOR AI TWIN:\n"
+            f"1. Synthesize all context from the visitor's question and Real Arun Yadav's live comments.\n"
+            f"2. Generate an accurate, helpful, and natural response for the visitor.\n"
+            f"3. Acknowledge Real Arun Yadav's live presence if relevant."
+        ))
+
+        messages = prompt.format_messages(
+            running_summary=memory.running_summary,
+            chat_history=memory.get_messages(),
+            input=extra_prompt or last_user_input or "Please answer the visitor's question based on our 3-way conversation.",
+            agent_scratchpad=[],
+        )
+        messages.insert(1, system_notice)
+
+        ai_msg = await asyncio.to_thread(main_llm.invoke, messages)
+        final_reply = ai_msg.content.strip()
+
+        if final_reply:
+            self.store.record_message(
+                session_id, "twin", final_reply,
+                thoughts=["AI Twin triggered via /answer command."],
+            )
+            memory.add_interaction(last_user_input or "Visitor Query", final_reply)
+
+        return final_reply
+
+    async def _stream_tokens(self, llm, messages):
+        """Yields {'type':'token', 'content': ...} chunks by streaming the LLM.
+
+        Mirrors the original implementation: the sync `llm.stream` generator is
+        consumed in place so tokens arrive incrementally over the wire.
+        """
+        for chunk in llm.stream(messages):
+            if getattr(chunk, "content", None):
+                yield {"type": "token", "content": chunk.content}
+                await asyncio.sleep(0.005)
+
+    def sync_reply(
+        self,
+        session_id: str,
+        user_input: str,
+        llm: Any,
+        prompt: Any,
+        memory: Any,
+        tool_map: Optional[Dict[str, Any]] = None,
+        user_metadata: Optional[Dict[str, Any]] = None,
+        max_iterations: int = 3,
+    ) -> str:
+        """Blocking agent turn for non-streaming channels (Telegram bot).
+
+        Replicates the legacy bot loop exactly (3 iterations, tool errors
+        swallowed into the reply context, final fallback message).
+        """
+        from backend.app.services.notification_service import (
+            queue_debug_event,
+            queue_maybe_notify_arun,
+        )
+
+        scratchpad: List[Any] = []
+        tool_map = tool_map or {}
+
+        try:
+            queue_debug_event(
+                "user_message",
+                user_input,
+                {"channel": "telegram", "chat_id": session_id, **(user_metadata or {})},
+            )
+
+            pre_escalation = run_pre_escalation(
+                user_input,
+                tool_map,
+                {"channel": "telegram", "chat_id": session_id, **(user_metadata or {})},
+                False,
+            )
+            if pre_escalation:
+                queue_debug_event(
+                    "pre_escalation",
+                    pre_escalation.get("result", ""),
+                    {
+                        "channel": "telegram",
+                        "chat_id": session_id,
+                        "category": pre_escalation.get("category"),
+                        "reason": pre_escalation.get("reason"),
+                        **(user_metadata or {}),
+                    },
+                )
+
+            final_response = None
+
+            for _ in range(max_iterations):
+                messages = prompt.format_messages(
+                    running_summary=memory.running_summary,
+                    chat_history=memory.get_messages(),
+                    input=user_input,
+                    agent_scratchpad=scratchpad,
+                )
+                ai_msg = llm.invoke(messages)
+
+                if ai_msg.tool_calls:
+                    scratchpad.append(ai_msg)
+                    for tc in ai_msg.tool_calls:
+                        tool_name = tc["name"]
+                        tool_args = tc.get("args", {})
+                        queue_debug_event(
+                            "tool_call",
+                            json.dumps(tool_args, ensure_ascii=False, indent=2, default=str),
+                            {
+                                "channel": "telegram",
+                                "chat_id": session_id,
+                                "tool_name": tool_name,
+                                **(user_metadata or {}),
+                            },
+                        )
+
+                        tool_func = tool_map.get(tool_name)
+                        try:
+                            result = tool_func.invoke(tool_args)
+                        except Exception as e:
+                            result = f"Tool error: {e}"
+
+                        scratchpad.append({
+                            "role": "tool",
+                            "name": tool_name,
+                            "tool_call_id": tc["id"],
+                            "content": str(result)[:2000],
+                        })
+                        queue_debug_event(
+                            "tool_result",
+                            str(result),
+                            {
+                                "channel": "telegram",
+                                "chat_id": session_id,
+                                "tool_name": tool_name,
+                                **(user_metadata or {}),
+                            },
+                        )
+                else:
+                    final_response = ai_msg.content
+                    break
+
+            if not final_response:
+                final_response = "I ran into an issue internally. Please try again."
+
+            queue_debug_event(
+                "assistant_reply",
+                final_response,
+                {"channel": "telegram", "chat_id": session_id, **(user_metadata or {})},
+            )
+
+            queue_maybe_notify_arun(
+                user_input=user_input,
+                final_response=final_response,
+                scratchpad=scratchpad,
+                tool_map=tool_map,
+                user_metadata={"channel": "telegram", "chat_id": session_id, **(user_metadata or {})},
+                pre_notified=bool(pre_escalation and pre_escalation.get("handled")),
+            )
+
+            memory.add_interaction(user_input, final_response)
+            return final_response
+        except Exception as e:
+            queue_debug_event(
+                "error",
+                str(e),
+                {"channel": "telegram", "chat_id": session_id, **(user_metadata or {})},
+            )
+            raise
+
+
+agent_runner = AgentRunner(agent_factory=None)

backend/app/services/auth_service.py

@@ -0,0 +1,22 @@
+"""Admin authentication for the 3-way live human chat takeover.
+
+Real Arun logs in through a signed admin token embedded in the 1-Click Join
+link generated by Telegram alerts. The token is derived from the session id
+plus a shared secret, keeping the webhook/join flow stateless.
+"""
+import hashlib
+import os
+
+ADMIN_SECRET_KEY = os.getenv("ADMIN_SECRET_KEY", "aruncore_secret_key_2026")
+
+
+def generate_admin_token(session_id: str) -> str:
+    raw = f"{session_id}:{ADMIN_SECRET_KEY}".encode("utf-8")
+    return hashlib.sha256(raw).hexdigest()[:16]
+
+
+def verify_admin_token(session_id: str, token: str) -> bool:
+    if not session_id or not token:
+        return False
+    expected = generate_admin_token(session_id)
+    return token.strip() == expected

backend/app/services/background.py

@@ -0,0 +1,49 @@
+"""Single-responsibility background task queue.
+
+The ArunCore backend performs several fire-and-forget jobs (Telegram delivery,
+chat history logging, debug event logging, re-ingestion triggering). All of
+that work is submitted here instead of being swallowed by the request /
+agent loop, so slow network calls never block chat streaming.
+"""
+import queue
+import threading
+from typing import Any, Callable
+
+
+_task_queue: "queue.Queue[Any]" = queue.Queue()
+
+
+def _background_worker() -> None:
+    import urllib3
+
+    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+
+    while True:
+        try:
+            task = _task_queue.get()
+            if task is None:
+                break
+            func, args, kwargs = task
+            try:
+                func(*args, **kwargs)
+            except Exception as e:
+                print(f"[BACKGROUND WORKER ERROR] Failed in {getattr(func, '__name__', func)}: {e}")
+            finally:
+                _task_queue.task_done()
+        except Exception as outer_e:
+            print(f"[BACKGROUND WORKER FATAL] Queue fetch failed: {outer_e}")
+
+
+_thread = threading.Thread(target=_background_worker, daemon=True, name="aruncore-background")
+_thread.start()
+
+
+def submit_background_task(name: str, func: Callable, *args: Any, **kwargs: Any) -> bool:
+    """Enqueue a callable to run on the shared background worker thread."""
+    try:
+        _task_queue.put((func, args, kwargs))
+        print(f"[BACKGROUND] {name}: Task queued.")
+        return True
+    except Exception as e:
+        print(f"[BACKGROUND ERROR] Failed to queue {name}: {e}")
+        return False

backend/app/services/knowledge_service.py

@@ -0,0 +1,453 @@
+"""Knowledge retrieval layer of the ArunCore digital twin.
+
+Owns every read against the on-disk knowledge base:
+
+* `data/github/<repo>/README.md` historical READMEs
+* `data/linkedin/posts.md` scraped posts
+* `data/static/*.md` profile, rules of engagement
+* `data/raw/personal_background.md`
+* `data/raw/unknown_questions.json` verified Q&A pairs
+
+It also handles live GitHub lookups and persists newly learned Q&A pairs
+(the active-learning feedback loop).
+"""
+import os
+import re
+import json
+import datetime
+import subprocess
+from typing import Dict, Any, List, Optional, Tuple
+
+import requests
+
+from backend.app.services.notification_service import safe_truncate, schedule_notify_arun
+
+
+def _project_root() -> str:
+    """profile/ (the repository root holding `data/` and `backend/`)."""
+    return os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
+
+
+# ------------------------------------------------------------------ #
+# Chunking + relevance scoring helpers                                #
+# ------------------------------------------------------------------ #
+_QUERY_STOP_WORDS = {
+    "how", "does", "the", "a", "an", "is", "for", "to", "of", "with", "work",
+    "what", "tell", "me", "about", "can", "you", "who", "where", "why", "arun",
+    "and", "or", "that", "this", "it", "in", "on", "at", "your", "my", "i", "we",
+}
+
+_BOILERPLATE_LINE_RE = re.compile(
+    r"^(>\s*\*\*GitHub Repository:|\*\*Primary Language:|>\s*\*\*Primary Language:|"
+    r">\s*\*\*Description:|>\s*\*\*Stars:|\*\*GitHub|#\s+[\w_-]+\s*$)"
+)
+
+
+def _query_terms(cleaned_query: str) -> List[str]:
+    """Significant query words (stopwords removed, len>2)."""
+
+    def _tok(w: str) -> str:
+        return w.strip("?.,!;:'\"()[]{}")
+
+    return [
+        _tok(w) for w in cleaned_query.split()
+        if _tok(w) and _tok(w).lower() not in _QUERY_STOP_WORDS and len(_tok(w)) > 2
+    ]
+
+
+def _relevance(score: int, alpha: float = 1.0) -> float:
+    """Normalize a raw hit count into a 0..˜2 ordering score."""
+    return alpha * (1.0 + min(score, 6))
+
+
+def _term_hits(text: str, terms: List[str]) -> int:
+    """Count word-boundary, case-insensitive term occurrences."""
+    if not terms or not text:
+        return 0
+    lowered = text.lower()
+    hits = 0
+    for term in terms:
+        hits += len(re.findall(rf"(?<![a-z0-9]){re.escape(term)}(?![a-z0-9])", lowered))
+    return hits
+
+
+def _term_score(heading: str, body: str, terms: List[str]) -> int:
+    """Weighted relevance: heading hits count double body hits."""
+    if not terms:
+        return 0
+    return 2 * _term_hits(heading, terms) + _term_hits(body, terms)
+
+
+def _score_term_streak(text: str, terms: List[str]) -> int:
+    """Consecutive-significant-term density bonus (e.g. 'clinical reasoning tutor')."""
+    if not terms or not text:
+        return 0
+    lowered = text.lower()
+    streak = 0
+    best = 0
+    words = re.findall(r"[a-z0-9]+", lowered)
+    term_set_low = {t.lower() for t in terms}
+    for w in words:
+        if w in term_set_low:
+            streak += 1
+            best = max(best, streak)
+        else:
+            streak = 0
+    return 2 * best + _term_hits(text, terms)
+
+
+def _clean_readme(content: str) -> str:
+    """Strip YAML frontmatter + GitHub API boilerplate blockquote lines."""
+    if content.startswith("---"):
+        parts = content.split("---", 2)
+        if len(parts) >= 3:
+            content = parts[2]
+    out = []
+    for line in content.splitlines():
+        stripped = line.strip()
+        if _BOILERPLATE_LINE_RE.match(stripped):
+            continue
+        if stripped.startswith("> **GitHub Repository:") or stripped.startswith("> **Primary Language:") or stripped.startswith(">-"):
+            continue
+        out.append(line)
+    return "\n".join(out)
+
+
+def _score_readme_sections(content: str, terms: List[str], max_chars: int = 1800) -> List[Tuple[str, str, int]]:
+    """Split a project README into heading-level chunks, scored by query hits.
+
+    Returns (heading, body, score) triples sorted best-first (score desc).
+    """
+    cleaned = _clean_readme(content)
+    sections: List[Tuple[str, str]] = []
+    cur_heading: Optional[str] = None
+    cur_body: List[str] = []
+
+    def _flush() -> None:
+        nonlocal cur_body
+        body = "\n".join(cur_body).strip()
+        cur_body = []
+        if body:
+            sections.append((cur_heading or "Overview", body))
+
+    for line in cleaned.splitlines():
+        if re.match(r"^#{1,4} ", line):
+            _flush()
+            cur_heading = re.sub(r"^#{1,4} ", "", line).strip()
+        else:
+            cur_body.append(line)
+    _flush()
+
+    scored = []
+    for heading, body in sections:
+        if not body:
+            continue
+        score = _term_score(heading, body, terms)
+        if score > 0:
+            truncated = body[:max_chars].rstrip()
+            if len(body) > max_chars:
+                truncated += "..."
+            scored.append((heading, truncated, score))
+    scored.sort(key=lambda x: x[2], reverse=True)
+    return scored
+
+
+def _score_profile_sections(content: str, terms: List[str], max_chars: int = 1200) -> List[Tuple[str, str, int]]:
+    """Split a profile / background doc into heading-level chunks, scored."""
+    sections: List[Tuple[str, str]] = []
+    current_heading = ""
+    cur: List[str] = []
+
+    def _flush() -> None:
+        nonlocal cur
+        body = "\n".join(cur).strip()
+        cur = []
+        if body:
+            sections.append((current_heading, body))
+
+    for line in content.splitlines():
+        if re.match(r"^#{1,3} ", line):
+            _flush()
+            current_heading = line.lstrip("# ").strip()
+        elif line.strip() == "---":
+            _flush()
+        else:
+            cur.append(line)
+    _flush()
+
+    scored = []
+    for heading, body in sections:
+        if not body.strip():
+            continue
+        score = _score_term_streak(heading + "\n" + body, terms)
+        if score > 0:
+            truncated = body[:max_chars].rstrip()
+            if len(body) > max_chars:
+                truncated += "..."
+            scored.append((f"### {heading}", truncated, score))
+    scored.sort(key=lambda x: x[2], reverse=True)
+    return scored
+
+
+def _clean_linkedin_posts(posts_text: str) -> List[str]:
+    """Strip YAML/meta boilerplate and without params, return bare post bodies."""
+    if posts_text.startswith("---"):
+        parts = posts_text.split("---", 2)
+        if len(parts) >= 3:
+            posts_text = parts[2]
+    posts = []
+    for block in re.split(r"(?m)^## ", posts_text):
+        block = block.strip()
+        if not block:
+            continue
+        lines: List[str] = []
+        for line in block.splitlines():
+            ls = line.strip()
+            if ls.startswith("**Date:") or ls.startswith("**Post Title") or ls.startswith("> **Link:"):
+                continue
+            lines.append(line)
+        body = "\n".join(lines).strip()
+        if body:
+            posts.append("# " + body)
+    return posts
+
+
+class KnowledgeService:
+    """Single responsibility: every read/write against Arun's knowledge data."""
+
+    def __init__(self, root_dir: Optional[str] = None):
+        self.root_dir = root_dir or _project_root()
+        self.data_dir = os.path.join(self.root_dir, "data")
+
+    # ------------------------------------------------------------------ #
+    # Static persona documents (2-tuple used by the legacy API layer)     #
+    # ------------------------------------------------------------------ #
+    def load_static_profile_and_rules(self) -> Tuple[str, str]:
+        profile_content = ""
+        rules_content = ""
+
+        profile_path = os.path.join(self.data_dir, "static", "public_profile.md")
+        rules_path = os.path.join(self.data_dir, "static", "rules_of_engagement.md")
+
+        if os.path.exists(profile_path):
+            with open(profile_path, "r", encoding="utf-8") as f:
+                profile_content = f.read()
+
+        if os.path.exists(rules_path):
+            with open(rules_path, "r", encoding="utf-8") as f:
+                rules_content = f.read()
+
+        return profile_content, rules_content
+
+    # ------------------------------------------------------------------ #
+    # GitHub live data
+    # ------------------------------------------------------------------ #
+    def fetch_live_github(self, username: str = "neural-arun") -> str:
+        try:
+            res = requests.get(
+                f"https://api.github.com/users/{username}/repos?sort=updated&per_page=10",
+                timeout=5,
+            )
+            if res.status_code == 200:
+                repos = res.json()
+                lines = [
+                    f"• [{r['name']}]({r['html_url']}) - {r.get('description', 'No description')} (Updated: {r['updated_at'][:10]})"
+                    for r in repos
+                ]
+                return "### Live GitHub Repositories:\n" + "\n".join(lines)
+        except Exception as e:
+            return f"GitHub fetch error: {e}"
+        return "Could not fetch GitHub data."
+
+    # ------------------------------------------------------------------ #
+    # Free-form knowledge search (previously search_arun_knowledge)       #
+    # ------------------------------------------------------------------ #
+    def search(self, query: str) -> str:
+        """Retrieve a compact, relevance-ranked set of knowledge chunks.
+
+        Every source (project READMEs, LinkedIn posts, static profile docs,
+        verified Q&A pairs) is split into heading-level chunks, the YAML/GitHub
+        boilerplate is stripped, each chunk is scored against the query's
+        significant terms, and only the best-scoring chunks are returned. The
+        tool output is therefore a bounded set of *clean sections* instead of
+        whole raw files.
+        """
+        data_dir = self.data_dir
+        cleaned_query = (query or "").lower().replace("-", " ").replace("_", " ")
+        query_terms = _query_terms(cleaned_query)
+
+        project_aliases = {
+            "med_coach": ["med_coach", "medcoach", "med coach", "medical tutor", "clinical reasoning", "clinical tutor"],
+            "legal_RAG_system": ["legal_rag_system", "legal rag", "legal", "ipc", "indian penal code", "chunking"],
+            "neet-bot": ["neet-bot", "neet bot", "neet 2027", "cbt simulator", "ncert", "mcq"],
+            "real_state_listing_scraper": ["real_state_listing_scraper", "99acres", "cloudflare", "scraper", "real estate"],
+            "ArunCore": ["aruncore", "profile", "assistant", "vector database", "reranking", "fastapi"],
+        }
+
+        matched_folders = []
+        for folder_name, synonyms in project_aliases.items():
+            if any(syn in cleaned_query for syn in synonyms):
+                matched_folders.append(folder_name)
+
+        results: List[Tuple[float, str]] = []
+
+        # 1. PROJECT READMES -> section-level chunks, ranked by relevance.
+        for folder in matched_folders:
+            target_readme = os.path.join(data_dir, "github", folder, "README.md")
+            if os.path.exists(target_readme):
+                with open(target_readme, "r", encoding="utf-8") as f:
+                    content = f.read()
+                for heading, body, score in _score_readme_sections(content, query_terms):
+                    if score > 0 and body:
+                        block = f"## {heading}\n{body.strip()}"
+                        results.append((_relevance(score), f"--- Project README ({folder}) ---\n{block}"))
+
+        stop_words = {"how", "does", "the", "a", "an", "is", "for", "to", "of", "with", "work", "what", "tell", "me", "about", "can", "you", "who", "where", "why", "arun"}
+        significant_words = [w.strip("?,.!") for w in cleaned_query.split() if w.strip("?,.!") not in stop_words and len(w) > 2]
+
+        # 2. LINKEDIN POSTS: strip meta boilerplate, chunk per post, rank.
+        posts_path = os.path.join(data_dir, "linkedin", "posts.md")
+        if os.path.exists(posts_path):
+            with open(posts_path, "r", encoding="utf-8") as f:
+                posts_text = f.read()
+            for post in _clean_linkedin_posts(posts_text):
+                body = post.strip()
+                if not body:
+                    continue
+                score = sum(1 for w in significant_words if w.lower() in body.lower())
+                if score > 0 and query_terms:
+                    results.append((_relevance(score, 0.8), f"--- Relevant LinkedIn Post ---\n{body[:2200]}"))
+
+        # 3. CORE PROFILE & BACKGROUND DOCS: heading-level chunks, ranked.
+        other_static_files = [
+            os.path.join(data_dir, "static", "public_profile.md"),
+            os.path.join(data_dir, "static", "rules_of_engagement.md"),
+            os.path.join(data_dir, "raw", "personal_background.md"),
+        ]
+        static_hits: List[Tuple[str, str, float]] = []
+        for s_file in other_static_files:
+            if os.path.exists(s_file):
+                with open(s_file, "r", encoding="utf-8") as f:
+                    content = f.read()
+                for heading, chunk, score in _score_profile_sections(content, query_terms):
+                    if score > 0:
+                        block = f"{heading}\n{chunk.strip()}"
+                        if len(block.strip()) > 80:
+                            static_hits.append((_relevance(score, 0.9), f"--- Document Insight ({os.path.basename(s_file)}) ---\n{block.strip()}"))
+        results.extend(static_hits)
+
+        # Domain fallback: "stack/tech/language/framework/tools" queries that
+        # the lexical matcher missed still deserve the Technical Stack section.
+        stack_terms = {"stack", "tech", "technology", "technologies", "language", "framework", "tool", "tools", "languages"}
+        if static_hits and not any(
+            "Technical Stack" in block for _, block in static_hits
+        ) and any(t in stack_terms for t in query_terms):
+            profile_path = os.path.join(data_dir, "static", "public_profile.md")
+            if os.path.exists(profile_path):
+                with open(profile_path, "r", encoding="utf-8") as f:
+                    profile_content = f.read()
+                for heading, chunk, score in _score_profile_sections(profile_content, stack_terms):
+                    if "Technical Stack" in heading:
+                        block = f"{heading}\n{chunk.strip()}"
+                        results.append((2.0, f"--- Document Insight (public_profile.md) ---\n{block}"))
+
+        # 4. VERIFIED Q&A PAIRS from the active-learning store.
+        unknown_questions_path = os.path.join(data_dir, "raw", "unknown_questions.json")
+        if os.path.exists(unknown_questions_path):
+            try:
+                with open(unknown_questions_path, "r", encoding="utf-8") as f:
+                    uq_data = json.load(f)
+                    if isinstance(uq_data, list):
+                        for item in uq_data:
+                            q_item = item.get("question", "")
+                            a_item = item.get("answer", "")
+                            if any(w in q_item.lower() or w in a_item.lower() for w in significant_words):
+                                results.append((0.6, f"--- Verified Q&A Pair ---\nQuestion: {q_item}\nAnswer: {a_item}"))
+            except Exception as e:
+                print(f"[UNKNOWN QUESTIONS READ ERROR] {e}")
+
+        # Sort by relevance score, dedupe, cap the total payload.
+        results.sort(key=lambda x: x[0], reverse=True)
+        unique_results = []
+        seen = set()
+        payload_len = 0
+        for _, block in results:
+            block = block.strip()
+            if block in seen:
+                continue
+            if len(block) > 3000:
+                block = block[:3000].rstrip() + "..."
+            if payload_len + len(block) > 9000:
+                break
+            unique_results.append(block)
+            seen.add(block)
+            payload_len += len(block) + 2
+
+        if unique_results:
+            return "\n\n".join(unique_results)
+
+        # Run out of chunks -> auto-trigger notification
+        schedule_notify_arun(
+            "UNKNOWN_QUESTION",
+            f"Unknown Question (No KB Match): {query}",
+        )
+        return (
+            "No exact match found in knowledge base. Auto-triggered UNKNOWN_QUESTION alert to Arun's phone. "
+            "YOU MUST CALL notify_arun AND ASK THE USER FOR THEIR CONTACT INFO (Name, Email, Phone/WhatsApp)."
+        )
+
+    # ------------------------------------------------------------------ #
+    # Active learning: persist a verified Q&A and re-ingest into memory   #
+    # ------------------------------------------------------------------ #
+    def save_verified_answer(self, question: str, answer: str) -> str:
+        """Writes an answered unknown question into data/raw/unknown_questions.json and triggers vector DB re-ingestion."""
+        target_file = os.path.join(self.data_dir, "raw", "unknown_questions.json")
+        os.makedirs(os.path.dirname(target_file), exist_ok=True)
+
+        now_iso = datetime.datetime.utcnow().isoformat() + "Z"
+
+        entry = {
+            "question": question.strip(),
+            "answer": answer.strip(),
+            "timestamp": now_iso,
+        }
+
+        existing = []
+        if os.path.exists(target_file):
+            try:
+                with open(target_file, "r", encoding="utf-8") as f:
+                    existing = json.load(f)
+                    if not isinstance(existing, list):
+                        existing = []
+            except Exception:
+                existing = []
+
+        updated = False
+        for item in existing:
+            if item.get("question", "").lower() == question.strip().lower():
+                item["answer"] = answer.strip()
+                item["timestamp"] = now_iso
+                updated = True
+                break
+
+        if not updated:
+            existing.append(entry)
+
+        with open(target_file, "w", encoding="utf-8") as f:
+            json.dump(existing, f, indent=2)
+
+        ingest_script = os.path.join(self.root_dir, "backend", "app", "core", "ingest.py")
+        try:
+            subprocess.Popen(["python3", ingest_script])
+        except Exception as e:
+            print(f"[REINGEST ERROR] {e}")
+
+        return "SUCCESS: Saved to unknown_questions.json and ingested into memory."
+
+
+knowledge_service = KnowledgeService()
+
+
+def load_static_context() -> Tuple[str, str]:
+    """Backward-compatible 2-tuple reader (profile, rules)."""
+    return knowledge_service.load_static_profile_and_rules()

backend/app/services/memory_manager.py

@@ -0,0 +1,69 @@
+"""Rolling conversation memory with summary-compression.
+
+A single session's chat lives in a shrinking window. Every N turns the oldest
+messages are folded into a compact running summary by the summary LLM, so long
+conversations keep full context without blowing the token budget.
+"""
+from typing import Any, List, Optional
+
+from langchain_core.messages import BaseMessage, HumanMessage, AIMessage, SystemMessage
+
+
+class RollingMemory:
+    """Stateful per-session memory used across API and Telegram chat loops."""
+
+    def __init__(self, summary_llm: Any, max_turns: int = 4):
+        self.summary_llm = summary_llm
+        self.max_turns = max_turns
+        self.history: List[BaseMessage] = []
+        self.running_summary: str = "No prior summary. This is the start of the conversation."
+        self.invocation_count = 0
+
+    def add_interaction(self, human_text: str, ai_text: str):
+        self.history.append(HumanMessage(content=human_text))
+        self.history.append(AIMessage(content=ai_text))
+        self.invocation_count += 1
+
+        if self.invocation_count >= self.max_turns:
+            self._summarize_and_prune()
+
+    def _summarize_and_prune(self):
+        print("\n[SYSTEM] Triggering background summarization...")
+        messages_to_summarize = self.history[:-4]
+
+        if not messages_to_summarize:
+            return
+
+        chat_transcript = "\n".join(
+            [f"{'User' if isinstance(m, HumanMessage) else 'Arun Assistant'}: {m.content}" for m in messages_to_summarize]
+        )
+
+        prompt = (
+            "You are an internal memory compression engine for Arun's AI Assistant.\n"
+            "Merge the existing summary with the new transcript. Preserve technical context, names, project mentions, user goals, and important decisions. "
+            "Keep it concise and stable. Return no more than 5 sentences.\n\n"
+            f"--- EXISTING SUMMARY ---\n{self.running_summary}\n\n"
+            f"--- NEW CHAT TO MERGE ---\n{chat_transcript}"
+        )
+
+        try:
+            res = self.summary_llm.invoke([SystemMessage(content=prompt)])
+            self.running_summary = res.content.strip()
+            self.history = self.history[-4:]
+            self.invocation_count = len(self.history) // 2
+            print(f"[SYSTEM] Memory compressed. New summary: {self.running_summary[:120]}...")
+        except Exception as e:
+            print(f"[SYSTEM ERROR] Failed to summarize memory: {e}")
+
+    def get_messages(self) -> List[BaseMessage]:
+        return self.history
+
+    def clear(self):
+        self.history.clear()
+        self.running_summary = "No prior summary. This is the start of the conversation."
+        self.invocation_count = 0
+
+
+# Backwards-compatible alias: the decoupled API exposes this class as
+# `MemoryManager` while the agent layer keeps the historical `RollingMemory` name.
+MemoryManager = RollingMemory

backend/app/services/notification_service.py

@@ -0,0 +1,508 @@
+"""Telegram notification & alert delivery.
+
+All outgoing Telegram traffic is consolidated here: chat history logging,
+debug events, per-message live chat alerts with the 1-Click Join link, and
+the category-based notify_arun escalation system. Every call is fire-and-forget
+via the shared background queue so the chat loop never blocks on the network.
+"""
+import os
+import json
+import time
+from typing import Dict, Any, List, Optional, Tuple
+
+from backend.app.services.background import submit_background_task
+from backend.app.services.auth_service import generate_admin_token
+
+
+def _is_truthy_env(val: Optional[str], default: bool = True) -> bool:
+    if val is None:
+        return default
+    return val.strip().lower() in ("1", "true", "yes", "on")
+
+
+def _is_telegram_debug_enabled() -> bool:
+    return _is_truthy_env(os.getenv("TELEGRAM_DEBUG_ENABLED"), default=True)
+
+
+def _get_telegram_target(debug: bool = False, alert: bool = False) -> Tuple[Optional[str], Optional[str]]:
+    if alert:
+        token = os.getenv("TELEGRAM_ALERT_BOT_TOKEN") or os.getenv("TELEGRAM_BOT_TOKEN")
+        chat_id = os.getenv("TELEGRAM_ALERT_CHAT_ID") or os.getenv("TELEGRAM_CHAT_ID")
+        return token, chat_id
+    if debug:
+        token = os.getenv("TELEGRAM_DEBUG_BOT_TOKEN") or os.getenv("TELEGRAM_BOT_TOKEN")
+        chat_id = os.getenv("TELEGRAM_DEBUG_CHAT_ID") or os.getenv("TELEGRAM_CHAT_ID")
+        return token, chat_id
+
+    return os.getenv("TELEGRAM_BOT_TOKEN"), os.getenv("TELEGRAM_CHAT_ID")
+
+
+def safe_truncate(text: str, limit: int = 1500) -> str:
+    cleaned = (text or "").strip()
+    if len(cleaned) <= limit:
+        return cleaned
+    return cleaned[:limit] + "..."
+
+
+def _escape_html(text: str) -> str:
+    return (
+        (text or "")
+        .replace("&", "&amp;")
+        .replace("<", "&lt;")
+        .replace(">", "&gt;")
+    )
+
+
+def _chunk_text(text: str, limit: int = 2200) -> List[str]:
+    cleaned = (text or "").strip() or "(empty)"
+    parts: List[str] = []
+    remaining = cleaned
+
+    while len(remaining) > limit:
+        split_at = remaining.rfind("\n", 0, limit)
+        if split_at < int(limit * 0.5):
+            split_at = remaining.rfind(" ", 0, limit)
+        if split_at <= 0:
+            split_at = limit
+
+        parts.append(remaining[:split_at].strip())
+        remaining = remaining[split_at:].lstrip()
+
+    if remaining:
+        parts.append(remaining)
+
+    return parts or ["(empty)"]
+
+
+TELEGRAM_DELIVERY_LOGS: List[Dict[str, Any]] = []
+
+
+def _record_telegram_log(label: str, status: str, detail: str) -> None:
+    entry = {
+        "time": time.strftime("%Y-%m-%d %H:%M:%S UTC", time.gmtime()),
+        "label": label,
+        "status": status,
+        "detail": detail,
+    }
+    TELEGRAM_DELIVERY_LOGS.append(entry)
+    if len(TELEGRAM_DELIVERY_LOGS) > 30:
+        TELEGRAM_DELIVERY_LOGS.pop(0)
+
+
+def _send_telegram_message(
+    token: str,
+    chat_id: str,
+    text: str,
+    parse_mode: str = "HTML",
+    max_attempts: int = 3,
+    delivery_label: str = "default",
+    retry_sleep_seconds: float = 1.0,
+) -> str:
+    import urllib.request
+    import urllib.parse
+    import ssl
+
+    token = (token or "").strip()
+    if token.lower().startswith("bot"):
+        token = token[3:].strip()
+
+    chat_id = (chat_id or "").strip()
+
+    if not token or not chat_id:
+        msg = f"FAILED: Token or Chat ID empty (token_len={len(token)}, chat_id_len={len(chat_id)})"
+        _record_telegram_log(delivery_label, "MISSING_CREDS", msg)
+        print(f"[TELEGRAM:{delivery_label}] {msg}")
+        return msg
+
+    chunks = _chunk_text(text)
+    total_chunks = len(chunks)
+
+    ssl_ctx = ssl.create_default_context()
+    ssl_ctx.check_hostname = False
+    ssl_ctx.verify_mode = ssl.CERT_NONE
+
+    for idx, chunk in enumerate(chunks, 1):
+        payload: Dict[str, Any] = {
+            "token": token,
+            "chat_id": chat_id,
+            "text": chunk,
+            "disable_web_page_preview": True,
+        }
+        if parse_mode:
+            payload["parse_mode"] = parse_mode
+
+        sent_chunk = False
+        last_error = ""
+
+        # Attempt 1: Try Vercel Serverless Relay (bypasses HF Space Telegram firewall block)
+        relay_url = "https://aruncore.vercel.app/api/telegram"
+        try:
+            relay_data = json.dumps(payload).encode("utf-8")
+            relay_req = urllib.request.Request(
+                relay_url,
+                data=relay_data,
+                headers={"Content-Type": "application/json", "User-Agent": "ArunCore/1.0", "Connection": "close"},
+            )
+            with urllib.request.urlopen(relay_req, timeout=8, context=ssl_ctx) as resp:
+                resp_bytes = resp.read()
+                resp_data = json.loads(resp_bytes.decode("utf-8"))
+                if resp.status == 200 and resp_data.get("ok"):
+                    sent_chunk = True
+        except Exception as e:
+            last_error = f"Vercel Relay error: {e}"
+
+        if not sent_chunk:
+            # Direct Telegram API Fallback
+            for attempt in range(1, max_attempts + 1):
+                try:
+                    url = f"https://api.telegram.org/bot{token}/sendMessage"
+                    direct_payload = {
+                        "chat_id": chat_id,
+                        "text": chunk,
+                        "disable_web_page_preview": True,
+                    }
+                    if parse_mode:
+                        direct_payload["parse_mode"] = parse_mode
+
+                    data = json.dumps(direct_payload).encode("utf-8")
+                    req = urllib.request.Request(
+                        url,
+                        data=data,
+                        headers={
+                            "Content-Type": "application/json",
+                            "User-Agent": "ArunCore/1.0",
+                            "Connection": "close",
+                        },
+                    )
+                    with urllib.request.urlopen(req, timeout=10, context=ssl_ctx) as resp:
+                        resp_bytes = resp.read()
+                        resp_data = json.loads(resp_bytes.decode("utf-8"))
+                        if resp.status == 200 and resp_data.get("ok"):
+                            sent_chunk = True
+                            break
+                        else:
+                            last_error = f"HTTP {resp.status}: {resp_bytes.decode('utf-8')[:200]}"
+                except Exception as e:
+                    last_error = str(e)
+
+                if attempt < max_attempts:
+                    time.sleep(retry_sleep_seconds)
+
+        if not sent_chunk and parse_mode == "HTML":
+            fallback_payload = {
+                "chat_id": chat_id,
+                "text": f"[{delivery_label}] (Plain Text Fallback {idx}/{total_chunks})\n{chunk}",
+                "disable_web_page_preview": True,
+            }
+            try:
+                url = f"https://api.telegram.org/bot{token}/sendMessage"
+                data = json.dumps(fallback_payload).encode("utf-8")
+                req = urllib.request.Request(
+                    url,
+                    data=data,
+                    headers={
+                        "Content-Type": "application/json",
+                        "User-Agent": "ArunCore/1.0",
+                        "Connection": "close",
+                    },
+                )
+                with urllib.request.urlopen(req, timeout=12, context=ssl_ctx) as resp:
+                    resp_bytes = resp.read()
+                    resp_data = json.loads(resp_bytes.decode("utf-8"))
+                    if resp.status == 200 and resp_data.get("ok"):
+                        sent_chunk = True
+            except Exception as e:
+                last_error = f"Fallback error: {e}"
+
+        if not sent_chunk:
+            err_msg = f"FAILED chunk {idx}/{total_chunks}: {last_error}"
+            print(f"[TELEGRAM:{delivery_label}] {err_msg}")
+            _record_telegram_log(delivery_label, "ERROR", err_msg)
+            return err_msg
+
+    print(f"[TELEGRAM:{delivery_label}] SUCCESS")
+    _record_telegram_log(delivery_label, "SUCCESS", f"Delivered {total_chunks} chunk(s)")
+    return "SUCCESS: message delivered."
+
+
+def send_chat_history_to_telegram(
+    session_id: str,
+    user_input: str,
+    assistant_response: str,
+    thoughts: Optional[List[str]] = None,
+    tool_calls: Optional[List[str]] = None,
+    retrieved_chunks: Optional[List[str]] = None,
+    github_data: Optional[List[str]] = None,
+) -> str:
+    token, chat_id = _get_telegram_target(debug=False)
+    if not token or not chat_id:
+        return "FAILED: Telegram credentials missing."
+
+    clean_user = _escape_html(safe_truncate(user_input, 1500))
+    clean_ai = _escape_html(safe_truncate(assistant_response, 2500))
+
+    tools_html = ""
+    if tool_calls and len(tool_calls) > 0:
+        tools_str = "\n".join([f"• <code>{_escape_html(safe_truncate(t, 250))}</code>" for t in tool_calls])
+        tools_html = f"\n\n<b>🧠 AI Decisions & Tool Calls:</b>\n{tools_str}"
+
+    chunks_html = ""
+    if retrieved_chunks and len(retrieved_chunks) > 0:
+        chunks_str = "\n".join([f"--- Chunk {i+1} ---\n{_escape_html(safe_truncate(c, 500))}" for i, c in enumerate(retrieved_chunks[:3])])
+        chunks_html = f"\n\n<b>📚 RAG Knowledge Chunks Retrieved:</b>\n<code>{chunks_str}</code>"
+
+    github_html = ""
+    if github_data and len(github_data) > 0:
+        github_str = "\n".join([f"--- Repo Data ---\n{_escape_html(safe_truncate(g, 500))}" for g in github_data[:2]])
+        github_html = f"\n\n<b>🐙 Live GitHub Repositories Fetched:</b>\n<code>{github_str}</code>"
+
+    thoughts_html = ""
+    if thoughts and len(thoughts) > 0:
+        thoughts_str = "\n".join([f"• {_escape_html(t)}" for t in thoughts])
+        thoughts_html = f"\n\n<b>⚙️ Execution Steps:</b>\n{thoughts_str}"
+
+    html = (
+        f"<b>📊 ARUNCORE FULL EXECUTION TRACE</b>\n"
+        f"<b>Session ID:</b> <code>{_escape_html(session_id)}</code>\n\n"
+        f"<b>👤 User Question:</b>\n{clean_user}"
+        f"{tools_html}"
+        f"{chunks_html}"
+        f"{github_html}"
+        f"{thoughts_html}\n\n"
+        f"<b>🤖 AI Twin Final Reply:</b>\n{clean_ai}"
+    )
+
+    return _send_telegram_message(
+        token=token,
+        chat_id=chat_id,
+        text=html,
+        parse_mode="HTML",
+        delivery_label="chat_log",
+    )
+
+
+def queue_chat_history_to_telegram(
+    session_id: str,
+    user_input: str,
+    assistant_response: str,
+    thoughts: Optional[List[str]] = None,
+    tool_calls: Optional[List[str]] = None,
+    retrieved_chunks: Optional[List[str]] = None,
+    github_data: Optional[List[str]] = None,
+) -> str:
+    submit_background_task(
+        "chat_history_log",
+        send_chat_history_to_telegram,
+        session_id,
+        user_input,
+        assistant_response,
+        thoughts or [],
+        tool_calls or [],
+        retrieved_chunks or [],
+        github_data or [],
+    )
+    return "QUEUED: chat history scheduled."
+
+
+def send_debug_event_to_telegram(
+    event_type: str,
+    payload_summary: str,
+    metadata: Optional[Dict[str, Any]] = None,
+) -> str:
+    if not _is_telegram_debug_enabled():
+        return "SKIPPED: debug disabled."
+
+    token, chat_id = _get_telegram_target(debug=True)
+    if not token or not chat_id:
+        return "FAILED: Telegram debug credentials missing."
+
+    clean_event = _escape_html(event_type.upper())
+    clean_payload = _escape_html(safe_truncate(payload_summary, 2000))
+
+    meta_str = ""
+    if metadata:
+        meta_str = "\n".join([f"• <b>{_escape_html(str(k))}:</b> {_escape_html(str(v))}" for k, v in metadata.items()])
+
+    html = f"<b>🔧 DEBUG EVENT: {clean_event}</b>\n{meta_str}\n\n<code>{clean_payload}</code>"
+
+    return _send_telegram_message(
+        token=token,
+        chat_id=chat_id,
+        text=html,
+        parse_mode="HTML",
+        delivery_label="debug_event",
+    )
+
+
+def queue_debug_event(
+    event_type: str,
+    payload_summary: str,
+    metadata: Optional[Dict[str, Any]] = None,
+) -> None:
+    submit_background_task(
+        "debug_event",
+        send_debug_event_to_telegram,
+        event_type,
+        payload_summary,
+        metadata,
+    )
+
+
+def send_automated_chat_alert(
+    session_id: str,
+    user_input: str,
+    assistant_response: str,
+) -> str:
+    token, chat_id = _get_telegram_target(alert=True)
+    if not token or not chat_id:
+        return "FAILED: Telegram credentials missing."
+
+    admin_tok = generate_admin_token(session_id)
+    join_link = f"https://aruncore.vercel.app/?session_id={session_id}&admin_token={admin_tok}"
+
+    clean_user = _escape_html(safe_truncate(user_input, 1000))
+    clean_ai = _escape_html(safe_truncate(assistant_response, 1500))
+
+    html = (
+        f"🚨 <b>LIVE WEBSITE CHAT ACTIVITY</b>\n"
+        f"<b>Session ID:</b> <code>{_escape_html(session_id)}</code>\n\n"
+        f"<b>👤 User Question:</b>\n{clean_user}\n\n"
+        f"<b>🤖 AI Response:</b>\n{clean_ai}\n\n"
+        f"🔗 <b><a href=\"{join_link}\">👉 CLICK HERE TO JOIN LIVE CHAT AS REAL ARUN</a></b>"
+    )
+
+    return _send_telegram_message(
+        token=token,
+        chat_id=chat_id,
+        text=html,
+        parse_mode="HTML",
+        delivery_label="every_chat_alert",
+    )
+
+
+def queue_automated_chat_alert(session_id: str, user_input: str, assistant_response: str) -> None:
+    submit_background_task(
+        "automated_chat_alert_bg",
+        send_automated_chat_alert,
+        session_id,
+        user_input,
+        assistant_response,
+    )
+
+
+def queue_maybe_notify_arun(
+    user_input: str,
+    reason: str = "",
+    channel: str = "api",
+    session_id: str = "",
+    final_response: str = "",
+    scratchpad: Optional[list] = None,
+    tool_map: Optional[dict] = None,
+    user_metadata: Optional[dict] = None,
+    pre_notified: bool = False,
+) -> None:
+    """Escalation hook: schedules an URGENT notify when contact intent is detected."""
+    submit_background_task(
+        "maybe_notify_arun",
+        _deliver_notify_arun,
+        "URGENT",
+        f"Reason: {reason or 'General check'}\nQuery: {user_input}",
+    )
+
+
+_RECENT_ALERTS: Dict[str, float] = {}
+_ALERT_DEDUP_WINDOW_SECONDS = 10.0
+
+
+def _should_send_alert(category: str, user_input: str) -> bool:
+    now = time.time()
+
+    expired_keys = [k for k, timestamp in _RECENT_ALERTS.items() if now - timestamp > _ALERT_DEDUP_WINDOW_SECONDS]
+    for k in expired_keys:
+        _RECENT_ALERTS.pop(k, None)
+
+    key = f"{category.upper()}:{user_input.strip().lower()}"
+    if key in _RECENT_ALERTS:
+        return False
+
+    _RECENT_ALERTS[key] = now
+    return True
+
+
+ALLOWED_NOTIFY_CATEGORIES = {
+    "UNKNOWN_QUESTION",
+    "URGENT",
+    "FEEDBACK",
+    "SYSTEM_ALERT",
+    "LEAD",
+    "ABUSE",
+    "WEIRD",
+    "SUSPICIOUS",
+    "OFF_TOPIC",
+}
+
+_CATEGORY_HEADERS = {
+    "UNKNOWN_QUESTION": "🤷 UNKNOWN QUESTION",
+    "LEAD":             "💼 NEW LEAD / HIRING INQUIRY",
+    "URGENT":           "🚨 URGENT ALERT",
+    "ABUSE":            "🤬 ABUSE / RUDE MESSAGE",
+    "WEIRD":            "👀 WEIRD / UNUSUAL MESSAGE",
+    "SUSPICIOUS":       "🕵️ SUSPICIOUS ACTIVITY",
+    "OFF_TOPIC":        "🚫 OFF-TOPIC / IRRELEVANT",
+    "FEEDBACK":         "💬 FEEDBACK",
+    "SYSTEM_ALERT":     "⚙️ SYSTEM ALERT",
+}
+
+
+def _deliver_notify_arun(
+    category: str,
+    user_input: str,
+    user_metadata_json: str = "",
+    _fast: bool = False,
+) -> str:
+    token, chat_id = _get_telegram_target(alert=True)
+
+    if not token or not chat_id:
+        msg = "FAILED: Telegram credentials missing."
+        _record_telegram_log("notify_arun_bg", "MISSING_CREDS", msg)
+        return msg
+
+    category = (category or "UNKNOWN_QUESTION").strip().upper()
+    if category not in ALLOWED_NOTIFY_CATEGORIES:
+        category = "UNKNOWN_QUESTION"
+
+    cleaned_input = safe_truncate(user_input, 1200)
+
+    if not _should_send_alert(category, cleaned_input):
+        msg = f"SKIPPED: duplicate {category} alert suppressed within 10s window."
+        _record_telegram_log("notify_arun_bg", "SKIPPED_DEDUP", msg)
+        return msg
+
+    header = _CATEGORY_HEADERS.get(category, f"🚨 ALERT: {category}")
+    html = (
+        f"<b>{header}</b>\n\n"
+        f"<b>User Query / Details:</b>\n{_escape_html(cleaned_input)}\n\n"
+        f"<b>Category:</b> {category}\n"
+        f"<b>Contact:</b> +91 8881109193 | neural.arun.dev@gmail.com\n\n"
+        f"<i>💡 Reply directly to this message to save your answer into AI memory!</i>"
+    )
+
+    return _send_telegram_message(
+        token=token,
+        chat_id=chat_id,
+        text=html,
+        parse_mode="HTML",
+        delivery_label="notify_arun_bg",
+    )
+
+
+def schedule_notify_arun(category: str, user_input: str, user_metadata_json: str = "") -> str:
+    """Queue an urgent notify_arun Telegram alert (used by the agent tool)."""
+    submit_background_task(
+        "notify_arun_bg",
+        _deliver_notify_arun,
+        category,
+        user_input,
+        user_metadata_json or "",
+    )
+    return "QUEUED: notify_arun alert scheduled."

backend/app/services/prompt_builder.py

@@ -0,0 +1,162 @@
+"""System prompt assembly for the ArunCore agent.
+
+Every tenant-facing prompt (both the legacy demos dictionary and the split
+tenant configs) flows through here. The builder returns the exact same prompt
+templates the legacy monolith produced, so prompt behavior is unchanged while
+the assembly itself lives in a single-responsibility service.
+"""
+import os
+from typing import Any, Dict, List, Optional, Tuple
+
+from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
+from langchain_core.messages import SystemMessage
+
+from backend.app.services.tenant_service import TenantService
+
+
+def _backend_app_dir() -> str:
+    """backend/app/ (parent of services/, core/, prompts/, ...)."""
+    return os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def _project_root() -> str:
+    """profile/ (repository root)."""
+    return os.path.dirname(_backend_app_dir())
+
+
+def load_static_context() -> Tuple[str, str, str, str, str]:
+    """Returns (system_prompt, guardrails, handoff, profile, rules).
+
+    The prompts/ directory may be absent on a fresh checkout; every missing
+    file simply resolves to an empty string, matching the legacy loader.
+    """
+    backend_app = _backend_app_dir()
+    root_dir = _project_root()
+
+    sys_prompt_path = os.path.join(backend_app, "prompts", "system_prompt.md")
+    guardrails_path = os.path.join(backend_app, "prompts", "guardrails.md")
+    handoff_path = os.path.join(backend_app, "prompts", "handoff_prompt.md")
+
+    profile_path = os.path.join(root_dir, "data", "static", "public_profile.md")
+    rules_path = os.path.join(root_dir, "data", "static", "rules_of_engagement.md")
+
+    def _read(path: str) -> str:
+        if os.path.exists(path):
+            with open(path, "r", encoding="utf-8") as f:
+                return f.read()
+        return ""
+
+    return (
+        _read(sys_prompt_path),
+        _read(guardrails_path),
+        _read(handoff_path),
+        _read(profile_path),
+        _read(rules_path),
+    )
+
+
+class PromptBuilder:
+    """Constructs the final system prompt + chat template for an agent run."""
+
+    def __init__(self, tenant_service_: Optional[TenantService] = None):
+        self.tenant_service = tenant_service_ or TenantService()
+
+    def build_system_prompt(self, tutor_id: Optional[str] = None) -> str:
+        """Assembles the system prompt for either a tenant (demos dict) or the
+        default Arun twin persona. Contains a literal ``{running_summary}``
+        placeholder that the chat template fills per message."""
+        tutor_cfg = self.tenant_service.load_legacy_tutor_config(tutor_id)
+
+        if tutor_cfg:
+            backend_cfg = tutor_cfg.get("backend_llm_configuration", {})
+            sys_prompt_cfg = backend_cfg.get("system_prompt", {})
+
+            custom_prompt = sys_prompt_cfg.get("persona_identity") or tutor_cfg.get("custom_system_prompt") or "You are an AI Course Advisor."
+            rules_list = sys_prompt_cfg.get("project_guidelines") or tutor_cfg.get("custom_rules") or []
+            courses_data = tutor_cfg.get("courses") or []
+            about_bio = tutor_cfg.get("frontend_ui_dictionary", {}).get("about_view", {}).get("bio_paragraphs") or tutor_cfg.get("about_text") or ""
+
+            rules_txt = "\n".join([f"- {r}" for r in rules_list])
+            courses_txt = json_dumps_safe(courses_data)
+            about_txt = "\n".join(about_bio) if isinstance(about_bio, list) else str(about_bio)
+
+            return f"""
+{custom_prompt}
+
+--- INSTRUCTOR & COURSE KNOWLEDGE ---
+About Instructor:
+{about_txt}
+
+Available Courses & Cohorts:
+{courses_txt}
+
+--- SPECIAL INSTRUCTIONS & RULES ---
+{rules_txt}
+
+--- LANGUAGE RULES ---
+Always respond in the exact language used by the student (English or natural Hinglish).
+
+--- PAST CONVERSATION SUMMARY ---
+{{running_summary}}
+"""
+
+        sys_content, guard_content, handoff_content, profile, rules = load_static_context()
+
+        return f"""
+--- MASTER RULES OF ENGAGEMENT & PERSONA (PRIMARY CORE) ---
+{rules}
+
+--- IDENTITY PROFILE & TECHNICAL SPECIFICATIONS ---
+{profile}
+
+--- GUARDRAILS & STEERING RULES ---
+{guard_content}
+
+--- 3-WAY LIVE CHAT & HANDOFF RULES ---
+{handoff_content}
+
+--- PAST CONVERSATION SUMMARY ---
+{{running_summary}}
+"""
+
+    @staticmethod
+    def build_chat_prompt(system_prompt: str) -> ChatPromptTemplate:
+        return ChatPromptTemplate.from_messages(
+            [
+                ("system", system_prompt),
+                MessagesPlaceholder(variable_name="chat_history"),
+                ("human", "{input}"),
+                MessagesPlaceholder(variable_name="agent_scratchpad"),
+            ]
+        )
+
+    @staticmethod
+    def inject_live_human_notice(messages: List[Any], arun_human_msgs: List[Dict[str, Any]]) -> List[Any]:
+        """Inserts the 3-way live chat presence notice when the real human is active."""
+        if not arun_human_msgs:
+            return messages
+
+        formatted_msgs = "\n".join([f"• Real Human Instructor (��‍💻): \"{m.get('text')}\"" for m in arun_human_msgs])
+        live_notice = SystemMessage(content=(
+            f"🟢 CRITICAL LIVE 3-WAY CHAT NOTICE (REAL HUMAN INSTRUCTOR IS PRESENT):\n"
+            f"The REAL HUMAN INSTRUCTOR (👨‍💻) HAS JOINED THIS CHAT ROOM LIVE AND IS CURRENTLY CHATTING!\n\n"
+            f"REAL INSTRUCTOR'S MESSAGES IN THIS SESSION:\n{formatted_msgs}\n\n"
+            f"MANDATORY INSTRUCTIONS FOR AI ASSISTANT IN THIS 3-WAY CHAT:\n"
+            f"1. Acknowledge that the REAL human instructor is present right next to you in this chat session!\n"
+            f"2. If the user asks how the instructor came here or questions about their arrival, explain enthusiastically: \"The real instructor tapped their 1-Click Telegram link and joined our chat live from their phone! So both of us (Real Instructor + AI Assistant) are here together with you!\"\n"
+            f"3. Never confuse yourself as the human — you are the AI Assistant co-piloting alongside the Real Instructor!"
+        ))
+
+        new_messages = list(messages)
+        if len(new_messages) > 1:
+            new_messages.insert(1, live_notice)
+        else:
+            new_messages.append(live_notice)
+        return new_messages
+
+
+def json_dumps_safe(obj: Any) -> str:
+    """dump with braces escaped so the LLM output isn't treated as prompt vars."""
+    import json
+
+    return json.dumps(obj, indent=2).replace("{", "{{").replace("}", "}}")

backend/app/services/rag_service.py

@@ -0,0 +1,48 @@
+"""Hybrid RAG coordinator.
+
+Composes the knowledge-retrieval pipeline for a tenant: dense vector search
+(ChromaDB), sparse keyword search (BM25), and reranking get their turn here
+in future integration; today ChromaDB is built at ingest time and runtime
+retrieval is served by `KnowledgeService` (project READMEs + LinkedIn posts +
+profile sections + verified Q&A pairs). This service also doubles as the
+active-learning ingestion point for owner-verified answers.
+"""
+import hashlib
+from typing import List, Dict, Any
+
+from backend.app.services.knowledge_service import knowledge_service
+
+
+class RAGService:
+    """Hybrid RAG coordinator: retrieval + active-learning ingestion per tenant."""
+
+    def __init__(self, tenant_id: str = "arun"):
+        self.tenant_id = tenant_id
+
+    def retrieve_context(self, query: str, top_k: int = 5) -> List[Dict[str, Any]]:
+        """Executes high-precision hybrid retrieval for the specified tenant."""
+        print(f"[RAG_SERVICE] Retrieving context for tenant '{self.tenant_id}' query: '{query[:80]}'")
+
+        raw = knowledge_service.search(query)
+
+        chunks: List[Dict[str, Any]] = []
+        if raw:
+            parts = [p for p in raw.split("\n\n--- ") if p.strip()]
+            for i, part in enumerate(parts[:top_k]):
+                chunks.append({
+                    "id": f"{self.tenant_id}_chunk_{i}_{hashlib.md5(part.encode('utf-8')).hexdigest()[:8]}",
+                    "content": part.strip(),
+                    "score": max(0.0, 0.95 - (i * 0.03)),
+                    "source": "knowledge_base",
+                })
+        return chunks
+
+    def add_knowledge_entry(self, question: str, answer: str) -> bool:
+        """Persists a verified Q&A into unknown_questions.json and re-ingests."""
+        try:
+            result = knowledge_service.save_verified_answer(question, answer)
+            print(f"[RAG_SERVICE] Active-learning entry for tenant '{self.tenant_id}': {result}")
+            return True
+        except Exception as e:
+            print(f"[RAG_SERVICE ERROR] Failed to persist Q&A for tenant '{self.tenant_id}': {e}")
+            return False

backend/app/services/session_store.py

@@ -0,0 +1,121 @@
+"""Thread-safe in-memory session state for the ArunCore backend.
+
+Consolidates all live websocket-like stores that were previously scattered
+through the API module:
+
+* message history per session (visitor + twin + real-human entries)
+* human-owned messages
+* human-control flags (3-way live chat takeover)
+* rolling memory objects per session
+
+All reads/writes are guarded by a global lock so concurrent chat streams
+cannot corrupt shared dicts.
+"""
+import os
+import time
+import datetime
+import threading
+from typing import Any, Dict, List, Optional
+
+from langchain_openai import ChatOpenAI
+
+from backend.app.services.memory_manager import RollingMemory
+
+
+def _default_summary_llm():
+    return ChatOpenAI(
+        temperature=0.0,
+        model=os.getenv("MEMORY_SUMMARY_MODEL", "gpt-4.1-nano"),
+        api_key=os.getenv("OPENAI_API_KEY"),
+    )
+
+
+class SessionStore:
+    """Single ownership point for every in-memory session artifact."""
+
+    def __init__(self, summary_llm_factory: Any = None):
+        self._lock = threading.RLock()
+        self._history: Dict[str, List[Dict[str, Any]]] = {}
+        self._human_messages: Dict[str, List[Dict[str, Any]]] = {}
+        self._human_control: Dict[str, bool] = {}
+        self._memories: Dict[str, RollingMemory] = {}
+        self._summary_llm_factory = summary_llm_factory or _default_summary_llm
+
+    @property
+    def liveness(self) -> int:
+        return len(self._memories)
+
+    # ------------------------------------------------------------------ #
+    # Message history
+    # ------------------------------------------------------------------ #
+    def record_message(self, session_id, sender, text, name="", thoughts=None):
+        with self._lock:
+            now_str = datetime.datetime.now().strftime("%I:%M %p")
+            entry = {
+                "id": f"msg_{sender}_{int(time.time() * 1000)}_{len(self._history.get(session_id, []))}",
+                "sender": sender,
+                "name": name or (
+                    "Arun Yadav" if sender == "human_arun"
+                    else "Arun's AI Assistant" if sender == "twin"
+                    else "You"
+                ),
+                "text": text,
+                "timestamp": now_str,
+            }
+            if thoughts:
+                entry["thoughts"] = thoughts
+
+            if session_id not in self._history:
+                self._history[session_id] = []
+            self._history[session_id].append(entry)
+            return entry
+
+    def get_history(self, session_id: str) -> List[Dict[str, Any]]:
+        with self._lock:
+            return list(self._history.get(session_id, []))
+
+    def get_human_messages(self, session_id: str) -> List[Dict[str, Any]]:
+        with self._lock:
+            return list(self._human_messages.get(session_id, []))
+
+    def append_human_message(self, session_id: str, entry: Dict[str, Any]) -> None:
+        with self._lock:
+            self._human_messages.setdefault(session_id, []).append(entry)
+
+    def get_human_entries(self, session_id: str) -> List[Dict[str, Any]]:
+        """Channel messages authored by the real human within this session."""
+        with self._lock:
+            return [m for m in self._history.get(session_id, []) if m.get("sender") == "human_arun"]
+
+    def get_last_user_message(self, session_id: str) -> str:
+        with self._lock:
+            for m in reversed(self._history.get(session_id, [])):
+                if m.get("sender") == "user":
+                    return m.get("text", "")
+        return ""
+
+    # ------------------------------------------------------------------ #
+    # Human-control (3-way live takeover)
+    # ------------------------------------------------------------------ #
+    def is_human_control(self, session_id: str) -> bool:
+        with self._lock:
+            return bool(self._human_control.get(session_id, False))
+
+    def set_human_control(self, session_id: str, enabled: bool) -> None:
+        with self._lock:
+            self._human_control[session_id] = enabled
+
+    # ------------------------------------------------------------------ #
+    # Rolling memory per session
+    # ------------------------------------------------------------------ #
+    def get_or_create_memory(self, session_id: str) -> RollingMemory:
+        with self._lock:
+            if session_id not in self._memories:
+                self._memories[session_id] = RollingMemory(
+                    summary_llm=self._summary_llm_factory(),
+                )
+            return self._memories[session_id]
+
+
+# Backwards-compatible global store (used by the FastAPI layer).
+session_store = SessionStore()

backend/app/services/tenant_service.py

@@ -0,0 +1,99 @@
+import os
+import json
+from typing import Optional, Dict, Any
+from backend.app.schemas.tenant import (
+    BrandConfig, AgentConfig, ChatConfig, VoiceConfig, SEOConfig, SocialConfig, TenantFullConfig
+)
+
+
+class TenantService:
+    def __init__(self, base_dir: Optional[str] = None):
+        if base_dir:
+            self.base_dir = base_dir
+        else:
+            self.base_dir = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
+        self.tenants_dir = os.path.join(self.base_dir, "tenants")
+
+    def get_tenant_path(self, tutor_id: str) -> str:
+        clean_id = (tutor_id or "arun").strip().lower()
+        path = os.path.join(self.tenants_dir, clean_id)
+        if os.path.exists(path):
+            return path
+        return os.path.join(self.tenants_dir, "tenant_starter")
+
+    def load_tenant_config(self, tutor_id: Optional[str] = None) -> TenantFullConfig:
+        clean_id = (tutor_id or "arun").strip().lower()
+        if clean_id in ("arun", "default", "none"):
+            clean_id = "tenant_starter"
+
+        tenant_path = self.get_tenant_path(clean_id)
+        resolved_id = os.path.basename(tenant_path)
+        config_dir = os.path.join(tenant_path, "config")
+
+        # Load split JSON configs
+        brand_data = self._read_json(os.path.join(config_dir, "brand.json"), {"tutor_id": resolved_id, "name": "Arun Yadav", "title": "Arun's AI Assistant", "role": "AI Systems Architect • Healthcare & Education", "avatar_url": "/profile_photo.png", "logo_url": "/logo.jpg", "primary_color": "#6366f1", "accent_color": "#818cf8", "theme": "dark", "cta_text": "Consult Arun"})
+        agent_data = self._read_json(os.path.join(config_dir, "agent.json"), {"tutor_id": resolved_id, "system_prompt": "You are Arun's AI Assistant.", "temperature": 0.3, "model": "gpt-4o", "enabled_tools": ["search_courses", "book_calendar", "faq_lookup"], "guardrails": []})
+        chat_data = self._read_json(os.path.join(config_dir, "chat.json"), {"tutor_id": resolved_id, "welcome_message": "Hi! I'm Arun's AI Assistant.", "suggested_questions": ["What projects has Arun built?", "How can I contact Arun?"]})
+        voice_data = self._read_json(os.path.join(config_dir, "voice.json"), {"tutor_id": resolved_id, "voice_id": "alloy", "model": "tts-1", "speed": 1.0})
+        seo_data = self._read_json(os.path.join(config_dir, "seo.json"), {"tutor_id": resolved_id, "meta_title": "Arun Core", "meta_description": "AI Platform", "keywords": []})
+        social_data = self._read_json(os.path.join(config_dir, "social.json"), {"tutor_id": resolved_id, "website": "https://neuralarun.in"})
+
+        brand_data["tutor_id"] = resolved_id
+        agent_data["tutor_id"] = resolved_id
+        chat_data["tutor_id"] = resolved_id
+        voice_data["tutor_id"] = resolved_id
+        seo_data["tutor_id"] = resolved_id
+        social_data["tutor_id"] = resolved_id
+
+        return TenantFullConfig(
+            tutor_id=resolved_id,
+            brand=BrandConfig(**brand_data),
+            agent=AgentConfig(**agent_data),
+            chat=ChatConfig(**chat_data),
+            voice=VoiceConfig(**voice_data),
+            seo=SEOConfig(**seo_data),
+            social=SocialConfig(**social_data),
+        )
+
+    def _read_json(self, filepath: str, fallback: Dict[str, Any]) -> Dict[str, Any]:
+        if os.path.exists(filepath):
+            try:
+                with open(filepath, "r", encoding="utf-8") as f:
+                    return json.load(f)
+            except Exception as e:
+                print(f"[TENANT_SERVICE WARNING] Failed to parse {filepath}: {e}")
+        return fallback
+
+    # ------------------------------------------------------------------ #
+    # Legacy demos-dictionary loader (pre-dates split configs).           #
+    # Kept here so both config systems resolve in one service and the     #
+    # agent loop reuses it for `?tutor=<id>` responses.                   #
+    # ------------------------------------------------------------------ #
+    def load_legacy_tutor_config(self, tutor_id: Optional[str]) -> Optional[Dict[str, Any]]:
+        slug = (tutor_id or "arun").strip().lower()
+
+        candidate_paths = [
+            os.path.join(self.base_dir, "demos", f"{slug}_enterprise_dictionary.json"),
+            os.path.join(self.base_dir, "demos", f"{slug}.json"),
+            os.path.join(self.base_dir, "data", "leads", f"{slug}_enterprise_dictionary.json"),
+            os.path.join(self.base_dir, "data", "leads", f"{slug}.json"),
+            os.path.join(self.base_dir, "demos", "general.json"),
+        ]
+
+        for path in candidate_paths:
+            if os.path.exists(path):
+                try:
+                    with open(path, "r", encoding="utf-8") as f:
+                        data = json.load(f)
+                        if data.get("client_id") == "{{TUTOR_ID}}":
+                            data["client_id"] = slug
+                            data["tutor_id"] = slug
+                        return data
+                except Exception as e:
+                    print(f"[LOAD TUTOR CONFIG ERROR] Failed reading {path}: {e}")
+
+        return None
+
+
+# Singleton instance helper
+tenant_service = TenantService()

backend/app/services/tool_executor.py

@@ -0,0 +1,92 @@
+"""Config-driven tool registry for the ArunCore agent.
+
+Every tool the LLM may call is defined here and exposed through `ToolExecutor`.
+A tenant's `enabled_tools` array (agent.json) simply selects which tools get
+bound into the LLM execution loop — no Python edits required to add a client.
+"""
+from typing import Dict, Any, List, Optional
+
+from langchain_core.tools import tool
+
+from backend.app.services.knowledge_service import knowledge_service
+from backend.app.services.notification_service import schedule_notify_arun
+
+
+# ---------------------------------------------------------------------- #
+# Legacy placeholder tools (kept for tenant agent.json enabled_tools and  #
+# the demo dictionary path; not bound by the core Arun twin agent).       #
+# ---------------------------------------------------------------------- #
+@tool
+def search_courses(query: str) -> str:
+    """Search tenant course catalog and curriculum."""
+    return f"Retrieved course details for query: '{query}'."
+
+
+@tool
+def book_calendar(date: str, topic: str) -> str:
+    """Book a consultation session or mentorship call."""
+    return f"Consultation session requested for '{topic}' on {date}."
+
+
+@tool
+def faq_lookup(question: str) -> str:
+    """Search tenant FAQ knowledge base."""
+    return f"Retrieved verified FAQ answers for: '{question}'."
+
+
+# ---------------------------------------------------------------------- #
+# ArunCore real agent tools                                              #
+# ---------------------------------------------------------------------- #
+@tool
+def search_arun_knowledge(query: str) -> str:
+    """Search Arun's local knowledge base for information about his projects, architecture, philosophy, and background."""
+    return knowledge_service.search(query)
+
+
+@tool
+def get_github_live_data(username: str = "neural-arun") -> str:
+    """Fetch live GitHub repository data and recent commits for Arun Yadav."""
+    return knowledge_service.fetch_live_github(username)
+
+
+@tool
+def notify_arun(category: str, user_input: str, user_metadata_json: str = "") -> str:
+    """Send an instant Telegram alert to Arun's phone ONLY when a visitor explicitly asks to hire, consult, or contact Arun (LEAD), asks an unknown technical question (UNKNOWN_QUESTION), or requests urgent assistance (URGENT). Do NOT call this tool for general questions or identity questions like 'who are you'."""
+    schedule_notify_arun(category, user_input, user_metadata_json)
+    return (
+        f"Successfully sent Telegram alert to Arun's phone (Category: {category.upper()}).\n"
+        "YOU MUST NOW OUTPUT ARUN'S DIRECT CONTACT DETAILS IN BULLET POINTS:\n"
+        "- 📞 Phone: +91 8881109193\n"
+        "- 💬 WhatsApp: https://wa.me/918881109193\n"
+        "- ✉️ Email: neural.arun.dev@gmail.com\n"
+        "- 💼 LinkedIn: https://www.linkedin.com/in/arun-yadav-768052368\n"
+        "- 🌐 GitHub: https://github.com/neural-arun"
+    )
+
+
+class ToolExecutor:
+    """Dynamic, config-driven tool registry bound into the LLM loop."""
+
+    DEFAULT_TOOLS = ["search_arun_knowledge", "get_github_live_data", "notify_arun"]
+
+    AVAILABLE_TOOLS = {
+        "search_courses": search_courses,
+        "book_calendar": book_calendar,
+        "faq_lookup": faq_lookup,
+        "search_arun_knowledge": search_arun_knowledge,
+        "get_github_live_data": get_github_live_data,
+        "notify_arun": notify_arun,
+    }
+
+    @classmethod
+    def get_enabled_tools(cls, enabled_tool_names: Optional[List[str]]) -> List[Any]:
+        names = enabled_tool_names or cls.DEFAULT_TOOLS
+        tools: List[Any] = []
+        for name in names:
+            if name in cls.AVAILABLE_TOOLS:
+                tools.append(cls.AVAILABLE_TOOLS[name])
+        return tools
+
+    @classmethod
+    def get_tool_map(cls, enabled_tool_names: Optional[List[str]]) -> Dict[str, Any]:
+        return {t.name: t for t in cls.get_enabled_tools(enabled_tool_names)}

backend/app/services/voice_service.py

@@ -0,0 +1,39 @@
+import os
+import io
+import requests
+from typing import Optional
+
+
+class VoiceService:
+    @staticmethod
+    def generate_tts_audio(text: str, voice: str = "alloy") -> bytes:
+        openai_key = os.getenv("OPENAI_API_KEY")
+        if not openai_key:
+            raise ValueError("OPENAI_API_KEY environment variable missing.")
+
+        clean_text = (
+            text.replace("*", "")
+            .replace("#", "")
+            .replace("`", "")
+            .replace("\n", " ")
+            [:1000]
+        )
+
+        res = requests.post(
+            "https://api.openai.com/v1/audio/speech",
+            headers={
+                "Authorization": f"Bearer {openai_key}",
+                "Content-Type": "application/json",
+            },
+            json={
+                "model": "tts-1",
+                "input": clean_text,
+                "voice": voice or "alloy",
+            },
+            timeout=15,
+        )
+
+        if res.status_code == 200:
+            return res.content
+        else:
+            raise RuntimeError(f"OpenAI TTS API error {res.status_code}: {res.text}")

data/HOW_TO_UPDATE_DATA.md

@@ -0,0 +1,51 @@
+# How to Update ArunCore Knowledge Data
+
+ArunCore uses a modular single-source-of-truth structure for your project data.
+
+---
+
+## 📁 Data Structure Overview
+
+```
+data/
+├── static/
+│   ├── public_profile.md         # Your core identity, principles, vision & loop
+│   └── rules_of_engagement.md   # LLM guidelines & zero-hallucination rules
+│
+└── github/                        # ONE clean folder per GitHub project
+    ├── <project_name>/
+    │   ├── README.md              # Project overview, tech stack, & direct GitHub URL link
+    │   └── metadata.json          # Structured metadata (URL, stack, stars, topics)
+    └── ...
+```
+
+---
+
+## 🚀 How to Add or Update a Project
+
+### Option A: Automatic GitHub Sync (Recommended)
+Run the automated sync script to pull your latest READMEs and GitHub URLs directly from your GitHub profile (`neural-arun`):
+
+```bash
+python scripts/sync_github_data.py
+```
+
+### Option B: Manual Add
+1. Create a new folder under `data/github/<your_repo_name>/`.
+2. Place your project `README.md` and `metadata.json` inside it.
+3. Make sure `README.md` starts with a link header:
+   ```markdown
+   > **GitHub Repository:** [https://github.com/neural-arun/your_repo_name](https://github.com/neural-arun/your_repo_name)
+   ```
+
+---
+
+## 🔄 Re-Ingest into Vector DB
+
+After adding or modifying files in `data/`:
+
+```bash
+python core/ingest.py
+```
+
+`ingest.py` incrementally updates ChromaDB so your agent instantly gains access to the new knowledge.

data/README.md

@@ -0,0 +1,37 @@
+# 📊 ArunCore Data Directory (`data/`)
+
+This directory contains the knowledge base, static profile documents, raw backgrounds, and GitHub repository README files used for retrieval-augmented generation.
+
+---
+
+## 📁 Data Subdirectories & Files
+
+```text
+data/
+├── github/                    # 21 subdirectories containing raw README.md files for each repo
+│   ├── aruncore/              # ArunCore RAG Engine README
+│   ├── legal_RAG_system/      # Legal RAG System README
+│   ├── med_coach/             # MedCoach Clinical Reasoning README
+│   ├── neet-bot/              # NEET Medical Bot README
+│   ├── real_state_listing_scraper/ # 99acres Scraper README
+│   └── ... (21 total public repositories)
+├── linkedin/
+│   ├── posts.md               # Scraped public LinkedIn posts & technical insights
+│   └── profile_summary.md     # Public LinkedIn profile summary
+├── raw/
+│   ├── personal_background.md # Deeper origin story, NEET pivot, JEE journey, & working style
+│   └── unknown_questions.json # Verified Q&A store for active learning alerts
+└── static/
+    ├── public_profile.md      # Master public technical profile & architecture specs
+    ├── rules_of_engagement.md # Steering rules, search mandates, & guardrails
+    └── voice_persona.md       # Custom sharp & witty voice persona definition
+```
+
+---
+
+## 🔄 Auto-Syncing Data
+
+- **Sync GitHub READMEs**: `python3 scripts/sync_github.py`
+- **Sync LinkedIn Posts**: `python3 scripts/sync_linkedin.py`
+- **1-Click Master Sync**: `python3 scripts/sync_all.py`
+- **Vector Ingestion**: `python3 scripts/ingest.py`

data/github/01_manage_patient_task/README.md

@@ -0,0 +1,87 @@
+---
+project_name: 01_manage_patient_task
+github_url: https://github.com/neural-arun/01_manage_patient_task
+language: Python
+stars: 0
+topics: [None]
+updated_at: 2026-06-28T11:34:30Z
+---
+
+# 01_manage_patient_task
+
+> **GitHub Repository:** [https://github.com/neural-arun/01_manage_patient_task](https://github.com/neural-arun/01_manage_patient_task)  
+> **Primary Language:** Python | **Stars:** 0 | **Forks:** 0  
+> **Description:** No description provided.
+
+---
+
+# Patient Task Management API
+
+**Build 1** of the FastAPI for Healthcare + Medical Education Systems roadmap.
+
+A CRUD API for managing patient tasks — medication reminders, daily activities, and completion tracking — built with FastAPI.
+
+## Features
+
+- Create a patient task
+- List all tasks (with optional status filtering)
+- Get a single task by ID
+- Update a task
+- Mark a task as complete
+- Delete a task
+
+## Tech Stack
+
+- **FastAPI** — web framework
+- **Uvicorn** — ASGI server
+- **Pydantic** — request/response validation
+- **Python 3.10+**
+
+## Project Structure
+
+```
+01_manage_patient_task/
+├── main.py              # App entry point & route definitions
+├── models.py            # Pydantic schemas
+├── notes/               # Study notes (learning reference)
+├── requirements.txt     # Dependencies
+└── README.md
+```
+
+## Setup
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+## Run
+
+```bash
+uvicorn main:app --reload
+```
+
+- API: `http://127.0.0.1:8000`
+- Docs: `http://127.0.0.1:8000/docs`
+- ReDoc: `http://127.0.0.1:8000/redoc`
+
+## API Endpoints
+
+| Method   | Endpoint                 | Description        |
+|----------|--------------------------|--------------------|
+| `POST`   | `/tasks`                 | Create a task      |
+| `GET`    | `/tasks`                 | List tasks         |
+| `GET`    | `/tasks/{task_id}`       | Get a task         |
+| `PUT`    | `/tasks/{task_id}`       | Update a task      |
+| `PATCH`  | `/tasks/{task_id}/complete` | Mark complete   |
+| `DELETE` | `/tasks/{task_id}`       | Delete a task      |
+
+## Topics Covered
+
+- FastAPI fundamentals & ASGI
+- Routing (GET, POST, PUT, PATCH, DELETE)
+- Path & query parameters
+- Request body & Pydantic validation
+- Response models & status codes
+- Error handling with HTTPException

data/github/01_manage_patient_task/metadata.json

@@ -0,0 +1,10 @@
+{
+    "name": "01_manage_patient_task",
+    "github_url": "https://github.com/neural-arun/01_manage_patient_task",
+    "description": "No description provided.",
+    "language": "Python",
+    "stars": 0,
+    "forks": 0,
+    "topics": [],
+    "updated_at": "2026-06-28T11:34:30Z"
+}

data/github/Agentic_AI_Projects/README.md

@@ -0,0 +1,21 @@
+---
+project_name: Agentic_AI_Projects
+github_url: https://github.com/neural-arun/Agentic_AI_Projects
+language: Python
+stars: 0
+topics: [None]
+updated_at: 2026-03-22T07:20:56Z
+---
+
+# Agentic_AI_Projects
+
+> **GitHub Repository:** [https://github.com/neural-arun/Agentic_AI_Projects](https://github.com/neural-arun/Agentic_AI_Projects)  
+> **Primary Language:** Python | **Stars:** 0 | **Forks:** 0  
+> **Description:** No description provided.
+
+---
+
+*No README.md file found in this repository on GitHub.*
+
+## Overview
+No description provided.

data/github/Agentic_AI_Projects/metadata.json

@@ -0,0 +1,10 @@
+{
+    "name": "Agentic_AI_Projects",
+    "github_url": "https://github.com/neural-arun/Agentic_AI_Projects",
+    "description": "No description provided.",
+    "language": "Python",
+    "stars": 0,
+    "forks": 0,
+    "topics": [],
+    "updated_at": "2026-03-22T07:20:56Z"
+}

data/github/ArunCore/README.md

@@ -0,0 +1,16 @@
+# ArunCore — Production Personal AI Assistant & Portfolio
+
+> **GitHub Repository:** [https://github.com/neural-arun/ArunCore](https://github.com/neural-arun/ArunCore)  
+> **Live Web Application:** [https://aruncore.vercel.app](https://aruncore.vercel.app)
+
+ArunCore is an agentic, stateful personal AI twin built for **Arun Yadav** (AI Systems Architect specializing in Healthcare & Education).
+
+---
+
+## Key Features
+- **100% Automated Telegram Alerts**: Every visitor interaction automatically sends a notification to Arun's phone.
+- **1-Click Magic Link 3-Way Real Human Takeover**: Enables Arun to join ongoing web chat sessions live from Telegram via a 1-click magic link.
+- **Vercel Serverless Egress Relay**: Routes Telegram API traffic through Vercel serverless routes (`/api/telegram`) to eliminate firewall timeouts.
+- **Active Learning Vector Store**: Stores verified human answers from Telegram replies into `data/raw/unknown_questions.json` and re-indexes ChromaDB.
+- **Zero-Hallucination RAG**: Dense vector retrieval + BM25 keyword search + Cohere V3 Reranker.
+- **Dynamic Language Rules**: Responds in 100% articulate English for English queries, matching user tone cleanly.