Remove excess documentation and CLI scripts to keep HF repo lean

NITDAA_ARCHITECTURE_DESIGN.md

@@ -1,32 +0,0 @@
-# Nitdaa Sibling Project - Architecture Design
-**Version 1.0 Public Release**
-
-## 1. System Overview
-Nitdaa is the mobile-friendly MVP sibling to the full HealthExpert enterprise platform. It removes administrative overhead, heavy RBAC UI components, and complex multi-pane layouts in favor of a lean, mobile-first edge interface.
-
-## 2. Core Differences from Base HealthExpert
-- **UI/UX**: Transitioned from a 3-pane desktop layout to a single-pane vertical mobile layout, featuring dynamic inline controls (e.g. the bottom input panel temporarily transforms into a 5-star & thumbs feedback interface after response generation).
-- **Admin Constraints**: Admin controls (Kill Switch, DB Purge) are entirely hidden in the HTML to prevent public tampering.
-- **Session Audit**: Uses `nitdaa_sessions.json` and `nitdaa_summary.json` for granular system telemetry rather than database-backed logs. Captures granular feedback (1-5 stars, thumbs, and text) instantly upon user action.
-- **Resumable Event Streaming**: The architecture uses an asynchronous Job ID system (`/api/query/start` and `/api/query/stream`) to survive mobile background disconnections.
-- **Concurrent RAG Isolation**: Vector and Graph retrieval pipelines operate inside a Thread Pool with thread-local database connections. Heavy PyTorch inference layers (CrossEncoder and SentenceTransformer) are strictly serialized via threading locks to prevent `OutOfMemory` crashes on resource-constrained infrastructure, while keeping I/O concurrency high.
-
-## 3. Data Flow
-1. **Startup & Sync**: Flask initializes. `sync_he_data()` runs asynchronously to pull the latest dataset from `Sam-max1/he-data`. If differences are found, it gracefully purges vector/graph databases and rebuilds. Concurrently, it performs a 2-way remote synchronization of logs with `Sam-max1/mat_data`.
-2. **Security & Connection**: Client connections face an immediate Math CAPTCHA. All API calls pass through `Flask-Limiter` and Strict HTTP Content Security Policies.
-3. **Ingestion**: If new files are detected, `process_document_pipeline` splits them into 512-token chunks, vectorizes them into ChromaDB, and extracts graph entities locally using `spaCy`.
-4. **Retrieval (Concurrent)**: When a query hits `/api/query/start`, an asynchronous thread pool handles Tri-modal Hybrid Retrieval (Vector + BM25 + Graph) allowing multiple users to fetch context simultaneously.
-5. **Synthesis & Feedback**: A Cross-Encoder reranks the chunks, and the context is fed into the local LLM generation queue. The frontend streams events using Server-Sent Events with offset recovery. Upon completion, users optionally submit ratings securely stored in `nitdaa_summary.json`.
-
-## 4. Resource Allocation
-Nitdaa is tuned for the HuggingFace Spaces free tier:
-- **vCPU constraints**: Threads are explicitly clamped to prevent CPU thrashing.
-- **Vector Limit**: A hard limit of 10,000 chunks is enforced globally to prevent `OutOfMemory` crashes.
-- **Session Quotas**: Max 5 file uploads per session to prevent abuse.
-
-## 5. Project Intelligence & Deployment
-- **HuggingFace Space**: [Sam-max1/nitdaa](https://huggingface.co/spaces/Sam-max1/nitdaa)
-- **HuggingFace ID**: `sam-max1`
-- **Authentication**: `HF_TOKEN` is provided via the system keyring for secure access to HuggingFace resources and pushing updates.
-- **Deployment Protocol**: Code base updates are pushed directly to the HuggingFace Space repository. Note: Core functionality should not be altered, deleted, or impaired unless explicitly requested.
-- **Minimalist Deployment**: Nitdaa must always contain only the bare minimum files required to run the application. Extraneous resources, local databases, or unneeded assets must be excluded from tracking to ensure only the bare minimum is pushed to HuggingFace.

NITDAA_ARCHITECTURE_DESIGN.pdf

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d06a069f37aa6efe19cc97d445336a38689930c94aabb1e059948b90238ed5c4
-size 72541

NITDAA_HEALTHEXPERT_USER_GUIDE.md

@@ -1,36 +0,0 @@
-# Nitdaa HealthExpert Mobile MVP - User Guide
-**Version 1.0 Public Release**
-
-## Overview
-Nitdaa HealthExpert is a mobile-optimized, single-pane web interface designed to provide users with direct access to the HealthExpert AI knowledge base. Deployed on HuggingFace Spaces, Nitdaa brings the power of Agentic AI, Hybrid RAG, and Edge Compute into a sandboxed, responsive environment.
-
-## 1. Accessing the Application
-- Navigate to the official public application link: [https://sam-max1-nitdaa.hf.space/](https://sam-max1-nitdaa.hf.space/)
-- **Security Check (CAPTCHA)**: Upon your first visit, a security modal will challenge you with a simple math problem (e.g. `5 + 3 = ?`). You must correctly solve this problem to unlock the application dashboard.
-- The application will then load automatically. Note that the first boot may take 3-5 minutes as the system pulls the latest knowledge datasets securely from the cloud.
-- Once the **Status Pill** in the top-right corner turns **Green** (Ready), the system is online.
-
-## 2. Using the Dashboard
-The dashboard is designed for mobile efficiency.
-- **Top Header**: Displays the application branding ("Healthcare Policy Expert") and current status.
-- **Scrollable Output Window**: The large central area displays the conversation history. It automatically scrolls down as new answers are generated.
-- **Input Area**: Located at the bottom of the screen.
-
-## 3. Asking Questions
-You can query the system in two ways:
-1. **Quick Questions Dropdown**: Tap the dropdown menu above the text input to select a preset question. Selecting a question will automatically submit it to the AI.
-2. **Manual Input**: Type your query directly into the text box and press the **Send** button.
-
-## 4. Understanding Output
-When a question is submitted:
-- Your question will instantly appear in the output window.
-- The system will display a **"Generating answer.. (1.2s)"** timer.
-- Once context is retrieved from the Vector and Graph databases, the LLM will stream the Markdown-formatted answer back to the screen.
-- Analysis may take upto 300 seconds due to slow shared resources. Please keep patience.
-- **Providing Feedback**: Once an answer completes, the bottom input area will temporarily switch to a Feedback panel. You will see Thumbs Up (👍) / Thumbs Down (👎) buttons alongside a 5-star rating widget (where colors transition from red to green). Selecting an icon captures your feedback immediately. You may optionally enter text feedback and click **Submit**, or click **Cancel** to revert the panel back to the standard question box.
-- Responses remain preserved in the scrollable window for the duration of your session.
-
-## 5. Security & Session Handling
-- **Multi-tenant Isolation**: Your conversation is strictly isolated from other users using an invisible `X-Session-Token`. Multiple users can query the system concurrently without interference.
-- **Resumable Mobile Sessions**: If your mobile device goes to sleep or you navigate to another app, the system safely buffers your answer. When you return, the streaming will seamlessly resume without missing a word.
-- **Session Expiration**: If the page is reloaded, or if you are inactive for an extended period, your session token resets, and previous context is cleared to ensure privacy.

NITDAA_HEALTHEXPERT_USER_GUIDE.pdf

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:0107e7f398e33e944638cc391375987e0eaa82cfd50f2e15525c94be98fad98e
-size 65952

NITDAA_HF_DEPLOYMENT_GUIDE.md

@@ -1,36 +0,0 @@
-# Nitdaa Sibling Project - HF Deployment Guide
-**Version 1.0 Public Release**
-
-## Overview
-This document outlines the specific steps and constraints required to deploy the Nitdaa mobile sibling to HuggingFace Spaces free tier.
-
-## 1. Prerequisites
-- A HuggingFace account.
-- A HuggingFace Space configured with the **Docker** SDK.
-- Hardware allocation: Free Tier (2 vCPU, 16GB Disk, 12GB RAM).
-
-## 2. HuggingFace Secrets Configuration
-To enable dynamic Dataset Synchronization, you must configure the following variable:
-- `HF_TOKEN`: A HuggingFace access token with "Read" and "Write" permissions to the `Sam-max1/he-data` and `Sam-max1/mat_data` datasets. This token is available via the system keyring or environment variables.
-- `ADMIN_MODE`: Set to `0` to completely lock down the UI for public usage.
-
-## 2.1 Minimalist Deployment Rule
-Nitdaa must always contain **only the bare minimum files required** to run the application in HF and the bare minimum files to be pushed to HF. Exclude all extraneous assets, development scripts, or unnecessary documentation from the Hugging Face repository.
-
-## 3. Deployment Steps
-1. Clone the `nitdaa` folder locally.
-2. Initialize a git repository pointing to your HF Space remote.
-3. Use the HuggingFace CLI to upload large directories without timing out:
-   ```bash
-   huggingface-cli upload <your_username>/nitdaa . . --repo-type space
-   ```
-4. HuggingFace will automatically detect the `Dockerfile` in the root and begin building.
-
-## 4. Cold Boot Times
-Due to the CPU-bound nature of the free tier:
-- **Docker Build**: ~5-8 minutes (first boot only).
-- **Dataset Sync**: Variable depending on the size of `he_data`.
-- **Model Load**: The `Qwen3.5-2B` GGUF model and PyTorch embeddings take ~1-2 minutes to spin up into memory on boot.
-
-## 5. Security Safeguards
-Nitdaa implements headless API blocking. Any direct programmatic access to the REST endpoints (without proper CORS headers or session tokens) will automatically return `403 Forbidden` to prevent bot-net abuse on the public URL.

docker-compose.yml

@@ -1,29 +0,0 @@
-services:
-  neo4j:
-    image: neo4j:5.18-community
-    container_name: healthexpert-neo4j
-    ports:
-      - "7474:7474"   # Neo4j Browser UI
-      - "7687:7687"   # Bolt protocol
-    environment:
-      - NEO4J_AUTH=neo4j/healthexpert
-      - NEO4J_server_memory_heap_initial__size=512m
-      - NEO4J_server_memory_heap_max__size=1G
-      - NEO4J_PLUGINS=["apoc"]
-      - NEO4J_dbms_security_procedures_unrestricted=apoc.*
-      - NEO4J_dbms_security_procedures_allowlist=apoc.*
-    volumes:
-      - neo4j_data:/data
-      - neo4j_logs:/logs
-      - neo4j_import:/var/lib/neo4j/import
-    healthcheck:
-      test: ["CMD", "wget", "-q", "--spider", "http://localhost:7474"]
-      interval: 10s
-      timeout: 5s
-      retries: 10
-    restart: unless-stopped
-
-volumes:
-  neo4j_data:
-  neo4j_logs:
-  neo4j_import:

healthexpert.py

@@ -1,111 +0,0 @@
-#!/usr/bin/env python3
-"""
-Document AI Expert — Standalone CLI
-=========================================
-Usage:
-  python healthexpert.py ingest <file_path>
-  python healthexpert.py query "<question>"
-  python healthexpert.py list
-  python healthexpert.py clear <source_name>
-  python healthexpert.py status
-"""
-from __future__ import annotations
-import sys, os
-sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
-
-import argparse
-import config
-from pipeline import vector_store, graph_store, embedder, document_loader, chunker
-from agents.crew import run_ingest_crew, run_query_crew
-
-
-def cmd_ingest(file_path: str) -> None:
-    if not os.path.exists(file_path):
-        print(f"[ERROR] File not found: {file_path}")
-        sys.exit(1)
-    print(f"[INGEST] Processing: {file_path}")
-    result = run_ingest_crew(file_path)
-    print(f"\n[RESULT]\n{result}")
-
-
-def cmd_query(question: str) -> None:
-    if vector_store.count() == 0:
-        print("[ERROR] No documents ingested. Run: python healthexpert.py ingest <file>")
-        sys.exit(1)
-    print(f"[QUERY] {question}\n")
-    answer = run_query_crew(question)
-    print("\n" + "=" * 60)
-    print(answer)
-    print("=" * 60)
-
-
-def cmd_list() -> None:
-    docs = vector_store.list_documents()
-    if not docs:
-        print("[INFO] No documents ingested yet.")
-        return
-    print(f"\n{'SOURCE':<40} {'TYPE':<10}")
-    print("-" * 52)
-    for d in docs:
-        print(f"{d['source']:<40} {d['file_type']:<10}")
-    print(f"\nTotal: {len(docs)} document(s), {vector_store.count()} chunk(s)")
-
-
-def cmd_clear(source_name: str) -> None:
-    n = vector_store.delete_document(source_name)
-    graph_store.delete_source(source_name)
-    print(f"[CLEAR] Deleted {n} chunks for '{source_name}'")
-
-
-def cmd_status() -> None:
-    print("\n── Document AI Expert Status ──")
-    print(f"  LLM Endpoint : {config.LLM_BASE_URL}")
-    print(f"  LLM Model    : {config.LLM_MODEL_ID}")
-    print(f"  Embedding    : {config.EMBEDDING_MODEL} ({config.EMBEDDING_DEVICE})")
-    print(f"  Vector DB    : {vector_store.count()} chunks  [{config.WEAVIATE_URL}]")
-    g = graph_store.get_stats()
-    if g.get("available"):
-        print(f"  Graph DB     : {g['nodes']} nodes, {g['relationships']} relationships")
-    else:
-        print("  Graph DB     : OFFLINE (vector-only mode)")
-    print()
-
-
-def main():
-    parser = argparse.ArgumentParser(
-        description="Document AI Expert — CLI",
-        formatter_class=argparse.RawDescriptionHelpFormatter,
-        epilog=__doc__,
-    )
-    sub = parser.add_subparsers(dest="command")
-
-    p_ingest = sub.add_parser("ingest", help="Ingest a document")
-    p_ingest.add_argument("file", help="Path to document file")
-
-    p_query = sub.add_parser("query", help="Ask a question")
-    p_query.add_argument("question", help="Question string")
-
-    sub.add_parser("list",   help="List ingested documents")
-    sub.add_parser("status", help="Show system status")
-
-    p_clear = sub.add_parser("clear", help="Remove a document")
-    p_clear.add_argument("source", help="Source filename to remove")
-
-    args = parser.parse_args()
-
-    if args.command == "ingest":
-        cmd_ingest(args.file)
-    elif args.command == "query":
-        cmd_query(args.question)
-    elif args.command == "list":
-        cmd_list()
-    elif args.command == "clear":
-        cmd_clear(args.source)
-    elif args.command == "status":
-        cmd_status()
-    else:
-        parser.print_help()
-
-
-if __name__ == "__main__":
-    main()

manage_db.py

@@ -1,93 +0,0 @@
-#!/usr/bin/env python3
-"""Manage HealthExpert databases.
-
-Database plan:
-  ┌─────────────────┬────────────────────────────────────────────────────┐
-  │ Database        │ Management                                          │
-  ├─────────────────┼────────────────────────────────────────────────────┤
-  │ ChromaDB        │ Embedded (in-process). No Docker container.        │
-  │  (Vector Store) │ Data stored at: data/chroma_db/                    │
-  │                 │ Managed by: pipeline/vector_store.py               │
-  │                 │ Use /api/admin/purge (UI) or wipe data/chroma_db/  │
-  ├─────────────────┼────────────────────────────────────────────────────┤
-  │ Kuzu            │ Embedded (in-process). No Docker container.        │
-  │  (Graph DB)     │ Data stored at: data/kuzu_db/                      │
-  │                 │ Managed by: pipeline/graph_store.py                │
-  │                 │ Use /api/admin/purge (UI) or wipe data/kuzu_db/    │
-  └─────────────────┴────────────────────────────────────────────────────┘
-
-Usage:
-  python manage_db.py          # show status of all databases
-  python manage_db.py -chroma  # show ChromaDB data directory info
-  python manage_db.py -kuzu    # show Kuzu data directory info
-"""
-import argparse
-import sys
-import os
-from pathlib import Path
-
-CHROMA_DIR   = Path(__file__).parent / "data" / "chroma_db"
-KUZU_DIR     = Path(__file__).parent / "data" / "kuzu_db"
-
-
-def _dir_info(path: Path) -> dict:
-    exists = path.exists()
-    size_mb = 0.0
-    file_count = 0
-    if exists:
-        for f in path.rglob("*"):
-            if f.is_file():
-                size_mb    += f.stat().st_size / 1024 ** 2
-                file_count += 1
-    return {"exists": exists, "path": str(path), "size_mb": size_mb, "files": file_count}
-
-
-def chroma_status() -> None:
-    info = _dir_info(CHROMA_DIR)
-    print("\n── ChromaDB (Vector Store) ──────────────────────────────────")
-    print(f"  Type      : Embedded (in-process, no server)")
-    print(f"  Data dir  : {info['path']}")
-    if info["exists"]:
-        print(f"  Status    : PRESENT  ({info['files']} files, {info['size_mb']:.2f} MB)")
-    else:
-        print(f"  Status    : EMPTY (will be created on first ingest)")
-    print("─" * 60)
-
-def kuzu_status() -> None:
-    info = _dir_info(KUZU_DIR)
-    print("\n── Kuzu (Graph DB) ──────────────────────────────────────────")
-    print(f"  Type      : Embedded (in-process, no server)")
-    print(f"  Data dir  : {info['path']}")
-    if info["exists"]:
-        print(f"  Status    : PRESENT  ({info['files']} files, {info['size_mb']:.2f} MB)")
-    else:
-        print(f"  Status    : EMPTY (will be created on first ingest)")
-    print("─" * 60)
-
-
-def status() -> None:
-    print("\n══════════════════════════════════════════════════════════════")
-    print("  HealthExpert — Database Status")
-    print("══════════════════════════════════════════════════════════════")
-    chroma_status()
-    kuzu_status()
-    print("")
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Manage HealthExpert databases",
-        formatter_class=argparse.RawDescriptionHelpFormatter,
-        epilog=__doc__,
-    )
-    parser.add_argument("-chroma", action="store_true", help="Show ChromaDB info")
-    parser.add_argument("-kuzu",   action="store_true", help="Show Kuzu DB info")
-    args = parser.parse_args()
-
-    if args.chroma: chroma_status()
-    elif args.kuzu: kuzu_status()
-    else: status()
-
-
-if __name__ == "__main__":
-    main()

manage_llm.py

@@ -1,159 +0,0 @@
-#!/usr/bin/env python3
-"""Manage HealthExpert LLM Microservices.
-
-This script allows you to spin up, spin down, and check the status
-of the local LLM inference endpoints (gen_llm.py and embed_llm.py).
-
-Usage:
-  python manage_llm.py status   # show status of LLM servers
-  python manage_llm.py up       # start both LLM servers
-  python manage_llm.py down     # stop both LLM servers
-"""
-import argparse
-import subprocess
-import time
-import os
-import sys
-from pathlib import Path
-import urllib.request
-
-GEN_PORT = 8002
-EMBED_PORT = 8003
-BASE_DIR = Path(__file__).parent.resolve()
-
-def _check_port(port: int) -> bool:
-    """Check if a port is actively listening by making a simple HTTP request."""
-    try:
-        # Just a healthcheck to see if server responds, we expect 404 or 200
-        req = urllib.request.Request(f"http://127.0.0.1:{port}/")
-        urllib.request.urlopen(req, timeout=1)
-        return True
-    except urllib.error.URLError as e:
-        # If it's an HTTPError (e.g. 404 Not Found), the server is alive
-        if hasattr(e, 'code'):
-            return True
-        # ConnectionRefusedError usually means nothing is listening
-        return False
-    except Exception:
-        return False
-
-def _kill_port(port: int) -> None:
-    """Kill any process listening on the given port."""
-    try:
-        # Using fuser to kill processes on the port
-        subprocess.run(["fuser", "-k", f"{port}/tcp"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
-    except FileNotFoundError:
-        try:
-            # Fallback to lsof if fuser is not available
-            pids = subprocess.check_output(["lsof", "-t", f"-i:{port}"]).decode().strip().split('\n')
-            for pid in pids:
-                if pid:
-                    subprocess.run(["kill", "-9", pid])
-        except Exception:
-            pass
-
-def status() -> None:
-    """Show the status of the LLM servers."""
-    gen_alive = _check_port(GEN_PORT)
-    embed_alive = _check_port(EMBED_PORT)
-
-    print("\n══════════════════════════════════════════════════════════════")
-    print("  HealthExpert — LLM Microservices Status")
-    print("══════════════════════════════════════════════════════════════")
-    
-    print("\n── Generation LLM (agents/gen_llm.py) ───────────────────────")
-    print(f"  Port      : {GEN_PORT}")
-    print(f"  Status    : {'🟢 RUNNING' if gen_alive else '🔴 STOPPED'}")
-    
-    print("\n── Embedding LLM (agents/embed_llm.py) ──────────────────────")
-    print(f"  Port      : {EMBED_PORT}")
-    print(f"  Status    : {'🟢 RUNNING' if embed_alive else '🔴 STOPPED'}")
-    print("─" * 62)
-    print("")
-
-def down() -> None:
-    """Stop the LLM servers."""
-    print("Stopping LLM services...")
-    _kill_port(GEN_PORT)
-    _kill_port(EMBED_PORT)
-    
-    # Also kill by script name as a fallback
-    subprocess.run(["pkill", "-f", "agents/gen_llm.py"], stderr=subprocess.DEVNULL)
-    subprocess.run(["pkill", "-f", "agents/embed_llm.py"], stderr=subprocess.DEVNULL)
-    
-    time.sleep(1)
-    print("LLM services stopped.")
-
-def up(hf_mode: bool = False) -> None:
-    """Start the LLM servers."""
-    gen_alive = _check_port(GEN_PORT)
-    embed_alive = _check_port(EMBED_PORT)
-
-    if gen_alive and embed_alive:
-        print("Both LLM services are already running.")
-        return
-
-    print("Starting LLM services...")
-    
-    env = os.environ.copy()
-    if hf_mode:
-        env["HF_MODE"] = "1"
-        print("Running in HF CPU mode (HF_MODE=1)")
-
-    # Start Embed LLM
-    if not embed_alive:
-        print(f"[1/2] Starting embed_llm on port {EMBED_PORT}...")
-        subprocess.Popen(
-            [sys.executable, str(BASE_DIR / "agents" / "embed_llm.py")],
-            cwd=BASE_DIR,
-            stdout=subprocess.DEVNULL,
-            stderr=subprocess.DEVNULL,
-            start_new_session=True,
-            env=env
-        )
-    else:
-        print(f"[1/2] embed_llm is already running on port {EMBED_PORT}.")
-
-    # Start Gen LLM
-    if not gen_alive:
-        print(f"[2/2] Starting gen_llm on port {GEN_PORT}...")
-        subprocess.Popen(
-            [sys.executable, str(BASE_DIR / "agents" / "gen_llm.py")],
-            cwd=BASE_DIR,
-            stdout=subprocess.DEVNULL,
-            stderr=subprocess.DEVNULL,
-            start_new_session=True,
-            env=env
-        )
-    else:
-        print(f"[2/2] gen_llm is already running on port {GEN_PORT}.")
-
-    print("\nWaiting for services to initialize...")
-    time.sleep(3)
-    status()
-
-
-def main() -> None:
-    parser = argparse.ArgumentParser(
-        description="Manage HealthExpert LLM microservices",
-        formatter_class=argparse.RawDescriptionHelpFormatter,
-        epilog=__doc__,
-    )
-    
-    # Optional positional argument for the command
-    parser.add_argument("command", nargs="?", choices=["up", "down", "status"], default="status",
-                        help="Action to perform (default: status)")
-    parser.add_argument("-hf", "--hf", action="store_true", help="Start servers in HuggingFace/CPU mode")
-    
-    args = parser.parse_args()
-
-    if args.command == "up":
-        up(hf_mode=args.hf)
-    elif args.command == "down":
-        down()
-    else:
-        status()
-
-
-if __name__ == "__main__":
-    main()

pytest.ini

@@ -1,17 +0,0 @@
-[pytest]
-testpaths = tests
-python_files = test_*.py
-python_classes = Test*
-python_functions = test_*
-markers =
-    unit: Unit tests for individual components
-    integration: Integration tests for workflows
-    slow: Tests that take longer than 1 second
-    requires_gpu: Tests that require GPU
-    requires_services: Tests that require running services
-addopts = 
-    -v
-    --tb=short
-    --strict-markers
-    -ra
-minversion = 7.0