Upload 9 files

.dockerignore

@@ -0,0 +1,8 @@
+.git
+.pytest_cache
+__pycache__
+*.py[cod]
+.env
+.venv
+tests
+data/knowledge_raw_archive

.env.example

@@ -0,0 +1,11 @@
+ENVIRONMENT=development
+ORIGIN_TOKEN=replace-with-the-same-random-value-as-worker-HF_ORIGIN_TOKEN
+OPENAI_API_KEY=
+ANTHROPIC_API_KEY=
+PRIMO_API_KEY=
+ADMIN_PASSWORD=replace-with-a-strong-admin-password
+CLOUDFLARE_WORKER_URL=https://REPLACE-WITH-YOUR-WORKER.workers.dev
+CLOUDFLARE_ADMIN_TOKEN=replace-with-the-same-random-value-as-worker-ADMIN_TOKEN
+CONTACT_EMAIL=libse@ku.ac.ae
+RUNTIME_DIR=./libbee_runtime
+ALLOWED_CORS_ORIGINS=http://localhost:8080,http://127.0.0.1:5500

.gitignore

@@ -0,0 +1,7 @@
+.env
+.venv/
+.pytest_cache/
+__pycache__/
+*.py[cod]
+data/rag_cache/
+libbee_runtime/

Dockerfile

@@ -1,11 +1,24 @@
-FROM python:3.9-slim
-
-WORKDIR /app
-
-COPY requirements.txt .
-RUN pip install --no-cache-dir -r requirements.txt
-
-COPY . .
-
-# HF Spaces requires port 7860
-CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7860"]
+FROM python:3.12-slim
+
+ENV PYTHONDONTWRITEBYTECODE=1 \
+    PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1
+
+WORKDIR /app
+
+RUN addgroup --system libbee \
+    && adduser --system --ingroup libbee --home /app libbee \
+    && mkdir -p /data/libbee_runtime \
+    && chown -R libbee:libbee /app /data/libbee_runtime
+
+COPY --chown=libbee:libbee requirements.txt ./
+RUN pip install --no-cache-dir --requirement requirements.txt
+
+COPY --chown=libbee:libbee . .
+USER libbee
+
+EXPOSE 7860
+HEALTHCHECK --interval=30s --timeout=8s --start-period=90s --retries=3 \
+  CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:7860/health/live', timeout=5)"
+
+CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "7860", "--proxy-headers", "--forwarded-allow-ips", "*"]

README.md

@@ -8,75 +8,36 @@ app_file: app.py
 pinned: false
 ---
 
-# 🐝 LibBee — KU Library AI Assistant
+# LibBee private origin v4.0
 
-An open-source, institution-aware AI library assistant for Khalifa University Library, Abu Dhabi, UAE.
+This Hugging Face Space is the private application origin behind the Cloudflare Worker. Browsers must not call `/agent`, `/search`, or `/feedback` directly. Those routes require `X-LibBee-Origin-Token`, which the Worker adds server-side.
 
-Built with FastAPI, hybrid FAISS + BM25 RAG, and live LibCal integration.
+## Required Space secrets
 
-**Live interface:** https://ku-library.github.io/LibBee  
-**GitHub:** https://github.com/ku-library/LibBee
+- `ORIGIN_TOKEN`: a new random secret; must match the Worker's `HF_ORIGIN_TOKEN`.
+- `OPENAI_API_KEY`: required for the default model, classification, embeddings, and RAG.
+- `ADMIN_PASSWORD`: strong password for `/admin`.
+- `CLOUDFLARE_WORKER_URL`: the deployed Worker URL or custom domain; used for live LibCal data and the server-side analytics proxy.
+- `CLOUDFLARE_ADMIN_TOKEN`: must match the Worker's `ADMIN_TOKEN`; never sent to browser JavaScript.
 
----
-
-## Architecture
-
-- **Backend:** FastAPI + Python 3.9 (this Space)
-- **Frontend:** GitHub Pages (single-file HTML/JS)
-- **RAG:** FAISS dense + BM25 sparse + RRF fusion
-- **Live data:** LibCal API via Cloudflare Worker proxy
-- **Analytics:** Cloudflare D1
-
-## Secrets required
-
-Set these in the Space **Settings → Variables and Secrets**:
-
-| Secret | Description |
-|---|---|
-| `OPENAI_API_KEY` | Required for classification and answering |
-| `ANTHROPIC_API_KEY` | Optional — enables Claude Haiku toggle |
-| `PRIMO_API_KEY` | Required for PRIMO discovery search |
-| `ADMIN_PASSWORD` | Required for /admin dashboard |
-| `CLOUDFLARE_WORKER_URL` | Required for live hours/events and analytics |
-| `SESSION_SECRET` | Required for admin session |
-
-## Version
-
-**v3.7** — Navigation-first agentic AI. See GitHub for changelog.
-
-## Architecture (v3.8)
-
-The pipeline is split by stage under `src/agentcore/` - each module's docstring
-explains its role:
+Optional secrets: `ANTHROPIC_API_KEY` and `PRIMO_API_KEY`.
 
-| Module | Role |
-|---|---|
-| `models.py` | Pydantic I/O contracts |
-| `constants.py` | Regexes, URLs, prompts (no logic) |
-| `utils.py` | Shared helpers, LLM factory, query/URL builders |
-| `classify.py` | Rule pre-classifier + LLM intent classifier |
-| `libcal.py` | Live hours/events via Cloudflare Worker proxy |
-| `scholarly.py` | OpenAlex, Semantic Scholar, Crossref, Unpaywall clients |
-| `rendering.py` | HTML fragments (badges, trace blocks, AI-tools footer) |
-| `intents_library.py` | library_info handlers |
-| `intents_search.py` | search_academic / search_medical machinery |
-| `intents_general.py` | general / social handlers |
-| `orchestrator.py` | Injection gate, routing pipeline, analytics logging |
+Use persistent HF storage and keep `RUNTIME_DIR=/data/libbee_runtime` so runtime configuration, metrics, feedback fallback data, and the signed RAG cache survive restarts.
 
-`src/api/agent.py` is a thin router. Tests: `pytest tests/ -q` (20 tests, no
-API keys required).
+## Security properties
 
-## Environment variables (new in v3.8)
+- Protected origin routes fail closed if `ORIGIN_TOKEN` is absent or incorrect.
+- Request models have strict size and enum limits.
+- Answer HTML is sanitized with a tag and attribute allow-list before it leaves the origin; the LibGuides client sanitizes it again.
+- The signed FAISS/BM25 cache is ignored if it was modified or if no origin secret is configured.
+- Raw scrape dumps are kept in `data/knowledge_raw_archive` but excluded from indexing. Curated, review-dated files are indexed.
+- Persistent analytics is written by the Worker, not by the browser or the HF origin.
 
-| Variable | Purpose |
-|---|---|
-| `CLOUDFLARE_WORKER_TOKEN` | Bearer token for the Worker analytics surface; must equal the Worker's `ANALYTICS_TOKEN` secret |
-| `CONTACT_EMAIL` | Polite-pool contact sent to OpenAlex / Unpaywall / Crossref (default `library@ku.ac.ae`) |
+## Local checks
 
-## Privacy
+```bash
+python -m compileall -q app.py src tests
+pytest -q
+```
 
-Before any query reaches metrics or the D1 analytics store it passes
-`_redact_for_analytics`: e-mail addresses and digit runs of 6+ are masked and
-text is truncated to 200 characters. Answer excerpts logged for the daily
-relevance agent get the same treatment. The Worker applies the identical
-redaction server-side as defence in depth.
+See the complete package `DEPLOYMENT_GUIDE.md` for coordinated HF, Worker, D1, and LibGuides deployment.

app.py

@@ -1,64 +1,26 @@
-"""
-app.py — LibBee v3.5
-FastAPI entry point: CORS, middleware, lifespan, router mounting.
-
-Changes over v3.1:
-- Maintenance mode middleware now reads from JsonRuntimeStore (admin-editable)
-  instead of settings.maintenance_mode (env var, static at startup).
-- CORS origins updated: added ku-library.github.io.
-- Section headers added throughout.
-"""
-
-# ── Imports ────────────────────────────────────────────────────────────────────
+"""LibBee FastAPI application for the private Hugging Face origin."""
 
 import asyncio
+import hmac
 import logging
-import time
 from contextlib import asynccontextmanager
 
 from fastapi import FastAPI, Request
 from fastapi.middleware.cors import CORSMiddleware
-from fastapi.responses import JSONResponse
-
-try:
-    from cachetools import TTLCache
-    _ttl_cache_available = True
-except ImportError:
-    _ttl_cache_available = False
+from fastapi.responses import JSONResponse, Response
 
-from src.config import get_settings, LIBBEE_VERSION
-from src.services.cache_service import CacheService
+from src.config import LIBBEE_VERSION, get_settings
 from src.services.metrics_service import MetricsService
 from src.services.rag_service import RAGService
 
 logging.basicConfig(level=logging.INFO)
 logger = logging.getLogger(__name__)
+settings = get_settings()
+rag_service = RAGService()
+metrics_service = MetricsService(settings.metrics_path)
+_agent_slots = asyncio.Semaphore(settings.max_concurrent_agent_requests)
 
 
-# ── Service Instances ──────────────────────────────────────────────────────────
-
-rag_service     = RAGService()
-cache_service   = CacheService()
-metrics_service = MetricsService(get_settings().metrics_path)
-
-
-# ── Rate-limit Request Log ─────────────────────────────────────────────────────
-# TTLCache auto-evicts entries older than ttl seconds.
-# Falls back to a plain deque-based dict if cachetools is not installed.
-
-if _ttl_cache_available:
-    _request_log      = TTLCache(maxsize=10_000, ttl=120)
-    _request_log_lock = asyncio.Lock()
-else:
-    from collections import defaultdict, deque
-    _request_log      = defaultdict(deque)   # type: ignore[assignment]
-    _request_log_lock = asyncio.Lock()
-
-
-# ── Public Accessors ───────────────────────────────────────────────────────────
-# Routers import these instead of importing module-level globals directly,
-# which avoids circular-import issues at import time.
-
 def get_rag_service() -> RAGService:
     return rag_service
 
@@ -67,228 +29,167 @@ def get_metrics_service() -> MetricsService:
     return metrics_service
 
 
-# ── Background Tasks ───────────────────────────────────────────────────────────
+def _runtime_config() -> dict:
+    from src.services.runtime_store import JsonRuntimeStore
+
+    defaults = {
+        "maintenance_mode": settings.maintenance_mode,
+        "maintenance_message": settings.maintenance_message,
+        "welcome_message": "Hi! I'm LibBee, the Khalifa University Library AI Assistant.",
+        "announcement": "",
+        "max_results": settings.max_results,
+    }
+    try:
+        return JsonRuntimeStore(settings.config_path, default=defaults).load()
+    except Exception:
+        return defaults
+
 
-async def _metrics_flush_loop(interval: int = 30) -> None:
-    """Flush in-memory metric counters to disk every `interval` seconds."""
+async def _metrics_flush_loop() -> None:
     while True:
-        await asyncio.sleep(interval)
+        await asyncio.sleep(30)
         try:
             await metrics_service.flush()
         except Exception as exc:
             logger.warning("Metrics flush error: %s", exc)
 
 
-# ── Lifespan ───────────────────────────────────────────────────────────────────
-
 @asynccontextmanager
-async def lifespan(app: FastAPI):
+async def lifespan(_app: FastAPI):
     from src.services.staff_service import build_staff_index
 
-    settings = get_settings()
-
-    # Build staff index
     try:
         build_staff_index()
-        logger.info("Staff index built")
     except Exception as exc:
         logger.error("Staff index build failed: %s", exc)
-
-    # Initialise RAG service
     if settings.openai_api_key:
         try:
             await rag_service.initialize(openai_api_key=settings.openai_api_key)
-            logger.info("RAG service ready — chunks: %d", len(rag_service.bm25_corpus))
         except Exception as exc:
             logger.error("RAG initialization failed: %s", exc, exc_info=True)
     else:
-        logger.warning("OPENAI_API_KEY not set — RAG disabled")
-
-    # Start background metrics flush
-    flush_task = asyncio.create_task(_metrics_flush_loop(interval=30))
-    logger.info("LibBee startup complete")
-
+        logger.warning("OPENAI_API_KEY not set; RAG is disabled")
+    flush_task = asyncio.create_task(_metrics_flush_loop())
     yield
-
-    # Graceful shutdown — flush remaining metrics
     flush_task.cancel()
     try:
-        await asyncio.wait_for(metrics_service.flush(), timeout=5.0)
+        await metrics_service.flush()
     except Exception:
         pass
-    logger.info("LibBee shutting down")
 
 
-# ── FastAPI App ────────────────────────────────────────────────────────────────
-
 app = FastAPI(
     title="LibBee - KU Library AI",
-    description="Khalifa University Library AI Assistant",
+    description="Private origin API for the Khalifa University Library AI Assistant",
     version=LIBBEE_VERSION,
     lifespan=lifespan,
+    docs_url=None if settings.environment == "production" else "/docs",
+    redoc_url=None,
 )
 
-
-# ── CORS Middleware ────────────────────────────────────────────────────────────
-
-_extra_origins = [
-    o.strip().rstrip("/")
-    for o in (get_settings().extra_cors_origins or "").split(",")
-    if o.strip()
-]
-
 app.add_middleware(
     CORSMiddleware,
-    allow_origins=_extra_origins + [
-        "https://nikeshn.github.io",
-        "https://ku-library.github.io",       # added v3.5
-        "https://library.ku.ac.ae",           # LibGuides embed (added v3.11.1)
-        "http://localhost:8080",
-        "http://localhost:3000",
-        "http://127.0.0.1:5500",
-    ],
-    allow_credentials=True,
-    allow_methods=["GET", "POST", "PUT", "DELETE", "OPTIONS"],
-    allow_headers=["*"],
-    max_age=86400,
+    allow_origins=settings.cors_origins,
+    allow_credentials=False,
+    allow_methods=["GET", "POST", "OPTIONS"],
+    allow_headers=["Content-Type", "X-LibBee-Origin-Token", "X-Request-ID", "X-LibBee-Canary"],
+    max_age=3600,
 )
 
 
-# ── Operational Guardrails Middleware ──────────────────────────────────────────
-
 @app.middleware("http")
-async def security_headers(request: Request, call_next):
-    """
-    Add security headers to every response.
-    X-Content-Type-Options — prevents MIME sniffing
-    X-Frame-Options        — prevents clickjacking
-    Referrer-Policy        — limits referrer leakage
-    X-XSS-Protection       — legacy XSS filter (belt-and-braces)
-    Permissions-Policy     — disables unused browser features
-    """
-    response = await call_next(request)
-    response.headers["X-Content-Type-Options"]  = "nosniff"
-    response.headers["X-Frame-Options"]          = "SAMEORIGIN"
-    response.headers["Referrer-Policy"]          = "strict-origin-when-cross-origin"
-    response.headers["X-XSS-Protection"]         = "1; mode=block"
-    response.headers["Permissions-Policy"]       = (
-        "geolocation=(), microphone=(), camera=(), payment=()"
-    )
-    return response
+async def secure_origin(request: Request, call_next):
+    path = request.url.path
+    protected = path in {"/agent", "/agent/rag-status", "/search", "/feedback"}
+    acquired = False
+
+    if path.startswith("/admin") and request.method not in {"GET", "HEAD", "OPTIONS"}:
+        fetch_site = request.headers.get("sec-fetch-site", "")
+        if fetch_site and fetch_site not in {"same-origin", "none"}:
+            return JSONResponse(status_code=403, content={"detail": "Cross-site admin request denied."})
+
+    if request.method in {"POST", "PUT", "PATCH"}:
+        length = request.headers.get("content-length")
+        if length and length.isdigit() and int(length) > 32_768:
+            return JSONResponse(status_code=413, content={"detail": "Request body too large."})
+
+    if protected:
+        supplied = request.headers.get("x-libbee-origin-token", "")
+        configured = settings.origin_token
+        local_insecure = settings.environment in {"development", "test"} and not configured
+        if not local_insecure and (not configured or not hmac.compare_digest(supplied, configured)):
+            return JSONResponse(status_code=401, content={"detail": "Origin authentication required."})
+
+        runtime = _runtime_config()
+        if runtime.get("maintenance_mode") and request.headers.get("x-libbee-canary") != "1":
+            return JSONResponse(
+                status_code=503,
+                content={"detail": str(runtime.get("maintenance_message") or settings.maintenance_message)[:300]},
+            )
+
+        if path == "/agent":
+            try:
+                await asyncio.wait_for(_agent_slots.acquire(), timeout=0.25)
+                acquired = True
+            except TimeoutError:
+                return JSONResponse(status_code=503, content={"detail": "LibBee is busy. Please retry shortly."})
 
-
-@app.middleware("http")
-async def operational_guardrails(request: Request, call_next):
-    settings = get_settings()
-
-    # ── Maintenance mode ───────────────────────────────────────────────────────
-    # FIX v3.5: read from JsonRuntimeStore (admin-editable) rather than
-    # settings.maintenance_mode (env var, static at startup). This means
-    # toggling maintenance via the admin dashboard actually takes effect.
-    _maintenance_on = False
-    try:
-        from src.services.runtime_store import JsonRuntimeStore
-        _store = JsonRuntimeStore(settings.config_path, default={"maintenance_mode": False})
-        _maintenance_on = bool(_store.load().get("maintenance_mode", False))
-    except Exception:
-        # If store is unreadable fall back to env-var setting
-        _maintenance_on = settings.maintenance_mode
-
-    _exempt_paths = {"/", "/config", "/admin", "/admin/login", "/admin/auth"}
-    if _maintenance_on and request.url.path not in _exempt_paths:
-        return JSONResponse(
-            status_code=503,
-            content={"detail": "LibBee is in maintenance mode. Please try again shortly."},
-        )
-
-    # ── Rate limiting ──────────────────────────────────────────────────────────
-    client_ip = request.client.host if request.client else "unknown"
-    now = time.time()
-
-    async with _request_log_lock:
-        if _ttl_cache_available:
-            timestamps = _request_log.get(client_ip, [])
-            timestamps = [t for t in timestamps if now - t < 60]
-            if len(timestamps) >= settings.rate_limit_per_minute:
-                metrics_service.incr_bucket("errors", "rate_limit")
-                return JSONResponse(
-                    status_code=429,
-                    content={"detail": "Too many requests. Please slow down and try again."},
-                )
-            timestamps.append(now)
-            _request_log[client_ip] = timestamps
-        else:
-            from collections import deque
-            bucket = _request_log[client_ip]  # type: ignore[index]
-            while bucket and now - bucket[0] > 60:
-                bucket.popleft()
-            if len(bucket) >= settings.rate_limit_per_minute:
-                metrics_service.incr_bucket("errors", "rate_limit")
-                return JSONResponse(
-                    status_code=429,
-                    content={"detail": "Too many requests. Please slow down and try again."},
-                )
-            bucket.append(now)
-
-    # ── Request execution ──────────────────────────────────────────────────────
     try:
         response = await call_next(request)
-        return response
     except Exception as exc:
         logger.exception("Unhandled application error: %s", exc)
         metrics_service.incr_bucket("errors", "unhandled_exception")
-        return JSONResponse(status_code=500, content={"detail": "Unexpected server error."})
-
+        response = JSONResponse(status_code=500, content={"detail": "Unexpected server error."})
+    finally:
+        if acquired:
+            _agent_slots.release()
+
+    response.headers["X-Content-Type-Options"] = "nosniff"
+    response.headers["Referrer-Policy"] = "no-referrer"
+    response.headers["Permissions-Policy"] = "geolocation=(), microphone=(), camera=(), payment=()"
+    response.headers["X-Frame-Options"] = "SAMEORIGIN" if path.startswith("/admin") else "DENY"
+    if protected or path.startswith("/admin"):
+        response.headers["Cache-Control"] = "no-store"
+    return response
 
-# ── Router Mounting ────────────────────────────────────────────────────────────
 
 from src.api import admin, agent, feedback, search  # noqa: E402
 
-app.include_router(agent.router,    prefix="/agent",    tags=["Agent"])
-app.include_router(search.router,   prefix="/search",   tags=["Search"])
-app.include_router(admin.router,    prefix="/admin",    tags=["Admin"])
+app.include_router(agent.router, prefix="/agent", tags=["Agent"])
+app.include_router(search.router, prefix="/search", tags=["Search"])
+app.include_router(admin.router, prefix="/admin", tags=["Admin"])
 app.include_router(feedback.router, prefix="/feedback", tags=["Feedback"])
 
 
-# ── Health & Public Endpoints ────────────────────────────────────────���─────────
-
-@app.get("/favicon.ico", include_in_schema=False)
-async def favicon():
-    """
-    Return 204 No Content for favicon requests.
-    Prevents 404 errors in browser console — the backend has no favicon,
-    the frontend (GitHub Pages) serves its own at assets/libbee-mascot.png.
-    """
-    from fastapi.responses import Response
-    return Response(status_code=204)
+@app.get("/")
+@app.get("/health/live")
+def health_live():
+    return {"status": "ok", "version": LIBBEE_VERSION, "service": "LibBee private origin"}
 
 
-@app.get("/")
-def health_check():
-    settings = get_settings()
+@app.get("/health/ready")
+def health_ready():
+    runtime = _runtime_config()
     return {
-        "status": "ok",
+        "status": "ok" if not runtime.get("maintenance_mode") else "maintenance",
         "version": LIBBEE_VERSION,
-        "service": "LibBee KU Library AI",
         "rag_ready": rag_service.is_ready(),
-        "maintenance_mode": settings.maintenance_mode,
-        "endpoints": ["/agent", "/search", "/admin", "/feedback"],
+        "maintenance_mode": bool(runtime.get("maintenance_mode")),
     }
 
 
 @app.get("/config")
+@app.get("/admin/public-config")
 def public_config():
-    """Legacy public config endpoint — frontend uses /admin/public-config instead."""
-    settings = get_settings()
+    runtime = _runtime_config()
     return {
-        "welcome_message": "Hi! I'm LibBee, the Khalifa University Library AI Assistant.",
-        "max_results": settings.max_results,
-        "maintenance_mode": settings.maintenance_mode,
+        "welcome_message": str(runtime.get("welcome_message", ""))[:500],
+        "announcement": str(runtime.get("announcement", ""))[:500],
+        "maintenance_mode": bool(runtime.get("maintenance_mode")),
+        "maintenance_message": str(runtime.get("maintenance_message", ""))[:300],
     }
 
 
-@app.get("/year")
-def get_year():
-    from datetime import datetime
-    now = datetime.utcnow()
-    return {"year": now.year, "month": now.month, "date": now.strftime("%Y-%m-%d")}
+@app.get("/favicon.ico", include_in_schema=False)
+def favicon():
+    return Response(status_code=204)

requirements-dev.txt

@@ -0,0 +1,3 @@
+-r requirements.txt
+pytest==8.3.5
+ruff==0.11.8

requirements.txt

@@ -1,16 +1,16 @@
-fastapi>=0.104.0
-uvicorn[standard]>=0.24.0
-python-multipart>=0.0.6
-pydantic>=2.5.0
-pydantic-settings>=2.1.0
-langchain>=0.1.0
-langchain-openai>=0.0.5
-langchain-anthropic>=0.1.0
-langchain-community>=0.0.10
-anthropic>=0.34.0
-faiss-cpu>=1.7.4
-rank-bm25>=0.2.2
-httpx>=0.25.0
-numpy>=1.26.0
-openai>=1.40.0
-cachetools>=5.3.0
+fastapi==0.115.12
+uvicorn[standard]==0.34.2
+python-multipart==0.0.20
+pydantic==2.11.4
+pydantic-settings==2.9.1
+langchain==0.3.25
+langchain-openai==0.3.16
+langchain-anthropic==0.3.13
+langchain-community==0.3.24
+anthropic==0.51.0
+faiss-cpu==1.11.0
+rank-bm25==0.2.2
+httpx==0.28.1
+numpy==2.2.5
+openai==1.79.0
+cachetools==5.5.2