Spaces:
Sleeping
Sleeping
| # Feature & Infrastructure Gap Analysis | |
| **This project** (`youtube-kb` → *Amana*, the T&S campaign-triage copilot) vs. | |
| **the general project** (`../supportops_ai_markdown_docs`, the support-ticket triage app). | |
| This document lists what exists in **supportops** but is **missing or weaker here**. It is | |
| deliberately one-directional — it does **not** catalogue what Amana does better (deterministic | |
| policy gate, prompt-injection defense, layered eval harness + LLM-as-judge, citation-by-rule-ID). | |
| Several supportops features are domain-specific to support tickets and **not worth porting**; those | |
| are flagged. Treat this as a menu, not a to-do list. | |
| > **Stack context.** The two apps are built on different stacks. supportops is a single Next.js 16 + | |
| > TypeScript app (Vercel / Neon Postgres + pgvector / Upstash Redis / Clerk / Gemini). Amana is a | |
| > Python backend (FastAPI + Pydantic AI + Claude + Chroma) with a React/Vite SPA, deployed as a | |
| > Docker image on Hugging Face Spaces. So "adopt this" usually means *adopt the capability*, not the | |
| > exact library. | |
| --- | |
| ## Tier 1 — Real gaps that matter for a deployed app | |
| These are genuine capability gaps, ordered by how much they'd matter for Amana as a live demo. | |
| ### 1. No persistent / relational data store | |
| - **supportops:** Neon Postgres via Drizzle ORM with an **8-table schema** (`tickets`, | |
| `ticket_classifications`, `documents`, `document_chunks`, `retrieval_results`, `draft_responses`, | |
| `review_items`, `pipeline_metrics`), UUID PKs, cascade deletes, enums, and per-ticket transactional | |
| persistence (`server/db/persist.ts`). Plus Drizzle **migrations** (`db:generate`/`db:migrate`). | |
| - **Here:** the only durable record is an append-only JSONL audit log (`src/audit.py` → | |
| `data/audit_log.jsonl`). Triage results live in an **in-memory dict** (`api.py:44`) that is wiped on | |
| every restart. The vector store (Chroma) holds the corpus but no operational/decision history. | |
| - **Why it matters:** on Hugging Face Spaces the container restarts and its filesystem is ephemeral — | |
| **the audit log itself is not durable across rebuilds.** A real deployment needs a persistent store | |
| for decision history and ideally for triage runs. | |
| - **Worth it?** *Partially.* Full relational modeling is overkill for an 18-campaign demo, but a | |
| durable backing store for the audit log (even SQLite committed to a volume, or a hosted Postgres) | |
| closes the biggest deploy-readiness gap. | |
| ### 2. No authentication / authorization | |
| - **supportops:** Clerk auth (`@clerk/nextjs`), middleware-protected routes (`proxy.ts`, | |
| `auth.protect()`), every API handler re-validates `auth()`, and the acting user's ID is captured on | |
| each decision (`createdBy`, `reviewerId`). | |
| - **Here:** **no auth at all.** Every `/api/*` endpoint in `api.py` is open; the audit log records a | |
| decision but **not *who* made it** (no moderator identity). | |
| - **Why it matters:** the whole product premise is "a *human moderator* makes the final decision and | |
| owns accountability." Without identity, the audit trail can't actually attribute an override to a | |
| person — a notable hole for a Trust & Safety tool specifically. | |
| - **Worth it?** *Yes, at least lightly.* Full Clerk-style auth may be more than a demo needs, but | |
| capturing a moderator identity (even a simple login or header) would make the human/AI-boundary | |
| story complete and auditable. | |
| ### 3. Cache does not survive restart | |
| - **supportops:** Upstash Redis (REST) cache with a 24h TTL and SHA-256 keys, **wrapped in `safe()`** | |
| so cache outages degrade to a miss instead of failing the request (`server/cache/redis.ts`). | |
| - **Here:** the triage cache is a plain in-process dict (`api.py:44`); restart = full re-bill of all | |
| campaigns; no TTL; no cross-instance sharing. | |
| - **Why it matters:** the spend-cap story ("max 18 calls per restart, then cache hits") **resets on | |
| every restart**. A persistent cache would make the cost guarantee hold across the container's life. | |
| - **Worth it?** *Nice-to-have.* The PUBLIC_DEMO cap already bounds worst-case spend, so this is an | |
| optimization rather than a safety fix. | |
| ### 4. No runtime observability / metrics dashboard | |
| - **supportops:** a `pipeline_metrics` table + a `/metrics` page + a live "pulse" on the inbox | |
| (runs, latency `totalMs`, cache-hit %, avg confidence, escalation %, **prompt/completion token | |
| counts**). Token usage is tracked per run end-to-end. | |
| - **Here:** rich *eval-time* metrics (accuracy, escalation recall, reject precision, citation | |
| validity) but **no runtime/ops metrics** — no latency tracking, no token/cost accounting, no | |
| operational dashboard. `/api/stats` only returns corpus counts and provider name. | |
| - **Why it matters:** for a billed Anthropic demo, **token/cost tracking per run** is the single most | |
| useful operational signal and is currently absent. | |
| - **Worth it?** *Token + latency tracking: yes* (cheap to add, high value). A full metrics page is | |
| optional polish. | |
| --- | |
| ## Tier 2 — Architectural patterns / quality improvements | |
| Transferable engineering practices that would strengthen the codebase regardless of domain. | |
| ### 5. Dependency-injection pipeline for full testability | |
| - **supportops:** the pipeline takes a `PipelineDeps` interface (`server/pipeline/deps.ts`, | |
| `process-ticket.ts`); production wiring vs. test mocks are swapped cleanly, so the **entire** | |
| orchestration is unit-tested without credentials (`tests/process-ticket.test.ts`). | |
| - **Here:** the deterministic gate is well unit-tested (`scripts/smoke_test.py::t_policy_gate`), but | |
| the agent orchestration in `src/agent.py::triage` is harder to test in isolation (relies on | |
| `TestModel` for wiring only). There's no injected-deps seam for the tool/RAG/sanctions layer. | |
| - **Worth it?** *Selectively.* The gate is already the safety-critical, well-tested core; a deps seam | |
| around `triage()` would mainly help if you grow the pipeline. | |
| ### 6. Graceful-degradation wrapper (`safe()`) | |
| - **supportops:** all side effects (cache, persistence) run through a `safe()` helper so any single | |
| failure degrades to a miss or an escalation — **never a crash**. | |
| - **Here:** error handling is more ad hoc — `api.py` catches triage exceptions into a 502, but the | |
| pattern isn't systematized; a Chroma/embedding hiccup mid-request isn't uniformly caught. | |
| - **Worth it?** *Yes, low-cost.* A small wrapper that converts infra failures into an ESCALATE (which | |
| is already Amana's safe default) fits the calibrated-humility philosophy perfectly. | |
| ### 7. Linter / formatter + build check in CI | |
| - **supportops:** ESLint 9 config + `npm run lint` and `npm run build` both run in CI | |
| (`.github/workflows/ci.yml`). | |
| - **Here:** CLAUDE.md states **"No test runner/linter/formatter is configured."** CI (`eval.yml`) | |
| runs the eval gate only — it does **not** lint Python, and **does not build the React frontend**, so | |
| a broken SPA build wouldn't be caught until deploy. | |
| - **Worth it?** *Yes for the frontend build check* (catches deploy-breakers cheaply); *optional* for | |
| adding `ruff`/`black` to the Python side. | |
| ### 8. Weighted, computed confidence score | |
| - **supportops:** final confidence is a deterministic weighted blend | |
| (`0.4·classify + 0.35·retrieval + 0.25·draft`, `server/pipeline/score.ts`), so it's reproducible | |
| and explainable independent of the model's self-report. | |
| - **Here:** confidence is the LLM's own self-reported `low/medium/high`. The gate uses it | |
| (`DEC-5` low-confidence → escalate) but it isn't corroborated by a computed signal. | |
| - **Worth it?** *Maybe.* A computed confidence (e.g., from retrieval similarity + signal count) could | |
| make the gate's low-confidence rule less dependent on the model's honesty — aligned with Amana's | |
| "invariants in code" philosophy. Worth considering, not urgent. | |
| ### 9. Retry-then-fallback at each LLM stage | |
| - **supportops:** classify/draft each retry once on invalid JSON, then fall back to a safe default | |
| (escalation / conservative response) rather than failing. | |
| - **Here:** relies on Pydantic AI's structured-output validation + the gate as backstop; there's no | |
| explicit retry-then-escalate around the agent call. | |
| - **Worth it?** *Low priority* — the gate already turns a bad/empty decision toward ESCALATE, so the | |
| safety outcome is covered; an explicit retry would just improve success rate. | |
| --- | |
| ## Tier 3 — Features that are domain-specific (likely NOT worth porting) | |
| Listed for completeness so the comparison is honest, but these reflect supportops being a | |
| support-desk product rather than gaps in Amana. | |
| - **Runtime KB upload/ingestion UI** (`/kb` page, `POST /api/kb/documents`): supportops lets users | |
| add knowledge-base docs at runtime. Amana's corpus (`data/policy.md`) is a **single source of truth | |
| that should *not* be user-editable** — a mutable policy would undermine the citation/audit model. | |
| *Don't port.* | |
| - **Rich ticket state machine** (`new → processing → auto_drafted → needs_review → approved → | |
| escalated → rejected`): useful for a ticket queue; Amana's decided/not-decided + 3 outcomes is | |
| intentionally simpler and fits the "human decides once" model. | |
| - **Server Actions / form-driven ticket creation**: Next.js-specific; Amana's 18 fixed campaigns are | |
| by design (spend cap by construction). | |
| - **Multi-model split** (separate cheap classify model, richer draft model): supportops uses 3 Gemini | |
| models for cost; Amana deliberately runs one Claude model + free local embeddings. | |
| --- | |
| ## Quick-reference matrix | |
| | Capability | supportops | Amana (here) | Recommendation | | |
| |---|---|---|---| | |
| | Persistent relational store | ✅ Neon Postgres + Drizzle (8 tables, migrations) | ❌ JSONL audit log + ephemeral in-mem | **Add durable store for audit log** | | |
| | Authentication / moderator identity | ✅ Clerk, protected routes, user IDs | ❌ none | **Add lightweight auth/identity** | | |
| | Persistent cache | ✅ Upstash Redis (TTL, safe-degrade) | ⚠️ in-process dict, reset on restart | Optional (cap already bounds spend) | | |
| | Runtime metrics dashboard | ✅ `/metrics`, token + latency tracking | ❌ eval metrics only, no token/cost | **Add token + latency tracking** | | |
| | DI pipeline / mockable orchestration | ✅ `PipelineDeps` | ⚠️ gate tested; agent wiring partial | Selective | | |
| | Graceful-degradation wrapper | ✅ `safe()` everywhere | ⚠️ ad hoc | **Adopt (cheap, fits philosophy)** | | |
| | Lint + frontend build in CI | ✅ ESLint + `next build` | ❌ eval gate only, no SPA build check | **Add SPA build to CI** | | |
| | Computed/weighted confidence | ✅ 40/35/25 blend | ⚠️ LLM self-report | Consider | | |
| | Retry-then-fallback per LLM stage | ✅ | ⚠️ gate backstop only | Low priority | | |
| | Deterministic policy gate | ❌ | ✅ `src/gate.py` | *(Amana strength)* | | |
| | Prompt-injection defense | ⚠️ generic | ✅ DEC-6, camp-015 test | *(Amana strength)* | | |
| | Layered eval + LLM-as-judge | ⚠️ unit tests | ✅ `eval/run_eval.py` | *(Amana strength)* | | |
| | Citation by stable rule ID | ❌ | ✅ | *(Amana strength)* | | |
| --- | |
| ## If you only do three things before deploying | |
| 1. **Make decision history durable** (Tier 1 #1) — back the audit log with something that survives a | |
| Spaces restart; the current JSONL on an ephemeral filesystem can disappear. | |
| 2. **Capture moderator identity** (Tier 1 #2) — even minimal auth, so overrides are attributable. | |
| This directly serves the human/AI-accountability story the project is graded on. | |
| 3. **Track tokens + latency per triage** (Tier 1 #4) — cheap to add to `api.py`, and the most useful | |
| operational signal for a billed live demo. | |