COMPARISON.md

Feature & Infrastructure Gap Analysis

This project (youtube-kb → Amana, the T&S campaign-triage copilot) vs. the general project (../supportops_ai_markdown_docs, the support-ticket triage app).

This document lists what exists in supportops but is missing or weaker here. It is deliberately one-directional — it does not catalogue what Amana does better (deterministic policy gate, prompt-injection defense, layered eval harness + LLM-as-judge, citation-by-rule-ID). Several supportops features are domain-specific to support tickets and not worth porting; those are flagged. Treat this as a menu, not a to-do list.

Stack context. The two apps are built on different stacks. supportops is a single Next.js 16 + TypeScript app (Vercel / Neon Postgres + pgvector / Upstash Redis / Clerk / Gemini). Amana is a Python backend (FastAPI + Pydantic AI + Claude + Chroma) with a React/Vite SPA, deployed as a Docker image on Hugging Face Spaces. So "adopt this" usually means adopt the capability, not the exact library.

---

Tier 1 — Real gaps that matter for a deployed app

These are genuine capability gaps, ordered by how much they'd matter for Amana as a live demo.

1. No persistent / relational data store

• supportops: Neon Postgres via Drizzle ORM with an 8-table schema (tickets, ticket_classifications, documents, document_chunks, retrieval_results, draft_responses, review_items, pipeline_metrics), UUID PKs, cascade deletes, enums, and per-ticket transactional persistence (server/db/persist.ts). Plus Drizzle migrations (db:generate/db:migrate).
• Here: the only durable record is an append-only JSONL audit log (src/audit.py → data/audit_log.jsonl). Triage results live in an in-memory dict (api.py:44) that is wiped on every restart. The vector store (Chroma) holds the corpus but no operational/decision history.
• Why it matters: on Hugging Face Spaces the container restarts and its filesystem is ephemeral — the audit log itself is not durable across rebuilds. A real deployment needs a persistent store for decision history and ideally for triage runs.
• Worth it? Partially. Full relational modeling is overkill for an 18-campaign demo, but a durable backing store for the audit log (even SQLite committed to a volume, or a hosted Postgres) closes the biggest deploy-readiness gap.

2. No authentication / authorization

• supportops: Clerk auth (@clerk/nextjs), middleware-protected routes (proxy.ts, auth.protect()), every API handler re-validates auth(), and the acting user's ID is captured on each decision (createdBy, reviewerId).
• Here: no auth at all. Every /api/* endpoint in api.py is open; the audit log records a decision but not who made it (no moderator identity).
• Why it matters: the whole product premise is "a human moderator makes the final decision and owns accountability." Without identity, the audit trail can't actually attribute an override to a person — a notable hole for a Trust & Safety tool specifically.
• Worth it? Yes, at least lightly. Full Clerk-style auth may be more than a demo needs, but capturing a moderator identity (even a simple login or header) would make the human/AI-boundary story complete and auditable.

3. Cache does not survive restart

• supportops: Upstash Redis (REST) cache with a 24h TTL and SHA-256 keys, wrapped in safe() so cache outages degrade to a miss instead of failing the request (server/cache/redis.ts).
• Here: the triage cache is a plain in-process dict (api.py:44); restart = full re-bill of all campaigns; no TTL; no cross-instance sharing.
• Why it matters: the spend-cap story ("max 18 calls per restart, then cache hits") resets on every restart. A persistent cache would make the cost guarantee hold across the container's life.
• Worth it? Nice-to-have. The PUBLIC_DEMO cap already bounds worst-case spend, so this is an optimization rather than a safety fix.

4. No runtime observability / metrics dashboard

• supportops: a pipeline_metrics table + a /metrics page + a live "pulse" on the inbox (runs, latency totalMs, cache-hit %, avg confidence, escalation %, prompt/completion token counts). Token usage is tracked per run end-to-end.
• Here: rich eval-time metrics (accuracy, escalation recall, reject precision, citation validity) but no runtime/ops metrics — no latency tracking, no token/cost accounting, no operational dashboard. /api/stats only returns corpus counts and provider name.
• Why it matters: for a billed Anthropic demo, token/cost tracking per run is the single most useful operational signal and is currently absent.
• Worth it? Token + latency tracking: yes (cheap to add, high value). A full metrics page is optional polish.

---

Tier 2 — Architectural patterns / quality improvements

Transferable engineering practices that would strengthen the codebase regardless of domain.

5. Dependency-injection pipeline for full testability

• supportops: the pipeline takes a PipelineDeps interface (server/pipeline/deps.ts, process-ticket.ts); production wiring vs. test mocks are swapped cleanly, so the entire orchestration is unit-tested without credentials (tests/process-ticket.test.ts).
• Here: the deterministic gate is well unit-tested (scripts/smoke_test.py::t_policy_gate), but the agent orchestration in src/agent.py::triage is harder to test in isolation (relies on TestModel for wiring only). There's no injected-deps seam for the tool/RAG/sanctions layer.
• Worth it? Selectively. The gate is already the safety-critical, well-tested core; a deps seam around triage() would mainly help if you grow the pipeline.

6. Graceful-degradation wrapper (safe())

• supportops: all side effects (cache, persistence) run through a safe() helper so any single failure degrades to a miss or an escalation — never a crash.
• Here: error handling is more ad hoc — api.py catches triage exceptions into a 502, but the pattern isn't systematized; a Chroma/embedding hiccup mid-request isn't uniformly caught.
• Worth it? Yes, low-cost. A small wrapper that converts infra failures into an ESCALATE (which is already Amana's safe default) fits the calibrated-humility philosophy perfectly.

7. Linter / formatter + build check in CI

• supportops: ESLint 9 config + npm run lint and npm run build both run in CI (.github/workflows/ci.yml).
• Here: CLAUDE.md states "No test runner/linter/formatter is configured." CI (eval.yml) runs the eval gate only — it does not lint Python, and does not build the React frontend, so a broken SPA build wouldn't be caught until deploy.
• Worth it? Yes for the frontend build check (catches deploy-breakers cheaply); optional for adding ruff/black to the Python side.

8. Weighted, computed confidence score

• supportops: final confidence is a deterministic weighted blend (0.4·classify + 0.35·retrieval + 0.25·draft, server/pipeline/score.ts), so it's reproducible and explainable independent of the model's self-report.
• Here: confidence is the LLM's own self-reported low/medium/high. The gate uses it (DEC-5 low-confidence → escalate) but it isn't corroborated by a computed signal.
• Worth it? Maybe. A computed confidence (e.g., from retrieval similarity + signal count) could make the gate's low-confidence rule less dependent on the model's honesty — aligned with Amana's "invariants in code" philosophy. Worth considering, not urgent.

9. Retry-then-fallback at each LLM stage

• supportops: classify/draft each retry once on invalid JSON, then fall back to a safe default (escalation / conservative response) rather than failing.
• Here: relies on Pydantic AI's structured-output validation + the gate as backstop; there's no explicit retry-then-escalate around the agent call.
• Worth it? Low priority — the gate already turns a bad/empty decision toward ESCALATE, so the safety outcome is covered; an explicit retry would just improve success rate.

---

Tier 3 — Features that are domain-specific (likely NOT worth porting)

Listed for completeness so the comparison is honest, but these reflect supportops being a support-desk product rather than gaps in Amana.

• Runtime KB upload/ingestion UI (/kb page, POST /api/kb/documents): supportops lets users add knowledge-base docs at runtime. Amana's corpus (data/policy.md) is a single source of truth that should not be user-editable — a mutable policy would undermine the citation/audit model. Don't port.
• Rich ticket state machine (new → processing → auto_drafted → needs_review → approved → escalated → rejected): useful for a ticket queue; Amana's decided/not-decided + 3 outcomes is intentionally simpler and fits the "human decides once" model.
• Server Actions / form-driven ticket creation: Next.js-specific; Amana's 18 fixed campaigns are by design (spend cap by construction).
• Multi-model split (separate cheap classify model, richer draft model): supportops uses 3 Gemini models for cost; Amana deliberately runs one Claude model + free local embeddings.

---

Quick-reference matrix

Capability	supportops	Amana (here)	Recommendation
Persistent relational store	✅ Neon Postgres + Drizzle (8 tables, migrations)	❌ JSONL audit log + ephemeral in-mem	Add durable store for audit log
Authentication / moderator identity	✅ Clerk, protected routes, user IDs	❌ none	Add lightweight auth/identity
Persistent cache	✅ Upstash Redis (TTL, safe-degrade)	⚠️ in-process dict, reset on restart	Optional (cap already bounds spend)
Runtime metrics dashboard	✅ /metrics, token + latency tracking	❌ eval metrics only, no token/cost	Add token + latency tracking
DI pipeline / mockable orchestration	✅ PipelineDeps	⚠️ gate tested; agent wiring partial	Selective
Graceful-degradation wrapper	✅ safe() everywhere	⚠️ ad hoc	Adopt (cheap, fits philosophy)
Lint + frontend build in CI	✅ ESLint + next build	❌ eval gate only, no SPA build check	Add SPA build to CI
Computed/weighted confidence	✅ 40/35/25 blend	⚠️ LLM self-report	Consider
Retry-then-fallback per LLM stage	✅	⚠️ gate backstop only	Low priority
Deterministic policy gate	❌	✅ src/gate.py	(Amana strength)
Prompt-injection defense	⚠️ generic	✅ DEC-6, camp-015 test	(Amana strength)
Layered eval + LLM-as-judge	⚠️ unit tests	✅ eval/run_eval.py	(Amana strength)
Citation by stable rule ID	❌	✅	(Amana strength)

---

If you only do three things before deploying

1. Make decision history durable (Tier 1 #1) — back the audit log with something that survives a Spaces restart; the current JSONL on an ephemeral filesystem can disappear.
2. Capture moderator identity (Tier 1 #2) — even minimal auth, so overrides are attributable. This directly serves the human/AI-accountability story the project is graded on.
3. Track tokens + latency per triage (Tier 1 #4) — cheap to add to api.py, and the most useful operational signal for a billed live demo.