data/docs/implementation_plan.md

Support Docs Copilot — Audit & Enhanced Implementation Plan

Honest Assessment of Current State

I've read every single file in the project. Here's the unfiltered truth:

What Already Exists (Skeleton-Complete)

The project has the file structure for all 13 phases you outlined — files exist for config, loaders, chunking, indexer, registry, context builder, query transform, retriever, guardrails, auth, observability, evaluation, UI tabs, Docker, and tests. The git log shows 12 commits covering the roadmap.

Critical Problems I Found

Most of these implementations are shallow scaffolds, not production-grade code. An experienced interviewer will see through them in minutes.

Area	Problem	Severity
Query Transform	Just regex heuristics ("error" → "troubleshoot", "how" → "steps"). No actual LLM-based query rewriting.	🔴 Critical
Guardrails	Duplicate keyword lists between input.py and validators.py. Both are simple substring checks — trivially bypassable. No real prompt injection detection.	🔴 Critical
Auth	Static API keys compared as strings. No JWT, no password hashing, no token expiry. AUTH_ENABLED=false by default.	🟡 Weak
Observability	Only RequestMetrics with manual timing. No structured JSON logging, no correlation IDs in responses, no OpenTelemetry.	🟡 Weak
Eval	token_overlap is a naive set intersection of tokens. No semantic similarity, no faithfulness scoring, no RAGAS integration despite listing ragas in requirements.	🔴 Critical
Tests	4 test files with 7 total tests. All pure unit tests on trivial logic. Zero integration tests, zero API tests, no pytest fixtures.	🔴 Critical
Golden QA	Only 2 questions. No realistic scenario coverage.	🟡 Weak
Sample Docs	Only 2 tiny .txt files. No PDF/DOCX/HTML/MD samples to prove multi-format works.	🟡 Weak
Streaming endpoint	/chat/stream duplicates the entire RAG logic instead of reusing the workflow. The prompt template is hardcoded separately from the workflow's prompt.	🔴 Critical
README	Still says python app/engine/ingestion.py instead of python -m app.engine.ingestion. Missing architecture diagram, security section, eval section.	🟡 Weak
Docker	No health checks in compose. No .dockerignore. Frontend Dockerfile installs streamlit requests from pip instead of requirements.txt.	🟡 Weak
Rate Limiter	In-memory deque per client. Resets on every server restart. Not usable in multi-worker setup.	🟡 Weak
Logging	configure_logging() is a 3-line basicConfig. Uses print() statements everywhere in the workflow.	🟡 Weak
\r\n line endings	Mixed CRLF/LF throughout workflow.py and other files. Looks sloppy in diffs.	🟢 Minor
Error handling	CopilotError in errors.py is defined but never used anywhere. No FastAPI exception handler registered.	🟡 Weak

---

Additional Enhancements to Make This Genuinely Resume-Worthy

These are the things NOT in your 13-phase plan that a senior engineer or interviewer would look for.

1. Conversation Memory / Multi-Turn Chat 🔴 CRITICAL MISS

Your copilot currently has zero conversation history. Each query is independent. A real support copilot needs:

• Session-level chat history (at minimum last N turns)
• Context-aware follow-up ("What about for enterprise users?" should work after "What is the refund policy?")
• LangGraph state should carry conversation history

2. Async Architecture 🔴 CRITICAL MISS

workflow.py uses rag_agent.invoke() which is synchronous. The FastAPI endpoints are async def but block on sync LangChain calls. This means:

• Your server blocks on every LLM call
• Under load, it's a single-threaded bottleneck
• Should use rag_agent.ainvoke() with async LangChain chains

3. Proper Error Handling with FastAPI Exception Handlers 🟡 IMPORTANT

CopilotError exists but has zero usage. You need:

• @app.exception_handler(CopilotError) registered
• All business logic raising CopilotError instead of raw HTTPException
• Consistent error response schema

4. Answer Confidence Score 🟡 IMPORTANT

Your roadmap mentions "show answer confidence" in Phase 10 but there's no mechanism to compute it. You need:

• Retrieval relevance score aggregation
• Hallucination check result exposed as confidence level
• Display in both API response and UI

5. Proper Structured JSON Logging 🟡 IMPORTANT

Replace print() and basicConfig with:

• JSON-structured log output (for log aggregators)
• Request ID propagation across all log lines
• Log level configurable from env

6. Feedback Loop / Thumbs Up-Down 🟡 NICE-TO-HAVE

Store user feedback on answers for future evaluation and fine-tuning signals:

• POST /feedback endpoint
• SQLite or JSON file store
• Display in eval dashboard

7. Proper .dockerignore 🟢 QUICK WIN

Missing entirely. Without it, your Docker builds copy .venv/, qdrant_data/, __pycache__/, .git/ into the image.

8. Makefile / scripts/run.ps1 🟢 QUICK WIN

One-command shortcuts for common operations. Shows engineering maturity.

9. pyproject.toml instead of requirements.txt 🟢 POLISH

Modern Python projects use pyproject.toml. It shows you know current tooling.

10. CORS Middleware 🟢 QUICK WIN

Not configured. The frontend works because it uses server-side requests, but any other client would fail.

11. Realistic Sample Documents 🟡 IMPORTANT

You need 5-8 real-ish support documents across formats (.md, .pdf, .txt, .html) to demonstrate the multi-format pipeline actually works in demos.

12. CI/CD Pipeline (GitHub Actions) 🟡 IMPORTANT

A .github/workflows/ci.yml that runs tests + linting on push. Makes the repo look professional.

---

Open Questions

Please answer these before I start implementing:

1. Conversation memory: Do you want full multi-turn chat history, or is single-turn acceptable for now?
2. Async refactor: Should I convert the LangGraph workflow to async (ainvoke), or keep sync for simplicity?
3. pyproject.toml: Do you want me to migrate from requirements.txt to pyproject.toml?
4. GitHub Actions CI: Do you want a CI workflow file?
5. Feedback endpoint: Should I add a /feedback thumbs-up/down endpoint with local storage?
6. Which of the additional enhancements above do you want included? (I recommend all of them — they're what separates a "tutorial project" from a "I built this" project)

---

Proposed Implementation Order

I'll implement in this order (stabilize → depth → polish), creating natural git commits at each step:

Phase A: Fix Foundation & Consistency

1. Normalize line endings, fix print() → logger, wire up CopilotError + exception handlers
2. Fix README commands, add .dockerignore, add CORS middleware
3. Fix streaming endpoint to reuse workflow instead of duplicating logic
4. Commit: fix: normalize logging, error handling, and streaming consistency

Phase B: Real Query Rewriting

5. Replace regex heuristics with LLM-based query rewriting via Ollama
6. Commit: feat: add LLM-based query rewriting for improved retrieval

Phase C: Stronger Guardrails

7. Deduplicate injection patterns, add more patterns, add encoding-aware detection
8. Add SSN/credit-card PII patterns to output redaction
9. Commit: feat: strengthen prompt injection and PII detection guardrails

Phase D: Proper Auth

10. Add JWT token generation, password hashing, token expiry
11. Add proper auth middleware
12. Commit: feat: implement JWT authentication with role-based access

Phase E: Conversation Memory

13. Add conversation history to LangGraph state
14. Wire up session-based chat history in API and UI
15. Commit: feat: add multi-turn conversation memory to chat workflow

Phase F: Answer Confidence

16. Compute confidence from retrieval scores + hallucination check
17. Expose in API response and UI
18. Commit: feat: add answer confidence scoring to responses

Phase G: Structured Logging & Observability

19. JSON structured logging with request ID propagation
20. Granular stage timing (retrieval, reranking, generation separately)
21. Commit: feat: add structured JSON logging with request correlation

Phase H: Real Evaluation

22. Expand golden QA to 10+ questions
23. Add semantic similarity scoring (not just token overlap)
24. Add retrieval precision/recall metrics
25. Commit: feat: expand RAG evaluation with semantic metrics and larger dataset

Phase I: Realistic Sample Docs

26. Create 5-8 sample docs in multiple formats (.md, .txt, .pdf placeholder, .html)
27. Commit: docs: add multi-format sample documents for demo

Phase J: Test Coverage

28. Add API integration tests with FastAPI TestClient
29. Add proper pytest fixtures and conftest.py
30. Add test for auth, streaming, ingestion flow
31. Commit: test: add integration tests for API, auth, and retrieval pipeline

Phase K: Docker & Deployment Polish

32. Fix Dockerfiles, add health checks, add .dockerignore
33. Add Makefile with common commands
34. Commit: deploy: harden Docker setup with health checks and build optimization

Phase L: Feedback System

35. Add /feedback endpoint with local JSON storage
36. Display feedback stats in eval tab
37. Commit: feat: add user feedback collection for answer quality tracking

Phase M: CI + README + Architecture

38. Add GitHub Actions CI workflow
39. Rewrite README with architecture diagram, security section, eval section
40. Commit: docs: comprehensive README with architecture diagram and CI pipeline

Phase N: Async Refactor

41. Convert workflow to async, use ainvoke
42. Commit: perf: convert RAG workflow to async for concurrent request handling

---

Verification Plan

Automated Tests

pytest tests/ -v
pytest tests/ -v --tb=short  # CI mode

Manual Verification

• Backend starts: uvicorn app.main:app --reload
• UI connects: streamlit run ui/app.py
• Ingestion works: python -m app.engine.ingestion ingest
• Chat returns cited answers
• Guardrails block injection attempts
• Auth restricts admin endpoints
• Docker compose brings up full stack