Spaces:
Running
Running
| # TASKS — Conversational Analytics Assistant (chat-service) | |
| Step-by-step build checklist, derived from the implementation plan. Check items off as you go. Order matters — each milestone assumes the previous one works. | |
| --- | |
| ## Milestone 0 — Repo & environment setup | |
| - [x] Create `chat-service/` folder structure | |
| - [x] `git init`, set `origin` remote | |
| - [x] Create virtual environment (`python -m venv venv`) | |
| - [x] Activate venv | |
| - [x] Create `.env.example` with all vars from plan §10 (placeholders) | |
| - [x] Create local `.env` (real values, gitignored) | |
| - [x] Add `.gitignore` (venv, `.env`, `__pycache__`, `*.pyc`) | |
| - [x] Initial commit ("chore: project skeleton") | |
| --- | |
| ## Milestone 1 — Service skeleton (boots + Groq round-trip) | |
| - [x] Add core deps to `requirements.txt`: `fastapi`, `uvicorn[standard]`, `pydantic-settings`, `groq`, `python-dotenv` | |
| - [x] `pip install -r requirements.txt` | |
| - [x] `app/config.py` — `Settings(BaseSettings)` class reading env vars | |
| - [x] `app/agent/llm.py` — thin Groq client wrapper (single `ask(prompt: str) -> str` function) | |
| - [x] `app/main.py`: | |
| - [x] `GET /health` → `{"status": "ok"}` | |
| - [x] Temporary `POST /chat` → calls Groq wrapper directly, no LangGraph yet, just to prove the chain works | |
| - [x] Run locally: `uvicorn app.main:app --reload --port 8002` | |
| - [x] Test `/health` with curl/Postman | |
| - [x] Test `/chat` with curl/Postman — confirm real Groq response comes back | |
| - [x] Write `Dockerfile` (Python 3.12 base, matches existing services) | |
| - [x] Build + run container locally, re-test `/health` and `/chat` inside Docker | |
| - [x] Commit ("feat: service skeleton with Groq round-trip") | |
| **Done when:** `/health` and `/chat` both work locally and in Docker. | |
| --- | |
| ## Milestone 2 — First real tool + LangGraph agent (`sql_query_tool`) | |
| - [x] Add deps: `langgraph`, `langchain-groq`, `sqlalchemy`, `psycopg[binary]` (or `asyncpg`) | |
| - [x] Create **read-only Postgres role** (`chat_ro_user`) — DB side, not app code | |
| - [x] `app/db/postgres.py` — read-only connection pool using `POSTGRES_READONLY_URL` | |
| - [x] Confirm exact schema fields needed (resolve plan §12.1: `ArticleFakeNews`, `ArticleTopic`, etc. field names) | |
| - [x] `app/tools/sql_tool.py`: | |
| - [x] Define `metric` enum: `fake_news_count`, `articles_by_topic`, `sentiment_breakdown`, `propaganda_count`, `hate_speech_count`, `article_lookup`, `dialect_breakdown` | |
| - [x] Write one parametrized SQL query per metric | |
| - [x] Return shape: `{"rows": [...], "sql_used": "...", "source_refs": [...]}` | |
| - [x] Write unit tests for each metric in `tests/test_sql_tool.py` (against a test/staging DB) | |
| - [x] `app/agent/state.py` — `AgentState` TypedDict (`messages`, `sources`, `session_id`, `user_id`) | |
| - [x] `app/agent/prompts.py` — system prompt (identity, tool-preference rule, citation rule, language-matching rule, no-fabrication rule) | |
| - [x] `app/agent/graph.py` — LangGraph ReAct-style agent wired to `sql_query_tool` only | |
| - [x] Wire `/chat` in `main.py` to call the LangGraph agent instead of the raw Groq wrapper | |
| - [x] `app/schemas/request.py` / `app/schemas/response.py` — formalize `ChatRequest` / `ChatResponse` models per plan §4 | |
| - [x] Manually test: "How many fake news articles were detected last month?" → correct tool call + correct answer | |
| - [x] Commit ("feat: sql_query_tool + LangGraph agent") | |
| **Done when:** agent correctly answers at least 3 different stat questions using real DB data, citing article IDs. | |
| --- | |
| ## Milestone 3 — Django proxy endpoint | |
| - [x] In Django: add `CHAT_SERVICE_URL` and `CHAT_SERVICE_INTERNAL_TOKEN` to settings/env | |
| - [x] New view: `POST /api/v1/chat/` | |
| - [x] Verify JWT (reuse existing auth) | |
| - [ ] Apply DRF throttling/rate limit (no need for now) | |
| - [x] Forward `{session_id, message, user_id}` to chat service with internal token header | |
| - [x] Return chat service's JSON response unchanged | |
| - [x] Confirm chat service rejects requests without the internal token | |
| - [x] Test end-to-end: frontend-style request → Django → chat service → Groq → back | |
| - [x] Commit on Django repo ("feat: chat proxy endpoint") | |
| **Done when:** a JWT-authenticated request through Django reaches the chat service and gets a real answer. | |
| --- | |
| ## Milestone 4 — `graph_query_tool` (Neo4j) | |
| - [x] Confirm current Neo4j label/relationship conventions used by `kg_sync` (resolve plan §12.2) | |
| - [x] ~~Create read-only Neo4j role/user~~ — N/A, Aura Free has no RBAC; enforcement moved to app-level (read-only transactions + no write Cypher in tool code) | |
| - [x] `app/db/neo4j.py` — driver wrapper using `NEO4J_*` vars, enforcing read-only via explicit read transactions | |
| - [x] `app/tools/graph_tool.py`: | |
| - [x] Define `query_type` enum (7 total): `entity_connections`, `entity_mentions`, `shared_entities_between_articles`, `most_connected_entities`, `article_verdict`, `claims_for_article`, `analysis_for_article` | |
| - [x] Write one parametrized Cypher template per query_type | |
| - [x] Return shape: `{"rows": [...], "source_refs": [...]}` | |
| - [x] Add `graph_query_tool` to the LangGraph agent's tool list | |
| - [x] Unit tests in `tests/test_graph_tool.py` | |
| - [x] Manually test: "Who is connected to [entity]?" → correct Cypher template used, correct answer | |
| - [x] Commit ("feat: graph_query_tool") | |
| **Done when:** agent correctly answers at least 2 relationship questions using real graph data. | |
| --- | |
| ## Milestone 5 — Memory (multi-turn) | |
| - [x] Add dep: Redis client (`redis` / `redis[hiredis]`) | |
| - [x] `app/memory/redis_checkpointer.py` — LangGraph checkpointer backed by Redis, key prefix `chat:checkpoint:{session_id}` | |
| - [x] Wire checkpointer into `app/agent/graph.py` compilation | |
| - [x] Set `CHAT_SESSION_TTL_SECONDS` TTL on session keys | |
| - [x] Implement history trimming: keep last `CHAT_HISTORY_MAX_TURNS` verbatim, summarize older turns beyond that | |
| - [x] Manual test: ask a question, then a follow-up ("and what about last month?") using the same `session_id` — confirm context carries over | |
| - [x] Manual test: new `session_id` → confirm no memory leaks across sessions | |
| - [x] Commit ("feat: Redis-backed multi-turn memory") | |
| **Done when:** follow-up questions correctly resolve using prior turn context, and TTL/trimming work as expected. | |
| --- | |
| ## Milestone 6 — Citations end-to-end | |
| - [x] `app/tools/article_tool.py` — `get_article_detail(article_id)` for resolving full title/snippet | |
| - [x] Confirm every tool (`sql_query_tool`, `graph_query_tool`) consistently returns `source_refs` | |
| - [x] Agent post-processing step: merge all `source_refs` collected during the turn into final `sources[]` in the response | |
| - [x] Update system prompt to explicitly require citing every ID it used | |
| - [x] Manual test: ask a question, verify `sources[]` in the JSON response matches what was actually used | |
| - [x] Commit ("feat: citation resolution") | |
| **Done when:** every factual answer includes a non-empty, accurate `sources[]` array. | |
| --- | |
| ## Milestone 7 — `hybrid_search_tool` | |
| - [x] Confirm reuse path for `_embed` (call existing Django/analysis embedding function - the function is in app/services/nlp_client.py) | |
| - [x] `app/tools/hybrid_tool.py`: | |
| - [x] Step 1: embed query text | |
| - [x] Step 2: pgvector cosine similarity search over `search_vector` | |
| - [x] Step 3: for each top article, pull mentioned entities from Neo4j | |
| - [x] Step 4: merge into single ranked result set with combined `source_refs` | |
| - [x] Add tool to agent's tool list | |
| - [ ] Manual test: "Articles about the election, and who's mentioned" → correct combined results | |
| - [x] Commit ("feat: hybrid_search_tool") | |
| **Done when:** at least one combined semantic+graph question is answered correctly with merged citations. | |
| --- | |
| ## Milestone 8 — Frontend chat widget | |
| - [ ] Generate/persist `session_id` client-side (e.g. `localStorage`, created on first load) | |
| - [ ] Chat UI component: message list, input box, send button | |
| - [ ] Call `POST /api/v1/chat/` with `{session_id, message}` | |
| - [ ] Render `answer` text | |
| - [ ] Render `sources[]` as clickable links (article/entity) | |
| - [ ] Loading state while waiting for response | |
| - [ ] Basic error state (service down / timeout) | |
| - [ ] Commit ("feat: chat widget") | |
| **Done when:** a real user can open the widget, ask a question, get an answer with clickable citations, and ask a follow-up. | |
| --- | |
| ## Milestone 9 — Tests, docs, hardening | |
| - [ ] Unit tests for all tools (`sql_tool`, `graph_tool`, `hybrid_tool`) with mocked/test DB data | |
| - [ ] `tests/test_agent_e2e.py` — at least 3 end-to-end scenarios (stat question, relationship question, follow-up question) | |
| - [ ] `chat-service/README.md` — setup instructions, env vars, how to run locally + in Docker | |
| - [ ] Verify `.env.example` is complete and matches plan §10 | |
| - [ ] Security checklist review (plan §8) — confirm every box is actually true in the running system: | |
| - [ ] Read-only DB roles confirmed (test that a write attempt fails) | |
| - [ ] No raw SQL/Cypher ever comes from the LLM — code review confirms only enum params reach tools | |
| - [ ] Internal token required and enforced | |
| - [ ] Rate limiting active on Django proxy | |
| - [ ] Groq key not logged anywhere | |
| - [ ] Tool calls logged (name + params, not raw rows) | |
| - [ ] Add `chat_service` to `docker-compose.yml` (no published host port in prod config) | |
| - [ ] Final walkthrough / demo run-through for defense | |
| - [ ] Tag/commit ("chore: v1 complete") | |
| **Done when:** everything above is checked and you can demo the full flow live without surprises. | |
| --- | |
| ## Open items to resolve before/while coding (carry over from plan §12) | |
| - [ ] Confirm exact Postgres field names per metric | |
| - [ ] Confirm Neo4j label/relationship naming conventions | |
| - [ ] Decide: authenticated-only sessions, or also support anonymous/demo sessions for defense | |
| - [ ] Decide: citation links deep-link into real frontend routes, or raw IDs for now | |