TASKS.md

# TASKS — Conversational Analytics Assistant (chat-service)

Step-by-step build checklist, derived from the implementation plan. Check items off as you go. Order matters — each milestone assumes the previous one works.

---

## Milestone 0 — Repo & environment setup

- [x] Create `chat-service/` folder structure
- [x] `git init`, set `origin` remote
- [x] Create virtual environment (`python -m venv venv`)
- [x] Activate venv
- [x] Create `.env.example` with all vars from plan §10 (placeholders)
- [x] Create local `.env` (real values, gitignored)
- [x] Add `.gitignore` (venv, `.env`, `__pycache__`, `*.pyc`)
- [x] Initial commit ("chore: project skeleton")

---

## Milestone 1 — Service skeleton (boots + Groq round-trip)

- [x] Add core deps to `requirements.txt`: `fastapi`, `uvicorn[standard]`, `pydantic-settings`, `groq`, `python-dotenv`
- [x] `pip install -r requirements.txt`
- [x] `app/config.py` — `Settings(BaseSettings)` class reading env vars
- [x] `app/agent/llm.py` — thin Groq client wrapper (single `ask(prompt: str) -> str` function)
- [x] `app/main.py`:
  - [x] `GET /health` → `{"status": "ok"}`
  - [x] Temporary `POST /chat` → calls Groq wrapper directly, no LangGraph yet, just to prove the chain works
- [x] Run locally: `uvicorn app.main:app --reload --port 8002`
- [x] Test `/health` with curl/Postman
- [x] Test `/chat` with curl/Postman — confirm real Groq response comes back
- [x] Write `Dockerfile` (Python 3.12 base, matches existing services)
- [x] Build + run container locally, re-test `/health` and `/chat` inside Docker
- [x] Commit ("feat: service skeleton with Groq round-trip")

**Done when:** `/health` and `/chat` both work locally and in Docker.

---

## Milestone 2 — First real tool + LangGraph agent (`sql_query_tool`)

- [x] Add deps: `langgraph`, `langchain-groq`, `sqlalchemy`, `psycopg[binary]` (or `asyncpg`)
- [x] Create **read-only Postgres role** (`chat_ro_user`) — DB side, not app code
- [x] `app/db/postgres.py` — read-only connection pool using `POSTGRES_READONLY_URL`
- [x] Confirm exact schema fields needed (resolve plan §12.1: `ArticleFakeNews`, `ArticleTopic`, etc. field names)
- [x] `app/tools/sql_tool.py`:
  - [x] Define `metric` enum: `fake_news_count`, `articles_by_topic`, `sentiment_breakdown`, `propaganda_count`, `hate_speech_count`, `article_lookup`, `dialect_breakdown`
  - [x] Write one parametrized SQL query per metric
  - [x] Return shape: `{"rows": [...], "sql_used": "...", "source_refs": [...]}`
- [x] Write unit tests for each metric in `tests/test_sql_tool.py` (against a test/staging DB)
- [x] `app/agent/state.py` — `AgentState` TypedDict (`messages`, `sources`, `session_id`, `user_id`)
- [x] `app/agent/prompts.py` — system prompt (identity, tool-preference rule, citation rule, language-matching rule, no-fabrication rule)
- [x] `app/agent/graph.py` — LangGraph ReAct-style agent wired to `sql_query_tool` only
- [x] Wire `/chat` in `main.py` to call the LangGraph agent instead of the raw Groq wrapper
- [x] `app/schemas/request.py` / `app/schemas/response.py` — formalize `ChatRequest` / `ChatResponse` models per plan §4
- [x] Manually test: "How many fake news articles were detected last month?" → correct tool call + correct answer
- [x] Commit ("feat: sql_query_tool + LangGraph agent")

**Done when:** agent correctly answers at least 3 different stat questions using real DB data, citing article IDs.

---

## Milestone 3 — Django proxy endpoint

- [x] In Django: add `CHAT_SERVICE_URL` and `CHAT_SERVICE_INTERNAL_TOKEN` to settings/env
- [x] New view: `POST /api/v1/chat/`
  - [x] Verify JWT (reuse existing auth)
  - [ ] Apply DRF throttling/rate limit (no need for now)
  - [x] Forward `{session_id, message, user_id}` to chat service with internal token header
  - [x] Return chat service's JSON response unchanged
- [x] Confirm chat service rejects requests without the internal token
- [x] Test end-to-end: frontend-style request → Django → chat service → Groq → back
- [x] Commit on Django repo ("feat: chat proxy endpoint")

**Done when:** a JWT-authenticated request through Django reaches the chat service and gets a real answer.

---

## Milestone 4 — `graph_query_tool` (Neo4j)

- [x] Confirm current Neo4j label/relationship conventions used by `kg_sync` (resolve plan §12.2)
- [x] ~~Create read-only Neo4j role/user~~ — N/A, Aura Free has no RBAC; enforcement moved to app-level (read-only transactions + no write Cypher in tool code)
- [x] `app/db/neo4j.py` — driver wrapper using `NEO4J_*` vars, enforcing read-only via explicit read transactions
- [x] `app/tools/graph_tool.py`:
  - [x] Define `query_type` enum (7 total): `entity_connections`, `entity_mentions`, `shared_entities_between_articles`, `most_connected_entities`, `article_verdict`, `claims_for_article`, `analysis_for_article`
  - [x] Write one parametrized Cypher template per query_type
  - [x] Return shape: `{"rows": [...], "source_refs": [...]}`
- [x] Add `graph_query_tool` to the LangGraph agent's tool list
- [x] Unit tests in `tests/test_graph_tool.py`
- [x] Manually test: "Who is connected to [entity]?" → correct Cypher template used, correct answer
- [x] Commit ("feat: graph_query_tool")

**Done when:** agent correctly answers at least 2 relationship questions using real graph data.

---

## Milestone 5 — Memory (multi-turn)

- [x] Add dep: Redis client (`redis` / `redis[hiredis]`)
- [x] `app/memory/redis_checkpointer.py` — LangGraph checkpointer backed by Redis, key prefix `chat:checkpoint:{session_id}`
- [x] Wire checkpointer into `app/agent/graph.py` compilation
- [x] Set `CHAT_SESSION_TTL_SECONDS` TTL on session keys
- [x] Implement history trimming: keep last `CHAT_HISTORY_MAX_TURNS` verbatim, summarize older turns beyond that
- [x] Manual test: ask a question, then a follow-up ("and what about last month?") using the same `session_id` — confirm context carries over
- [x] Manual test: new `session_id` → confirm no memory leaks across sessions
- [x] Commit ("feat: Redis-backed multi-turn memory")

**Done when:** follow-up questions correctly resolve using prior turn context, and TTL/trimming work as expected.

---

## Milestone 6 — Citations end-to-end

- [x] `app/tools/article_tool.py` — `get_article_detail(article_id)` for resolving full title/snippet
- [x] Confirm every tool (`sql_query_tool`, `graph_query_tool`) consistently returns `source_refs`
- [x] Agent post-processing step: merge all `source_refs` collected during the turn into final `sources[]` in the response
- [x] Update system prompt to explicitly require citing every ID it used
- [x] Manual test: ask a question, verify `sources[]` in the JSON response matches what was actually used
- [x] Commit ("feat: citation resolution")

**Done when:** every factual answer includes a non-empty, accurate `sources[]` array.

---

## Milestone 7 — `hybrid_search_tool`

- [x] Confirm reuse path for `_embed` (call existing Django/analysis embedding function - the function is in app/services/nlp_client.py)
- [x] `app/tools/hybrid_tool.py`:
  - [x] Step 1: embed query text
  - [x] Step 2: pgvector cosine similarity search over `search_vector`
  - [x] Step 3: for each top article, pull mentioned entities from Neo4j
  - [x] Step 4: merge into single ranked result set with combined `source_refs`
- [x] Add tool to agent's tool list
- [ ] Manual test: "Articles about the election, and who's mentioned" → correct combined results
- [x] Commit ("feat: hybrid_search_tool")

**Done when:** at least one combined semantic+graph question is answered correctly with merged citations.

---

## Milestone 8 — Frontend chat widget

- [ ] Generate/persist `session_id` client-side (e.g. `localStorage`, created on first load)
- [ ] Chat UI component: message list, input box, send button
- [ ] Call `POST /api/v1/chat/` with `{session_id, message}`
- [ ] Render `answer` text
- [ ] Render `sources[]` as clickable links (article/entity)
- [ ] Loading state while waiting for response
- [ ] Basic error state (service down / timeout)
- [ ] Commit ("feat: chat widget")

**Done when:** a real user can open the widget, ask a question, get an answer with clickable citations, and ask a follow-up.

---

## Milestone 9 — Tests, docs, hardening

- [ ] Unit tests for all tools (`sql_tool`, `graph_tool`, `hybrid_tool`) with mocked/test DB data
- [ ] `tests/test_agent_e2e.py` — at least 3 end-to-end scenarios (stat question, relationship question, follow-up question)
- [ ] `chat-service/README.md` — setup instructions, env vars, how to run locally + in Docker
- [ ] Verify `.env.example` is complete and matches plan §10
- [ ] Security checklist review (plan §8) — confirm every box is actually true in the running system:
  - [ ] Read-only DB roles confirmed (test that a write attempt fails)
  - [ ] No raw SQL/Cypher ever comes from the LLM — code review confirms only enum params reach tools
  - [ ] Internal token required and enforced
  - [ ] Rate limiting active on Django proxy
  - [ ] Groq key not logged anywhere
  - [ ] Tool calls logged (name + params, not raw rows)
- [ ] Add `chat_service` to `docker-compose.yml` (no published host port in prod config)
- [ ] Final walkthrough / demo run-through for defense
- [ ] Tag/commit ("chore: v1 complete")

**Done when:** everything above is checked and you can demo the full flow live without surprises.

---

## Open items to resolve before/while coding (carry over from plan §12)

- [ ] Confirm exact Postgres field names per metric
- [ ] Confirm Neo4j label/relationship naming conventions
- [ ] Decide: authenticated-only sessions, or also support anonymous/demo sessions for defense
- [ ] Decide: citation links deep-link into real frontend routes, or raw IDs for now