learning.md

# Project Journal — Screen Ireland RAG Chatbot

> A living document. Append to it as the project grows. Sections are stable;
> dated entries at the bottom of each section show the evolution over time.

---

## 1. What this project is

A retrieval-augmented chatbot for **screenireland.ie**. Visitors ask plain-English
questions about funding, filming, courses, or programmes; the bot answers
**only** from Screen Ireland's own published HTML and PDFs — with citations
— or says *"I don't have that"*. The hard requirement: **never fabricate**.

The architecture document (`architecture.md`) is the binding design spec. This
file (`learning.md`) is a journal — what's been built, what works, what's
pending, and the *why* behind decisions that aren't obvious from the code.

---

## 2. What we've achieved (POC — Stage 1)

### 2.1 Ingestion pipeline (Stage 1A) — ✅ complete

- **Crawler** (`ingest/crawl.py`): two-phase async crawler.
  - Phase 1: parse `sitemap.xml` → fetch every HTML page with `httpx` (3 concurrent slots, 0.4 s/slot for politeness, descriptive User-Agent).
  - Phase 2: scan downloaded HTML for `<a href="*.pdf">` links → download each PDF.
  - Resume-safe (skips already-fetched files).
  - **Result:** 2,551 HTML pages + 196 PDFs downloaded. ~500 MB on disk.

- **Extractor** (`ingest/extract.py`):
  - HTML → `trafilatura.extract(output_format="markdown")` — preserves heading hierarchy (`#`, `##`, `###`) for downstream chunking, strips nav/footer/cookie banners.
  - PDF → `PyMuPDF` per page (page numbers retained for citation).
  - PDF title quality fix: when embedded title metadata is empty, fall back to the URL filename (`IFB_Annual_Report_full2009`), not the hashed local path. This was a key quality fix found during acceptance.
  - **Result:** 2,523 HTML + 191 PDFs survived extraction.

- **Chunker** (`ingest/chunk.py`):
  - Parses markdown headings to build a `heading_path` like `Funding > Development > Limits`.
  - Heading-aware split, then ~600-token sliding window with ~80-token overlap inside each section.
  - **Heading path prepended to each chunk's text** so embeddings carry context.
  - Filters chunks with <25 tokens of content (drops news-listing fragments, subscription forms).
  - **Result:** 13,414 chunks.

- **Corpus enrichment** (added after first quality review):
  - **`category` field** — URL-derived (news / funding / report / policy / about / courses / skills / filming / strategy / catalogue / festivals / spotlight / sustainability / audience / insights / other). Tightened PDF heuristics to capture competency framework booklets, industry studies, etc.
  - **`lang` field** — `en` or `ga`. Detected via known Irish URL prefixes (`/clair-mhaoiniuchain/`, `/feilte-margai/`, `/mar-gheall-ar-bse/`, `/plean-straiteiseach-2025-2029/`) plus a Gaeilge marker-word heuristic for bilingual press releases.
  - **Decision: `lang=ga` chunks are excluded at index time** (1,419 of 13,414). The embedder is English-only; embedding Irish content with `bge-small-en` wastes compute and dilutes results. Re-indexable later when we move to a multilingual embedder.

### 2.2 Indexing (Stage 1B) — ✅ complete

- **Embedder** (`app/providers/local_bge.py`): `BAAI/bge-small-en-v1.5` via **fastembed** (ONNX runtime).
  - Original plan was `sentence-transformers`, but a broken PyTorch ABI in the local Anaconda made it unusable. Pivot to fastembed cost nothing (same model, same weights) and bought ~3× faster CPU inference plus zero torch dependency.
  - Applies the bge query prefix (`"Represent this sentence for searching relevant passages: "`) at retrieval time only.
  - L2-normalised output → cosine similarity becomes a dot product.

- **Vector store** (`ingest/index.py`): LanceDB, file-based, schema:
  ```
  id, vector(384), text, source_url, title, source_type,
  page, heading_path, category, lang
  ```
  - Resume-safe: skips IDs already in the table.
  - Indexed throughput: ~18 chunks/s on M-series CPU.
  - **Result:** 11,995 English chunks indexed in ~11 minutes. ~30 MB on disk.

### 2.3 Retrieval + answer (Stage 1C) — ✅ complete

- **`/chat` endpoint** (`app/main.py`, FastAPI + sse-starlette): POST `{"question": "..."}`, returns SSE stream with events `token`, `sources`, `done`.
- **Retrieval** (`app/retrieve.py`): embed query → LanceDB cosine top-k (default 5) → apply **relevance gate**.
- **Relevance gate (Safeguard 1)**: if the top score is below `RELEVANCE_THRESHOLD` (default 0.35), the LLM is **never called**; the bot streams the canned IDK and a closest-page link. This is the single most important component — it kills hallucinations on out-of-corpus topics at zero LLM cost. Empirically:
  - "What is the current weather in Dublin?" → score 0.341, gate fails. ✓
  - "What is the development funding limit?" → score 0.546, gate passes. ✓
  - Prompt-injection probe ("Ignore previous instructions…") → score 0.173, gate fails before the LLM sees it. ✓
- **Grounding prompt (Safeguard 2)** (`app/prompt.py`): forbids outside knowledge, demands citations on every claim, adds "verify on page" sentence for funding figures, treats question + passages as **data not instructions** (prompt-injection defense), refuses to reveal the system prompt.
- **LLM provider abstraction** (`app/providers/`): `Embedder` and `LLMProvider` ABCs. Concrete:
  - `gemini.py` — gemini-2.5-flash (primary, free).
  - `groq.py` — llama-3.3-70b-versatile (fallback).
  - `anthropic.py` — claude-haiku-4-5 (paid-switch ready, zero code change to flip).
  - `factory.py` — automatic fallback chain (primary → secondary on quota/error).

### 2.4 Widget (Stage 1D) — ✅ complete

- **`widget/widget.js`** — single self-mounting script. Renders inside a **Shadow DOM** so host-site CSS can't break it. Floating chat bubble bottom-right; click to expand a 380×560 panel. Streams from `/chat` via `fetch` + `ReadableStream` (parses SSE blocks manually since the request is POST).
- **`widget/demo.html`** — minimal landing page hosting the widget for the client demo.
- Served from FastAPI at `/widget.js` and `/demo` so a single `uvicorn app.main:app` boots the whole stack.
- Accessible defaults (keyboard `Enter` sends, `aria-live` on messages, `aria-label` on bubble/close, focus management on open). **WCAG AA work remains for Stage 2** (full contrast audit, screen-reader pass).

### 2.5 Evaluation (Stage 1E) — ✅ complete

- **`eval/questions.jsonl`** — 15 questions: 10 answerable (about, funding, tax, filming, courses, audience), 4 deliberately out-of-corpus, 1 prompt-injection probe.
- **`eval/run_eval.py`** — scores each question on `gate_ok`, `answered`, `cited`, `refused`, `leaked_prompt` as appropriate. Rate-limit-aware (`--delay 7` keeps Gemini under 10 RPM). `--no-llm` runs gate-only (fast, free).
- **Baseline numbers** (2026-06-19):
  - Without rate-limit spacing: 11/15 pass — the 4 "failures" were all HTTP 429s from Gemini's per-minute quota, not bugs.
  - With `--delay 7` (the canonical run): **15/15 pass (100%)**. Every topic green: about, audience, courses, filming, funding, injection, tax, unanswerable.
  - Use this 15/15 as the regression baseline. Any change that drops it must justify the drop.

### 2.6 Quality checkpoints we deliberately committed

- **Hands-on inspection of 20 random chunks** before declaring Stage 1A done (architecture acceptance criterion). All clean — no cookie banners, no nav menus, correct headings, correct attribution. Tightened the PDF title heuristic (URL-derived fallback) after the first pass surfaced ugly hashed titles.
- **Honest corpus audit** before moving to Stage 1B: identified that 63% of HTML is news articles (which would compete for retrieval slots vs. funding content), 15.7% is Irish-language (poor fit for English embedder), and ~4% had a "Top Picks" sidebar widget bleeding into chunks. Acted on the top two by tagging and language-filtering at index time. Left the sidebar issue documented and deferred — the relevance gate handles it in practice.

---

## 3. Key decisions and why

| Decision | Why we made it | When to revisit |
|---|---|---|
| Local bge embeddings, not hosted (Voyage / OpenAI) | Free forever, no rate limits, 384d cheap to store, privacy (text never leaves the box) | If eval recall is weak, upgrade to `bge-base` (768d) — one model swap + re-embed |
| `fastembed` over `sentence-transformers` | Local torch was broken; fastembed needs no torch and is ~3× faster on CPU | Never, unless we need PyTorch-only features |
| LanceDB over pgvector / Pinecone | Zero servers, file-based, scales to millions, cosine native | Move to Supabase pgvector when we need re-index without redeploy |
| Markdown-aware chunking, not blind 600-token splits | Preserves `H1 > H2 > H3` context per chunk; without it, sentences split mid-section and lose meaning | Tune size against `eval/questions.jsonl` |
| Exclude Irish chunks at index, keep them in `chunks.jsonl` | English embedder can't represent Irish well — wastes compute and dilutes results. Keeping them in chunks.jsonl makes re-indexing under a multilingual model a one-command job | When we want bilingual answers (would also need query language detection) |
| Two safeguards, not one (gate **and** prompt) | Gate handles "topic not in corpus"; prompt handles "topic in corpus but model wants to embellish". Defense in depth — one isn't enough for a state body | Never — both are mandatory |
| Gemini primary, Groq fallback, Anthropic paid-switch | Architecture's choice. Gemini has the most generous free tier; Groq is the fastest TTFT; Anthropic is what we move to when budget exists | `LLM_PROVIDER=anthropic` + a key is the entire migration |

---

## 4. What's NOT done yet

### 4.1 Architecture items still pending

| Stage | Item | Effort | Priority |
|---|---|---|---|
| 1.5  | Hybrid search (BM25 + bge fusion) + cross-encoder rerank | Half-day | **Medium** — add if Stage 1E shows specific-term misses (Section 481, named schemes, exact € figures) |
| 1.5  | bge-base upgrade for higher recall | 1 hour (just re-embed) | Low — only if 1.5 doesn't help |
| 2    | Embed widget on screenireland.ie via `<script>` tag | Day | High — needs client sign-off + CORS lock |
| 2    | CORS origin-locked to screenireland.ie | Trivial | High |
| 2    | HTTPS in production | Hosting concern | High |
| 2    | Cookie/consent integration with site banner | Day | High (legal — ePrivacy) |
| 2    | Rate limiting per IP + max question length | Half-day | High (abuse/cost control) |
| 2    | WCAG 2.1 AA accessibility audit on widget | 2 days | **High — legal** under EU Web Accessibility Directive for public-sector sites |
| 2    | GDPR-aware logging policy (scrub PII, retention SLA) | Day | High (legal) |
| 2    | Monitoring & metrics: latency, error rate, fallback rate, **IDK rate** | Day | High |
| 2    | Thumbs up/down feedback loop | Half-day | Medium |
| 2    | Scheduled re-crawl + incremental indexing (hash-then-skip-unchanged) | Day | **High** — funding deadlines change; stale answers are the main prod risk |
| 2    | "Sources last updated on X" stamp in widget | Trivial | Medium |
| 12.2 | Multi-turn conversation with query rewriting | 2 days | Medium |
| 12.3 | Analytics dashboard (top questions, IDK rate, satisfaction) | Week | **High value upsell** — surfaces content gaps on their own site |
| 12.4 | Eval in CI — block regressions on every deploy | Half-day | High |
| 12.5 | Paid Claude upgrade (env var swap) | 10 min | Medium |
| 12.6 | Multilingual (Gaeilge) | Week | Low — client decision |
| 12.7 | Anthropic contextual retrieval (LLM-generated context blurb per chunk) | 2 days | Medium |

### 4.2 Known gaps / tech debt

- **Google Gemini SDK deprecation**: we use `google.generativeai` (legacy). Google's recommended replacement is `google.genai`. The migration is a contained 20-line job in `app/providers/gemini.py`. Defer until paid tier or until the legacy SDK breaks.
- **"Top Picks" sidebar contamination** (~192 HTML chunks, ~4%): trafilatura is including a sidebar widget on some course pages. Fix is a small `prune_xpath` rule. Deferred — the relevance gate catches it in practice (sidebar chunks rarely beat real content for any reasonable query).
- **Annual reports dominate PDFs** (~50% of PDF chunks): 2014–2024 annual reports give ~190 chunks each. Genuine content but historical and high-noise. Consider per-year tagging in Stage 2 so an old £ amount can't beat a current one.
- **No FAQ-specific chunking**: a generic 600-token window cuts FAQ Q+A pairs unpredictably. Stage 1.5 candidate: detect FAQ pages and keep Q+A pairs together.
- **No rerank**: top-k from raw cosine. A cross-encoder reranker on the top-20 would lift precision noticeably on named terms ("Section 481", scheme names).

---

## 5. Operational notes (the stuff that bit us)

- **Gemini free tier is 10 RPM**, not "1500 RPD comfortable for demos". The first full eval run hit 429s on requests 6–10 because we fired them with no spacing. Fix was `--delay 7` in `run_eval.py`. For live demos either (a) configure `GROQ_API_KEY` so the auto-fallback kicks in, or (b) space questions.
- **Anaconda + PyTorch ABI mismatch** silently breaks `sentence-transformers`. Switching to fastembed (ONNX) sidestepped the whole class of problem.
- **`.env.example` is a template** (safe to commit). **`.env` is the real secrets** (gitignored, mode 600). Easy mistake during setup; document explicitly in the README and the file headers.
- **LanceDB locks the file while indexing** — concurrent `count_rows()` from another process blocks. Just wait for the indexer to finish.

---

## 6. How to keep this document useful

Every meaningful change to the system should land a short dated entry below.
Aim for *why*, not *what* — the code shows what changed; this file is where
the rationale survives so the next engineer (or future-you) doesn't have to
re-derive it.

### Changelog

- **2026-06-18** — Stage 1A built end-to-end. 13,414 chunks from 2,551 pages + 196 PDFs. PDF title heuristic tightened after acceptance review.
- **2026-06-19** — Stage 1A+: added `category` and `lang` tagging; Irish chunks excluded at index time. Stage 1B complete: 11,995 chunks embedded in LanceDB via fastembed (~11 min on CPU). Stage 1C complete: FastAPI + Gemini-primary/Groq-fallback streaming, both safeguards working in eval. Stage 1D complete: Shadow-DOM widget self-mounts, SSE streaming verified end-to-end. Stage 1E formalised with rate-limit-aware harness. Initial GitHub push.