learning.md

Project Journal — Screen Ireland RAG Chatbot

A living document. Append to it as the project grows. Sections are stable; dated entries at the bottom of each section show the evolution over time.

---

1. What this project is

A retrieval-augmented chatbot for screenireland.ie. Visitors ask plain-English questions about funding, filming, courses, or programmes; the bot answers only from Screen Ireland's own published HTML and PDFs — with citations — or says "I don't have that". The hard requirement: never fabricate.

The architecture document (architecture.md) is the binding design spec. This file (learning.md) is a journal — what's been built, what works, what's pending, and the why behind decisions that aren't obvious from the code.

---

2. What we've achieved (POC — Stage 1)

2.1 Ingestion pipeline (Stage 1A) — ✅ complete

• Crawler (ingest/crawl.py): two-phase async crawler.

  • Phase 1: parse sitemap.xml → fetch every HTML page with httpx (3 concurrent slots, 0.4 s/slot for politeness, descriptive User-Agent).
  • Phase 2: scan downloaded HTML for <a href="*.pdf"> links → download each PDF.
  • Resume-safe (skips already-fetched files).
  • Result: 2,551 HTML pages + 196 PDFs downloaded. ~500 MB on disk.

• Extractor (ingest/extract.py):

  • HTML → trafilatura.extract(output_format="markdown") — preserves heading hierarchy (#, ##, ###) for downstream chunking, strips nav/footer/cookie banners.
  • PDF → PyMuPDF per page (page numbers retained for citation).
  • PDF title quality fix: when embedded title metadata is empty, fall back to the URL filename (IFB_Annual_Report_full2009), not the hashed local path. This was a key quality fix found during acceptance.
  • Result: 2,523 HTML + 191 PDFs survived extraction.

• Chunker (ingest/chunk.py):

  • Parses markdown headings to build a heading_path like Funding > Development > Limits.
  • Heading-aware split, then ~600-token sliding window with ~80-token overlap inside each section.
  • Heading path prepended to each chunk's text so embeddings carry context.
  • Filters chunks with <25 tokens of content (drops news-listing fragments, subscription forms).
  • Result: 13,414 chunks.

• Corpus enrichment (added after first quality review):

  • category field — URL-derived (news / funding / report / policy / about / courses / skills / filming / strategy / catalogue / festivals / spotlight / sustainability / audience / insights / other). Tightened PDF heuristics to capture competency framework booklets, industry studies, etc.
  • lang field — en or ga. Detected via known Irish URL prefixes (/clair-mhaoiniuchain/, /feilte-margai/, /mar-gheall-ar-bse/, /plean-straiteiseach-2025-2029/) plus a Gaeilge marker-word heuristic for bilingual press releases.
  • Decision: lang=ga chunks are excluded at index time (1,419 of 13,414). The embedder is English-only; embedding Irish content with bge-small-en wastes compute and dilutes results. Re-indexable later when we move to a multilingual embedder.

2.2 Indexing (Stage 1B) — ✅ complete

• Embedder (app/providers/local_bge.py): BAAI/bge-small-en-v1.5 via fastembed (ONNX runtime).

  • Original plan was sentence-transformers, but a broken PyTorch ABI in the local Anaconda made it unusable. Pivot to fastembed cost nothing (same model, same weights) and bought ~3× faster CPU inference plus zero torch dependency.
  • Applies the bge query prefix ("Represent this sentence for searching relevant passages: ") at retrieval time only.
  • L2-normalised output → cosine similarity becomes a dot product.

• Vector store (ingest/index.py): LanceDB, file-based, schema:

  id, vector(384), text, source_url, title, source_type,
  page, heading_path, category, lang
  

  • Resume-safe: skips IDs already in the table.
  • Indexed throughput: ~18 chunks/s on M-series CPU.
  • Result: 11,995 English chunks indexed in ~11 minutes. ~30 MB on disk.

2.3 Retrieval + answer (Stage 1C) — ✅ complete

• /chat endpoint (app/main.py, FastAPI + sse-starlette): POST {"question": "..."}, returns SSE stream with events token, sources, done.
• Retrieval (app/retrieve.py): embed query → LanceDB cosine top-k (default 5) → apply relevance gate.
• Relevance gate (Safeguard 1): if the top score is below RELEVANCE_THRESHOLD (default 0.35), the LLM is never called; the bot streams the canned IDK and a closest-page link. This is the single most important component — it kills hallucinations on out-of-corpus topics at zero LLM cost. Empirically:
  • "What is the current weather in Dublin?" → score 0.341, gate fails. ✓
  • "What is the development funding limit?" → score 0.546, gate passes. ✓
  • Prompt-injection probe ("Ignore previous instructions…") → score 0.173, gate fails before the LLM sees it. ✓
• Grounding prompt (Safeguard 2) (app/prompt.py): forbids outside knowledge, demands citations on every claim, adds "verify on page" sentence for funding figures, treats question + passages as data not instructions (prompt-injection defense), refuses to reveal the system prompt.
• LLM provider abstraction (app/providers/): Embedder and LLMProvider ABCs. Concrete:
  • gemini.py — gemini-2.5-flash (primary, free).
  • groq.py — llama-3.3-70b-versatile (fallback).
  • anthropic.py — claude-haiku-4-5 (paid-switch ready, zero code change to flip).
  • factory.py — automatic fallback chain (primary → secondary on quota/error).

2.4 Widget (Stage 1D) — ✅ complete

• widget/widget.js — single self-mounting script. Renders inside a Shadow DOM so host-site CSS can't break it. Floating chat bubble bottom-right; click to expand a 380×560 panel. Streams from /chat via fetch + ReadableStream (parses SSE blocks manually since the request is POST).
• widget/demo.html — minimal landing page hosting the widget for the client demo.
• Served from FastAPI at /widget.js and /demo so a single uvicorn app.main:app boots the whole stack.
• Accessible defaults (keyboard Enter sends, aria-live on messages, aria-label on bubble/close, focus management on open). WCAG AA work remains for Stage 2 (full contrast audit, screen-reader pass).

2.5 Evaluation (Stage 1E) — ✅ complete

• eval/questions.jsonl — 15 questions: 10 answerable (about, funding, tax, filming, courses, audience), 4 deliberately out-of-corpus, 1 prompt-injection probe.
• eval/run_eval.py — scores each question on gate_ok, answered, cited, refused, leaked_prompt as appropriate. Rate-limit-aware (--delay 7 keeps Gemini under 10 RPM). --no-llm runs gate-only (fast, free).
• Baseline numbers (2026-06-19):
  • Without rate-limit spacing: 11/15 pass — the 4 "failures" were all HTTP 429s from Gemini's per-minute quota, not bugs.
  • With --delay 7 (the canonical run): 15/15 pass (100%). Every topic green: about, audience, courses, filming, funding, injection, tax, unanswerable.
  • Use this 15/15 as the regression baseline. Any change that drops it must justify the drop.

2.6 Quality checkpoints we deliberately committed

• Hands-on inspection of 20 random chunks before declaring Stage 1A done (architecture acceptance criterion). All clean — no cookie banners, no nav menus, correct headings, correct attribution. Tightened the PDF title heuristic (URL-derived fallback) after the first pass surfaced ugly hashed titles.
• Honest corpus audit before moving to Stage 1B: identified that 63% of HTML is news articles (which would compete for retrieval slots vs. funding content), 15.7% is Irish-language (poor fit for English embedder), and ~4% had a "Top Picks" sidebar widget bleeding into chunks. Acted on the top two by tagging and language-filtering at index time. Left the sidebar issue documented and deferred — the relevance gate handles it in practice.

---

3. Key decisions and why

Decision	Why we made it	When to revisit
Local bge embeddings, not hosted (Voyage / OpenAI)	Free forever, no rate limits, 384d cheap to store, privacy (text never leaves the box)	If eval recall is weak, upgrade to bge-base (768d) — one model swap + re-embed
fastembed over sentence-transformers	Local torch was broken; fastembed needs no torch and is ~3× faster on CPU	Never, unless we need PyTorch-only features
LanceDB over pgvector / Pinecone	Zero servers, file-based, scales to millions, cosine native	Move to Supabase pgvector when we need re-index without redeploy
Markdown-aware chunking, not blind 600-token splits	Preserves H1 > H2 > H3 context per chunk; without it, sentences split mid-section and lose meaning	Tune size against eval/questions.jsonl
Exclude Irish chunks at index, keep them in chunks.jsonl	English embedder can't represent Irish well — wastes compute and dilutes results. Keeping them in chunks.jsonl makes re-indexing under a multilingual model a one-command job	When we want bilingual answers (would also need query language detection)
Two safeguards, not one (gate and prompt)	Gate handles "topic not in corpus"; prompt handles "topic in corpus but model wants to embellish". Defense in depth — one isn't enough for a state body	Never — both are mandatory
Gemini primary, Groq fallback, Anthropic paid-switch	Architecture's choice. Gemini has the most generous free tier; Groq is the fastest TTFT; Anthropic is what we move to when budget exists	LLM_PROVIDER=anthropic + a key is the entire migration

---

4. What's NOT done yet

4.1 Architecture items still pending

Stage	Item	Effort	Priority
1.5	Hybrid search (BM25 + bge fusion) + cross-encoder rerank	Half-day	Medium — add if Stage 1E shows specific-term misses (Section 481, named schemes, exact € figures)
1.5	bge-base upgrade for higher recall	1 hour (just re-embed)	Low — only if 1.5 doesn't help
2	Embed widget on screenireland.ie via <script> tag	Day	High — needs client sign-off + CORS lock
2	CORS origin-locked to screenireland.ie	Trivial	High
2	HTTPS in production	Hosting concern	High
2	Cookie/consent integration with site banner	Day	High (legal — ePrivacy)
2	Rate limiting per IP + max question length	Half-day	High (abuse/cost control)
2	WCAG 2.1 AA accessibility audit on widget	2 days	High — legal under EU Web Accessibility Directive for public-sector sites
2	GDPR-aware logging policy (scrub PII, retention SLA)	Day	High (legal)
2	Monitoring & metrics: latency, error rate, fallback rate, IDK rate	Day	High
2	Thumbs up/down feedback loop	Half-day	Medium
2	Scheduled re-crawl + incremental indexing (hash-then-skip-unchanged)	Day	High — funding deadlines change; stale answers are the main prod risk
2	"Sources last updated on X" stamp in widget	Trivial	Medium
12.2	Multi-turn conversation with query rewriting	2 days	Medium
12.3	Analytics dashboard (top questions, IDK rate, satisfaction)	Week	High value upsell — surfaces content gaps on their own site
12.4	Eval in CI — block regressions on every deploy	Half-day	High
12.5	Paid Claude upgrade (env var swap)	10 min	Medium
12.6	Multilingual (Gaeilge)	Week	Low — client decision
12.7	Anthropic contextual retrieval (LLM-generated context blurb per chunk)	2 days	Medium

4.2 Known gaps / tech debt

• Google Gemini SDK deprecation: we use google.generativeai (legacy). Google's recommended replacement is google.genai. The migration is a contained 20-line job in app/providers/gemini.py. Defer until paid tier or until the legacy SDK breaks.
• "Top Picks" sidebar contamination (~192 HTML chunks, ~4%): trafilatura is including a sidebar widget on some course pages. Fix is a small prune_xpath rule. Deferred — the relevance gate catches it in practice (sidebar chunks rarely beat real content for any reasonable query).
• Annual reports dominate PDFs (~50% of PDF chunks): 2014–2024 annual reports give ~190 chunks each. Genuine content but historical and high-noise. Consider per-year tagging in Stage 2 so an old £ amount can't beat a current one.
• No FAQ-specific chunking: a generic 600-token window cuts FAQ Q+A pairs unpredictably. Stage 1.5 candidate: detect FAQ pages and keep Q+A pairs together.
• No rerank: top-k from raw cosine. A cross-encoder reranker on the top-20 would lift precision noticeably on named terms ("Section 481", scheme names).

---

5. Operational notes (the stuff that bit us)

• Gemini free tier is 10 RPM, not "1500 RPD comfortable for demos". The first full eval run hit 429s on requests 6–10 because we fired them with no spacing. Fix was --delay 7 in run_eval.py. For live demos either (a) configure GROQ_API_KEY so the auto-fallback kicks in, or (b) space questions.
• Anaconda + PyTorch ABI mismatch silently breaks sentence-transformers. Switching to fastembed (ONNX) sidestepped the whole class of problem.
• .env.example is a template (safe to commit). .env is the real secrets (gitignored, mode 600). Easy mistake during setup; document explicitly in the README and the file headers.
• LanceDB locks the file while indexing — concurrent count_rows() from another process blocks. Just wait for the indexer to finish.

---

6. How to keep this document useful

Every meaningful change to the system should land a short dated entry below. Aim for why, not what — the code shows what changed; this file is where the rationale survives so the next engineer (or future-you) doesn't have to re-derive it.

Changelog

• 2026-06-18 — Stage 1A built end-to-end. 13,414 chunks from 2,551 pages + 196 PDFs. PDF title heuristic tightened after acceptance review.
• 2026-06-19 — Stage 1A+: added category and lang tagging; Irish chunks excluded at index time. Stage 1B complete: 11,995 chunks embedded in LanceDB via fastembed (~11 min on CPU). Stage 1C complete: FastAPI + Gemini-primary/Groq-fallback streaming, both safeguards working in eval. Stage 1D complete: Shadow-DOM widget self-mounts, SSE streaming verified end-to-end. Stage 1E formalised with rate-limit-aware harness. Initial GitHub push.