Spaces:
Sleeping
Sleeping
EUR-Lex AI Chat — Full Implementation Plan
Project: EUR-Lex AI Chat — Ask questions about EU law in plain language, get answers with citations to real EUR-Lex documents. Build target: ~15 hours total (one-time, laptop-based) Ongoing cost: $0.00/month — runs 100% autonomously after deployment
Overview
Architecture
DATA FLOW (fully automated):
EUR-Lex Cellar SPARQL (public)
│ GitHub Actions queries daily
▼
Cellar REST RDF → XHTML (public, no auth)
│ GitHub Actions downloads via RDF graph traversal
▼
GitHub Actions Runner (free, 7GB RAM, 2-core CPU)
│ pip install → chunk → embed (sentence-transformers ONNX) → merge
▼
HuggingFace Hub Dataset (public, free storage)
│ vectors.npy + chunks.json
│ ▲ │
│ │ ▼
│ │ Render (512MB, free)
│ │ ├── startup: download from HF Hub
│ │ ├── hourly: check HF Hub for updates via /refresh
│ │ └── live: FastAPI + numpy KNN (brute-force cosine similarity)
│ │ │
│ │ ▼
│ │ ┌─────────────────┐
│ └──────────────┤ Groq API │
│ │ (Llama 3.3 70B) │
│ └────────┬─────────┘
│ │
USER FLOW: │
[Google Search] │
│ user finds FAQ/blog post
▼
[Browser] ─► [Vercel: Astro site] ─► [Render: FastAPI API] ─► [Groq]
Static HTML pages /chat endpoint answer
+ React chat island numpy KNN + RAG
KEEP ALIVE:
[cron-job.org] ──► [Render /health] every 5 min
[cron-job.org] ──► [Render /refresh] every 60 min
Files Layout
eur-lex-ai-chat/
├── backend/
│ ├── main.py # FastAPI: /chat, /health, /refresh
│ ├── search.py # numpy KNN search over pre-loaded vectors
│ ├── rag.py # Build prompt, call Groq, parse citations
│ ├── data_loader.py # Download index from HF Hub at startup
│ ├── rate_limit.py # Per-IP + global rate limiting
│ ├── requirements.txt # fastapi, uvicorn, numpy, huggingface_hub, httpx
│ └── startup.sh # Render entry point
├── frontend/
│ ├── src/
│ │ ├── pages/
│ │ │ ├── index.astro # Landing page (SEO + chat island)
│ │ │ ├── faq.astro # FAQ page (JSON-LD)
│ │ │ └── blog/
│ │ │ ├── index.astro # Blog listing
│ │ │ └── posts/ # Markdown blog posts
│ │ ├── components/
│ │ │ ├── ChatWidget.jsx # React chat island
│ │ │ └── SeoHead.astro # JSON-LD + OG tags
│ │ └── layouts/
│ │ └── Base.astro # Main layout
│ ├── astro.config.mjs
│ ├── tailwind.config.mjs
│ └── package.json
├── scripts/
│ ├── build_index.py # Laptop: FULL build from scratch (one-time)
│ └── update_index.py # GitHub Actions: INCREMENTAL update (daily)
├── .github/
│ └── workflows/
│ └── update-index.yml # GitHub Actions workflow
├── proposal.md
├── implementation-plan.md
└── README.md
Phase 0: Pre-flight Checklist
Goal: Verify every tool, API key, and service is working before we start building.
Step 0.1 — Verify Node.js (needs >=22.12.0 for Astro 5)
source ~/.nvm/nvm.sh
nvm use v22.22.3
node --version # must show v22.22.3
npm --version # must show 10.x or higher
If nvm is not found, install it:
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.5/install.sh | bash
source ~/.bashrc
nvm install v22.22.3
nvm use v22.22.3
Step 0.2 — Verify Python venv + packages
source ~/Desktop/EUProjects/.venv/bin/activate
python3 --version # must show 3.12.x
pip list | grep -E "fastapi|uvicorn|numpy|httpx|huggingface|polars|eurlxp|beautifulsoup4|pymupdf|tqdm|pydantic"
Expected: fastapi, uvicorn, numpy, httpx, huggingface_hub, polars, eurlxp, beautifulsoup4, pymupdf, tqdm, pydantic.
If any are missing:
pip install fastapi uvicorn numpy httpx huggingface_hub polars eurlxp[sparql] beautifulsoup4 pymupdf tqdm
Step 0.3 — Install sentence-transformers (needed for Phase 1)
source ~/Desktop/EUProjects/.venv/bin/activate
pip install sentence-transformers
This installs torch (~2GB). It's a one-time install for the laptop build. The build script runs once.
Step 0.4 — Verify GROQ_API_KEY
source ~/Desktop/EUProjects/.venv/bin/activate
python3 -c "
import os, httpx
key = os.environ.get('GROQ_API_KEY')
if not key: print('MISSING: GROQ_API_KEY not found in env')
else:
r = httpx.get('https://api.groq.com/openai/v1/models',
headers={'Authorization': f'Bearer {key}'}, timeout=10)
assert r.status_code == 200, f'GROQ API returned {r.status_code}'
models = [m['id'] for m in r.json()['data']]
print(f'GROQ OK — {len(models)} models available')
print(f' Default: llama-3.3-70b-versatile')
"
If GROQ_API_KEY is missing, it should be in /home/nedaktov/Desktop/NedCode3/.env:
source /home/nedaktov/Desktop/NedCode3/.env
echo "GROQ_API_KEY is set: ${GROQ_API_KEY:0:10}..."
Step 0.5 — Verify HuggingFace Hub access
source ~/Desktop/EUProjects/.venv/bin/activate
python3 -c "
from huggingface_hub import HfApi
api = HfApi()
# Test anonymous read access
try:
api.dataset_info('hf-internal-testing/dummy_dataset')
print('HF Hub accessible (anonymous)')
except Exception as e:
print(f'HF Hub error: {e}')
"
Step 0.6 — Create HuggingFace token and log in
We need a write token to upload the vector index. Create one at: https://huggingface.co/settings/tokens
Create a token with "write" permissions. Then:
source ~/Desktop/EUProjects/.venv/bin/activate
huggingface-cli login --token YOUR_HF_TOKEN
# Or use the Python API:
python3 -c "
from huggingface_hub import login
login(token='YOUR_HF_TOKEN', add_to_git_credential=True)
print('Logged in to HuggingFace')
"
Step 0.7 — Create project directories
mkdir -p ~/Desktop/EUProjects/eur-lex-ai-chat/{backend,frontend/src/{pages,components,layouts,pages/blog/posts},scripts,.github/workflows,data}
Step 0.8 — Verify EUR-Lex SPARQL endpoint + eurlxp
source ~/Desktop/EUProjects/.venv/bin/activate
python3 -c "
from eurlxp import get_documents
docs = get_documents(types=['REG'], limit=3)
print(f'SPARQL OK — got {len(docs)} documents')
for d in docs:
print(f' {d[\"celex\"]} ({d[\"type\"]}) — {d[\"date\"]}')
"
Expected output:
SPARQL OK — got 3 documents
32025R1355R(02) (REG) — 2026-03-27
...
Step 0.9 — Verify EUR-Lex HTML fetch + parse
⚠️ Known issue: eurlxp.parse_html() has a Polars schema bug — the modifier field is conditionally included, causing schema mismatch. Use the internal parser directly with a fixed schema instead.
source ~/Desktop/EUProjects/.venv/bin/activate
python3 -c "
from eurlxp import get_html_by_celex_id
from eurlxp.parser import _parse_html_with_beautifulsoup as internal_parse
import polars as pl
html = get_html_by_celex_id('32019R0947', language='en')
print(f'HTML fetched: {len(html)} bytes')
results = internal_parse(html)
print(f'Parsed: {len(results)} elements')
# Build records with fixed schema (workaround for eurlxp Polars bug)
records = []
for r in results:
records.append({
'text': r.text,
'type': r.item_type,
'ref': str(r.ref),
'modifier': r.modifier,
'document': r.context.document,
'article': r.context.article,
'article_subtitle': r.context.article_subtitle,
'paragraph': r.context.paragraph,
'group': r.context.group,
'section': r.context.section,
})
df = pl.DataFrame(records, schema={
'text': pl.Utf8, 'type': pl.Utf8, 'ref': pl.Utf8,
'modifier': pl.Utf8, 'document': pl.Utf8, 'article': pl.Utf8,
'article_subtitle': pl.Utf8, 'paragraph': pl.Utf8,
'group': pl.Utf8, 'section': pl.Utf8,
})
print(f'DataFrame: {len(df)} rows')
texts = df.filter(pl.col('type') == 'text')
print(f'Text elements: {len(texts)}')
print(texts.head(5))
"
Step 0.10 — Pre-flight checklist summary
| # | Check | Status | Fix if failing |
|---|---|---|---|
| 1 | Node.js >=22.12.0 | nvm install v22.22.3 |
|
| 2 | Python venv active | source ~/Desktop/EUProjects/.venv/bin/activate |
|
| 3 | All Python packages installed | pip install -r requirements.txt |
|
| 4 | sentence-transformers installed | pip install sentence-transformers |
|
| 5 | GROQ_API_KEY in env | Add to .env and source it | |
| 6 | HF Hub accessible | Check internet, try again | |
| 7 | HF write token + login | Create token at huggingface.co/settings/tokens | |
| 8 | Project directories exist | Run mkdir command above | |
| 9 | EUR-Lex SPARQL works | Check internet, try later | |
| 10 | EUR-Lex HTML fetch works | Check eurlxp version, try again |
Phase 1: Build the Data Index
Goal: Create the vector index of all EU legislative acts. This runs ONCE on your laptop (~3-5 hours). Output: vectors.npy + chunks.json uploaded to HuggingFace Hub.
Scope: All regulations (REG), directives (DIR), implementing regulations (REG_IMPL), and implementing directives (DIR_IMPL) with English titles, published from 2000-01-01 onwards. This covers ~25 years of EU legislation — the laws people actually search for.
Step 1.1 — Write scripts/build_index.py
This is the main build script. Save it as ~/Desktop/EUProjects/eur-lex-ai-chat/scripts/build_index.py:
#!/usr/bin/env python3
"""
build_index.py — One-time full build of the EUR-Lex vector index.
Pipeline:
1. SPARQL query → list of CELEX IDs with metadata
2. For each CELEX ID: RDF graph traversal → XHTML content
3. Parse HTML → structured text via eurlxp parser
4. Chunk into passages (by article/paragraph)
5. Embed with sentence-transformers (all-MiniLM-L6-v2, 384-dim)
6. Upload vectors.npy + chunks.json to HuggingFace Hub
Usage:
source ~/Desktop/EUProjects/.venv/bin/activate
HF_TOKEN=hf_yourtoken python3 scripts/build_index.py
Output:
data/vectors.npy — numpy array of shape (N, 384), float32
data/chunks.json — list of dicts with text + metadata
data/last_updated.txt — ISO timestamp of build
"""
import json
import logging
import os
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
import numpy as np
import requests
from tqdm import tqdm
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)
DATA_DIR = os.path.join(os.path.dirname(os.path.dirname(__file__)), "data")
os.makedirs(DATA_DIR, exist_ok=True)
# ── Configuration ──────────────────────────────────────────────────────────
SPARQL_ENDPOINT = "https://publications.europa.eu/webapi/rdf/sparql"
# Document types to include
DOC_TYPES = [
"REG", # Regulations
"DIR", # Directives
"REG_IMPL", # Implementing regulations
"DIR_IMPL", # Implementing directives
]
# Date range: 2000-01-01 onwards (covers 25+ years of EU law)
FROM_DATE = "2000-01-01"
# How many documents to fetch per SPARQL page
SPARQL_PAGE_SIZE = 1000
# How many parallel download workers
DOWNLOAD_WORKERS = 20
# Embedding model
EMBEDDING_MODEL = "all-MiniLM-L6-v2" # 384-dim, runs on CPU
# HuggingFace dataset name
HF_DATASET_NAME = "eurlex-chat-data"
# ── Step 1: SPARQL Query ───────────────────────────────────────────────────
def make_type_filter(types):
"""Build SPARQL FILTER for document types."""
type_uris = [
f"<http://publications.europa.eu/resource/authority/resource-type/{t}>"
for t in types
]
return " ||\n ".join(f"?type = {uri}" for uri in type_uris)
def query_all_documents():
"""Query all documents matching our criteria via SPARQL with pagination."""
logger.info("Querying SPARQL for documents...")
all_docs = []
offset = 0
while True:
type_filter = make_type_filter(DOC_TYPES)
query = f"""
PREFIX cdm: <http://publications.europa.eu/ontology/cdm#>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
SELECT DISTINCT ?doc ?type ?celex ?title ?date
WHERE {{
?doc cdm:work_has_resource-type ?type .
?doc cdm:resource_legal_id_celex ?celex .
?doc dc:title ?title .
?doc cdm:work_date_document ?date .
FILTER({type_filter})
FILTER(LANG(?title) = "en")
FILTER(?date >= "{FROM_DATE}"^^xsd:date)
}}
ORDER BY DESC(?date)
OFFSET {offset}
LIMIT {SPARQL_PAGE_SIZE}
"""
r = requests.get(
SPARQL_ENDPOINT,
params={"query": query, "format": "json"},
timeout=60,
)
r.raise_for_status()
data = r.json()
bindings = data["results"]["bindings"]
if not bindings:
break
for b in bindings:
all_docs.append({
"celex": b["celex"]["value"],
"title": b["title"]["value"],
"date": b["date"]["value"],
"type": b["type"]["value"].split("/")[-1],
"cellar_url": b["doc"]["value"],
})
offset += SPARQL_PAGE_SIZE
logger.info(f" Fetched {len(all_docs)} documents so far...")
if len(bindings) < SPARQL_PAGE_SIZE:
break
logger.info(f"Total documents found: {len(all_docs)}")
return all_docs
# ── Step 2: Download XHTML Content ─────────────────────────────────────────
def fetch_document_xhtml(doc):
"""Fetch a document's XHTML content via Cellar RDF graph traversal.
Uses the same approach as eurlxp's _fetch_html_via_sparql:
1. Get work RDF → find English expression
2. Get expression RDF → find XHTML manifestation
3. Download XHTML
"""
celex = doc["celex"]
try:
# Step 1: Get work RDF graph
work_url = f"http://publications.europa.eu/resource/celex/{celex}?language=eng"
r = requests.get(work_url, timeout=30)
r.raise_for_status()
# Parse RDF/XML to find English expression
from xml.etree import ElementTree as ET
root = ET.fromstring(r.content)
# Namespaces
ns = {
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"cdm": "http://publications.europa.eu/ontology/cdm#",
}
# Find the English expression (ENG)
expressions = root.findall(".//cdm:work_has_expression", ns)
expression_url = None
for expr in expressions:
resource = expr.get("{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource")
if resource and resource.endswith(".ENG"):
expression_url = resource
break
if not expression_url and expressions:
# Fallback to first expression
resource = expressions[0].get(
"{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource"
)
if resource:
expression_url = resource
if not expression_url:
logger.warning(f" No expression found for {celex}")
return None
# Step 2: Get expression RDF to find XHTML manifestation
r2 = requests.get(expression_url, timeout=30)
r2.raise_for_status()
expr_root = ET.fromstring(r2.content)
manifestations = expr_root.findall(
".//cdm:expression_manifested_by_manifestation", ns
)
xhtml_url = None
for manif in manifestations:
resource = manif.get(
"{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource"
)
if resource and resource.endswith(".xhtml"):
xhtml_url = resource
break
if not xhtml_url and manifestations:
# Fallback to .fmx4 (Formex 4 format)
for manif in manifestations:
resource = manif.get(
"{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource"
)
if resource and resource.endswith(".fmx4"):
xhtml_url = resource
break
if not xhtml_url:
logger.warning(f" No XHTML manifestation for {celex}")
return None
# Step 3: Download the XHTML content
r3 = requests.get(
xhtml_url,
headers={"Accept": "application/xhtml+xml, text/html"},
timeout=30,
)
r3.raise_for_status()
html = r3.text
if len(html) < 100:
logger.warning(f" Empty content for {celex}")
return None
return html
except requests.RequestException as e:
logger.warning(f" HTTP error for {celex}: {e}")
return None
except ET.ParseError as e:
logger.warning(f" XML parse error for {celex}: {e}")
return None
except Exception as e:
logger.warning(f" Unexpected error for {celex}: {e}")
return None
# ── Step 3: Parse HTML → Structured Text ───────────────────────────────────
def parse_html_to_chunks(html, celex_id, title):
"""Parse EUR-Lex HTML into text chunks using eurlxp internal parser.
⚠️ Uses _parse_html_with_beautifulsoup directly instead of parse_html()
because eurlxp's parse_html() has a Polars schema bug (conditional modifier
field causes schema mismatch). This workaround uses a fixed schema.
"""
from eurlxp.parser import _parse_html_with_beautifulsoup as internal_parse
import polars as pl
try:
results = internal_parse(html)
except Exception as e:
logger.warning(f" Parse error for {celex_id}: {e}")
return []
if not results:
return []
# Convert to DataFrame with fixed schema (bypasses eurlxp bug)
records = []
for r in results:
records.append({
'text': r.text,
'type': r.item_type,
'ref': str(r.ref),
'modifier': r.modifier,
'document': r.context.document,
'article': r.context.article,
'article_subtitle': r.context.article_subtitle,
'paragraph': r.context.paragraph,
'group': r.context.group,
'section': r.context.section,
})
df = pl.DataFrame(records, schema={
'text': pl.Utf8, 'type': pl.Utf8, 'ref': pl.Utf8,
'modifier': pl.Utf8, 'document': pl.Utf8, 'article': pl.Utf8,
'article_subtitle': pl.Utf8, 'paragraph': pl.Utf8,
'group': pl.Utf8, 'section': pl.Utf8,
})
if len(df) == 0:
return []
chunks = []
current_article = None
current_text = []
# Process rows in order
for row in df.to_dicts():
text = row.get("text", "").strip()
row_type = row.get("type", "")
article = row.get("article", None)
if not text:
continue
# Skip very short fragments (titles, headers)
if len(text) < 40 and row_type in ("doc-title", "art-title", "art-subtitle", "group-title"):
if row_type == "doc-title":
pass # We already have the title from SPARQL
elif row_type == "art-title" and article:
# Finalize previous article chunk
if current_text:
chunk_text = " ".join(current_text)
if len(chunk_text) > 50:
chunks.append({
"text": chunk_text,
"celex": celex_id,
"title": title,
"article": current_article,
"type": "article",
})
current_text = []
current_article = article
continue
# Skip pure metadata rows
if row_type in ("doc-title", "art-subtitle", "group-title", "section-title"):
continue
# Skip notice/note rows
if row.get("modifier") in ("note", "signatory"):
continue
current_text.append(text)
# If text is a full paragraph and we have enough context, finalize
if len(text) > 100 and current_text:
# Check if this is a natural break (new article, new section)
pass
# Flush remaining text
if current_text:
chunk_text = " ".join(current_text)
if len(chunk_text) > 50:
chunks.append({
"text": chunk_text,
"celex": celex_id,
"title": title,
"article": current_article,
"type": "article",
})
# If no article-based chunks found, fall back to paragraph splitting
if not chunks:
# Try to split by paragraphs (double newlines)
paragraphs = [
p.strip() for p in html.replace("</p>", "\n\n")
.replace("</div>", "\n\n")
.replace("<br/>", "\n")
.split("\n\n")
if len(p.strip()) > 80
]
for i, para in enumerate(paragraphs):
chunks.append({
"text": para,
"celex": celex_id,
"title": title,
"article": None,
"type": "paragraph",
})
return chunks
# ── Step 4: Embed ──────────────────────────────────────────────────────────
def embed_chunks(all_chunks, batch_size=128):
"""Embed all chunks using sentence-transformers."""
from sentence_transformers import SentenceTransformer
logger.info(f"Loading embedding model: {EMBEDDING_MODEL}")
model = SentenceTransformer(EMBEDDING_MODEL)
logger.info("Model loaded")
texts = [c["text"] for c in all_chunks]
all_embeddings = []
logger.info(f"Embedding {len(texts)} chunks in batches of {batch_size}...")
for i in range(0, len(texts), batch_size):
batch = texts[i : i + batch_size]
embeddings = model.encode(batch, show_progress_bar=False, normalize_embeddings=True)
all_embeddings.append(embeddings)
if (i // batch_size) % 10 == 0:
logger.info(f" Embedded {min(i + batch_size, len(texts))}/{len(texts)}")
return np.vstack(all_embeddings).astype(np.float32)
# ── Step 5: Upload to HuggingFace Hub ──────────────────────────────────────
def upload_to_hub(vectors, chunks, dataset_name, token):
"""Upload vectors.npy + chunks.json + last_updated.txt to HF Hub."""
from huggingface_hub import HfApi, create_repo
api = HfApi()
repo_id = f"{api.whoami()['name']}/{dataset_name}"
# Create repo if it doesn't exist
try:
create_repo(repo_id, repo_type="dataset", exist_ok=True, token=token)
logger.info(f"HF dataset repo: {repo_id}")
except Exception as e:
logger.warning(f"Repo creation warning (may already exist): {e}")
# Save vectors
vectors_path = os.path.join(DATA_DIR, "vectors.npy")
np.save(vectors_path, vectors)
logger.info(f"Saved vectors: {vectors.shape} ({os.path.getsize(vectors_path) / 1e6:.1f} MB)")
# Save chunks
chunks_path = os.path.join(DATA_DIR, "chunks.json")
with open(chunks_path, "w") as f:
json.dump(chunks, f, indent=2)
logger.info(f"Saved chunks: {len(chunks)} items ({os.path.getsize(chunks_path) / 1e6:.1f} MB)")
# Save timestamp
ts_path = os.path.join(DATA_DIR, "last_updated.txt")
ts = datetime.now(timezone.utc).isoformat()
with open(ts_path, "w") as f:
f.write(ts)
# Upload files
api.upload_file(
repo_id=repo_id,
path_in_repo="vectors.npy",
path_or_fileobj=vectors_path,
repo_type="dataset",
token=token,
)
api.upload_file(
repo_id=repo_id,
path_in_repo="chunks.json",
path_or_fileobj=chunks_path,
repo_type="dataset",
token=token,
)
api.upload_file(
repo_id=repo_id,
path_in_repo="last_updated.txt",
path_or_fileobj=ts_path,
repo_type="dataset",
token=token,
)
logger.info(f"Uploaded to HF Hub: {repo_id}")
return repo_id
# ── Main Pipeline ──────────────────────────────────────────────────────────
def main():
hf_token = os.environ.get("HF_TOKEN")
if not hf_token:
logger.error("HF_TOKEN environment variable required")
logger.error("Usage: HF_TOKEN=hf_yourtoken python3 scripts/build_index.py")
return
total_start = time.time()
# Step 1: Query SPARQL
docs = query_all_documents()
if not docs:
logger.error("No documents found — SPARQL query returned empty")
return
logger.info(f"Documents to process: {len(docs)}")
# Step 2: Download all documents in parallel
logger.info(f"Downloading documents ({DOWNLOAD_WORKERS} workers)...")
html_results = {}
with ThreadPoolExecutor(max_workers=DOWNLOAD_WORKERS) as executor:
future_map = {
executor.submit(fetch_document_xhtml, doc): doc for doc in docs
}
for future in tqdm(as_completed(future_map), total=len(docs), desc="Downloading"):
doc = future_map[future]
try:
html = future.result()
if html:
html_results[doc["celex"]] = html
except Exception as e:
logger.debug(f"Failed {doc['celex']}: {e}")
logger.info(f"Downloaded {len(html_results)}/{len(docs)} documents successfully")
# Step 3: Parse all documents into chunks
logger.info("Parsing HTML into chunks...")
all_chunks = []
for doc in tqdm(docs, desc="Parsing"):
celex = doc["celex"]
html = html_results.get(celex)
if not html:
continue
chunks = parse_html_to_chunks(html, celex, doc["title"])
all_chunks.extend(chunks)
logger.info(f"Total chunks: {len(all_chunks)}")
if not all_chunks:
logger.error("No chunks produced — check parsing")
return
# Step 4: Embed all chunks
vectors = embed_chunks(all_chunks)
logger.info(f"Embedding complete: {vectors.shape}")
# Step 5: Upload to HF Hub
repo_id = upload_to_hub(vectors, all_chunks, HF_DATASET_NAME, hf_token)
total_time = time.time() - total_start
logger.info(f"Build complete in {total_time / 60:.1f} minutes")
logger.info(f"Dataset: {repo_id}")
logger.info(f"Documents: {len(html_results)} | Chunks: {len(all_chunks)} | Dims: {vectors.shape[1]}")
if __name__ == "__main__":
main()
Step 1.2 — Run the build
⏱ Estimated runtime: 3-5 hours. This downloads ~30K-80K documents (depending on date filter) and embeds ~100K-200K chunks. You can leave it running overnight.
source ~/Desktop/EUProjects/.venv/bin/activate
cd ~/Desktop/EUProjects/eur-lex-ai-chat
HF_TOKEN=hf_your_write_token python3 scripts/build_index.py 2>&1 | tee build_log.txt
Expected output milestones:
Querying SPARQL for documents...
Fetched 1000 documents so far...
Fetched 2000 documents so far...
...
Total documents found: XXXX
Downloading documents (20 workers)...
Downloading: 100%|████████████| XXXX/XXXX [XX:XX<00:00]
Downloaded XXXX/XXXX documents successfully
Parsing HTML into chunks...
Total chunks: XXXX
Loading embedding model: all-MiniLM-L6-v2
Embedding XXXX chunks in batches of 128...
Embedding complete: (XXXX, 384)
Saved vectors: (XXXX, 384) (XX.X MB)
Saved chunks: XXXX items (XX.X MB)
Uploaded to HF Hub: yourusername/eurlex-chat-data
Build complete in XXX.X minutes
Step 1.3 — Verify the upload
source ~/Desktop/EUProjects/.venv/bin/activate
python3 -c "
from huggingface_hub import hf_hub_download
import numpy as np, json
# Download vectors
vectors_path = hf_hub_download('yourusername/eurlex-chat-data', 'vectors.npy')
vectors = np.load(vectors_path)
print(f'Vectors: {vectors.shape}, dtype={vectors.dtype}')
# Download chunks
chunks_path = hf_hub_download('yourusername/eurlex-chat-data', 'chunks.json')
with open(chunks_path) as f:
chunks = json.load(f)
print(f'Chunks: {len(chunks)}')
print(f'Sample: {chunks[0][\"text\"][:100]}...')
print(f'Source: CELEX {chunks[0][\"celex\"]} — {chunks[0][\"title\"][:60]}...')
"
If verification passes, Phase 1 is complete. The vector index is live on HuggingFace Hub.
Phase 2: FastAPI Backend
Goal: A FastAPI server that loads the vector index from HF Hub at startup, accepts chat queries, finds relevant chunks via numpy KNN, calls Groq to generate answers with citations, and provides /health and /refresh endpoints.
Step 2.1 — Write backend/requirements.txt
fastapi==0.136.1
uvicorn==0.47.0
numpy==2.4.6
httpx==0.28.1
huggingface_hub==1.15.0
Step 2.2 — Write backend/data_loader.py
"""Download and manage the vector index from HuggingFace Hub."""
import json
import logging
import os
import time
from datetime import datetime, timezone
import numpy as np
from huggingface_hub import hf_hub_download, HfApi
logger = logging.getLogger(__name__)
# HF dataset configuration
HF_USERNAME = os.environ.get("HF_USERNAME", "yourusername")
HF_DATASET = os.environ.get("HF_DATASET", "eurlex-chat-data")
HF_TOKEN = os.environ.get("HF_TOKEN", None)
REPO_ID = f"{HF_USERNAME}/{HF_DATASET}"
# Current index in memory
_index_data = {
"vectors": None,
"chunks": None,
"last_updated": None,
"loaded_at": None,
}
def download_index():
"""Download vectors.npy + chunks.json from HF Hub into memory."""
logger.info(f"Downloading index from {REPO_ID}...")
try:
vectors_path = hf_hub_download(
repo_id=REPO_ID,
filename="vectors.npy",
repo_type="dataset",
token=HF_TOKEN,
)
chunks_path = hf_hub_download(
repo_id=REPO_ID,
filename="chunks.json",
repo_type="dataset",
token=HF_TOKEN,
)
except Exception as e:
logger.error(f"Failed to download from HF Hub: {e}")
raise
vectors = np.load(vectors_path)
with open(chunks_path, "r") as f:
chunks = json.load(f)
_index_data["vectors"] = vectors
_index_data["chunks"] = chunks
_index_data["last_updated"] = _get_last_updated()
_index_data["loaded_at"] = datetime.now(timezone.utc).isoformat()
logger.info(f"Index loaded: {vectors.shape[0]} vectors, {len(chunks)} chunks")
return _index_data
def _get_last_updated():
"""Get the last_updated timestamp from HF Hub."""
try:
ts_path = hf_hub_download(
repo_id=REPO_ID,
filename="last_updated.txt",
repo_type="dataset",
token=HF_TOKEN,
)
with open(ts_path, "r") as f:
return f.read().strip()
except Exception:
return None
def check_for_updates():
"""Check if the index has been updated on HF Hub since we loaded it."""
current_remote = _get_last_updated()
if current_remote and current_remote != _index_data["last_updated"]:
logger.info(f"Remote index updated: {current_remote}")
return True
return False
def reload_index():
"""Re-download and reload the index."""
return download_index()
def get_index():
"""Get the current in-memory index."""
return _index_data
def get_stats():
"""Get index statistics."""
data = get_index()
return {
"vectors": data["vectors"].shape if data["vectors"] is not None else None,
"chunks": len(data["chunks"]) if data["chunks"] is not None else 0,
"last_updated": data["last_updated"],
"loaded_at": data["loaded_at"],
}
Step 2.3 — Write backend/search.py
"""numpy KNN search over pre-loaded vectors."""
import logging
import numpy as np
logger = logging.getLogger(__name__)
def search(query_vector, top_k=10):
"""Find top-k most similar chunks using brute-force cosine similarity.
Uses numpy dot product since vectors are L2-normalized.
Returns list of dicts with chunk data and similarity score.
"""
from data_loader import get_index
index = get_index()
vectors = index["vectors"]
chunks = index["chunks"]
if vectors is None or chunks is None:
logger.error("Index not loaded")
return []
# Cosine similarity = dot product (vectors are normalized)
if query_vector.ndim == 1:
query_vector = query_vector.reshape(1, -1)
similarities = np.dot(vectors, query_vector.T).flatten()
# Get top-k indices
top_indices = np.argpartition(similarities, -top_k)[-top_k:]
top_indices = top_indices[np.argsort(-similarities[top_indices])]
results = []
for idx in top_indices:
results.append({
"score": float(similarities[idx]),
"text": chunks[idx]["text"],
"celex": chunks[idx]["celex"],
"title": chunks[idx]["title"],
"article": chunks[idx].get("article"),
})
return results
Step 2.4 — Write backend/rag.py
"""Build prompts and call Groq API for RAG."""
import json
import logging
import os
import re
import httpx
logger = logging.getLogger(__name__)
GROQ_API_KEY = os.environ.get("GROQ_API_KEY")
GROQ_MODEL = os.environ.get("GROQ_MODEL", "llama-3.3-70b-versatile")
GROQ_API_URL = "https://api.groq.com/openai/v1/chat/completions"
SYSTEM_PROMPT = """You are a legal AI assistant specialized in EU law. You help users understand EU legislation by answering their questions based on provided context from EUR-Lex documents.
Guidelines:
1. Answer based ONLY on the provided context. If the context doesn't contain enough information, say so.
2. Always cite the specific EUR-Lex document(s) you used with their CELEX numbers.
3. When citing articles, include the article number and CELEX number.
4. Keep answers clear and accessible — explain legal concepts in plain language.
5. If the user asks in a non-English language, respond in that language.
6. Do not make up legal citations or references. Only cite what's in the context.
7. Be honest about limitations — if you're unsure, say so."""
def build_prompt(query, context_chunks):
"""Build the prompt with context chunks and user query."""
context_parts = []
for i, chunk in enumerate(context_chunks):
source = f"[{i+1}] CELEX {chunk['celex']}"
if chunk.get("article"):
source += f", Article {chunk['article']}"
context_parts.append(f"Context {i+1} ({source}):\n{chunk['text']}")
context_str = "\n\n---\n\n".join(context_parts)
prompt = f"""Here are relevant excerpts from EU law documents:
{context_str}
Based on the above legal texts, please answer the following question:
{query}"""
return prompt
def call_groq(prompt, max_retries=3):
"""Call Groq API with the prompt and return the response."""
if not GROQ_API_KEY:
logger.error("GROQ_API_KEY not set")
return None
headers = {
"Authorization": f"Bearer {GROQ_API_KEY}",
"Content-Type": "application/json",
}
payload = {
"model": GROQ_MODEL,
"messages": [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": prompt},
],
"temperature": 0.2,
"max_tokens": 2048,
}
for attempt in range(max_retries):
try:
r = httpx.post(
GROQ_API_URL,
headers=headers,
json=payload,
timeout=30,
)
if r.status_code == 429:
logger.warning(f"Groq rate limited (attempt {attempt + 1})")
if attempt < max_retries - 1:
import time
time.sleep(2 ** attempt)
continue
r.raise_for_status()
data = r.json()
return data["choices"][0]["message"]["content"]
except httpx.HTTPStatusError as e:
if e.response.status_code == 429 and attempt < max_retries - 1:
import time
time.sleep(2 ** attempt)
continue
logger.error(f"Groq API error: {e}")
return None
except Exception as e:
logger.error(f"Groq API error: {e}")
return None
return None
def extract_citations(text):
"""Extract CELEX citations from the response text."""
celex_pattern = r"CELEX\s+(\d{2,4}[A-Z0-9]+(?:\([A-Z0-9]+\))?(?:\([0-9]+\))?)"
return re.findall(celex_pattern, text)
def answer_question(query, context_chunks):
"""Full RAG pipeline: build prompt → call Groq → return answer with citations."""
prompt = build_prompt(query, context_chunks)
answer = call_groq(prompt)
if not answer:
return {
"answer": "Sorry, I couldn't generate an answer right now. Please try again.",
"citations": [],
}
citations = extract_citations(answer)
# Also add sources from our context
source_citations = []
seen = set()
for chunk in context_chunks:
if chunk["celex"] not in seen:
seen.add(chunk["celex"])
source_citations.append({
"celex": chunk["celex"],
"title": chunk["title"],
"article": chunk.get("article"),
"score": chunk["score"],
})
return {
"answer": answer,
"citations": citations,
"sources": source_citations,
}
Step 2.5 — Write backend/rate_limit.py
"""Per-IP rate limiting for the /chat endpoint."""
import time
from collections import defaultdict
# Rate limit configuration
MAX_REQUESTS_PER_IP = 20 # Max requests per window
WINDOW_SECONDS = 60 # Window size in seconds
MAX_GLOBAL_PER_MINUTE = 100 # Global max
_ip_counters = defaultdict(list)
_global_counters = []
def is_rate_limited(client_ip):
"""Check if a client IP has exceeded the rate limit.
Returns True if rate limited, False if request is allowed.
"""
now = time.time()
window_start = now - WINDOW_SECONDS
# Clean old entries
_ip_counters[client_ip] = [
t for t in _ip_counters[client_ip] if t > window_start
]
# Check IP limit
if len(_ip_counters[client_ip]) >= MAX_REQUESTS_PER_IP:
return True
# Check global limit
global _global_counters
_global_counters = [t for t in _global_counters if t > window_start]
if len(_global_counters) >= MAX_GLOBAL_PER_MINUTE:
return True
# Record this request
_ip_counters[client_ip].append(now)
_global_counters.append(now)
return False
Step 2.6 — Write backend/main.py
"""FastAPI application for EUR-Lex AI Chat."""
import logging
import os
from contextlib import asynccontextmanager
from fastapi import FastAPI, HTTPException, Request
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import JSONResponse
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
)
logger = logging.getLogger(__name__)
# Global model instance — loaded once at startup, reused for all requests
_embedding_model = None
def get_embedding_model():
"""Get the singleton embedding model instance."""
global _embedding_model
if _embedding_model is None:
from sentence_transformers import SentenceTransformer
logger.info("Loading embedding model: all-MiniLM-L6-v2")
_embedding_model = SentenceTransformer("all-MiniLM-L6-v2")
return _embedding_model
@asynccontextmanager
async def lifespan(app: FastAPI):
"""Load the vector index and embedding model on startup."""
from data_loader import download_index
logger.info("Starting up — loading index...")
try:
download_index()
logger.info("Index loaded successfully")
except Exception as e:
logger.error(f"Failed to load index: {e}")
# Pre-load embedding model
logger.info("Pre-loading embedding model...")
get_embedding_model()
logger.info("Embedding model loaded")
yield
logger.info("Shutting down")
app = FastAPI(title="EUR-Lex AI Chat API", version="1.0.0", lifespan=lifespan)
# CORS — allow frontend on Vercel
app.add_middleware(
CORSMiddleware,
allow_origins=[
"https://eurlex-chat.vercel.app",
"http://localhost:4321",
"http://localhost:3000",
],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
@app.get("/health")
async def health():
"""Health check endpoint — used by cron-job.org keepalive."""
from data_loader import get_stats
stats = get_stats()
return {
"status": "ok",
"index_loaded": stats["vectors"] is not None,
"vector_count": stats["vectors"][0] if stats["vectors"] else 0,
"chunk_count": stats["chunks"],
"last_updated": stats["last_updated"],
"loaded_at": stats["loaded_at"],
}
@app.get("/refresh")
async def refresh():
"""Check for index updates on HF Hub and reload if newer."""
from data_loader import check_for_updates, reload_index, get_stats
try:
has_updates = check_for_updates()
if has_updates:
logger.info("New index available, reloading...")
reload_index()
return {"status": "reloaded", "message": "Index updated successfully"}
else:
stats = get_stats()
return {"status": "current", "message": "Index is up to date"}
except Exception as e:
logger.error(f"Refresh failed: {e}")
return JSONResponse(
status_code=500,
content={"status": "error", "message": str(e)},
)
@app.post("/chat")
async def chat(request: Request):
"""Main chat endpoint. Accepts {query: string}, returns {answer, sources}."""
from rate_limit import is_rate_limited
from search import search
from rag import answer_question
# Get client IP
client_ip = request.client.host if request.client else "unknown"
# Rate limit check
if is_rate_limited(client_ip):
raise HTTPException(
status_code=429,
detail="Rate limit exceeded. Max 20 requests per minute per IP.",
)
# Parse request
body = await request.json()
query = body.get("query", "").strip()
if not query:
raise HTTPException(status_code=400, detail="Query is required")
if len(query) > 2000:
raise HTTPException(status_code=400, detail="Query too long (max 2000 chars)")
# Embed query using the singleton model
model = get_embedding_model()
query_vector = model.encode(query, normalize_embeddings=True)
# Search
chunks = search(query_vector, top_k=10)
if not chunks:
return {
"answer": "I don't have enough information to answer that question. Try asking about a specific EU regulation or directive.",
"citations": [],
"sources": [],
}
# Generate answer via RAG
result = answer_question(query, chunks)
return result
if __name__ == "__main__":
import uvicorn
port = int(os.environ.get("PORT", 8000))
uvicorn.run("main:app", host="0.0.0.0", port=port, reload=False)
Step 2.7 — Write backend/startup.sh
#!/bin/bash
# Render entry point — downloads index and starts uvicorn
set -e
cd "$(dirname "$0")"
echo "=== EUR-Lex AI Chat Backend Startup ==="
echo "Python: $(python3 --version)"
# The FastAPI app loads the index at startup via lifespan
echo "Starting uvicorn..."
exec uvicorn main:app --host 0.0.0.0 --port ${PORT:-8000} --workers 1
Make it executable:
chmod +x ~/Desktop/EUProjects/eur-lex-ai-chat/backend/startup.sh
Step 2.8 — Test the backend locally
source ~/Desktop/EUProjects/.venv/bin/activate
cd ~/Desktop/EUProjects/eur-lex-ai-chat/backend
# Start server in background
uvicorn main:app --host 0.0.0.0 --port 8000 &
sleep 5
# Test health endpoint
curl -s http://localhost:8000/health | python3 -m json.tool
# Test chat endpoint
curl -s -X POST http://localhost:8000/chat \
-H "Content-Type: application/json" \
-d '{"query": "What are the requirements of the GDPR?"}' \
| python3 -m json.tool
# Kill the server
kill %1 2>/dev/null
Expected /health response:
{
"status": "ok",
"index_loaded": true,
"vector_count": 123456,
"chunk_count": 123456,
"last_updated": "2026-05-20T12:00:00+00:00",
"loaded_at": "2026-05-20T12:00:00+00:00"
}
Step 2.9 — Create backend/render.yaml for Render deployment
# render.yaml — Render Blueprint for EUR-Lex AI Chat backend
services:
- type: web
name: eurlex-chat-api
runtime: python
repo: https://github.com/yourusername/eur-lex-ai-chat
branch: main
buildCommand: pip install -r backend/requirements.txt
startCommand: cd backend && ./startup.sh
envVars:
- key: GROQ_API_KEY
sync: false
- key: HF_TOKEN
sync: false
- key: HF_USERNAME
value: yourusername
- key: HF_DATASET
value: eurlex-chat-data
- key: PYTHON_VERSION
value: 3.12.3
healthCheckPath: /health
We'll set the environment variables in the Render dashboard (not in the yaml for security).
Phase 3: Astro Frontend
Goal: A fully SEO-optimized Astro website with a React chat island. Ships zero-JS HTML by default. Chat widget is only interactive JS on the page.
Step 3.1 — Scaffold Astro project
source ~/.nvm/nvm.sh
nvm use v22.22.3
cd ~/Desktop/EUProjects/eur-lex-ai-chat
# Create Astro project in frontend/ directory
npm create astro@latest frontend -- --template basics --typescript --no-install --no-git
cd frontend
# Install dependencies
npm install astro @astrojs/react @astrojs/tailwind @astrojs/sitemap tailwindcss react react-dom
# Add integrations
npx astro add react --yes
npx astro add tailwind --yes
npx astro add sitemap --yes
Step 3.2 — Write frontend/astro.config.mjs
import { defineConfig } from "astro/config";
import react from "@astrojs/react";
import tailwind from "@astrojs/tailwind";
import sitemap from "@astrojs/sitemap";
export default defineConfig({
site: "https://eurlex-chat.vercel.app",
integrations: [
react(),
tailwind(),
sitemap({
changefreq: "weekly",
priority: 0.7,
lastmod: new Date(),
}),
],
});
Step 3.3 — Write frontend/tailwind.config.mjs
/** @type {import('tailwindcss').Config} */
export default {
content: ["./src/**/*.{astro,html,js,jsx,mdx,tsx}"],
theme: {
extend: {
colors: {
eu: {
blue: "#003399",
gold: "#FFCC00",
navy: "#002266",
light: "#F0F4FF",
},
},
},
},
plugins: [],
};
Step 3.4 — Write frontend/src/layouts/Base.astro
---
// Base layout — used by all pages
// Includes: SEO meta tags, JSON-LD, nav, footer
export interface Props {
title: string;
description: string;
canonical?: string;
ogType?: string;
jsonLd?: Record<string, any>;
}
const {
title,
description,
canonical = "https://eurlex-chat.vercel.app",
ogType = "website",
jsonLd,
} = Astro.props;
---
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>{title} — EUR-Lex AI Chat</title>
<meta name="description" content={description} />
<link rel="canonical" href={canonical} />
<!-- Open Graph -->
<meta property="og:title" content={title} />
<meta property="og:description" content={description} />
<meta property="og:url" content={canonical} />
<meta property="og:type" content={ogType} />
<meta property="og:site_name" content="EUR-Lex AI Chat" />
<meta property="og:locale" content="en_US" />
<!-- Twitter Card -->
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:title" content={title} />
<meta name="twitter:description" content={description} />
{/* JSON-LD */}
{jsonLd && <script type="application/ld+json" set:html={JSON.stringify(jsonLd)} />}
{/* Google Search Console verification */}
<meta name="google-site-verification" content="YOUR_VERIFICATION_CODE" />
</head>
<body class="bg-white text-gray-900 min-h-screen flex flex-col">
<!-- Navigation -->
<nav class="bg-eu-blue text-white shadow-lg">
<div class="max-w-6xl mx-auto px-4 py-3 flex items-center justify-between">
<a href="/" class="text-xl font-bold tracking-tight">
EUR-Lex AI Chat
</a>
<div class="flex gap-4 text-sm">
<a href="/" class="hover:text-eu-gold transition">Home</a>
<a href="/faq" class="hover:text-eu-gold transition">FAQ</a>
<a href="/blog" class="hover:text-eu-gold transition">Blog</a>
</div>
</div>
</nav>
<!-- Main content -->
<main class="flex-1">
<slot />
</main>
<!-- Footer -->
<footer class="bg-gray-50 border-t py-8 mt-12">
<div class="max-w-6xl mx-auto px-4 text-center text-sm text-gray-600">
<p class="mb-2">
EUR-Lex AI Chat — An open-source tool for exploring EU law.
Uses data from
<a href="https://eur-lex.europa.eu/" class="text-eu-blue underline" target="_blank" rel="noopener">EUR-Lex</a>
via the Cellar API.
</p>
<p>
<a href="https://github.com/yourusername/eur-lex-ai-chat" class="text-eu-blue underline" target="_blank" rel="noopener">Source code (MIT)</a>
·
<a href="/faq" class="text-eu-blue underline">FAQ</a>
·
<a href="/blog" class="text-eu-blue underline">Blog</a>
</p>
</div>
</footer>
</body>
</html>
Step 3.5 — Write frontend/src/components/SeoHead.astro
---
// Helper component for page-specific SEO metadata
export interface Props {
pageTitle: string;
pageDescription: string;
pageUrl?: string;
jsonLd?: Record<string, any>;
}
const {
pageTitle,
pageDescription,
pageUrl = "https://eurlex-chat.vercel.app",
jsonLd,
} = Astro.props;
---
<!-- Page title suffix -->
{pageTitle && <title>{pageTitle} — EUR-Lex AI Chat</title>}
{pageDescription && <meta name="description" content={pageDescription} />}
<link rel="canonical" href={pageUrl} />
<meta property="og:title" content={pageTitle} />
<meta property="og:description" content={pageDescription} />
<meta property="og:url" content={pageUrl} />
{jsonLd && <script type="application/ld+json" set:html={JSON.stringify(jsonLd)} />}
Step 3.6 — Write frontend/src/components/ChatWidget.jsx
import { useState, useRef, useEffect } from "react";
const API_URL = import.meta.env.PUBLIC_API_URL || "http://localhost:8000";
export default function ChatWidget() {
const [messages, setMessages] = useState([
{
role: "assistant",
content:
"Hi! I'm an AI assistant specialized in EU law. Ask me anything about EU regulations, directives, or legislation.",
},
]);
const [input, setInput] = useState("");
const [loading, setLoading] = useState(false);
const [error, setError] = useState(null);
const messagesEndRef = useRef(null);
useEffect(() => {
messagesEndRef.current?.scrollIntoView({ behavior: "smooth" });
}, [messages]);
const sendMessage = async () => {
const query = input.trim();
if (!query || loading) return;
setInput("");
setError(null);
setMessages((prev) => [...prev, { role: "user", content: query }]);
setLoading(true);
try {
const res = await fetch(`${API_URL}/chat`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ query }),
});
if (!res.ok) {
const data = await res.json();
throw new Error(data.detail || `Error ${res.status}`);
}
const data = await res.json();
// Format answer with sources
let answer = data.answer;
if (data.sources && data.sources.length > 0) {
answer += "\n\n**Sources:**";
for (const s of data.sources) {
answer += `\n- CELEX ${s.celex}: ${s.title}`;
if (s.article) answer += ` (Article ${s.article})`;
}
}
setMessages((prev) => [
...prev,
{ role: "assistant", content: answer },
]);
} catch (err) {
setError(err.message);
setMessages((prev) => [
...prev,
{
role: "assistant",
content: `Sorry, I encountered an error: ${err.message}. Please try again.`,
},
]);
} finally {
setLoading(false);
}
};
const handleKeyDown = (e) => {
if (e.key === "Enter" && !e.shiftKey) {
e.preventDefault();
sendMessage();
}
};
return (
<div className="bg-white rounded-xl shadow-lg border border-gray-200 overflow-hidden max-w-2xl mx-auto">
{/* Chat header */}
<div className="bg-eu-blue text-white px-4 py-3 font-semibold">
Ask about EU Law
</div>
{/* Messages */}
<div className="h-96 overflow-y-auto p-4 space-y-3 bg-gray-50">
{messages.map((msg, i) => (
<div
key={i}
className={`flex ${msg.role === "user" ? "justify-end" : "justify-start"}`}
>
<div
className={`max-w-[80%] rounded-lg px-4 py-2 text-sm whitespace-pre-wrap ${
msg.role === "user"
? "bg-eu-blue text-white"
: "bg-white border border-gray-200 text-gray-800"
}`}
>
{msg.content}
</div>
</div>
))}
{loading && (
<div className="flex justify-start">
<div className="bg-white border border-gray-200 rounded-lg px-4 py-2 text-sm text-gray-500 italic">
Thinking...
</div>
</div>
)}
<div ref={messagesEndRef} />
</div>
{/* Input */}
<div className="border-t border-gray-200 p-3 flex gap-2">
<textarea
value={input}
onChange={(e) => setInput(e.target.value)}
onKeyDown={handleKeyDown}
placeholder="Ask about EU law..."
rows={1}
className="flex-1 border border-gray-300 rounded-lg px-3 py-2 text-sm focus:outline-none focus:ring-2 focus:ring-eu-blue resize-none"
disabled={loading}
/>
<button
onClick={sendMessage}
disabled={loading || !input.trim()}
className="bg-eu-blue text-white px-4 py-2 rounded-lg text-sm font-medium hover:bg-eu-navy transition disabled:opacity-50 disabled:cursor-not-allowed"
>
Send
</button>
</div>
{error && (
<div className="bg-red-50 text-red-700 text-xs px-4 py-2 border-t border-red-200">
{error}
</div>
)}
</div>
);
}
Step 3.7 — Write frontend/src/pages/index.astro
---
import Base from "../layouts/Base.astro";
import ChatWidget from "../components/ChatWidget";
const jsonLd = {
"@context": "https://schema.org",
"@type": "SoftwareApplication",
name: "EUR-Lex AI Chat",
applicationCategory: "WebApplication",
operatingSystem: "Web",
description:
"Chat about EU law in plain English. Get answers with citations to real EUR-Lex documents.",
url: "https://eurlex-chat.vercel.app",
offers: {
"@type": "Offer",
price: "0",
priceCurrency: "EUR",
},
};
---
<Base
title="EUR-Lex AI Chat — Ask EU Law Questions in Plain English"
description="Free AI-powered chat over EU legislation. Ask questions about GDPR, AI Act, Digital Markets Act, and thousands of other EU regulations and directives. Get answers with citations."
jsonLd={jsonLd}
>
<!-- Hero section -->
<section class="bg-gradient-to-br from-eu-blue to-eu-navy text-white">
<div class="max-w-4xl mx-auto px-4 py-16 text-center">
<h1 class="text-3xl md:text-4xl font-bold mb-4">
Ask EU Law Questions in Plain English
</h1>
<p class="text-lg text-blue-200 max-w-2xl mx-auto mb-8">
EUR-Lex AI Chat helps you understand EU regulations, directives, and legislation.
Ask any question and get answers with citations to real EUR-Lex documents.
</p>
<div class="flex justify-center gap-3 flex-wrap">
<span class="bg-white/10 rounded-full px-4 py-1.5 text-sm">Free</span>
<span class="bg-white/10 rounded-full px-4 py-1.5 text-sm">No account needed</span>
<span class="bg-white/10 rounded-full px-4 py-1.5 text-sm">20 queries/day</span>
<span class="bg-white/10 rounded-full px-4 py-1.5 text-sm">Open source</span>
</div>
</div>
</section>
<!-- Chat section -->
<section class="max-w-4xl mx-auto px-4 py-12">
<h2 class="text-2xl font-bold text-center mb-8">Try It Now</h2>
<ChatWidget client:load />
</section>
<!-- Features section -->
<section class="bg-gray-50 py-12">
<div class="max-w-4xl mx-auto px-4">
<h2 class="text-2xl font-bold text-center mb-8">How It Works</h2>
<div class="grid md:grid-cols-3 gap-6">
<div class="bg-white rounded-xl p-6 shadow-sm border border-gray-100">
<div class="text-eu-blue text-2xl font-bold mb-2">1</div>
<h3 class="font-semibold mb-2">Ask a Question</h3>
<p class="text-sm text-gray-600">
Type any question about EU law in plain language. No legal knowledge needed.
</p>
</div>
<div class="bg-white rounded-xl p-6 shadow-sm border border-gray-100">
<div class="text-eu-blue text-2xl font-bold mb-2">2</div>
<h3 class="font-semibold mb-2">AI Searches EU Law</h3>
<p class="text-sm text-gray-600">
We search over 100,000 passages from EUR-Lex documents to find relevant legal texts.
</p>
</div>
<div class="bg-white rounded-xl p-6 shadow-sm border border-gray-100">
<div class="text-eu-blue text-2xl font-bold mb-2">3</div>
<h3 class="font-semibold mb-2">Get Cited Answers</h3>
<p class="text-sm text-gray-600">
Receive clear answers with CELEX numbers and article references so you can verify the source.
</p>
</div>
</div>
</div>
</section>
<!-- Example questions section -->
<section class="py-12">
<div class="max-w-4xl mx-auto px-4">
<h2 class="text-2xl font-bold text-center mb-8">Example Questions You Can Ask</h2>
<div class="grid md:grid-cols-2 gap-4">
<div class="bg-white border border-gray-200 rounded-lg p-4 text-sm text-gray-700">
"What are the data protection requirements under the GDPR?"
</div>
<div class="bg-white border border-gray-200 rounded-lg p-4 text-sm text-gray-700">
"When does the EU AI Act come into force?"
</div>
<div class="bg-white border border-gray-200 rounded-lg p-4 text-sm text-gray-700">
"What are the obligations for gatekeepers under the DMA?"
</div>
<div class="bg-white border border-gray-200 rounded-lg p-4 text-sm text-gray-700">
"What is a CELEX number and how do I use it?"
</div>
</div>
</div>
</section>
</Base>
Step 3.8 — Write frontend/src/pages/faq.astro
---
import Base from "../layouts/Base.astro";
const faqQuestions = [
{
question: "What is EUR-Lex?",
answer:
"EUR-Lex is the official online portal for European Union law. It provides access to the Official Journal of the EU, treaties, legislation, case law, and other legal documents. EUR-Lex AI Chat uses data from EUR-Lex via the Cellar API.",
},
{
question: "What is a CELEX number?",
answer:
"A CELEX number is a unique identifier assigned to every document in EUR-Lex. It follows a specific format: for example, 32019R0947 breaks down as 3 (sector: legislation), 2019 (year), R (regulation), 0947 (document number). CELEX numbers are used to cite and locate EU legal documents precisely.",
},
{
question: "How accurate is the AI?",
answer:
"The AI searches over actual EUR-Lex documents and cites its sources with CELEX numbers. However, you should always verify important legal information by checking the official documents on EUR-Lex. The AI can make mistakes and should not be used as a substitute for professional legal advice.",
},
{
question: "Is this service free?",
answer:
"Yes, completely free. You can make up to 20 queries per day per IP address. There are no accounts, no subscriptions, and no hidden costs. This is an open-source project built with free cloud services.",
},
{
question: "What documents are included?",
answer:
"The index includes EU regulations, directives, and implementing acts published since 2000, available in English. This covers over 25 years of EU legislation, including major laws like the GDPR, AI Act, Digital Markets Act, and thousands more.",
},
{
question: "How often is the data updated?",
answer:
"The index is refreshed daily via automated GitHub Actions workflows. New regulations and amendments are picked up within 24 hours of publication in the Official Journal.",
},
{
question: "What is the Cellar API?",
answer:
"The Cellar API is the official Semantic Web interface for EUR-Lex. It provides structured access to EU legal documents via SPARQL queries and REST endpoints. This project uses Cellar APIs exclusively, not web scraping.",
},
{
question: "What does a CELEX number look like?",
answer:
"CELEX numbers have a standard format: 32019R0947 (a regulation from 2019), 32016R0679 (the GDPR), or 32024R1689 (the AI Act). The first digit indicates the sector, the next four digits are the year, followed by a document type code and number.",
},
{
question: "Can I cite answers from this tool in legal documents?",
answer:
"No. This tool is for informational purposes only. Always verify legal information against the official text published in the Official Journal of the EU (OJEU). The AI-generated answers should not be used as legal authority.",
},
{
question: "What is the difference between a regulation and a directive?",
answer:
"A regulation is a binding legislative act that must be applied in its entirety across all EU member states. A directive is a legislative act that sets a goal that all EU countries must achieve, but it's up to the individual countries to devise their own laws to reach that goal.",
},
{
question: "What is the GDPR?",
answer:
"The General Data Protection Regulation (GDPR, Regulation 2016/679) is the EU's primary data protection law. It governs how personal data must be collected, processed, stored, and protected by organizations operating in the EU.",
},
{
question: "What is the EU AI Act?",
answer:
"The EU AI Act (Regulation 2024/1689) is the world's first comprehensive legal framework for artificial intelligence. It categorizes AI systems by risk level and imposes requirements ranging from transparency obligations for minimal-risk AI to bans on unacceptable-risk AI.",
},
{
question: "What is the Digital Markets Act (DMA)?",
answer:
"The Digital Markets Act (Regulation 2022/1925) is an EU law that aims to make digital markets fairer and more contestable. It sets obligations for large online platforms designated as 'gatekeepers' to prevent anti-competitive practices.",
},
{
question: "What is the Digital Services Act (DSA)?",
answer:
"The Digital Services Act (Regulation 2022/2065) is an EU law that regulates online intermediary platforms. It sets rules for content moderation, transparency, and accountability of platforms from the largest social networks to small online marketplaces.",
},
{
question: "Where can I find the official text of an EU regulation?",
answer:
"The official text is published in the Official Journal of the EU (OJEU) and is available on EUR-Lex at eur-lex.europa.eu. You can search by CELEX number, title, or date. The OJ L series contains legislation, while the OJ C series contains information and notices.",
},
{
question: "Can I search for documents from a specific year?",
answer:
"Yes. The index covers documents from 2000 onwards. You can ask questions like 'What regulations were passed in 2023 regarding digital platforms?' and the AI will search the relevant documents.",
},
{
question: "How do I find the CELEX number for a specific law?",
answer:
"You can search directly on this chat by asking 'What is the CELEX number for the GDPR?' or 'Find CELEX for the AI Act.' The AI will return the CELEX number along with the title and other metadata.",
},
{
question: "What languages are supported?",
answer:
"The index currently contains documents in English. However, you can ask questions in any language, and the AI will respond in the language you used. The underlying legal texts are in English.",
},
{
question: "Can I download or export the results?",
answer:
"Currently, the chat provides answers with citations on screen. You can copy the text directly. We're working on adding export functionality in future updates.",
},
{
question: "Is this project open source?",
answer:
"Yes! The full source code is available on GitHub under the MIT license. You can view, fork, and contribute to the project. The data pipeline, AI backend, and frontend are all open source.",
},
{
question: "What happens if I reach the daily limit?",
answer:
"After 20 queries in a minute, you'll see a rate limit message. Wait a minute and try again. This limit helps keep the service free and available for everyone.",
},
];
const faqJsonLd = {
"@context": "https://schema.org",
"@type": "FAQPage",
mainEntity: faqQuestions.map((q) => ({
"@type": "Question",
name: q.question,
acceptedAnswer: {
"@type": "Answer",
text: q.answer,
},
})),
};
---
<Base
title="Frequently Asked Questions"
description="Answers to common questions about EUR-Lex AI Chat, CELEX numbers, EU regulations, directives, GDPR, AI Act, DMA, DSA, and how to use the service."
canonical="https://eurlex-chat.vercel.app/faq"
jsonLd={faqJsonLd}
>
<div class="max-w-3xl mx-auto px-4 py-12">
<h1 class="text-3xl font-bold mb-2">Frequently Asked Questions</h1>
<p class="text-gray-600 mb-8">
Everything you need to know about EUR-Lex AI Chat and EU law.
</p>
<div class="space-y-4">
{faqQuestions.map((item) => (
<details class="bg-white border border-gray-200 rounded-lg overflow-hidden">
<summary class="px-4 py-3 cursor-pointer font-medium hover:bg-gray-50 transition">
{item.question}
</summary>
<div class="px-4 py-3 text-sm text-gray-700 border-t border-gray-100">
{item.answer}
</div>
</details>
))}
</div>
</div>
</Base>
Step 3.9 — Write blog listing page frontend/src/pages/blog/index.astro
---
import Base from "../../layouts/Base.astro";
const posts = await Astro.glob("./posts/*.md");
// Sort by date descending
posts.sort((a, b) => new Date(b.frontmatter.date) - new Date(a.frontmatter.date));
---
<Base
title="Blog"
description="Articles about EU law, including guides to GDPR, AI Act, CELEX numbers, and other European legislation topics."
canonical="https://eurlex-chat.vercel.app/blog"
>
<div class="max-w-3xl mx-auto px-4 py-12">
<h1 class="text-3xl font-bold mb-2">Blog</h1>
<p class="text-gray-600 mb-8">Articles about EU law and how to use EUR-Lex AI Chat.</p>
<div class="space-y-6">
{posts.map((post) => (
<article class="bg-white border border-gray-200 rounded-lg p-6 hover:shadow-md transition">
<time class="text-xs text-gray-500">{post.frontmatter.date}</time>
<h2 class="text-xl font-semibold mt-1 mb-2">
<a href={post.url} class="text-eu-blue hover:underline">{post.frontmatter.title}</a>
</h2>
<p class="text-sm text-gray-600">{post.frontmatter.description}</p>
</article>
))}
</div>
</div>
</Base>
Step 3.10 — Create blog post directory and write first post
mkdir -p ~/Desktop/EUProjects/eur-lex-ai-chat/frontend/src/pages/blog/posts
Write blog posts in Markdown. Save each as .md in frontend/src/pages/blog/posts/. Each needs frontmatter with title, description, date, and optionally tags.
Step 3.11 — Test the frontend locally
source ~/.nvm/nvm.sh
nvm use v22.22.3
cd ~/Desktop/EUProjects/eur-lex-ai-chat/frontend
npm run dev
Open http://localhost:4321 in a browser. Verify:
- Landing page renders with chat widget
- FAQ page loads with all 20 questions expandable
- Blog index lists posts
- Test the chat widget (it will try to connect to local backend)
Step 3.12 — Create vercel.json (optional)
For custom Vercel config:
{
"buildCommand": "npm run build",
"outputDirectory": "dist",
"framework": "astro"
}
Phase 4: SEO Content
Goal: Write the FAQ content (already done in Phase 3 — 20 questions) and 3 blog posts. This is pure content writing with Markdown.
Step 4.1 — Blog Post 1: eu-ai-act-explained.md
---
title: "EU AI Act Explained: What You Need to Know"
description: "A comprehensive guide to the EU AI Act (Regulation 2024/1689) — the world's first legal framework for artificial intelligence. Learn about risk categories, requirements, and timelines."
date: "2026-05-20"
tags: ["AI Act", "artificial intelligence", "regulation"]
---
The EU AI Act (Regulation 2024/1689) entered into force on August 1, 2024, marking a historic moment in technology regulation. It is the world's first comprehensive legal framework for artificial intelligence.
## What Is the AI Act?
The AI Act establishes a risk-based classification system for AI systems:
- **Unacceptable risk** — AI systems that pose a clear threat to safety, livelihoods, or rights (banned)
- **High risk** — AI systems in critical areas like healthcare, law enforcement, employment (strict requirements)
- **Limited risk** — AI systems with specific transparency obligations (e.g., chatbots must disclose they are AI)
- **Minimal risk** — All other AI systems (no additional obligations)
## Key Requirements
High-risk AI systems must comply with:
1. **Risk management system** (Article 9)
2. **Data governance** — training data must be relevant, representative, and free from errors (Article 10)
3. **Technical documentation** — detailed description of the system's development and capabilities (Article 11)
4. **Record-keeping** — automatic logging of system events (Article 12)
5. **Transparency** — users must be informed they are interacting with AI (Article 13)
6. **Human oversight** — humans must be able to override or stop the system (Article 14)
7. **Accuracy, robustness, and cybersecurity** (Article 15)
## Timeline
- **August 2024**: AI Act enters into force
- **February 2025**: Prohibitions on unacceptable risk AI apply
- **August 2025**: Rules for general-purpose AI models apply
- **August 2026**: Most rules apply (including high-risk AI systems)
- **August 2027**: Rules for high-risk AI systems in certain product categories apply
## CELEX Number
The AI Act has CELEX number **32024R1689**. You can find the full text on EUR-Lex at this CELEX identifier.
## How EUR-Lex AI Chat Can Help
Use EUR-Lex AI Chat to ask specific questions about the AI Act, such as:
- "What are the transparency obligations for chatbots under the AI Act?"
- "Which AI systems are banned under the AI Act?"
- "What are the penalties for non-compliance with the AI Act?"
Step 4.2 — Blog Post 2: gdpr-guide.md
---
title: "GDPR Explained: A Practical Guide to EU Data Protection Law"
description: "Everything you need to know about the General Data Protection Regulation (GDPR, Regulation 2016/679). Covers data subject rights, obligations for controllers and processors, and key requirements."
date: "2026-05-20"
tags: ["GDPR", "data protection", "privacy", "regulation"]
---
The General Data Protection Regulation (GDPR, Regulation 2016/679) is the European Union's flagship data protection law. Since its application on May 25, 2018, it has become the global benchmark for privacy regulation.
## Scope
The GDPR applies to:
- Organizations established in the EU that process personal data
- Organizations outside the EU that offer goods or services to individuals in the EU
- Organizations outside the EU that monitor the behavior of individuals in the EU
## Key Principles (Article 5)
Personal data must be:
1. **Lawfulness, fairness, and transparency** — processed lawfully, fairly, and transparently
2. **Purpose limitation** — collected for specified, explicit, and legitimate purposes
3. **Data minimization** — adequate, relevant, and limited to what is necessary
4. **Accuracy** — accurate and kept up to date
5. **Storage limitation** — kept no longer than necessary
6. **Integrity and confidentiality** — processed securely
7. **Accountability** — the controller is responsible for compliance
## Data Subject Rights (Chapter III)
Individuals have the right to:
- **Access** their personal data (Article 15)
- **Rectification** of inaccurate data (Article 16)
- **Erasure** ('right to be forgotten') (Article 17)
- **Restriction of processing** (Article 18)
- **Data portability** (Article 20)
- **Object** to processing (Article 21)
## CELEX Number
The GDPR has CELEX number **32016R0679**. You can find the full text on EUR-Lex at this CELEX identifier.
## Use EUR-Lex AI Chat
Ask questions like:
- "What are the requirements for data protection impact assessments under the GDPR?"
- "When can personal data be transferred outside the EU?"
- "What are the penalties for GDPR violations?"
Step 4.3 — Blog Post 3: what-is-celex.md
---
title: "What Is a CELEX Number? A Complete Guide to EUR-Lex Document Identifiers"
description: "Learn what CELEX numbers are, how to read them, and how to use them to find EU legal documents on EUR-Lex. Includes examples for GDPR, AI Act, and more."
date: "2026-05-20"
tags: ["CELEX", "EUR-Lex", "document identifiers", "guide"]
---
If you've ever tried to cite an EU legal document precisely, you've probably encountered CELEX numbers. This guide explains what they are, how they work, and how to use them.
## What Is a CELEX Number?
A CELEX number is a unique identifier assigned to every document in EUR-Lex, the official online portal for EU law. Think of it as a digital fingerprint for EU legal documents — no two documents share the same CELEX number.
## How to Read a CELEX Number
A CELEX number follows a specific format. Let's break down the GDPR's CELEX number: **32016R0679**
| Digit(s) | Meaning | Example |
|----------|---------|---------|
| 1st digit | Sector | 3 = Legislation |
| 2nd-5th digits | Year | 2016 |
| Letter(s) | Document type | R = Regulation |
| Remaining digits | Document number | 0679 |
### Sector Codes
- **1**: Treaties
- **2**: International agreements
- **3**: Legislation (most common for our purposes)
- **4**: Complementary legislation
- **5**: Preparatory acts
- **6**: Case law
- **7**: National transposition measures
- **8**: References
- **9**: Parliamentary questions
- **0**: Consolidated texts
### Document Type Letters
- **R**: Regulation
- **L**: Directive
- **D**: Decision
- **H**: Recommendation
- **A**: Opinion
- **C**: Cost of the Treaties
## Examples of Common CELEX Numbers
| Law | CELEX Number |
|-----|-------------|
| GDPR | 32016R0679 |
| EU AI Act | 32024R1689 |
| Digital Markets Act | 32022R1925 |
| Digital Services Act | 32022R2065 |
| Uber ruling (ECJ) | 62017CJ0434 |
## How to Use CELEX Numbers
1. **Search**: Go to eur-lex.europa.eu and paste a CELEX number into the search box
2. **Cite**: Use CELEX numbers in legal citations for precision
3. **API**: Developers can use CELEX numbers to retrieve documents programmatically via the Cellar API
## Use EUR-Lex AI Chat
Ask questions like:
- "What is the CELEX number for the GDPR?"
- "Find the CELEX number for the Digital Markets Act"
- "How do I search for documents by CELEX number?"
Step 4.4 — Create frontend/public/robots.txt
User-agent: *
Allow: /
Sitemap: https://eurlex-chat.vercel.app/sitemap-index.xml
Phase 5: GitHub Actions Automation
Goal: A daily GitHub Actions workflow that scrapes EUR-Lex for new/changed documents, processes them incrementally, and uploads the updated index to HuggingFace Hub.
Step 5.1 — Write scripts/update_index.py
This incremental update script:
- Gets the current last_updated timestamp from HF Hub
- Queries SPARQL for documents modified since that timestamp
- Downloads and processes only new/changed documents
- Merges with existing index
- Re-embeds (only new vectors)
- Uploads updated index
#!/usr/bin/env python3
"""
update_index.py — Incremental daily update of the EUR-Lex vector index.
Runs in GitHub Actions on a schedule.
Pipeline:
1. Download current index from HF Hub
2. Query SPARQL for documents modified since last update
3. Only process new/changed documents
4. Merge with existing index (replace changed, append new)
5. Upload updated index to HF Hub
Usage:
HF_TOKEN=hf_yourtoken python3 scripts/update_index.py
"""
import json
import logging
import os
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
import numpy as np
import requests
from huggingface_hub import hf_hub_download, HfApi, upload_file
from tqdm import tqdm
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)
SPARQL_ENDPOINT = "https://publications.europa.eu/webapi/rdf/sparql"
DOWNLOAD_WORKERS = 10
EMBEDDING_MODEL = "all-MiniLM-L6-v2"
HF_USERNAME = os.environ.get("HF_USERNAME", "yourusername")
HF_DATASET = os.environ.get("HF_DATASET", "eurlex-chat-data")
HF_TOKEN = os.environ.get("HF_TOKEN")
REPO_ID = f"{HF_USERNAME}/{HF_DATASET}"
def download_current_index():
"""Download the current vectors.npy and chunks.json from HF Hub."""
logger.info("Downloading current index from HF Hub...")
vectors_path = hf_hub_download(REPO_ID, "vectors.npy", repo_type="dataset", token=HF_TOKEN)
chunks_path = hf_hub_download(REPO_ID, "chunks.json", repo_type="dataset", token=HF_TOKEN)
ts_path = hf_hub_download(REPO_ID, "last_updated.txt", repo_type="dataset", token=HF_TOKEN)
vectors = np.load(vectors_path)
with open(chunks_path, "r") as f:
chunks = json.load(f)
with open(ts_path, "r") as f:
last_updated = f.read().strip()
logger.info(f"Current index: {vectors.shape[0]} vectors, {len(chunks)} chunks")
logger.info(f"Last updated: {last_updated}")
return vectors, chunks, last_updated
def query_modified_documents(since_date):
"""Query SPARQL for documents modified since the given date."""
query = f"""
PREFIX cdm: <http://publications.europa.eu/ontology/cdm#>
PREFIX dc: <http://purl.org/dc/elements/1.1/>
SELECT DISTINCT ?doc ?type ?celex ?title ?date
WHERE {{
?doc cdm:work_has_resource-type ?type .
?doc cdm:resource_legal_id_celex ?celex .
?doc dc:title ?title .
?doc cdm:work_date_lastUpdate ?date .
FILTER(LANG(?title) = "en")
FILTER(?date >= "{since_date}"^^xsd:dateTime)
}}
ORDER BY DESC(?date)
"""
r = requests.get(
SPARQL_ENDPOINT,
params={"query": query, "format": "json"},
timeout=60,
)
r.raise_for_status()
data = r.json()
docs = []
for b in data["results"]["bindings"]:
docs.append({
"celex": b["celex"]["value"],
"title": b["title"]["value"],
"date": b["date"]["value"],
"type": b["type"]["value"].split("/")[-1],
})
logger.info(f"Modified documents since {since_date}: {len(docs)}")
return docs
def fetch_document_html(celex):
"""Fetch XHTML content via Cellar RDF graph traversal."""
try:
from xml.etree import ElementTree as ET
ns = {
"rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
"cdm": "http://publications.europa.eu/ontology/cdm#",
}
work_url = f"http://publications.europa.eu/resource/celex/{celex}?language=eng"
r = requests.get(work_url, timeout=30)
r.raise_for_status()
root = ET.fromstring(r.content)
expressions = root.findall(".//cdm:work_has_expression", ns)
expression_url = None
for expr in expressions:
resource = expr.get("{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource")
if resource and resource.endswith(".ENG"):
expression_url = resource
break
if not expression_url and expressions:
resource = expressions[0].get("{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource")
if resource:
expression_url = resource
if not expression_url:
return None
r2 = requests.get(expression_url, timeout=30)
r2.raise_for_status()
expr_root = ET.fromstring(r2.content)
manifestations = expr_root.findall(".//cdm:expression_manifested_by_manifestation", ns)
xhtml_url = None
for manif in manifestations:
resource = manif.get("{http://www.w3.org/1999/02/22-rdf-syntax-ns#}resource")
if resource and (resource.endswith(".xhtml") or resource.endswith(".fmx4")):
xhtml_url = resource
break
if not xhtml_url:
return None
r3 = requests.get(xhtml_url, headers={"Accept": "application/xhtml+xml, text/html"}, timeout=30)
r3.raise_for_status()
return r3.text
except Exception as e:
logger.warning(f"Failed to fetch {celex}: {e}")
return None
def parse_to_chunks(html, celex, title):
"""Parse HTML into text chunks.
⚠️ Uses _parse_html_with_beautifulsoup directly (eurlxp parse_html has
a Polars schema bug with conditional modifier field).
"""
try:
from eurlxp.parser import _parse_html_with_beautifulsoup as internal_parse
import polars as pl
results = internal_parse(html)
except Exception:
return []
if not results:
return []
# Fixed-schema conversion (bypasses eurlxp parse_html bug)
records = []
for r in results:
records.append({
'text': r.text, 'type': r.item_type, 'ref': str(r.ref),
'modifier': r.modifier, 'document': r.context.document,
'article': r.context.article, 'article_subtitle': r.context.article_subtitle,
'paragraph': r.context.paragraph, 'group': r.context.group,
'section': r.context.section,
})
df = pl.DataFrame(records, schema={
'text': pl.Utf8, 'type': pl.Utf8, 'ref': pl.Utf8,
'modifier': pl.Utf8, 'document': pl.Utf8, 'article': pl.Utf8,
'article_subtitle': pl.Utf8, 'paragraph': pl.Utf8,
'group': pl.Utf8, 'section': pl.Utf8,
})
if len(df) == 0:
return []
chunks = []
current_article = None
current_text = []
for row in df.to_dicts():
text = row.get("text", "").strip()
row_type = row.get("type", "")
article = row.get("article")
if not text or len(text) < 40:
continue
if row_type in ("doc-title", "art-subtitle", "group-title", "section-title"):
continue
if row.get("modifier") in ("note", "signatory"):
continue
current_text.append(text)
if current_text:
chunk_text = " ".join(current_text)
if len(chunk_text) > 50:
chunks.append({
"text": chunk_text,
"celex": celex,
"title": title,
"article": current_article,
"type": "article",
})
if not chunks:
paragraphs = [p.strip() for p in html.split("\n\n") if len(p.strip()) > 80]
for para in paragraphs:
chunks.append({
"text": para,
"celex": celex,
"title": title,
"article": None,
"type": "paragraph",
})
return chunks
def embed_chunks(chunks, batch_size=128):
"""Embed chunks using sentence-transformers."""
from sentence_transformers import SentenceTransformer
model = SentenceTransformer(EMBEDDING_MODEL)
texts = [c["text"] for c in chunks]
all_embeddings = []
for i in range(0, len(texts), batch_size):
batch = texts[i : i + batch_size]
embeddings = model.encode(batch, show_progress_bar=False, normalize_embeddings=True)
all_embeddings.append(embeddings)
return np.vstack(all_embeddings).astype(np.float32)
def upload_index(vectors, chunks):
"""Upload updated index to HF Hub."""
api = HfApi()
import tempfile
with tempfile.NamedTemporaryFile(suffix=".npy", delete=False) as vf:
np.save(vf.name, vectors)
api.upload_file(
repo_id=REPO_ID,
path_in_repo="vectors.npy",
path_or_fileobj=vf.name,
repo_type="dataset",
token=HF_TOKEN,
)
with tempfile.NamedTemporaryFile(suffix=".json", mode="w", delete=False) as cf:
json.dump(chunks, cf, indent=2)
cf.flush()
api.upload_file(
repo_id=REPO_ID,
path_in_repo="chunks.json",
path_or_fileobj=cf.name,
repo_type="dataset",
token=HF_TOKEN,
)
with tempfile.NamedTemporaryFile(suffix=".txt", mode="w", delete=False) as tf:
tf.write(datetime.now(timezone.utc).isoformat())
tf.flush()
api.upload_file(
repo_id=REPO_ID,
path_in_repo="last_updated.txt",
path_or_fileobj=tf.name,
repo_type="dataset",
token=HF_TOKEN,
)
logger.info("Index uploaded to HF Hub")
def main():
if not HF_TOKEN:
logger.error("HF_TOKEN environment variable required")
return
# Step 1: Download current index
vectors, chunks, last_updated = download_current_index()
# Build CELEX → chunk mapping for dedup
existing_celex_set = set(c["celex"] for c in chunks)
# Step 2: Query modified documents
docs = query_modified_documents(last_updated)
if not docs:
logger.info("No new/changed documents")
return
# Step 3: Download and process only new docs
logger.info("Downloading new/changed documents...")
html_map = {}
new_celex_ids = [d["celex"] for d in docs if d["celex"] not in existing_celex_set]
changed_celex_ids = [d["celex"] for d in docs if d["celex"] in existing_celex_set]
logger.info(f"New: {len(new_celex_ids)}, Changed: {len(changed_celex_ids)}")
all_to_process = docs
with ThreadPoolExecutor(max_workers=DOWNLOAD_WORKERS) as executor:
future_map = {executor.submit(fetch_document_html, d["celex"]): d for d in all_to_process}
for future in tqdm(as_completed(future_map), total=len(all_to_process), desc="Downloading"):
doc = future_map[future]
try:
html = future.result()
if html:
html_map[doc["celex"]] = html
except Exception:
pass
# Step 4: Parse and chunk new docs
logger.info("Parsing new documents...")
new_chunks = []
for doc in tqdm(all_to_process, desc="Parsing"):
html = html_map.get(doc["celex"])
if html:
parsed = parse_to_chunks(html, doc["celex"], doc["title"])
new_chunks.extend(parsed)
if not new_chunks:
logger.info("No new chunks to add")
return
# Step 5: Remove changed docs from existing, add new ones
celexes_to_remove = set(changed_celex_ids)
filtered_chunks = [c for c in chunks if c["celex"] not in celexes_to_remove]
updated_chunks = filtered_chunks + new_chunks
# Step 6: Embed new chunks only
logger.info(f"Embedding {len(new_chunks)} new chunks...")
new_vectors = embed_chunks(new_chunks)
# Step 7: Merge vectors
# For changed docs: need to figure out which vector indices to remove
# Simple approach: re-embed everything (safer but slower for large changes)
# For small daily updates: just append new vectors
if celexes_to_remove:
# Re-embed all chunks to keep vectors in sync
logger.info("Changed docs detected — re-embedding all chunks...")
updated_vectors = embed_chunks(updated_chunks)
else:
# Just append new vectors
updated_vectors = np.vstack([vectors, new_vectors])
logger.info(f"Updated index: {updated_vectors.shape[0]} vectors, {len(updated_chunks)} chunks")
# Step 8: Upload
upload_index(updated_vectors, updated_chunks)
logger.info("Daily update complete")
if __name__ == "__main__":
main()
Step 5.2 — Write .github/workflows/update-index.yml
name: Daily EUR-Lex Index Update
on:
schedule:
- cron: "0 6 * * *" # Every day at 06:00 UTC
workflow_dispatch: # Allow manual trigger
jobs:
update:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install dependencies
run: |
pip install --upgrade pip
pip install numpy huggingface_hub sentence-transformers requests tqdm eurlxp[sparql] polars
- name: Run incremental update
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
HF_USERNAME: ${{ secrets.HF_USERNAME }}
HF_DATASET: "eurlex-chat-data"
run: python scripts/update_index.py
Step 5.3 — Create the .github/workflows/ directory
mkdir -p ~/Desktop/EUProjects/eur-lex-ai-chat/.github/workflows
Step 5.4 — Set up GitHub secrets
After pushing to GitHub, add these secrets in the repository Settings → Secrets and variables → Actions:
| Secret | Value |
|---|---|
HF_TOKEN |
Your HuggingFace write token |
HF_USERNAME |
Your HuggingFace username |
Phase 6: Deployment
Goal: Deploy the backend to Render, frontend to Vercel, set up cron-job.org keepalives, and submit to Google Search Console.
Step 6.1 — Push code to GitHub
cd ~/Desktop/EUProjects/eur-lex-ai-chat
# Initialize git repo
git init
git add -A
git commit -m "Initial commit: EUR-Lex AI Chat — full implementation"
# Create GitHub repo and push
gh repo create eur-lex-ai-chat --public --push --source=.
# If gh isn't authenticated:
# 1. Go to https://github.com/new
# 2. Create repo named "eur-lex-ai-chat"
# 3. Run:
# git remote add origin https://github.com/YOUR_USERNAME/eur-lex-ai-chat.git
# git branch -M main
# git push -u origin main
Step 6.2 — Deploy backend to Render
- Go to https://dashboard.render.com
- Click New + → Web Service
- Connect your GitHub repository
- Configure:
- Name:
eurlex-chat-api - Runtime: Python 3
- Region: Frankfurt (EU) — for low latency in Europe
- Branch:
main - Build Command:
pip install -r backend/requirements.txt - Start Command:
cd backend && ./startup.sh - Instance Type: Free ($0/month, 512MB RAM)
- Name:
- Add environment variables:
GROQ_API_KEY: your_groq_keyHF_TOKEN: your_hf_tokenHF_USERNAME: your_hf_usernameHF_DATASET: eurlex-chat-data
- Click Deploy Web Service
The deploy takes ~2-3 minutes. Render will:
- Clone the repo
- Install dependencies
- Start the FastAPI server
- Load the vector index from HF Hub at startup
Step 6.3 — Note the Render URL
After deployment, Render assigns a URL like: https://eurlex-chat-api.onrender.com
Test it:
curl https://eurlex-chat-api.onrender.com/health
Expected response:
{"status": "ok", "index_loaded": true, "vector_count": ..., "chunk_count": ...}
Step 6.4 — Configure frontend API URL
Before deploying frontend, set the API URL:
cd ~/Desktop/EUProjects/eur-lex-ai-chat/frontend
# Set the production API URL
echo "PUBLIC_API_URL=https://eurlex-chat-api.onrender.com" > .env.production
Step 6.5 — Deploy frontend to Vercel
- Go to https://vercel.com/new
- Import your GitHub repository
- Configure:
- Framework Preset: Astro
- Root Directory:
frontend - Build Command:
npm run build(default) - Output Directory:
dist(default)
- Add environment variables:
PUBLIC_API_URL:https://eurlex-chat-api.onrender.com
- Click Deploy
Vercel auto-deploys on every push to main.
Step 6.6 — Configure custom domain (optional)
- Buy a domain (e.g.,
eurlex-chat.comfrom Namecheap, Cloudflare, etc.) — this is the ONLY potential cost- OR use Vercel's free subdomain:
eurlex-chat.vercel.app(free, no domain needed)
- OR use Vercel's free subdomain:
- In Vercel dashboard → Project → Settings → Domains → Add your domain
- Add the DNS records provided by Vercel to your domain registrar
- Wait for DNS propagation (5 min to 48 hours)
For $0 hosting, skip the custom domain. Use eurlex-chat.vercel.app — it's free, HTTPS by default, and performs equally well in SEO.
Step 6.7 — Update the sitemap URL
If using a custom domain, update astro.config.mjs:
site: "https://yourdomain.com",
Also update robots.txt and Base.astro canonical URLs.
Step 6.8 — Set up cron-job.org
- Go to https://cron-job.org
- Create account (free, no credit card)
- Create two cron jobs:
Job 1: Keep-alive (prevents Render from sleeping)
- URL:
https://eurlex-chat-api.onrender.com/health - Interval: Every 5 minutes
- Method: GET
Job 2: Data refresh (hourly check for new index)
- URL:
https://eurlex-chat-api.onrender.com/refresh - Interval: Every 60 minutes
- Method: GET
Render free tier sleeps after 15 minutes of inactivity. Cron-job.org pings every 5 minutes to prevent sleep. The /refresh endpoint checks HF Hub for a newer index and reloads if available.
Step 6.9 — Submit to Google Search Console
- Go to https://search.google.com/search-console
- Add your property:
- URL prefix:
https://eurlex-chat.vercel.app
- URL prefix:
- Verify ownership:
- Recommended: Add DNS TXT record at your domain registrar (if using custom domain)
- Alternative: HTML file upload — download the verification HTML file and add it to
frontend/public/
- After verification, submit sitemap:
- Go to Sitemaps section
- Enter:
https://eurlex-chat.vercel.app/sitemap-index.xml - Click Submit
The sitemap tells Google about all pages on the site:
/(landing page)/faq(FAQ with 20 questions)/blog(blog listing)/blog/eu-ai-act-explained/blog/gdpr-guide/blog/what-is-celex
Step 6.10 — Verify deployments
Frontend check:
curl -s https://eurlex-chat.vercel.app | head -50
Should return HTML content (not blank JS shell).
FAQ rich snippets check:
Go to https://search.google.com/test/rich-results
Enter: https://eurlex-chat.vercel.app/faq
Should detect FAQPage schema with 20 questions.
Backend health check:
curl -s https://eurlex-chat-api.onrender.com/health
Chat endpoint check:
curl -s -X POST https://eurlex-chat-api.onrender.com/chat \
-H "Content-Type: application/json" \
-d '{"query": "What is the GDPR?"}'
Phase 7: Go Live Checklist
Goal: Verify everything works end-to-end and the system runs autonomously.
Step 7.1 — Run SEO audit with seofor.dev
# Start the dev server
cd ~/Desktop/EUProjects/eur-lex-ai-chat/frontend
npm run dev &
# Run SEO audit
seo audit run --port 4321
# Or audit the live site
seo audit run --url https://eurlex-chat.vercel.app
Check for:
- ✅ All pages have
<title>and<meta description> - ✅ No broken links
- ✅ Proper heading hierarchy (H1 → H2 → H3)
- ✅ Canonical URLs set correctly
- ✅ Open Graph + Twitter Card tags present
- ✅ Sitemap is valid and reachable
- ✅ robots.txt is valid
Step 7.2 — Verify cron-job.org keepalive
Wait 5 minutes after setting up cron-job.org jobs. Check Render logs:
curl -s https://eurlex-chat-api.onrender.com/health
The loaded_at field should show the server has been up continuously (not sleeping).
Step 7.3 — Test full user flow
- Open
https://eurlex-chat.vercel.appin a browser - Verify landing page loads with chat widget visible
- Send a test query: "What is the GDPR?"
- Verify response includes CELEX number citations
- Navigate to FAQ page — verify all 20 questions expand
- Navigate to Blog — verify all 3 posts listed
- Open blog post — verify Article JSON-LD renders
- Test on mobile — verify responsive design
Step 7.4 — Verify Google Search Console indexing
- In Search Console, go to URL Inspection
- Enter
https://eurlex-chat.vercel.app/ - Click Request Indexing
- Repeat for
/faqand/blog - Check back in 24-48 hours for index status
Step 7.5 — Verify GitHub Actions runs
- Go to GitHub → Actions tab
- Verify the first scheduled run happens at 06:00 UTC next day
- Or trigger manually: click Run workflow → Daily EUR-Lex Index Update
- Monitor the run — check logs for errors
Step 7.6 — Final autonomy checklist
| # | Check | Verified |
|---|---|---|
| 1 | Frontend serves from Vercel (no laptop needed) | |
| 2 | Backend serves from Render (no laptop needed) | |
| 3 | cron-job.org pings /health every 5 min | |
| 4 | cron-job.org pings /refresh every 60 min | |
| 5 | GitHub Actions runs daily at 06:00 UTC | |
| 6 | Google Search Console has sitemap submitted | |
| 7 | SEO structured data validates (JSON-LD) | |
| 8 | Chat widget works with production backend | |
| 9 | All pages return 200, no broken links | |
| 10 | Rate limiting active (20 req/min/IP) |
When all 10 are verified, the system is fully autonomous.
Your laptop can power off. The project runs itself forever at $0/month.
Step 7.7 — What to monitor
| What | How | Frequency |
|---|---|---|
| Render logs | Render dashboard → Logs | Weekly (5 min) |
| GitHub Actions runs | GitHub → Actions tab | Daily (2 min) |
| Google Search Console | Search Console → Performance | Weekly (5 min) |
| cron-job.org stats | cron-job.org dashboard | Monthly (2 min) |
| Groq API usage | Groq console | Monthly (2 min) |
| HF Hub dataset size | HF Hub → dataset | Monthly (2 min) |
Step 7.8 — Future improvements
After the system is live, possible enhancements:
- More document types: Add decisions, international agreements, case law
- More languages: Process documents in French, German, Spanish, etc.
- Multi-query expansion: Send 3-5 related queries to retrieve more relevant context
- Citation highlighting: Highlight exact cited paragraphs in the answer
- Conversation history: Multi-turn chat with context
- Related documents: "What other regulations are related to this topic?"
- PDF export: Download chat conversations as PDF
- Advanced search: Filter by year, document type, CELEX number
- Rate limit per user: Token bucket algorithm instead of fixed window
- Backlinks: List on awesome-opensource, GitHub topics, product hunt
Appendix: Troubleshooting
SPARQL query returns 0 results
If the SPARQL query in build_index.py returns 0 documents:
# Test with a minimal query
python3 -c "
import requests
r = requests.get('https://publications.europa.eu/webapi/rdf/sparql',
params={'query': 'SELECT * WHERE { ?s ?p ?o } LIMIT 1', 'format': 'json'},
timeout=30)
print(r.status_code, len(r.json()['results']['bindings']))
"
If this also fails, the endpoint might be temporarily down. Retry later.
eurlxp HTML fetch fails with WAF
The build_index.py uses Cellar RDF graph traversal (not the EUR-Lex website), so WAF should not be triggered. If it fails:
# Test with eurlxp directly (which has SPARQL fallback)
python3 -c "
from eurlxp import get_html_by_celex_id
html = get_html_by_celex_id('32019R0947')
print('Success:', len(html), 'bytes')
"
Render won't start / out of memory
The index size should fit in 512MB RAM (~150MB vectors + ~50MB chunks.json + ~100MB overhead = ~300MB). If Render runs out of memory:
- Reduce index size by filtering fewer document types
- Or upgrade to Render's $7/month Starter plan (512MB → 1GB RAM)
Frontend can't reach backend (CORS error)
Check:
PUBLIC_API_URLenv var is set correctly in Vercel- Backend CORS
allow_originsincludes the Vercel domain - Backend is not sleeping (cron-job.org /health ping working)
Groq API returns 429 rate limited
The backend has 20 req/min/IP limit. If Groq itself rate-limits:
# Check remaining rate limit
curl -s -I https://api.groq.com/openai/v1/models \
-H "Authorization: Bearer $GROQ_API_KEY" \
| grep -i ratelimit
Free tier: 1,000 requests/day, 20 requests/min. Backend rate limiting prevents hitting Groq limits.
Appendix: Key Links
| Service | URL | Purpose |
|---|---|---|
| Render dashboard | https://dashboard.render.com | Backend hosting |
| Vercel dashboard | https://vercel.com | Frontend hosting |
| cron-job.org | https://cron-job.org | Keepalive + refresh |
| Google Search Console | https://search.google.com/search-console | SEO monitoring |
| HuggingFace Hub | https://huggingface.co/ | Vector index storage |
| Groq Console | https://console.groq.com | AI API key + usage |
| GitHub repo | https://github.com/YOUR_USERNAME/eur-lex-ai-chat | Source code |
| seofor.dev | npx seo audit | SEO auditing |
| EUR-Lex SPARQL | https://publications.europa.eu/webapi/rdf/sparql | Document querying |
| EUR-Lex Cellar | https://publications.europa.eu/resource/ | Document fetching |
| Rich Results Test | https://search.google.com/test/rich-results | JSON-LD validation |
| PageSpeed Insights | https://pagespeed.web.dev/ | Performance testing |