README.md

metadata

title: AskTheHandbook
emoji: 🎓
colorFrom: blue
colorTo: indigo
sdk: docker
pinned: false
app_port: 7860

AskTheHandbook

A production-style RAG system that answers student questions about the IITM BS Degree Programme (fees, grading, exams, eligibility, academic structure) strictly grounded in the official handbook and grading documents, with source citations.

🔗 Live demo: huggingface.co/spaces/rishitpant/AskTheHandbook

Features:

• Hybrid retrieval + cross-encoder reranking
• Versioned prompts
• Automated evaluation harness with checkpointing
• Langfuse observability
• Rate limiting via slowapi (5 requests/minute per IP, proxy-aware)
• CI quality gate with auto-deploy to Hugging Face Spaces
• FastAPI backend + web chat UI, Dockerized for production

---

Architecture

source PDFs  →  LlamaParse  →  data/*.md  →  ingest.py  →  Chroma Cloud (or local ChromaDB)
                                                                  │
question  →  retrieve.py  ────────────────────────────────────────┴─── BM25 (keyword index)
                  │
          EnsembleRetriever (RRF fusion, weights 0.6/0.4)
                  │
          Cross-encoder reranker (ms-marco-MiniLM-L-6-v2)
                  │
          top-N chunks (with source + section metadata)
                  │
generate.py  →  prompts.yaml (versioned prompts)
                  │
          Groq LLM (llama-3.3-70b-versatile), streamed
                  │
          Grounded answer + cited sources
                  │
   app.py (FastAPI) → static/index.html (chat UI) / CLI
                  │
          Langfuse trace (latency, chunk metadata, rerank scores)

---

Project layout

RAG/
├── app.py                         # FastAPI app (chat/stream + static UI + rate limiting)
├── static/
│   └── index.html                 # chat frontend
├── data/
│   ├── student-handbook.md        # source document 1
│   └── grading-document.md        # source document 2
├── src/
│   ├── ingest.py                  # chunk + embed + index (local or Chroma Cloud)
│   ├── retrieve.py                # hybrid search + rerank
│   ├── generate.py                # Generator class, RAG chain + CLI chat
│   └── tracing.py                 # Langfuse observability (opt-in)
├── eval/
│   ├── eval_prompts.json          # 10-question set
│   ├── evaluate.py                # DeepEval harness
│   └── report.json                # latest run's results
├── db/                            # local ChromaDB vector store (generated)
├── prompts.yaml                   # versioned prompts (prod + eval)
├── Dockerfile
├── requirements.txt               # full dev environment
├── requirements-prod.txt          # slim deps for Docker/CI
└── .github/workflows/ci.yaml      # eval gate + HF Spaces deploy

---

Ingestion (src/ingest.py)

• Source docs: PDFs (exported from Google Docs) converted to Markdown via LlamaParse (student-handbook.md, grading-document.md)
• Cleans residual Google Docs export noise (page numbers, "Published using Google Docs" footers, info-icon artifacts) carried over from the source PDFs; converts HTML tables to Markdown tables
• Header-aware splitting (MarkdownHeaderTextSplitter) + RecursiveCharacterTextSplitter (2000 char chunks, 200 overlap)
• Wraps each chunk with [Course: h1 > h2 > h3] tags for both vector and BM25 search
• Embeddings: BAAI/bge-small-en-v1.5
• Writes to Chroma Cloud if CHROMA_API_KEY is set, else to local ChromaDB

Retrieval (src/retrieve.py)

• Hybrid search: vector + BM25, fused via EnsembleRetriever (weights [0.6, 0.4]), 8 candidates each
• Cross-encoder reranking: cross-encoder/ms-marco-MiniLM-L-6-v2 rescores every (query, chunk) pair
• ToC penalty: deprioritizes table-of-contents-like chunks
• Scrubs residual export noise before chunks reach the LLM
• Returns top N chunks (default 5) with text, source, page (section path), rerank_score
• Same Chroma Cloud / local switch as ingestion, via USE_CHROMA_CLOUD

Generation (src/generate.py)

• Generator class wraps the retriever + Groq LLM client, used by both the CLI and the API
• Prompts loaded from prompts.yaml (not hardcoded), version printed at startup
• System prompt enforces: strict grounding, no hallucination, table extraction, mandatory citations, defining acronyms, flagging overlapping fee figures, one course at a time
• Streams tokens via Groq's OpenAI-compatible endpoint
• Model overridable via RAG_MODEL env var (default: llama-3.3-70b-versatile)
• get_sources() returns the deduplicated source list for the last answer
• Each turn wrapped in an @observe span for tracing

Web app (app.py, static/index.html)

• FastAPI app exposing:
  • GET /api/health — pipeline/backend status
  • POST /api/chat — non-streaming chat
  • POST /api/chat/stream — Server-Sent Events streaming chat
• RAG pipeline loaded once at startup (FastAPI lifespan), not per-request
• Static chat UI served from static/ at /
• Rate limiting: 5 requests/minute per IP via slowapi; proxy-aware (reads X-Forwarded-For)
• CORS open by default (allow_origins=["*"])

Observability (src/tracing.py)

• Langfuse tracing is opt-in via LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST
• When enabled: each turn produces a trace with a hybrid_retrieval span (chunks, sections, rerank scores, model costs, latency)
• When disabled: observe becomes a no-op — no tracing dependency required

---

Evaluation (eval/evaluate.py)

• 20-question set (eval/eval_prompts.json), curated for category coverage, not size due to api rate limits
• Weak-scoring questions are kept deliberately to surface gaps, not flatter the score

Pipeline per question:

• Retrieve top-4 chunks via the full hybrid + rerank pipeline
• Generate an answer with llama-3.3-70b-versatile (eval_system + shared human prompt, 150-token cap)
• Fast keyword-hit check (no LLM) due to api rate limits
• Three DeepEval LLM-judge metrics (same model as judge): Faithfulness, Answer Relevancy, Contextual Precision.

Reliability features:

• Auto-retry with backoff on Groq 429s
• Throttling between calls/metrics to stay under rate limits
• Per-question checkpointing (eval_checkpoint.json) — resumes after a crash, clears on a clean pass
• Every run writes eval/report.json: model/prompt versions, per-metric averages, per-question breakdown, pass/fail gate
• Judge model overridable via JUDGE_MODEL env var (default: llama-3.3-70b-versatile); CI uses llama-3.1-8b-instant for speed

Latest result (20 questions, judge = llama-3.3-70b-versatile, threshold 0.8):

Metric	Score
Keyword Hit Rate	100%
Faithfulness (avg)	91%
Answer Relevancy (avg)	91%
Contextual Precision (avg)	96%
Gate (min of keyword & faithfulness)	0.908 — PASSED

• Judge choice matters: an 8B judge produced a gate score of 0.618 with false-zero faithfulness scores on verified-correct answers — it lacks reasoning capacity for compound eligibility/fee questions. The 70B judge raised the gate to 0.908, reflecting actual system quality. CI keeps the 8B model for speed; 70B is used for local runs.

---

Prompt versioning (prompts.yaml)

• system — full production system prompt (generate.py)
• eval_system — leaner version for the 150-token eval budget, same core grounding/refusal behavior
• human — shared {context} / {question} template for both prod and eval
• version — bumped on any prompt change, recorded in every report.json

Current version: 1.4.0

---

CI/CD (.github/workflows/ci.yaml)

On push to main:

1. Quality gate — installs requirements-prod.txt, runs the keyword-only eval against Chroma Cloud using llama-3.1-8b-instant (python eval/evaluate.py --no-deepeval --threshold 0.8)
2. Deploy — if the gate passes, force-pushes the repo to the linked Hugging Face Space, which rebuilds the Docker image and redeploys

• DeepEval (faithfulness/relevancy/precision) gate is not yet wired into CI due to api rate limits — roadmap item, planned for PRs into main only (~5–10 min throttled runtime)

---

Deployment

• Docker: Dockerfile builds a slim image (requirements-prod.txt), runs uvicorn app:app on port 7860
• Hugging Face Spaces: repo includes HF Spaces metadata (top of this file); CI auto-deploys on every successful main push
• Vector store in prod: the image does not ship data/ or a local db/ — it expects CHROMA_API_KEY/CHROMA_TENANT/CHROMA_DATABASE pointing at a pre-ingested Chroma Cloud collection

---

How to run

pip install -r requirements.txt

# Build the vector index (required before first run)
python src/ingest.py

# Chat with the assistant (CLI)
python src/generate.py

# Run the web app locally
uvicorn app:app --reload
# → open http://localhost:8000

# Run the full eval suite (DeepEval + keyword gate)
python eval/evaluate.py --save-report

# Run keyword-only gate (fast, no LLM judge)
python eval/evaluate.py --no-deepeval --threshold 0.5

# Or run via Docker
docker build -t askthehandbook .
docker run -p 7860:7860 --env-file .env askthehandbook

Environment variables (create a .env file):

GROQ_API_KEY=your_key_here

# Optional — use Chroma Cloud instead of local persistent ChromaDB
CHROMA_API_KEY=...
CHROMA_TENANT=...
CHROMA_DATABASE=...

# Optional — enables Langfuse tracing
LANGFUSE_PUBLIC_KEY=...
LANGFUSE_SECRET_KEY=...
LANGFUSE_HOST=https://cloud.langfuse.com

---

Tech stack

LangChain (LCEL) · ChromaDB / Chroma Cloud · BM25 (rank-bm25) · BAAI/bge-small-en-v1.5 embeddings · cross-encoder/ms-marco-MiniLM-L-6-v2 reranker · Groq (llama-3.3-70b-versatile) · FastAPI · slowapi · DeepEval · Langfuse · GitHub Actions · Docker · Hugging Face Spaces

---

What's next

• Wire the DeepEval gate into CI (currently keyword-only; scope to PRs into main)
• Improve Contextual Precision on scheduling/fee-overlap questions — tune BM25/vector weights, increase HYBRID_TOP_K, or add query expansion
• Add auth to the public API endpoints before wider sharing