README.md

metadata

title: Ask GreenMetric — Sustainable University Rankings Q&A
emoji: 📚
colorFrom: green
colorTo: blue
sdk: gradio
sdk_version: 6.14.0
app_file: app.py
pinned: false

📚 RAG — Multi Source Information Retrieval

A hybrid Retrieval-Augmented Generation (RAG) project built to learn and implement context injection using the DeepSeek API.

The system answers complex queries about the UI GreenMetric Sustainable University Rankings by combining unstructured narrative guidelines with structured tabular appendices.

v1.0 · Python 3.12.13 · Live Demo

---

🛠️ Tech Stack

Layer	Component
Language	Python 3.12.13
Embedding	BAAI/bge-m3 (SentenceTransformers, 1024-dim)
Retrieval	RAG Fusion: paraphrase ×3 + ChromaDB (cosine, top-k=10) + RRF (k=60), top-n=7
Data	pandas, openpyxl
LLM (pipeline)	DeepSeek API via OpenAI SDK (v4-pro generation, v4-flash routing/paraphrase)
LLM (evaluation)	deepseek-v4-flash with thinking disabled
UI	Gradio
Evaluation	DeepEval 4.0.4

---

🗂️ Data Sources

The UI GreenMetric guidelines document is split into 7 files — 1 narrative document and 6 structured tables. This separation preserves each piece's natural format and lets the retriever target the right structure per query.

File	Format	Content
guidelines_markdown.md	Markdown	Full narrative: methodology, categories, indicators, evidence rules, coordinator info
appendix1_questionnairemasterandscoring.csv	CSV (question-grouped)	118 indicators across 7 categories with answer options and calculated scores
appendix2_listofgreenbuildingelements.csv	CSV (category-grouped)	Green building elements for existing buildings and new construction
appendix3_listanddescriptionofsmartbuildingrequirements.csv	CSV (field-code-grouped)	Smart building requirements (Automation, Safety, Energy, Water, Indoor, Lighting)
table1_nationalcoordinators.csv	CSV (country-grouped)	35 national coordinator universities across 30 countries
table2_categoriesusedandweighting.csv	CSV (category-grouped)	7 categories with their percentage weights
table4_greenhousegasemissionsources.csv	CSV (scope-grouped)	Emission sources classified by Scope 1, 2, and 3

Why split tables out of the PDF? Tables extracted during PDF-to-markdown conversion lose their row-column structure. By storing them as separate CSVs, the retriever can fetch data by natural grouping keys (question number, category, country) instead of parsing broken table text from paragraphs.

---

🧱 Source Code

File	Role
src/chunker.py	Splits markdown (heading-level) and CSV tables (grouped by column) into embeddable chunks
src/embedder.py	Loads BGE-M3, encodes text into vectors, persists to ChromaDB
src/retriever.py	Single-query retrieval + multi-query RRF merge; dispatches by source (pdf/csv/both) and query type (lookup/aggregate)
src/router.py	LLM-based query classifier — routes to source (PDF/CSV/Both/None) and query type (lookup/aggregate); generates paraphrase variants for RAG Fusion
src/generator.py	Formats context + calls DeepSeek to produce answers; flags low-confidence results
src/pipeline.py	Orchestrator — wires route → paraphrase → multi-query RRF → (opt-in reranker) → generate
src/budget.py	Token budget tracking with HF Datasets persistence; guards against API overspend
src/conversation.py	Logs user prompts + responses to HF Datasets for quality monitoring
src/reranker.py	BGE V2-M3 cross-encoder reranker (enabled by default)
src/evaluate.py	Pre-computes routes + paraphrases, runs pipeline, DeepEval batch scorer, 5-section report with context debugging and per-case timing
build_collection.py	Chunks all 7 sources, prints sanity check (317 expected), builds ChromaDB collection
app.py	Gradio chat UI

---

🛠️ Dependencies

Key libraries beyond the standard Python data stack:

Package	Version	Purpose
chromadb	1.5.9	Vector database
sentence-transformers	5.5.1	Embedding model (BGE-M3)
openai	2.38.0	DeepSeek API client
gradio	6.14.0	Web UI
deepeval	4.0.4	Evaluation metrics
huggingface-hub	≥0.20	HF Datasets storage for budget + conversation logs
transformers	4.57.6	LLM model loading
FlagEmbedding	1.4.0	BGE reranker + BGE-M3 embeddings

---

⚙️ Pipeline Architecture

1. Ingestion: Markdown → Heading-Level Chunking, CSV → Group-Based Chunking (118 question groups, 6 green building categories, 6 smart building fields, 30 coordinator countries, 7 category weights, 3 emission scopes) → Embed with BGE-M3 → Store in ChromaDB.
2. Retrieval & Generation: User Query → Budget Guard → Router (LLM) → Paraphrase (3 variants via DeepSeek) → Multi-query ChromaDB search (top-k=10 each) → RRF (k=60) → top 7 chunks → (reranker, enabled by default) → Context Concatenation → DeepSeek LLM Generation.

---

📊 Evaluation (DeepEval)

Metric	v0.5 (MiniLM)	v0.6 (BGE-M3)	v1.0 (BGE-M3 + BGE)
Faithfulness	0.91	0.93	0.96
Contextual Recall	0.74	0.81	0.83
Contextual Precision (NDCG@K)	0.45	0.56	0.72
G-Eval Correctness	0.43	0.51	0.59
Router Accuracy	80.0%	77.5%	91.5%

47 test cases. LLM-as-judge ±0.05-0.08 variance. See test_cases/RERANKER_REPORT_V2.md for full benchmark.

---

🧠 Design Decisions

Decision	Reason
Cosine similarity	Matches the training metric of the embedding model
BGE-M3 over Qwen3	Better CP on structured CSV + markdown data (0.60 solo vs 0.44). Native dense+sparse training signal. No instruction prompt needed. Qwen3 evaluated and reverted.
RAG Fusion (paraphrase ×3 + RRF)	Resolves vocabulary mismatches that cosine search alone misses
Question-grouped CSV chunks	Prevents partial/orphaned indicators — the LLM always sees a complete criterion
Formula injection in chunks	Embedding formulas directly into chunk text reduces hallucination on calculation questions
No cosine distance threshold	Removed 0.5 threshold — was discarding relevant chunks; RRF handles quality ordering
Single ChromaDB collection	At 317 chunks, per-source collections add complexity with no performance gain
No conversation history	Degrades router accuracy — few-shot training uses single queries, and prior-turn vocabulary pulls the router toward stale sources
BGE reranker enabled	Evaluated 5 rerankers across 2 embedders. BGE V2-M3 + BGE-M3 embedder gives best CP (0.72) and CR (0.83). Qwen3+reranker hits 0.61 CP. Reranker enabled by default.

---

🗺️ v0.5 → v1.0 Roadmap

• Embedding model upgrade to BGE-M3 (re-adopted after Qwen3 evaluation)
• Rebuild ChromaDB collection (317 chunks, 1024-dim)
• Embedding model upgrade to Qwen3-Embedding-0.6B (evaluated, CP inferior, reverted to BGE-M3)
• Implement RAG Fusion (paraphrase + multi-query RRF)
• Evaluate 5 rerankers — BGE V2-M3 adopted, enabled by default
• Budget management for API spending
• Deploy on HuggingFace Spaces (fortunius/rag-uigreenmetric)
• Aggregate query optimization (metadata-driven stats, zero LLM, token reduction 15K→2K)
• Router tuned to 91-94% with 26 few-shot examples — LLM variance (±5-8%), some queries span both sources
• Dense + sparse hybrid retrieval using BGE-M3 flag embeddings | Tested, result in table below:

Metric	Dense	Hybrid	Δ
Faithfulness	0.96	0.92	-0.04
Contextual Recall	0.83	0.84	+0.01
Contextual Precision	0.72	0.71	-0.01
G-Eval	0.59	0.51	-0.08
Router Accuracy	91.5%	89.4%	-2.1%
Avg latency	10,188ms	12,444ms	+2,256ms

Sparse added 2.2s latency with no CP gain and degraded G-Eval. Not worth the cost.

---

🗺️ v1.5 Roadmap

• Show retrieved context (collapsible view of chunks used per answer)
• Source citation (display which data source answered the query)
• Token cost per query (exact usage and estimated cost under each answer)
• Pipeline timing (response generation time displayed)
• Route badge color-coding (green/blue/purple/gray for pdf/csv/both/none)
• Markdown rendering (properly formatted lists, tables, bold text)
• Copy answer button (clipboard copy on each response)
• Welcome + example chips (clickable sample questions on empty chat)
• Feedback thumbs (±1 per answer, logged for future evaluation)
• Dark mode toggle (OS preference detection + manual toggle)

🚀 v2.0 Roadmap

• FastAPI backend (POST /ask, GET /budget, GET /health)
• Streaming generator (DeepSeek stream=True → StreamingResponse)
• Conversation history (generator-only, router stays single-query)
• Dockerize backend (Dockerfile + docker-compose.yml)
• Svelte chat UI with Vercel AI SDK (replaces Gradio)
• Strip Gradio (app.py, gradio from requirements)

---

⚠️ Known Limitations

• Router accuracy ~91-94%: Improved with few-shot tuning but 1-4 cases still misrouted per run due to LLM variance (±5-8%). Some queries genuinely span both PDF and CSV sources — neither route is wrong, just incomplete.
• CP bottleneck (0.72): Contextual Precision remains the weakest metric, improving from 0.60 (BGE-M3 solo) to 0.72 (BGE-M3 + reranker + aggregate stats). Sparse hybrid retrieval tested and rejected. Further gains likely require embedder fine-tuning on domain-specific data.
• G-Eval language sensitivity: Scoring dips when the answer and ground truth differ in language (EN ↔ ID) despite being semantically equivalent.
• RAG Fusion latency: Paraphrase LLM call + 4× embeddings adds ~1-2s per query vs single-query retrieval.

---