| # Multi-Document Research Assistant — RAG with Retrieval Quality Evaluation | |
| > Upload multiple PDFs. Ask questions. Get cited answers grounded in your documents — with automatic retrieval quality scoring. | |
| [](https://neurotic-evidence-defendant.ngrok-free.dev/) | |
| [](https://python.org) | |
| [](https://langchain.com) | |
| [](https://trychroma.com) | |
| --- | |
| ## 🎯 Problem | |
| Most LLMs hallucinate when asked about private or domain-specific documents. They generate confident answers from training data — not from your actual content. | |
| Standard RAG systems fix this but introduce a new problem: **you can't tell when retrieval fails.** Wrong chunks get retrieved. The LLM answers confidently from bad context. No warning. | |
| This project solves both problems: | |
| 1. Grounds all answers in your uploaded documents | |
| 2. Scores retrieval quality on every query so you know when to trust the answer | |
| --- | |
| ## 🏗️ Architecture | |
| ``` | |
| PDF Documents | |
| │ | |
| ▼ | |
| ┌─────────────────┐ | |
| │ PyPDFLoader │ Extract text page by page | |
| └────────┬────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────┐ | |
| │ RecursiveText │ Split into 512-token chunks | |
| │ Splitter │ with 50-token overlap | |
| └────────┬────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────┐ | |
| │ all-MiniLM-L6 │ Embed each chunk → 384-dim vector | |
| │ Embeddings │ | |
| └────────┬────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────┐ | |
| │ ChromaDB │ Store vectors + text + metadata | |
| │ (Vector Store) │ (source filename + page number) | |
| └────────┬────────┘ | |
| │ | |
| User Query | |
| │ | |
| ▼ | |
| ┌─────────────────┐ | |
| │ Similarity │ Cosine search → Top-k chunks | |
| │ Search │ | |
| └────────┬────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────┐ | |
| │ Gemini Flash │ Generate answer from chunks only | |
| │ (LLM) │ | |
| └────────┬────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────┐ | |
| │ Custom Evaluator│ Score grounding, relevance, | |
| │ │ completeness | |
| └─────────────────┘ | |
| │ | |
| ▼ | |
| Answer + Sources + Quality Scores | |
| ``` | |
| --- | |
| ## 📊 Evaluation Metrics | |
| Instead of using RAGAS (which requires paid OpenAI API), this project implements a custom lightweight evaluator — free, fast, and interpretable. | |
| | Metric | What It Measures | How | | |
| |---|---|---| | |
| | **Grounding Score** | Is the answer based on retrieved chunks? | Word overlap: answer ∩ context / answer | | |
| | **Retrieval Relevance** | Did we retrieve the right chunks? | Cosine similarity: query vector vs chunk vectors | | |
| | **Answer Completeness** | Did the LLM use the retrieved context? | Word overlap: answer ∩ context / context | | |
| ### Sample Results (Attention Is All You Need paper) | |
| | Query | Grounding | Relevance | Completeness | | |
| |---|---|---|---| | |
| | "What is the attention mechanism?" | 0.97 | 0.61 | 0.16 | | |
| | "Who are the authors?" | 0.95 | 0.58 | 0.12 | | |
| | "What is the conclusion?" | 0.91 | 0.55 | 0.14 | | |
| **Key insight:** High grounding (0.97) but low completeness (0.16) means the LLM extracted a precise answer from a small portion of retrieved context — expected behavior for factual queries. | |
| --- | |
| ## 🚀 Quick Start | |
| ### Prerequisites | |
| - Python 3.10+ | |
| - Google AI Studio API key (free at [aistudio.google.com](https://aistudio.google.com)) | |
| ### Installation | |
| ```bash | |
| git clone https://github.com/aneebnaqvi15/rag-research-assistant | |
| cd rag-research-assistant | |
| python -m venv venv | |
| venv\Scripts\activate # Windows | |
| # source venv/bin/activate # Mac/Linux | |
| pip install -r requirements.txt | |
| ``` | |
| ### Configuration | |
| Create a `.env` file: | |
| ``` | |
| GOOGLE_API_KEY=your_key_here | |
| ``` | |
| ### Run | |
| ```bash | |
| streamlit run app.py | |
| ``` | |
| Open `http://localhost:8501` | |
| --- | |
| ## 🛠️ Tech Stack | |
| | Component | Tool | Why | | |
| |---|---|---| | |
| | PDF Loading | `pypdf` + LangChain | Page-level metadata for citations | | |
| | Chunking | `RecursiveCharacterTextSplitter` | Respects sentence boundaries | | |
| | Embeddings | `all-MiniLM-L6-v2` | Fast, free, 384-dim, runs on CPU | | |
| | Vector Store | `ChromaDB` | Local, zero-config, stores metadata | | |
| | LLM | `Gemini 2.5 Flash` | Free tier, strong reasoning | | |
| | Orchestration | `LangChain` | Connects all RAG components | | |
| | UI | `Streamlit` | Rapid prototyping, real-time logs | | |
| | Evaluation | Custom (NumPy + sklearn) | Free, interpretable, no OpenAI dependency | | |
| --- | |
| ## ✨ Features | |
| - **Multi-document support** — Upload and query across multiple PDFs simultaneously | |
| - **Source citations** — Every answer shows exact filename and page number | |
| - **Real-time processing logs** — Watch the pipeline run: load → chunk → embed → index | |
| - **Retrieval quality scores** — Three metrics scored on every query | |
| - **Bring your own API key** — Toggle in sidebar to use your own Gemini key | |
| - **Custom model selection** — Enter any Gemini model string from AI Studio | |
| - **Adjustable retrieval** — Control chunk size and top-k via sidebar sliders | |
| --- | |
| ## 📁 Project Structure | |
| ``` | |
| rag-research-assistant/ | |
| ├── app.py # Streamlit UI + full RAG pipeline | |
| ├── requirements.txt # Pinned dependencies | |
| ├── .env # API keys (not committed) | |
| ├── .gitignore | |
| └── README.md | |
| ``` | |
| --- | |
| ## 💡 Key Learnings | |
| **1. Dependency hell is real.** | |
| ChromaDB, NumPy 2.0, and OpenTelemetry have a war inside Colab's environment. The fix: use `EphemeralClient()` instead of file-based persistence, and never pin NumPy manually. | |
| **2. RAGAS is not free.** | |
| RAGAS v0.4+ requires OpenAI API for their `InstructorLLM`. Building a custom evaluator is not a compromise — it's better engineering. You understand every metric you ship. | |
| **3. Chunk size is a hyperparameter.** | |
| 512 tokens with 50-token overlap is a starting point, not a truth. Smaller chunks = sharper vectors but less context. Tune based on your document type. | |
| **4. Retrieval can fail silently.** | |
| A grounding score of 1.0 does not mean a correct answer. It means every word in the answer exists somewhere in retrieved chunks. You need multiple metrics to trust a RAG system. | |
| **5. LLM quality matters more than RAG architecture.** | |
| `flan-t5-base` produced gibberish from perfect retrieval. `Gemini 2.5 Flash` produced accurate answers from the same chunks. The retrieval pipeline is only as useful as the model that reads it. | |
| --- | |
| ## 🔗 Related Projects | |
| This project is **Part 2** of a connected AI/ML portfolio: | |
| | Project | Focus | Link | | |
| |---|---|---| | |
| | **Banking77 Intent Classifier** | NLP fine-tuning with DistilBERT + LoRA | [GitHub](https://github.com/aneebnaqvi15/banking77-intent-classifier) | | |
| | **RAG Research Assistant** | Retrieval-Augmented Generation + Evaluation | This repo | | |
| Together these demonstrate: fine-tuning (shaping model behavior) vs RAG (shaping model knowledge) — two complementary approaches to applied NLP. | |
| --- | |
| ## 📄 References | |
| - [Attention Is All You Need](https://arxiv.org/abs/1706.03762) — Vaswani et al. (2017) | |
| - [REALM: Retrieval-Augmented Language Model Pre-Training](https://arxiv.org/abs/2002.08909) | |
| - [LangChain Documentation](https://docs.langchain.com) | |
| - [ChromaDB Documentation](https://docs.trychroma.com) | |
| - [Sentence Transformers — all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | |
| --- | |
| ## 👤 Author | |
| **Syed Muhammad Aneeb** | |
| CS Graduate · Full-Stack Developer · AI Engineer | |
| [](https://github.com/aneebnaqvi15) | |
| --- | |
| *Built with zero GPU budget on Google Colab free tier. Constraints breed creativity.* |