README.md

---
title: Enterprise RAG System
emoji: 🏢
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 4.44.0
app_file: app.py
pinned: true
---

# Enterprise Knowledge Retrieval System

A production-oriented Retrieval-Augmented Generation (RAG) pipeline built for enterprise document Q&A. Demonstrates AI engineering depth across ingestion, retrieval, generation, evaluation, and observability — fully free to run.

## What This System Does

Upload any enterprise PDF (policy documents, financial reports, technical manuals, compliance frameworks) and ask natural language questions. The system retrieves semantically relevant sections and generates grounded answers using Mistral-7B — refusing to answer when evidence is insufficient rather than hallucinating.

## System Architecture

```
PDF Upload → Text Extraction → Chunking → Embedding → FAISS Index
                                                            ↓
User Query → Embed Query → Cosine Similarity Search → Top-K Chunks
                                                            ↓
                                              Relevance Threshold Check
                                              ↓              ↓
                                         Fallback      Prompt Assembly
                                                            ↓
                                              Mistral-7B via HF API
                                                            ↓
                                    Grounded Answer + Evaluation Scores
                                                            ↓
                                          Langfuse Trace + Local Log
```

## Tech Stack

| Layer | Technology | Rationale |
|---|---|---|
| LLM | Mistral-7B-Instruct (HF API) | Free hosted inference, strong instruction following |
| Embeddings | all-MiniLM-L6-v2 | Runs locally, no API cost, 384-dim, fast on CPU |
| Vector Store | FAISS IndexFlatIP | Exact cosine search, zero infra, ideal for < 500k chunks |
| Evaluation | Custom cosine similarity metrics | No LLM calls required, millisecond latency |
| Observability | Langfuse (optional) + JSONL logs | Free tier, self-hostable, structured traces |
| UI | Gradio | HF Spaces native, minimal setup |

## Engineering Decisions & Tradeoffs

**Chunking strategy:** Fixed-size character chunking with sentence-boundary snapping. Faster than semantic chunking (no extra embeddings), cleaner than hard character splits. The 512-token / 64-token overlap defaults work well for most English documents. For financial tables or code, increase chunk size; for FAQ-style documents, decrease it.

**Relevance threshold (0.35):** Below this cosine similarity, the retrieved chunks are unlikely to contain the answer. Rather than hallucinate, we return a fallback message. This threshold should be tuned empirically on your document corpus.

**Evaluation without an LLM judge:** Ragas-style metrics use a second LLM to evaluate answers — expensive in both cost and latency. Our proxy metrics (answer-context cosine similarity for faithfulness, answer-query similarity for relevance) run in ~20ms and correlate well with human judgment. For a production system, augment with periodic human eval on a golden test set.

## Deployment on Hugging Face Spaces

1. Create a new Space (SDK: Gradio)
2. Add files from this repository
3. Go to **Settings → Repository secrets** and add:
   - `HF_TOKEN` — your HF read token (huggingface.co → Settings → Access Tokens)
   - `LANGFUSE_PUBLIC_KEY` (optional)
   - `LANGFUSE_SECRET_KEY` (optional)
4. The Space will build and launch automatically

## Known Limitations & Production Concerns

- **Scanned PDFs:** PyPDF cannot extract text from image-based PDFs. Add Tesseract OCR for production use.
- **Multi-document retrieval:** Current design supports one document per session. For a multi-document corpus, add document-level metadata to chunks and filter by source.
- **FAISS persistence:** The index lives in memory and resets when the Space restarts. For production, serialize the index with `faiss.write_index()` and store in persistent storage.
- **Concurrent users:** Gradio's `gr.State()` isolates per-session state, but the embedding model is shared. Under high concurrency, add a request queue.
- **HF Inference API rate limits:** Free tier allows ~1000 requests/day per token. For production, deploy a dedicated Inference Endpoint.

## Scalability Path

| Scale | Recommended Change |
|---|---|
| > 500k chunks | Switch FAISS to IndexIVFFlat with nprobe tuning |
| > 10 users | Add Redis for session state isolation |
| Production SLA | Move to dedicated HF Inference Endpoint or vLLM |
| Multi-document | Add metadata filtering layer + document registry |
| Compliance | Add PII detection before ingestion, audit log retention |

## Evaluation Methodology

Three proxy metrics computed without an LLM judge:

- **Faithfulness (0–1):** For each answer sentence, maximum cosine similarity to any retrieved chunk. Scores < 0.5 indicate potential hallucination.
- **Answer Relevance (0–1):** Cosine similarity between the answer and the original query embeddings. Measures whether the answer addresses what was asked.
- **Context Precision (0–1):** Rank-weighted average of retrieval similarity scores. Measures retrieval quality independent of generation.

Overall score is an unweighted average. For compliance applications, weight faithfulness higher.

## Future Improvements

- [ ] Add OCR fallback for scanned PDFs (Tesseract/AWS Textract)
- [ ] Implement hybrid search (BM25 + dense) for better recall on keyword queries
- [ ] Add re-ranking layer (cross-encoder) between retrieval and generation
- [ ] Persist FAISS index to HF dataset for cross-session memory
- [ ] Add streaming response support for lower perceived latency
- [ ] Build evaluation dataset from uploaded documents automatically