Abeshith commited on
Commit
dbb1f7a
·
verified ·
1 Parent(s): dfa6a46

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +8 -186
README.md CHANGED
@@ -1,186 +1,8 @@
1
- # RAG Project - Learn About Transformers
2
-
3
- A production-ready **Corrective Retrieval-Augmented Generation (CRAG)** system built with LangChain, LangGraph, and FastAPI. This project implements an intelligent RAG pipeline that not only retrieves relevant documents but also **validates, corrects, and improves** retrieval quality through an agent-based workflow.
4
-
5
- ## What Makes This Different from Traditional RAG?
6
-
7
- ### Traditional RAG:
8
- ```
9
- Query → Retrieve Documents → Generate Answer
10
- ```
11
- **Problem**: If retrieved documents are irrelevant or low-quality, the answer will be poor.
12
-
13
- ### This Project (Corrective RAG):
14
- ```
15
- Query → Retrieve → Grade Quality → Transform Query if Needed → Web Search if Necessary → Generate
16
- ```
17
- **Solution**: Intelligent agent workflow that **self-corrects** by grading document relevance and taking corrective actions.
18
-
19
- ## Architecture
20
-
21
- ```mermaid
22
- graph LR
23
- A[User Query] --> B[Retrieve]
24
- B --> C[FAISS+MMR]
25
- C --> D[Rerank]
26
- D --> E{Grade}
27
- E -->|Relevant| F[Generate]
28
- E -->|Partial| G[Filter]
29
- E -->|Poor| H[Transform]
30
- G --> F
31
- H --> I[Web Search]
32
- I --> F
33
- F --> J[Groq LLM]
34
- J --> K[Answer]
35
- ```
36
-
37
- ## Key Features
38
-
39
- ### 1. **Intelligent Document Grading**
40
- - LLM evaluates retrieved documents for relevance
41
- - Filters out low-quality results automatically
42
- - Ensures only useful context reaches generation
43
-
44
- ### 2. **Query Transformation**
45
- - Rewrites ambiguous or poor queries
46
- - Improves retrieval on second attempt
47
- - Adaptive query refinement
48
-
49
- ### 3. **Web Search Fallback**
50
- - Tavily API integration for external knowledge
51
- - Activates when local documents insufficient
52
- - Combines local + web results
53
-
54
- ### 4. **Advanced Retrieval Stack**
55
- - **FAISS** vector store with MMR search
56
- - **FastEmbed** (BAAI/bge-small-en-v1.5) embeddings
57
- - **FlashRank** (rank-T5-flan) reranking
58
- - Self-query retriever support
59
-
60
- ### 5. **LangGraph Agent Workflow**
61
- - State machine orchestration
62
- - Conditional routing logic
63
- - Transparent decision-making
64
-
65
- ## Tech Stack
66
-
67
- | Component | Technology |
68
- |-----------|------------|
69
- | **LLM** | Groq (openai/gpt-oss-120b) |
70
- | **Embeddings** | FastEmbed (BAAI/bge-small-en-v1.5) |
71
- | **Vector Store** | FAISS |
72
- | **Reranker** | FlashRank (rank-T5-flan) |
73
- | **Agent Framework** | LangGraph |
74
- | **RAG Framework** | LangChain 0.3.x |
75
- | **Web Search** | Tavily API |
76
- | **Web Framework** | FastAPI + Uvicorn |
77
- | **Observability** | LangSmith (optional) |
78
- | **Document Source** | "Attention Is All You Need" (Transformer paper) |
79
-
80
- ## Project Structure
81
-
82
- ```
83
- RAG Project/
84
- ├── project/
85
- │ ├── config/
86
- │ │ └── config.yaml # Model & pipeline configuration
87
- │ ├── logger/
88
- │ │ └── logging.py # Centralized logging
89
- │ ├── exception/
90
- │ │ └── except.py # Custom exception handling
91
- │ ├── utils/
92
- │ │ ├── config_loader.py # YAML config loader
93
- │ │ └── model_loader.py # LLM & embedding initialization
94
- │ ├── source/
95
- │ │ └── data_preparation.py # PDF/ArXiv document loading
96
- │ ├── model/
97
- │ │ ├── retriever.py # FAISS retriever with MMR
98
- │ │ └── reranking.py # FlashRank reranking
99
- │ ├── prompts/
100
- │ │ └── prompt_template.py # RAG, Router, WebSearch prompts
101
- │ └── pipeline/
102
- │ ├── rag.py # Core RAG pipeline
103
- │ └── agents.py # CRAG agent workflow
104
- ├── templates/
105
- │ └── index.html # Web UI template
106
- ├── static/
107
- │ └── styles.css # Purple gradient theme
108
- ├── data/
109
- │ └── attention-is-all-you-need.pdf
110
- ├── app.py # FastAPI application
111
- ├── main.py # CLI entry point
112
- ├── Dockerfile # Docker containerization
113
- └── requirements.txt # Dependencies
114
-
115
- ```
116
-
117
- ## Quick Start
118
-
119
- ### 1. Clone & Install
120
- ```bash
121
- git clone https://github.com/Abeshith/RAG-Project-PipeLine.git
122
- cd RAG-Project-PipeLine
123
- pip install -r requirements.txt
124
- ```
125
-
126
- ### 2. Set Environment Variables
127
- Create `.env` file:
128
- ```env
129
- GROQ_API_KEY=your_groq_api_key
130
- GOOGLE_API_KEY=your_google_api_key
131
- LANGSMITH_API_KEY=your_langsmith_key
132
- TAVILY_API_KEY=your_tavily_key
133
- ```
134
-
135
- ### 3. Run Web Interface
136
- ```bash
137
- python app.py
138
- ```
139
- Visit: http://localhost:8000
140
-
141
- ### 4. Run CLI
142
- ```bash
143
- python main.py
144
- ```
145
-
146
- ## Docker Deployment
147
-
148
- ### Build & Run
149
- ```bash
150
- docker build -t rag-project .
151
- docker run -d -p 8000:8000 --env-file .env rag-project
152
- ```
153
-
154
- ## How It Works
155
-
156
- ### Workflow Example
157
-
158
- **Query**: "What is the attention mechanism in transformers?"
159
-
160
- 1. **Retrieval**: FAISS finds top 3 most similar chunks from "Attention Is All You Need" paper
161
- 2. **Reranking**: FlashRank reorders by relevance (top 3 kept)
162
- 3. **Grading**: LLM evaluates each document:
163
- - ✅ Doc 1: Relevant (explains attention)
164
- - ✅ Doc 2: Relevant (shows formula)
165
- - ❌ Doc 3: Not relevant (talks about training data)
166
- 4. **Decision**: 2/3 relevant → Use filtered docs
167
- 5. **Generation**: Groq LLM synthesizes answer from relevant docs
168
- 6. **Output**: Comprehensive answer with LaTeX formulas (rendered via MathJax)
169
-
170
- ### When Retrieval Fails
171
-
172
- **Query**: "What are the latest improvements to transformers in 2024?"
173
-
174
- 1. **Retrieval**: Finds documents from 2017 paper
175
- 2. **Grading**: ❌ All documents marked "not relevant" (outdated info)
176
- 3. **Transform**: Rewrites query → "Recent transformer architecture improvements 2024"
177
- 4. **Web Search**: Tavily searches current web content
178
- 5. **Generation**: Answer combines paper fundamentals + recent developments
179
-
180
- ## Web Interface Features
181
-
182
- - **Modern UI**: Purple gradient design with responsive layout
183
- - **MathJax Integration**: Renders LaTeX formulas beautifully
184
- - **Transformer Visualization**: Architecture diagram in header
185
- - **Real-time Search**: Fast async FastAPI backend
186
- - **Error Handling**: Graceful degradation with user-friendly messages
 
1
+ ---
2
+ title: RAG Project - Learn with Transformers
3
+ emoji: 🤖
4
+ colorFrom: purple
5
+ colorTo: blue
6
+ sdk: docker
7
+ pinned: false
8
+ ---