Cheh Kit Hong commited on
Commit
04d4d26
Β·
1 Parent(s): c92c6b2

cleaning uncessary files

Browse files
README.md CHANGED
@@ -1,20 +1,288 @@
1
- rag_agent/
2
- β”œβ”€β”€ app.py # Main Gradio application entry point
3
- β”œβ”€β”€ config.py # Configuration hub (models, chunk sizes, providers)
4
- β”œβ”€β”€ util.py # PDF to markdown conversion
5
- β”œβ”€β”€ core/ # Core RAG components orchestration
6
- β”‚ β”œβ”€β”€ chat_interface.py
7
- β”‚ β”œβ”€β”€ document_manager.py
8
- β”‚ └── rag_system.py
9
- β”œβ”€β”€ knowledge_base/ # for create chromadb
10
- β”œβ”€β”€ chroma_data/ # chroma vectorstore data
11
- β”œβ”€β”€ agent_logic/ # LangGraph agent workflow
12
- β”‚ β”œβ”€β”€ edges.py # Conditional routing logic
13
- β”‚ β”œβ”€β”€ graph.py # Graph construction and compilation
14
- β”‚ β”œβ”€β”€ graph_state.py # State definitions
15
- β”‚ β”œβ”€β”€ nodes.py # Processing nodes (summarize, rewrite, agent)
16
- β”‚ β”œβ”€β”€ prompts.py # System prompts
17
- β”‚ β”œβ”€β”€ schemas.py # Pydantic data models
18
- β”‚ └── tools.py # Retrieval tools
19
- └── ui/ # User interface
20
- └── gradio_app.py # Gradio interface components
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: RAG Agent
3
+ emoji: πŸ•΅πŸ»β€β™‚οΈ
4
+ colorFrom: indigo
5
+ colorTo: indigo
6
+ sdk: gradio
7
+ sdk_version: 6.0.1
8
+ app_file: main.py
9
+ pinned: false
10
+ hf_oauth: true
11
+ hf_oauth_expiration_minutes: 480
12
+ ---
13
+
14
+ # πŸ“ Project Structure
15
+
16
+ ```
17
+ mai-rag-agent/
18
+ β”‚
19
+ β”œβ”€β”€ πŸ“‚ agent/ # Core agent logic
20
+ β”‚ β”œβ”€β”€ graph.py # LangGraph workflow definition
21
+ β”‚ β”œβ”€β”€ nodes.py # Agent nodes (router, vectordb, web_search, generate)
22
+ β”‚ β”œβ”€β”€ prompts.py # System prompts and templates
23
+ β”‚ β”œβ”€β”€ state.py # Agent state management (AgentState, RAG_method)
24
+ β”‚ └── tools.py # Tool definitions (Tavily, Wikipedia, ArXiv, ChromaDB)
25
+ β”‚
26
+ β”œβ”€β”€ πŸ“‚ core/ # Business logic layer
27
+ β”‚ β”œβ”€β”€ llm.py # LLM initialization (Anthropic Claude)
28
+ β”‚ └── rag_agent.py # Main RAGAgent class with graph orchestration
29
+ β”‚
30
+ β”œβ”€β”€ πŸ“‚ ui/ # User interface
31
+ β”‚ └── gradio_components.py # Gradio web interface components
32
+ β”‚
33
+ β”œβ”€β”€ πŸ“‚ knowledge_base/ # scripts for setting up Chroma
34
+ β”‚
35
+ β”œβ”€β”€ πŸ“‚ chroma_data/ # Artifacts for Chroma
36
+ β”‚
37
+ β”œβ”€β”€ πŸ“‚ docs/ # Source documents (PDFs, text files)
38
+ β”‚
39
+ β”œβ”€β”€ πŸ“„ main.py # Application entry point
40
+ β”œβ”€β”€ πŸ“„ config.py # Configuration settings
41
+ β”œβ”€β”€ πŸ“„ test_scripts.py # Agent testing script
42
+ β”‚
43
+ β”œβ”€β”€ πŸ“„ .env # Environment variables (API keys)
44
+ β”œβ”€β”€ πŸ“„ .gitignore # Git ignore rules
45
+ β”‚
46
+ β”œβ”€β”€ πŸ“„ requirements.txt # Python dependencies
47
+ β”œβ”€β”€ πŸ“„ pyproject.toml # Project metadata (if using uv)
48
+ β”‚
49
+ └── πŸ“„ README.md # Project documentation (this file)
50
+ ```
51
+
52
+ ## πŸ“‹ Key Components
53
+
54
+ ### πŸ€– Agent Module (`agent/`)
55
+ - **`graph.py`**: Defines the LangGraph workflow with conditional routing
56
+ - **`nodes.py`**: Implements agent nodes:
57
+ - `router_node`: Classifies queries (RAG/WEBSEARCH/GENERAL)
58
+ - `vectordb_node`: Retrieves from local ChromaDB
59
+ - `web_search_agent_node`: Executes web searches
60
+ - `generate_node`: Generates final responses
61
+ - **`state.py`**: Defines `AgentState` with message history, routing method, and context
62
+ - **`tools.py`**: Tool implementations for Tavily, Wikipedia, ArXiv, and ChromaDB
63
+ - **`prompts.py`**: System prompts for routing and generation
64
+
65
+ ### 🎯 Core Module (`core/`)
66
+ - **`llm.py`**: Initializes the LLM (Anthropic Claude Sonnet 4.5)
67
+ - **`rag_agent.py`**: Main `RAGAgent` class that orchestrates the graph
68
+
69
+ ### πŸ–₯️ UI Module (`ui/`)
70
+ - **`gradio_components.py`**: Gradio web interface with chat functionality
71
+
72
+ ### πŸ“Š Data Module (`data/`)
73
+ - **`documents/`**: Raw source documents for ingestion
74
+ - **`chroma_db/`**: Persisted vector embeddings
75
+
76
+ ### βš™οΈ Configuration
77
+ - **`config.py`**: Centralized configuration (model names, paths, API settings)
78
+ - **`.env`**: API keys (ANTHROPIC_API_KEY, TAVILY_API_KEY)
79
+
80
+ ### πŸš€ Entry Points
81
+ - **`main.py`**: Launches the Gradio UI
82
+ - **`test_scripts.py`**: Runs agent tests
83
+
84
+ ## πŸ”„ Data Flow
85
+
86
+ ```
87
+ User Query
88
+ ↓
89
+ [Router Node] β†’ Classifies intent (RAG/WEBSEARCH/GENERAL)
90
+ ↓
91
+ β”œβ”€β†’ [VectorDB Node] β†’ Retrieves from ChromaDB β†’ [Generate Node]
92
+ β”œβ”€β†’ [Web Search Agent] β†’ Calls Tavily/Wikipedia β†’ [Generate Node]
93
+ └─→ [Generate Node] β†’ Uses LLM knowledge only
94
+ ↓
95
+ Response to User
96
+ ```
97
+
98
+ ## πŸ› οΈ Technology Stack
99
+
100
+ - **LangChain**: Framework for LLM applications
101
+ - **LangGraph**: Workflow orchestration
102
+ - **Anthropic Claude**: LLM (Sonnet 4.5)
103
+ - **ChromaDB**: Vector database
104
+ - **Gradio**: Web UI framework
105
+ - **HuggingFace**: Embeddings model
106
+ - **Tavily**: Web search API
107
+ - **UV**: Python package manager
108
+
109
+
110
+ ## πŸš€ Quick Start with UV
111
+
112
+ ### Prerequisites
113
+ - Python 3.10+
114
+ - UV package manager ([Install UV](https://github.com/astral-sh/uv))
115
+ - API Keys: Anthropic, Tavily
116
+
117
+ ### 1️⃣ Clone the Repository
118
+ ```bash
119
+ git clone https://github.com/yourusername/mai-rag-agent.git
120
+ cd mai-rag-agent
121
+ ```
122
+
123
+ ### 2️⃣ Create Virtual Environment with UV
124
+ ```bash
125
+ # Create a new virtual environment
126
+ uv venv
127
+
128
+ # Activate the environment
129
+ source .venv/bin/activate # Linux/macOS
130
+ # or
131
+ .venv\Scripts\activate # Windows
132
+ ```
133
+
134
+ ### 3️⃣ Install Dependencies
135
+ ```bash
136
+ # Install all dependencies from requirements.txt
137
+ uv pip install -r requirements.txt
138
+
139
+ # Or install directly from pyproject.toml (if available)
140
+ uv pip install -e .
141
+ ```
142
+
143
+ ### 4️⃣ Set Up Environment Variables
144
+ ```bash
145
+ # Copy example environment file
146
+ cp .env.example .env
147
+
148
+ # Edit .env and add your API keys
149
+ nano .env # or use your preferred editor
150
+ ```
151
+
152
+ **Required environment variables:**
153
+ ```bash
154
+ ANTHROPIC_API_KEY=sk-ant-xxxxxxxxxxxxx
155
+ TAVILY_API_KEY=tvly-xxxxxxxxxxxxx
156
+ ```
157
+
158
+ ### 5️⃣ Prepare Data
159
+ ```bash
160
+ # Create necessary directories
161
+ mkdir -p data/documents data/chroma_db
162
+
163
+ # Add your documents to data/documents/
164
+ # Then run ingestion (if you have an ingestion script)
165
+ # python ingest_data.py
166
+ ```
167
+
168
+ ### 6️⃣ Run the Application
169
+ ```bash
170
+ # Launch the Gradio UI
171
+ python main.py
172
+ ```
173
+
174
+ The app will be available at: **http://127.0.0.1:7860**
175
+
176
+ ### 7️⃣ Run Tests (Optional)
177
+ ```bash
178
+ # Test the agent functionality
179
+ python test_scripts.py
180
+ ```
181
+
182
+ ---
183
+
184
+ ## 🐳 Quick Start with Dev Container (Alternative)
185
+
186
+ If you're using VS Code with Dev Containers:
187
+
188
+ ```bash
189
+ # 1. Open in VS Code
190
+ code .
191
+
192
+ # 2. Reopen in Container
193
+ # Command Palette (Ctrl+Shift+P) β†’ "Dev Containers: Reopen in Container"
194
+
195
+ # 3. Inside container, install dependencies
196
+ uv pip install -r requirements.txt
197
+
198
+ # 4. Set up .env file
199
+ cp .env.example .env
200
+ # Edit .env with your API keys
201
+
202
+ # 5. Run the app
203
+ python main.py
204
+ ```
205
+
206
+ ---
207
+
208
+ ## πŸ“¦ UV-Specific Commands
209
+
210
+ ```bash
211
+ # Update all dependencies
212
+ uv pip install --upgrade -r requirements.txt
213
+
214
+ # List installed packages
215
+ uv pip list
216
+
217
+ # Freeze current environment
218
+ uv pip freeze > requirements.txt
219
+
220
+ # Install a new package
221
+ uv pip install package-name
222
+
223
+ # Uninstall a package
224
+ uv pip uninstall package-name
225
+
226
+ # Sync environment (removes unused packages)
227
+ uv pip sync requirements.txt
228
+ ```
229
+
230
+ ---
231
+
232
+ ## πŸ”§ Troubleshooting
233
+
234
+ ### Issue: `uv` command not found
235
+ ```bash
236
+ # Install UV
237
+ curl -LsSf https://astral.sh/uv/install.sh | sh
238
+
239
+ # Add to PATH (if needed)
240
+ export PATH="$HOME/.cargo/bin:$PATH"
241
+ ```
242
+
243
+ ### Issue: API key not loading
244
+ ```bash
245
+ # Check if .env exists
246
+ cat .env | grep -i api
247
+
248
+ # Ensure no typos in variable names
249
+ # Should be: ANTHROPIC_API_KEY and TAVILY_API_KEY
250
+ ```
251
+
252
+ ### Issue: ChromaDB not found
253
+ ```bash
254
+ # Ensure data directories exist
255
+ mkdir -p data/chroma_db
256
+
257
+ # Check permissions
258
+ chmod -R 755 data/
259
+ ```
260
+
261
+ ### Issue: Port 7860 already in use
262
+ ```bash
263
+ # Find and kill the process
264
+ lsof -ti:7860 | xargs kill -9
265
+
266
+ # Or use a different port in main.py
267
+ # demo.launch(server_port=7861)
268
+ ```
269
+
270
+ ---
271
+
272
+ ## 🎯 Next Steps
273
+
274
+ 1. βœ… Add your documents to `data/documents/`
275
+ 2. βœ… Configure embeddings model in `config.py`
276
+ 3. βœ… Customize prompts in `agent/prompts.py`
277
+ 4. βœ… Test with sample queries in the Gradio UI
278
+ 5. βœ… Deploy to production (see deployment docs)
279
+
280
+ ---
281
+
282
+ ## πŸ“š Additional Resources
283
+
284
+ - [UV Documentation](https://github.com/astral-sh/uv)
285
+ - [LangGraph Docs](https://langchain-ai.github.io/langgraph/)
286
+ - [Anthropic API](https://docs.anthropic.com/)
287
+ - [Tavily API](https://docs.tavily.com/)
288
+ - [ChromaDB Docs](https://docs.trychroma.com/)
core/chat_interface.py DELETED
@@ -1,196 +0,0 @@
1
- import gradio as gr
2
- from core.rag_agent import RAGAgent
3
- from core.document_manager import DocumentManager
4
- import os
5
-
6
- # Initialize components
7
- doc_manager = DocumentManager()
8
- rag_agent = None
9
-
10
- def initialize_agent():
11
- """Initialize RAG agent lazily"""
12
- global rag_agent
13
- if rag_agent is None:
14
- rag_agent = RAGAgent()
15
- return rag_agent
16
-
17
- def upload_files(files):
18
- """Handle file uploads"""
19
- if not files:
20
- return "No files selected", get_file_list()
21
-
22
- results = []
23
- for file in files:
24
- try:
25
- result = doc_manager.add_document(file.name)
26
- results.append(result)
27
- except Exception as e:
28
- results.append(f"Error processing {os.path.basename(file.name)}: {str(e)}")
29
-
30
- return "\n".join(results), get_file_list()
31
-
32
- def get_file_list():
33
- """Get list of documents in the knowledge base"""
34
- try:
35
- files = doc_manager.list_documents()
36
- if not files:
37
- return "No documents in knowledge base"
38
- return "\n".join([f"β€’ {f}" for f in files])
39
- except Exception as e:
40
- return f"Error listing files: {str(e)}"
41
-
42
- def clear_database():
43
- """Clear all documents from the knowledge base"""
44
- try:
45
- result = doc_manager.clear_all()
46
- return result, get_file_list()
47
- except Exception as e:
48
- return f"Error clearing database: {str(e)}", get_file_list()
49
-
50
- def chat_with_agent(message, history):
51
- """Handle chat interactions with the RAG agent"""
52
- if not message.strip():
53
- return history
54
-
55
- try:
56
- agent = initialize_agent()
57
-
58
- # Stream the agent's response
59
- response_text = ""
60
- for event in agent.agent_graph.stream(
61
- {"messages": [("user", message)]},
62
- agent.get_config(),
63
- stream_mode="values"
64
- ):
65
- if "messages" in event and len(event["messages"]) > 0:
66
- last_message = event["messages"][-1]
67
- if hasattr(last_message, "content"):
68
- response_text = last_message.content
69
-
70
- if not response_text:
71
- response_text = "I apologize, but I couldn't generate a response. Please try again."
72
-
73
- return response_text
74
-
75
- except Exception as e:
76
- return f"Error: {str(e)}"
77
-
78
- def reset_conversation():
79
- """Reset the conversation thread"""
80
- global rag_agent
81
- if rag_agent:
82
- rag_agent.reset_thread()
83
- return None # Clear chat history
84
-
85
- def create_gradio_ui():
86
- """Create the complete Gradio interface"""
87
-
88
- with gr.Blocks(title="RAG Agent with Agentic Memory", theme=gr.themes.Soft()) as demo:
89
- gr.Markdown("""
90
- # πŸ€– RAG Agent with Agentic Memory
91
-
92
- Upload documents and chat with an intelligent agent that uses:
93
- - πŸ“š **Local Knowledge Base** (ChromaDB)
94
- - πŸ” **Web Search** (Tavily)
95
- - πŸ“– **Wikipedia**
96
- - πŸŽ“ **ArXiv** (Academic Papers)
97
- """)
98
-
99
- with gr.Tabs():
100
- # Documents Tab
101
- with gr.Tab("πŸ“„ Documents"):
102
- gr.Markdown("### Upload and Manage Documents")
103
- gr.Markdown("Upload PDF or Markdown files to add them to the knowledge base.")
104
-
105
- with gr.Row():
106
- with gr.Column(scale=2):
107
- file_upload = gr.File(
108
- label="Upload Documents",
109
- file_count="multiple",
110
- file_types=[".pdf", ".md"]
111
- )
112
- upload_btn = gr.Button("πŸ“€ Add to Knowledge Base", variant="primary")
113
- upload_status = gr.Textbox(label="Upload Status", lines=3)
114
-
115
- with gr.Column(scale=1):
116
- file_list = gr.Textbox(
117
- label="Documents in Knowledge Base",
118
- lines=10,
119
- value=get_file_list()
120
- )
121
- refresh_btn = gr.Button("πŸ”„ Refresh List")
122
- clear_btn = gr.Button("πŸ—‘οΈ Clear All Documents", variant="stop")
123
-
124
- # Connect document management buttons
125
- upload_btn.click(
126
- fn=upload_files,
127
- inputs=[file_upload],
128
- outputs=[upload_status, file_list]
129
- )
130
-
131
- refresh_btn.click(
132
- fn=get_file_list,
133
- outputs=[file_list]
134
- )
135
-
136
- clear_btn.click(
137
- fn=clear_database,
138
- outputs=[upload_status, file_list]
139
- )
140
-
141
- # Chat Tab
142
- with gr.Tab("πŸ’¬ Chat"):
143
- gr.Markdown("### Chat with Your Documents")
144
- gr.Markdown("Ask questions about your documents or any topic. The agent will search multiple sources.")
145
-
146
- chatbot = gr.Chatbot(
147
- label="Conversation",
148
- height=500,
149
- show_label=True,
150
- avatar_images=(None, "πŸ€–")
151
- )
152
-
153
- with gr.Row():
154
- msg = gr.Textbox(
155
- label="Your Message",
156
- placeholder="Ask me anything about your documents or general knowledge...",
157
- scale=4
158
- )
159
- submit_btn = gr.Button("Send", variant="primary", scale=1)
160
-
161
- with gr.Row():
162
- clear_chat_btn = gr.Button("πŸ”„ Reset Conversation")
163
- gr.Markdown("*Note: Resetting clears the conversation history*")
164
-
165
- # Chat interface
166
- chat_interface = gr.ChatInterface(
167
- fn=chat_with_agent,
168
- chatbot=chatbot,
169
- textbox=msg,
170
- submit_btn=submit_btn,
171
- retry_btn=None,
172
- undo_btn=None,
173
- clear_btn=None
174
- )
175
-
176
- clear_chat_btn.click(
177
- fn=reset_conversation,
178
- outputs=[chatbot]
179
- )
180
-
181
- gr.Markdown("""
182
- ---
183
- ### πŸ”§ How it works:
184
- 1. **Upload documents** in the Documents tab
185
- 2. **Ask questions** in the Chat tab
186
- 3. The agent will:
187
- - Analyze your query
188
- - Search relevant sources
189
- - Provide comprehensive answers with citations
190
- """)
191
-
192
- return demo
193
-
194
- if __name__ == "__main__":
195
- demo = create_gradio_ui()
196
- demo.launch(share=False, server_name="127.0.0.1", server_port=7860)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
knowledge_base/chroma.py CHANGED
@@ -9,13 +9,12 @@ from langchain_chroma import Chroma
9
  from config import configs
10
 
11
  if __name__ == "__main__":
12
- # --- 1. Load Documents ---
13
  print("Loading documents from directory...")
14
  loader = DirectoryLoader(
15
  path=configs["DATA_PATH"],
16
  glob="*.md",
17
  loader_cls=TextLoader,
18
- silent_errors=True # Set to False if you want to see loader errors
19
  )
20
 
21
  raw_documents = loader.load()
@@ -23,25 +22,22 @@ if __name__ == "__main__":
23
  print(f"Error: No documents found in {configs['DATA_PATH']}. Check your path and file types.")
24
  exit()
25
 
26
- # --- 2. Split Documents into Chunks ---
27
  print(f"Loaded {len(raw_documents)} raw documents. Splitting into chunks...")
28
- # Recursive splitting is better than simple splitting, preserving context.
29
  text_splitter = RecursiveCharacterTextSplitter(
30
  chunk_size=1000,
31
  chunk_overlap=200,
32
- separators=["\n\n", "\n", " ", ""] # Optimal separators for markdown/text
33
  )
34
 
35
  documents_to_embed = text_splitter.split_documents(raw_documents)
36
  print(f"Split into {len(documents_to_embed)} chunks.")
37
 
38
- # --- 3. Define Custom Embedding Model ---
39
  print(f"Initializing custom embedding model: {configs['EMBEDDING_MODEL_NAME']}...")
40
  dense_embeddings = HuggingFaceEmbeddings(
41
  model_name=configs["EMBEDDING_MODEL_NAME"]
42
  )
43
 
44
- # --- 4. Create and Persist the Vector Store ---
45
  print(f"Creating Chroma vector store and persisting data to {configs['PERSIST_PATH']}...")
46
  vectorstore = Chroma.from_documents(
47
  documents=documents_to_embed, # The prepared Document chunks
 
9
  from config import configs
10
 
11
  if __name__ == "__main__":
 
12
  print("Loading documents from directory...")
13
  loader = DirectoryLoader(
14
  path=configs["DATA_PATH"],
15
  glob="*.md",
16
  loader_cls=TextLoader,
17
+ silent_errors=True
18
  )
19
 
20
  raw_documents = loader.load()
 
22
  print(f"Error: No documents found in {configs['DATA_PATH']}. Check your path and file types.")
23
  exit()
24
 
25
+ # Split Documents into Chunks
26
  print(f"Loaded {len(raw_documents)} raw documents. Splitting into chunks...")
 
27
  text_splitter = RecursiveCharacterTextSplitter(
28
  chunk_size=1000,
29
  chunk_overlap=200,
30
+ separators=["\n\n", "\n", " ", ""]
31
  )
32
 
33
  documents_to_embed = text_splitter.split_documents(raw_documents)
34
  print(f"Split into {len(documents_to_embed)} chunks.")
35
 
 
36
  print(f"Initializing custom embedding model: {configs['EMBEDDING_MODEL_NAME']}...")
37
  dense_embeddings = HuggingFaceEmbeddings(
38
  model_name=configs["EMBEDDING_MODEL_NAME"]
39
  )
40
 
 
41
  print(f"Creating Chroma vector store and persisting data to {configs['PERSIST_PATH']}...")
42
  vectorstore = Chroma.from_documents(
43
  documents=documents_to_embed, # The prepared Document chunks
knowledge_base/test_retrieval.py CHANGED
@@ -1,17 +1,14 @@
1
  from langchain_community.embeddings import HuggingFaceEmbeddings
2
  from langchain_chroma import Chroma
3
 
4
- # Configuration must match the creation step
5
  PERSIST_PATH = "./knowledge_base/chroma_data"
6
  EMBEDDING_MODEL_NAME = "sentence-transformers/all-mpnet-base-v2"
7
  COLLECTION_NAME = "langchain_mpnet_collection"
8
 
9
- # 1. Define the custom embedding object (Crucial for query vectorization)
10
  dense_embeddings = HuggingFaceEmbeddings(
11
  model_name=EMBEDDING_MODEL_NAME
12
  )
13
 
14
- # 2. Load the existing vector store from disk
15
  try:
16
  vectorstore = Chroma(
17
  persist_directory=PERSIST_PATH,
@@ -25,8 +22,7 @@ except Exception as e:
25
 
26
  query = "Tell me about SAM3 general architecture."
27
 
28
- # Perform the search
29
- # k=3 means it will return the top 3 most similar document chunks
30
  retrieved_docs = vectorstore.similarity_search(query, k=3)
31
 
32
  print(f"\n--- Search Results for: '{query}' ---")
 
1
  from langchain_community.embeddings import HuggingFaceEmbeddings
2
  from langchain_chroma import Chroma
3
 
 
4
  PERSIST_PATH = "./knowledge_base/chroma_data"
5
  EMBEDDING_MODEL_NAME = "sentence-transformers/all-mpnet-base-v2"
6
  COLLECTION_NAME = "langchain_mpnet_collection"
7
 
 
8
  dense_embeddings = HuggingFaceEmbeddings(
9
  model_name=EMBEDDING_MODEL_NAME
10
  )
11
 
 
12
  try:
13
  vectorstore = Chroma(
14
  persist_directory=PERSIST_PATH,
 
22
 
23
  query = "Tell me about SAM3 general architecture."
24
 
25
+
 
26
  retrieved_docs = vectorstore.similarity_search(query, k=3)
27
 
28
  print(f"\n--- Search Results for: '{query}' ---")
notebook.ipynb DELETED
@@ -1,37 +0,0 @@
1
- {
2
- "cells": [
3
- {
4
- "cell_type": "code",
5
- "execution_count": null,
6
- "id": "524b8568",
7
- "metadata": {},
8
- "outputs": [],
9
- "source": [
10
- "from langchain_community.document_loaders.text import DirectoryLoader, TextLoader"
11
- ]
12
- },
13
- {
14
- "cell_type": "code",
15
- "execution_count": null,
16
- "id": "babc2558",
17
- "metadata": {},
18
- "outputs": [],
19
- "source": [
20
- "print(\"Y\")"
21
- ]
22
- }
23
- ],
24
- "metadata": {
25
- "kernelspec": {
26
- "display_name": "rag_agent",
27
- "language": "python",
28
- "name": "python3"
29
- },
30
- "language_info": {
31
- "name": "python",
32
- "version": "3.10.17"
33
- }
34
- },
35
- "nbformat": 4,
36
- "nbformat_minor": 5
37
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
quick_check.py DELETED
@@ -1,16 +0,0 @@
1
- import gradio as gr
2
-
3
- print("Gradio Version:", gr.__version__)
4
-
5
- import gradio as gr
6
-
7
- history = [
8
- gr.ChatMessage(role="assistant", content="How can I help you?"),
9
- gr.ChatMessage(role="user", content="Can you make me a plot of quarterly sales?"),
10
- gr.ChatMessage(role="assistant", content="I am happy to provide you that report and plot.")
11
- ]
12
-
13
- with gr.Blocks() as demo:
14
- gr.Chatbot(history)
15
-
16
- demo.launch()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
requirements.txt CHANGED
@@ -1,20 +1,184 @@
1
- langchain
2
- langgraph
3
- langchain-huggingface
4
- langchain-google-genai
5
- langchain-chroma
6
- fastapi
7
- uvicorn
8
- pydantic
9
- chromadb
10
- pymupdf
11
- pymupdf4llm
12
- langchain-community
13
- langchain_text_splitters
14
- pymupdf-layout
15
- sentence_transformers
16
- gradio
17
- python-dotenv
18
- langchain-tavily
19
- arxiv
20
- wikipedia
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ aiofiles==24.1.0
2
+ aiohappyeyeballs==2.6.1
3
+ aiohttp==3.13.2
4
+ aiosignal==1.4.0
5
+ annotated-doc==0.0.4
6
+ annotated-types==0.7.0
7
+ anyio==4.12.0
8
+ arxiv==2.3.1
9
+ async-timeout==5.0.1
10
+ attrs==25.4.0
11
+ backoff==2.2.1
12
+ bcrypt==5.0.0
13
+ beautifulsoup4==4.14.2
14
+ brotli==1.2.0
15
+ build==1.3.0
16
+ cachetools==6.2.2
17
+ certifi==2025.11.12
18
+ charset-normalizer==3.4.4
19
+ chromadb==1.3.5
20
+ click==8.3.1
21
+ coloredlogs==15.0.1
22
+ dataclasses-json==0.6.7
23
+ distro==1.9.0
24
+ durationpy==0.10
25
+ exceptiongroup==1.3.1
26
+ fastapi==0.122.0
27
+ feedparser==6.0.12
28
+ ffmpy==1.0.0
29
+ filelock==3.20.0
30
+ filetype==1.2.0
31
+ flatbuffers==25.9.23
32
+ frozenlist==1.8.0
33
+ fsspec==2025.10.0
34
+ google-ai-generativelanguage==0.9.0
35
+ google-api-core==2.28.1
36
+ google-auth==2.43.0
37
+ googleapis-common-protos==1.72.0
38
+ gradio==6.0.1
39
+ gradio-client==2.0.0
40
+ greenlet==3.2.4
41
+ groovy==0.1.2
42
+ grpcio==1.76.0
43
+ grpcio-status==1.76.0
44
+ h11==0.16.0
45
+ hf-xet==1.2.0
46
+ httpcore==1.0.9
47
+ httptools==0.7.1
48
+ httpx==0.28.1
49
+ httpx-sse==0.4.3
50
+ huggingface-hub==0.36.0
51
+ humanfriendly==10.0
52
+ idna==3.11
53
+ importlib-metadata==8.7.0
54
+ importlib-resources==6.5.2
55
+ jinja2==3.1.6
56
+ joblib==1.5.2
57
+ jsonpatch==1.33
58
+ jsonpointer==3.0.0
59
+ jsonschema==4.25.1
60
+ jsonschema-specifications==2025.9.1
61
+ kubernetes==34.1.0
62
+ langchain==1.1.0
63
+ langchain-chroma==1.0.0
64
+ langchain-classic==1.0.0
65
+ langchain-community==0.4.1
66
+ langchain-core==1.1.0
67
+ langchain-google-genai==3.2.0
68
+ langchain-huggingface==1.1.0
69
+ langchain-tavily==0.2.13
70
+ langchain-text-splitters==1.0.0
71
+ langgraph==1.0.4
72
+ langgraph-checkpoint==3.0.1
73
+ langgraph-prebuilt==1.0.5
74
+ langgraph-sdk==0.2.10
75
+ langsmith==0.4.49
76
+ markdown-it-py==4.0.0
77
+ markupsafe==3.0.3
78
+ marshmallow==3.26.1
79
+ mdurl==0.1.2
80
+ mmh3==5.2.0
81
+ mpmath==1.3.0
82
+ multidict==6.7.0
83
+ mypy-extensions==1.1.0
84
+ networkx==3.4.2
85
+ numpy==2.2.6
86
+ nvidia-cublas-cu12==12.8.4.1
87
+ nvidia-cuda-cupti-cu12==12.8.90
88
+ nvidia-cuda-nvrtc-cu12==12.8.93
89
+ nvidia-cuda-runtime-cu12==12.8.90
90
+ nvidia-cudnn-cu12==9.10.2.21
91
+ nvidia-cufft-cu12==11.3.3.83
92
+ nvidia-cufile-cu12==1.13.1.3
93
+ nvidia-curand-cu12==10.3.9.90
94
+ nvidia-cusolver-cu12==11.7.3.90
95
+ nvidia-cusparse-cu12==12.5.8.93
96
+ nvidia-cusparselt-cu12==0.7.1
97
+ nvidia-nccl-cu12==2.27.5
98
+ nvidia-nvjitlink-cu12==12.8.93
99
+ nvidia-nvshmem-cu12==3.3.20
100
+ nvidia-nvtx-cu12==12.8.90
101
+ oauthlib==3.3.1
102
+ onnxruntime==1.23.2
103
+ opentelemetry-api==1.38.0
104
+ opentelemetry-exporter-otlp-proto-common==1.38.0
105
+ opentelemetry-exporter-otlp-proto-grpc==1.38.0
106
+ opentelemetry-proto==1.38.0
107
+ opentelemetry-sdk==1.38.0
108
+ opentelemetry-semantic-conventions==0.59b0
109
+ orjson==3.11.4
110
+ ormsgpack==1.12.0
111
+ overrides==7.7.0
112
+ packaging==25.0
113
+ pandas==2.3.3
114
+ pillow==12.0.0
115
+ posthog==5.4.0
116
+ propcache==0.4.1
117
+ proto-plus==1.26.1
118
+ protobuf==6.33.1
119
+ pyasn1==0.6.1
120
+ pyasn1-modules==0.4.2
121
+ pybase64==1.4.2
122
+ pydantic==2.12.5
123
+ pydantic-core==2.41.5
124
+ pydantic-settings==2.12.0
125
+ pydub==0.25.1
126
+ pygments==2.19.2
127
+ pymupdf==1.26.6
128
+ pymupdf-layout==1.26.6
129
+ pymupdf4llm==0.2.4
130
+ pypika==0.48.9
131
+ pyproject-hooks==1.2.0
132
+ python-dateutil==2.9.0.post0
133
+ python-dotenv==1.2.1
134
+ python-multipart==0.0.20
135
+ pytz==2025.2
136
+ pyyaml==6.0.3
137
+ referencing==0.37.0
138
+ regex==2025.11.3
139
+ requests==2.32.5
140
+ requests-oauthlib==2.0.0
141
+ requests-toolbelt==1.0.0
142
+ rich==14.2.0
143
+ rpds-py==0.29.0
144
+ rsa==4.9.1
145
+ safehttpx==0.1.7
146
+ safetensors==0.7.0
147
+ scikit-learn==1.7.2
148
+ scipy==1.15.3
149
+ semantic-version==2.10.0
150
+ sentence-transformers==5.1.2
151
+ sgmllib3k==1.0.0
152
+ shellingham==1.5.4
153
+ six==1.17.0
154
+ sniffio==1.3.1
155
+ soupsieve==2.8
156
+ sqlalchemy==2.0.44
157
+ starlette==0.50.0
158
+ sympy==1.14.0
159
+ tabulate==0.9.0
160
+ tenacity==9.1.2
161
+ threadpoolctl==3.6.0
162
+ tokenizers==0.22.1
163
+ tomli==2.3.0
164
+ tomlkit==0.13.3
165
+ torch==2.9.1
166
+ tqdm==4.67.1
167
+ transformers==4.57.3
168
+ triton==3.5.1
169
+ typer==0.20.0
170
+ typing-extensions==4.15.0
171
+ typing-inspect==0.9.0
172
+ typing-inspection==0.4.2
173
+ tzdata==2025.2
174
+ urllib3==2.5.0
175
+ uvicorn==0.38.0
176
+ uvloop==0.22.1
177
+ watchfiles==1.1.1
178
+ websocket-client==1.9.0
179
+ websockets==15.0.1
180
+ wikipedia==1.4.0
181
+ xxhash==3.6.0
182
+ yarl==1.22.0
183
+ zipp==3.23.0
184
+ zstandard==0.25.0
testing_main.py DELETED
@@ -1,11 +0,0 @@
1
- from config import configs
2
- from knowledge_base.test_retrieval import PERSIST_PATH, EMBEDDING_MODEL_NAME, COLLECTION_NAME
3
-
4
- if __name__ == "__main__":
5
- print("Testing configuration values...")
6
- for key, value in configs.items():
7
- print(f"{key}: {value}")
8
- print("βœ… Configuration test completed successfully.")
9
- print(f"PERSIST_PATH: {PERSIST_PATH}")
10
- print(f"EMBEDDING_MODEL_NAME: {EMBEDDING_MODEL_NAME}")
11
- print(f"COLLECTION_NAME: {COLLECTION_NAME}")