initial app

DEPLOYMENT.md

@@ -0,0 +1,214 @@
+# Deploying to Hugging Face Spaces
+
+This guide will walk you through deploying your RAG chatbot to Hugging Face Spaces.
+
+## Prerequisites
+
+1. **Hugging Face Account**: Sign up at https://huggingface.co/join
+2. **Access Token**: Get your token from https://huggingface.co/settings/tokens
+
+## Step-by-Step Deployment
+
+### Method 1: Using the Web Interface (Easiest)
+
+1. **Create a New Space**:
+   - Go to https://huggingface.co/new-space
+   - Fill in:
+     - **Space name**: Choose a name (e.g., `my-rag-chatbot`)
+     - **SDK**: Select **Gradio**
+     - **Hardware**: Choose based on your needs:
+       - **CPU basic**: Free, good for testing
+       - **CPU upgrade**: Better performance
+       - **GPU**: If you need faster model inference
+     - **Visibility**: Public or Private
+   - Click **Create Space**
+
+2. **Upload Your Files**:
+   - In your new Space, click the **Files and versions** tab
+   - Click **Add file** → **Upload files**
+   - Upload these files:
+     - `app.py`
+     - `ingestion.py`
+     - `requirements.txt`
+     - `README.md` (optional but recommended)
+     - `pdfs/` folder (if you want to include sample PDFs)
+     - `ingest_documents.py` (optional, for manual ingestion)
+
+3. **Important Notes**:
+   - **Vector Store**: The `vector_store/` folder is in `.gitignore` and won't be uploaded. You have two options:
+     - **Option A**: Run `ingest_documents.py` on the Space after deployment (via the Space's terminal)
+     - **Option B**: Upload the vector store files manually if they're not too large
+   - **PDFs**: If your PDFs are large (>50MB), consider hosting them elsewhere or using Hugging Face Datasets
+
+4. **Wait for Build**: Hugging Face will automatically:
+   - Install dependencies from `requirements.txt`
+   - Start your Gradio app
+   - Your Space will be live at: `https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME`
+
+### Method 2: Using Git (Recommended for Updates)
+
+1. **Install Git** (if not already installed):
+   - Windows: Download from https://git-scm.com/download/win
+   - Or use Git that comes with GitHub Desktop
+
+2. **Install Hugging Face CLI**:
+   ```bash
+   py -m pip install huggingface_hub
+   ```
+
+3. **Login to Hugging Face**:
+   ```bash
+   huggingface-cli login
+   ```
+   Enter your access token when prompted.
+
+4. **Create a New Space** (via web interface):
+   - Go to https://huggingface.co/new-space
+   - Create the space with Gradio SDK
+   - Note your space name (e.g., `YOUR_USERNAME/my-rag-chatbot`)
+
+5. **Initialize Git in Your Project**:
+   ```bash
+   cd C:\Users\DanielSimeone\Desktop\testing-hugging-face
+   git init
+   git add app.py ingestion.py requirements.txt README.md ingest_documents.py pdfs/
+   git commit -m "Initial commit for Hugging Face Space"
+   ```
+
+6. **Add Hugging Face Remote and Push**:
+   ```bash
+   git remote add origin https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
+   git push -u origin main
+   ```
+
+7. **For Future Updates**:
+   ```bash
+   git add .
+   git commit -m "Update description"
+   git push
+   ```
+
+## Post-Deployment Setup
+
+### Setting Up the Vector Store on Hugging Face Spaces
+
+Since the vector store isn't included in the repository, you need to create it on the Space:
+
+1. **Option A: Use the Space Terminal** (if available):
+   - Go to your Space → **Settings** → Enable **Embedded Gradio SDK**
+   - Or use the Space's built-in terminal/console
+   - Run: `python ingest_documents.py`
+
+2. **Option B: Upload Vector Store Files**:
+   - If your vector store files are small enough:
+     - Upload `vector_store/index.faiss`
+     - Upload `vector_store/documents.pkl`
+     - Upload `vector_store/embeddings.pkl`
+   - The app will automatically load them on startup
+
+3. **Option C: Pre-build in a Script**:
+   - Create a `setup.py` or modify the Space to run ingestion on first launch
+   - This is more complex but ensures the vector store is always ready
+
+## Required Files for Deployment
+
+Your Space needs these files:
+
+- ✅ `app.py` - Main Gradio application
+- ✅ `ingestion.py` - Document ingestion module
+- ✅ `requirements.txt` - Python dependencies
+- ✅ `README.md` - Documentation (optional but recommended)
+- ⚠️ `vector_store/` - Will be created on the Space
+- ⚠️ `pdfs/` - Optional, include if you want sample PDFs
+
+## Configuration for Hugging Face Spaces
+
+### Update app.py for Spaces
+
+The current `app.py` should work, but you might want to adjust:
+
+1. **Port**: Hugging Face Spaces uses port 7860 automatically
+2. **Server name**: Use `0.0.0.0` (already set)
+3. **Share**: Set to `False` (already set)
+
+Your current launch code is fine:
+```python
+app.launch(
+    share=False,
+    server_name="0.0.0.0",
+    server_port=7861,  # Note: Spaces uses 7860, but this should auto-adjust
+    theme=MinimalistTheme()
+)
+```
+
+### Optional: Add README Frontmatter
+
+Add this to the top of your `README.md` for better Space display:
+
+```yaml
+---
+title: RAG Chatbot
+emoji: 🤖
+colorFrom: blue
+colorTo: purple
+sdk: gradio
+sdk_version: 6.3.0
+app_file: app.py
+pinned: false
+---
+```
+
+## Troubleshooting
+
+### Build Fails
+
+- Check that all dependencies in `requirements.txt` are correct
+- Ensure Python version compatibility (Spaces uses Python 3.10 by default)
+- Check the build logs in your Space's **Logs** tab
+
+### Vector Store Not Loading
+
+- Verify the vector store files are in the correct location
+- Check file permissions
+- Ensure the path in `app.py` is correct (should be `./vector_store`)
+
+### Model Loading Issues
+
+- Large models may take time to download on first run
+- Consider using smaller models for faster startup
+- Check available disk space in your Space
+
+### Memory Issues
+
+- If you get out-of-memory errors, consider:
+  - Using a smaller embedding model
+  - Reducing chunk size in `ingestion.py`
+  - Upgrading to a Space with more memory
+
+## Updating Your Space
+
+After making changes locally:
+
+```bash
+git add .
+git commit -m "Description of changes"
+git push
+```
+
+Hugging Face will automatically rebuild your Space.
+
+## Sharing Your Space
+
+Once deployed, your Space will be available at:
+```
+https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
+```
+
+You can share this URL with others!
+
+## Next Steps
+
+- Add more PDFs to the `pdfs/` folder
+- Update URLs in `ingest_documents.py`
+- Customize the theme further
+- Add more features to the chatbot

README.md

@@ -1,13 +1,196 @@
 ---
-title: Testing
-emoji: 🏃
-colorFrom: purple
-colorTo: red
+title: RAG Chatbot
+emoji: 🤖
+colorFrom: blue
+colorTo: purple
 sdk: gradio
 sdk_version: 6.3.0
 app_file: app.py
 pinned: false
-short_description: testing
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Hugging Face RAG Chatbot
+
+A Retrieval-Augmented Generation (RAG) chatbot that can ingest PDFs and URLs, then answer questions based on the ingested documents. Built for deployment on Hugging Face Spaces.
+
+## Features
+
+- 📄 **PDF Ingestion**: Upload and process PDF documents
+- 🌐 **URL Ingestion**: Extract and process content from web URLs
+- 🔍 **Vector Search**: Semantic search using sentence transformers
+- 💬 **Chatbot Interface**: Interactive Gradio interface for querying documents
+- 🚀 **Hugging Face Ready**: Configured for easy deployment to Hugging Face Spaces
+
+## Setup
+
+### Local Development
+
+1. **Install dependencies:**
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+2. **Run the application:**
+   ```bash
+   python app.py
+   ```
+
+3. **Access the interface:**
+   - Open your browser to `http://localhost:7860`
+
+### Usage
+
+1. **Ingest Documents (Run this first or periodically to update):**
+   - Add PDF files to the `pdfs/` folder
+   - Edit `ingest_documents.py` and add your URLs to the `URLS` list
+   - Run the ingestion script:
+     ```bash
+     py ingest_documents.py
+     ```
+   - Wait for processing to complete (this creates/updates the vector store)
+
+2. **Chat with Documents:**
+   - Run the chatbot app:
+     ```bash
+     py app.py
+     ```
+   - Open your browser to `http://localhost:7860`
+   - Toggle "Use RAG" to enable/disable document retrieval
+   - Ask questions about your ingested documents
+   - The chatbot will retrieve relevant context and generate answers
+
+## Deployment to Hugging Face Spaces
+
+### Option 1: Using Hugging Face CLI
+
+1. **Install Hugging Face CLI:**
+   ```bash
+   pip install huggingface_hub
+   ```
+
+2. **Login to Hugging Face:**
+   ```bash
+   huggingface-cli login
+   ```
+
+3. **Create a new Space:**
+   - Go to https://huggingface.co/new-space
+   - Choose a name and select "Gradio" as the SDK
+   - Create the space
+
+4. **Clone and push your code:**
+   ```bash
+   git clone https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
+   cd YOUR_SPACE_NAME
+   # Copy your files here
+   git add .
+   git commit -m "Initial commit"
+   git push
+   ```
+
+### Option 2: Using Git
+
+1. **Initialize git repository:**
+   ```bash
+   git init
+   git add .
+   git commit -m "Initial commit"
+   ```
+
+2. **Add Hugging Face remote:**
+   ```bash
+   git remote add origin https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
+   git push -u origin main
+   ```
+
+### Required Files for Hugging Face Spaces
+
+Your Space needs these files:
+- `app.py` - Main Gradio application
+- `requirements.txt` - Python dependencies
+- `README.md` - This file (optional but recommended)
+
+### Optional: Add app.py to README
+
+For Hugging Face Spaces, you can also add a `app.py` reference in your README:
+
+```yaml
+---
+title: RAG Chatbot
+emoji: 🤖
+colorFrom: blue
+colorTo: purple
+sdk: gradio
+sdk_version: 4.0.0
+app_file: app.py
+pinned: false
+---
+```
+
+## Configuration
+
+### Changing the Chatbot Model
+
+Edit `app.py` and change the `model_name` parameter in `RAGChatbot`:
+
+```python
+chatbot = RAGChatbot(model_name="your-preferred-model")
+```
+
+Popular options:
+- `microsoft/DialoGPT-medium` (default)
+- `gpt2`
+- `facebook/blenderbot-400M-distill`
+- `microsoft/DialoGPT-large`
+
+### Changing the Embedding Model
+
+Edit the `embedding_model` parameter:
+
+```python
+chatbot = RAGChatbot(embedding_model="sentence-transformers/all-mpnet-base-v2")
+```
+
+## Project Structure
+
+```
+.
+├── app.py                  # Main Gradio chatbot application
+├── ingest_documents.py     # Standalone script to ingest PDFs and URLs
+├── ingestion.py            # Document ingestion and vector store module
+├── requirements.txt        # Python dependencies
+├── README.md              # This file
+├── pdfs/                  # Folder for PDF files (add your PDFs here)
+│   └── README.md
+└── vector_store/          # Saved vector store (created after ingestion)
+    ├── index.faiss
+    ├── documents.pkl
+    └── embeddings.pkl
+```
+
+## Limitations
+
+- Vector store is stored locally (not persistent on Hugging Face Spaces by default)
+- Large documents may take time to process
+- Some URLs may be blocked or require authentication
+- GPU recommended for better performance with larger models
+
+## Troubleshooting
+
+### Out of Memory Errors
+- Use smaller models
+- Reduce chunk size in `ingestion.py`
+- Process fewer documents at once
+
+### URL Fetching Issues
+- Some websites block automated requests
+- Try different URLs or use PDF uploads instead
+
+### Model Loading Issues
+- Ensure you have sufficient disk space
+- Check your internet connection for model downloads
+- Some models require GPU - check model requirements
+
+## License
+
+This project is open source and available under the MIT License.

SETUP_PATH.md

@@ -0,0 +1,70 @@
+# Fixing Python PATH on Windows
+
+Your Python 3.12 is installed but not in your PATH. Here's how to fix it:
+
+## Quick Fix (Temporary - Current Session Only)
+
+For now, you can use these commands:
+- `py -m pip` instead of `pip`
+- `py` instead of `python`
+- `C:\Users\DanielSimeone\AppData\Local\Programs\Python\Python312\python.exe` for direct access
+
+## Permanent Fix - Add Python to PATH
+
+### Method 1: Using Windows Settings (Recommended)
+
+1. **Open System Environment Variables:**
+   - Press `Win + X` and select "System"
+   - Click "Advanced system settings"
+   - Click "Environment Variables" button
+
+2. **Edit PATH:**
+   - Under "User variables" (or "System variables" if you want it for all users), find and select "Path"
+   - Click "Edit"
+   - Click "New" and add these two paths:
+     ```
+     C:\Users\DanielSimeone\AppData\Local\Programs\Python\Python312
+     C:\Users\DanielSimeone\AppData\Local\Programs\Python\Python312\Scripts
+     ```
+   - Click "OK" on all dialogs
+
+3. **Restart your terminal/PowerShell** for changes to take effect
+
+### Method 2: Using PowerShell (Run as Administrator)
+
+```powershell
+[Environment]::SetEnvironmentVariable(
+    "Path",
+    [Environment]::GetEnvironmentVariable("Path", "User") + ";C:\Users\DanielSimeone\AppData\Local\Programs\Python\Python312;C:\Users\DanielSimeone\AppData\Local\Programs\Python\Python312\Scripts",
+    "User"
+)
+```
+
+Then restart your terminal.
+
+### Method 3: Reinstall Python with "Add to PATH" option
+
+If you reinstall Python, make sure to check "Add Python to PATH" during installation.
+
+## Verify It Works
+
+After adding to PATH and restarting your terminal, test:
+
+```bash
+python --version
+pip --version
+```
+
+Both should work without errors.
+
+## For This Project
+
+You can run the app using:
+```bash
+py app.py
+```
+
+Or after fixing PATH:
+```bash
+python app.py
+```

app.py

@@ -0,0 +1,295 @@
+"""
+Gradio app for Hugging Face chatbot with RAG capabilities.
+"""
+import gradio as gr
+from gradio.themes.base import Base
+from gradio.themes.utils import colors, fonts, sizes
+import os
+from typing import List, Tuple
+from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
+from ingestion import DocumentIngestion
+import torch
+
+
+# Create a clean minimalist theme
+class MinimalistTheme(Base):
+    """A clean, minimalist theme with subtle colors and simple styling."""
+    def __init__(self):
+        super().__init__(
+            primary_hue=colors.blue,
+            secondary_hue=colors.gray,
+            neutral_hue=colors.gray,
+            spacing_size=sizes.spacing_md,
+            radius_size=sizes.radius_sm,
+            text_size=sizes.text_md,
+            font=(
+                fonts.GoogleFont("Inter"),
+                "ui-sans-serif",
+                "system-ui",
+                "sans-serif",
+            ),
+            font_mono=(
+                fonts.GoogleFont("JetBrains Mono"),
+                "ui-monospace",
+                "monospace",
+            ),
+        )
+        super().set(
+            # Clean backgrounds
+            body_background_fill="#ffffff",
+            body_background_fill_dark="#0f0f0f",
+            block_background_fill="#ffffff",
+            block_background_fill_dark="#1a1a1a",
+            
+            # Subtle borders
+            block_border_width="1px",
+            block_border_color="#e0e0e0",
+            block_border_color_dark="#2a2a2a",
+            block_shadow="none",
+            
+            # Clean buttons
+            button_primary_background_fill="#2563eb",
+            button_primary_background_fill_hover="#1d4ed8",
+            button_primary_text_color="#ffffff",
+            button_primary_background_fill_dark="#3b82f6",
+            button_primary_background_fill_hover_dark="#2563eb",
+            button_secondary_background_fill="#f3f4f6",
+            button_secondary_background_fill_hover="#e5e7eb",
+            button_secondary_text_color="#111827",
+            button_secondary_background_fill_dark="#374151",
+            button_secondary_background_fill_hover_dark="#4b5563",
+            button_border_width="1px",
+            
+            # Input fields
+            input_background_fill="#ffffff",
+            input_background_fill_dark="#1a1a1a",
+            input_border_width="1px",
+            input_border_color="#d1d5db",
+            input_border_color_dark="#374151",
+            
+            # Text colors
+            body_text_color="#111827",
+            body_text_color_dark="#e5e7eb",
+            block_label_text_color="#374151",
+            block_label_text_color_dark="#9ca3af",
+        )
+
+
+class RAGChatbot:
+    """Chatbot with RAG capabilities."""
+    
+    def __init__(
+        self,
+        model_name: str = "microsoft/DialoGPT-medium",
+        embedding_model: str = "all-MiniLM-L6-v2",
+        vector_store_path: str = "./vector_store"
+    ):
+        """
+        Initialize the RAG chatbot.
+        
+        Args:
+            model_name: Hugging Face model name for the chatbot
+            embedding_model: Model for document embeddings
+            vector_store_path: Path to saved vector store
+        """
+        self.model_name = model_name
+        self.device = "cuda" if torch.cuda.is_available() else "cpu"
+        
+        # Load chatbot model
+        print(f"Loading chatbot model: {model_name}")
+        try:
+            self.tokenizer = AutoTokenizer.from_pretrained(model_name)
+            self.model = AutoModelForCausalLM.from_pretrained(model_name)
+            self.tokenizer.pad_token = self.tokenizer.eos_token
+        except Exception as e:
+            print(f"Warning: Could not load {model_name}. Using a simpler pipeline.")
+            self.model = None
+            self.tokenizer = None
+            self.chatbot_pipeline = pipeline(
+                "text-generation",
+                model="gpt2",
+                device=0 if self.device == "cuda" else -1
+            )
+        
+        # Initialize document ingestion
+        self.ingestion = DocumentIngestion(embedding_model=embedding_model)
+        
+        # Load vector store if it exists
+        if os.path.exists(vector_store_path) and os.path.exists(
+            os.path.join(vector_store_path, "index.faiss")
+        ):
+            try:
+                self.ingestion.load(vector_store_path)
+                print("Loaded existing vector store")
+            except Exception as e:
+                print(f"Could not load vector store: {e}")
+        
+        self.chat_history = []
+    
+    def generate_response(self, query: str, use_rag: bool = True, num_results: int = 3) -> str:
+        """
+        Generate a response to the user query.
+        
+        Args:
+            query: User's question
+            use_rag: Whether to use RAG (retrieve relevant documents)
+            num_results: Number of document chunks to retrieve
+            
+        Returns:
+            Generated response
+        """
+        # If RAG is enabled and we have a vector store, return relevant context
+        if use_rag and self.ingestion.index is not None:
+            try:
+                results = self.ingestion.search(query, k=num_results)
+                if results:
+                    # Format the response with relevant context
+                    response_parts = []
+                    response_parts.append(f"Based on the documents, here's what I found regarding your question: '{query}'\n\n")
+                    
+                    for i, result in enumerate(results, 1):
+                        source = result['metadata']['source']
+                        text = result['text']
+                        # Clean up the text
+                        text = text.strip()
+                        if text:
+                            response_parts.append(f"**Relevant information {i}** (from {source}):\n{text}\n")
+                    
+                    response = "\n".join(response_parts)
+                    return response
+            except Exception as e:
+                print(f"Error in RAG retrieval: {e}")
+                return f"I encountered an error while searching the documents: {str(e)}"
+        
+        # If no RAG or no results, try to generate a response using the model
+        # But DialoGPT isn't great for this, so we'll keep it simple
+        if self.model and self.tokenizer:
+            # Simple generation without complex prompts
+            inputs = self.tokenizer.encode(query, return_tensors="pt")
+            inputs = inputs.to(self.device)
+            
+            with torch.no_grad():
+                outputs = self.model.generate(
+                    inputs,
+                    max_new_tokens=100,
+                    num_return_sequences=1,
+                    temperature=0.7,
+                    do_sample=True,
+                    pad_token_id=self.tokenizer.eos_token_id,
+                    eos_token_id=self.tokenizer.eos_token_id,
+                )
+            
+            # Decode only new tokens
+            input_length = inputs.shape[1]
+            generated_tokens = outputs[0][input_length:]
+            response = self.tokenizer.decode(generated_tokens, skip_special_tokens=True)
+            
+            # Clean up
+            response = response.replace("<|endoftext|>", "").strip()
+            
+            if not response or len(response.strip()) < 3:
+                return "I understand your question, but I don't have relevant information in my knowledge base. Please enable RAG to search the documents."
+            
+            return response
+        else:
+            return "I understand your question, but I don't have relevant information in my knowledge base. Please enable RAG to search the documents."
+    
+    def chat(self, message: str, history, use_rag: bool):
+        """
+        Handle chat interaction.
+        
+        Args:
+            message: User message
+            history: Chat history (list of ChatMessage or dicts with 'role' and 'content')
+            use_rag: Whether to use RAG
+            
+        Returns:
+            Updated history
+        """
+        if not message or not message.strip():
+            return "", history or []
+        
+        # Ensure history is a list
+        if history is None:
+            history = []
+        
+        # Add user message as dictionary
+        history.append({"role": "user", "content": message})
+        
+        # Generate response
+        try:
+            response = self.generate_response(message, use_rag=use_rag)
+            # Ensure response is not empty
+            if not response or not response.strip():
+                response = "I'm sorry, I couldn't generate a response. Please try again."
+        except Exception as e:
+            print(f"Error generating response: {e}")
+            import traceback
+            traceback.print_exc()
+            response = f"I encountered an error: {str(e)}"
+        
+        # Add assistant response as dictionary
+        history.append({"role": "assistant", "content": response})
+        
+        print(f"Debug - History length: {len(history)}")
+        print(f"Debug - Response: {response[:100] if response else 'None'}...")
+        
+        return "", history
+
+
+# Initialize chatbot
+chatbot = RAGChatbot()
+
+
+# Create Gradio interface
+with gr.Blocks(title="Hugging Face RAG Chatbot") as app:
+    gr.Markdown("# 🤖 Hugging Face RAG Chatbot")
+    gr.Markdown("Chat with your documents! Run `ingest_documents.py` to update the knowledge base.")
+    
+    chatbot_interface = gr.Chatbot(
+        label="Chat",
+        height=500,
+        value=[]  # Initialize with empty list
+    )
+    
+    with gr.Row():
+        msg = gr.Textbox(
+            label="Your Message",
+            placeholder="Ask a question about your documents...",
+            scale=4
+        )
+        use_rag = gr.Checkbox(
+            label="Use RAG",
+            value=True,
+            scale=1
+        )
+    
+    with gr.Row():
+        submit_btn = gr.Button("Send", variant="primary")
+        clear_btn = gr.Button("Clear")
+    
+    msg.submit(
+        chatbot.chat,
+        inputs=[msg, chatbot_interface, use_rag],
+        outputs=[msg, chatbot_interface]
+    )
+    submit_btn.click(
+        chatbot.chat,
+        inputs=[msg, chatbot_interface, use_rag],
+        outputs=[msg, chatbot_interface]
+    )
+    def clear_chat():
+        return [], ""
+    
+    clear_btn.click(clear_chat, outputs=[chatbot_interface, msg])
+
+
+if __name__ == "__main__":
+    # Get port from environment variable (Hugging Face Spaces sets this) or default to 7860
+    port = int(os.environ.get("PORT", 7860))
+    app.launch(
+        share=False,
+        server_name="0.0.0.0",
+        server_port=port,
+        theme=MinimalistTheme()
+    )

example_usage.py

@@ -0,0 +1,53 @@
+"""
+Example script showing how to use the document ingestion system programmatically.
+"""
+from ingestion import DocumentIngestion
+
+
+def main():
+    # Initialize the ingestion system
+    ingestion = DocumentIngestion(embedding_model="all-MiniLM-L6-v2")
+    
+    # Example 1: Process PDFs
+    pdf_paths = [
+        # Add your PDF file paths here
+        # "path/to/document1.pdf",
+        # "path/to/document2.pdf",
+    ]
+    
+    # Example 2: Process URLs
+    urls = [
+        # Add URLs here
+        # "https://en.wikipedia.org/wiki/Artificial_intelligence",
+        # "https://huggingface.co/docs/transformers",
+    ]
+    
+    # Process documents
+    if pdf_paths or urls:
+        print("Processing documents...")
+        documents = ingestion.process_documents(pdf_paths=pdf_paths, urls=urls)
+        print(f"Processed {len(documents)} document chunks")
+        
+        # Build vector store
+        ingestion.build_vector_store()
+        
+        # Save vector store
+        ingestion.save("./vector_store")
+        
+        # Example search
+        query = "What is artificial intelligence?"
+        results = ingestion.search(query, k=3)
+        
+        print(f"\nSearch results for: '{query}'")
+        print("-" * 50)
+        for i, result in enumerate(results, 1):
+            print(f"\nResult {i}:")
+            print(f"Source: {result['metadata']['source']}")
+            print(f"Score: {result['score']:.4f}")
+            print(f"Text: {result['text'][:200]}...")
+    else:
+        print("Please add PDF paths or URLs to the script to test ingestion.")
+
+
+if __name__ == "__main__":
+    main()

ingest_documents.py

@@ -0,0 +1,87 @@
+"""
+Standalone script to ingest PDFs and URLs into the vector store.
+Run this script periodically to update your document knowledge base.
+"""
+import os
+from pathlib import Path
+from ingestion import DocumentIngestion
+
+
+# Configuration
+PDF_FOLDER = "./pdfs"  # Folder containing PDF files
+URLS = [
+    # Add your URLs here, one per line
+    "https://www.ontario.ca/page/organic-crop-and-livestock-production-ontario"
+]
+
+
+def main():
+    """Main ingestion function."""
+    print("=" * 60)
+    print("Document Ingestion Script")
+    print("=" * 60)
+    
+    # Initialize ingestion system
+    print("\nInitializing document ingestion system...")
+    ingestion = DocumentIngestion(embedding_model="all-MiniLM-L6-v2")
+    
+    # Collect PDF files
+    pdf_paths = []
+    if os.path.exists(PDF_FOLDER):
+        pdf_files = list(Path(PDF_FOLDER).glob("*.pdf"))
+        pdf_paths = [str(f) for f in pdf_files]
+        print(f"\nFound {len(pdf_paths)} PDF file(s) in {PDF_FOLDER}:")
+        for pdf in pdf_paths:
+            print(f"  - {os.path.basename(pdf)}")
+    else:
+        print(f"\nPDF folder '{PDF_FOLDER}' not found. Creating it...")
+        os.makedirs(PDF_FOLDER, exist_ok=True)
+        print(f"Please add PDF files to {PDF_FOLDER} and run again.")
+    
+    # Filter out empty URLs
+    urls = [url.strip() for url in URLS if url.strip()]
+    
+    if urls:
+        print(f"\nFound {len(urls)} URL(s) to process:")
+        for url in urls:
+            print(f"  - {url}")
+    else:
+        print("\nNo URLs configured. Add URLs to the URLS list in this script.")
+    
+    if not pdf_paths and not urls:
+        print("\n[ERROR] No documents to process. Please add PDFs or URLs.")
+        return
+    
+    # Process documents
+    print("\n" + "=" * 60)
+    print("Processing documents...")
+    print("=" * 60)
+    
+    try:
+        documents = ingestion.process_documents(pdf_paths=pdf_paths, urls=urls)
+        print(f"\n[SUCCESS] Successfully processed {len(documents)} document chunks")
+        
+        # Build vector store
+        print("\nBuilding vector store...")
+        ingestion.build_vector_store()
+        
+        # Save vector store
+        print("\nSaving vector store...")
+        ingestion.save("./vector_store")
+        
+        print("\n" + "=" * 60)
+        print("[SUCCESS] Ingestion complete!")
+        print("=" * 60)
+        print(f"\nTotal document chunks: {len(documents)}")
+        print(f"Vector store saved to: ./vector_store")
+        print("\nYou can now run 'py app.py' to start the chatbot.")
+        
+    except Exception as e:
+        print(f"\n[ERROR] Error during ingestion: {str(e)}")
+        import traceback
+        traceback.print_exc()
+        return
+
+
+if __name__ == "__main__":
+    main()

ingestion.py

@@ -0,0 +1,231 @@
+"""
+Document ingestion module for processing PDFs and URLs.
+"""
+import os
+from typing import List, Dict
+from pathlib import Path
+import requests
+from bs4 import BeautifulSoup
+from pypdf import PdfReader
+from langchain_text_splitters import RecursiveCharacterTextSplitter
+from sentence_transformers import SentenceTransformer
+import numpy as np
+import faiss
+import pickle
+
+
+class DocumentIngestion:
+    """Handles ingestion of PDFs and URLs into a searchable vector store."""
+    
+    def __init__(self, embedding_model: str = "all-MiniLM-L6-v2"):
+        """
+        Initialize the document ingestion system.
+        
+        Args:
+            embedding_model: Hugging Face model name for embeddings
+        """
+        self.embedding_model = SentenceTransformer(embedding_model)
+        self.text_splitter = RecursiveCharacterTextSplitter(
+            chunk_size=1000,
+            chunk_overlap=200,
+            length_function=len,
+        )
+        self.documents = []
+        self.embeddings = None
+        self.index = None
+        
+    def read_pdf(self, file_path: str) -> str:
+        """
+        Extract text from a PDF file.
+        
+        Args:
+            file_path: Path to the PDF file
+            
+        Returns:
+            Extracted text content
+        """
+        try:
+            reader = PdfReader(file_path)
+            text = ""
+            for page in reader.pages:
+                text += page.extract_text() + "\n"
+            return text
+        except Exception as e:
+            raise Exception(f"Error reading PDF {file_path}: {str(e)}")
+    
+    def read_url(self, url: str) -> str:
+        """
+        Extract text from a URL.
+        
+        Args:
+            url: URL to fetch and extract text from
+            
+        Returns:
+            Extracted text content
+        """
+        try:
+            headers = {
+                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
+            }
+            response = requests.get(url, headers=headers, timeout=10)
+            response.raise_for_status()
+            
+            soup = BeautifulSoup(response.content, 'html.parser')
+            
+            # Remove script and style elements
+            for script in soup(["script", "style"]):
+                script.decompose()
+            
+            # Get text
+            text = soup.get_text()
+            
+            # Clean up whitespace
+            lines = (line.strip() for line in text.splitlines())
+            chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
+            text = ' '.join(chunk for chunk in chunks if chunk)
+            
+            return text
+        except Exception as e:
+            raise Exception(f"Error reading URL {url}: {str(e)}")
+    
+    def process_documents(self, pdf_paths: List[str] = None, urls: List[str] = None) -> List[Dict]:
+        """
+        Process PDFs and URLs into chunks.
+        
+        Args:
+            pdf_paths: List of PDF file paths
+            urls: List of URLs to process
+            
+        Returns:
+            List of document chunks with metadata
+        """
+        all_texts = []
+        all_metadata = []
+        
+        # Process PDFs
+        if pdf_paths:
+            for pdf_path in pdf_paths:
+                if not os.path.exists(pdf_path):
+                    print(f"Warning: PDF file not found: {pdf_path}")
+                    continue
+                    
+                text = self.read_pdf(pdf_path)
+                chunks = self.text_splitter.split_text(text)
+                
+                for i, chunk in enumerate(chunks):
+                    all_texts.append(chunk)
+                    all_metadata.append({
+                        'source': pdf_path,
+                        'type': 'pdf',
+                        'chunk_index': i
+                    })
+        
+        # Process URLs
+        if urls:
+            for url in urls:
+                try:
+                    text = self.read_url(url)
+                    chunks = self.text_splitter.split_text(text)
+                    
+                    for i, chunk in enumerate(chunks):
+                        all_texts.append(chunk)
+                        all_metadata.append({
+                            'source': url,
+                            'type': 'url',
+                            'chunk_index': i
+                        })
+                except Exception as e:
+                    print(f"Warning: Failed to process URL {url}: {str(e)}")
+                    continue
+        
+        # Create document objects
+        documents = []
+        for text, metadata in zip(all_texts, all_metadata):
+            documents.append({
+                'text': text,
+                'metadata': metadata
+            })
+        
+        self.documents = documents
+        return documents
+    
+    def build_vector_store(self):
+        """Build FAISS vector store from processed documents."""
+        if not self.documents:
+            raise ValueError("No documents processed. Call process_documents() first.")
+        
+        # Extract texts
+        texts = [doc['text'] for doc in self.documents]
+        
+        # Generate embeddings
+        print("Generating embeddings...")
+        self.embeddings = self.embedding_model.encode(texts, show_progress_bar=True)
+        
+        # Build FAISS index
+        dimension = self.embeddings.shape[1]
+        self.index = faiss.IndexFlatL2(dimension)
+        self.index.add(self.embeddings.astype('float32'))
+        
+        print(f"Vector store built with {len(self.documents)} documents")
+    
+    def search(self, query: str, k: int = 5) -> List[Dict]:
+        """
+        Search for similar documents.
+        
+        Args:
+            query: Search query
+            k: Number of results to return
+            
+        Returns:
+            List of relevant document chunks with scores
+        """
+        if self.index is None:
+            raise ValueError("Vector store not built. Call build_vector_store() first.")
+        
+        # Encode query
+        query_embedding = self.embedding_model.encode([query])
+        
+        # Search
+        distances, indices = self.index.search(query_embedding.astype('float32'), k)
+        
+        # Format results
+        results = []
+        for i, idx in enumerate(indices[0]):
+            if idx < len(self.documents):
+                results.append({
+                    'text': self.documents[idx]['text'],
+                    'metadata': self.documents[idx]['metadata'],
+                    'score': float(distances[0][i])
+                })
+        
+        return results
+    
+    def save(self, directory: str = "./vector_store"):
+        """Save the vector store to disk."""
+        os.makedirs(directory, exist_ok=True)
+        
+        # Save index
+        faiss.write_index(self.index, os.path.join(directory, "index.faiss"))
+        
+        # Save documents and embeddings
+        with open(os.path.join(directory, "documents.pkl"), "wb") as f:
+            pickle.dump(self.documents, f)
+        
+        with open(os.path.join(directory, "embeddings.pkl"), "wb") as f:
+            pickle.dump(self.embeddings, f)
+        
+        print(f"Vector store saved to {directory}")
+    
+    def load(self, directory: str = "./vector_store"):
+        """Load the vector store from disk."""
+        # Load index
+        self.index = faiss.read_index(os.path.join(directory, "index.faiss"))
+        
+        # Load documents and embeddings
+        with open(os.path.join(directory, "documents.pkl"), "rb") as f:
+            self.documents = pickle.load(f)
+        
+        with open(os.path.join(directory, "embeddings.pkl"), "rb") as f:
+            self.embeddings = pickle.load(f)
+        
+        print(f"Vector store loaded from {directory}")

pdfs/.gitkeep

@@ -0,0 +1,2 @@
+# This file ensures the pdfs folder is tracked in git
+# Add your PDF files to this folder

requirements.txt

@@ -0,0 +1,12 @@
+gradio>=4.0.0
+transformers>=4.35.0
+torch>=2.0.0
+sentence-transformers>=2.2.0
+langchain>=0.1.0
+langchain-community>=0.0.20
+pypdf>=3.17.0
+beautifulsoup4>=4.12.0
+requests>=2.31.0
+faiss-cpu>=1.7.4
+numpy>=1.24.0
+accelerate>=0.25.0