README.md

---
title: RAG Chatbot
emoji: 🤖
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 6.3.0
app_file: app.py
pinned: false
---

# Hugging Face RAG Chatbot

A Retrieval-Augmented Generation (RAG) chatbot that can ingest PDFs and URLs, then answer questions based on the ingested documents. Built for deployment on Hugging Face Spaces.

## Features

- 📄 **PDF Ingestion**: Upload and process PDF documents
- 🌐 **URL Ingestion**: Extract and process content from web URLs
- 🔍 **Vector Search**: Semantic search using sentence transformers
- 💬 **Chatbot Interface**: Interactive Gradio interface for querying documents
- 🚀 **Hugging Face Ready**: Configured for easy deployment to Hugging Face Spaces

## Setup

### Local Development

1. **Install dependencies:**
   ```bash
   pip install -r requirements.txt
   ```

2. **Run the application:**
   ```bash
   python app.py
   ```

3. **Access the interface:**
   - Open your browser to `http://localhost:7860`

### Usage

1. **Ingest Documents (Run this first or periodically to update):**
   - Add PDF files to the `pdfs/` folder
   - Edit `ingest_documents.py` and add your URLs to the `URLS` list
   - Run the ingestion script:
     ```bash
     py ingest_documents.py
     ```
   - Wait for processing to complete (this creates/updates the vector store)

2. **Chat with Documents:**
   - Run the chatbot app:
     ```bash
     py app.py
     ```
   - Open your browser to `http://localhost:7860`
   - Toggle "Use RAG" to enable/disable document retrieval
   - Ask questions about your ingested documents
   - The chatbot will retrieve relevant context and generate answers

## Deployment to Hugging Face Spaces

### Option 1: Using Hugging Face CLI

1. **Install Hugging Face CLI:**
   ```bash
   pip install huggingface_hub
   ```

2. **Login to Hugging Face:**
   ```bash
   huggingface-cli login
   ```

3. **Create a new Space:**
   - Go to https://huggingface.co/new-space
   - Choose a name and select "Gradio" as the SDK
   - Create the space

4. **Clone and push your code:**
   ```bash
   git clone https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
   cd YOUR_SPACE_NAME
   # Copy your files here
   git add .
   git commit -m "Initial commit"
   git push
   ```

### Option 2: Using Git

1. **Initialize git repository:**
   ```bash
   git init
   git add .
   git commit -m "Initial commit"
   ```

2. **Add Hugging Face remote:**
   ```bash
   git remote add origin https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
   git push -u origin main
   ```

### Required Files for Hugging Face Spaces

Your Space needs these files:
- `app.py` - Main Gradio application
- `requirements.txt` - Python dependencies
- `README.md` - This file (optional but recommended)

### Optional: Add app.py to README

For Hugging Face Spaces, you can also add a `app.py` reference in your README:

```yaml
---
title: RAG Chatbot
emoji: 🤖
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 4.0.0
app_file: app.py
pinned: false
---
```

## Configuration

### Setting Up Hugging Face Token (Required)

The chatbot uses Hugging Face Inference API to access high-quality models. You need to set up an API token:

1. **Get your token:**
   - Go to https://huggingface.co/settings/tokens
   - Create a new token with "Read" permissions
   - Copy the token

2. **For local development:**
   - Set environment variable: `export HF_TOKEN=your_token_here` (Linux/Mac)
   - Or: `set HF_TOKEN=your_token_here` (Windows)
   - Or create a `.env` file with `HF_TOKEN=your_token_here`

3. **For Hugging Face Spaces:**
   - Go to your Space → Settings → Secrets
   - Add a new secret: Name = `HF_TOKEN`, Value = your token
   - The app will automatically use this token

### Chatbot Model and Finding Available Models

The chatbot uses the Hugging Face Inference API. **Which models you can use depends on which providers you have enabled.**

**How to see which models are available to you:**

1. **Browse models that support Inference API**
   - https://huggingface.co/inference/models — lists providers and models
   - https://huggingface.co/models?inference_provider=hf-inference — filter Hub models by “HF Inference API”

2. **Enable providers (required)**  
   See **“How to enable a provider and model”** below.

3. **Pick a chat model**
   - From the links above, choose a **text generation / chat** model that’s supported by a provider you enabled. Note its **model id** (e.g. `meta-llama/Llama-3.2-1B-Instruct`).

4. **Use it in the app**
   - In `app.py`, pass that model id when creating the chatbot:
   ```python
   chatbot = RAGChatbot(model_name="meta-llama/Llama-3.2-1B-Instruct")  # use an id you enabled
   ```
   - The app also tries fallbacks (Phi-2, Zephyr, Qwen) by default; if none are available, enable a provider that supports at least one of them, or set `model_name` to a model you enabled as above.

### How to enable a provider and model

1. **Log in** to [Hugging Face](https://huggingface.co).

2. **Open Inference Provider settings** (one of these, depending on the current UI):
   - https://huggingface.co/settings/inference-providers  
   - https://huggingface.co/settings/inference-api  

3. **Enable a provider**
   - On that page you’ll see a list of **providers** (e.g. Hugging Face, Together, Groq, etc.).
   - **Turn on** the provider that serves your model (e.g. **Together** for `ServiceNow-AI/Apriel-1.6-15b-Thinker:together`).
   - You can set the **order** of providers; “auto” uses this order to pick which provider handles the request.

4. **Credits / billing**
   - Free accounts get a small amount of monthly credits; usage is deducted from that.
   - If you use a third-party provider (e.g. Together), you can either use HF-routed billing (credits on your HF account) or add a **custom provider API key** (e.g. Together API key) in the same settings so that provider is billed directly.

5. **Confirm the model**
   - Browse models for that provider:  
     [Together models on the Hub](https://huggingface.co/models?inference_provider=together&sort=trending)  
   - Open the model page (e.g. `ServiceNow-AI/Apriel-1.6-15b-Thinker`) and check the inference widget; if you see “Together” and can run it, that model is available with your enabled provider.

6. **Use it in this app**
   - The app default is already `ServiceNow-AI/Apriel-1.6-15b-Thinker:together`.  
   - Ensure **Together** is enabled in your Inference Provider settings and you have credits (or a Together API key). Then restart the app and send a message.

### Embedding Model

The default embedding model is **all-mpnet-base-v2**, which provides high-quality embeddings for better retrieval.

To change the embedding model, edit both `app.py` and `ingest_documents.py`:

```python
# In app.py
chatbot = RAGChatbot(embedding_model="all-mpnet-base-v2")

# In ingest_documents.py
ingestion = DocumentIngestion(embedding_model="all-mpnet-base-v2")
```

**Note:** If you change the embedding model, you must re-run `ingest_documents.py` to rebuild the vector store.

### Ingestion Parameters

The ingestion system uses optimized parameters:
- **Chunk size**: 600 characters (for precise retrieval)
- **Chunk overlap**: 150 characters (to avoid cutting sentences)
- **Retrieval count**: 5 chunks (for comprehensive context)

These parameters are set in `ingestion.py` and can be adjusted if needed.

## Project Structure

```
.
├── app.py                  # Main Gradio chatbot application
├── ingest_documents.py     # Standalone script to ingest PDFs and URLs
├── ingestion.py            # Document ingestion and vector store module
├── requirements.txt        # Python dependencies
├── README.md              # This file
├── pdfs/                  # Folder for PDF files (add your PDFs here)
│   └── README.md
└── data/
    └── vector_store/     # Saved vector store (created after ingestion)
        ├── index.faiss
        ├── documents.pkl
        └── embeddings.pkl
```

## How It Works

The chatbot uses **Retrieval-Augmented Generation (RAG)**:

1. **Document Ingestion**: PDFs and URLs are processed into chunks and embedded using sentence transformers
2. **Vector Search**: When you ask a question, the system searches for the most relevant document chunks
3. **Answer Generation**: The retrieved context is sent to Mistral-7B-Instruct via Inference API, which synthesizes a coherent answer based on the context

This approach combines the accuracy of document retrieval with the natural language capabilities of a large language model.

## Limitations

- Vector store is stored locally (not persistent on Hugging Face Spaces by default)
- Large documents may take time to process
- Some URLs may be blocked or require authentication
- Requires HF_TOKEN for Inference API access (free tier available)
- If you change embedding model or chunk parameters, you must re-run ingestion

## Troubleshooting

### Out of Memory Errors
- Use smaller models
- Reduce chunk size in `ingestion.py`
- Process fewer documents at once

### URL Fetching Issues
- Some websites block automated requests
- Try different URLs or use PDF uploads instead

### Inference API Issues
- Verify your `HF_TOKEN` is set correctly
- Check that the token has "Read" permissions
- Ensure you have API access (free tier available)
- If you get rate limit errors, you may need to upgrade your Hugging Face account

### Ingestion Issues
- If you changed embedding model or chunk parameters, re-run `ingest_documents.py`
- Ensure you have enough disk space for the vector store
- Large documents may take time to process

## License

This project is open source and available under the MIT License.