README.md

---
license: agpl-3.0
sdk: gradio
---
# Vector Search Demo App

This is a Gradio web application that demonstrates vector search capabilities using MongoDB Atlas and OpenAI embeddings.

## Prerequisites

1. MongoDB Atlas account with vector search enabled
2. OpenAI API key
3. Python 3.8+
4. Sample movie data loaded in MongoDB Atlas (sample_mflix database)

## Setup

1. Clone this repository

2. Install dependencies:
```bash
pip install -r requirements.txt
```

3. Set up environment variables:
```bash
export OPENAI_API_KEY="your-openai-api-key"
export ATLAS_URI="your-mongodb-atlas-connection-string"
```

4. Ensure your MongoDB Atlas setup:
   - Database name: sample_mflix
   - Collection: embedded_movies
   - Vector search index: idx_plot_embedding
   - Index configuration:
   ```json
   {
     "fields": [
       {
         "type": "vector",
         "path": "plot_embedding",
         "numDimensions": 1536,
         "similarity": "dotProduct"
       }
     ]
   }
   ```

## Running the App

Start the application:
```bash
python app.py
```

The app will be available at http://localhost:7860

## Usage

### Generating Embeddings
1. Select your database and collection from the dropdowns
2. Choose the field to generate embeddings for
3. Specify the embedding field name (defaults to "embedding")
4. Set a document limit (0 for all documents)
5. Click "Generate Embeddings" to start processing

The app uses memory-efficient cursor-based batch processing that can handle large collections:
- Documents are processed in batches (default 20 documents per batch)
- Memory usage is optimized through cursor-based iteration
- Real-time progress tracking shows completed/total documents
- Supports processing of large collections (100,000+ documents)
- Automatically resumes from where it left off if embeddings already exist

### Searching
1. Enter a natural language query in the text box (e.g., "humans fighting aliens")
2. Click "Submit" to search
3. View the results showing matching documents with their similarity scores

## Example Queries

- "humans fighting aliens"
- "relationship drama between two good friends"
- "comedy about family vacation"
- "detective solving mysterious murder"

## Performance Notes

The application is optimized for handling large datasets:
- Uses cursor-based batch processing to avoid memory issues
- Processes documents in configurable batch sizes (default: 20)
- Implements parallel processing with ThreadPoolExecutor
- Provides real-time progress tracking
- Automatically handles memory cleanup during processing
- Supports resuming interrupted operations

## Notes

- The search uses OpenAI's text-embedding-ada-002 model to create embeddings
- Results are limited to top 5 matches
- Similarity scores range from 0 to 1, with higher scores indicating better matches