README.md

---
license: mit
title: crawling rag
sdk: gradio
emoji: 👁
colorFrom: gray
colorTo: purple
short_description: a rag that can crawle website
sdk_version: 5.7.1
---
# 🔍 Intelligent Web Search Engine

An advanced AI-powered search engine that provides deep understanding of web content, code analysis, and intelligent insights.

## 🌟 Features

- Multi-model AI analysis
- Semantic search and caching
- Automatic insights generation
- Smart follow-up questions
- Code-aware analysis
- Related searches

## 🚀 Deployment to Hugging Face Spaces

1. Create a new Space:
   - Go to [huggingface.co/spaces](https://huggingface.co/spaces)
   - Click "Create new Space"
   - Choose "Gradio" as the SDK
   - Select "CPU" as the hardware
   - Name your space (e.g., "intelligent-web-search")

2. Upload Files:
   - Upload all files from the `aiws` directory
   - Make sure to include:
     - `app.py`
     - `search_engine.py`
     - `requirements.txt`
     - `packages.txt`

3. Space Settings:
   - Go to the "Settings" tab
   - Under "Repository secrets", add any required API keys
   - Under "Variables", set:
     ```
     PYTHON_PACKAGES_PATH=/home/user/.local/lib/python3.9/site-packages
     ```

4. The space will automatically build and deploy your app

## 📦 Local Development

1. Clone the repository:
```bash
git clone [your-repo-url]
cd aiws
```

2. Install dependencies:
```bash
pip install -r requirements.txt
```

3. Run the app:
```bash
python app.py
```

## 🔧 Configuration

The search engine uses several AI models:
- Summarization: facebook/bart-base
- Code Understanding: Salesforce/codet5-small
- General QA: google/flan-t5-base
- Embeddings: sentence-transformers/all-MiniLM-L6-v2

## 📝 Usage

1. Enter your search query
2. Adjust the maximum number of results (1-20)
3. Click "Search"
4. View results including:
   - Key insights
   - Follow-up questions
   - Detailed analysis
   - Related searches

## 🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.