Spaces:
Runtime error
Runtime error
| title: web search MCP-server | |
| sdk: gradio | |
| colorFrom: green | |
| colorTo: green | |
| short_description: MCP server for general and custom search on web | |
| sdk_version: 5.34.0 | |
| tags: | |
| - mcp-server-track | |
| app_file: app.py | |
| pinned: true | |
| # Search Tool | |
| ## Overview | |
| **Search Tool** is a modular Python framework for performing advanced web searches, scraping content from search results, and analyzing the retrieved information using AI-powered models. The project is designed for extensibility, allowing easy integration of new search engines, scrapers, and analyzers. | |
| ## Demo video | |
| Link: https://drive.google.com/file/d/11bHRCr0tdAkCEtwKOiuzzfAp7RgZk-si/view?usp=sharing | |
|  | |
| ## Features | |
| - **Custom Site Search:** Search within a specified list of websites. | |
| - **Custom Domain Search:** Restrict searches to specific domains (e.g., `.edu`, `.gov`). | |
| - **General Web Search:** Perform open web searches. | |
| - **Content Scraping:** Extracts main textual content from URLs using [trafilatura](https://trafilatura.readthedocs.io/). | |
| - **AI Analysis:** Summarizes and analyzes scraped content using OpenAI models. | |
| - **Validation:** Ensures URLs are valid before processing. | |
| - **Extensible Architecture:** Easily add new searchers, scrapers, or analyzers. | |
| ## Project Structure | |
| ``` | |
| search_tool/ | |
| ├── src/ | |
| │ ├── analyzer/ # AI-powered analyzers (e.g., OpenAI) | |
| │ ├── core/ | |
| │ │ ├── factory/ # Factories for searcher, scraper, | |
| │ │ ├── interface/ # Abstract interfaces for extensibility | |
| │ │ └── types.py # Enums and constants | |
| │ ├── mcp_servers/ # MCP server integration | |
| │ ├── models/ # Pydantic models for data validation | |
| │ ├── scraper/ # Web scrapers (e.g., Trafilatura) | |
| │ ├── searcher/ # Search engine integrations | |
| │ ├── tools/ # User-facing tool functions | |
| │ └── utils/ # Utility functions (e.g., URL validation) | |
| ├── test.py # Example/test script | |
| ├── requirements.txt # Python dependencies | |
| ├── pyproject.toml # Project metadata and dependencies | |
| ├── .env # Environment variables (e.g., API keys) | |
| └── README.md # Project documentation | |
| ``` | |
| ## Installation | |
| 1. **Clone the repository:** | |
| ```sh | |
| git clone https://github.com/ola172/web-search-mcp-server.git | |
| cd search_tool | |
| ``` | |
| 2. **Set up a virtual environment (recommended):** | |
| ```sh | |
| python3 -m venv .venv | |
| source .venv/bin/activate | |
| ``` | |
| 3. **Install dependencies:** | |
| ```sh | |
| pip install -r requirements.txt | |
| ``` | |
| 4. **Configure environment variables:** | |
| - Copy `.env.example` to `.env` | |
| - Add your secrets: | |
| ## Usage | |
| ### Core Tools | |
| Each tool validates input, performs the search, scrapes the results, and analyzes the content. | |
| - **General Web Search:** `search_on_web` | |
| - **Custom Sites Search:** `search_custom_sites` | |
| - **Custom Domains Search:** `search_custom_domain` | |
| ### MCP Server Integration | |
| The project includes an MCP server (`web_search_server.py`) for exposing search tools as mcp tools. | |
| ## Extending the Framework | |
| - **Add a new searcher:** Implement the `SearchInterface` and register it in `SearcherFactory`. | |
| - **Add a new scraper:** Implement the `ScraperInterface` and register it in `ScraperFactory`. | |
| - **Add a new analyzer:** Implement the `AnalyzerInterface` and register it in `AnalyzerFactory`. | |
| ## Configuration | |
| - **API Keys:** Store sensitive keys (e.g., OpenAI) in the `.env` file. | |
| - **Search Engine IDs:** For Google Custom Search, configure `API_KEY` and `SEARCH_ENGINE_ID` in the relevant modules. | |
| ## Dependencies | |
| - `openai` | |
| - `trafilatura` | |
| - `pydantic` | |
| - `googlesearch-python` | |
| - `python-dotenv` | |
| - `google-api-python-client` | |
| See `requirements.txt` for the full list. | |
| ## License | |
| This project is for educational and research purposes. Please ensure compliance with the terms of service of any third-party APIs used. | |
| ## Acknowledgements | |
| - OpenAI | |
| - Trafilatura | |
| - Google Custom Search | |
| For questions or contributions, please open an issue or pull request. |