| # RAG System | |
| A modular Retrieval-Augmented Generation (RAG) system for document-based question answering. | |
| ## Features | |
| - **Document Processing**: Extract and chunk text from PDF, DOCX, and TXT files | |
| - **Semantic Search**: Embed and search documents based on meaning, not just keywords | |
| - **Flexible Architecture**: Support for multiple embedding models and vector databases | |
| - **REST API**: API for integrating with other applications | |
| - **Web UI**: User-friendly Streamlit interface for document upload and querying | |
| ## Architecture | |
| The system consists of the following components: | |
| - **Embedding Model**: Converts text to vector embeddings | |
| - **Vector Database**: Stores and searches document embeddings | |
| - **Document Processor**: Extracts and chunks text from documents | |
| - **RAG Engine**: Combines retrieval and generation for question answering | |
| - **API**: Exposes functionality through a RESTful API | |
| - **UI**: Provides a user interface for interacting with the system | |
| ## Installation | |
| ### Prerequisites | |
| - Python 3.8+ | |
| - pip | |
| ### Setup | |
| 1. Clone the repository: | |
| ```bash | |
| git clone https://github.com/yourusername/rag-system.git | |
| cd rag-system | |
| ``` | |
| 2. Install dependencies: | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| 3. Set up environment variables (optional): | |
| ```bash | |
| cp .env.example .env | |
| # Edit .env with your settings | |
| ``` | |
| ## Usage | |
| ### API Server | |
| Run the API server: | |
| ```bash | |
| python app.py | |
| ``` | |
| The API will be available at http://localhost:8000 | |
| ### Streamlit UI | |
| Run the Streamlit UI: | |
| ```bash | |
| streamlit run ui/app.py | |
| ``` | |
| The UI will be available at http://localhost:8501 | |
| ## API Endpoints | |
| - `POST /documents`: Add documents | |
| - `POST /upload`: Upload and process document files | |
| - `POST /query`: Query the RAG system | |
| - `GET /search`: Search for documents | |
| - `DELETE /documents`: Clear all documents | |
| - `GET /health`: Check system health | |
| ## Configuration | |
| The system can be configured through environment variables or the `config.py` file: | |
| - `EMBEDDING_MODEL_NAME`: Name of the embedding model | |
| - `VECTOR_DB_TYPE`: Type of vector database to use | |
| - `CHUNK_SIZE`: Size of document chunks | |
| - `CHUNK_OVERLAP`: Overlap between chunks | |
| - `TOP_K`: Number of documents to retrieve | |
| - `SEARCH_TYPE`: Type of search (semantic, keyword, hybrid) | |
| - `LLM_MODEL_NAME`: Name of the language model for generation | |
| - `LLM_API_KEY`: API key for the language model | |
| ## Extending | |
| The modular architecture makes it easy to extend the system: | |
| - Add new embedding models in `embedding/model.py` | |
| - Add new vector databases in `storage/vector_db.py` | |
| - Add support for new document types in `document/processor.py` | |
| - Add new LLM integrations in `llm/model.py` | |
| ## License | |
| [MIT License](LICENSE) | |