Upload README.md

README.md

@@ -1,245 +1,53 @@
-# RAG System for Hugging Face Spaces
-
-A simplified Retrieval-Augmented Generation (RAG) system optimized for deployment on Hugging Face Spaces.
-
-## 🚀 Features
-
-- **FAISS Vector Search**: Fast similarity search using FAISS
-- **BM25 Keyword Search**: Traditional keyword-based retrieval
-- **Hybrid Search**: Combines both dense and sparse retrieval
-- **Qwen 2.5 1.5B**: Advanced language model for answer generation
-- **Streamlit UI**: Clean, interactive web interface
-- **PDF Processing**: Extract and process PDF documents
-- **Persistent Storage**: Saves embeddings and metadata locally
-
-## 📁 Project Structure
-
-```
-huggingface_deploy/
-├── app.py                 # Main Streamlit application
-├── rag_system.py          # Simplified RAG system
-├── pdf_processor.py       # PDF processing utilities
-├── requirements.txt       # Python dependencies
-├── README.md             # This file
-└── vector_store/         # FAISS index and metadata (created automatically)
-```
-
-## 🛠️ Technologies Used
-
-- **Streamlit**: Web interface
-- **FAISS**: Vector similarity search
-- **BM25**: Keyword-based retrieval
-- **Sentence Transformers**: Text embeddings
-- **Transformers**: Qwen 2.5 1.5B model
-- **PyPDF**: PDF text extraction
-- **PyTorch**: Deep learning framework
-
-## 🚀 Quick Start
-
-### Local Development
-
-1. **Install dependencies:**
-```bash
-pip install -r requirements.txt
-```
-
-2. **Run the application:**
-```bash
-streamlit run app.py
-```
-
-3. **Open in browser:**
-Navigate to `http://localhost:8501`
-
-### Hugging Face Spaces Deployment
-
-1. **Create a new Space:**
-   - Go to [Hugging Face Spaces](https://huggingface.co/spaces)
-   - Click "Create new Space"
-   - Choose "Streamlit" as the SDK
-   - Set visibility (public or private)
-
-2. **Upload files:**
-   - Upload all files from this directory to your Space
-   - The Space will automatically install dependencies and run the app
-
-3. **Access your app:**
-   - Your RAG system will be available at your Space URL
-
-## 📖 How to Use
-
-### 1. Upload Documents
-- Use the sidebar to upload PDF documents
-- The system will automatically process and index the content
-- Multiple documents can be uploaded
-
-### 2. Ask Questions
-- Type your question in the chat interface
-- Choose your preferred retrieval method:
-  - **Hybrid**: Combines FAISS and BM25 (recommended)
-  - **Dense**: Uses only FAISS vector similarity
-  - **Sparse**: Uses only BM25 keyword matching
-
-### 3. View Results
-- See the generated answer
-- View search results with confidence scores
-- Check response time and method used
-
-## ⚙️ Configuration
-
-### Environment Variables
-
-You can customize the system by setting these environment variables:
-
-```bash
-# Model configuration
-EMBEDDING_MODEL=all-MiniLM-L6-v2
-GENERATIVE_MODEL=Qwen/Qwen2.5-1.5B-Instruct
-
-# Chunk sizes for document processing
-CHUNK_SIZES=100,400
-
-# Vector store path
-VECTOR_STORE_PATH=./vector_store
-```
-
-### Model Options
-
-**Embedding Models:**
-- `all-MiniLM-L6-v2` (default, 384 dimensions)
-- `all-mpnet-base-v2` (768 dimensions)
-- `multi-qa-MiniLM-L6-cos-v1` (384 dimensions)
-
-**Generative Models:**
-- `Qwen/Qwen2.5-1.5B-Instruct` (default)
-- `distilgpt2` (fallback)
-- `microsoft/DialoGPT-medium`
-
-## 🔧 Customization
-
-### Adding New Models
-
-To use different models, modify the `SimpleRAGSystem` initialization in `app.py`:
-
-```python
-st.session_state.rag_system = SimpleRAGSystem(
-    embedding_model="your-embedding-model",
-    generative_model="your-generative-model"
-)
-```
-
-### Custom Chunk Sizes
-
-Modify the chunk sizes for different document types:
-
-```python
-chunk_sizes = [50, 200, 800]  # Smaller chunks for technical docs
-```
-
-### Custom Search Methods
-
-Add new search methods in `rag_system.py`:
-
-```python
-def custom_search(self, query: str, top_k: int = 5):
-    # Your custom search implementation
-    pass
-```
-
-## 📊 Performance Optimization
-
-### Memory Usage
-- Use smaller embedding models for limited memory
-- Reduce chunk sizes for large documents
-- Enable model quantization
-
-### Speed Optimization
-- Use GPU acceleration when available
-- Optimize FAISS index parameters
-- Cache embeddings for repeated queries
-
-### Storage
-- FAISS index and metadata are saved locally
-- Consider cloud storage for production deployments
-
-## 🐛 Troubleshooting
-
-### Common Issues
-
-1. **Model Loading Errors**
-   - Check internet connection for model downloads
-   - Verify model names are correct
-   - Ensure sufficient disk space
-
-2. **Memory Issues**
-   - Reduce batch sizes
-   - Use smaller models
-   - Enable gradient checkpointing
-
-3. **PDF Processing Errors**
-   - Verify PDF files are not corrupted
-   - Check file permissions
-   - Ensure PyPDF is properly installed
-
-### Debug Mode
-
-Enable debug logging by adding to `app.py`:
-
-```python
-import logging
-logging.basicConfig(level=logging.DEBUG)
-```
-
-## 🔒 Security Considerations
+---
+title: RAG System with PDF Documents
+emoji: 🤖
+colorFrom: blue
+colorTo: purple
+sdk: docker
+sdk_version: latest
+app_file: app.py
+pinned: false
+---
 
-- **File Upload**: Validate PDF files before processing
-- **Model Access**: Use appropriate model access tokens
-- **Data Privacy**: Consider data retention policies
-- **Rate Limiting**: Implement query rate limiting for production
+# RAG System - Hugging Face Spaces
 
-## 📈 Monitoring
+A comprehensive Retrieval-Augmented Generation (RAG) system that processes PDF documents and answers questions using advanced AI models.
 
-### System Metrics
-- Document count and chunk count
-- Response times
-- Search result quality
-- Model performance
+## Features
 
-### Logs
-- Application logs in Streamlit
-- Model loading and inference logs
-- Error tracking and debugging
+- **PDF Processing**: Automatically loads and processes PDF documents
+- **Hybrid Search**: Combines FAISS vector search with BM25 keyword search
+- **Multiple Retrieval Methods**: Hybrid, dense, and sparse retrieval options
+- **Advanced AI Models**: Uses Qwen 2.5 1.5B for response generation
+- **Real-time Chat Interface**: Interactive Streamlit-based UI
+- **Parallel Document Loading**: Fast document processing with concurrent loading
 
-## 🤝 Contributing
+## How to Use
 
-1. Fork the repository
-2. Create a feature branch
-3. Make your changes
-4. Test thoroughly
-5. Submit a pull request
+1. **Wait for Initialization**: The system automatically loads pre-configured PDF documents
+2. **Ask Questions**: Use the chat interface to ask questions about the documents
+3. **Choose Method**: Select from hybrid, dense, or sparse retrieval methods
+4. **View Results**: See answers with confidence scores and search results
 
-## 📄 License
+## Technology Stack
 
-This project is licensed under the MIT License - see the LICENSE file for details.
+- **Vector Database**: FAISS for efficient similarity search
+- **Sparse Retrieval**: BM25 for keyword-based search
+- **Embedding Model**: all-MiniLM-L6-v2 for document embeddings
+- **Generative Model**: Qwen 2.5 1.5B for answer generation
+- **UI Framework**: Streamlit for interactive interface
+- **Containerization**: Docker for deployment
 
-## 🆘 Support
+## Configuration
 
-For issues and questions:
-1. Check the troubleshooting section
-2. Review the logs for error messages
-3. Create an issue on GitHub
-4. Contact the maintainers
+The system is pre-configured with RIL quarterly reports and automatically loads them on startup. Users can also upload additional PDF documents through the interface.
 
-## 🎯 Roadmap
+## Performance
 
-- [ ] Add support for more document formats
-- [ ] Implement advanced search algorithms
-- [ ] Add model fine-tuning capabilities
-- [ ] Improve UI/UX design
-- [ ] Add export/import functionality
-- [ ] Implement user authentication
-- [ ] Add analytics dashboard
+- **Parallel Processing**: Documents are loaded concurrently for faster initialization
+- **Optimized Search**: Hybrid retrieval combines the best of vector and keyword search
+- **Memory Efficient**: Uses CPU-optimized models for deployment compatibility
 
 ---
 
-**Happy RAG-ing! 🚀**
+*Built with ❤️ for efficient document question-answering*