| --- |
| title: GenAI Hackathon |
| emoji: 🔥 |
| colorFrom: indigo |
| colorTo: green |
| sdk: gradio |
| sdk_version: 6.19.0 |
| python_version: '3.13' |
| app_file: app.py |
| pinned: false |
| --- |
| |
| # 👁️ HybridSight |
|
|
| > A multimodal AI assistant that combines Retrieval-Augmented Generation (RAG), Web Search, and Vision into a single application through a custom reasoning agent. |
|
|
| HybridSight is a lightweight GenAI application capable of answering questions from uploaded documents, uploaded images, and the web. Instead of relying on a fixed pipeline, it uses a custom-built reasoning agent that dynamically decides which tool(s) should be used for every query while exposing its reasoning process to the user. |
|
|
| --- |
|
|
| ## ✨ Features |
|
|
| ### 💬 Hybrid Chat |
| - Free-form conversational interface |
| - Dynamic tool selection using a custom reasoning agent |
| - Supports document retrieval, web search, and image understanding |
| - Multi-tool reasoning for complex queries |
| - Collapsible reasoning trace for every response |
|
|
| ### 📄 Document QA |
| - PDF upload and indexing into ChromaDB |
| - Semantic retrieval using sentence embeddings |
| - Multi-turn document conversations |
| - Streaming responses |
| - Source-aware answers |
|
|
| ### 🖼️ Image Studio |
| - Image upload with adjustable resolution |
| - Vision-based question answering |
| - Image metadata preview |
| - Token-aware image preprocessing |
| - Automatic validation before sending images to the Vision model |
|
|
| --- |
|
|
| # 🏗️ Architecture |
|
|
| ``` |
| User |
| │ |
| ┌──────────────┼──────────────┐ |
| │ │ │ |
| ▼ ▼ ▼ |
| Hybrid Chat Document QA Image Studio |
| │ │ │ |
| ▼ ▼ ▼ |
| Custom Agent RAG Engine Vision Pipeline |
| │ │ │ |
| ├──────── search_documents ───┤ |
| ├────────── web_search ───────┤ |
| └──────── describe_image ─────┘ |
| │ |
| ▼ |
| Groq API |
| ``` |
|
|
| --- |
|
|
| # ⚙️ Technology Stack |
|
|
| | Component | Technology | |
| |------------|------------| |
| | LLM | Llama 3.3 70B (Groq) | |
| | Vision Model | Llama 4 Scout Vision | |
| | Framework | Gradio | |
| | Retrieval | LangChain | |
| | Vector Database | ChromaDB | |
| | Embeddings | all-MiniLM-L6-v2 | |
| | Search | DuckDuckGo Search | |
| | Image Processing | Pillow | |
|
|
| --- |
|
|
| # 📂 Project Structure |
|
|
| ``` |
| HybridSight |
| │ |
| ├── app.py |
| ├── agent.py |
| ├── tools.py |
| ├── rag.py |
| ├── retrival.py |
| ├── indexing.py |
| ├── vision.py |
| ├── safety.py |
| ├── requirements.txt |
| ├── README.md |
| └── .gitignore |
| ``` |
|
|
| --- |
|
|
| # 🚀 Running Locally |
|
|
| Clone the repository |
|
|
| ```bash |
| git clone <repository-url> |
| ``` |
|
|
| Install dependencies |
|
|
| ```bash |
| pip install -r requirements.txt |
| ``` |
|
|
| Create a `.env` file |
|
|
| ``` |
| GROQ_API_KEY=YOUR_API_KEY |
| ``` |
|
|
| Run the application |
|
|
| ```bash |
| python app.py |
| ``` |
|
|
| --- |
|
|
|
|
| # 🧪 Tested Scenarios |
|
|
| ✅ Pure document questions |
|
|
| ✅ Pure web-search questions |
|
|
| ✅ Pure image questions |
|
|
| ✅ Image + document reasoning |
|
|
| ✅ Graceful handling of API failures |
|
|
| ✅ Missing document handling |
|
|
| ✅ Missing image handling |
|
|
| ✅ Session-specific document retrieval |
|
|
| --- |
|
|
| # 💡 Engineering Decisions |
|
|
| ## Custom Reasoning Agent |
|
|
| Instead of relying on a fixed workflow, HybridSight implements its own reasoning loop. |
|
|
| The agent repeatedly: |
|
|
| ``` |
| Question |
| │ |
| Planner |
| │ |
| Tool |
| │ |
| Observation |
| │ |
| Planner |
| │ |
| Final Answer |
| ``` |
|
|
| This provides full control over: |
|
|
| - tool routing |
| - stopping conditions |
| - reasoning traces |
| - memory |
| - runtime behaviour |
| - tool-specific execution |
|
|
| It also makes it straightforward to add new tools without changing the overall architecture. |
|
|
| --- |
|
|
| ## Runtime Image Placeholder |
|
|
| A common challenge with multimodal agents is handling image data efficiently. |
|
|
| Instead of sending the complete Base64 image to the planner model, the planner reasons only with a lightweight placeholder: |
|
|
| ``` |
| CURRENT_IMAGE |
| ``` |
|
|
| During execution, the runtime replaces this placeholder with the actual image before invoking the vision tool. |
|
|
| ``` |
| Planner |
| │ |
| CURRENT_IMAGE |
| │ |
| Runtime |
| │ |
| Vision Tool |
| ``` |
|
|
| This approach: |
|
|
| - reduces prompt size |
| - lowers token usage |
| - avoids unnecessary serialization |
| - keeps reasoning independent of image size |
| - prevents duplicate image processing |
|
|
| --- |
|
|
| ## Token-Aware Vision Pipeline |
|
|
| The resolution slider is not merely a UI feature. |
|
|
| Large images generate significantly larger Base64 payloads, increasing latency, token consumption, and the likelihood of request failures. |
|
|
| Before an image reaches the Vision model, HybridSight performs: |
|
|
| ``` |
| Upload |
| │ |
| Resize |
| │ |
| JPEG Compression |
| │ |
| Token Estimation |
| │ |
| Metadata Generation |
| │ |
| Validation |
| │ |
| Vision Model |
| ``` |
|
|
| Only images that satisfy the configured token threshold are analyzed. |
|
|
| This protects the application from oversized requests while improving responsiveness. |
|
|
| --- |
|
|
| ## Dedicated Document QA |
|
|
| Hybrid Chat and Document QA serve different purposes. |
|
|
| Hybrid Chat focuses on open-ended multimodal reasoning. |
|
|
| ``` |
| Question |
| │ |
| Agent |
| │ |
| Multiple Tools |
| │ |
| Answer |
| ``` |
|
|
| Document QA is optimized specifically for retrieval. |
|
|
| ``` |
| PDF |
| │ |
| Embedding |
| │ |
| ChromaDB |
| │ |
| Retriever |
| │ |
| LLM |
| ``` |
|
|
| Separating these workflows keeps document conversations faster, source-grounded, and free from unnecessary planner overhead. |
|
|
| --- |
|
|
| ## Session Isolation |
|
|
| Each user receives an independent ChromaDB collection. |
|
|
| ``` |
| collection_name = session_id |
| ``` |
|
|
| This ensures uploaded documents remain isolated between users and prevents retrieval across sessions. |
|
|
| --- |
|
|
| ## Reasoning Trace |
|
|
| Every tool invocation is recorded and displayed. |
|
|
| For each reasoning step, the application exposes: |
|
|
| - selected tool |
| - tool input |
| - tool output |
|
|
| This makes the decision-making process transparent and simplifies debugging. |
|
|
| --- |
|
|
| ## Unified Safety Layer |
|
|
| External dependencies are wrapped with a unified safety decorator. |
|
|
| Instead of terminating the application, common failures are handled gracefully, including: |
|
|
| - rate limits |
| - invalid API keys |
| - network errors |
| - JSON parsing failures |
| - timeout errors |
| - missing files |
| - invalid tool requests |
|
|
| This keeps the UI responsive even when external services fail. |
|
|
| --- |
|
|
| ## Progressive Document Indexing |
|
|
| Document indexing provides real-time progress updates. |
|
|
| ``` |
| Loading PDF |
| ↓ |
| Splitting |
| ↓ |
| Preparing Metadata |
| ↓ |
| Generating Embeddings |
| ↓ |
| Writing ChromaDB |
| ↓ |
| Completed |
| ``` |
|
|
| This improves user feedback, especially for larger documents. |
|
|
| --- |
|
|
| # 🔮 Future Improvements |
|
|
| - Multi-document collections |
| - OCR support for scanned PDFs |
| - Citation highlighting |
| - Streaming reasoning traces |
| - Persistent cloud memory |
| - Additional knowledge tools (Wikipedia, ArXiv, etc.) |
| - Parallel tool execution |
|
|
| --- |
|
|
|
|
|
|
