|
|
--- |
|
|
license: mit |
|
|
tags: |
|
|
- rag |
|
|
- multimodal |
|
|
- faiss |
|
|
- sentence-transformers |
|
|
- clip |
|
|
- mistral |
|
|
- information-retrieval |
|
|
--- |
|
|
|
|
|
# Multimodal RAG System |
|
|
|
|
|
This repository contains a complete Multimodal Retrieval-Augmented Generation (RAG) system that combines text and image search with LLM-based answer generation. |
|
|
|
|
|
## System Components |
|
|
|
|
|
- **Text Embeddings**: Sentence-BERT (all-MiniLM-L6-v2) - 384 dimensions |
|
|
- **Image Embeddings**: CLIP (ViT-B/32) - 512 dimensions |
|
|
- **Vector Database**: FAISS indices for efficient similarity search |
|
|
- **LLM**: Mistral-7B-Instruct (4-bit quantized) |
|
|
- **Total Vectors**: 446 (161 text + 285 images) |
|
|
|
|
|
## Files |
|
|
|
|
|
- `text_index.faiss`: FAISS index for text embeddings |
|
|
- `image_index.faiss`: FAISS index for image embeddings |
|
|
- `text_metadata.pkl`: Metadata for text chunks (source, page, content) |
|
|
- `image_metadata.pkl`: Metadata for images (source, page, image_id) |
|
|
- `config.json`: System configuration |
|
|
- `image_summary.json`: Reference summary of images |
|
|
|
|
|
## Usage |
|
|
|
|
|
See the load cells in the notebook for loading and using this RAG system. |
|
|
|
|
|
## Features |
|
|
|
|
|
- Semantic text search |
|
|
- Cross-modal image search (text query → image results) |
|
|
- Multiple prompting strategies (Standard, Chain-of-Thought, Few-shot, Zero-shot) |
|
|
- Source attribution and traceability |
|
|
- Real-time answer generation |
|
|
|
