README.md

metadata

title: Multimodal Rag Nemotron
emoji: ⚡
colorFrom: green
colorTo: gray
sdk: gradio
sdk_version: 6.2.0
python_version: '3.11'
app_file: app.py
pinned: false
short_description: End-to-end Multimodal RAG for recipe discovery.

👁️📑 Multimodal RAG with Nemotron & LangGraph

This repository contains a full-fledged Multimodal Retrieval-Augmented Generation (RAG) application built with state-of-the-art vision-language models, vector databases, and semantic caching.

The application allows users to search for recipes (both through text and image queries), retrieve relevant recipe documents, rerank them for precision, and optionally generate an AI-powered summary—all orchestrated through LangGraph and served via a Gradio interface.

---

✨ Features

• Multimodal Retrieval: Search recipes using either text queries or by uploading an image.
• Advanced Embeddings & Reranking: Utilizes NVIDIA's Llama-Nemotron Vision-Language models for accurate, multimodal document matching and reranking.
• Semantic Caching: Integrates RedisVL to instantly return cached results for semantically similar queries, saving time and compute resources. The vector distance threshold is dynamically adjustable via the UI.
• Agentic Workflow: The entire pipeline (Cache Check -> Pinecone Retrieval -> Reranking -> Generation -> Cache Update) is orchestrated as a state machine using LangGraph.
• Parametric UI: Fully interactive Gradio interface with adjustable sliders for Retrieval Top-K and Cache Thresholds, alongside optional Rerank and Summary generation toggles.

---

🛠️ Architecture & Tech Stack

Models

• Embedding: nvidia/llama-nemotron-embed-vl-1b-v2
• Reranking: nvidia/llama-nemotron-rerank-vl-1b-v2
• Generation: Qwen/Qwen3-VL-2B-Instruct

Infrastructure

• Vector Store: Pinecone (for storing and querying recipe vectors)
• Semantic Cache: Redis / RedisVL (for caching responses to avoid redundant compute)
• Orchestration: LangGraph
• UI: Gradio
• Dataset: mrdbourke/recipe-synthetic-images-10k (HuggingFace)

---

🚀 Getting Started

Prerequisites

You need a machine with a CUDA-compatible GPU (VRAM requirement ~16GB+ depending on the models used). Ensure you have Python 3.10+ installed.

1. Clone the repository

git clone https://github.com/devsu88/multimodal-rag-nemotron.git
cd multimodal-rag-nemotron

2. Install Dependencies

You can install the required packages using pip (a requirements.txt should be generated, but here are the core dependencies):

pip install torch transformers datasets pinecone-client redisvl python-dotenv gradio langgraph pillow numpy

3. Environment Variables

Create a .env file in the root directory of the project and populate it with your API keys and connection strings:

PINECONE_API_KEY="your_pinecone_api_key_here"
PINECONE_INDEX_NAME="your_pinecone_index_name"
REDIS_URL="redis://your_redis_host:port"
REDIS_DBNAME="your_redis_index_name"

4. Populate Pinecone (One-time setup)

Before running the application for the first time, you need to embed and upload the dataset to your Pinecone index running:

python ingest_to_pinecone.py

(Note: Make sure your Pinecone index is created with 2048 dimensions and the cosine metric before running the population script).

5. Run the Application

Start the Gradio server by running:

python app.py

The server will automatically download the dataset and the HuggingFace models upon the first execution. Once loaded, a local URL (usually http://127.0.0.1:7860) will be provided in the terminal.

---

📂 Project Structure

• config.py: Environment loading and global configuration constants.
• models.py: Initialization of Pinecone, Redis, and loading of HuggingFace models & processors into GPU memory.
• graph.py: The LangGraph state definitions, nodes, and conditional edges that orchestrate the RAG flow.
• utils.py: Helper functions for LLM generation and HTML formatting.
• app.py: The presentation layer; contains the Gradio UI definitions and the main entry point.
• ingest_to_pinecone.py: The script to embed and upload the dataset to Pinecone.