merge local branch

.env.example

@@ -0,0 +1,22 @@
+# Qdrant Cloud Configuration (required)
+QDRANT_URL=https://your-cluster.qdrant.tech
+QDRANT_API_KEY=your-api-key
+
+# HuggingFace Token (optional but recommended for faster downloads)
+HF_TOKEN=your-huggingface-token
+
+# LLM Provider (default: huggingface)
+LLM_PROVIDER=huggingface
+
+# HuggingFace Model (default: Sarvam-1 for Indic languages)
+HUGGINGFACE_MODEL=sarvamai/sarvam-1
+
+# Generation Parameters
+TEMPERATURE=0.7
+MAX_NEW_TOKENS=1024
+REPETITION_PENALTY=1.1
+TOP_P=0.9
+TOP_K=50
+
+# Backend URL for frontend
+BACKEND_URL=http://localhost:8000

.gitignore

@@ -0,0 +1,51 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+env/
+venv/
+ENV/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Environment
+.env
+.env.local
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Qdrant
+qdrant_storage/
+
+# Logs
+*.log
+
+# Model cache
+.cache/
+models/
+
+# Dataset hashes
+hf_datasets_hashes.json

README.md

@@ -1,10 +1,245 @@
 ---
-title: IndicRAG
-emoji: 🏃
-colorFrom: blue
-colorTo: pink
+title: HindiRAG
+emoji: 💻
+colorFrom: pink
+colorTo: purple
 sdk: docker
+app_port: 7860
+sdk_version: latest
+app_file: Dockerfile
 pinned: false
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+# Simple RAG
+
+A simplified Retrieval-Augmented Generation (RAG) system for Indic languages using Sarvam-1 model and Qdrant Cloud.
+
+## Features
+
+- **10 Indic Languages Support**: Hindi, Bengali, Gujarati, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu
+- **User-Selected Language**: Choose your preferred language from the UI
+- **Sarvam-1 Model**: Optimized for Indic language generation
+- **Qdrant Cloud**: Managed vector database service
+- **Simple UI**: Clean interface with just query input and response area
+- **No Language Detection**: Direct query processing without auto-detection
+- **HuggingFace Datasets**: Load data directly from HuggingFace (no local files needed)
+
+## Supported Languages
+
+| Language | Native Name | Code |
+|----------|-------------|------|
+| Hindi | हिंदी | hi |
+| Bengali | বাংলা | bn |
+| Gujarati | ગુજરાતી | gu |
+| Kannada | ಕನ್ನಡ | kn |
+| Malayalam | മലയാളം | ml |
+| Marathi | मराठी | mr |
+| Odia | ଓଡ଼ିଆ | or |
+| Punjabi | ਪੰਜਾਬੀ | pa |
+| Tamil | தமிழ் | ta |
+| Telugu | తెలుగు | te |
+
+## Quick Start
+
+### 1. Clone and Setup
+
+```bash
+cd simple-rag
+```
+
+### 2. Configure Environment
+
+```bash
+cp .env.example .env
+```
+
+Edit `.env` and add your Qdrant Cloud credentials:
+```bash
+QDRANT_URL=https://your-cluster.qdrant.tech
+QDRANT_API_KEY=your-api-key
+```
+
+### 3. Install Dependencies
+
+```bash
+# Backend
+cd backend
+pip install -r requirements.txt
+
+# Frontend (in a new terminal)
+cd ../frontend
+pip install -r requirements.txt
+```
+
+### 4. Load Data to Qdrant (Optional)
+
+If your Qdrant collection is empty, you can load data from HuggingFace datasets:
+
+```bash
+cd backend
+
+# Set the datasets you want to load (MIRACL corpus for all 10 Indic languages)
+export HF_DATASETS=miracl/miracl-corpus:hi:train,miracl/miracl-corpus:bn:train,miracl/miracl-corpus:ta:train,miracl/miracl-corpus:te:train,miracl/miracl-corpus:mr:train,miracl/miracl-corpus:gu:train,miracl/miracl-corpus:kn:train,miracl/miracl-corpus:ml:train,miracl/miracl-corpus:or:train,miracl/miracl-corpus:pa:train
+
+# Run the data loader
+python src/load_huggingface_dataset.py
+```
+
+### 5. Run the Application
+
+```bash
+# Terminal 1: Start Backend
+cd backend
+python main.py
+
+# Terminal 2: Start Frontend
+cd frontend
+streamlit run app.py
+```
+
+Then open http://localhost:8501 in your browser.
+
+## Architecture
+
+```
+┌─────────────────┐     ┌──────────────┐     ┌─────────────────┐
+│   User Query    │────▶│  Qdrant Cloud│────▶│  Sarvam-1 LLM   │
+│  (Selected Lang)│     │ Vector DB    │     │  (Generation)   │
+└─────────────────┘     └──────────────┘     └─────────────────┘
+       │                      │
+       │                      │
+       ▼                      ▼
+┌─────────────────┐     ┌─────────────────┐
+│  Streamlit UI   │     │   Embedding     │
+│  (Frontend)     │     │   Generator     │
+└─────────────────┘     └─────────────────┘
+```
+
+## Project Structure
+
+```
+simple-rag/
+├── backend/
+│   ├── src/
+│   │   ├── rag_system.py           # Simplified RAG system
+│   │   ├── llm_manager.py          # Sarvam-1 LLM management
+│   │   ├── embedding_generator.py  # Embedding generation
+│   │   ├── qdrant_setup.py         # Qdrant Cloud setup
+│   │   ├── language_constants.py   # Supported languages config
+│   │   └── load_huggingface_dataset.py  # HF dataset loader
+│   ├── main.py                     # FastAPI application
+│   ├── requirements.txt
+│   └── Dockerfile
+├── frontend/
+│   ├── app.py                      # Streamlit frontend
+│   ├── requirements.txt
+│   └── Dockerfile
+├── docker-compose.yml
+├── .env.example
+└── README.md
+```
+
+## API Endpoints
+
+### POST /query
+Query the RAG system.
+
+```json
+{
+  "query": "प्रकृति का वर्णन कैसे किया गया है?",
+  "top_k": 5,
+  "language": "hi"
+}
+```
+
+Response:
+```json
+{
+  "question": "प्रकृति का वर्णन कैसे किया गया है?",
+  "answer": "...",
+  "user_selected_language": "hi",
+  "supported_languages": [...]
+}
+```
+
+### GET /languages
+Get list of supported languages.
+
+### GET /health
+Health check endpoint.
+
+## Docker Deployment
+
+```bash
+# Set environment variables
+export QDRANT_URL=https://your-cluster.qdrant.tech
+export QDRANT_API_KEY=your-api-key
+
+# Build and run all services
+docker-compose up --build
+```
+
+## Differences from HindiRAG
+
+- **No Language Detection**: User selects language manually
+- **Simplified UI**: No sidebar, no document display, no document count
+- **Direct Processing**: Query goes directly to Qdrant without language analysis
+- **LLM Language Handling**: The LLM prompt instructs it to detect and respond in the query language
+- **Separate Backend/Frontend**: FastAPI backend, Streamlit frontend
+- **Qdrant Cloud**: Uses managed Qdrant Cloud instead of local instance
+- **No Local Data**: Uses HuggingFace datasets only, no local file dependencies
+
+## Troubleshooting
+
+### LLM Initialization Failed
+- Ensure you have enough memory for Sarvam-1 model (~8GB)
+- Check internet connection for model download
+
+### Qdrant Connection Error
+- Verify your QDRANT_URL and QDRANT_API_KEY in `.env`
+- Check that your Qdrant Cloud cluster is running
+- Ensure network access to Qdrant Cloud
+
+### Backend Connection Error
+- Ensure backend is running on port 8000
+- Check `BACKEND_URL` in frontend configuration
+
+### No Documents Found
+- Make sure your Qdrant collection has documents loaded
+- Run the HuggingFace dataset loader to populate Qdrant
+
+## Setting up Qdrant Cloud
+
+1. Go to https://cloud.qdrant.io/
+2. Create a free account
+3. Create a new cluster
+4. Copy the cluster URL and API key
+5. Add them to your `.env` file
+
+## Loading HuggingFace Datasets
+
+The system uses the MIRACL corpus for Indic languages. To load data:
+
+```bash
+cd backend
+
+# Load all 10 Indic languages (506K+ documents)
+export HF_DATASETS=miracl/miracl-corpus:hi:train,miracl/miracl-corpus:bn:train,miracl/miracl-corpus:ta:train,miracl/miracl-corpus:te:train,miracl/miracl-corpus:mr:train,miracl/miracl-corpus:gu:train,miracl/miracl-corpus:kn:train,miracl/miracl-corpus:ml:train,miracl/miracl-corpus:or:train,miracl/miracl-corpus:pa:train
+
+# Or load a single language (e.g., Hindi only - 506K docs)
+export HF_DATASETS=miracl/miracl-corpus:hi:train
+
+# Run the loader
+python src/load_huggingface_dataset.py
+```
+
+## License
+
+MIT License
+
+## Acknowledgments
+
+- [Sarvam-1](https://sarvam.ai) for the Indic language model
+- [Qdrant](https://qdrant.tech) for vector database
+- [HuggingFace](https://huggingface.co) for MIRACL corpus dataset

backend/Dockerfile

@@ -0,0 +1,33 @@
+FROM python:3.10-slim
+
+WORKDIR /app
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Create non-root user
+RUN useradd -m -u 1000 appuser
+
+# Copy requirements first for better caching
+COPY requirements.txt .
+
+# Install Python dependencies (CPU-only torch)
+RUN pip install --no-cache-dir -r requirements.txt \
+    --extra-index-url https://download.pytorch.org/whl/cpu
+
+# Copy application code
+COPY . .
+
+# Change ownership to non-root user
+RUN chown -R appuser:appuser /app
+
+# Switch to non-root user
+USER appuser
+
+# Expose port
+EXPOSE 8000
+
+# Run the application
+CMD ["python", "main.py"]

backend/main.py

@@ -0,0 +1,107 @@
+"""
+FastAPI Backend for Simple RAG System
+"""
+import os
+import sys
+from pathlib import Path
+
+from fastapi import FastAPI, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel
+from typing import Optional, List, Dict, Any
+
+# Add src directory to path
+sys.path.insert(0, str(Path(__file__).parent / "src"))
+
+from rag_system import SimpleRAGSystem
+from language_constants import get_supported_languages
+
+# Initialize FastAPI app
+app = FastAPI(
+    title="Simple RAG API",
+    description="A simplified RAG system API for Indic languages",
+    version="1.0.0"
+)
+
+# Add CORS middleware
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+
+class QueryRequest(BaseModel):
+    query: str
+    top_k: int = 5
+    language: Optional[str] = None
+
+
+class QueryResponse(BaseModel):
+    question: str
+    answer: str
+    user_selected_language: Optional[str]
+    supported_languages: List[Dict[str, str]]
+
+
+# Global RAG system instance
+rag_system: Optional[SimpleRAGSystem] = None
+
+
+def get_rag_system() -> SimpleRAGSystem:
+    """Get or initialize the RAG system"""
+    global rag_system
+    if rag_system is None:
+        rag_system = SimpleRAGSystem()
+    return rag_system
+
+
+@app.get("/")
+async def root():
+    """Root endpoint"""
+    return {"message": "Simple RAG API is running", "status": "healthy"}
+
+
+@app.get("/health")
+async def health_check():
+    """Health check endpoint"""
+    return {"status": "healthy"}
+
+
+@app.get("/languages")
+async def get_languages():
+    """Get list of supported languages"""
+    return {"languages": get_supported_languages()}
+
+
+@app.post("/query", response_model=QueryResponse)
+async def query(request: QueryRequest):
+    """
+    Query the RAG system
+    
+    - query: The user's question
+    - top_k: Number of documents to retrieve (default: 5)
+    - language: Optional language code selected by user
+    """
+    try:
+        system = get_rag_system()
+        result = system.query(
+            question=request.query,
+            top_k=request.top_k,
+            user_selected_language=request.language
+        )
+        return QueryResponse(
+            question=result["question"],
+            answer=result["answer"],
+            user_selected_language=result["user_selected_language"],
+            supported_languages=result["supported_languages"]
+        )
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+
+
+if __name__ == "__main__":
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=8000)

backend/requirements.txt

@@ -0,0 +1,27 @@
+# Core dependencies
+qdrant-client>=1.7.0
+fastapi>=0.104.0
+uvicorn[standard]>=0.24.0
+pydantic>=2.4.2
+python-dotenv>=1.0.0
+
+# ML/NLP dependencies
+langchain>=0.0.331
+langchain-huggingface>=0.0.1
+sentence-transformers>=2.2.2
+torch>=2.0.1
+transformers>=4.35.0
+tokenizers>=0.14.0
+datasets>=2.14.6,<4.0.0
+numpy>=1.24.3
+pandas>=2.1.1
+huggingface-hub>=0.17.3
+accelerate>=0.23.0
+sentencepiece>=0.1.99
+
+# Utilities
+tiktoken>=0.5.1
+
+# Frontend
+streamlit>=1.28.1
+httpx>=0.25.0

backend/src/embedding_generator.py

@@ -0,0 +1,58 @@
+import os
+from typing import List
+import numpy as np
+from transformers import AutoTokenizer, AutoModel
+from dotenv import load_dotenv
+import torch
+
+load_dotenv()
+
+
+class EmbeddingGenerator:
+    def __init__(self, model_name="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"):
+        """Initialize embedding generator for multilingual text"""
+        self.model_name = model_name
+        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
+        self.model = AutoModel.from_pretrained(
+            model_name,
+            dtype=torch.float32,
+            device_map="cpu",
+            low_cpu_mem_usage=False
+        )
+        self.model = self.model.to("cpu")
+        self.model.eval()
+
+    def get_embedding(self, text: str) -> List[float]:
+        """Generate embedding for text"""
+        inputs = self.tokenizer(
+            text,
+            return_tensors="pt",
+            padding=True,
+            truncation=True,
+            max_length=512,
+            add_special_tokens=True
+        )
+
+        with torch.no_grad():
+            outputs = self.model(**inputs)
+
+        embeddings = outputs.last_hidden_state.mean(dim=1).squeeze().cpu().numpy()
+        return embeddings.tolist()
+
+    def get_embeddings_batch(self, texts: List[str]) -> List[List[float]]:
+        """Generate embeddings for a batch of texts"""
+        return [self.get_embedding(text) for text in texts]
+
+
+def get_embedding_function():
+    """Return the embedding function"""
+    embedder = EmbeddingGenerator()
+    return embedder.get_embedding
+
+
+if __name__ == "__main__":
+    embed_gen = EmbeddingGenerator()
+    test_text = "Hello world, नमस्ते दुनिया"
+    embedding = embed_gen.get_embedding(test_text)
+    print(f"Embedding length: {len(embedding)}")
+    print(f"First 10 values: {embedding[:10]}")

backend/src/language_constants.py

@@ -0,0 +1,41 @@
+"""
+Supported languages configuration for the RAG system
+No language detection - just defines supported languages
+"""
+
+SUPPORTED_LANGUAGES = {
+    "hi": {"name": "Hindi", "native_name": "हिंदी"},
+    "bn": {"name": "Bengali", "native_name": "বাংলা"},
+    "gu": {"name": "Gujarati", "native_name": "ગુજરાતી"},
+    "kn": {"name": "Kannada", "native_name": "ಕನ್ನಡ"},
+    "ml": {"name": "Malayalam", "native_name": "മലയാളം"},
+    "mr": {"name": "Marathi", "native_name": "मराठी"},
+    "or": {"name": "Odia", "native_name": "ଓଡ଼ିଆ"},
+    "pa": {"name": "Punjabi", "native_name": "ਪੰਜਾਬੀ"},
+    "ta": {"name": "Tamil", "native_name": "தமிழ்"},
+    "te": {"name": "Telugu", "native_name": "తెలుగు"},
+}
+
+
+def get_supported_languages():
+    """Get list of supported languages"""
+    return [
+        {"code": code, "name": info["name"], "native_name": info["native_name"]}
+        for code, info in SUPPORTED_LANGUAGES.items()
+    ]
+
+
+def is_language_supported(lang_code: str) -> bool:
+    """Check if a language code is supported"""
+    return lang_code in SUPPORTED_LANGUAGES
+
+
+def get_language_info(lang_code: str) -> dict:
+    """Get language information by code"""
+    if lang_code in SUPPORTED_LANGUAGES:
+        return {
+            "code": lang_code,
+            "name": SUPPORTED_LANGUAGES[lang_code]["name"],
+            "native_name": SUPPORTED_LANGUAGES[lang_code]["native_name"],
+        }
+    return {"code": lang_code, "name": lang_code, "native_name": lang_code}

backend/src/llm_manager.py

@@ -0,0 +1,100 @@
+"""
+LLM Manager module with Sarvam-1 model support for Indic languages
+Optimized for CPU-only environments
+"""
+from typing import Optional, Dict, Any
+from langchain_huggingface import HuggingFacePipeline
+from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
+from huggingface_hub import login
+import torch
+import os
+import warnings
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+warnings.filterwarnings("ignore", category=UserWarning)
+
+
+class LLMManager:
+    _instance = None
+    _llm_instance = None
+    _initialization_error = None
+
+    def __new__(cls):
+        if cls._instance is None:
+            cls._instance = super(LLMManager, cls).__new__(cls)
+        return cls._instance
+
+    def get_llm(self, provider: str = "huggingface", model_kwargs: Optional[Dict[str, Any]] = None):
+        """Get LLM instance based on provider"""
+        if self._initialization_error is not None:
+            logger.error(f"LLM initialization failed: {self._initialization_error}")
+            return None
+
+        if self._llm_instance is not None:
+            return self._llm_instance
+
+        self._llm_instance = self._get_sarvam_llm(model_kwargs)
+
+        if self._llm_instance is None:
+            logger.error("Failed to initialize Sarvam-1 LLM")
+            self._initialization_error = "Sarvam-1 initialization failed"
+
+        return self._llm_instance
+
+    def _get_sarvam_llm(self, model_kwargs: Optional[Dict[str, Any]] = None):
+        """Initialize Sarvam-1 model for Indic text generation"""
+        model_id = "sarvamai/sarvam-1"
+
+        try:
+            # Authenticate with HuggingFace if token is provided
+            hf_token = os.getenv("HF_TOKEN")
+            if hf_token:
+                logger.info("Authenticating with HuggingFace...")
+                login(token=hf_token)
+            else:
+                logger.warning("No HF_TOKEN provided. Downloads may be slower.")
+
+            logger.info(f"Initializing Sarvam-1 model: {model_id}")
+            logger.info("Sarvam-1: 2B parameters, optimized for 10 Indic languages")
+            logger.info("Loading model with CPU-first approach...")
+
+            pipe = pipeline(
+                "text-generation",
+                model=model_id,
+                model_kwargs={
+                    "torch_dtype": torch.float32,
+                    "low_cpu_mem_usage": False,
+                },
+                device_map="cpu"
+            )
+
+            logger.info("Sarvam-1 pipeline initialized successfully on CPU")
+            llm = HuggingFacePipeline(pipeline=pipe)
+            return llm
+
+        except Exception as e:
+            logger.error(f"Failed to initialize Sarvam-1 model: {e}")
+            if "meta tensor" in str(e).lower():
+                logger.error("Meta tensor error: Insufficient RAM for model loading")
+            self._initialization_error = str(e)
+            return None
+
+    def is_available(self) -> bool:
+        return self._llm_instance is not None and self._initialization_error is None
+
+    def get_initialization_error(self) -> Optional[str]:
+        return self._initialization_error
+
+
+def get_llm(provider: str = "huggingface", model_kwargs: Optional[Dict[str, Any]] = None):
+    """Convenience function to get LLM instance"""
+    manager = LLMManager()
+    return manager.get_llm(provider, model_kwargs)
+
+
+def get_llm_with_provider(provider: str = "huggingface", model_kwargs: Optional[Dict[str, Any]] = None):
+    """Get LLM with specific provider and model kwargs"""
+    manager = LLMManager()
+    return manager.get_llm(provider, model_kwargs)

backend/src/load_huggingface_dataset.py

@@ -0,0 +1,223 @@
+"""
+Load HuggingFace datasets and ingest into Qdrant
+No local file dependencies - uses only HF datasets
+"""
+import os
+import sys
+import hashlib
+import json
+from pathlib import Path
+from typing import List, Dict
+
+# Add src directory to path
+sys.path.insert(0, str(Path(__file__).parent))
+
+from qdrant_setup import QdrantSetup
+from embedding_generator import EmbeddingGenerator
+from datasets import load_dataset
+
+# Hash file to track ingested documents
+HASH_FILE = "hf_datasets_hashes.json"
+
+
+def get_dataset_hashes() -> Dict[str, str]:
+    """Load existing dataset hashes"""
+    if os.path.exists(HASH_FILE):
+        with open(HASH_FILE, "r") as f:
+            return json.load(f)
+    return {}
+
+
+def save_dataset_hashes(hashes: Dict[str, str]):
+    """Save dataset hashes"""
+    with open(HASH_FILE, "w") as f:
+        json.dump(hashes, f, indent=2)
+
+
+def compute_dataset_hash(dataset_name: str, config: str, split: str, data) -> str:
+    """Compute a hash for the dataset to detect changes"""
+    # Use dataset info for hashing
+    info = f"{dataset_name}:{config}:{split}:{len(data)}"
+    return hashlib.md5(info.encode()).hexdigest()
+
+
+def parse_dataset_spec(spec: str) -> tuple:
+    """Parse dataset specification: name:config:split"""
+    parts = spec.strip().split(":")
+    if len(parts) == 3:
+        return parts[0], parts[1], parts[2]
+    elif len(parts) == 2:
+        return parts[0], parts[1], "train"
+    else:
+        return parts[0], None, "train"
+
+
+def load_and_ingest_dataset(qdrant_client, collection_name: str, embedding_func,
+                           dataset_name: str, config: str, split: str):
+    """Load a dataset from HuggingFace and ingest into Qdrant"""
+    print(f"Loading dataset: {dataset_name} (config={config}, split={split})")
+
+    try:
+        # Load dataset
+        if config:
+            dataset = load_dataset(dataset_name, config, split=split)
+        else:
+            dataset = load_dataset(dataset_name, split=split)
+
+        print(f"  Loaded {len(dataset)} documents")
+
+        # Prepare documents for ingestion
+        texts_to_ingest = []
+        metadatas_to_ingest = []
+
+        for item in dataset:
+            # Extract text - handle different dataset formats
+            text = None
+            if "text" in item:
+                text = item["text"]
+            elif "content" in item:
+                text = item["content"]
+            elif "passage" in item:
+                text = item["passage"]
+            elif "document" in item:
+                text = item["document"]
+
+            if text and isinstance(text, str) and text.strip():
+                texts_to_ingest.append(text)
+
+                # Extract metadata
+                metadata = {
+                    "title": item.get("title", "") or "",
+                    "author": item.get("author", "") or "",
+                    "genre": item.get("genre", "") or "",
+                    "source": f"{dataset_name}:{config}:{split}",
+                }
+
+                # Add language info if available
+                if "language" in item:
+                    metadata["language"] = item["language"]
+
+                metadatas_to_ingest.append(metadata)
+
+        if not texts_to_ingest:
+            print(f"  No valid texts found in dataset")
+            return 0
+
+        print(f"  Found {len(texts_to_ingest)} valid texts to ingest")
+
+        # Ingest in batches
+        batch_size = 100
+        ingested_count = 0
+
+        from qdrant_client.http import models
+
+        for i in range(0, len(texts_to_ingest), batch_size):
+            batch_texts = texts_to_ingest[i:i + batch_size]
+            batch_metadatas = metadatas_to_ingest[i:i + batch_size]
+
+            # Generate embeddings
+            embeddings = []
+            for text in batch_texts:
+                embedding = embedding_func(text)
+                embeddings.append(embedding)
+
+            # Create points
+            points = []
+            for j, (text, metadata, embedding) in enumerate(zip(batch_texts, batch_metadatas, embeddings)):
+                point = models.PointStruct(
+                    id=ingested_count + j,
+                    vector=embedding,
+                    payload={
+                        "full_text": text,
+                        **metadata
+                    }
+                )
+                points.append(point)
+
+            # Upload to Qdrant
+            qdrant_client.upsert(
+                collection_name=collection_name,
+                points=points
+            )
+
+            ingested_count += len(batch_texts)
+            print(f"  Ingested {ingested_count}/{len(texts_to_ingest)} documents")
+
+        print(f"  ✓ Successfully ingested {ingested_count} documents")
+        return ingested_count
+
+    except Exception as e:
+        print(f"  Error loading dataset: {e}")
+        return 0
+
+
+def main():
+    """Main function to load and ingest all configured datasets"""
+    print("=" * 60)
+    print("HuggingFace Dataset Loader for Simple RAG")
+    print("=" * 60)
+
+    # Get datasets from environment
+    hf_datasets = os.getenv("HF_DATASETS", "")
+
+    if not hf_datasets:
+        print("No HF_DATASETS environment variable set.")
+        print("Set HF_DATASETS to load datasets (e.g., miracl/miracl-corpus:hi:train)")
+        return
+
+    # Initialize Qdrant
+    print("\nInitializing Qdrant...")
+    qdrant_setup = QdrantSetup()
+    qdrant_client = qdrant_setup.get_client()
+    collection_name = qdrant_setup.get_collection_name()
+
+    # Create collection if not exists
+    qdrant_setup.create_collection()
+
+    # Initialize embedding generator
+    print("Initializing embedding generator...")
+    embedding_func = EmbeddingGenerator().get_embedding
+
+    # Load existing hashes
+    dataset_hashes = get_dataset_hashes()
+
+    # Parse and process datasets
+    dataset_specs = [spec.strip() for spec in hf_datasets.split(",")]
+
+    total_ingested = 0
+    for spec in dataset_specs:
+        if not spec:
+            continue
+
+        dataset_name, config, split = parse_dataset_spec(spec)
+        dataset_key = f"{dataset_name}:{config}:{split}"
+
+        # Check if already ingested
+        if dataset_key in dataset_hashes:
+            print(f"\nSkipping {dataset_key} (already ingested)")
+            continue
+
+        print(f"\nProcessing: {dataset_key}")
+
+        # Load and ingest
+        count = load_and_ingest_dataset(
+            qdrant_client, collection_name, embedding_func,
+            dataset_name, config, split
+        )
+
+        if count > 0:
+            # Save hash
+            dataset_hashes[dataset_key] = compute_dataset_hash(
+                dataset_name, config, split,
+                {"count": count}
+            )
+            save_dataset_hashes(dataset_hashes)
+            total_ingested += count
+
+    print("\n" + "=" * 60)
+    print(f"Total documents ingested: {total_ingested}")
+    print("=" * 60)
+
+
+if __name__ == "__main__":
+    main()

backend/src/qdrant_setup.py

@@ -0,0 +1,59 @@
+import qdrant_client
+from qdrant_client.http import models
+from qdrant_client.http.models import Distance, VectorParams
+import os
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+class QdrantSetup:
+    def __init__(self, host=None, port=None, api_key=None, https=True):
+        """
+        Initialize Qdrant client - supports both local and cloud instances
+        """
+        cloud_url = os.getenv("QDRANT_URL")
+        cloud_api_key = os.getenv("QDRANT_API_KEY")
+
+        if cloud_url:
+            self.client = qdrant_client.QdrantClient(
+                url=cloud_url,
+                api_key=cloud_api_key,
+                https=https
+            )
+        else:
+            host = host or os.getenv("QDRANT_HOST", "localhost")
+            port = port or int(os.getenv("QDRANT_PORT", 6333))
+            self.client = qdrant_client.QdrantClient(
+                host=host,
+                port=port
+            )
+
+        self.collection_name = "hindi_poems_stories"
+
+    def create_collection(self, vector_size=384):
+        """Create a collection in Qdrant for storing text embeddings"""
+        collections = self.client.get_collections()
+        collection_names = [col.name for col in collections.collections]
+
+        if self.collection_name in collection_names:
+            print(f"Collection '{self.collection_name}' already exists.")
+            return
+
+        self.client.create_collection(
+            collection_name=self.collection_name,
+            vectors_config=VectorParams(size=vector_size, distance=Distance.COSINE),
+        )
+        print(f"Collection '{self.collection_name}' created successfully.")
+
+    def get_client(self):
+        return self.client
+
+    def get_collection_name(self):
+        return self.collection_name
+
+
+if __name__ == "__main__":
+    qdrant_setup = QdrantSetup()
+    qdrant_setup.create_collection()
+    print("Qdrant setup completed!")

backend/src/rag_system.py

@@ -0,0 +1,195 @@
+"""
+Simplified RAG system - No language detection
+Passes query directly to Qdrant and LLM
+"""
+import os
+import logging
+from typing import List, Dict, Optional
+
+from qdrant_setup import QdrantSetup
+from embedding_generator import EmbeddingGenerator
+from llm_manager import get_llm
+from language_constants import (
+    get_supported_languages,
+    is_language_supported,
+    get_language_info,
+    SUPPORTED_LANGUAGES,
+)
+from langchain_core.output_parsers import StrOutputParser
+from qdrant_client.http import models
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+# Simplified prompt - checks if language is supported, then processes with context
+QA_PROMPT = """You are a helpful assistant for literature and poetry.
+
+IMPORTANT: Check if the query language is supported. The supported languages are:
+{supported_languages_list}
+
+If the query is in a supported language:
+- Respond in the SAME language as the query
+- Use the provided context to answer accurately
+- If the answer is not in the context, say "I don't know based on the available documents" in the query language
+- Do not hallucinate. Provide comprehensive answers based on context.
+
+If the query is NOT in a supported language:
+- Respond in English saying you only support the listed languages
+
+Context:
+{context_str}
+
+Query: {query}
+
+Answer:"""
+
+UNSUPPORTED_LANG_PROMPT = """I apologize, but I only support the following languages:
+
+{supported_languages_list}
+
+Please try asking your question in one of these supported languages."""
+
+
+class SimpleRAGSystem:
+    def __init__(self, llm_provider=None, model_kwargs=None):
+        """Initialize the simplified RAG system"""
+        # Setup Qdrant client
+        qdrant_setup = QdrantSetup()
+        self.qdrant_client = qdrant_setup.get_client()
+        self.collection_name = qdrant_setup.get_collection_name()
+
+        # Setup embedding generator
+        self.embedding_generator = EmbeddingGenerator()
+
+        # Setup LLM
+        if llm_provider or model_kwargs:
+            self.llm = get_llm(provider=llm_provider, model_kwargs=model_kwargs)
+        else:
+            self.llm = get_llm()
+
+        if self.llm is None:
+            logger.error("LLM initialization failed. RAG system will have limited functionality.")
+
+        self.supported_languages = get_supported_languages()
+        self.supported_language_codes = set(lang["code"] for lang in self.supported_languages)
+
+    def _get_supported_languages_list(self) -> str:
+        """Get formatted list of supported languages"""
+        return "\n".join([
+            f"  - {lang['name']} ({lang['native_name']})"
+            for lang in self.supported_languages
+        ])
+
+    def retrieve_relevant_documents(self, query: str, top_k: int = 5) -> List[Dict]:
+        """Retrieve relevant documents from Qdrant based on the query"""
+        query_embedding = self.embedding_generator.get_embedding(query)
+
+        search_result = self.qdrant_client.query_points(
+            collection_name=self.collection_name,
+            query=query_embedding,
+            limit=top_k
+        )
+
+        retrieved_docs = []
+        for result in search_result.points:
+            payload = result.payload if result.payload is not None else {}
+            doc = {
+                "score": result.score or 0,
+                "title": payload.get("title", "") if payload else "",
+                "author": payload.get("author", "") if payload else "",
+                "genre": payload.get("genre", "") if payload else "",
+                "text": payload.get("full_text", "") if payload else "",
+                "source_file": payload.get("source_file", "") if payload else "",
+            }
+            retrieved_docs.append(doc)
+
+        return retrieved_docs
+
+    def generate_answer(self, query: str, context_docs: List[Dict], user_selected_language: Optional[str] = None) -> str:
+        """
+        Generate an answer based on the query and retrieved documents
+        
+        Args:
+            query: User query
+            context_docs: Retrieved context documents
+            user_selected_language: Optional language code selected by user in frontend
+        """
+        if self.llm is None:
+            logger.error("LLM not initialized. Returning synthesized answer from documents.")
+            return self._synthesize_answer(query, context_docs)
+
+        # Use user-selected language or try to infer from query context
+        # For simplicity, we just pass the query and let the LLM handle it
+        # The prompt instructs the LLM to respond in the same language as the query
+        
+        limited_docs = context_docs[:5] if len(context_docs) > 5 else context_docs
+
+        # Format context
+        formatted_contexts = []
+        for i, doc in enumerate(limited_docs, 1):
+            text_snippet = doc["text"][:400] + "..." if len(doc["text"]) > 400 else doc["text"]
+            formatted_context = f"[{i}] Title: {doc['title']}\nAuthor: {doc['author']}\nGenre: {doc['genre']}\nContent: {text_snippet}\nScore: {doc['score']:.3f}\n"
+            formatted_contexts.append(formatted_context)
+
+        context_str = "\n\n".join(formatted_contexts)
+
+        # Build prompt
+        prompt_text = QA_PROMPT.format(
+            supported_languages_list=self._get_supported_languages_list(),
+            context_str=context_str,
+            query=query
+        )
+
+        try:
+            chain = self.llm | StrOutputParser()
+            response = chain.invoke(prompt_text)
+
+            if response and len(response.strip()) > 10:
+                return response.strip()
+            else:
+                logger.warning("LLM returned empty or minimal response. Using document synthesis.")
+        except Exception as e:
+            logger.error(f"LLM generation failed: {e}")
+
+        return self._synthesize_answer(query, context_docs)
+
+    def _synthesize_answer(self, query: str, context_docs: List[Dict]) -> str:
+        """Synthesize an answer from retrieved documents when LLM is unavailable"""
+        if not context_docs:
+            return f"Sorry, no relevant documents found for: '{query}'"
+
+        synthesized_answer = f"Question: {query}\n\n"
+        synthesized_answer += "Information from retrieved documents:\n\n"
+
+        for i, doc in enumerate(context_docs[:3], 1):
+            synthesized_answer += f"{i}. {doc['title']} - {doc['author']} (Score: {doc['score']:.3f})\n"
+            text_preview = doc["text"][:500] + "..." if len(doc["text"]) > 500 else doc["text"]
+            synthesized_answer += f"   Summary: {text_preview}\n\n"
+
+        synthesized_answer += "Information synthesized from the above documents."
+        return synthesized_answer
+
+    def query(self, question: str, top_k: int = 5, user_selected_language: Optional[str] = None) -> Dict:
+        """
+        Main query method that retrieves documents and generates an answer
+
+        Args:
+            question: User question
+            top_k: Number of documents to retrieve
+            user_selected_language: Optional language code selected by user
+        """
+        relevant_docs = self.retrieve_relevant_documents(question, top_k)
+        answer = self.generate_answer(question, relevant_docs, user_selected_language)
+
+        return {
+            "question": question,
+            "answer": answer,
+            "user_selected_language": user_selected_language,
+            "relevant_documents": relevant_docs,
+            "supported_languages": self.supported_languages,
+        }
+
+    def get_supported_languages(self) -> List[Dict]:
+        """Get list of supported languages"""
+        return self.supported_languages

docker-compose.yml

@@ -0,0 +1,22 @@
+services:
+  backend:
+    build:
+      context: ./backend
+      dockerfile: Dockerfile
+    ports:
+      - "8000:8000"
+    env_file:
+      - .env
+    environment:
+      - BACKEND_URL=http://localhost:8000
+
+  frontend:
+    build:
+      context: ./frontend
+      dockerfile: Dockerfile
+    ports:
+      - "8501:8501"
+    environment:
+      - BACKEND_URL=http://backend:8000
+    depends_on:
+      - backend

frontend/Dockerfile

@@ -0,0 +1,23 @@
+FROM python:3.10-slim
+
+WORKDIR /app
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y \
+    build-essential \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy requirements first for better caching
+COPY requirements.txt .
+
+# Install Python dependencies
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy application code
+COPY . .
+
+# Expose port
+EXPOSE 8501
+
+# Run the application
+CMD ["streamlit", "run", "app.py", "--server.address", "0.0.0.0", "--server.port", "8501"]

frontend/app.py

@@ -0,0 +1,109 @@
+"""
+Streamlit Frontend for Simple RAG System
+Simplified UI - No sidebar, just query input and response
+"""
+import streamlit as st
+import httpx
+import os
+
+# Configuration
+BACKEND_URL = os.getenv("BACKEND_URL", "http://localhost:8000")
+
+st.set_page_config(
+    page_title="Simple RAG",
+    page_icon="🤖",
+    layout="centered"
+)
+
+# Page title
+st.title("🤖 Simple RAG System")
+st.markdown("Ask questions in any supported Indic language. The system will respond in the same language.")
+
+# Supported languages
+SUPPORTED_LANGUAGES = [
+    {"code": "hi", "name": "Hindi", "native": "हिंदी"},
+    {"code": "bn", "name": "Bengali", "native": "বাংলা"},
+    {"code": "gu", "name": "Gujarati", "native": "ગુજરાતી"},
+    {"code": "kn", "name": "Kannada", "native": "ಕನ್ನಡ"},
+    {"code": "ml", "name": "Malayalam", "native": "മലയാളം"},
+    {"code": "mr", "name": "Marathi", "native": "मराठी"},
+    {"code": "or", "name": "Odia", "native": "ଓଡ଼ିଆ"},
+    {"code": "pa", "name": "Punjabi", "native": "ਪੰਜਾਬੀ"},
+    {"code": "ta", "name": "Tamil", "native": "தமிழ்"},
+    {"code": "te", "name": "Telugu", "native": "తెలుగు"},
+]
+
+# Language selection
+st.subheader("Select Language")
+language_options = [f"{lang['native']} ({lang['name']})" for lang in SUPPORTED_LANGUAGES]
+selected_language = st.selectbox(
+    "Choose a language",
+    options=language_options,
+    index=0,
+    label_visibility="collapsed"
+)
+
+# Get language code from selection
+selected_lang_code = SUPPORTED_LANGUAGES[language_options.index(selected_language)]["code"]
+
+# Query input
+st.subheader("Ask Your Question")
+query = st.text_area(
+    "Enter your question",
+    height=100,
+    placeholder=f"Type your question in {selected_language}...",
+    label_visibility="collapsed"
+)
+
+# Submit button
+col1, col2 = st.columns([1, 4])
+with col1:
+    submit_button = st.button("🔍 Search", type="primary", use_container_width=True)
+
+# Process query
+if submit_button and query.strip():
+    with st.spinner("Searching and generating answer..."):
+        try:
+            # Make API request
+            response = httpx.post(
+                f"{BACKEND_URL}/query",
+                json={
+                    "query": query,
+                    "top_k": 5,
+                    "language": selected_lang_code
+                },
+                timeout=120.0
+            )
+            
+            if response.status_code == 200:
+                result = response.json()
+                
+                # Display answer
+                st.subheader("Answer")
+                st.write(result["answer"])
+                
+            else:
+                st.error(f"Error: {response.status_code} - {response.text}")
+                
+        except httpx.ConnectError:
+            st.error(f"Could not connect to backend at {BACKEND_URL}. Make sure the backend is running.")
+        except Exception as e:
+            st.error(f"Error: {str(e)}")
+
+# Footer
+st.divider()
+st.markdown(
+    """
+    <style>
+    .footer {
+        text-align: center;
+        padding: 20px;
+        color: #666;
+    }
+    </style>
+    <div class="footer">
+        Simple RAG System - Powered by Sarvam-1 and Qdrant
+    </div>
+    """,
+    unsafe_allow_html=True
+)

frontend/requirements.txt

@@ -0,0 +1,2 @@
+streamlit>=1.28.1
+httpx>=0.25.0