Initial SourceLink AI demo

.env.example

@@ -0,0 +1,12 @@
+VECTOR_DB_BACKEND=zilliz
+ZILLIZ_URI=your-zilliz-public-endpoint
+ZILLIZ_TOKEN=your-zilliz-token
+COLLECTION_NAME=vectorEMBD
+
+EMBEDDING_PROVIDER=huggingface
+EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
+EMBEDDING_DIMENSION=384
+
+CHAT_PROVIDER=groq
+CHAT_MODEL=llama-3.1-8b-instant
+GROQ_API_KEY=your-groq-api-key

.gitignore

@@ -0,0 +1,16 @@
+.env
+.env.*
+!.env.example
+.venv/
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+
+data/raw/
+data/chroma/
+data/processed/
+
+.streamlit/secrets.toml
+Untitled
+*.log

.streamlit/config.toml

@@ -0,0 +1,5 @@
+[server]
+fileWatcherType = "none"
+
+[client]
+showErrorDetails = true

MIGRATION.md

@@ -0,0 +1,118 @@
+Migration Toward The Demo Goal
+==============================
+
+Goal
+----
+
+Build a deployable demo where users connect document sources, index them, search semantically, open the original source, and chat with retrieved document context.
+
+The app should avoid storing original files unless the user explicitly uploads them. For linked sources, it stores:
+
+- chunk text
+- embedding vectors
+- document metadata
+- source references such as Drive/GitHub URLs
+
+Current State
+-------------
+
+The app now supports:
+
+- public GitHub repository ingestion
+- public Google Drive file ingestion
+- public Google Drive folder ingestion through `gdown`
+- upload-based ingestion
+- local `data/raw` ingestion
+- retrieved-document chat with a local Ollama chat model
+- a vector store interface so Chroma can later be swapped out
+
+Available vector backends:
+
+```text
+VECTOR_DB_BACKEND=chroma
+VECTOR_DB_BACKEND=zilliz
+```
+
+Cloud Migration Path
+--------------------
+
+Recommended demo stack:
+
+```text
+App hosting: Hugging Face Spaces or Render
+Source files: stay in Google Drive / GitHub
+Metadata: vector DB metadata first, Supabase later if auth is added
+Vector DB: Zilliz Cloud or Qdrant Cloud
+Embeddings: local sentence-transformers on the app server, or Ollama on a VPS
+Chat: local small model on VPS, or API-based model for hosted demos
+```
+
+Zilliz Setup
+------------
+
+Install dependencies:
+
+```powershell
+pip install -r requirements.txt
+```
+
+Create a free Zilliz Cloud cluster, then set:
+
+Expected environment variables:
+
+```text
+VECTOR_DB_BACKEND=zilliz
+ZILLIZ_URI=<your-zilliz-endpoint>
+ZILLIZ_TOKEN=<your-zilliz-token>
+COLLECTION_NAME=vectorEMBD
+EMBEDDING_PROVIDER=huggingface
+EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
+EMBEDDING_DIMENSION=384
+CHAT_PROVIDER=groq
+CHAT_MODEL=llama-3.1-8b-instant
+GROQ_API_KEY=<your-groq-api-key>
+```
+
+These can be placed in `.env` at the project root. The app loads `.env` automatically through `python-dotenv`.
+
+The Zilliz backend stores:
+
+- vector
+- chunk text
+- filename
+- source_url
+- source_type
+- document_id
+- other scalar metadata
+
+Keep secrets and OAuth tokens outside Zilliz.
+
+Next Code Step
+--------------
+
+Add user identity and source ownership metadata:
+
+```text
+user_id
+source_id
+tenant_id
+```
+
+Then filter search results by user/source so one user's indexed chunks cannot appear for another user.
+
+Production Notes
+----------------
+
+For a public demo, public Drive/GitHub links are enough.
+
+For real users, use OAuth:
+
+- Google Drive API for private Drive access
+- GitHub OAuth or GitHub App installation for private repos
+- Supabase Auth for app users
+
+For original files:
+
+- Keep linked source files in Drive/GitHub.
+- Store only source references in vector metadata.
+- Use Supabase Storage only for manual uploads that need persistence.

README.md

@@ -0,0 +1,93 @@
+Document Search System
+======================
+
+Demo app for indexing document sources, searching them semantically, opening the original source file, and chatting with the retrieved document context.
+
+Supported demo sources:
+
+- Public GitHub repository URLs, such as `https://github.com/owner/repo`
+- Public Google Drive file and folder links
+- Manual uploads through the Streamlit sidebar
+- Local demo files in `data/raw`
+
+Current Google Drive note: public folder ingestion uses `gdown`, which is good for demos. Private folders or per-user permissions should use the Google Drive API with OAuth.
+
+Setup
+-----
+
+Install dependencies:
+
+```powershell
+pip install -r requirements.txt
+```
+
+Run Ollama and pull the models you want to use:
+
+```powershell
+ollama pull bge-m3
+ollama pull llama3.2:3b
+```
+
+Optional environment overrides:
+
+```env
+EMBEDDING_PROVIDER=ollama
+EMBEDDING_MODEL=bge-m3:567m
+EMBEDDING_DIMENSION=1024
+CHAT_MODEL=llama3.2:3b
+VECTOR_DB_BACKEND=chroma
+```
+
+For Zilliz Cloud, put this in `.env`:
+
+```env
+VECTOR_DB_BACKEND=zilliz
+ZILLIZ_URI=your-zilliz-endpoint
+ZILLIZ_TOKEN=your-zilliz-token
+COLLECTION_NAME=vectorEMBD
+EMBEDDING_PROVIDER=huggingface
+EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
+EMBEDDING_DIMENSION=384
+CHAT_PROVIDER=groq
+CHAT_MODEL=llama-3.1-8b-instant
+GROQ_API_KEY=your-groq-api-key
+```
+
+`.env` is ignored by git because it contains secrets.
+
+Start the app:
+
+```powershell
+streamlit run app/ui/main.py
+```
+
+Verify Vector Storage
+---------------------
+
+Check which vector store is active and how many chunks are stored:
+
+```powershell
+python scripts/check_vector_store.py
+```
+
+Run a quick search against the active vector store:
+
+```powershell
+python scripts/check_vector_store.py --query "machine learning"
+```
+
+How It Works
+------------
+
+1. A user provides a source link or uploads files.
+2. The app extracts supported documents.
+3. Text is chunked and embedded with Ollama.
+4. Chunks and metadata are stored in ChromaDB.
+5. Search returns relevant chunks grouped by original document.
+6. The UI shows excerpts and an `Open source` or `Download file` action.
+7. The chat panel answers follow-up questions using the most recent retrieved chunks.
+
+Migration
+---------
+
+This project is being moved toward a deployable source-connected demo. See `MIGRATION.md` for the current architecture, cloud backend plan, and the next vector database migration step.

app/config/settings.py

@@ -0,0 +1,129 @@
+import os
+from pathlib import Path
+from dataclasses import dataclass
+
+from dotenv import load_dotenv
+
+
+PROJECT_ROOT = Path(__file__).parent.parent.parent
+load_dotenv(PROJECT_ROOT / ".env")
+
+
+@dataclass
+class Settings:
+    """
+    Centralized configuration for the document search system.
+    
+    All paths, model names, and system parameters are defined here.
+    Can be overridden via environment variables.
+    """
+    
+    # ============================================================
+    # Project Paths
+    # ============================================================
+    PROJECT_ROOT: Path = PROJECT_ROOT
+    DATA_DIR: Path = PROJECT_ROOT / "data"
+    RAW_DATA_DIR: Path = DATA_DIR / "raw"
+    PROCESSED_DATA_DIR: Path = DATA_DIR / "processed"
+    CHROMA_PERSIST_DIR: Path = DATA_DIR / "chroma"
+    
+    # ============================================================
+    # Embedding / Chat Configuration
+    # ============================================================
+    EMBEDDING_PROVIDER: str = "ollama"
+    OLLAMA_BASE_URL: str = "http://localhost:11434"
+    EMBEDDING_MODEL: str = "bge-m3:567m"
+    EMBEDDING_DIMENSION: int = 1024  # bge-m3 outputs 1024-dim vectors
+    CHAT_PROVIDER: str = "ollama"
+    CHAT_MODEL: str = "llama3.2:3b"
+    GROQ_API_KEY: str = ""
+    GROQ_BASE_URL: str = "https://api.groq.com/openai/v1"
+    
+    # ============================================================
+    # Vector Database Configuration
+    # ============================================================
+    VECTOR_DB_BACKEND: str = "chroma"
+    COLLECTION_NAME: str = "documents"
+    ZILLIZ_URI: str = ""
+    ZILLIZ_TOKEN: str = ""
+    
+    # ============================================================
+    # Document Processing
+    # ============================================================
+    CHUNK_SIZE: int = 500  # Characters per chunk
+    CHUNK_OVERLAP: int = 50  # Overlap between chunks
+    
+    SUPPORTED_FILE_TYPES: tuple = (".pdf", ".txt", ".docx", ".md")
+    
+    # ============================================================
+    # Retrieval Configuration
+    # ============================================================
+    DEFAULT_TOP_K: int = 5  # Number of chunks to retrieve
+    SIMILARITY_THRESHOLD: float = 0.7  # Minimum similarity score (0-1)
+    
+    # ============================================================
+    # Streamlit UI
+    # ============================================================
+    APP_TITLE: str = "📚 Document Search System"
+    APP_ICON: str = "📄"
+    MAX_UPLOAD_SIZE_MB: int = 200
+    
+    # ============================================================
+    # Logging
+    # ============================================================
+    LOG_LEVEL: str = os.getenv("LOG_LEVEL", "INFO")
+    LOG_FORMAT: str = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+    
+    def __post_init__(self):
+        """Create necessary directories on initialization."""
+        self.RAW_DATA_DIR.mkdir(parents=True, exist_ok=True)
+        self.PROCESSED_DATA_DIR.mkdir(parents=True, exist_ok=True)
+        self.CHROMA_PERSIST_DIR.mkdir(parents=True, exist_ok=True)
+    
+    @classmethod
+    def from_env(cls) -> "Settings":
+        """
+        Load settings with environment variable overrides.
+        
+        Example .env file:
+            OLLAMA_BASE_URL=http://localhost:11434
+            EMBEDDING_PROVIDER=huggingface
+            EMBEDDING_MODEL=bge-m3:567m
+            EMBEDDING_DIMENSION=1024
+            CHAT_PROVIDER=groq
+            CHAT_MODEL=llama3.2:3b
+            VECTOR_DB_BACKEND=chroma
+            CHUNK_SIZE=1000
+            DEFAULT_TOP_K=10
+        """
+        return cls(
+            EMBEDDING_PROVIDER=os.getenv("EMBEDDING_PROVIDER", cls.EMBEDDING_PROVIDER),
+            OLLAMA_BASE_URL=os.getenv("OLLAMA_BASE_URL", cls.OLLAMA_BASE_URL),
+            EMBEDDING_MODEL=os.getenv("EMBEDDING_MODEL", cls.EMBEDDING_MODEL),
+            EMBEDDING_DIMENSION=int(os.getenv("EMBEDDING_DIMENSION", cls.EMBEDDING_DIMENSION)),
+            CHAT_PROVIDER=os.getenv("CHAT_PROVIDER", cls.CHAT_PROVIDER),
+            CHAT_MODEL=os.getenv("CHAT_MODEL", cls.CHAT_MODEL),
+            GROQ_API_KEY=os.getenv("GROQ_API_KEY", cls.GROQ_API_KEY),
+            GROQ_BASE_URL=os.getenv("GROQ_BASE_URL", cls.GROQ_BASE_URL),
+            CHUNK_SIZE=int(os.getenv("CHUNK_SIZE", cls.CHUNK_SIZE)),
+            CHUNK_OVERLAP=int(os.getenv("CHUNK_OVERLAP", cls.CHUNK_OVERLAP)),
+            DEFAULT_TOP_K=int(os.getenv("DEFAULT_TOP_K", cls.DEFAULT_TOP_K)),
+            VECTOR_DB_BACKEND=os.getenv("VECTOR_DB_BACKEND", cls.VECTOR_DB_BACKEND),
+            COLLECTION_NAME=os.getenv("COLLECTION_NAME", cls.COLLECTION_NAME),
+            ZILLIZ_URI=os.getenv("ZILLIZ_URI", cls.ZILLIZ_URI),
+            ZILLIZ_TOKEN=os.getenv("ZILLIZ_TOKEN", cls.ZILLIZ_TOKEN),
+        )
+
+
+# ============================================================
+# Global Settings Instance
+# ============================================================
+settings = Settings.from_env()
+
+
+# ============================================================
+# Convenience function for other modules
+# ============================================================
+def get_settings() -> Settings:
+    """Returns the global settings instance."""
+    return settings

app/ingestion/chunker.py

@@ -0,0 +1,292 @@
+from typing import List, Dict, Any
+import logging
+import re
+
+from app.config.settings import settings
+
+logging.basicConfig(
+    level=settings.LOG_LEVEL,
+    format=settings.LOG_FORMAT
+)
+
+logger = logging.getLogger(__name__)
+
+
+class TextChunker:
+    """
+    Smart text chunking with overlap.
+    
+    Splits documents into manageable chunks for embedding while:
+    - Preserving sentence boundaries
+    - Adding overlap between chunks for context continuity
+    - Maintaining metadata for each chunk
+    """
+
+    def __init__(
+        self,
+        chunk_size: int = None,
+        chunk_overlap: int = None,
+    ):
+        """
+        Args:
+            chunk_size: Target characters per chunk (default: from settings)
+            chunk_overlap: Overlap between consecutive chunks (default: from settings)
+        """
+        self.chunk_size = chunk_size or settings.CHUNK_SIZE
+        self.chunk_overlap = chunk_overlap or settings.CHUNK_OVERLAP
+        
+        if self.chunk_overlap >= self.chunk_size:
+            raise ValueError("chunk_overlap must be less than chunk_size")
+        
+        logger.info(
+            f"TextChunker initialized: chunk_size={self.chunk_size}, "
+            f"overlap={self.chunk_overlap}"
+        )
+
+    def chunk_text(
+        self,
+        text: str,
+        metadata: Dict[str, Any] = None,
+    ) -> List[Dict[str, Any]]:
+        """
+        Split text into overlapping chunks.
+        
+        Args:
+            text: Input text to chunk
+            metadata: Optional metadata to attach to each chunk
+            
+        Returns:
+            List of chunk dictionaries, each containing:
+                - 'text': Chunk text
+                - 'metadata': Chunk metadata (includes chunk_index)
+        """
+        if not text or not text.strip():
+            logger.warning("Empty text provided to chunker")
+            return []
+        
+        # Split into sentences for better boundary detection
+        sentences = self._split_into_sentences(text)
+        
+        chunks = []
+        current_chunk = []
+        current_length = 0
+        chunk_index = 0
+        
+        for sentence in sentences:
+            sentence_length = len(sentence)
+            
+            # If adding this sentence exceeds chunk_size, finalize current chunk
+            if current_length + sentence_length > self.chunk_size and current_chunk:
+                # Create chunk
+                chunk_text = " ".join(current_chunk)
+                chunks.append(self._create_chunk(chunk_text, chunk_index, metadata))
+                chunk_index += 1
+                
+                # Start new chunk with overlap
+                overlap_text = chunk_text[-self.chunk_overlap:] if len(chunk_text) > self.chunk_overlap else chunk_text
+                current_chunk = [overlap_text]
+                current_length = len(overlap_text)
+            
+            # Add sentence to current chunk
+            current_chunk.append(sentence)
+            current_length += sentence_length + 1  # +1 for space
+        
+        # Add final chunk
+        if current_chunk:
+            chunk_text = " ".join(current_chunk)
+            chunks.append(self._create_chunk(chunk_text, chunk_index, metadata))
+        
+        logger.info(f"✓ Created {len(chunks)} chunks from {len(text)} characters")
+        
+        return chunks
+
+    def _split_into_sentences(self, text: str) -> List[str]:
+        """
+        Split text into sentences using regex.
+        
+        Handles common sentence boundaries like:
+        - Period followed by space and capital letter
+        - Question marks and exclamation marks
+        - Preserves abbreviations like "Dr." and "U.S."
+        """
+        # Simple sentence splitting pattern
+        # Matches: . ! ? followed by space and capital letter
+        sentence_pattern = r'(?<=[.!?])\s+(?=[A-Z])'
+        
+        sentences = re.split(sentence_pattern, text)
+        
+        # Clean up sentences
+        sentences = [s.strip() for s in sentences if s.strip()]
+        
+        return sentences
+
+    def _create_chunk(
+        self,
+        text: str,
+        chunk_index: int,
+        base_metadata: Dict[str, Any] = None,
+    ) -> Dict[str, Any]:
+        """
+        Create a chunk dictionary with metadata.
+        
+        Args:
+            text: Chunk text
+            chunk_index: Index of this chunk in the document
+            base_metadata: Base metadata from the document
+            
+        Returns:
+            Dictionary with 'text' and 'metadata' keys
+        """
+        metadata = base_metadata.copy() if base_metadata else {}
+        
+        # Add chunk-specific metadata
+        metadata.update({
+            "chunk_index": chunk_index,
+            "chunk_length": len(text),
+        })
+        
+        return {
+            "text": text,
+            "metadata": metadata,
+        }
+
+    def chunk_document(
+        self,
+        document: Dict[str, Any],
+    ) -> List[Dict[str, Any]]:
+        """
+        Chunk a loaded document (output from DocumentLoader).
+        
+        Args:
+            document: Dictionary with 'text' and 'metadata' keys
+            
+        Returns:
+            List of chunks with metadata
+        """
+        text = document.get("text", "")
+        metadata = document.get("metadata", {})
+        
+        return self.chunk_text(text, metadata)
+
+    def chunk_documents(
+        self,
+        documents: Dict[str, Dict[str, Any]],
+    ) -> List[Dict[str, Any]]:
+        """
+        Chunk multiple documents.
+        
+        Args:
+            documents: Dictionary mapping filename -> document data
+            
+        Returns:
+            Flattened list of all chunks from all documents
+        """
+        all_chunks = []
+        
+        for filename, doc_data in documents.items():
+            logger.info(f"Chunking document: {filename}")
+            chunks = self.chunk_document(doc_data)
+            all_chunks.extend(chunks)
+        
+        logger.info(f"✓ Total chunks created: {len(all_chunks)}")
+        
+        return all_chunks
+
+
+# ============================================================
+# Advanced Chunking Strategies (Optional)
+# ============================================================
+
+class SemanticChunker(TextChunker):
+    """
+    Advanced chunker that tries to preserve semantic boundaries.
+    
+    Uses paragraph breaks and section headers as primary split points.
+    Falls back to sentence-based chunking when needed.
+    """
+    
+    def chunk_text(
+        self,
+        text: str,
+        metadata: Dict[str, Any] = None,
+    ) -> List[Dict[str, Any]]:
+        """Override to use paragraph-aware chunking."""
+        
+        if not text or not text.strip():
+            return []
+        
+        # Split by paragraphs (double newline)
+        paragraphs = re.split(r'\n\s*\n', text)
+        paragraphs = [p.strip() for p in paragraphs if p.strip()]
+        
+        chunks = []
+        current_chunk = []
+        current_length = 0
+        chunk_index = 0
+        
+        for para in paragraphs:
+            para_length = len(para)
+            
+            # If paragraph itself is too large, split it
+            if para_length > self.chunk_size:
+                # Finalize current chunk first
+                if current_chunk:
+                    chunk_text = "\n\n".join(current_chunk)
+                    chunks.append(self._create_chunk(chunk_text, chunk_index, metadata))
+                    chunk_index += 1
+                    current_chunk = []
+                    current_length = 0
+                
+                # Split large paragraph using parent method
+                para_chunks = super().chunk_text(para, metadata)
+                for pc in para_chunks:
+                    pc['metadata']['chunk_index'] = chunk_index
+                    chunks.append(pc)
+                    chunk_index += 1
+                
+                continue
+            
+            # Check if adding this paragraph exceeds limit
+            if current_length + para_length > self.chunk_size and current_chunk:
+                chunk_text = "\n\n".join(current_chunk)
+                chunks.append(self._create_chunk(chunk_text, chunk_index, metadata))
+                chunk_index += 1
+                current_chunk = []
+                current_length = 0
+            
+            current_chunk.append(para)
+            current_length += para_length
+        
+        # Add final chunk
+        if current_chunk:
+            chunk_text = "\n\n".join(current_chunk)
+            chunks.append(self._create_chunk(chunk_text, chunk_index, metadata))
+        
+        logger.info(f"✓ Semantic chunking: {len(chunks)} chunks created")
+        
+        return chunks
+
+
+# ============================================================
+# Global Chunker Instance
+# ============================================================
+_chunker_instance = None
+
+
+def get_chunker(semantic: bool = False) -> TextChunker:
+    """
+    Returns a chunker instance.
+    
+    Args:
+        semantic: If True, returns SemanticChunker (paragraph-aware)
+                  If False, returns standard TextChunker
+    """
+    global _chunker_instance
+    
+    if _chunker_instance is None:
+        if semantic:
+            _chunker_instance = SemanticChunker()
+        else:
+            _chunker_instance = TextChunker()
+    
+    return _chunker_instance

app/ingestion/embedder.py

@@ -0,0 +1,139 @@
+from typing import List, Union
+import logging
+
+import requests
+
+from app.config.settings import settings
+
+logger = logging.getLogger(__name__)
+
+
+class OllamaEmbedder:
+    """Wrapper for Ollama's local embedding API."""
+
+    def __init__(self, base_url: str = None, model: str = None):
+        self.base_url = base_url or settings.OLLAMA_BASE_URL
+        self.model = model or settings.EMBEDDING_MODEL
+        self.embed_endpoint = f"{self.base_url}/api/embeddings"
+
+        logger.info("Initialized OllamaEmbedder with model: %s", self.model)
+        self._check_ollama_connection()
+
+    def _check_ollama_connection(self) -> None:
+        try:
+            response = requests.get(f"{self.base_url}/api/tags", timeout=5)
+            response.raise_for_status()
+        except requests.exceptions.RequestException as exc:
+            raise ConnectionError(
+                f"Ollama server not reachable at {self.base_url}. "
+                f"Make sure Ollama is running. Error: {exc}"
+            ) from exc
+
+    def embed_single(self, text: str) -> List[float]:
+        if not text or not text.strip():
+            raise ValueError("Cannot embed empty text")
+
+        response = requests.post(
+            self.embed_endpoint,
+            json={"model": self.model, "prompt": text},
+            timeout=30,
+        )
+        response.raise_for_status()
+
+        embedding = response.json().get("embedding")
+        if not embedding:
+            raise RuntimeError("No embedding returned from Ollama")
+
+        return embedding
+
+    def embed_batch(self, texts: List[str]) -> List[List[float]]:
+        embeddings = []
+        for index, text in enumerate(texts):
+            try:
+                embeddings.append(self.embed_single(text))
+            except Exception as exc:
+                logger.error("Failed to embed text %s: %s", index, exc)
+                embeddings.append([0.0] * settings.EMBEDDING_DIMENSION)
+
+        return embeddings
+
+    def embed(self, text: Union[str, List[str]]) -> Union[List[float], List[List[float]]]:
+        if isinstance(text, str):
+            return self.embed_single(text)
+        if isinstance(text, list):
+            return self.embed_batch(text)
+        raise TypeError("Input must be str or List[str]")
+
+    def get_embedding_dimension(self) -> int:
+        return settings.EMBEDDING_DIMENSION
+
+
+class HuggingFaceEmbedder:
+    """Local sentence-transformers embedder for deployable Python hosting."""
+
+    def __init__(self, model: str = None):
+        try:
+            from sentence_transformers import SentenceTransformer
+        except ImportError as exc:
+            raise RuntimeError(
+                "Hugging Face embeddings require sentence-transformers. "
+                "Install requirements.txt again."
+            ) from exc
+
+        self.model_name = model or settings.EMBEDDING_MODEL
+        self.model = SentenceTransformer(self.model_name)
+        logger.info("Initialized HuggingFaceEmbedder with model: %s", self.model_name)
+
+        actual_dimension = self.model.get_sentence_embedding_dimension()
+        if actual_dimension != settings.EMBEDDING_DIMENSION:
+            raise ValueError(
+                f"EMBEDDING_DIMENSION={settings.EMBEDDING_DIMENSION} does not match "
+                f"{self.model_name} output dimension {actual_dimension}."
+            )
+
+    def embed_single(self, text: str) -> List[float]:
+        if not text or not text.strip():
+            raise ValueError("Cannot embed empty text")
+
+        return self.model.encode(text, normalize_embeddings=True).tolist()
+
+    def embed_batch(self, texts: List[str]) -> List[List[float]]:
+        if not texts:
+            return []
+
+        clean_texts = [text if text and text.strip() else " " for text in texts]
+        return self.model.encode(clean_texts, normalize_embeddings=True).tolist()
+
+    def embed(self, text: Union[str, List[str]]) -> Union[List[float], List[List[float]]]:
+        if isinstance(text, str):
+            return self.embed_single(text)
+        if isinstance(text, list):
+            return self.embed_batch(text)
+        raise TypeError("Input must be str or List[str]")
+
+    def get_embedding_dimension(self) -> int:
+        return settings.EMBEDDING_DIMENSION
+
+
+_embedder_instance = None
+
+
+def get_embedder():
+    """Return a singleton embedder for the configured provider."""
+    global _embedder_instance
+
+    if _embedder_instance is not None:
+        return _embedder_instance
+
+    provider = settings.EMBEDDING_PROVIDER.lower()
+    if provider == "ollama":
+        _embedder_instance = OllamaEmbedder()
+    elif provider in {"huggingface", "sentence-transformers", "sentence_transformers"}:
+        _embedder_instance = HuggingFaceEmbedder()
+    else:
+        raise ValueError(
+            f"Unsupported EMBEDDING_PROVIDER='{settings.EMBEDDING_PROVIDER}'. "
+            "Use 'ollama' or 'huggingface'."
+        )
+
+    return _embedder_instance

app/ingestion/ingest.py

@@ -0,0 +1,307 @@
+from typing import List, Dict, Any, Optional
+from pathlib import Path
+import logging
+import uuid
+import tempfile
+
+from app.ingestion.loader import get_loader
+from app.ingestion.chunker import get_chunker
+from app.ingestion.embedder import get_embedder
+from app.sources.connectors import get_source_connector
+from app.vectordb.factory import get_vector_store
+from app.config.settings import settings
+
+logging.basicConfig(
+    level=settings.LOG_LEVEL,
+    format=settings.LOG_FORMAT
+)
+
+logger = logging.getLogger(__name__)
+
+
+class IngestionPipeline:
+    """
+    End-to-end document ingestion pipeline.
+    
+    Pipeline stages:
+    1. Load documents (PDF, TXT, DOCX, MD)
+    2. Chunk text into manageable pieces
+    3. Generate embeddings using Ollama
+    4. Store in ChromaDB with metadata
+    """
+
+    def __init__(
+        self,
+        chroma_persist_dir: str = None,
+        collection_name: str = None,
+        use_semantic_chunking: bool = False,
+    ):
+        """
+        Args:
+            chroma_persist_dir: Where ChromaDB stores data
+            collection_name: ChromaDB collection name
+            use_semantic_chunking: Use paragraph-aware chunking
+        """
+        self.loader = get_loader()
+        self.chunker = get_chunker(semantic=use_semantic_chunking)
+        self.embedder = get_embedder()
+        
+        self.vector_store = get_vector_store()
+        
+        logger.info("✓ Ingestion pipeline initialized")
+
+    def ingest_file(self, file_path: str, extra_metadata: Optional[Dict[str, Any]] = None) -> int:
+        """
+        Ingest a single document file.
+        
+        Args:
+            file_path: Path to document
+            
+        Returns:
+            Number of chunks ingested
+        """
+        logger.info(f"Starting ingestion for: {file_path}")
+        
+        # 1. Load document
+        document = self.loader.load(file_path)
+        if extra_metadata:
+            document["metadata"].update(extra_metadata)
+        document["metadata"].setdefault("source_type", "local")
+        document["metadata"].setdefault("source_url", "")
+        document["metadata"].setdefault("source_root", "")
+        document["metadata"].setdefault("source_path", str(file_path))
+        document["metadata"].setdefault("document_id", self._document_id(document["metadata"], file_path))
+        
+        # 2. Chunk document
+        chunks = self.chunker.chunk_document(document)
+        
+        if not chunks:
+            logger.warning(f"No chunks created from {file_path}")
+            return 0
+        
+        # 3. Generate embeddings
+        chunk_texts = [chunk["text"] for chunk in chunks]
+        embeddings = self.embedder.embed_batch(chunk_texts)
+        
+        # 4. Prepare data for ChromaDB
+        ids = [self._generate_chunk_id(file_path, i) for i in range(len(chunks))]
+        metadatas = [chunk["metadata"] for chunk in chunks]
+        
+        # 5. Store in vector database
+        self.vector_store.add_documents(
+            ids=ids,
+            documents=chunk_texts,
+            embeddings=embeddings,
+            metadatas=metadatas,
+        )
+        
+        logger.info(f"✓ Ingested {len(chunks)} chunks from {Path(file_path).name}")
+        
+        return len(chunks)
+
+    def ingest_directory(self, directory_path: str) -> Dict[str, int]:
+        """
+        Ingest all supported documents from a directory.
+        
+        Args:
+            directory_path: Path to directory containing documents
+            
+        Returns:
+            Dictionary mapping filename -> number of chunks ingested
+        """
+        logger.info(f"Starting batch ingestion from: {directory_path}")
+        
+        dir_path = Path(directory_path)
+        
+        if not dir_path.exists() or not dir_path.is_dir():
+            raise ValueError(f"Invalid directory: {directory_path}")
+        
+        results = {}
+        total_chunks = 0
+        
+        # Get all supported files
+        supported_files = [
+            f for f in dir_path.iterdir()
+            if f.is_file() and f.suffix.lower() in settings.SUPPORTED_FILE_TYPES
+        ]
+        
+        logger.info(f"Found {len(supported_files)} supported documents")
+        
+        for file_path in supported_files:
+            try:
+                num_chunks = self.ingest_file(
+                    str(file_path),
+                    extra_metadata={
+                        "source_type": "local",
+                        "source_path": str(file_path),
+                        "document_id": f"local:{file_path.resolve()}",
+                    },
+                )
+                results[file_path.name] = num_chunks
+                total_chunks += num_chunks
+            except Exception as e:
+                logger.error(f"Failed to ingest {file_path.name}: {e}")
+                results[file_path.name] = 0
+        
+        logger.info(f"✓ Batch ingestion complete: {total_chunks} total chunks from {len(results)} files")
+        
+        return results
+
+    def ingest_documents(self, documents: Dict[str, Dict[str, Any]]) -> int:
+        """
+        Ingest pre-loaded documents (from DocumentLoader.load_directory()).
+        
+        Args:
+            documents: Dictionary mapping filename -> document data
+            
+        Returns:
+            Total number of chunks ingested
+        """
+        logger.info(f"Ingesting {len(documents)} pre-loaded documents")
+        
+        total_chunks = 0
+        
+        for filename, doc_data in documents.items():
+            try:
+                # Chunk document
+                chunks = self.chunker.chunk_document(doc_data)
+                
+                if not chunks:
+                    continue
+                
+                # Generate embeddings
+                chunk_texts = [chunk["text"] for chunk in chunks]
+                embeddings = self.embedder.embed_batch(chunk_texts)
+                
+                # Prepare for ChromaDB
+                ids = [self._generate_chunk_id(filename, i) for i in range(len(chunks))]
+                metadatas = [chunk["metadata"] for chunk in chunks]
+                
+                # Store
+                self.vector_store.add_documents(
+                    ids=ids,
+                    documents=chunk_texts,
+                    embeddings=embeddings,
+                    metadatas=metadatas,
+                )
+                
+                total_chunks += len(chunks)
+                logger.info(f"✓ Ingested {len(chunks)} chunks from {filename}")
+                
+            except Exception as e:
+                logger.error(f"Failed to ingest {filename}: {e}")
+        
+        logger.info(f"✓ Total ingestion: {total_chunks} chunks")
+        
+        return total_chunks
+
+    def ingest_source_url(self, source_url: str) -> Dict[str, int]:
+        """
+        Ingest supported documents from a public source URL.
+
+        Demo sources:
+        - Public GitHub repository URL
+        - Public Google Drive file URL
+        """
+        connector = get_source_connector()
+        results = {}
+
+        with tempfile.TemporaryDirectory(prefix="document_source_") as temp_dir:
+            source_documents = connector.fetch(source_url, Path(temp_dir))
+
+            if not source_documents:
+                return results
+
+            for source_document in source_documents:
+                try:
+                    num_chunks = self.ingest_file(
+                        str(source_document.path),
+                        extra_metadata=source_document.metadata,
+                    )
+                    display_name = source_document.metadata.get("source_path", source_document.path.name)
+                    results[display_name] = num_chunks
+                except Exception as e:
+                    logger.error("Failed to ingest %s: %s", source_document.path.name, e)
+                    results[source_document.path.name] = 0
+
+        return results
+
+    def _generate_chunk_id(self, source: str, chunk_index: int) -> str:
+        """
+        Generate unique ID for a chunk.
+        
+        Format: {source_name}_{chunk_index}_{uuid}
+        """
+        source_name = Path(source).stem  # filename without extension
+        unique_id = str(uuid.uuid4())[:8]  # Short UUID
+        
+        return f"{source_name}_chunk{chunk_index}_{unique_id}"
+
+    def _document_id(self, metadata: Dict[str, Any], file_path: str) -> str:
+        source_url = metadata.get("source_url")
+        if source_url:
+            return source_url
+
+        return f"local:{Path(file_path).resolve()}"
+
+    def get_status(self) -> Dict[str, Any]:
+        """
+        Get current status of the vector database.
+        
+        Returns:
+            Dictionary with collection info and document count
+        """
+        info = self.vector_store.get_collection_info()
+        
+        return {
+            "collection_name": info["name"],
+            "total_chunks": info["count"],
+            "metadata": info["metadata"],
+        }
+
+    def reset_database(self) -> None:
+        """
+        Delete all data from the vector database.
+        
+        WARNING: This is irreversible!
+        """
+        logger.warning("Resetting vector database - all data will be deleted!")
+        self.vector_store.delete_all()
+        logger.info("✓ Database reset complete")
+
+
+# ============================================================
+# Convenience Functions
+# ============================================================
+
+def ingest_file(file_path: str) -> int:
+    """
+    Quick function to ingest a single file.
+    
+    Args:
+        file_path: Path to document
+        
+    Returns:
+        Number of chunks ingested
+    """
+    pipeline = IngestionPipeline()
+    return pipeline.ingest_file(file_path)
+
+
+def ingest_directory(directory_path: str) -> Dict[str, int]:
+    """
+    Quick function to ingest a directory.
+    
+    Args:
+        directory_path: Path to directory
+        
+    Returns:
+        Dictionary mapping filename -> chunk count
+    """
+    pipeline = IngestionPipeline()
+    return pipeline.ingest_directory(directory_path)
+
+
+def get_pipeline() -> IngestionPipeline:
+    """Returns a configured ingestion pipeline instance."""
+    return IngestionPipeline()

app/ingestion/loader.py

@@ -0,0 +1,218 @@
+from pathlib import Path
+from typing import Dict, Optional
+import logging
+
+# Document parsing libraries
+import PyPDF2
+from docx import Document as DocxDocument
+import markdown
+from bs4 import BeautifulSoup
+
+from app.config.settings import settings
+
+logging.basicConfig(
+    level=settings.LOG_LEVEL,
+    format=settings.LOG_FORMAT
+)
+
+logger = logging.getLogger(__name__)
+
+
+class DocumentLoader:
+    """
+    Unified document loader for multiple file formats.
+    
+    Supports:
+    - PDF (.pdf)
+    - Text files (.txt)
+    - Word documents (.docx)
+    - Markdown files (.md)
+    """
+
+    def __init__(self):
+        self.supported_types = settings.SUPPORTED_FILE_TYPES
+        logger.info(f"DocumentLoader initialized. Supported types: {self.supported_types}")
+
+    def load(self, file_path: str) -> Dict[str, any]:
+        """
+        Load a document and extract its text content.
+        
+        Args:
+            file_path: Path to the document file
+            
+        Returns:
+            Dictionary containing:
+                - 'text': Extracted text content
+                - 'metadata': File metadata (name, type, size, etc.)
+                
+        Raises:
+            FileNotFoundError: If file doesn't exist
+            ValueError: If file type is not supported
+        """
+        path = Path(file_path)
+        
+        if not path.exists():
+            raise FileNotFoundError(f"File not found: {file_path}")
+        
+        file_ext = path.suffix.lower()
+        
+        if file_ext not in self.supported_types:
+            raise ValueError(
+                f"Unsupported file type: {file_ext}. "
+                f"Supported types: {self.supported_types}"
+            )
+        
+        logger.info(f"Loading document: {path.name}")
+        
+        # Route to appropriate loader
+        if file_ext == ".pdf":
+            text = self._load_pdf(path)
+        elif file_ext == ".txt":
+            text = self._load_txt(path)
+        elif file_ext == ".docx":
+            text = self._load_docx(path)
+        elif file_ext == ".md":
+            text = self._load_markdown(path)
+        else:
+            raise ValueError(f"No loader implemented for {file_ext}")
+        
+        # Build metadata
+        metadata = {
+            "filename": path.name,
+            "file_type": file_ext,
+            "file_size_bytes": path.stat().st_size,
+            "char_count": len(text),
+        }
+        
+        logger.info(f"✓ Loaded {path.name}: {len(text)} characters")
+        
+        return {
+            "text": text,
+            "metadata": metadata,
+        }
+
+    # ------------------------------------------------------------------
+    # Format-specific loaders
+    # ------------------------------------------------------------------
+
+    def _load_pdf(self, path: Path) -> str:
+        """Extract text from PDF using PyPDF2."""
+        text = []
+        
+        try:
+            with open(path, 'rb') as file:
+                pdf_reader = PyPDF2.PdfReader(file)
+                num_pages = len(pdf_reader.pages)
+                
+                for page_num in range(num_pages):
+                    page = pdf_reader.pages[page_num]
+                    page_text = page.extract_text()
+                    
+                    if page_text:
+                        text.append(page_text)
+                
+                logger.info(f"  Extracted {num_pages} pages from PDF")
+                
+        except Exception as e:
+            logger.error(f"Failed to parse PDF {path.name}: {e}")
+            raise RuntimeError(f"PDF parsing failed: {e}")
+        
+        return "\n\n".join(text)
+
+    def _load_txt(self, path: Path) -> str:
+        """Load plain text file."""
+        try:
+            with open(path, 'r', encoding='utf-8') as file:
+                text = file.read()
+            return text
+        except UnicodeDecodeError:
+            # Fallback to Latin-1 encoding
+            logger.warning(f"UTF-8 decode failed for {path.name}, trying latin-1")
+            with open(path, 'r', encoding='latin-1') as file:
+                text = file.read()
+            return text
+
+    def _load_docx(self, path: Path) -> str:
+        """Extract text from Word document."""
+        try:
+            doc = DocxDocument(path)
+            paragraphs = [para.text for para in doc.paragraphs if para.text.strip()]
+            
+            logger.info(f"  Extracted {len(paragraphs)} paragraphs from DOCX")
+            
+            return "\n\n".join(paragraphs)
+            
+        except Exception as e:
+            logger.error(f"Failed to parse DOCX {path.name}: {e}")
+            raise RuntimeError(f"DOCX parsing failed: {e}")
+
+    def _load_markdown(self, path: Path) -> str:
+        """
+        Load Markdown file and convert to plain text.
+        Strips HTML tags from rendered markdown.
+        """
+        try:
+            with open(path, 'r', encoding='utf-8') as file:
+                md_text = file.read()
+            
+            # Convert markdown to HTML
+            html = markdown.markdown(md_text)
+            
+            # Strip HTML tags to get plain text
+            soup = BeautifulSoup(html, 'html.parser')
+            text = soup.get_text(separator='\n\n')
+            
+            return text
+            
+        except Exception as e:
+            logger.error(f"Failed to parse Markdown {path.name}: {e}")
+            raise RuntimeError(f"Markdown parsing failed: {e}")
+
+    # ------------------------------------------------------------------
+    # Batch loading
+    # ------------------------------------------------------------------
+
+    def load_directory(self, directory_path: str) -> Dict[str, Dict]:
+        """
+        Load all supported documents from a directory.
+        
+        Args:
+            directory_path: Path to directory containing documents
+            
+        Returns:
+            Dictionary mapping filename -> loaded document data
+        """
+        dir_path = Path(directory_path)
+        
+        if not dir_path.exists() or not dir_path.is_dir():
+            raise ValueError(f"Invalid directory: {directory_path}")
+        
+        documents = {}
+        
+        for file_path in dir_path.iterdir():
+            if file_path.is_file() and file_path.suffix.lower() in self.supported_types:
+                try:
+                    doc_data = self.load(str(file_path))
+                    documents[file_path.name] = doc_data
+                except Exception as e:
+                    logger.error(f"Failed to load {file_path.name}: {e}")
+        
+        logger.info(f"✓ Loaded {len(documents)} documents from {dir_path.name}")
+        
+        return documents
+
+
+# ============================================================
+# Global Loader Instance
+# ============================================================
+_loader_instance = None
+
+
+def get_loader() -> DocumentLoader:
+    """Returns a singleton instance of DocumentLoader."""
+    global _loader_instance
+    
+    if _loader_instance is None:
+        _loader_instance = DocumentLoader()
+    
+    return _loader_instance

app/retrieval/aggregator.py

@@ -0,0 +1,181 @@
+from typing import List, Dict, Any
+from collections import defaultdict
+import logging
+
+from app.config.settings import settings
+
+logging.basicConfig(
+    level=settings.LOG_LEVEL,
+    format=settings.LOG_FORMAT
+)
+
+logger = logging.getLogger(__name__)
+
+
+class ResultAggregator:
+    """
+    Aggregates and ranks search results by document.
+    
+    Takes raw chunk-level results and groups them by source document,
+    computing document-level relevance scores.
+    """
+
+    def __init__(self):
+        logger.info("ResultAggregator initialized")
+
+    def aggregate_by_document(
+        self,
+        results: List[Dict[str, Any]],
+        max_chunks_per_doc: int = 3,
+    ) -> List[Dict[str, Any]]:
+        """
+        Group search results by source document.
+        
+        Args:
+            results: List of chunk-level search results
+            max_chunks_per_doc: Max chunks to include per document
+            
+        Returns:
+            List of document-level results, each containing:
+                - 'filename': Source document name
+                - 'relevance_score': Aggregated relevance
+                - 'chunks': Top matching chunks from this document
+                - 'metadata': Document metadata
+        """
+        if not results:
+            return []
+        
+        # Group chunks by stable document identity. Filenames are not enough
+        # once the same repo/folder can contain duplicate names.
+        doc_groups = defaultdict(list)
+        
+        for result in results:
+            document_key = result['metadata'].get('document_id') or result['metadata'].get('source_url') or result['metadata'].get('filename', 'unknown')
+            doc_groups[document_key].append(result)
+        
+        # Aggregate and rank documents
+        aggregated = []
+        
+        for document_key, chunks in doc_groups.items():
+            # Sort chunks by similarity (highest first)
+            chunks = sorted(chunks, key=lambda x: x['similarity'], reverse=True)
+            
+            # Take top N chunks
+            top_chunks = chunks[:max_chunks_per_doc]
+            
+            # Calculate document-level relevance score
+            # Use average of top chunks' similarities
+            relevance_score = sum(c['similarity'] for c in top_chunks) / len(top_chunks)
+            
+            # Extract document metadata (from first chunk)
+            doc_metadata = self._extract_document_metadata(chunks[0]['metadata'])
+            filename = doc_metadata.get('filename', document_key)
+            
+            aggregated.append({
+                'filename': filename,
+                'document_id': document_key,
+                'relevance_score': relevance_score,
+                'num_matching_chunks': len(chunks),
+                'chunks': top_chunks,
+                'metadata': doc_metadata,
+            })
+        
+        # Sort documents by relevance
+        aggregated = sorted(aggregated, key=lambda x: x['relevance_score'], reverse=True)
+        
+        logger.info(f"✓ Aggregated {len(results)} chunks into {len(aggregated)} documents")
+        
+        return aggregated
+
+    def _extract_document_metadata(self, chunk_metadata: Dict[str, Any]) -> Dict[str, Any]:
+        """
+        Extract document-level metadata from chunk metadata.
+        
+        Removes chunk-specific fields like chunk_index.
+        """
+        doc_metadata = chunk_metadata.copy()
+        
+        # Remove chunk-specific fields
+        chunk_fields = ['chunk_index', 'chunk_length']
+        for field in chunk_fields:
+            doc_metadata.pop(field, None)
+        
+        return doc_metadata
+
+    def format_for_display(
+        self,
+        aggregated_results: List[Dict[str, Any]],
+        include_chunk_text: bool = True,
+    ) -> List[Dict[str, Any]]:
+        """
+        Format aggregated results for UI display.
+        
+        Args:
+            aggregated_results: Output from aggregate_by_document()
+            include_chunk_text: Whether to include full chunk text
+            
+        Returns:
+            Formatted results suitable for display
+        """
+        formatted = []
+        
+        for doc in aggregated_results:
+            formatted_doc = {
+                'filename': doc['filename'],
+                'relevance_score': round(doc['relevance_score'], 3),
+                'num_matches': doc['num_matching_chunks'],
+                'file_type': doc['metadata'].get('file_type', 'unknown'),
+            }
+            
+            if include_chunk_text:
+                formatted_doc['excerpts'] = []
+                
+                for chunk in doc['chunks']:
+                    excerpt = {
+                        'text': self._create_excerpt(chunk['text']),
+                        'similarity': round(chunk['similarity'], 3),
+                        'chunk_index': chunk['metadata'].get('chunk_index', 0),
+                    }
+                    formatted_doc['excerpts'].append(excerpt)
+            
+            formatted.append(formatted_doc)
+        
+        return formatted
+
+    def _create_excerpt(self, text: str, max_length: int = 200) -> str:
+        """
+        Create a display excerpt from full chunk text.
+        
+        Truncates long text and adds ellipsis.
+        """
+        if len(text) <= max_length:
+            return text
+        
+        return text[:max_length].strip() + "..."
+
+
+# ============================================================
+# Convenience Functions
+# ============================================================
+
+def aggregate_results(
+    results: List[Dict[str, Any]],
+    max_chunks_per_doc: int = 3,
+) -> List[Dict[str, Any]]:
+    """
+    Quick function to aggregate search results.
+    
+    Args:
+        results: Raw chunk-level search results
+        max_chunks_per_doc: Max chunks to show per document
+        
+    Returns:
+        Document-level aggregated results
+    """
+    aggregator = ResultAggregator()
+    return aggregator.aggregate_by_document(results, max_chunks_per_doc)
+
+
+def get_aggregator() -> ResultAggregator:
+    """Returns a ResultAggregator instance."""
+    return ResultAggregator()

app/retrieval/chat.py

@@ -0,0 +1,158 @@
+from typing import Dict, List
+import logging
+
+import requests
+
+from app.config.settings import settings
+
+logger = logging.getLogger(__name__)
+
+
+class BaseDocumentChat:
+    """Common retrieved-context prompt builder."""
+
+    def answer(self, question: str, chunks: List[Dict], max_context_chars: int = 4000) -> str:
+        raise NotImplementedError
+
+    def healthcheck(self) -> None:
+        raise NotImplementedError
+
+    def _build_prompt(self, question: str, chunks: List[Dict], max_context_chars: int) -> str:
+        context = self._build_context(chunks, max_context_chars)
+        if not context:
+            return ""
+
+        return (
+            "Answer the user's question using only the document excerpts below. "
+            "If the excerpts do not contain the answer, say that the indexed documents do not show it.\n\n"
+            f"Document excerpts:\n{context}\n\n"
+            f"Question: {question}\n"
+            "Answer:"
+        )
+
+    def _build_context(self, chunks: List[Dict], max_context_chars: int) -> str:
+        parts = []
+        used_chars = 0
+
+        for chunk in chunks:
+            metadata = chunk.get("metadata", {})
+            filename = metadata.get("filename", "unknown")
+            chunk_index = metadata.get("chunk_index", 0)
+            text = chunk.get("text", "")
+            part = f"[{filename} chunk {chunk_index}]\n{text}"
+
+            if used_chars + len(part) > max_context_chars:
+                break
+
+            parts.append(part)
+            used_chars += len(part)
+
+        return "\n\n".join(parts)
+
+
+class OllamaDocumentChat(BaseDocumentChat):
+    """Answer questions with a local Ollama chat model."""
+
+    def __init__(self, base_url: str = None, model: str = None):
+        self.base_url = base_url or settings.OLLAMA_BASE_URL
+        self.model = model or settings.CHAT_MODEL
+        self.generate_endpoint = f"{self.base_url}/api/generate"
+
+    def answer(self, question: str, chunks: List[Dict], max_context_chars: int = 4000) -> str:
+        prompt = self._build_prompt(question, chunks, max_context_chars)
+        if not prompt:
+            return "I could not find enough retrieved context to answer this."
+
+        response = requests.post(
+            self.generate_endpoint,
+            json={"model": self.model, "prompt": prompt, "stream": False},
+            timeout=90,
+        )
+        response.raise_for_status()
+        return response.json().get("response", "").strip() or "The model returned an empty answer."
+
+    def healthcheck(self) -> None:
+        response = requests.get(f"{self.base_url}/api/tags", timeout=5)
+        response.raise_for_status()
+
+        models = response.json().get("models", [])
+        model_names = {model.get("name") for model in models}
+        if self.model not in model_names:
+            available = ", ".join(sorted(name for name in model_names if name)) or "none"
+            raise RuntimeError(
+                f"Chat model '{self.model}' is not installed in Ollama. "
+                f"Available models: {available}. Run: ollama pull {self.model}"
+            )
+
+
+class GroqDocumentChat(BaseDocumentChat):
+    """Answer questions with Groq's OpenAI-compatible chat completions API."""
+
+    def __init__(self, api_key: str = None, base_url: str = None, model: str = None):
+        self.api_key = api_key or settings.GROQ_API_KEY
+        self.base_url = (base_url or settings.GROQ_BASE_URL).rstrip("/")
+        self.model = model or settings.CHAT_MODEL
+        self.chat_endpoint = f"{self.base_url}/chat/completions"
+
+        if not self.api_key:
+            raise ValueError("GROQ_API_KEY is required when CHAT_PROVIDER=groq.")
+
+    def answer(self, question: str, chunks: List[Dict], max_context_chars: int = 4000) -> str:
+        prompt = self._build_prompt(question, chunks, max_context_chars)
+        if not prompt:
+            return "I could not find enough retrieved context to answer this."
+
+        response = requests.post(
+            self.chat_endpoint,
+            headers={
+                "Authorization": f"Bearer {self.api_key}",
+                "Content-Type": "application/json",
+            },
+            json={
+                "model": self.model,
+                "messages": [
+                    {
+                        "role": "system",
+                        "content": "You answer questions using only the retrieved document context.",
+                    },
+                    {"role": "user", "content": prompt},
+                ],
+                "temperature": 0.2,
+                "stream": False,
+            },
+            timeout=90,
+        )
+        response.raise_for_status()
+        choices = response.json().get("choices", [])
+        if not choices:
+            return "The chat model returned no answer."
+
+        return choices[0].get("message", {}).get("content", "").strip() or "The chat model returned an empty answer."
+
+    def healthcheck(self) -> None:
+        response = requests.get(
+            f"{self.base_url}/models",
+            headers={"Authorization": f"Bearer {self.api_key}"},
+            timeout=10,
+        )
+        response.raise_for_status()
+
+        models = response.json().get("data", [])
+        model_ids = {model.get("id") for model in models}
+        if self.model not in model_ids:
+            sample = ", ".join(sorted(model for model in model_ids if model)[:10]) or "none"
+            raise RuntimeError(
+                f"Groq model '{self.model}' was not found for this API key. "
+                f"Available examples: {sample}"
+            )
+
+
+def get_document_chat() -> BaseDocumentChat:
+    provider = settings.CHAT_PROVIDER.lower()
+
+    if provider == "ollama":
+        return OllamaDocumentChat()
+    if provider == "groq":
+        return GroqDocumentChat()
+
+    raise ValueError(f"Unsupported CHAT_PROVIDER='{settings.CHAT_PROVIDER}'. Use 'ollama' or 'groq'.")

app/retrieval/search.py

@@ -0,0 +1,205 @@
+from typing import List, Dict, Any, Optional
+import logging
+
+from app.ingestion.embedder import get_embedder
+from app.vectordb.factory import get_vector_store
+from app.config.settings import settings
+
+logging.basicConfig(
+    level=settings.LOG_LEVEL,
+    format=settings.LOG_FORMAT
+)
+
+logger = logging.getLogger(__name__)
+
+
+class DocumentSearcher:
+    """
+    Semantic search over document collection.
+    
+    Workflow:
+    1. Convert query text to embedding
+    2. Search vector database for similar chunks
+    3. Return ranked results with metadata
+    """
+
+    def __init__(
+        self,
+        chroma_persist_dir: str = None,
+        collection_name: str = None,
+    ):
+        """
+        Args:
+            chroma_persist_dir: Where ChromaDB stores data
+            collection_name: ChromaDB collection name
+        """
+        self.embedder = get_embedder()
+        
+        self.vector_store = get_vector_store()
+        
+        logger.info("✓ DocumentSearcher initialized")
+
+    def search(
+        self,
+        query: str,
+        top_k: int = None,
+        filter_metadata: Optional[Dict[str, Any]] = None,
+    ) -> List[Dict[str, Any]]:
+        """
+        Search for documents similar to the query.
+        
+        Args:
+            query: Search query text
+            top_k: Number of results to return
+            filter_metadata: Optional metadata filters (e.g., {"file_type": ".pdf"})
+            
+        Returns:
+            List of search results, each containing:
+                - 'text': Chunk text
+                - 'metadata': Chunk metadata
+                - 'similarity': Similarity score (lower distance = higher similarity)
+                - 'rank': Result rank (1-indexed)
+        """
+        if not query or not query.strip():
+            logger.warning("Empty query provided")
+            return []
+        
+        top_k = top_k or settings.DEFAULT_TOP_K
+        
+        logger.info(f"Searching for: '{query[:50]}...' (top_k={top_k})")
+        
+        # 1. Generate query embedding
+        try:
+            query_embedding = self.embedder.embed(query)
+        except Exception as e:
+            logger.error(f"Failed to generate query embedding: {e}")
+            raise RuntimeError(f"Query embedding failed: {e}")
+        
+        # 2. Search vector database
+        try:
+            raw_results = self.vector_store.similarity_search(
+                query_embedding=query_embedding,
+                top_k=top_k,
+                where=filter_metadata,
+            )
+        except Exception as e:
+            logger.error(f"Vector search failed: {e}")
+            raise RuntimeError(f"Search failed: {e}")
+        
+        # 3. Format results
+        results = self._format_results(raw_results)
+        
+        logger.info(f"✓ Found {len(results)} results")
+        
+        return results
+
+    def _format_results(self, raw_results: Dict[str, Any]) -> List[Dict[str, Any]]:
+        """
+        Convert ChromaDB results to standardized format.
+        
+        ChromaDB returns:
+            {
+                'ids': [[...]],
+                'documents': [[...]],
+                'metadatas': [[...]],
+                'distances': [[...]]
+            }
+        """
+        if not raw_results or not raw_results.get('documents'):
+            return []
+        
+        # ChromaDB returns nested lists (batch query support)
+        # We only send one query, so take first element
+        documents = raw_results['documents'][0]
+        metadatas = raw_results['metadatas'][0]
+        distances = raw_results['distances'][0]
+        
+        results = []
+        
+        for rank, (doc, metadata, distance) in enumerate(zip(documents, metadatas, distances), start=1):
+            # Convert cosine distance to similarity
+            # Cosine distance: 0 (identical) to 2 (opposite)
+            # Cosine similarity: 1 - distance (ranges 0 to 1)
+            similarity = 1.0 - distance
+            
+            result = {
+                'text': doc,
+                'metadata': metadata,
+                'distance': distance,
+                'similarity': similarity,
+                'rank': rank,
+            }
+            
+            results.append(result)
+        
+        return results
+
+    def search_with_threshold(
+        self,
+        query: str,
+        top_k: int = None,
+        similarity_threshold: float = None,
+    ) -> List[Dict[str, Any]]:
+        """
+        Search and filter by minimum similarity threshold.
+        
+        Args:
+            query: Search query
+            top_k: Max results to return
+            similarity_threshold: Minimum similarity score (0-1)
+            
+        Returns:
+            Filtered search results
+        """
+        threshold = similarity_threshold or settings.SIMILARITY_THRESHOLD
+        
+        results = self.search(query, top_k)
+        
+        # Filter by threshold
+        filtered_results = [
+            r for r in results
+            if r['similarity'] >= threshold
+        ]
+        
+        logger.info(
+            f"Filtered {len(results)} -> {len(filtered_results)} results "
+            f"(threshold={threshold})"
+        )
+        
+        return filtered_results
+
+    def get_collection_stats(self) -> Dict[str, Any]:
+        """
+        Get statistics about the indexed documents.
+        
+        Returns:
+            Dictionary with collection metadata
+        """
+        return self.vector_store.get_collection_info()
+
+
+# ============================================================
+# Convenience Function
+# ============================================================
+
+def search_documents(
+    query: str,
+    top_k: int = None,
+) -> List[Dict[str, Any]]:
+    """
+    Quick search function.
+    
+    Args:
+        query: Search query
+        top_k: Number of results
+        
+    Returns:
+        Search results
+    """
+    searcher = DocumentSearcher()
+    return searcher.search(query, top_k)
+
+
+def get_searcher() -> DocumentSearcher:
+    """Returns a configured DocumentSearcher instance."""
+    return DocumentSearcher()

app/sources/__init__.py

@@ -0,0 +1 @@
+"""Source connectors for external document locations."""

app/sources/connectors.py

@@ -0,0 +1,226 @@
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Dict, List
+from urllib.parse import urlparse
+import logging
+import re
+import zipfile
+
+import requests
+
+from app.config.settings import settings
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class SourceDocument:
+    """A downloaded source file plus metadata about where it came from."""
+
+    path: Path
+    metadata: Dict[str, str]
+
+
+class SourceConnector:
+    """Fetch public source links into temporary local files for ingestion."""
+
+    def fetch(self, source_url: str, workspace_dir: Path) -> List[SourceDocument]:
+        parsed = urlparse(source_url)
+        host = parsed.netloc.lower()
+
+        if "github.com" in host:
+            return self._fetch_github_repo(source_url, workspace_dir)
+
+        if "drive.google.com" in host:
+            folder_id = self._parse_google_drive_folder_id(source_url)
+            if folder_id:
+                return self._fetch_google_drive_folder(source_url, folder_id, workspace_dir)
+
+            return [self._fetch_google_drive_file(source_url, workspace_dir)]
+
+        raise ValueError("Supported demo sources are public GitHub repository URLs and public Google Drive file links.")
+
+    def _fetch_github_repo(self, source_url: str, workspace_dir: Path) -> List[SourceDocument]:
+        owner, repo = self._parse_github_repo(source_url)
+        repo_api_url = f"https://api.github.com/repos/{owner}/{repo}"
+
+        repo_response = requests.get(repo_api_url, timeout=20)
+        repo_response.raise_for_status()
+        default_branch = repo_response.json().get("default_branch", "main")
+
+        zip_url = f"https://codeload.github.com/{owner}/{repo}/zip/refs/heads/{default_branch}"
+        zip_path = workspace_dir / f"{owner}_{repo}.zip"
+
+        self._download_file(zip_url, zip_path)
+
+        extract_dir = workspace_dir / f"{owner}_{repo}"
+        with zipfile.ZipFile(zip_path) as archive:
+            archive.extractall(extract_dir)
+
+        supported_files = [
+            path
+            for path in extract_dir.rglob("*")
+            if path.is_file() and path.suffix.lower() in settings.SUPPORTED_FILE_TYPES
+        ]
+
+        documents = []
+        for path in supported_files:
+            relative_path = self._repo_relative_path(path, owner, repo, default_branch)
+            source_file_url = f"https://github.com/{owner}/{repo}/blob/{default_branch}/{relative_path}"
+            documents.append(
+                SourceDocument(
+                    path=path,
+                    metadata={
+                        "source_type": "github",
+                        "source_url": source_file_url,
+                        "source_root": source_url,
+                        "source_path": relative_path,
+                        "document_id": f"github:{owner}/{repo}:{default_branch}:{relative_path}",
+                    },
+                )
+            )
+
+        logger.info("Fetched %s supported documents from %s/%s", len(documents), owner, repo)
+        return documents
+
+    def _fetch_google_drive_file(self, source_url: str, workspace_dir: Path) -> SourceDocument:
+        file_id = self._parse_google_drive_file_id(source_url)
+        if not file_id:
+            raise ValueError("Google Drive URL must be a public file link or public folder link.")
+
+        download_url = f"https://drive.google.com/uc?export=download&id={file_id}"
+        response = requests.get(download_url, timeout=30, stream=True)
+        response.raise_for_status()
+
+        filename = self._filename_from_response(response) or f"google_drive_{file_id}"
+        suffix = Path(filename).suffix.lower()
+        if suffix not in settings.SUPPORTED_FILE_TYPES:
+            raise ValueError(f"Unsupported Google Drive file type: {suffix or 'unknown'}")
+
+        destination = workspace_dir / filename
+        with open(destination, "wb") as file:
+            for chunk in response.iter_content(chunk_size=1024 * 1024):
+                if chunk:
+                    file.write(chunk)
+
+        return SourceDocument(
+            path=destination,
+            metadata={
+                "source_type": "google_drive",
+                "source_url": source_url,
+                "source_root": source_url,
+                "source_path": filename,
+                "document_id": f"google_drive:{file_id}",
+            },
+        )
+
+    def _fetch_google_drive_folder(
+        self,
+        source_url: str,
+        folder_id: str,
+        workspace_dir: Path,
+    ) -> List[SourceDocument]:
+        try:
+            import gdown
+        except ImportError as exc:
+            raise RuntimeError("Google Drive folder ingestion requires the gdown package. Install requirements.txt again.") from exc
+
+        output_dir = workspace_dir / f"google_drive_{folder_id}"
+        output_dir.mkdir(parents=True, exist_ok=True)
+
+        downloaded_paths = gdown.download_folder(
+            url=source_url,
+            output=str(output_dir),
+            quiet=True,
+            use_cookies=False,
+        )
+
+        if downloaded_paths is None:
+            raise RuntimeError("Could not download the Google Drive folder. Confirm it is public and accessible.")
+
+        supported_files = [
+            Path(path)
+            for path in downloaded_paths
+            if Path(path).is_file() and Path(path).suffix.lower() in settings.SUPPORTED_FILE_TYPES
+        ]
+
+        documents = []
+        for path in supported_files:
+            relative_path = str(path.relative_to(output_dir)).replace("\\", "/")
+            documents.append(
+                SourceDocument(
+                    path=path,
+                    metadata={
+                        "source_type": "google_drive_folder",
+                        "source_url": source_url,
+                        "source_root": source_url,
+                        "source_path": relative_path,
+                        "document_id": f"google_drive_folder:{folder_id}:{relative_path}",
+                    },
+                )
+            )
+
+        logger.info("Fetched %s supported documents from Google Drive folder %s", len(documents), folder_id)
+        return documents
+
+    def _download_file(self, url: str, destination: Path) -> None:
+        response = requests.get(url, timeout=60, stream=True)
+        response.raise_for_status()
+
+        with open(destination, "wb") as file:
+            for chunk in response.iter_content(chunk_size=1024 * 1024):
+                if chunk:
+                    file.write(chunk)
+
+    def _parse_github_repo(self, source_url: str) -> tuple[str, str]:
+        parsed = urlparse(source_url)
+        parts = [part for part in parsed.path.split("/") if part]
+
+        if len(parts) < 2:
+            raise ValueError("GitHub URL must look like https://github.com/owner/repo")
+
+        return parts[0], parts[1].removesuffix(".git")
+
+    def _repo_relative_path(self, path: Path, owner: str, repo: str, branch: str) -> str:
+        marker = f"{repo}-{branch}"
+        parts = path.parts
+
+        if marker in parts:
+            marker_index = parts.index(marker)
+            return str(Path(*parts[marker_index + 1 :])).replace("\\", "/")
+
+        fallback = path.name
+        logger.warning("Could not derive repo-relative path for %s; using %s", path, fallback)
+        return fallback
+
+    def _parse_google_drive_file_id(self, source_url: str) -> str | None:
+        patterns = [
+            r"/file/d/([^/]+)",
+            r"[?&]id=([^&]+)",
+        ]
+
+        for pattern in patterns:
+            match = re.search(pattern, source_url)
+            if match:
+                return match.group(1)
+
+        return None
+
+    def _parse_google_drive_folder_id(self, source_url: str) -> str | None:
+        match = re.search(r"/drive/folders/([^/?]+)", source_url)
+        if match:
+            return match.group(1)
+
+        return None
+
+    def _filename_from_response(self, response: requests.Response) -> str | None:
+        content_disposition = response.headers.get("content-disposition", "")
+        match = re.search(r'filename="?([^";]+)"?', content_disposition)
+        if match:
+            return match.group(1)
+
+        return None
+
+
+def get_source_connector() -> SourceConnector:
+    return SourceConnector()

app/ui/main.py

@@ -0,0 +1,715 @@
+import html
+import sys
+import time
+from pathlib import Path
+from textwrap import dedent
+
+import streamlit as st
+
+PROJECT_ROOT = Path(__file__).resolve().parents[2]
+if str(PROJECT_ROOT) not in sys.path:
+    sys.path.insert(0, str(PROJECT_ROOT))
+
+from app.config.settings import settings
+from app.ingestion.ingest import get_pipeline
+from app.retrieval.aggregator import get_aggregator
+from app.retrieval.chat import get_document_chat
+from app.retrieval.search import get_searcher
+
+
+st.set_page_config(
+    page_title="SourceLink AI",
+    page_icon="SL",
+    layout="wide",
+    initial_sidebar_state="collapsed",
+)
+
+
+def inject_styles() -> None:
+    st.html(
+        dedent(
+            """
+        <style>
+        :root {
+            --ink: #101828;
+            --muted: #667085;
+            --line: #e6e9ef;
+            --soft: #f7f8fb;
+            --violet: #7047eb;
+            --pink: #d946ef;
+            --cyan: #06b6d4;
+            --green: #10b981;
+        }
+
+        .stApp {
+            background: #f8fafc;
+            color: var(--ink);
+        }
+
+        .block-container {
+            max-width: 1180px;
+            padding-top: 1.1rem;
+            padding-bottom: 3rem;
+        }
+
+        section[data-testid="stSidebar"] {
+            background: #ffffff;
+            border-right: 1px solid var(--line);
+        }
+
+        h1, h2, h3, p {
+            letter-spacing: 0;
+        }
+
+        div[data-testid="stButton"] > button,
+        div[data-testid="stDownloadButton"] > button,
+        a[data-testid="stLinkButton"] {
+            border-radius: 10px;
+            min-height: 42px;
+            font-weight: 700;
+            border: 1px solid #ded7ff;
+            box-shadow: none;
+            background: #ffffff;
+            color: #111827 !important;
+        }
+
+        div[data-testid="stButton"] > button[kind="primary"] {
+            background: linear-gradient(135deg, var(--violet), var(--pink));
+            border: 0;
+            color: #ffffff !important;
+        }
+
+        div[data-testid="stButton"] > button[kind="primary"] *,
+        div[data-testid="stButton"] > button[kind="secondary"] *,
+        div[data-testid="stDownloadButton"] > button *,
+        a[data-testid="stLinkButton"] * {
+            color: inherit !important;
+        }
+
+        div[data-testid="stButton"] > button[kind="secondary"] {
+            background: #ffffff !important;
+            color: #111827 !important;
+            border: 1px solid #ded7ff !important;
+        }
+
+        section[data-testid="stSidebar"] div[data-testid="stButton"] > button {
+            background: #ffffff !important;
+            color: #111827 !important;
+            border: 1px solid #e6e9ef !important;
+        }
+
+        div[data-testid="stFileUploader"] button {
+            background: #ffffff !important;
+            color: #111827 !important;
+            border: 1px solid #ded7ff !important;
+        }
+
+        div[data-testid="stFileUploader"] button * {
+            color: #111827 !important;
+        }
+
+        div[data-testid="stTextInput"] input {
+            border-radius: 12px;
+            min-height: 48px;
+            border-color: var(--line);
+            background: #ffffff;
+            color: #111827;
+        }
+
+        div[data-testid="stTextInput"] input::placeholder {
+            color: #8b95a7;
+        }
+
+        div[data-testid="stFileUploader"] section {
+            border-radius: 16px;
+            border: 1.5px dashed #d8ddec;
+            background: #fbfcff;
+            min-height: 108px;
+        }
+
+        div[data-testid="stExpander"] {
+            border: 1px solid var(--line);
+            border-radius: 16px;
+            overflow: hidden;
+            background: #ffffff;
+            box-shadow: 0 14px 40px rgba(16, 24, 40, 0.05);
+        }
+
+        .app-shell {
+            width: 100%;
+        }
+
+        .topbar {
+            display: flex;
+            align-items: center;
+            justify-content: space-between;
+            margin: 0 auto 4.8rem auto;
+        }
+
+        .brand {
+            display: flex;
+            align-items: center;
+            gap: 12px;
+            font-weight: 800;
+            color: #111827;
+            font-size: 1.05rem;
+        }
+
+        .brand-mark {
+            width: 42px;
+            height: 42px;
+            display: grid;
+            place-items: center;
+            border-radius: 14px;
+            color: #ffffff;
+            font-weight: 900;
+            background: linear-gradient(135deg, #6d5dfc, #e250d5);
+            box-shadow: 0 14px 30px rgba(112, 71, 235, 0.24);
+        }
+
+        .status-pill {
+            display: inline-flex;
+            align-items: center;
+            gap: 9px;
+            padding: 8px 16px;
+            border-radius: 999px;
+            border: 1px solid var(--line);
+            background: rgba(255, 255, 255, 0.82);
+            color: #6b7280;
+            font-size: 0.86rem;
+            font-weight: 700;
+        }
+
+        .dot {
+            width: 9px;
+            height: 9px;
+            border-radius: 999px;
+            background: linear-gradient(135deg, var(--violet), var(--cyan));
+        }
+
+        .hero {
+            text-align: center;
+            margin: 0 auto 2.8rem auto;
+        }
+
+        .hero h1 {
+            max-width: 860px;
+            margin: 1.7rem auto 1.2rem auto;
+            font-size: clamp(3.2rem, 6vw, 5.7rem);
+            line-height: 0.98;
+            font-weight: 900;
+            color: #111827;
+        }
+
+        .hero-gradient {
+            display: block;
+            background: linear-gradient(135deg, #6545f5 0%, #b34df0 48%, #e250d5 100%);
+            -webkit-background-clip: text;
+            background-clip: text;
+            color: transparent;
+        }
+
+        .hero p {
+            max-width: 760px;
+            margin: 0 auto;
+            font-size: 1.18rem;
+            line-height: 1.55;
+            color: var(--muted);
+        }
+
+        .feature-grid {
+            display: grid;
+            grid-template-columns: repeat(3, minmax(0, 1fr));
+            gap: 18px;
+            max-width: 840px;
+            margin: 2.8rem auto 3.2rem auto;
+        }
+
+        .feature-card {
+            background: #ffffff;
+            border: 1px solid var(--line);
+            border-radius: 16px;
+            padding: 24px;
+            min-height: 130px;
+            box-shadow: 0 20px 44px rgba(16, 24, 40, 0.06);
+        }
+
+        .feature-icon {
+            color: var(--violet);
+            font-size: 1.55rem;
+            line-height: 1;
+            margin-bottom: 20px;
+        }
+
+        .feature-title {
+            color: #1f2937;
+            font-weight: 800;
+            margin-bottom: 4px;
+        }
+
+        .feature-copy {
+            color: var(--muted);
+            font-size: 0.92rem;
+        }
+
+        .panel {
+            background: #ffffff;
+            border: 1px solid var(--line);
+            border-radius: 18px;
+            padding: 28px;
+            box-shadow: 0 24px 60px rgba(16, 24, 40, 0.07);
+            margin-bottom: 24px;
+        }
+
+        .panel-title {
+            display: flex;
+            align-items: center;
+            justify-content: space-between;
+            gap: 18px;
+            margin-bottom: 18px;
+        }
+
+        .panel-title h2 {
+            font-size: 1.25rem;
+            margin: 0;
+            color: #111827;
+        }
+
+        .panel-title p {
+            margin: 4px 0 0 0;
+            color: var(--muted);
+        }
+
+        .metric-row {
+            display: grid;
+            grid-template-columns: repeat(4, minmax(0, 1fr));
+            gap: 14px;
+            margin: 10px 0 26px 0;
+        }
+
+        .mini-metric {
+            border: 1px solid var(--line);
+            border-radius: 14px;
+            padding: 15px 16px;
+            background: #fbfcff;
+        }
+
+        .mini-metric strong {
+            display: block;
+            font-size: 1.25rem;
+            color: #111827;
+        }
+
+        .mini-metric span {
+            color: var(--muted);
+            font-size: 0.84rem;
+        }
+
+        .section-kicker {
+            color: var(--violet);
+            font-weight: 800;
+            margin-bottom: 8px;
+            font-size: 0.82rem;
+            text-transform: uppercase;
+        }
+
+        .excerpt-box {
+            border: 1px solid #e9ecf3;
+            border-radius: 14px;
+            padding: 16px;
+            background: #fbfcff;
+            margin: 12px 0;
+        }
+
+        .excerpt-meta {
+            color: var(--muted);
+            font-size: 0.84rem;
+            font-weight: 700;
+            margin-bottom: 8px;
+        }
+
+        .excerpt-text {
+            color: #1f2937;
+            line-height: 1.55;
+        }
+
+        @media (max-width: 780px) {
+            .topbar {
+                margin-bottom: 2.8rem;
+            }
+            .hero h1 {
+                font-size: 3rem;
+            }
+            .feature-grid,
+            .metric-row {
+                grid-template-columns: 1fr;
+            }
+            .panel {
+                padding: 20px;
+            }
+        }
+        </style>
+        """
+        ),
+    )
+
+
+def initialize_state() -> None:
+    if "pipeline" not in st.session_state:
+        st.session_state.pipeline = get_pipeline()
+        st.session_state.searcher = get_searcher()
+        st.session_state.aggregator = get_aggregator()
+        st.session_state.chat = get_document_chat()
+        st.session_state.last_chunks = []
+        st.session_state.last_documents = []
+
+
+def ingest_uploaded_files(uploaded_files) -> None:
+    with st.spinner("Indexing uploaded files..."):
+        progress_bar = st.progress(0)
+        total_chunks = 0
+
+        for index, uploaded_file in enumerate(uploaded_files):
+            temp_path = settings.RAW_DATA_DIR / uploaded_file.name
+            with open(temp_path, "wb") as file:
+                file.write(uploaded_file.getbuffer())
+
+            try:
+                num_chunks = st.session_state.pipeline.ingest_file(
+                    str(temp_path),
+                    extra_metadata={
+                        "source_type": "upload",
+                        "source_path": str(temp_path),
+                        "document_id": f"upload:{uploaded_file.name}",
+                    },
+                )
+                total_chunks += num_chunks
+                st.success(f"{uploaded_file.name}: {num_chunks} chunks")
+            except Exception as exc:
+                st.error(f"{uploaded_file.name}: {exc}")
+
+            progress_bar.progress((index + 1) / len(uploaded_files))
+
+        st.success(f"Indexed {total_chunks} chunks from {len(uploaded_files)} uploaded files.")
+        time.sleep(1)
+        st.rerun()
+
+
+def render_document_actions(doc) -> None:
+    source_url = doc["metadata"].get("source_url")
+    local_file = settings.RAW_DATA_DIR / doc["filename"]
+
+    if source_url:
+        st.link_button("Open source", source_url, use_container_width=True)
+    elif local_file.exists():
+        with open(local_file, "rb") as file:
+            st.download_button(
+                label="Download file",
+                data=file.read(),
+                file_name=doc["filename"],
+                mime="application/octet-stream",
+                use_container_width=True,
+            )
+    else:
+        st.caption("Original file is not available locally.")
+
+
+def render_results(aggregated_results) -> None:
+    for doc in aggregated_results:
+        safe_filename = html.escape(doc["filename"])
+        with st.expander(
+            f"{doc['filename']}  |  relevance {doc['relevance_score']:.3f}",
+            expanded=True,
+        ):
+            meta_col, size_col, match_col, action_col = st.columns(4)
+            with meta_col:
+                st.caption(f"Source: {doc['metadata'].get('source_type', 'local')}")
+            with size_col:
+                st.caption(f"Type: {doc['metadata'].get('file_type', 'unknown')}")
+            with match_col:
+                st.caption(f"Matches: {doc['num_matching_chunks']}")
+            with action_col:
+                render_document_actions(doc)
+
+            st.markdown(f"**Relevant excerpts from {safe_filename}**")
+            for index, chunk in enumerate(doc["chunks"], start=1):
+                text = html.escape(chunk["text"])
+                similarity = chunk["similarity"]
+                chunk_index = chunk["metadata"].get("chunk_index", 0)
+                st.html(
+                    dedent(
+                        f"""
+                    <div class="excerpt-box">
+                        <div class="excerpt-meta">Excerpt {index} - similarity {similarity:.3f} - chunk #{chunk_index}</div>
+                        <div class="excerpt-text">{text}</div>
+                    </div>
+                    """
+                    ),
+                )
+
+
+def render_sidebar() -> None:
+    with st.sidebar:
+        st.header("Controls")
+
+        status = st.session_state.pipeline.get_status()
+        st.metric("Indexed chunks", status["total_chunks"])
+        st.caption(f"Vector DB: {settings.VECTOR_DB_BACKEND}")
+
+        st.divider()
+
+        st.subheader("Local demo data")
+        if st.button("Index data/raw", use_container_width=True):
+            with st.spinner("Indexing local demo directory..."):
+                try:
+                    results = st.session_state.pipeline.ingest_directory(str(settings.RAW_DATA_DIR))
+                    st.success(f"Indexed {len(results)} files.")
+                    time.sleep(1)
+                    st.rerun()
+                except Exception as exc:
+                    st.error(str(exc))
+
+        st.divider()
+
+        st.subheader("Danger zone")
+        if st.button("Clear vector index", use_container_width=True):
+            if st.session_state.get("confirm_reset", False):
+                st.session_state.pipeline.reset_database()
+                st.session_state.confirm_reset = False
+                st.session_state.pipeline = get_pipeline()
+                st.session_state.searcher = get_searcher()
+                st.session_state.aggregator = get_aggregator()
+                st.session_state.last_chunks = []
+                st.session_state.last_documents = []
+                st.success("Vector index cleared.")
+                time.sleep(1)
+                st.rerun()
+            else:
+                st.session_state.confirm_reset = True
+                st.warning("Click again to confirm deletion.")
+
+
+def render_hero(status_count: int) -> None:
+    st.html(
+        dedent(
+            f"""
+        <div class="app-shell">
+            <div class="topbar">
+                <div class="brand">
+                    <div class="brand-mark">SL</div>
+                    <div>SourceLink AI</div>
+                </div>
+                <div class="status-pill"><span class="dot"></span>{status_count} indexed chunks</div>
+            </div>
+
+            <section class="hero">
+                <div class="status-pill"><span class="dot"></span>Your documents, now an AI assistant</div>
+                <h1>
+                    Connect your sources.
+                    <span class="hero-gradient">Chat with every document.</span>
+                </h1>
+                <p>
+                    Paste a Drive folder, connect a public repository, or upload files.
+                    SourceLink indexes the knowledge and answers only from retrieved document context.
+                </p>
+            </section>
+
+            <div class="feature-grid">
+                <div class="feature-card">
+                    <div class="feature-icon">01</div>
+                    <div class="feature-title">Connect Sources</div>
+                    <div class="feature-copy">Google Drive folders, public GitHub repos, and uploads.</div>
+                </div>
+                <div class="feature-card">
+                    <div class="feature-icon">02</div>
+                    <div class="feature-title">Index Knowledge</div>
+                    <div class="feature-copy">Chunks, embeddings, and source links stay searchable.</div>
+                </div>
+                <div class="feature-card">
+                    <div class="feature-icon">03</div>
+                    <div class="feature-title">Ask Questions</div>
+                    <div class="feature-copy">Chat with the retrieved files and open originals instantly.</div>
+                </div>
+            </div>
+        </div>
+        """
+        ),
+    )
+
+
+inject_styles()
+initialize_state()
+render_sidebar()
+
+status = st.session_state.pipeline.get_status()
+render_hero(status["total_chunks"])
+
+st.html(
+    dedent(
+        """
+    <div class="panel">
+        <div class="panel-title">
+            <div>
+                <div class="section-kicker">Index workspace</div>
+                <h2>Add documents to your assistant</h2>
+                <p>Use a public source link or upload files directly for this demo.</p>
+            </div>
+        </div>
+    </div>
+    """
+    ),
+)
+
+source_col, upload_col = st.columns([1, 1], gap="large")
+
+with source_col:
+    st.markdown("#### Source link")
+    st.caption(f"Indexing target: {settings.VECTOR_DB_BACKEND} / {settings.COLLECTION_NAME}")
+    source_url = st.text_input(
+        "Public GitHub repository or Google Drive link",
+        placeholder="https://drive.google.com/drive/folders/...",
+        help="Supports public GitHub repositories and public Google Drive file/folder links.",
+        label_visibility="collapsed",
+    )
+    if st.button("Index source link", type="primary", use_container_width=True):
+        if not source_url.strip():
+            st.warning("Paste a public source link first.")
+        else:
+            with st.spinner("Fetching and indexing source files..."):
+                try:
+                    results = st.session_state.pipeline.ingest_source_url(source_url.strip())
+                    if results:
+                        st.success(f"Indexed {len(results)} files.")
+                        for filename, count in results.items():
+                            st.write(f"{filename}: {count} chunks")
+                    else:
+                        st.warning("No supported files found in that source.")
+                    time.sleep(1)
+                    st.rerun()
+                except Exception as exc:
+                    st.error(str(exc))
+
+with upload_col:
+    st.markdown("#### Upload files")
+    st.caption(f"Indexing target: {settings.VECTOR_DB_BACKEND} / {settings.COLLECTION_NAME}")
+    uploaded_files = st.file_uploader(
+        "Upload PDF, TXT, DOCX, or Markdown",
+        type=["pdf", "txt", "docx", "md"],
+        accept_multiple_files=True,
+        label_visibility="collapsed",
+    )
+    if st.button("Index uploaded files", use_container_width=True):
+        if uploaded_files:
+            ingest_uploaded_files(uploaded_files)
+        else:
+            st.warning("Upload at least one file first.")
+
+st.html(
+    dedent(
+        f"""
+    <div class="metric-row">
+        <div class="mini-metric"><strong>{status["total_chunks"]}</strong><span>Indexed chunks</span></div>
+        <div class="mini-metric"><strong>{html.escape(settings.VECTOR_DB_BACKEND)}</strong><span>Vector backend</span></div>
+        <div class="mini-metric"><strong>{html.escape(settings.EMBEDDING_PROVIDER)}</strong><span>Embedding provider</span></div>
+        <div class="mini-metric"><strong>{html.escape(settings.CHAT_PROVIDER)}</strong><span>Chat provider</span></div>
+    </div>
+    """
+    ),
+)
+
+st.html(
+    dedent(
+        """
+    <div class="panel">
+        <div class="panel-title">
+            <div>
+                <div class="section-kicker">Semantic search</div>
+                <h2>Find the most relevant files</h2>
+                <p>Search across all indexed chunks, then open the source or ask follow-up questions.</p>
+            </div>
+        </div>
+    </div>
+    """
+    ),
+)
+
+query = st.text_input(
+    "Search documents",
+    placeholder="Ask about a topic, policy, chapter, API, or concept...",
+    label_visibility="collapsed",
+)
+
+search_col, top_k_col, threshold_col = st.columns([2, 1, 1])
+with search_col:
+    search_requested = st.button("Search documents", type="primary", use_container_width=True)
+with top_k_col:
+    top_k = st.slider("Results", min_value=1, max_value=20, value=5)
+with threshold_col:
+    similarity_threshold = st.slider("Min similarity", 0.0, 1.0, 0.3, 0.01)
+
+if search_requested and query.strip():
+    with st.spinner("Searching indexed documents..."):
+        try:
+            results = st.session_state.searcher.search(query, top_k=top_k * 3)
+            results = [result for result in results if result["similarity"] >= similarity_threshold]
+
+            if not results:
+                st.session_state.last_chunks = []
+                st.session_state.last_documents = []
+                st.warning("No results found. Try a broader query or lower the similarity threshold.")
+            else:
+                aggregated = st.session_state.aggregator.aggregate_by_document(results, max_chunks_per_doc=3)
+                st.session_state.last_chunks = [chunk for doc in aggregated for chunk in doc["chunks"]]
+                st.session_state.last_documents = aggregated
+                st.success(f"Found {len(aggregated)} relevant documents.")
+                render_results(aggregated)
+        except Exception as exc:
+            st.error(f"Search error: {exc}")
+elif st.session_state.last_documents:
+    render_results(st.session_state.last_documents)
+
+st.html(
+    dedent(
+        """
+    <div class="panel">
+        <div class="panel-title">
+            <div>
+                <div class="section-kicker">Document chat</div>
+                <h2>Ask the retrieved context</h2>
+                <p>The answer is grounded in the files returned by your most recent search.</p>
+            </div>
+        </div>
+    </div>
+    """
+    ),
+)
+
+chat_question = st.text_input(
+    "Ask a follow-up",
+    placeholder="What should I know from the returned documents?",
+    label_visibility="collapsed",
+)
+
+if st.button("Ask retrieved context", use_container_width=True):
+    if not chat_question.strip():
+        st.warning("Enter a follow-up question first.")
+    elif not st.session_state.last_chunks:
+        st.warning("Run a search first so the chat has document context.")
+    else:
+        with st.spinner("Asking the chat model..."):
+            try:
+                st.session_state.chat.healthcheck()
+                answer = st.session_state.chat.answer(chat_question, st.session_state.last_chunks)
+                st.success(answer)
+            except Exception as exc:
+                st.error(f"Chat model error: {exc}")
+                if settings.CHAT_PROVIDER.lower() == "groq":
+                    st.info("Make sure GROQ_API_KEY is set in .env and CHAT_MODEL is available in your Groq account.")
+                else:
+                    st.info(f"Make sure Ollama is running and the chat model is installed: ollama pull {settings.CHAT_MODEL}")
+
+st.caption(
+    f"Embeddings: {settings.EMBEDDING_PROVIDER} {settings.EMBEDDING_MODEL} - "
+    f"Chat: {settings.CHAT_PROVIDER} {settings.CHAT_MODEL} - "
+    f"Vector DB: {settings.VECTOR_DB_BACKEND}"
+)

app/vectordb/base.py

@@ -0,0 +1,28 @@
+from typing import Any, Dict, List, Optional, Protocol
+
+
+class VectorStore(Protocol):
+    """Interface every vector database backend must implement."""
+
+    def add_documents(
+        self,
+        ids: List[str],
+        documents: List[str],
+        embeddings: List[List[float]],
+        metadatas: List[Dict[str, Any]],
+    ) -> None:
+        ...
+
+    def similarity_search(
+        self,
+        query_embedding: List[float],
+        top_k: int = 5,
+        where: Optional[Dict[str, Any]] = None,
+    ) -> Dict[str, Any]:
+        ...
+
+    def delete_all(self) -> None:
+        ...
+
+    def get_collection_info(self) -> Dict[str, Any]:
+        ...

app/vectordb/chroma_client.py

@@ -0,0 +1,143 @@
+from typing import List, Dict, Any, Optional
+import os
+
+import chromadb
+from chromadb.config import Settings
+
+
+class ChromaClient:
+    """
+    Centralized ChromaDB client abstraction.
+
+    Responsibilities:
+    - Initialize persistent ChromaDB
+    - Create / load collections
+    - Add document chunks with embeddings
+    - Perform similarity search
+
+    This class is the SINGLE point of interaction with ChromaDB.
+    """
+
+    def __init__(
+        self,
+        persist_directory: str,
+        collection_name: str = "documents",
+        embedding_function=None,
+    ):
+        """
+        Args:
+            persist_directory: Path where ChromaDB will persist data
+            collection_name: Name of the Chroma collection
+            embedding_function: Optional embedding function (NOT required if
+                                embeddings are precomputed)
+        """
+
+        self.persist_directory = persist_directory
+        self.collection_name = collection_name
+
+        os.makedirs(self.persist_directory, exist_ok=True)
+
+        self.client = chromadb.PersistentClient(
+        path=self.persist_directory,
+        settings=Settings(anonymized_telemetry=False),
+        )
+
+        
+
+        self.collection = self.client.get_or_create_collection(
+            name=self.collection_name,
+            embedding_function=embedding_function,
+            metadata={
+                "description": "Semantic document retrieval collection",
+                "hnsw:space": "cosine"  # Use cosine similarity (0-1 scale)
+            },
+        )
+
+    # ------------------------------------------------------------------
+    # Ingestion API
+    # ------------------------------------------------------------------
+
+    def add_documents(
+        self,
+        ids: List[str],
+        documents: List[str],
+        embeddings: List[List[float]],
+        metadatas: List[Dict[str, Any]],
+    ) -> None:
+        """
+        Add document chunks to the vector store.
+
+        Args:
+            ids: Unique IDs for each chunk
+            documents: Chunk text
+            embeddings: Precomputed embedding vectors
+            metadatas: Metadata per chunk (doc name, page, chunk index, etc.)
+        """
+
+        if not (len(ids) == len(documents) == len(embeddings) == len(metadatas)):
+            raise ValueError("ids, documents, embeddings, and metadatas must be same length")
+
+        self.collection.add(
+            ids=ids,
+            documents=documents,
+            embeddings=embeddings,
+            metadatas=metadatas,
+        )
+
+    # ------------------------------------------------------------------
+    # Retrieval API
+    # ------------------------------------------------------------------
+
+    def similarity_search(
+        self,
+        query_embedding: List[float],
+        top_k: int = 5,
+        where: Optional[Dict[str, Any]] = None,
+    ) -> Dict[str, Any]:
+        """
+        Perform similarity search using a query embedding.
+
+        Args:
+            query_embedding: Embedded query vector
+            top_k: Number of nearest neighbors
+            where: Optional metadata filter
+
+        Returns:
+            Raw ChromaDB query result
+        """
+
+        result = self.collection.query(
+            query_embeddings=[query_embedding],
+            n_results=top_k,
+            where=where,
+            include=["documents", "metadatas", "distances"],
+        )
+
+        return result
+
+    # ------------------------------------------------------------------
+    # Utility / Maintenance
+    # ------------------------------------------------------------------
+
+    def count(self) -> int:
+        """Return number of stored embeddings."""
+        return self.collection.count()
+
+    def delete_all(self) -> None:
+        """Delete all data in the collection."""
+        self.client.delete_collection(self.collection_name)
+        self.collection = self.client.get_or_create_collection(
+            name=self.collection_name,
+            metadata={
+                "description": "Semantic document retrieval collection",
+                "hnsw:space": "cosine"
+            },
+        )
+
+    def get_collection_info(self) -> Dict[str, Any]:
+        """Return basic collection metadata."""
+        return {
+            "name": self.collection.name,
+            "count": self.collection.count(),
+            "metadata": self.collection.metadata,
+        }

app/vectordb/factory.py

@@ -0,0 +1,26 @@
+from app.config.settings import settings
+from app.vectordb.chroma_client import ChromaClient
+from app.vectordb.zilliz_client import ZillizClient
+
+
+def get_vector_store():
+    backend = settings.VECTOR_DB_BACKEND.lower()
+
+    if backend == "chroma":
+        return ChromaClient(
+            persist_directory=str(settings.CHROMA_PERSIST_DIR),
+            collection_name=settings.COLLECTION_NAME,
+        )
+
+    if backend == "zilliz":
+        return ZillizClient(
+            uri=settings.ZILLIZ_URI,
+            token=settings.ZILLIZ_TOKEN,
+            collection_name=settings.COLLECTION_NAME,
+            dimension=settings.EMBEDDING_DIMENSION,
+        )
+
+    raise ValueError(
+        f"Unsupported VECTOR_DB_BACKEND='{settings.VECTOR_DB_BACKEND}'. "
+        "Available backends: chroma, zilliz."
+    )

app/vectordb/zilliz_client.py

@@ -0,0 +1,182 @@
+from typing import Any, Dict, List, Optional
+
+from app.config.settings import settings
+
+
+class ZillizClient:
+    """
+    Zilliz Cloud / Milvus vector store backend.
+
+    Uses pymilvus.MilvusClient. The collection uses quick setup with a string
+    primary key, a vector field, and dynamic metadata fields.
+    """
+
+    def __init__(
+        self,
+        uri: str = None,
+        token: str = None,
+        collection_name: str = None,
+        dimension: int = None,
+    ):
+        try:
+            from pymilvus import DataType, MilvusClient
+        except ImportError as exc:
+            raise RuntimeError("Zilliz backend requires pymilvus. Install requirements.txt again.") from exc
+
+        self.data_type = DataType
+        self.uri = uri or settings.ZILLIZ_URI
+        self.token = token or settings.ZILLIZ_TOKEN
+        self.collection_name = collection_name or settings.COLLECTION_NAME
+        self.dimension = dimension or settings.EMBEDDING_DIMENSION
+        self.vector_field = "vector"
+        self.text_field = "text"
+
+        if not self.uri or not self.token:
+            raise ValueError("ZILLIZ_URI and ZILLIZ_TOKEN are required when VECTOR_DB_BACKEND=zilliz.")
+
+        self.client = MilvusClient(uri=self.uri, token=self.token)
+
+        if not self.client.has_collection(self.collection_name):
+            self.client.create_collection(
+                collection_name=self.collection_name,
+                dimension=self.dimension,
+                primary_field_name="id",
+                id_type=self.data_type.VARCHAR,
+                vector_field_name=self.vector_field,
+                metric_type="COSINE",
+                auto_id=False,
+                max_length=512,
+            )
+        self._load_collection()
+
+    def add_documents(
+        self,
+        ids: List[str],
+        documents: List[str],
+        embeddings: List[List[float]],
+        metadatas: List[Dict[str, Any]],
+    ) -> None:
+        if not (len(ids) == len(documents) == len(embeddings) == len(metadatas)):
+            raise ValueError("ids, documents, embeddings, and metadatas must be same length")
+
+        rows = []
+        for doc_id, document, embedding, metadata in zip(ids, documents, embeddings, metadatas):
+            row = {
+                "id": doc_id,
+                self.vector_field: embedding,
+                self.text_field: document,
+            }
+            row.update(self._sanitize_metadata(metadata))
+            rows.append(row)
+
+        result = self.client.upsert(collection_name=self.collection_name, data=rows)
+        self._flush_collection()
+        self._load_collection()
+        return result
+
+    def similarity_search(
+        self,
+        query_embedding: List[float],
+        top_k: int = 5,
+        where: Optional[Dict[str, Any]] = None,
+    ) -> Dict[str, Any]:
+        filter_expression = self._where_to_filter(where)
+        search_kwargs = {
+            "collection_name": self.collection_name,
+            "data": [query_embedding],
+            "limit": top_k,
+            "output_fields": ["*"],
+        }
+
+        if filter_expression:
+            search_kwargs["filter"] = filter_expression
+
+        raw_results = self.client.search(**search_kwargs)
+
+        documents = []
+        metadatas = []
+        distances = []
+
+        for hit in raw_results[0] if raw_results else []:
+            entity = hit.get("entity", {})
+            documents.append(entity.get(self.text_field, ""))
+            metadatas.append(self._extract_metadata(entity))
+            distances.append(1.0 - float(hit.get("distance", 0.0)))
+
+        return {
+            "documents": [documents],
+            "metadatas": [metadatas],
+            "distances": [distances],
+        }
+
+    def delete_all(self) -> None:
+        if self.client.has_collection(self.collection_name):
+            self.client.drop_collection(self.collection_name)
+
+        self.client.create_collection(
+            collection_name=self.collection_name,
+            dimension=self.dimension,
+            primary_field_name="id",
+            id_type=self.data_type.VARCHAR,
+            vector_field_name=self.vector_field,
+            metric_type="COSINE",
+            auto_id=False,
+            max_length=512,
+        )
+
+    def get_collection_info(self) -> Dict[str, Any]:
+        count = 0
+        if self.client.has_collection(self.collection_name):
+            stats = self.client.get_collection_stats(self.collection_name)
+            count = int(stats.get("row_count", 0))
+
+        return {
+            "name": self.collection_name,
+            "count": count,
+            "metadata": {
+                "backend": "zilliz",
+                "dimension": self.dimension,
+                "metric": "COSINE",
+            },
+        }
+
+    def _sanitize_metadata(self, metadata: Dict[str, Any]) -> Dict[str, Any]:
+        sanitized = {}
+        for key, value in metadata.items():
+            if value is None:
+                continue
+            if isinstance(value, (str, int, float, bool)):
+                sanitized[key] = value
+            else:
+                sanitized[key] = str(value)
+        return sanitized
+
+    def _extract_metadata(self, entity: Dict[str, Any]) -> Dict[str, Any]:
+        ignored_fields = {"id", self.vector_field, self.text_field}
+        return {key: value for key, value in entity.items() if key not in ignored_fields}
+
+    def _where_to_filter(self, where: Optional[Dict[str, Any]]) -> str:
+        if not where:
+            return ""
+
+        filters = []
+        for key, value in where.items():
+            if isinstance(value, str):
+                escaped = value.replace("\\", "\\\\").replace('"', '\\"')
+                filters.append(f'{key} == "{escaped}"')
+            elif isinstance(value, bool):
+                filters.append(f"{key} == {str(value).lower()}")
+            elif isinstance(value, (int, float)):
+                filters.append(f"{key} == {value}")
+
+        return " and ".join(filters)
+
+    def _flush_collection(self) -> None:
+        flush = getattr(self.client, "flush", None)
+        if callable(flush):
+            flush(collection_name=self.collection_name)
+
+    def _load_collection(self) -> None:
+        load_collection = getattr(self.client, "load_collection", None)
+        if callable(load_collection):
+            load_collection(collection_name=self.collection_name)

requirements.txt

@@ -0,0 +1,11 @@
+PyPDF2
+python-docx
+markdown
+beautifulsoup4
+streamlit
+requests
+chromadb
+gdown
+pymilvus
+python-dotenv
+sentence-transformers

run.py

@@ -0,0 +1,15 @@
+# run_app.py
+import sys
+from pathlib import Path
+
+# Add project root to path
+project_root = Path(__file__).parent
+sys.path.insert(0, str(project_root))
+
+# Now run streamlit
+if __name__ == "__main__":
+    import streamlit.web.cli as stcli
+    import sys
+    
+    sys.argv = ["streamlit", "run", "app/ui/main.py"]
+    sys.exit(stcli.main())

scripts/check_vector_store.py

@@ -0,0 +1,88 @@
+import argparse
+import sys
+from pathlib import Path
+
+
+PROJECT_ROOT = Path(__file__).resolve().parents[1]
+if str(PROJECT_ROOT) not in sys.path:
+    sys.path.insert(0, str(PROJECT_ROOT))
+
+from app.config.settings import settings
+from app.ingestion.embedder import get_embedder
+from app.vectordb.factory import get_vector_store
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Check the configured vector store.")
+    parser.add_argument("--query", help="Optional test query to run against the vector store.")
+    parser.add_argument("--top-k", type=int, default=3, help="Number of search results for --query.")
+    parser.add_argument("--list-collections", action="store_true", help="List collections when using Zilliz/Milvus.")
+    parser.add_argument("--probe-insert", action="store_true", help="Insert one tiny probe row into the active vector store.")
+    args = parser.parse_args()
+
+    print(f"Vector backend: {settings.VECTOR_DB_BACKEND}")
+    print(f"Collection: {settings.COLLECTION_NAME}")
+    print(f"Embedding provider: {settings.EMBEDDING_PROVIDER}")
+    print(f"Embedding model: {settings.EMBEDDING_MODEL}")
+    print(f"Embedding dimension: {settings.EMBEDDING_DIMENSION}")
+
+    store = get_vector_store()
+
+    if args.list_collections and hasattr(store, "client"):
+        list_collections = getattr(store.client, "list_collections", None)
+        if callable(list_collections):
+            print(f"Available collections: {list_collections()}")
+
+    if args.probe_insert:
+        probe_id = "debug_probe_row"
+        probe_text = "debug probe document for vector store verification"
+        probe_embedding = [0.0] * settings.EMBEDDING_DIMENSION
+        probe_embedding[0] = 1.0
+        store.add_documents(
+            ids=[probe_id],
+            documents=[probe_text],
+            embeddings=[probe_embedding],
+            metadatas=[
+                {
+                    "filename": "debug_probe.txt",
+                    "source_type": "debug",
+                    "source_path": "debug_probe",
+                    "document_id": "debug_probe",
+                    "chunk_index": 0,
+                }
+            ],
+        )
+        print("Inserted probe row.")
+
+    info = store.get_collection_info()
+
+    print(f"Stored rows/chunks: {info['count']}")
+    print(f"Store metadata: {info['metadata']}")
+
+    if args.query:
+        embedder = get_embedder()
+        query_embedding = embedder.embed(args.query)
+        results = store.similarity_search(query_embedding, top_k=args.top_k)
+
+        documents = results.get("documents", [[]])[0]
+        metadatas = results.get("metadatas", [[]])[0]
+        distances = results.get("distances", [[]])[0]
+
+        print(f"\nSearch results for: {args.query}")
+        if not documents:
+            print("No matches returned.")
+            return
+
+        for index, (document, metadata, distance) in enumerate(zip(documents, metadatas, distances), start=1):
+            filename = metadata.get("filename", "unknown")
+            source_type = metadata.get("source_type", "unknown")
+            source_path = metadata.get("source_path", "")
+            preview = document.replace("\n", " ")[:160]
+            print(f"{index}. {filename} | {source_type} | distance={distance:.4f}")
+            if source_path:
+                print(f"   source_path: {source_path}")
+            print(f"   preview: {preview}")
+
+
+if __name__ == "__main__":
+    main()

test_embeder.py

@@ -0,0 +1,95 @@
+# from app.retrieval.search import get_searcher
+# from app.retrieval.aggregator import get_aggregator
+
+# searcher = get_searcher()
+
+# # Test different queries
+# queries = [
+#     "machine learning",
+#     "python programming",
+#     "deep learning neural networks",
+#     "variance analysis"
+# ]
+
+# for query in queries:
+#     print(f"\n{'='*60}")
+#     print(f"Query: '{query}'")
+#     print('='*60)
+    
+#     results = searcher.search(query, top_k=3)
+    
+#     if results:
+#         for result in results:
+#             print(f"\nRank {result['rank']} - Similarity: {result['similarity']:.3f}")
+#             print(f"  File: {result['metadata']['filename']}")
+#             print(f"  Text: {result['text'][:100]}...")
+#     else:
+#         print("No results found")
+
+# print("\n" + "="*60)
+# print("AGGREGATED VIEW")
+# print("="*60)
+
+# results = searcher.search("machine learning", top_k=10)
+# aggregator = get_aggregator()
+# aggregated = aggregator.aggregate_by_document(results)
+
+# for doc in aggregated:
+#     print(f"\n📄 {doc['filename']}")
+#     print(f"   Relevance: {doc['relevance_score']:.3f}")
+#     print(f"   Matches: {doc['num_matching_chunks']}")
+
+import shutil
+from pathlib import Path
+from app.config.settings import settings
+from app.ingestion.ingest import get_pipeline
+
+# 1. Delete the entire ChromaDB directory
+# force_reset_v2.py
+import shutil
+from pathlib import Path
+from app.config.settings import settings
+
+# 1. Delete the entire ChromaDB directory
+chroma_dir = Path(settings.CHROMA_PERSIST_DIR)
+if chroma_dir.exists():
+    print(f"🗑️  Deleting old ChromaDB at {chroma_dir}")
+    shutil.rmtree(chroma_dir)
+    print("✓ Deleted")
+
+# Wait for imports to use new code
+print("\n📊 Importing fresh modules...")
+from app.ingestion.ingest import get_pipeline
+
+# 2. Create fresh pipeline
+print("Creating fresh ChromaDB with cosine similarity...")
+pipeline = get_pipeline()
+status = pipeline.get_status()
+print(f"✓ Collection: {status['collection_name']}")
+print(f"✓ Chunks: {status['total_chunks']}")
+print(f"✓ Metadata: {status['metadata']}")
+
+# 3. Ingest documents
+print("\n📂 Ingesting documents...")
+results = pipeline.ingest_directory("data/raw/")
+
+print(f"\n✅ Done! Ingested {len(results)} files:")
+for filename, count in results.items():
+    print(f"  • {filename}: {count} chunks")
+
+final_status = pipeline.get_status()
+print(f"\n📊 Final count: {final_status['total_chunks']} chunks")
+
+# 4. Test search immediately
+print("\n🔍 Testing search...")
+from app.retrieval.search import get_searcher
+
+searcher = get_searcher()
+results = searcher.search("machine learning", top_k=3)
+
+if results:
+    print(f"✓ Search works! Found {len(results)} results")
+    for r in results:
+        print(f"  - {r['metadata']['filename']}: similarity {r['similarity']:.3f}")
+else:
+    print("✗ No results found")

uv.lock

@@ -0,0 +1,3 @@
+version = 1
+revision = 3
+requires-python = ">=3.12"