# 🕷️ Code Crawler - Complete Architecture Walkthrough

## Table of Contents
1. [Project Overview](#project-overview)
2. [System Architecture](#system-architecture)
3. [Data Flow Pipeline](#data-flow-pipeline)
4. [RAG Implementation](#rag-implementation)
5. [AST Analysis & Graph Creation](#ast-analysis--graph-creation)
6. [Code Chunking Strategy](#code-chunking-strategy)
7. [Retrieval System](#retrieval-system)
8. [Agentic Workflow](#agentic-workflow)
9. [Frontend & API](#frontend--api)
10. [Component Deep Dives](#component-deep-dives)

---

## Project Overview

**Code Crawler** is an AI-powered codebase assistant that combines multiple advanced techniques:

- **RAG (Retrieval-Augmented Generation)**: Vector-based semantic search over code
- **AST Analysis**: Abstract Syntax Tree parsing for understanding code structure
- **Graph RAG**: Knowledge graph enhancement for relationship-aware retrieval
- **Agentic Workflows**: Multi-step reasoning with tool use (LangGraph)
- **Multi-LLM Support**: Gemini, Groq (Llama 3.3)

### Key Features
| Feature | Description |
|---------|-------------|
| 💬 Chat Mode | Natural language Q&A about codebase |
| 🔍 Search Mode | Regex pattern search across files |
| 🔧 Refactor Mode | AI-assisted code refactoring |
| ✨ Generate Mode | Spec generation (PO-friendly, Dev Specs, User Stories) |

---

## System Architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                            CODE CRAWLER SYSTEM                               │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  ┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐        │
│  │   DATA INGEST   │────▶│   PROCESSING    │────▶│    STORAGE      │        │
│  │                 │     │                 │     │                 │        │
│  │ • ZIP Files     │     │ • AST Parsing   │     │ • Vector DB     │        │
│  │ • GitHub URLs   │     │ • Chunking      │     │   (Chroma/FAISS)│        │
│  │ • Local Dirs    │     │ • Embeddings    │     │ • AST Graph     │        │
│  │ • Web Docs      │     │ • Graph Build   │     │   (GraphML)     │        │
│  └─────────────────┘     └─────────────────┘     └────────┬────────┘        │
│                                                           │                  │
│                                                           ▼                  │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                        RETRIEVAL LAYER                               │    │
│  │  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐ │    │
│  │  │   Vector    │  │    LLM      │  │   Graph     │  │  Reranker   │ │    │
│  │  │  Retriever  │──│  Retriever  │──│  Enhanced   │──│  (Cross-    │ │    │
│  │  │             │  │             │  │  Retriever  │  │   Encoder)  │ │    │
│  │  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘ │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                    │                                         │
│                                    ▼                                         │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                         CHAT ENGINE                                  │    │
│  │                                                                      │    │
│  │   ┌──────────────────┐        ┌──────────────────────────┐          │    │
│  │   │   Linear RAG     │   OR   │   Agentic Workflow       │          │    │
│  │   │   (Simple)       │        │   (LangGraph)            │          │    │
│  │   │                  │        │                          │          │    │
│  │   │  Query → Retrieve│        │  Agent → Tool → Agent    │          │    │
│  │   │      → Answer    │        │       ↓                  │          │    │
│  │   │                  │        │  search_codebase         │          │    │
│  │   │                  │        │  read_file               │          │    │
│  │   │                  │        │  list_files              │          │    │
│  │   │                  │        │  find_callers            │          │    │
│  │   └──────────────────┘        └──────────────────────────┘          │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
│                                    │                                         │
│                                    ▼                                         │
│  ┌─────────────────────────────────────────────────────────────────────┐    │
│  │                       FRONTEND LAYER                                 │    │
│  │                                                                      │    │
│  │   Streamlit App          FastAPI (REST)         Next.js (React)     │    │
│  │   ├── app.py             ├── /api/index         ├── /chat           │    │
│  │   └── Code_Studio.py     ├── /api/chat          ├── /generate       │    │
│  │                          └── /api/health        └── /search         │    │
│  └─────────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────────┘
```

---

## Data Flow Pipeline

### 1. Ingestion Flow

```
User Input (ZIP/GitHub/Local)
         │
         ▼
┌─────────────────────────────────────────┐
│      UniversalIngestor                  │
│      (universal_ingestor.py)            │
│                                         │
│  ┌─────────────┐  ┌─────────────────┐   │
│  │ _detect_    │  │ Handler Classes │   │
│  │  handler()  │──▶│                 │   │
│  └─────────────┘  │ • ZIPFileManager│   │
│                   │ • GitHubRepoMgr │   │
│                   │ • LocalDirMgr   │   │
│                   │ • WebDocManager │   │
│                   └─────────────────┘   │
└────────────────────┬────────────────────┘
                     │
                     ▼
            List[Document] + local_path
```

**Example: GitHub Repository Processing**

```python
# 1. User provides: "https://github.com/owner/repo"

# 2. UniversalIngestor detects GitHub URL
ingestor = UniversalIngestor(source)
# delegate = GitHubRepoManager

# 3. Download (clone or ZIP fallback)
ingestor.download()
# Clones to: /tmp/code_chatbot/owner_repo/

# 4. Walk files
for content, metadata in ingestor.walk():
    # content = "def hello(): ..."
    # metadata = {"file_path": "/tmp/.../main.py", "source": "main.py"}
```

### 2. Indexing Flow

```
Documents
    │
    ▼
┌─────────────────────────────────────────────────────────────────┐
│                        Indexer                                   │
│                       (indexer.py)                               │
│                                                                  │
│  ┌─────────────────┐   ┌─────────────────┐   ┌───────────────┐  │
│  │ StructuralChunker│──▶│ Embedding Model │──▶│  Vector Store │  │
│  │                  │   │ (Gemini/HF)     │   │ (Chroma/FAISS)│  │
│  └─────────────────┘   └─────────────────┘   └───────────────┘  │
│                                                                  │
│  Additionally:                                                   │
│  ┌─────────────────┐   ┌─────────────────┐                      │
│  │ ASTGraphBuilder │──▶│  GraphML File   │                      │
│  └─────────────────┘   └─────────────────┘                      │
└─────────────────────────────────────────────────────────────────┘
```

---

## RAG Implementation

The RAG system in this project is implemented in `code_chatbot/rag.py` with these key components:

### ChatEngine Class

```python
class ChatEngine:
    def __init__(self, retriever, model_name, provider, ...):
        # 1. Base retriever (from vector store)
        self.base_retriever = retriever

        # 2. Enhanced retriever with reranking
        self.vector_retriever = build_enhanced_retriever(
            base_retriever=retriever,
            use_multi_query=use_multi_query,
            use_reranking=True  # Uses Cross-Encoder
        )

        # 3. LLM Retriever (file-aware)
        self.llm_retriever = LLMRetriever(llm, repo_files)

        # 4. Ensemble Retriever (combines both)
        self.retriever = EnsembleRetriever(
            retrievers=[self.vector_retriever, self.llm_retriever],
            weights=[0.6, 0.4]  # 60% vector, 40% LLM
        )
```

### RAG Flow Example

```
User Query: "How does the authentication work?"
                    │
                    ▼
┌─────────────────────────────────────────────────────────────┐
│ 1. RETRIEVAL                                                │
│    ┌──────────────────┐      ┌──────────────────┐           │
│    │ Vector Retriever │      │ LLM Retriever    │           │
│    │                  │      │                  │           │
│    │ Semantic search  │      │ LLM picks files  │           │
│    │ in Chroma DB     │      │ from structure   │           │
│    └────────┬─────────┘      └────────┬─────────┘           │
│             │                         │                      │
│             └────────────┬────────────┘                      │
│                          ▼                                   │
│              ┌─────────────────────┐                         │
│              │ EnsembleRetriever   │                         │
│              │ (60% + 40% weighted)│                         │
│              └─────────┬───────────┘                         │
│                        │                                     │
│                        ▼                                     │
│              ┌─────────────────────┐                         │
│              │ Reranker            │                         │
│              │ (Cross-Encoder)     │                         │
│              │ ms-marco-MiniLM     │                         │
│              └─────────┬───────────┘                         │
│                        │                                     │
│                        ▼                                     │
│              Top 5 Most Relevant Docs                        │
└─────────────────────────────────────────────────────────────┘
                    │
                    ▼
┌─────────────────────────────────────────────────────────────┐
│ 2. GENERATION                                               │
│                                                             │
│    System Prompt + Context + History + Question             │
│                        │                                    │
│                        ▼                                    │
│              ┌─────────────────────┐                        │
│              │ LLM (Gemini/Groq)   │                        │
│              └─────────┬───────────┘                        │
│                        │                                    │
│                        ▼                                    │
│                   Answer + Sources                          │
└─────────────────────────────────────────────────────────────┘
```

---

## AST Analysis & Graph Creation

The AST analysis is implemented in `code_chatbot/ast_analysis.py` using **tree-sitter** for multi-language parsing.

### How AST Parsing Works

```python
# Example: Parsing a Python file

# Source code:
"""
from typing import List

class UserService:
    def __init__(self, db):
        self.db = db

    def get_user(self, user_id: int) -> User:
        return self.db.find(user_id)

    def create_user(self, name: str) -> User:
        user = User(name=name)
        self.db.save(user)
        return user
"""

# tree-sitter parses this into an AST:
"""
module
├── import_from_statement
│   ├── module: "typing"
│   └── names: ["List"]
├── class_definition
│   ├── name: "UserService"
│   └── block
│       ├── function_definition (name: "__init__")
│       ├── function_definition (name: "get_user")
│       │   └── call (function: "self.db.find")
│       └── function_definition (name: "create_user")
│           ├── call (function: "User")
│           └── call (function: "self.db.save")
"""
```

### EnhancedCodeAnalyzer

```python
class EnhancedCodeAnalyzer:
    """Builds a knowledge graph from code"""

    def __init__(self):
        self.graph = nx.DiGraph()  # NetworkX directed graph
        self.functions = {}         # node_id -> FunctionInfo
        self.classes = {}           # node_id -> ClassInfo
        self.imports = {}           # file_path -> [ImportInfo]
        self.definitions = {}       # name -> [node_ids]
```

### Graph Structure Example

```
┌─────────────────────────────────────────────────────────────────┐
│                    AST KNOWLEDGE GRAPH                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Nodes:                                                         │
│  ┌──────────────────┐                                          │
│  │ Type: "file"     │                                          │
│  │ Name: "api.py"   │                                          │
│  └────────┬─────────┘                                          │
│           │ defines                                             │
│           ▼                                                     │
│  ┌──────────────────┐         ┌──────────────────┐             │
│  │ Type: "class"    │         │ Type: "function" │             │
│  │ Name: "UserAPI"  │         │ Name: "main"     │             │
│  └────────┬─────────┘         └──────────────────┘             │
│           │ has_method                                          │
│           ▼                                                     │
│  ┌──────────────────┐                                          │
│  │ Type: "method"   │───calls───▶ UserService.get_user         │
│  │ Name: "get"      │                                          │
│  └──────────────────┘                                          │
│                                                                 │
│  Edges:                                                         │
│  • defines: file -> class/function                              │
│  • has_method: class -> method                                  │
│  • calls: function -> function                                  │
│  • imports: file -> module                                      │
│  • inherits_from: class -> class                                │
└─────────────────────────────────────────────────────────────────┘
```

### Call Graph Resolution

```python
def resolve_call_graph(self):
    """
    After parsing all files, resolve function calls to their definitions.

    Example:
    - File A has: service.get_user(id)
    - File B has: def get_user(self, id): ...

    Resolution:
    - Finds that "get_user" is defined in File B
    - Creates edge: A::caller_func --calls--> B::UserService.get_user
    """
    for caller_id, callee_name, line in self.unresolved_calls:
        # Try direct match
        if callee_name in self.definitions:
            for target_id in self.definitions[callee_name]:
                self.graph.add_edge(caller_id, target_id, relation="calls")
```

---

## Code Chunking Strategy

The chunking system in `code_chatbot/chunker.py` uses **structural chunking** based on AST boundaries.

### Chunking Philosophy

```
Traditional Text Chunking:
┌─────────────────────────────────────────┐
│ def process_data():        │ CHUNK 1    │
│     data = load()          │            │
│     # Some processing      │            │
│ ───────────────────────────┼────────────│
│     result = transform()   │ CHUNK 2    │  ← Breaks mid-function!
│     return result          │            │
└─────────────────────────────────────────┘

Structural Chunking (This Project):
┌─────────────────────────────────────────┐
│ def process_data():        │            │
│     data = load()          │ CHUNK 1    │  ← Complete function
│     result = transform()   │            │
│     return result          │            │
├─────────────────────────────────────────┤
│ def another_function():    │            │
│     ...                    │ CHUNK 2    │  ← Complete function
└─────────────────────────────────────────┘
```

### StructuralChunker Implementation

```python
class StructuralChunker:
    """Uses tree-sitter to chunk code at semantic boundaries"""

    def __init__(self, max_tokens: int = 800):
        self.max_tokens = max_tokens
        self._init_parsers()  # Python, JS, TS parsers

    def _chunk_node(self, node, file_content, file_metadata):
        """
        Recursive chunking algorithm:

        1. If node fits in max_tokens → return as single chunk
        2. If node is too large → recurse into children
        3. Merge neighboring small chunks
        """
        chunk = FileChunk(file_content, file_metadata,
                         node.start_byte, node.end_byte)

        # Fits? Return it
        if chunk.num_tokens <= self.max_tokens:
            return [chunk]

        # Too large? Recurse
        child_chunks = []
        for child in node.children:
            child_chunks.extend(self._chunk_node(child, ...))

        # Merge small neighbors
        return self._merge_small_chunks(child_chunks)
```

### Chunk Metadata (Rich Context)

Each chunk carries rich metadata:

```python
@dataclass
class FileChunk:
    file_content: str
    file_metadata: Dict
    start_byte: int
    end_byte: int

    # Enhanced metadata
    symbols_defined: List[str]    # ["UserService", "UserService.get_user"]
    imports_used: List[str]       # ["from typing import List"]
    complexity_score: int         # Cyclomatic complexity
    parent_context: str           # "UserService" (parent class)
```

This metadata is stored in the vector DB and used for filtering/ranking.

---

## Retrieval System

### Multi-Stage Retrieval Pipeline

```
Query: "How does user authentication work?"
                    │
                    ▼
┌───────────────────────────────────────────────────────────────┐
│  STAGE 1: Initial Retrieval (k=10)                            │
│                                                               │
│  ┌─────────────────────────────────────────────────────────┐  │
│  │               Vector Store (Chroma)                      │  │
│  │                                                          │  │
│  │  Query Embedding ──similarity──▶ Document Embeddings     │  │
│  │                                                          │  │
│  │  Returns: 10 candidate documents                         │  │
│  └─────────────────────────────────────────────────────────┘  │
└───────────────────────────────────────────────────────────────┘
                    │
                    ▼
┌───────────────────────────────────────────────────────────────┐
│  STAGE 2: LLM-Based File Selection                            │
│                                                               │
│  ┌─────────────────────────────────────────────────────────┐  │
│  │              LLMRetriever                                │  │
│  │                                                          │  │
│  │  File Tree:                                              │  │
│  │  ├── src/                                                │  │
│  │  │   ├── auth/                                           │  │
│  │  │   │   ├── login.py      ◄── LLM selects this         │  │
│  │  │   │   └── middleware.py ◄── And this                 │  │
│  │  │   └── api/                                            │  │
│  │  └── tests/                                              │  │
│  │                                                          │  │
│  │  LLM Prompt: "Select top 5 relevant files for: ..."      │  │
│  └─────────────────────────────────────────────────────────┘  │
└───────────────────────────────────────────────────────────────┘
                    │
                    ▼
┌───────────────────────────────────────────────────────────────┐
│  STAGE 3: Ensemble Combination                                │
│                                                               │
│  Vector Results (weight: 0.6) + LLM Results (weight: 0.4)     │
│                                                               │
│  Combined: 12-15 unique documents                             │
└───────────────────────────────────────────────────────────────┘
                    │
                    ▼
┌───────────────────────────────────────────────────────────────┐
│  STAGE 4: Graph Enhancement                                   │
│                                                               │
│  For each retrieved document:                                 │
│  1. Find its node in AST graph                                │
│  2. Get neighboring nodes (related files)                     │
│  3. Add related files to context                              │
│                                                               │
│  Example: login.py found → adds auth_utils.py (imports it)    │
└───────────────────────────────────────────────────────────────┘
                    │
                    ▼
┌───────────────────────────────────────────────────────────────┐
│  STAGE 5: Reranking                                           │
│                                                               │
│  ┌─────────────────────────────────────────────────────────┐  │
│  │              Cross-Encoder Reranker                      │  │
│  │              (ms-marco-MiniLM-L-6-v2)                     │  │
│  │                                                          │  │
│  │  For each (query, document) pair:                        │  │
│  │  score = cross_encoder.predict([query, doc.content])     │  │
│  │                                                          │  │
│  │  Sort by score, return top 5                             │  │
│  └─────────────────────────────────────────────────────────┘  │
└───────────────────────────────────────────────────────────────┘
                    │
                    ▼
            Final: Top 5 Documents
```

### Reranker (Cross-Encoder)

```python
class Reranker:
    """
    Uses a Cross-Encoder for precise relevance scoring.

    Unlike bi-encoders (used for initial retrieval), cross-encoders
    process query AND document together, giving more accurate scores.
    """

    def __init__(self, model_name="cross-encoder/ms-marco-MiniLM-L-6-v2"):
        self.model = CrossEncoder(model_name)

    def rerank(self, query: str, documents: List[Document], top_k=5):
        # Score each document against the query
        pairs = [[query, doc.page_content] for doc in documents]
        scores = self.model.predict(pairs)

        # Sort by score
        scored = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)
        return [doc for doc, _ in scored[:top_k]]
```

---

## Agentic Workflow

The agentic workflow uses **LangGraph** to enable multi-step reasoning with tool use.

### Agent Graph Structure

```
┌─────────────────────────────────────────────────────────────────┐
│                    LANGGRAPH AGENT                              │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│                    ┌─────────────┐                              │
│         ┌─────────│   START     │─────────┐                     │
│         │         └─────────────┘         │                     │
│         ▼                                 │                     │
│  ┌─────────────────────────────────────┐  │                     │
│  │           AGENT NODE                │  │                     │
│  │                                     │  │                     │
│  │  1. Process messages                │  │                     │
│  │  2. Call LLM with tools bound       │  │                     │
│  │  3. LLM decides:                    │  │                     │
│  │     - Call a tool? → go to TOOLS    │  │                     │
│  │     - Final answer? → go to END     │  │                     │
│  └──────────────┬──────────────────────┘  │                     │
│                 │                         │                     │
│       has_tool_call?                      │                     │
│         │     │                           │                     │
│    Yes  │     │  No                       │                     │
│         │     │                           │                     │
│         ▼     └──────────────────────────▶┤                     │
│  ┌─────────────────────────────────────┐  │                     │
│  │           TOOLS NODE                │  │                     │
│  │                                     │  │                     │
│  │  Execute tool calls:                │  │                     │
│  │  • search_codebase(query)           │  │                     │
│  │  • read_file(path)                  │  │                     │
│  │  • list_files(dir)                  │  │                     │
│  │  • find_callers(func)               │  │                     │
│  │  • find_callees(func)               │  │                     │
│  │  • find_call_chain(a, b)            │  │                     │
│  │                                     │  │                     │
│  │  Add tool results to messages       │  │                     │
│  └──────────────┬──────────────────────┘  │                     │
│                 │                         │                     │
│                 └─────────────────────────┘                     │
│                                                                 │
│                         ▼                                       │
│                  ┌─────────────┐                                │
│                  │     END     │                                │
│                  └─────────────┘                                │
└─────────────────────────────────────────────────────────────────┘
```

### Available Tools

```python
# 1. search_codebase - Semantic search in vector store
@tool("search_codebase")
def search_codebase(query: str):
    """Search the codebase for relevant code snippets."""
    docs = retriever.invoke(query)
    return format_results(docs[:5])

# 2. read_file - Read complete file content
@tool("read_file")
def read_file(file_path: str):
    """Read the content of a specific file."""
    with open(full_path, "r") as f:
        return f.read()

# 3. list_files - Directory listing
@tool("list_files")
def list_files(path: str = "."):
    """List files in a directory."""
    return "\n".join(os.listdir(target_path))

# 4. find_callers - Call graph: who calls this?
@tool("find_callers")
def find_callers(function_name: str):
    """Find all functions that call this function."""
    return analyzer.get_callers(function_name)

# 5. find_callees - Call graph: what does this call?
@tool("find_callees")
def find_callees(function_name: str):
    """Find all functions called by this function."""
    return analyzer.get_callees(function_name)

# 6. find_call_chain - Call graph: trace execution path
@tool("find_call_chain")
def find_call_chain(start: str, end: str):
    """Find the call path from one function to another."""
    return analyzer.get_call_chain(start, end)
```

### Agent Execution Example

```
User: "How does the login function handle invalid passwords?"

Agent Thought: I need to find the login function first.
Agent Action: search_codebase("login function authentication")
Tool Result: [login.py snippet showing authenticate() function]

Agent Thought: I found authenticate(). Let me see the full implementation.
Agent Action: read_file("src/auth/login.py")
Tool Result: [Full file content]

Agent Thought: I see it calls verify_password(). Let me check that.
Agent Action: find_callees("authenticate")
Tool Result: verify_password, get_user, create_session

Agent Action: search_codebase("verify_password invalid password")
Tool Result: [password_utils.py with error handling]

Agent Final Answer: The login function handles invalid passwords by...
```

---

## Frontend & API

### Streamlit App Structure

```
app.py (Main Entry)
    │
    ├── Ingestion Screen
    │   ├── Source Type Selection (ZIP/GitHub/Web)
    │   ├── File Upload / URL Input
    │   └── "Process & Index" Button
    │
    └── Redirects to → pages/1_⚡_Code_Studio.py

Code_Studio.py
    │
    ├── Left Panel (Tabs)
    │   ├── 📁 Explorer - File tree navigation
    │   ├── 🔍 Search - Regex pattern search
    │   ├── 💬 Chat - RAG conversation
    │   └── ✨ Generate - Spec generation
    │
    └── Right Panel
        └── Code Viewer - Syntax highlighted file view
```

### FastAPI REST API

```
/api
  ├── /health     GET   - Health check
  │
  ├── /index      POST  - Index a codebase
  │   Body: {
  │     source: "https://github.com/...",
  │     provider: "gemini",
  │     use_agent: true
  │   }
  │
  └── /chat       POST  - Ask questions
      Body: {
        question: "How does auth work?",
        provider: "gemini",
        use_agent: true
      }
      Response: {
        answer: "...",
        sources: [...],
        mode: "agent",
        processing_time: 2.5
      }
```

---

## Component Deep Dives

### Merkle Tree (Incremental Indexing)

```python
class MerkleTree:
    """
    Enables incremental indexing by detecting file changes.

    How it works:
    1. Build a hash tree mirroring directory structure
    2. Each file node has SHA-256 hash of content
    3. Each directory node has hash of children hashes
    4. Compare old vs new tree to find changes
    """

    def compare_trees(self, old, new) -> ChangeSet:
        # Returns: added, modified, deleted, unchanged files
```

**Example:**

```
First Index:
  project/
  ├── main.py    (hash: abc123)
  └── utils.py   (hash: def456)

  Root hash: sha256(abc123 + def456) = xyz789

Second Index (utils.py changed):
  project/
  ├── main.py    (hash: abc123)  ← unchanged
  └── utils.py   (hash: ghi012)  ← NEW HASH!

  Root hash changed! → Only re-index utils.py
```

### Path Obfuscation (Privacy)

```python
class PathObfuscator:
    """
    Obfuscates file paths for sensitive codebases.

    Original: /home/user/secret-project/src/auth/login.py
    Obfuscated: /f8a3b2c1/d4e5f6a7/89012345.py

    Mapping stored securely, reversible only with key.
    """
```

### Rate Limiter (API Management)

```python
class AdaptiveRateLimiter:
    """
    Handles rate limits for free-tier APIs.

    Gemini Free Tier: 15 RPM, 32K TPM, 1500 RPD

    Strategies:
    1. Track usage in rolling window
    2. Adaptive delay based on remaining quota
    3. Exponential backoff on 429 errors
    4. Model fallback chain (flash → pro → legacy)
    """
```

---

## Configuration System

```python
@dataclass
class RAGConfig:
    """Central configuration for entire pipeline"""

    # Chunking
    chunking: ChunkingConfig
        max_chunk_tokens: int = 800
        min_chunk_tokens: int = 100
        preserve_imports: bool = True
        calculate_complexity: bool = True

    # Privacy
    privacy: PrivacyConfig
        enable_path_obfuscation: bool = False

    # Indexing
    indexing: IndexingConfig
        enable_incremental_indexing: bool = True
        batch_size: int = 100
        ignore_patterns: List[str] = [...]

    # Retrieval
    retrieval: RetrievalConfig
        enable_reranking: bool = True
        retrieval_k: int = 10
        rerank_top_k: int = 5
        similarity_threshold: float = 0.5
```

---

## File Dependency Map

```
app.py
├── code_chatbot/universal_ingestor.py
├── code_chatbot/indexer.py
│   ├── code_chatbot/chunker.py (StructuralChunker)
│   ├── code_chatbot/merkle_tree.py (MerkleTree)
│   ├── code_chatbot/config.py (RAGConfig)
│   └── code_chatbot/db_connection.py (Chroma client)
├── code_chatbot/rag.py (ChatEngine)
│   ├── code_chatbot/retriever_wrapper.py
│   │   └── code_chatbot/reranker.py (Reranker)
│   ├── code_chatbot/llm_retriever.py (LLMRetriever)
│   ├── code_chatbot/agent_workflow.py
│   │   └── code_chatbot/tools.py
│   └── code_chatbot/prompts.py
├── code_chatbot/ast_analysis.py (EnhancedCodeAnalyzer)
└── code_chatbot/graph_rag.py (GraphEnhancedRetriever)

pages/1_⚡_Code_Studio.py
├── components/file_explorer.py
├── components/code_viewer.py
├── components/panels.py
└── components/style.py

api/main.py
├── api/routes/chat.py
├── api/routes/index.py
├── api/routes/health.py
├── api/schemas.py
└── api/state.py
```

---

## Summary

This project implements a sophisticated code understanding system with:

1. **Multi-Source Ingestion**: ZIP, GitHub, Local, Web
2. **Structural Chunking**: AST-aware code splitting
3. **Hybrid Retrieval**: Vector + LLM + Graph-enhanced
4. **Cross-Encoder Reranking**: Precision at the top
5. **Agentic Workflow**: Multi-step reasoning with tools
6. **Call Graph Analysis**: Function relationship tracking
7. **Incremental Indexing**: Merkle tree change detection
8. **Multi-LLM Support**: Gemini, Groq with fallbacks

The architecture is designed for scalability, accuracy, and developer experience.