docs/architecture.md

# Capstone RAG — Architecture

## System overview

A modular RAG explorer built on **Gradio + HuggingFace Spaces**. The index artifacts (FAISS, BM25, chunks) are precomputed offline and stored in an HF dataset repo. At runtime the app downloads them on demand, runs retrieval + generation, and evaluates the result against RAGBench gold labels.

```mermaid
graph TD
    RB[RAGBench Dataset<br/>rungalileo/ragbench]
    KB[KB Builder<br/>Colab Notebook]
    HF[(HF Dataset Repo<br/>Abhiram83/rag-capstone-indices)]

    RB -->|load corpus + questions| KB
    KB -->|push FAISS · BM25 · chunks<br/>+ index_config.json| HF

    subgraph APP [Gradio Space]
        QA["🔍 Single-question tab<br/>───────────────────<br/>pick config · type question<br/>see answer · retrieved chunks<br/>gold scores side-by-side"]

        BM["⚡ Benchmark tab<br/>───────────────────<br/>pick config · choose N<br/>run pipeline on N random Qs<br/>summary + per-question table<br/>RMSE / AUROC vs gold"]

        RP["📋 Reports tab<br/>───────────────────<br/>list saved runs<br/>load full detail · delete"]

        CP["⚖️ Compare tab<br/>───────────────────<br/>multi-select reports<br/>grouped bar chart<br/>scores table · config diff"]
    end

    HF -->|download indices on first use<br/>load index_config.json at startup| APP

    BM -->|Save as Report<br/>report JSON + index.json| HF
    HF -->|auto-load on startup · Refresh| RP
    HF -->|auto-load on startup · Refresh| CP
```

![System overview](images/system_overview.png)

---

## Pipeline flow

### Single-question path

```mermaid
sequenceDiagram
    participant U as User
    participant P as RagPipeline
    participant QT as QueryTransform
    participant R as HybridRetriever
    participant RR as Reranker
    participant LLM as Groq LLM
    participant E as Evaluator

    U->>P: question
    P->>QT: transform(question, strategy)
    QT-->>P: query / variants / hypothetical
    P->>R: retrieve(query, strategy)
    R-->>P: candidate chunks [(idx, score)]
    P->>RR: rerank(query, candidates, top_n)
    RR-->>P: reranked [(idx, cross_encoder_score)]
    P->>P: sentence_window_expand(reranked)
    P->>LLM: generate(context, question)
    LLM-->>P: answer
    P-->>U: {answer, context, relevance, trace}
    U->>E: score(result, eval_method)
    note over E: NLI path — local DeBERTa NLI model<br/>adherence · utilization · ROUGE-L completeness
    E-->>U: {relevance, adherence, utilization, completeness}<br/>displayed alongside gold scores
```

![Single-question flow](images/single_question_flow.png)

### Benchmark path

```mermaid
sequenceDiagram
    participant U as User
    participant B as run_benchmark
    participant P as RagPipeline
    participant NLI as NLI Scorer<br/>(local DeBERTa)
    participant J as LLM Judge<br/>(Groq / OpenAI)
    participant C as DiskCache

    U->>B: run(config, n_samples, eval_method)
    B->>B: sample(dataset, n, random_seed)
    loop each question
        B->>C: lookup(config_hash, row_idx)
        alt cache hit
            C-->>B: cached scores
        else
            B->>P: answer(question, trace=use_judge)
            P-->>B: {answer, context, relevance}
            alt eval_method = local_nli
                B->>NLI: adherence(context, answer)
                B->>NLI: utilization(context, answer)
                B->>NLI: completeness(answer, gold_response) via ROUGE-L
                NLI-->>B: {adherence, utilization, completeness}
            else eval_method = groq / openai
                B->>J: score(question, docs, answer)<br/>JSON mode + Pydantic validation
                J-->>B: {adherence, relevance, utilization, completeness}
            end
            B->>C: store(config_hash, row_idx, scores + gold_labels)
        end
    end
    B->>B: compute summary means + RMSE/AUROC vs gold
    B-->>U: summary_df, per_q_df, stats_df, bar_chart
```

![Benchmark flow](images/benchmark_flow.png)

---

## Retrieval

### Strategies

| Strategy | What happens |
|---|---|
| **Dense only** | Encode query with SentenceTransformer → FAISS inner-product search → top `dense_k` |
| **BM25 only** | Tokenise query (lowercase split) → BM25Okapi scores → top `bm25_k` |
| **Hybrid (RRF)** | Run both, fuse with Reciprocal Rank Fusion → top `fusion_top_k` |

### Reciprocal Rank Fusion

Each hit from each ranked list gets score `1 / (k + rank)` where `k = 60` (reduces sensitivity to rank-1 outliers). Scores are summed across lists; result is re-sorted descending.

```
rrf_score(doc) = Σ  1 / (60 + rank_in_list_i)
```

### Query rewrite strategies

| Strategy | LLM call | What changes |
|---|---|---|
| None | — | Raw question sent to retriever |
| Rewrite | 1× Groq | Single improved query replaces original |
| Multi-query | 1× Groq | `n` variants generated; each retrieves independently; RRF fuses all result lists |
| HyDE | 1× Groq | Hypothetical passage generated; its **embedding** used for dense search |
| HyDE + BM25 | 1× Groq | Same as HyDE for dense; original question used for BM25; RRF fuses both |

### Reranker

A **cross-encoder** (`cross-encoder/ms-marco-MiniLM-L-6-v2`) re-scores each `(query, chunk)` pair directly — no embedding compression. Takes the top `top_n` from the reranked list. When the reranker is off, the top `top_n` from retrieval are used directly.

After reranking, **sentence-window expansion** widens each selected chunk by ±1 sentence from its source document to improve context coverage without losing precision.

---

## Evaluation

Two paths are available, selected at benchmark time.

### Path 1 — Local NLI (TRACe-proxy)

Uses a local cross-encoder (`cross-encoder/nli-deberta-v3-base`) with no API calls.

```mermaid
graph LR
    A[context + answer] --> B[NLI cross-encoder]
    B --> C{label logits}
    C -->|softmax| D[P entailment / P contradiction / P neutral]
    D --> E[adherence]
    D --> F[utilization]
    G[answer + gold_response] --> H[ROUGE-L] --> I[completeness]
    J[reranker scores] --> K[sigmoid mean] --> L[relevance]
```

![Local NLI path](images/local_nli_path.png)

#### Relevance
Mean sigmoid of the cross-encoder scores from the **reranker** (not the NLI model):

```
relevance = mean( sigmoid(score_i) )   for each reranked chunk i
sigmoid(x) = 1 / (1 + e^−x)
```

#### Adherence
Single NLI call treating the **full context** as premise and the **generated answer** as hypothesis. Returns `P(entailment)`:

```
adherence = P( context ⊢ answer )
```

#### Utilization
Per-sentence NLI: for each sentence in the context, ask whether the answer entails it. Fraction above threshold 0.5:

```
utilization = |{s ∈ context_sentences : P(answer ⊢ s) > 0.5}| / |context_sentences|
```

#### Completeness
ROUGE-L F1 between the generated answer and the gold reference response from RAGBench:

```
completeness = ROUGE-L F1(answer, gold_response)
```

Only available when a gold reference exists; `nan` otherwise.

---

### Path 2 — LLM-as-Judge (Groq / OpenAI)

One structured LLM call per question. The judge receives sentence-keyed documents and the sentence-keyed answer, and returns a single JSON object. **`response_format={"type": "json_object"}`** is passed to the API so the model is token-level constrained to valid JSON. The response is validated with a **Pydantic model** (`JudgeOutput`) — missing fields fall back to safe defaults instead of raising.

```mermaid
graph TD
    A[docs + answer] --> B[sentence-key both]
    B --> C[build prompt with keyed text]
    C --> D[LLM call — JSON mode]
    D --> E[JudgeOutput.model_validate_json]
    E --> F{compute_judge_scores}
    F --> G[adherence]
    F --> H[relevance]
    F --> I[utilization]
    F --> J[completeness]
```

![LLM as judge](images/LLM_as_judge.png)

#### Judge output schema (`JudgeOutput`)

```json
{
  "overall_supported": true,
  "all_relevant_sentence_keys": ["0_0", "1_2"],
  "all_utilized_sentence_keys": ["0_0"],
  "sentence_support_information": [
    { "response_sentence_key": "r_0", "supporting_sentence_keys": ["0_0"], "fully_supported": true }
  ]
}
```

Document keys follow `{doc_idx}_{sent_idx}`. Answer sentence keys follow `r_{idx}`.

#### Metric derivation from judge output

Let `V` = set of all valid document sentence keys.

| Metric | Formula |
|---|---|
| **Relevance** | `|relevant ∩ V| / |V|` |
| **Utilization** | `|utilized ∩ V| / |V|` |
| **Completeness** | `|relevant ∩ utilized| / |relevant|` |
| **Adherence** | `1.0` if `overall_supported` else `0.0` |

`relevant` = `all_relevant_sentence_keys ∩ V` (invalid keys silently dropped).  
`utilized` = `all_utilized_sentence_keys ∩ V`.  
All values clipped to `[0, 1]`.

---

## Aggregate benchmark statistics

Computed once per benchmark run, comparing predicted scores against RAGBench gold labels.

### RMSE (per metric)

```
RMSE(metric) = sqrt( mean( (pred_i − gold_i)² ) )
```

Computed over all N questions for relevance, utilization, and completeness. NaN rows are masked before averaging.

### Hallucination AUROC

Uses `gold_adherence` (binary 0/1 from RAGBench) as the ground-truth label and `1 − pred_adherence` as the predicted hallucination score:

```
y_true  = round( 1 − gold_adherence )    # 1 = hallucinated, 0 = faithful
y_score = 1 − pred_adherence             # higher = more likely hallucinated

AUROC = sklearn.metrics.roc_auc_score(y_true, y_score)
```

Returns `NaN` when `y_true` has only one unique value (e.g., all samples are faithful in a small run). Displayed as `N/A` in the UI.

---

## Caching

### Benchmark result cache (`app/cache.py`)

Disk-backed CSV at `data/cache/benchmark_results.csv`. Key: `(config_hash, row_index)` where:
- `config_hash` = MD5 of `{dataset, index_id, retrieval_strategy, reranker, llm_model, dense_k, bm25_k, fusion_top_k, top_n, eval_method}`
- `row_index` = original DataFrame index of the RAGBench row (stable across runs — **not** a reset 0..n-1 position)

On re-run: only rows not already in cache are evaluated. Rows with the same config + same question reuse cached scores.

### Query transform cache (`app/data/cache/query_transforms.json`)

MD5-keyed JSON dict. Key: `"{transform_type}|{question}"`. Avoids re-calling Groq for the same rewrite/expand/HyDE on repeated demo runs.

---

## Index storage (HF Hub)

```
Abhiram83/rag-capstone-indices/
├── index_config.json          ← authoritative config downloaded at startup
├── {index_id}/
│   ├── faiss.index            ← FAISS IndexFlatIP
│   ├── embeddings.npy         ← L2-normalised passage embeddings
│   ├── bm25.pkl               ← BM25Okapi instance
│   ├── chunks.parquet         ← chunk text + doc_id
│   ├── docs.parquet           ← source documents
│   └── meta.json              ← build metadata (model, chunk config, prefixes)
└── reports/
    ├── index.json             ← lightweight list of all saved reports
    ├── report_{id}.json       ← full benchmark result (config + scores + per-question)
    └── deleted_reports.json   ← archive of deleted reports
```