docs/architecture.md

Capstone RAG — Architecture

System overview

A modular RAG explorer built on Gradio + HuggingFace Spaces. The index artifacts (FAISS, BM25, chunks) are precomputed offline and stored in an HF dataset repo. At runtime the app downloads them on demand, runs retrieval + generation, and evaluates the result against RAGBench gold labels.

graph TD
    RB[RAGBench Dataset<br/>rungalileo/ragbench]
    KB[KB Builder<br/>Colab Notebook]
    HF[(HF Dataset Repo<br/>Abhiram83/rag-capstone-indices)]

    RB -->|load corpus + questions| KB
    KB -->|push FAISS · BM25 · chunks<br/>+ index_config.json| HF

    subgraph APP [Gradio Space]
        QA["🔍 Single-question tab<br/>───────────────────<br/>pick config · type question<br/>see answer · retrieved chunks<br/>gold scores side-by-side"]

        BM["⚡ Benchmark tab<br/>───────────────────<br/>pick config · choose N<br/>run pipeline on N random Qs<br/>summary + per-question table<br/>RMSE / AUROC vs gold"]

        RP["📋 Reports tab<br/>───────────────────<br/>list saved runs<br/>load full detail · delete"]

        CP["⚖️ Compare tab<br/>───────────────────<br/>multi-select reports<br/>grouped bar chart<br/>scores table · config diff"]
    end

    HF -->|download indices on first use<br/>load index_config.json at startup| APP

    BM -->|Save as Report<br/>report JSON + index.json| HF
    HF -->|auto-load on startup · Refresh| RP
    HF -->|auto-load on startup · Refresh| CP

---

Pipeline flow

Single-question path

sequenceDiagram
    participant U as User
    participant P as RagPipeline
    participant QT as QueryTransform
    participant R as HybridRetriever
    participant RR as Reranker
    participant LLM as Groq LLM
    participant E as Evaluator

    U->>P: question
    P->>QT: transform(question, strategy)
    QT-->>P: query / variants / hypothetical
    P->>R: retrieve(query, strategy)
    R-->>P: candidate chunks [(idx, score)]
    P->>RR: rerank(query, candidates, top_n)
    RR-->>P: reranked [(idx, cross_encoder_score)]
    P->>P: sentence_window_expand(reranked)
    P->>LLM: generate(context, question)
    LLM-->>P: answer
    P-->>U: {answer, context, relevance, trace}
    U->>E: score(result, eval_method)
    note over E: NLI path — local DeBERTa NLI model<br/>adherence · utilization · ROUGE-L completeness
    E-->>U: {relevance, adherence, utilization, completeness}<br/>displayed alongside gold scores

Benchmark path

sequenceDiagram
    participant U as User
    participant B as run_benchmark
    participant P as RagPipeline
    participant NLI as NLI Scorer<br/>(local DeBERTa)
    participant J as LLM Judge<br/>(Groq / OpenAI)
    participant C as DiskCache

    U->>B: run(config, n_samples, eval_method)
    B->>B: sample(dataset, n, random_seed)
    loop each question
        B->>C: lookup(config_hash, row_idx)
        alt cache hit
            C-->>B: cached scores
        else
            B->>P: answer(question, trace=use_judge)
            P-->>B: {answer, context, relevance}
            alt eval_method = local_nli
                B->>NLI: adherence(context, answer)
                B->>NLI: utilization(context, answer)
                B->>NLI: completeness(answer, gold_response) via ROUGE-L
                NLI-->>B: {adherence, utilization, completeness}
            else eval_method = groq / openai
                B->>J: score(question, docs, answer)<br/>JSON mode + Pydantic validation
                J-->>B: {adherence, relevance, utilization, completeness}
            end
            B->>C: store(config_hash, row_idx, scores + gold_labels)
        end
    end
    B->>B: compute summary means + RMSE/AUROC vs gold
    B-->>U: summary_df, per_q_df, stats_df, bar_chart

---

Retrieval

Strategies

Strategy	What happens
Dense only	Encode query with SentenceTransformer → FAISS inner-product search → top dense_k
BM25 only	Tokenise query (lowercase split) → BM25Okapi scores → top bm25_k
Hybrid (RRF)	Run both, fuse with Reciprocal Rank Fusion → top fusion_top_k

Reciprocal Rank Fusion

Each hit from each ranked list gets score 1 / (k + rank) where k = 60 (reduces sensitivity to rank-1 outliers). Scores are summed across lists; result is re-sorted descending.

rrf_score(doc) = Σ  1 / (60 + rank_in_list_i)

Query rewrite strategies

Strategy	LLM call	What changes
None	—	Raw question sent to retriever
Rewrite	1× Groq	Single improved query replaces original
Multi-query	1× Groq	n variants generated; each retrieves independently; RRF fuses all result lists
HyDE	1× Groq	Hypothetical passage generated; its embedding used for dense search
HyDE + BM25	1× Groq	Same as HyDE for dense; original question used for BM25; RRF fuses both

Reranker

A cross-encoder (cross-encoder/ms-marco-MiniLM-L-6-v2) re-scores each (query, chunk) pair directly — no embedding compression. Takes the top top_n from the reranked list. When the reranker is off, the top top_n from retrieval are used directly.

After reranking, sentence-window expansion widens each selected chunk by ±1 sentence from its source document to improve context coverage without losing precision.

---

Evaluation

Two paths are available, selected at benchmark time.

Path 1 — Local NLI (TRACe-proxy)

Uses a local cross-encoder (cross-encoder/nli-deberta-v3-base) with no API calls.

graph LR
    A[context + answer] --> B[NLI cross-encoder]
    B --> C{label logits}
    C -->|softmax| D[P entailment / P contradiction / P neutral]
    D --> E[adherence]
    D --> F[utilization]
    G[answer + gold_response] --> H[ROUGE-L] --> I[completeness]
    J[reranker scores] --> K[sigmoid mean] --> L[relevance]

Relevance

Mean sigmoid of the cross-encoder scores from the reranker (not the NLI model):

relevance = mean( sigmoid(score_i) )   for each reranked chunk i
sigmoid(x) = 1 / (1 + e^−x)

Adherence

Single NLI call treating the full context as premise and the generated answer as hypothesis. Returns P(entailment):

adherence = P( context ⊢ answer )

Utilization

Per-sentence NLI: for each sentence in the context, ask whether the answer entails it. Fraction above threshold 0.5:

utilization = |{s ∈ context_sentences : P(answer ⊢ s) > 0.5}| / |context_sentences|

Completeness

ROUGE-L F1 between the generated answer and the gold reference response from RAGBench:

completeness = ROUGE-L F1(answer, gold_response)

Only available when a gold reference exists; nan otherwise.

---

Path 2 — LLM-as-Judge (Groq / OpenAI)

One structured LLM call per question. The judge receives sentence-keyed documents and the sentence-keyed answer, and returns a single JSON object. response_format={"type": "json_object"} is passed to the API so the model is token-level constrained to valid JSON. The response is validated with a Pydantic model (JudgeOutput) — missing fields fall back to safe defaults instead of raising.

graph TD
    A[docs + answer] --> B[sentence-key both]
    B --> C[build prompt with keyed text]
    C --> D[LLM call — JSON mode]
    D --> E[JudgeOutput.model_validate_json]
    E --> F{compute_judge_scores}
    F --> G[adherence]
    F --> H[relevance]
    F --> I[utilization]
    F --> J[completeness]

Judge output schema (JudgeOutput)

{
  "overall_supported": true,
  "all_relevant_sentence_keys": ["0_0", "1_2"],
  "all_utilized_sentence_keys": ["0_0"],
  "sentence_support_information": [
    { "response_sentence_key": "r_0", "supporting_sentence_keys": ["0_0"], "fully_supported": true }
  ]
}

Document keys follow {doc_idx}_{sent_idx}. Answer sentence keys follow r_{idx}.

Metric derivation from judge output

Let V = set of all valid document sentence keys.

Metric	Formula
Relevance	`
Utilization	`
Completeness	`
Adherence	1.0 if overall_supported else 0.0

relevant = all_relevant_sentence_keys ∩ V (invalid keys silently dropped).
utilized = all_utilized_sentence_keys ∩ V.
All values clipped to [0, 1].

---

Aggregate benchmark statistics

Computed once per benchmark run, comparing predicted scores against RAGBench gold labels.

RMSE (per metric)

RMSE(metric) = sqrt( mean( (pred_i − gold_i)² ) )

Computed over all N questions for relevance, utilization, and completeness. NaN rows are masked before averaging.

Hallucination AUROC

Uses gold_adherence (binary 0/1 from RAGBench) as the ground-truth label and 1 − pred_adherence as the predicted hallucination score:

y_true  = round( 1 − gold_adherence )    # 1 = hallucinated, 0 = faithful
y_score = 1 − pred_adherence             # higher = more likely hallucinated

AUROC = sklearn.metrics.roc_auc_score(y_true, y_score)

Returns NaN when y_true has only one unique value (e.g., all samples are faithful in a small run). Displayed as N/A in the UI.

---

Caching

Benchmark result cache (app/cache.py)

Disk-backed CSV at data/cache/benchmark_results.csv. Key: (config_hash, row_index) where:

• config_hash = MD5 of {dataset, index_id, retrieval_strategy, reranker, llm_model, dense_k, bm25_k, fusion_top_k, top_n, eval_method}
• row_index = original DataFrame index of the RAGBench row (stable across runs — not a reset 0..n-1 position)

On re-run: only rows not already in cache are evaluated. Rows with the same config + same question reuse cached scores.

Query transform cache (app/data/cache/query_transforms.json)

MD5-keyed JSON dict. Key: "{transform_type}|{question}". Avoids re-calling Groq for the same rewrite/expand/HyDE on repeated demo runs.

---

Index storage (HF Hub)

Abhiram83/rag-capstone-indices/
├── index_config.json          ← authoritative config downloaded at startup
├── {index_id}/
│   ├── faiss.index            ← FAISS IndexFlatIP
│   ├── embeddings.npy         ← L2-normalised passage embeddings
│   ├── bm25.pkl               ← BM25Okapi instance
│   ├── chunks.parquet         ← chunk text + doc_id
│   ├── docs.parquet           ← source documents
│   └── meta.json              ← build metadata (model, chunk config, prefixes)
└── reports/
    ├── index.json             ← lightweight list of all saved reports
    ├── report_{id}.json       ← full benchmark result (config + scores + per-question)
    └── deleted_reports.json   ← archive of deleted reports