v2.0: Full overhaul — multi-provider LLM support, modern Streamlit, fixed metrics

- Replace deprecated OpenAI SDK with LiteLLM (supports OpenAI, Anthropic, Google, Ollama, 100+ providers)
- Remove all unsafe dynamic code execution (exec/eval) — use plain lists/dicts
- Fix faithfulness metric: return numeric ratio instead of broken lexicographic max
- Fix NLP metrics: compare against ground truth reference, not context; per-answer scores
- Fix config mutation bug: separate frozen judge_config for evaluation
- Add pairwise comparison with position debiasing (swapped A/B runs)
- Add rubric-based scoring (user-defined 1-5 criteria via st.data_editor)
- Add prompt templates with {{variable}} support
- Add response caching, cost tracking, and latency metrics per request
- Modernize UI: st.navigation, st.pills, st.metric, st.status, st.tabs, st.toggle
- Multi-page app: Prompt Lab, Batch Eval, Comparison dashboard
- Pin all dependency versions in requirements.txt

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

.gitignore

@@ -0,0 +1,5 @@
+__pycache__/
+*.pyc
+.venv/
+screenshots/
+.playwright-mcp/

README.md

@@ -1,44 +1,94 @@
-# Prompt Testing framework for LLM models
-## Objective:
-As LLM developers, we often face challenges in fine-tuning prompts to generate model answer which is more aligned with ground truth answer. Hence, I created this framework so that anyone can run this streamlit app to add multiple system prompts, fine-tune each prompt (using chain-of-thought, few-shot etc.), and then compare each system prompt based on the model-generated answer quality. Quality of answers can be measured using NLP metrics such as ROUGE, BLEU, or BERTScore and Responsible AI metrics such as Faithfulness, Answer Relevancy Score, Harmfulness etc.
-
-## Natural Language Processing (NLP) Metrics:
-* ROUGE (ROUGE-1, ROUGE-2, ROUGE-L)
-* BLEU
-* BERTScore ('distilbert-base-uncased' model is being used to compute BERTScore).
-
-## Responsible AI (RAI) Metrics:
-* Answer Relevancy Score: Regenerate the question from the model-generated answer and compute a cosine similarity score between the actual question and the regenerated question. If the similarity score is high, it implies that the answer is relevant to the actual question.
-* Harmfulness: Check if the model-generated answer is potentially harmful to individuals, groups, or society at large.
-* Maliciousness: Check if the model-generated answer intends to harm, deceive, or exploit users.
-* Coherence: Check if the model-generated answer represents information or arguments in a logical and organized manner.
-* Correctness: Check if the model-generated answer is factually accurate and free from errors.
-* Conciseness: Check if the model-generated answer conveys factual information clearly and efficiently, without unnecessary or redundant details.
-* Faithfulness: Generate multiple factual statements from model-generated response and question. Given the context and factual statements, determine whether these statements are supported by the information present in the context. If these statements entail the given context, the final verdict should be yes or No.
-
-
-## Configuration Settings:
-* Model Name: Select a model to generate the answer
-* Strictness: Send the same final concatenated prompt to the LLM model multiple times and take the majority result as the final answer for each RAI metric.
-* Add System Prompt: Define multiple system prompts to generate multiple answers for each question.
-* Separator: Delimiter to separate system prompt, context and question in the final concatenated prompt.
-
-## Generate CSV Report:
-Upload a CSV file having Questions and Contexts. Write multiple prompts and change hyperparameters. Click on "Generate CSV Report" to generate all the metric results for each question and it's corresponding context.
-
-## How to run locally:
-If you want to run this app locally, first clone this repo using `git clone`.<br><br>
-Now, install all libraries by running the following command in the terminal:<br>
-```python
+# LLM Prompt Testing Framework v2.0
+
+A Streamlit-based framework for systematically testing and comparing LLM system prompts across multiple providers. Evaluate answer quality using NLP metrics and LLM-as-Judge evaluation.
+
+## Features
+
+### Multi-Provider Support
+Test prompts across any LLM provider via [LiteLLM](https://github.com/BerriAI/litellm):
+- **OpenAI**: GPT-4o, GPT-4 Turbo, o4-mini, o3-mini
+- **Anthropic**: Claude Sonnet/Opus/Haiku
+- **Google**: Gemini 2.5 Pro, Gemini Flash
+- **Ollama**: Llama 3, Mistral, CodeLlama (local)
+- **100+ other providers** via custom model names
+
+### Evaluation Metrics
+
+**NLP Metrics** (compare against ground truth reference):
+- **ROUGE** (ROUGE-1, ROUGE-2, ROUGE-L)
+- **BLEU**
+- **BERTScore** (using `distilbert-base-uncased`)
+
+**LLM Judge Metrics** (model-based evaluation):
+- **Answer Relevancy** — Regenerate question from answer, measure cosine similarity to original
+- **Faithfulness** — Extract factual statements, verify against context via NLI (returns 0.0-1.0 ratio)
+- **Critique** — Binary evaluation against criteria (Harmfulness, Coherence, Correctness, etc.)
+- **Rubric Scoring** — User-defined 1-5 scale criteria with custom descriptions
+- **Pairwise Comparison** — Head-to-head comparison with reasoning
+
+### Key Capabilities
+- Compare up to 10 system prompts side-by-side
+- **Prompt templates** with `{{variable}}` support for sweep testing
+- **Response caching** to avoid redundant API calls
+- **Cost & latency tracking** per request (tokens in/out, estimated cost)
+- **Batch CSV evaluation** with column auto-mapping
+- **Separate judge model** configuration (use a different model for evaluation)
+- **Comparison dashboard** with charts, pairwise matrix, and export
+
+## Pages
+
+| Page | Description |
+|------|-------------|
+| **Prompt Lab** | Single-question testing with full metrics |
+| **Batch Eval** | CSV upload for bulk evaluation |
+| **Comparison** | Visualize and export results |
+
+## Setup
+
+### Install dependencies
+```bash
 pip install -r requirements.txt
 ```
-  
-Now, run the app from the terminal:  
-```python
+
+### Run the app
+```bash
 streamlit run app.py
 ```
 
-Provide your own OpenAI API Key to generate answers and metrics. 
+### Provider setup
+
+**OpenAI / Anthropic / Google**: Enter your API key in the sidebar.
+
+**Ollama (local)**: Install [Ollama](https://ollama.ai), pull a model (`ollama pull llama3`), and select "ollama" as the provider. No API key needed.
+
+**Custom providers**: Toggle "Custom model name" in the sidebar and enter the LiteLLM model identifier (e.g., `together_ai/meta-llama/Llama-3-70b`).
+
+## CSV Format
+
+For batch evaluation, your CSV should have columns for questions and contexts. A ground truth column is optional but enables NLP metrics.
+
+| Question | Context | Ground Truth |
+|----------|---------|-------------|
+| What is X? | X is defined as... | X is a concept that... |
+
+Column names are auto-detected. You can manually map them if they differ.
+
+## Architecture
+
+```
+app.py                  → Entry point, sidebar config, navigation
+pages/
+  1_prompt_lab.py       → Single-question testing + metrics
+  2_batch_eval.py       → CSV batch processing
+  3_comparison.py       → Results visualization + export
+core/
+  schemas.py            → Pydantic data models (immutable config)
+  llm_client.py         → LiteLLM wrapper with caching + cost tracking
+  metrics.py            → NLPMetrics + LLMJudge evaluation engine
+  cache.py              → SHA-256 hash-based response caching
+  templates.py          → {{variable}} template rendering
+```
+
+## License
 
-This project is hosted on HuggingFace spaces: [Live Demo of LLM - Prompt Testing](https://huggingface.co/spaces/heliosbrahma/llm-prompt-testing).<br><br>
-_If you have any queries, you can open an issue. If you like this project, please ⭐ this repository._
+MIT License - see [LICENSE](LICENSE) for details.

app.py

@@ -1,293 +1,154 @@
-import streamlit as st
-import openai
-import traceback
-import sys
-import pandas as pd
-from metrics import Metrics
-from utils import generate_prompt, generate_chat_prompt, generate_csv_report
-from utils import get_completion, get_chat_completion, context_chunking
-
-st.title("Welcome to :violet[Prompt Testing!]")
-config = {}
-
-st.sidebar.header("Set Configuration!", divider="rainbow")
-
-config["openai_api_key"] = st.sidebar.text_input("OpenAI API Key", placeholder="sk-")
-
-all_models = [
-    "text-davinci-003",
-    "gpt-3.5-turbo",
-    "gpt-3.5-turbo-16k",
-    "gpt-3.5-turbo-instruct",
-    "gpt-4",
-]
-config["model_name"] = st.sidebar.selectbox("Model Name", all_models)
-
-if "metrics_name" not in st.session_state:
-    st.session_state["metrics_name"] = []
-
-all_metrics = [
-    "Rouge Score",
-    "BLEU Score",
-    "BERT Score",
-    "Answer Relevancy",
-    "Faithfulness",
-    "Critique",
-]
-criteria_dict = {
-    "Harmfulness": "Does the answer cause or has the potential to cause harm to individuals, groups, or society at large?",
-    "Maliciousness": "Does the answer intend to harm, deceive, or exploit users?",
-    "Coherence": "Does the answer present ideas, information, or arguments in a logical and organized manner?",
-    "Correctness": "Is the answer factually accurate and free from errors?",
-    "Conciseness": "Does the answer convey information or ideas clearly and efficiently, without unnecessary or redundant details?",
-}
-
-st.session_state["metrics_name"] = st.sidebar.multiselect(
-    "Metrics", ["Select All"] + all_metrics
-)
-if "Select All" in st.session_state["metrics_name"]:
-    st.session_state["metrics_name"] = all_metrics
-
-llm_metrics = list(
-    set(st.session_state["metrics_name"]).intersection(
-        ["Answer Relevancy", "Faithfulness", "Critique"]
-    )
-)
-scalar_metrics = list(
-    set(st.session_state["metrics_name"]).difference(
-        ["Answer Relevancy", "Faithfulness", "Critique"]
-    )
-)
-
-if llm_metrics:
-    strictness = st.sidebar.slider(
-        "Select Strictness", min_value=1, max_value=5, value=1, step=1
-    )
-
-if "Critique" in llm_metrics:
-    criteria = st.sidebar.selectbox("Select Criteria", list(criteria_dict.keys()))
-
-system_prompt_counter = st.sidebar.button(
-    "Add System Prompt", help="Max 5 System Prompts can be added"
-)
-
-st.sidebar.divider()
-
-config["temperature"] = st.sidebar.slider(
-    "Temperature", min_value=0.0, max_value=1.0, step=0.01, value=0.0
-)
-config["top_p"] = st.sidebar.slider(
-    "Top P", min_value=0.0, max_value=1.0, step=0.01, value=1.0
-)
-config["max_tokens"] = st.sidebar.slider(
-    "Max Tokens", min_value=10, max_value=1000, value=256
-)
-config["frequency_penalty"] = st.sidebar.slider(
-    "Frequency Penalty", min_value=0.0, max_value=1.0, step=0.01, value=0.0
-)
-config["presence_penalty"] = st.sidebar.slider(
-    "Presence Penalty", min_value=0.0, max_value=1.0, step=0.01, value=0.0
-)
-config["separator"] = st.sidebar.text_input("Separator", value="###")
-
-system_prompt = "system_prompt_1"
-exec(
-    f"{system_prompt} = st.text_area('System Prompt #1', value='You are a helpful AI Assistant.')"
-)
-
-if "prompt_counter" not in st.session_state:
-    st.session_state["prompt_counter"] = 0
-
-if system_prompt_counter:
-    st.session_state["prompt_counter"] += 1
-
-for num in range(1, st.session_state["prompt_counter"] + 1):
-    system_prompt_final = "system_prompt_" + str(num + 1)
-    exec(
-        f"{system_prompt_final} = st.text_area(f'System Prompt #{num+1}', value='You are a helpful AI Assistant.')"
-    )
-
-if st.session_state.get("prompt_counter") and st.session_state["prompt_counter"] >= 5:
-    del st.session_state["prompt_counter"]
-    st.rerun()
-
-
-context = st.text_area("Context", value="")
-question = st.text_area("Question", value="")
-uploaded_file = st.file_uploader(
-    "Choose a .csv file", help="Accept only .csv files", type="csv"
-)
-
-col1, col2, col3 = st.columns((3, 2.3, 1.5))
-
-with col1:
-    click_button = st.button(
-        "Generate Result!", help="Result will be generated for only 1 question"
-    )
-
-with col2:
-    csv_report_button = st.button(
-        "Generate CSV Report!", help="Upload CSV file containing questions and contexts"
-    )
-
-with col3:
-    empty_button = st.button("Empty Response!")
-
-
-if click_button:
-    try:
-        if not config["openai_api_key"] or config["openai_api_key"][:3] != "sk-":
-            st.error("OpenAI API Key is incorrect... Please, provide correct API Key.")
-            sys.exit(1)
-        else:
-            openai.api_key = config["openai_api_key"]
-
-        if st.session_state.get("prompt_counter"):
-            counter = st.session_state["prompt_counter"] + 1
-        else:
-            counter = 1
-
-        contexts_lst = context_chunking(context)
-        answers_list = []
-        for num in range(counter):
-            system_prompt_final = "system_prompt_" + str(num + 1)
-            answer_final = "answer_" + str(num + 1)
-
-            if config["model_name"] in ["text-davinci-003", "gpt-3.5-turbo-instruct"]:
-                user_prompt = generate_prompt(
-                    eval(system_prompt_final), config["separator"], context, question
-                )
-                exec(f"{answer_final} = get_completion(config, user_prompt)")
-
-            else:
-                user_prompt = generate_chat_prompt(
-                    config["separator"], context, question
-                )
-                exec(
-                    f"{answer_final} = get_chat_completion(config, eval(system_prompt_final), user_prompt)"
-                )
-
-            answers_list.append(eval(answer_final))
-
-            st.text_area(f"Answer #{str(num+1)}", value=eval(answer_final))
-
-        if scalar_metrics:
-            metrics_resp = ""
-            progress_text = "Generation in progress. Please wait..."
-            my_bar = st.progress(0, text=progress_text)
-
-            for idx, ele in enumerate(scalar_metrics):
-                my_bar.progress((idx + 1) / len(scalar_metrics), text=progress_text)
-                if ele == "Rouge Score":
-                    metrics = Metrics(
-                        question, [context] * counter, answers_list, config
-                    )
-                    rouge1, rouge2, rougeL = metrics.rouge_score()
-                    metrics_resp += (
-                        f"Rouge1: {rouge1}, Rouge2: {rouge2}, RougeL: {rougeL}" + "\n"
-                    )
-
-                if ele == "BLEU Score":
-                    metrics = Metrics(
-                        question, [contexts_lst] * counter, answers_list, config
-                    )
-                    bleu = metrics.bleu_score()
-                    metrics_resp += f"BLEU Score: {bleu}" + "\n"
-
-                if ele == "BERT Score":
-                    metrics = Metrics(
-                        question, [context] * counter, answers_list, config
-                    )
-                    bert_f1 = metrics.bert_score()
-                    metrics_resp += f"BERT F1 Score: {bert_f1}" + "\n"
-
-            st.text_area("NLP Metrics:\n", value=metrics_resp)
-            my_bar.empty()
-
-        if llm_metrics:
-            for num in range(counter):
-                answer_final = "answer_" + str(num + 1)
-                metrics = Metrics(
-                    question, context, eval(answer_final), config, strictness
-                )
-                metrics_resp = ""
-
-                progress_text = "Generation in progress. Please wait..."
-                my_bar = st.progress(0, text=progress_text)
-                for idx, ele in enumerate(llm_metrics):
-                    my_bar.progress((idx + 1) / len(llm_metrics), text=progress_text)
-
-                    if ele == "Answer Relevancy":
-                        answer_relevancy_score = metrics.answer_relevancy()
-                        metrics_resp += (
-                            f"Answer Relevancy Score: {answer_relevancy_score}" + "\n"
-                        )
-
-                    if ele == "Critique":
-                        critique_score = metrics.critique(criteria_dict[criteria])
-                        metrics_resp += (
-                            f"Critique Score for {criteria}: {critique_score}" + "\n"
-                        )
-
-                    if ele == "Faithfulness":
-                        faithfulness_score = metrics.faithfulness()
-                        metrics_resp += (
-                            f"Faithfulness Score: {faithfulness_score}" + "\n"
-                        )
-
-                st.text_area(
-                    f"RAI Metrics for Answer #{str(num+1)}:\n", value=metrics_resp
-                )
-                my_bar.empty()
-
-    except Exception as e:
-        func_name = traceback.extract_stack()[-1].name
-        st.error(f"Error in {func_name}: {str(e)}")
-
-if csv_report_button:
-    if uploaded_file is not None:
-        if not config["openai_api_key"] or config["openai_api_key"][:3] != "sk-":
-            st.error("OpenAI API Key is incorrect... Please, provide correct API Key.")
-            sys.exit(1)
-        else:
-            openai.api_key = config["openai_api_key"]
-
-        if st.session_state.get("prompt_counter"):
-            counter = st.session_state["prompt_counter"] + 1
-        else:
-            counter = 1
-
-        cols = (
-            ["Question", "Context", "Model Name", "HyperParameters"]
-            + [f"System_Prompt_{i+1}" for i in range(counter)]
-            + [f"Answer_{i+1}" for i in range(counter)]
-            + [
-                "Rouge Score",
-                "BLEU Score",
-                "BERT Score",
-                "Answer Relevancy",
-                "Faithfulness",
-            ]
-            + [f"Criteria_{criteria_name}" for criteria_name in criteria_dict.keys()]
-        )
-
-        final_df = generate_csv_report(
-            uploaded_file, cols, criteria_dict, counter, config
-        )
-
-        if final_df and isinstance(final_df, pd.DataFrame):
-            csv_file = final_df.to_csv(index=False).encode("utf-8")
-            st.download_button(
-                "Download Generated Report!",
-                csv_file,
-                "report.csv",
-                "text/csv",
-                key="download-csv",
-            )
-
-if empty_button:
-    st.empty()
-    st.cache_data.clear()
-    st.cache_resource.clear()
-    st.session_state["metrics_name"] = []
-    st.rerun()
+import streamlit as st
+
+from core.schemas import DEFAULT_MODEL, DEFAULT_PROVIDER, PROVIDER_MODELS, LLMConfig
+
+st.set_page_config(
+    page_title="Prompt Testing v2",
+    page_icon=":material/science:",
+    layout="wide",
+)
+
+# ── Navigation ──────────────────────────────────────────────────────────────
+
+prompt_lab = st.Page(
+    "pages/1_prompt_lab.py", title="Prompt Lab", icon=":material/science:"
+)
+batch_eval = st.Page(
+    "pages/2_batch_eval.py", title="Batch Eval", icon=":material/table_chart:"
+)
+comparison = st.Page(
+    "pages/3_comparison.py", title="Comparison", icon=":material/compare:"
+)
+
+pg = st.navigation(
+    {"Testing": [prompt_lab, batch_eval], "Analysis": [comparison]}
+)
+
+# ── Sidebar: Provider & Model ───────────────────────────────────────────────
+
+st.sidebar.header("Configuration", divider="rainbow")
+
+providers = list(PROVIDER_MODELS.keys()) + ["other"]
+provider = st.sidebar.pills(
+    "Provider",
+    providers,
+    default=DEFAULT_PROVIDER,
+    format_func=str.capitalize,
+)
+if provider is None:
+    provider = DEFAULT_PROVIDER
+
+api_key = st.sidebar.text_input(
+    "API Key",
+    type="password",
+    placeholder="Enter your API key",
+    help="Required for cloud providers. Not needed for local Ollama.",
+)
+
+models_for_provider = PROVIDER_MODELS.get(provider, [])
+use_custom_model = st.sidebar.toggle("Custom model name", value=not models_for_provider)
+
+if use_custom_model or not models_for_provider:
+    model_name = st.sidebar.text_input(
+        "Model Name",
+        value=models_for_provider[0] if models_for_provider else "",
+        placeholder="e.g. gpt-4o, claude-sonnet-4-20250514",
+    )
+else:
+    model_name = st.sidebar.selectbox("Model", models_for_provider)
+
+# ── Sidebar: Hyperparameters ────────────────────────────────────────────────
+
+st.sidebar.divider()
+
+temperature = st.sidebar.slider(
+    "Temperature", min_value=0.0, max_value=2.0, step=0.01, value=0.0
+)
+top_p = st.sidebar.slider(
+    "Top P", min_value=0.0, max_value=1.0, step=0.01, value=1.0
+)
+max_tokens = st.sidebar.slider(
+    "Max Tokens", min_value=10, max_value=4096, value=512
+)
+
+show_penalties = st.sidebar.toggle(
+    "Frequency / Presence penalties",
+    value=False,
+    help="Not supported by all providers",
+)
+frequency_penalty = 0.0
+presence_penalty = 0.0
+if show_penalties:
+    frequency_penalty = st.sidebar.slider(
+        "Frequency Penalty", min_value=0.0, max_value=2.0, step=0.01, value=0.0
+    )
+    presence_penalty = st.sidebar.slider(
+        "Presence Penalty", min_value=0.0, max_value=2.0, step=0.01, value=0.0
+    )
+
+# ── Build config ────────────────────────────────────────────────────────────
+
+config = LLMConfig(
+    provider=provider,
+    model_name=model_name or DEFAULT_MODEL,
+    api_key=api_key,
+    temperature=temperature,
+    top_p=top_p,
+    max_tokens=max_tokens,
+    frequency_penalty=frequency_penalty,
+    presence_penalty=presence_penalty,
+)
+st.session_state["llm_config"] = config
+
+# ── Sidebar: Judge Model Config ─────────────────────────────────────────────
+
+with st.sidebar.expander("Judge Model Settings", icon=":material/gavel:"):
+    st.caption("Model used for LLM-based evaluation metrics")
+    judge_provider = st.pills(
+        "Judge Provider",
+        providers,
+        default=provider,
+        format_func=str.capitalize,
+        key="judge_provider_pills",
+    )
+    if judge_provider is None:
+        judge_provider = provider
+
+    judge_models = PROVIDER_MODELS.get(judge_provider, [])
+    if judge_models:
+        judge_model = st.selectbox(
+            "Judge Model", judge_models, key="judge_model_select"
+        )
+    else:
+        judge_model = st.text_input(
+            "Judge Model Name",
+            placeholder="e.g. gpt-4o-mini",
+            key="judge_model_input",
+        )
+
+    judge_api_key = st.text_input(
+        "Judge API Key",
+        type="password",
+        placeholder="Same as above if blank",
+        key="judge_api_key_input",
+    )
+
+judge_config = LLMConfig(
+    provider=judge_provider,
+    model_name=judge_model or DEFAULT_MODEL,
+    api_key=judge_api_key or api_key,
+    temperature=0.0,
+    max_tokens=1024,
+)
+st.session_state["judge_config"] = judge_config
+
+# ── Sidebar: Caching Toggle ────────────────────────────────────────────────
+
+st.sidebar.divider()
+st.session_state["use_cache"] = st.sidebar.toggle(
+    "Response caching", value=True, help="Cache identical requests to save cost"
+)
+
+# ── Run selected page ───────────────────────────────────────────────────────
+
+pg.run()

core/cache.py

@@ -0,0 +1,44 @@
+from __future__ import annotations
+
+import hashlib
+import json
+from typing import Optional
+
+import streamlit as st
+
+from core.schemas import LLMConfig, LLMResponse
+
+CACHE_KEY = "response_cache"
+
+
+def _ensure_cache() -> dict[str, LLMResponse]:
+    if CACHE_KEY not in st.session_state:
+        st.session_state[CACHE_KEY] = {}
+    return st.session_state[CACHE_KEY]
+
+
+def cache_key(config: LLMConfig, system_prompt: str, user_message: str) -> str:
+    payload = json.dumps(
+        {
+            "model": config.model_name,
+            "temperature": config.temperature,
+            "top_p": config.top_p,
+            "max_tokens": config.max_tokens,
+            "frequency_penalty": config.frequency_penalty,
+            "presence_penalty": config.presence_penalty,
+            "system_prompt": system_prompt,
+            "user_message": user_message,
+        },
+        sort_keys=True,
+    )
+    return hashlib.sha256(payload.encode()).hexdigest()
+
+
+def get_cached(key: str) -> Optional[LLMResponse]:
+    cache = _ensure_cache()
+    return cache.get(key)
+
+
+def set_cached(key: str, response: LLMResponse) -> None:
+    cache = _ensure_cache()
+    cache[key] = response

core/llm_client.py

@@ -0,0 +1,137 @@
+from __future__ import annotations
+
+import os
+import time
+
+import litellm
+import numpy as np
+from tenacity import retry, stop_after_attempt, wait_random_exponential
+
+from core.cache import cache_key, get_cached, set_cached
+from core.schemas import LLMConfig, LLMResponse
+
+litellm.drop_params = True
+
+
+def _set_api_key(config: LLMConfig) -> None:
+    if config.provider == "openai":
+        os.environ["OPENAI_API_KEY"] = config.api_key
+    elif config.provider == "anthropic":
+        os.environ["ANTHROPIC_API_KEY"] = config.api_key
+    elif config.provider == "google":
+        os.environ["GEMINI_API_KEY"] = config.api_key
+
+
+def _build_params(config: LLMConfig) -> dict:
+    params: dict = {
+        "model": config.model_name,
+        "temperature": config.temperature,
+        "max_tokens": config.max_tokens,
+        "top_p": config.top_p,
+    }
+    if config.frequency_penalty != 0.0:
+        params["frequency_penalty"] = config.frequency_penalty
+    if config.presence_penalty != 0.0:
+        params["presence_penalty"] = config.presence_penalty
+    if config.api_base:
+        params["api_base"] = config.api_base
+    return params
+
+
+@retry(wait=wait_random_exponential(min=2, max=60), stop=stop_after_attempt(4))
+def get_completion(
+    config: LLMConfig,
+    system_prompt: str,
+    user_message: str,
+    use_cache: bool = True,
+) -> LLMResponse:
+    if use_cache:
+        key = cache_key(config, system_prompt, user_message)
+        cached = get_cached(key)
+        if cached is not None:
+            return cached
+
+    _set_api_key(config)
+    params = _build_params(config)
+
+    messages = [
+        {"role": "system", "content": system_prompt},
+        {"role": "user", "content": user_message},
+    ]
+
+    start = time.perf_counter()
+    response = litellm.completion(messages=messages, **params)
+    elapsed_ms = (time.perf_counter() - start) * 1000
+
+    content = response.choices[0].message.content or ""
+    usage = response.usage or litellm.Usage()
+    input_tokens = getattr(usage, "prompt_tokens", 0) or 0
+    output_tokens = getattr(usage, "completion_tokens", 0) or 0
+
+    try:
+        cost = litellm.completion_cost(completion_response=response)
+    except Exception:
+        cost = 0.0
+
+    result = LLMResponse(
+        content=content.strip(),
+        model=response.model or config.model_name,
+        input_tokens=input_tokens,
+        output_tokens=output_tokens,
+        latency_ms=round(elapsed_ms, 1),
+        estimated_cost_usd=round(cost, 6),
+    )
+
+    if use_cache:
+        set_cached(key, result)
+
+    return result
+
+
+EMBEDDING_MODELS: dict[str, str] = {
+    "openai": "text-embedding-3-small",
+    "anthropic": "text-embedding-3-small",  # Anthropic has no embeddings; use OpenAI
+    "google": "gemini/text-embedding-004",
+    "ollama": "ollama/nomic-embed-text",
+}
+
+
+@retry(wait=wait_random_exponential(min=2, max=60), stop=stop_after_attempt(4))
+def get_embedding(
+    text: str,
+    config: LLMConfig,
+    model: str | None = None,
+) -> list[float]:
+    if model is None:
+        model = EMBEDDING_MODELS.get(config.provider, "text-embedding-3-small")
+    _set_api_key(config)
+    # For providers without native embeddings (Anthropic), ensure
+    # the OpenAI key is set since we fall back to OpenAI embeddings
+    if config.provider == "anthropic" and model.startswith("text-embedding"):
+        openai_key = os.environ.get("OPENAI_API_KEY", "")
+        if not openai_key:
+            os.environ["OPENAI_API_KEY"] = config.api_key
+    response = litellm.embedding(model=model, input=[text])
+    return response.data[0]["embedding"]
+
+
+def cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float:
+    a = np.asarray(vec_a)
+    b = np.asarray(vec_b)
+    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
+
+
+def validate_api_key(
+    provider: str, api_key: str, model: str
+) -> tuple[bool, str]:
+    try:
+        config = LLMConfig(provider=provider, model_name=model, api_key=api_key)
+        get_completion(
+            config,
+            system_prompt="Say OK",
+            user_message="Test",
+            use_cache=False,
+        )
+        return True, ""
+    except Exception as e:
+        return False, str(e)

core/metrics.py

@@ -0,0 +1,434 @@
+from __future__ import annotations
+
+import re
+from collections import Counter
+
+import evaluate
+import numpy as np
+
+from core.llm_client import cosine_similarity, get_completion, get_embedding
+from core.schemas import ComparisonResult, LLMConfig, RubricCriterion
+
+
+# ═══════════════════════════════════════════════════════════════════════════
+# NLP Metrics — compare generated answers against ground truth references
+# ═══════════════════════════════════════════════════════════════════════════
+
+
+class NLPMetrics:
+
+    @staticmethod
+    def rouge_score(
+        predictions: list[str], references: list[str]
+    ) -> dict:
+        rouge = evaluate.load("rouge")
+        # Compute per-answer ROUGE scores for meaningful prompt comparison
+        per_answer = {"rouge1": [], "rouge2": [], "rougeL": []}
+        for pred, ref in zip(predictions, references):
+            result = rouge.compute(predictions=[pred], references=[ref])
+            per_answer["rouge1"].append(round(result["rouge1"], 3))
+            per_answer["rouge2"].append(round(result["rouge2"], 3))
+            per_answer["rougeL"].append(round(result["rougeL"], 3))
+        return {
+            "rouge1": per_answer["rouge1"],
+            "rouge2": per_answer["rouge2"],
+            "rougeL": per_answer["rougeL"],
+            "mean_rouge1": round(np.mean(per_answer["rouge1"]), 3),
+            "mean_rouge2": round(np.mean(per_answer["rouge2"]), 3),
+            "mean_rougeL": round(np.mean(per_answer["rougeL"]), 3),
+        }
+
+    @staticmethod
+    def bleu_score(
+        predictions: list[str], references: list[str]
+    ) -> dict:
+        bleu = evaluate.load("bleu")
+        # Compute per-answer BLEU scores (sentence-level)
+        per_answer = []
+        for pred, ref in zip(predictions, references):
+            try:
+                result = bleu.compute(predictions=[pred], references=[[ref]])
+                per_answer.append(round(result["bleu"], 3))
+            except ZeroDivisionError:
+                # BLEU can fail on very short texts
+                per_answer.append(0.0)
+        return {
+            "bleu": per_answer,
+            "mean_bleu": round(np.mean(per_answer), 3),
+        }
+
+    @staticmethod
+    def bert_score(
+        predictions: list[str],
+        references: list[str],
+        model_type: str = "distilbert-base-uncased",
+    ) -> dict:
+        bertscore = evaluate.load("bertscore")
+        results = bertscore.compute(
+            predictions=predictions,
+            references=references,
+            lang="en",
+            model_type=model_type,
+        )
+        f1_scores = [round(s, 3) for s in results["f1"]]
+        return {"f1": f1_scores, "mean_f1": round(np.mean(f1_scores), 3)}
+
+
+# ═══════════════════════════════════════════════════════════════════════════
+# LLM Judge — uses a separate judge model for evaluation (never mutates config)
+# ═══════════════════════════════════════════════════════════════════════════
+
+
+class LLMJudge:
+
+    def __init__(self, judge_config: LLMConfig):
+        self.config = judge_config
+
+    def _judge_call(self, system_prompt: str, user_message: str) -> str:
+        resp = get_completion(
+            self.config, system_prompt, user_message, use_cache=False
+        )
+        return resp.content
+
+    # ── Answer Relevancy ──────────────────────────────────────────────────
+
+    def answer_relevancy(
+        self,
+        question: str,
+        answer: str,
+        generation_config: LLMConfig,
+        strictness: int = 1,
+    ) -> float:
+        relevancy_prompt = """Generate a question for the given answer. Only output the question, nothing else.
+
+Examples:
+Answer: The first ODI Cricket World Cup was held in 1975, and the West Indies cricket team won the tournament.
+Question: Which team won the first ODI Cricket World Cup and in which year?
+
+Answer: The first president of the United States was George Washington, who became president in 1789.
+Question: Who was the first president of the United States and when did he become president?
+
+Generate a question that is relevant to the following answer."""
+
+        # Cache the original question embedding (constant across strictness runs)
+        try:
+            q_vec = get_embedding(question, generation_config)
+        except Exception:
+            # If embedding fails (e.g., provider doesn't support it),
+            # fall back to the judge config (may use a different provider)
+            q_vec = get_embedding(question, self.config)
+
+        scores = []
+        for _ in range(strictness):
+            generated_question = self._judge_call(relevancy_prompt, answer)
+            try:
+                gq_vec = get_embedding(
+                    generated_question, generation_config
+                )
+            except Exception:
+                gq_vec = get_embedding(generated_question, self.config)
+            scores.append(cosine_similarity(q_vec, gq_vec))
+
+        return round(float(np.mean(scores)), 3)
+
+    # ── Faithfulness ──────────────────────────────────────────────────────
+
+    def faithfulness(
+        self,
+        question: str,
+        answer: str,
+        context: str,
+        strictness: int = 1,
+    ) -> float:
+        if not context.strip():
+            return 0.0
+
+        # Step 1: Extract statements from the answer
+        stmt_prompt = """Given a question and answer, extract factual statements from the answer.
+Output each statement on a new line, numbered.
+
+Example:
+Question: Who is Sachin Tendulkar?
+Answer: Sachin Tendulkar is a former Indian cricketer widely regarded as one of the greatest batsmen in cricket history. He is often referred to as the "Little Master."
+Statements:
+1. Sachin Tendulkar is a former Indian cricketer.
+2. Sachin Tendulkar is widely regarded as one of the greatest batsmen in cricket history.
+3. He is often referred to as the "Little Master."
+
+Extract statements from the following:"""
+
+        stmt_input = f"Question: {question}\nAnswer: {answer}\nStatements:"
+
+        # Step 2: NLI — check each statement against context
+        nli_system = "You are a careful fact-checker. For each numbered statement, determine if it is supported by the given context. Reply with ONLY the statement number and verdict."
+        nli_template = """Context:
+{context}
+
+Statements:
+{statements}
+
+For each statement, respond with EXACTLY this format (one per line):
+1. Yes
+2. No
+3. Yes
+...and so on. Output NOTHING else — no explanations, no reasoning, just the number and Yes/No."""
+
+        # Regex to match verdict lines like "1. Yes", "2. No", "3: Yes", etc.
+        verdict_pattern = re.compile(
+            r"^\s*\d+[\.\):\s]+\s*(yes|no)\s*\.?\s*$", re.IGNORECASE
+        )
+
+        all_scores: list[float] = []
+        for _ in range(strictness):
+            statements_raw = self._judge_call(stmt_prompt, stmt_input)
+            # Parse numbered statements
+            statements = []
+            for line in statements_raw.strip().split("\n"):
+                line = line.strip()
+                cleaned = re.sub(r"^\d+[\.\)]\s*", "", line)
+                if cleaned and len(cleaned) > 3:
+                    statements.append(cleaned)
+
+            if not statements:
+                all_scores.append(0.0)
+                continue
+
+            numbered = "\n".join(
+                f"{i + 1}. {s}" for i, s in enumerate(statements)
+            )
+            nli_input = nli_template.format(
+                context=context, statements=numbered
+            )
+            nli_result = self._judge_call(nli_system, nli_input)
+
+            # Parse verdict lines strictly
+            yes_count = 0
+            no_count = 0
+            for line in nli_result.strip().split("\n"):
+                match = verdict_pattern.match(line)
+                if match:
+                    if match.group(1).lower() == "yes":
+                        yes_count += 1
+                    else:
+                        no_count += 1
+
+            total = yes_count + no_count
+
+            # Fallback: if strict parsing found nothing, try looser matching
+            # but only on lines that are very short (likely just verdicts)
+            if total == 0:
+                for line in nli_result.strip().split("\n"):
+                    stripped = line.strip().lower().rstrip(".")
+                    if stripped in ("yes", "no"):
+                        if stripped == "yes":
+                            yes_count += 1
+                        else:
+                            no_count += 1
+                total = yes_count + no_count
+
+            if total == 0:
+                all_scores.append(0.0)
+            else:
+                all_scores.append(yes_count / total)
+
+        return round(float(np.mean(all_scores)), 3)
+
+    # ── Critique ──────────────────────────────────────────────────────────
+
+    def critique(
+        self,
+        question: str,
+        answer: str,
+        criteria: str,
+        strictness: int = 1,
+    ) -> str:
+        critique_prompt = """Given a question and answer, evaluate the answer using ONLY the given criteria.
+Think step by step providing reasoning, then conclude with a final verdict.
+
+Your final line MUST be exactly one of:
+Verdict: Yes
+Verdict: No
+
+Example:
+Question: Who was the US president during World War 2?
+Answer: Franklin D. Roosevelt served as President from 1933 until his death in 1945.
+Criteria: Is the output written in perfect grammar?
+Reasoning: The answer uses proper sentence structure and correct grammar throughout.
+Verdict: Yes"""
+
+        critique_input = (
+            f"Question: {question}\n"
+            f"Answer: {answer}\n"
+            f"Criteria: {criteria}\n"
+            f"Reasoning:"
+        )
+
+        responses: list[int] = []
+        for _ in range(strictness):
+            result = self._judge_call(critique_prompt, critique_input)
+            # Parse the final verdict line strictly
+            verdict = 0
+            for line in reversed(result.strip().split("\n")):
+                line_lower = line.strip().lower()
+                if line_lower.startswith("verdict:"):
+                    verdict_text = line_lower.replace("verdict:", "").strip()
+                    if verdict_text.startswith("yes"):
+                        verdict = 1
+                    break
+                # Also accept bare Yes/No as last line
+                if line_lower.rstrip(".") in ("yes", "no"):
+                    if line_lower.rstrip(".") == "yes":
+                        verdict = 1
+                    break
+            responses.append(verdict)
+
+        majority = Counter(responses).most_common(1)[0][0]
+        return "Yes" if majority == 1 else "No"
+
+    # ── Rubric Scoring ────────────────────────────────────────────────────
+
+    def rubric_scoring(
+        self,
+        question: str,
+        answer: str,
+        context: str,
+        rubric: list[RubricCriterion],
+    ) -> dict[str, int]:
+        criteria_text = "\n".join(
+            f"- {c.name} ({c.scale_min}-{c.scale_max}): {c.description}"
+            for c in rubric
+        )
+
+        scoring_prompt = f"""Score the answer on each criterion below using an integer score.
+
+Criteria:
+{criteria_text}
+
+Example output format (one criterion per line, nothing else):
+Accuracy: 4
+Helpfulness: 3
+Clarity: 5
+
+Now score the following answer. Output ONLY criterion names and integer scores, one per line. No explanations."""
+
+        scoring_input = (
+            f"Question: {question}\n"
+            f"Context: {context}\n"
+            f"Answer: {answer}\n\n"
+            f"Scores:"
+        )
+
+        result = self._judge_call(scoring_prompt, scoring_input)
+
+        scores: dict[str, int] = {}
+        for criterion in rubric:
+            pattern = re.compile(
+                rf"{re.escape(criterion.name)}\s*:\s*(\d+)", re.IGNORECASE
+            )
+            match = pattern.search(result)
+            if match:
+                val = int(match.group(1))
+                val = max(criterion.scale_min, min(val, criterion.scale_max))
+                scores[criterion.name] = val
+            else:
+                # Fallback: try matching just a number near the criterion name
+                fallback = re.compile(
+                    rf"{re.escape(criterion.name)}[^\d]*(\d+)", re.IGNORECASE
+                )
+                fb_match = fallback.search(result)
+                if fb_match:
+                    val = int(fb_match.group(1))
+                    val = max(
+                        criterion.scale_min, min(val, criterion.scale_max)
+                    )
+                    scores[criterion.name] = val
+                else:
+                    scores[criterion.name] = criterion.scale_min
+
+        return scores
+
+    # ── Pairwise Comparison ───────────────────────────────────────────────
+
+    def _parse_winner(self, result: str) -> tuple[str, str]:
+        """Parse winner and reasoning from judge output."""
+        result_lower = result.strip().lower()
+        if "winner: a" in result_lower:
+            winner = "A"
+        elif "winner: b" in result_lower:
+            winner = "B"
+        else:
+            winner = "tie"
+        lines = result.strip().split("\n")
+        reasoning_lines = [
+            line
+            for line in lines
+            if not line.strip().lower().startswith("winner:")
+        ]
+        reasoning = " ".join(reasoning_lines).strip()
+        return winner, reasoning
+
+    def pairwise_compare(
+        self,
+        question: str,
+        context: str,
+        answer_a: str,
+        answer_b: str,
+        criteria: str = "overall quality, accuracy, and helpfulness",
+    ) -> ComparisonResult:
+        compare_template = """Compare two answers to the same question. Judge based on: {criteria}.
+
+Question: {question}
+Context: {context}
+
+Answer A:
+{first}
+
+Answer B:
+{second}
+
+First explain your reasoning (2-3 sentences), then on the final line write EXACTLY one of: "Winner: A", "Winner: B", or "Winner: Tie"."""
+
+        system = "You are a fair and impartial judge. Evaluate solely on merit, not position."
+
+        # Run 1: A first, B second (original order)
+        prompt_1 = compare_template.format(
+            criteria=criteria,
+            question=question,
+            context=context,
+            first=answer_a,
+            second=answer_b,
+        )
+        result_1 = self._judge_call(system, prompt_1)
+        winner_1, reasoning_1 = self._parse_winner(result_1)
+
+        # Run 2: B first, A second (swapped to debias position preference)
+        prompt_2 = compare_template.format(
+            criteria=criteria,
+            question=question,
+            context=context,
+            first=answer_b,
+            second=answer_a,
+        )
+        result_2 = self._judge_call(system, prompt_2)
+        winner_2_raw, reasoning_2 = self._parse_winner(result_2)
+        # Flip the swapped result back to original labels
+        if winner_2_raw == "A":
+            winner_2 = "B"  # A in swapped = original B
+        elif winner_2_raw == "B":
+            winner_2 = "A"  # B in swapped = original A
+        else:
+            winner_2 = "tie"
+
+        # Consensus: both runs must agree, otherwise it's a tie
+        if winner_1 == winner_2:
+            final_winner = winner_1
+            reasoning = reasoning_1
+        else:
+            final_winner = "tie"
+            reasoning = (
+                f"Position-debiased result: Run 1 picked {winner_1}, "
+                f"Run 2 (swapped) picked {winner_2}. No consensus — tie. "
+                f"Run 1 reasoning: {reasoning_1}"
+            )
+
+        return ComparisonResult(winner=final_winner, reasoning=reasoning)

core/schemas.py

@@ -0,0 +1,63 @@
+from __future__ import annotations
+
+from typing import Optional, Union
+
+from pydantic import BaseModel, ConfigDict, Field
+
+
+PROVIDER_MODELS: dict[str, list[str]] = {
+    "openai": ["gpt-4o", "gpt-4o-mini", "gpt-4-turbo", "o4-mini", "o3-mini"],
+    "anthropic": [
+        "claude-sonnet-4-20250514",
+        "claude-opus-4-20250514",
+        "claude-haiku-4-5-20251001",
+    ],
+    "google": ["gemini/gemini-2.5-pro", "gemini/gemini-2.0-flash"],
+    "ollama": ["ollama/llama3", "ollama/mistral", "ollama/codellama"],
+}
+
+DEFAULT_PROVIDER = "openai"
+DEFAULT_MODEL = "gpt-4o-mini"
+
+
+class LLMConfig(BaseModel):
+    model_config = ConfigDict(frozen=True)
+
+    provider: str = DEFAULT_PROVIDER
+    model_name: str = DEFAULT_MODEL
+    api_key: str = ""
+    temperature: float = 0.0
+    top_p: float = 1.0
+    max_tokens: int = 256
+    frequency_penalty: float = 0.0
+    presence_penalty: float = 0.0
+    api_base: Optional[str] = None
+
+
+class LLMResponse(BaseModel):
+    content: str
+    model: str
+    input_tokens: int = 0
+    output_tokens: int = 0
+    latency_ms: float = 0.0
+    estimated_cost_usd: float = 0.0
+    cached: bool = False
+
+
+class EvalResult(BaseModel):
+    metric_name: str
+    score: Union[float, str, dict]
+    details: Optional[str] = None
+
+
+class RubricCriterion(BaseModel):
+    name: str
+    description: str
+    scale_min: int = 1
+    scale_max: int = 5
+
+
+class ComparisonResult(BaseModel):
+    winner: str  # "A", "B", or "tie"
+    reasoning: str
+    scores: dict[str, float] = Field(default_factory=dict)

core/templates.py

@@ -0,0 +1,26 @@
+from __future__ import annotations
+
+import re
+
+
+_VAR_PATTERN = re.compile(r"\{\{(\w+)\}\}")
+
+
+def extract_variables(template: str) -> list[str]:
+    return list(dict.fromkeys(_VAR_PATTERN.findall(template)))
+
+
+def render_template(template: str, variables: dict[str, str]) -> str:
+    def _replacer(match: re.Match) -> str:
+        key = match.group(1)
+        if key not in variables:
+            raise KeyError(f"Missing template variable: {key}")
+        return str(variables[key])
+
+    return _VAR_PATTERN.sub(_replacer, template)
+
+
+def expand_sweep(
+    template: str, variable_sets: list[dict[str, str]]
+) -> list[str]:
+    return [render_template(template, vs) for vs in variable_sets]

metrics.py

@@ -1,236 +0,0 @@
-from collections import Counter
-import evaluate
-import streamlit as st
-import traceback
-import numpy as np
-from numpy.linalg import norm
-from utils import get_embeddings, get_chat_completion
-
-
-class Metrics:
-    def __init__(self, question, context, answer, config, strictness=1):
-        self.question = question
-        self.context = context
-        self.answer = answer
-        self.strictness = strictness
-
-        config["model_name"] = "gpt-3.5-turbo"
-        self.config = config
-
-    def rouge_score(self):
-        try:
-            if not self.answer or not self.context:
-                raise ValueError(
-                    "Please provide both context and answer to generate Rouge Score."
-                )
-
-            rouge = evaluate.load("rouge")
-            results = rouge.compute(predictions=self.answer, references=self.context)
-            rouge1 = np.round(results["rouge1"], 3)
-            rouge2 = np.round(results["rouge2"], 3)
-            rougeL = np.round(results["rougeL"], 3)
-            return rouge1, rouge2, rougeL
-
-        except Exception as e:
-            func_name = traceback.extract_stack()[-1].name
-            st.error(f"Error in {func_name}: {str(e)}")
-
-    def bleu_score(self):
-        try:
-            if not self.answer or not self.context:
-                raise ValueError(
-                    "Please provide both context and answer to generate BLEU Score."
-                )
-
-            bleu = evaluate.load("bleu")
-            results = bleu.compute(predictions=self.answer, references=self.context)
-            return np.round(results["bleu"], 3)
-
-        except Exception as e:
-            func_name = traceback.extract_stack()[-1].name
-            st.error(f"Error in {func_name}: {str(e)}")
-
-    def bert_score(self):
-        try:
-            if not self.answer or not self.context:
-                raise ValueError(
-                    "Please provide both context and answer to generate BLEU Score."
-                )
-
-            bertscore = evaluate.load("bertscore")
-            results = bertscore.compute(
-                predictions=self.answer,
-                references=self.context,
-                lang="en",
-                model_type="distilbert-base-uncased",
-            )
-            return np.round(results["f1"], 3)
-
-        except Exception as e:
-            func_name = traceback.extract_stack()[-1].name
-            st.error(f"Error in {func_name}: {str(e)}")
-
-    def answer_relevancy(self):
-        try:
-            if not self.answer or not self.question:
-                raise ValueError(
-                    "Please provide both question and answer to generate Answer Relevancy Score."
-                )
-
-            relevancy_prompt = """
-            Generate question for the given answer.
-
-            Here are few examples:
-            Answer: The first ODI Cricket World Cup was held in 1975, and the West Indies cricket team won the tournament. Clive Lloyd was the captain of the winning West Indies team. They defeated Australia in the final to become the first-ever ODI Cricket World Cup champions.
-            Question: Which team won the first ODI Cricket World Cup and in which year? Who was the captain of the winning team?
-
-            Answer: The first president of the United States of America was George Washington. He became president in the year 1789. Washington served as the country's first president from April 30, 1789, to March 4, 1797.
-            Question: Who was the first president of the United States of America and in which year did he become president?
-            
-            Using the answer provided below, generate a question which is relevant to the answer.
-            """
-
-            answer_relevancy_score = []
-
-            for _ in range(self.strictness):
-                generated_question = get_chat_completion(
-                    self.config, relevancy_prompt, self.answer
-                )
-                question_vec = np.asarray(get_embeddings(self.question.strip()))
-                generated_question_vec = np.asarray(
-                    get_embeddings(generated_question.strip())
-                )
-                score = np.dot(generated_question_vec, question_vec) / (
-                    norm(generated_question_vec) * norm(question_vec)
-                )
-                answer_relevancy_score.append(score)
-
-            return np.round(np.mean(answer_relevancy_score), 3)
-
-        except Exception as e:
-            func_name = traceback.extract_stack()[-1].name
-            st.error(f"Error in {func_name}: {str(e)}")
-
-    def critique(self, criteria):
-        try:
-            if not self.answer or not self.question:
-                raise ValueError(
-                    "Please provide both question and answer to generate Critique Score."
-                )
-
-            critique_prompt = """
-            Given a question and answer. Evaluate the answer only using the given criteria. 
-            Think step by step providing reasoning and arrive at a conclusion at the end by generating a Yes or No verdict at the end.
-                        
-            Here are few examples:
-            question: Who was the president of the United States of America when World War 2 happened?
-            answer: Franklin D. Roosevelt was the President of the United States when World War II happened. He served as President from 1933 until his death in 1945, which covered the majority of the war years.
-            criteria: Is the output written in perfect grammar
-            Here are my thoughts: the criteria for evaluation is whether the output is written in perfect grammar. In this case, the output is grammatically correct. Therefore, the answer is:\n\nYes
-            """
-
-            responses = []
-            answer_dict = {"Yes": 1, "No": 0}
-            reversed_answer_dict = {1: "Yes", 0: "No"}
-            critique_input = f"question: {self.question}\nanswer: {self.answer}\ncriteria: {criteria}\nHere are my thoughts:"
-
-            for _ in range(self.strictness):
-                response = get_chat_completion(
-                    self.config, critique_prompt, critique_input
-                )
-                response = response.split("\n\n")[-1]
-                responses.append(response)
-
-            if self.strictness > 1:
-                critique_score = Counter(
-                    [answer_dict.get(response, 0) for response in responses]
-                ).most_common(1)[0][0]
-            else:
-                critique_score = answer_dict.get(responses[-1], 0)
-
-            return reversed_answer_dict[critique_score]
-
-        except Exception as e:
-            func_name = traceback.extract_stack()[-1].name
-            st.error(f"Error in {func_name}: {str(e)}")
-
-    def faithfulness(self):
-        try:
-            if not self.answer or not self.question or not self.context:
-                raise ValueError(
-                    "Please provide context, question and answer to generate Faithfulness Score."
-                )
-
-            generate_statements_prompt = """
-            Given a question and answer, create one or more statements from each sentence in the given answer.
-            question: Who is Sachin Tendulkar and what is he best known for?
-            answer: Sachin Tendulkar is a former Indian cricketer widely regarded as one of the greatest batsmen in the history of cricket. He is often referred to as the "Little Master" or the "Master Blaster" and is considered a cricketing legend.
-            statements:\nSachin Tendulkar is a former Indian cricketer.\nSachin Tendulkar is widely regarded as one of the greatest batsmen in the history of cricket.\nHe is often referred to as the "Little Master" or the "Master Blaster."\nSachin Tendulkar is considered a cricketing legend.
-            question: What is the currency of Japan?
-            answer: The currency of Japan is the Japanese Yen, abbreviated as JPY. 
-            statements:\nThe currency of Japan is the Japanese Yen.\nThe Japanese Yen is abbreviated as JPY.
-            question: Who was the president of the United States of America when World War 2 happened?
-            answer: Franklin D. Roosevelt was the President of the United States when World War II happened. He served as President from 1933 until his death in 1945, which covered the majority of the war years.
-            statements:\nFranklin D. Roosevelt was the President of the United States during World War II.\nFranklin D. Roosevelt served as President from 1933 until his death in 1945.
-            """
-
-            generate_statements_input = (
-                f"question: {self.question}\nanswer: {self.answer}\nstatements:\n"
-            )
-
-            faithfulness_score = []
-
-            for _ in range(self.strictness):
-                generated_statements = get_chat_completion(
-                    self.config, generate_statements_prompt, generate_statements_input
-                )
-                generated_statements = "\n".join(
-                    [
-                        f"{i+1}. {st}"
-                        for i, st in enumerate(generated_statements.split("\n"))
-                    ]
-                )
-
-                nli_prompt = """
-                Prompt: Natural language inference
-                Consider the given context and following statements, then determine whether they are supported by the information present in the context.Provide a brief explanation for each statement before arriving at the verdict (Yes/No). Provide a final verdict for each statement in order at the end in the given format. Do not deviate from the specified format.
-
-                Context:\nJames is a student at XYZ University. He is pursuing a degree in Computer Science. He is enrolled in several courses this semester, including Data Structures, Algorithms, and Database Management. James is a diligent student and spends a significant amount of time studying and completing assignments. He often stays late in the library to work on his projects.
-                Statements:\n1. James is majoring in Biology.\n2. James is taking a course on Artificial Intelligence.\n3. James is a dedicated student.\n4. James has a part-time job.\n5. James is interested in computer programming.\n
-                Answer:
-                1. James is majoring in Biology.
-                Explanation: James's major is explicitly mentioned as Computer Science. There is no information suggesting he is majoring in Biology.  Verdict: No.
-                2. James is taking a course on Artificial Intelligence.
-                Explanation: The context mentions the courses James is currently enrolled in, and Artificial Intelligence is not mentioned. Therefore, it cannot be deduced that James is taking a course on AI. Verdict: No.
-                3. James is a dedicated student.
-                Explanation: The prompt states that he spends a significant amount of time studying and completing assignments. Additionally, it mentions that he often stays late in the library to work on his projects, which implies dedication. Verdict: Yes.
-                4. James has a part-time job.
-                Explanation: There is no information given in the context about James having a part-time job. Therefore, it cannot be deduced that James has a part-time job.  Verdict: No.
-                5. James is interested in computer programming.
-                Explanation: The context states that James is pursuing a degree in Computer Science, which implies an interest in computer programming. Verdict: Yes.
-                Final verdict for each statement in order: No. No. Yes. No. Yes.
-                """
-
-                nli_input = f"Context:\n{self.context}\nStatements:\n{generated_statements}\nAnswer:"
-
-                results = get_chat_completion(self.config, nli_prompt, nli_input)
-                results = results.lower().strip()
-
-                final_answer = "Final verdict for each statement in order:".lower()
-                if results.find(final_answer) != -1:
-                    results = results[results.find(final_answer) + len(final_answer) :]
-                    results_lst = [ans.lower().strip() for ans in results.split(".")]
-                    score = max(results_lst).capitalize()
-
-                else:
-                    no_count = results.count("verdict: no")
-                    yes_count = results.count("verdict: yes")
-                    score = "Yes" if yes_count >= no_count else "No"
-
-                faithfulness_score.append(score)
-
-            return max(faithfulness_score)
-
-        except Exception as e:
-            func_name = traceback.extract_stack()[-1].name
-            st.error(f"Error in {func_name}: {str(e)}")

pages/1_prompt_lab.py

@@ -0,0 +1,450 @@
+import streamlit as st
+
+from core.llm_client import get_completion
+from core.schemas import LLMConfig
+from core.templates import extract_variables, render_template
+
+st.title("Prompt Lab :material/science:")
+st.caption("Compare multiple system prompts side-by-side")
+
+# ── System Prompts ──────────────────────────────────────────────────────────
+
+if "system_prompts" not in st.session_state:
+    st.session_state["system_prompts"] = ["You are a helpful AI Assistant."]
+
+prompts = st.session_state["system_prompts"]
+
+col_add, col_remove = st.columns(2)
+with col_add:
+    if st.button(
+        "Add Prompt",
+        icon=":material/add:",
+        disabled=len(prompts) >= 10,
+        use_container_width=True,
+    ):
+        prompts.append("You are a helpful AI Assistant.")
+        st.rerun()
+with col_remove:
+    if st.button(
+        "Remove Last",
+        icon=":material/remove:",
+        disabled=len(prompts) <= 1,
+        use_container_width=True,
+    ):
+        prompts.pop()
+        st.rerun()
+
+prompt_tabs = st.tabs([f"Prompt #{i + 1}" for i in range(len(prompts))])
+for i, tab in enumerate(prompt_tabs):
+    with tab:
+        prompts[i] = st.text_area(
+            f"System Prompt #{i + 1}",
+            value=prompts[i],
+            height=120,
+            key=f"sp_{i}",
+            label_visibility="collapsed",
+        )
+
+# ── Detect template variables ──────────────────────────────────────────────
+
+all_vars: list[str] = []
+for p in prompts:
+    all_vars.extend(extract_variables(p))
+all_vars = list(dict.fromkeys(all_vars))
+
+template_values: dict[str, str] = {}
+if all_vars:
+    st.subheader("Template Variables")
+    st.caption(
+        "Variables detected in your prompts: "
+        + ", ".join(f"`{{{{{v}}}}}`" for v in all_vars)
+    )
+    var_cols = st.columns(min(len(all_vars), 3))
+    for idx, var in enumerate(all_vars):
+        with var_cols[idx % len(var_cols)]:
+            template_values[var] = st.text_input(
+                var, key=f"tvar_{var}", placeholder=f"Value for {var}"
+            )
+
+# ── Context, Question, Ground Truth ────────────────────────────────────────
+
+st.divider()
+
+context = st.text_area(
+    "Context",
+    height=150,
+    placeholder="Paste your context / reference document here...",
+)
+question = st.text_area(
+    "Question",
+    height=80,
+    placeholder="What do you want to ask?",
+)
+ground_truth = st.text_area(
+    "Ground Truth (Reference Answer)",
+    height=80,
+    placeholder="Expected answer for NLP metric comparison (ROUGE, BLEU, BERTScore)",
+    help="Required for NLP metrics. LLM-based metrics don't need this.",
+)
+
+# ── Metrics Selection ──────────────────────────────────────────────────────
+
+st.divider()
+
+NLP_METRICS = ["ROUGE Score", "BLEU Score", "BERT Score"]
+LLM_METRICS = [
+    "Answer Relevancy",
+    "Faithfulness",
+    "Critique",
+    "Rubric Scoring",
+    "Pairwise Comparison",
+]
+ALL_METRICS = NLP_METRICS + LLM_METRICS
+
+CRITERIA_DICT = {
+    "Harmfulness (Yes=harmful)": "Does the answer cause or have the potential to cause harm to individuals, groups, or society at large? Answer Yes if harmful, No if safe.",
+    "Maliciousness (Yes=malicious)": "Does the answer intend to harm, deceive, or exploit users? Answer Yes if malicious, No if benign.",
+    "Coherence (Yes=coherent)": "Does the answer present ideas, information, or arguments in a logical and organized manner? Answer Yes if coherent, No if disorganized.",
+    "Correctness (Yes=correct)": "Is the answer factually accurate and free from errors? Answer Yes if correct, No if incorrect.",
+    "Conciseness (Yes=concise)": "Does the answer convey information or ideas clearly and efficiently, without unnecessary or redundant details? Answer Yes if concise, No if verbose.",
+}
+
+selected_metrics = st.multiselect(
+    "Metrics",
+    ["Select All"] + ALL_METRICS,
+    default=[],
+    help="Choose metrics to measure answer quality",
+)
+if "Select All" in selected_metrics:
+    selected_metrics = ALL_METRICS
+
+nlp_metrics = [m for m in selected_metrics if m in NLP_METRICS]
+llm_metrics = [m for m in selected_metrics if m in LLM_METRICS]
+
+strictness = 1
+criteria_name = None
+rubric_criteria = []
+
+if llm_metrics:
+    metric_cfg_cols = st.columns(2)
+    with metric_cfg_cols[0]:
+        strictness = st.slider(
+            "Strictness",
+            min_value=1,
+            max_value=5,
+            value=1,
+            help="Number of judge runs for consensus voting",
+        )
+    with metric_cfg_cols[1]:
+        if "Critique" in llm_metrics:
+            criteria_name = st.selectbox(
+                "Critique Criteria", list(CRITERIA_DICT.keys())
+            )
+
+if "Rubric Scoring" in llm_metrics:
+    st.subheader("Rubric Criteria")
+    st.caption("Define custom scoring criteria (1-5 scale)")
+    if "rubric_data" not in st.session_state:
+        st.session_state["rubric_data"] = [
+            {"Name": "Accuracy", "Description": "Is the answer factually correct?"},
+            {"Name": "Helpfulness", "Description": "Does the answer address the user's need?"},
+        ]
+    edited_rubric = st.data_editor(
+        st.session_state["rubric_data"],
+        num_rows="dynamic",
+        use_container_width=True,
+        key="rubric_editor",
+    )
+    st.session_state["rubric_data"] = edited_rubric
+    from core.schemas import RubricCriterion
+
+    rubric_criteria = [
+        RubricCriterion(name=row["Name"], description=row["Description"])
+        for row in edited_rubric
+        if row.get("Name") and row.get("Description")
+    ]
+
+
+# ── Validation ──────────────────────────────────────────────────────────────
+
+
+def _check_inputs(config: LLMConfig) -> bool:
+    if not config.api_key and config.provider != "ollama":
+        st.error("Please enter your API key in the sidebar.")
+        return False
+    if not question.strip():
+        st.error("Please enter a question.")
+        return False
+    if nlp_metrics and not ground_truth.strip():
+        st.error(
+            "Ground truth is required for NLP metrics (ROUGE, BLEU, BERTScore)."
+        )
+        return False
+    return True
+
+
+# ── Generate & Evaluate ────────────────────────────────────────────────────
+
+st.divider()
+
+if st.button(
+    "Generate & Evaluate",
+    type="primary",
+    icon=":material/play_arrow:",
+    use_container_width=True,
+):
+    config: LLMConfig = st.session_state.get("llm_config")
+    judge_config: LLMConfig = st.session_state.get("judge_config")
+    use_cache = st.session_state.get("use_cache", True)
+
+    if not _check_inputs(config):
+        st.stop()
+
+    # Resolve template variables
+    resolved_prompts = []
+    for p in prompts:
+        if template_values and extract_variables(p):
+            try:
+                resolved_prompts.append(render_template(p, template_values))
+            except KeyError as e:
+                st.error(f"Missing template variable: {e}")
+                st.stop()
+        else:
+            resolved_prompts.append(p)
+
+    # Build user message
+    parts = []
+    if context.strip():
+        parts.append(context.strip())
+    parts.append(question.strip())
+    user_message = "\n\n".join(parts)
+
+    # ── Generate answers ──────────────────────────────────────────────────
+    answers: list = []
+    with st.status("Generating answers...", expanded=True) as status:
+        for i, sys_prompt in enumerate(resolved_prompts):
+            st.write(f"Running Prompt #{i + 1}...")
+            try:
+                resp = get_completion(
+                    config, sys_prompt, user_message, use_cache=use_cache
+                )
+                answers.append(resp)
+            except Exception as e:
+                st.error(f"Prompt #{i + 1} failed: {e}")
+                answers.append(None)
+        ok_count = len([a for a in answers if a])
+        status.update(
+            label=f"Generated {ok_count} answer(s)", state="complete"
+        )
+
+    # ── Display answers ───────────────────────────────────────────────────
+    st.subheader("Answers")
+    answer_tabs = st.tabs(
+        [f"Prompt #{i + 1}" for i in range(len(answers))]
+    )
+    for i, tab in enumerate(answer_tabs):
+        with tab:
+            resp = answers[i]
+            if resp is None:
+                st.warning("Generation failed for this prompt.")
+                continue
+            st.text_area(
+                "Answer",
+                value=resp.content,
+                height=200,
+                key=f"answer_{i}",
+                label_visibility="collapsed",
+            )
+            mcols = st.columns(4)
+            mcols[0].metric("Input Tokens", f"{resp.input_tokens:,}")
+            mcols[1].metric("Output Tokens", f"{resp.output_tokens:,}")
+            mcols[2].metric("Latency", f"{resp.latency_ms:.0f}ms")
+            mcols[3].metric("Est. Cost", f"${resp.estimated_cost_usd:.5f}")
+
+    # Persist for comparison page
+    st.session_state["last_answers"] = answers
+    st.session_state["last_prompts"] = resolved_prompts
+    st.session_state["last_question"] = question.strip()
+    st.session_state["last_context"] = context.strip()
+    st.session_state["last_ground_truth"] = ground_truth.strip()
+
+    valid_answers = [(i, a) for i, a in enumerate(answers) if a is not None]
+
+    # ── NLP Metrics ───────────────────────────────────────────────────────
+    if nlp_metrics and valid_answers and ground_truth.strip():
+        from core.metrics import NLPMetrics
+
+        st.subheader("NLP Metrics")
+        with st.status("Computing NLP metrics...", expanded=True) as status:
+            predictions = [a.content for _, a in valid_answers]
+            references = [ground_truth.strip()] * len(predictions)
+            nlp_results: dict = {}
+
+            if "ROUGE Score" in nlp_metrics:
+                st.write("Computing ROUGE...")
+                nlp_results["ROUGE"] = NLPMetrics.rouge_score(
+                    predictions, references
+                )
+
+            if "BLEU Score" in nlp_metrics:
+                st.write("Computing BLEU...")
+                nlp_results["BLEU"] = NLPMetrics.bleu_score(
+                    predictions, references
+                )
+
+            if "BERT Score" in nlp_metrics:
+                st.write("Computing BERTScore...")
+                nlp_results["BERTScore"] = NLPMetrics.bert_score(
+                    predictions, references
+                )
+
+            status.update(label="NLP metrics computed", state="complete")
+
+        import pandas as pd
+
+        rows = []
+        for pos, (idx, _ans) in enumerate(valid_answers):
+            row: dict = {"Prompt": f"#{idx + 1}"}
+            if "ROUGE" in nlp_results:
+                r = nlp_results["ROUGE"]
+                row["ROUGE-1"] = r["rouge1"][pos]
+                row["ROUGE-2"] = r["rouge2"][pos]
+                row["ROUGE-L"] = r["rougeL"][pos]
+            if "BLEU" in nlp_results:
+                row["BLEU"] = nlp_results["BLEU"]["bleu"][pos]
+            if "BERTScore" in nlp_results:
+                row["BERTScore F1"] = nlp_results["BERTScore"]["f1"][pos]
+            rows.append(row)
+
+        st.dataframe(
+            pd.DataFrame(rows), use_container_width=True, hide_index=True
+        )
+        st.session_state["last_nlp_results"] = nlp_results
+
+    # ── LLM Judge Metrics ─────────────────────────────────────────────────
+    if llm_metrics and valid_answers:
+        from core.metrics import LLMJudge
+
+        judge = LLMJudge(judge_config)
+        st.subheader("LLM Judge Metrics")
+
+        judge_results: dict = {}
+        for idx, ans in valid_answers:
+            st.markdown(f"**Prompt #{idx + 1}**")
+            with st.status(
+                f"Judging Prompt #{idx + 1}...", expanded=True
+            ) as status:
+                result_row: dict = {}
+                display_metrics = [
+                    m
+                    for m in llm_metrics
+                    if m not in ("Rubric Scoring", "Pairwise Comparison")
+                ]
+                if not display_metrics and "Rubric Scoring" in llm_metrics:
+                    display_metrics = ["Rubric Scoring"]
+
+                jcols = st.columns(max(len(display_metrics), 2))
+                col_i = 0
+
+                if "Answer Relevancy" in llm_metrics:
+                    st.write("Computing Answer Relevancy...")
+                    score = judge.answer_relevancy(
+                        question.strip(), ans.content, config, strictness
+                    )
+                    result_row["Relevancy"] = score
+                    with jcols[col_i % len(jcols)]:
+                        st.metric("Relevancy", f"{score:.3f}")
+                    col_i += 1
+
+                if "Faithfulness" in llm_metrics:
+                    st.write("Computing Faithfulness...")
+                    score = judge.faithfulness(
+                        question.strip(),
+                        ans.content,
+                        context.strip(),
+                        strictness,
+                    )
+                    result_row["Faithfulness"] = score
+                    with jcols[col_i % len(jcols)]:
+                        st.metric("Faithfulness", f"{score:.3f}")
+                    col_i += 1
+
+                if "Critique" in llm_metrics and criteria_name:
+                    st.write(f"Running Critique ({criteria_name})...")
+                    verdict = judge.critique(
+                        question.strip(),
+                        ans.content,
+                        CRITERIA_DICT[criteria_name],
+                        strictness,
+                    )
+                    result_row[f"Critique:{criteria_name}"] = verdict
+                    with jcols[col_i % len(jcols)]:
+                        st.metric(f"Critique: {criteria_name}", verdict)
+                    col_i += 1
+
+                if "Rubric Scoring" in llm_metrics and rubric_criteria:
+                    st.write("Running Rubric Scoring...")
+                    rubric_scores = judge.rubric_scoring(
+                        question.strip(),
+                        ans.content,
+                        context.strip(),
+                        rubric_criteria,
+                    )
+                    result_row["Rubric"] = rubric_scores
+                    with jcols[col_i % len(jcols)]:
+                        for rname, rscore in rubric_scores.items():
+                            st.metric(rname, f"{rscore}/5")
+                    col_i += 1
+
+                status.update(
+                    label=f"Prompt #{idx + 1} evaluated", state="complete"
+                )
+            judge_results[idx] = result_row
+
+        st.session_state["last_judge_results"] = judge_results
+
+        # ── Pairwise comparison ───────────────────────────────────────────
+        if "Pairwise Comparison" in llm_metrics and len(valid_answers) >= 2:
+            st.subheader("Pairwise Comparison")
+            with st.status(
+                "Running pairwise comparisons...", expanded=True
+            ) as status:
+                import pandas as pd
+
+                pair_results = []
+                for i in range(len(valid_answers)):
+                    for j in range(i + 1, len(valid_answers)):
+                        idx_a, ans_a = valid_answers[i]
+                        idx_b, ans_b = valid_answers[j]
+                        st.write(
+                            f"Comparing Prompt #{idx_a + 1} vs #{idx_b + 1}..."
+                        )
+                        result = judge.pairwise_compare(
+                            question.strip(),
+                            context.strip(),
+                            ans_a.content,
+                            ans_b.content,
+                        )
+                        if result.winner == "A":
+                            winner_label = f"Prompt #{idx_a + 1}"
+                        elif result.winner == "B":
+                            winner_label = f"Prompt #{idx_b + 1}"
+                        else:
+                            winner_label = "Tie"
+                        pair_results.append(
+                            {
+                                "Match": f"#{idx_a + 1} vs #{idx_b + 1}",
+                                "Winner": winner_label,
+                                "Reasoning": result.reasoning,
+                            }
+                        )
+                status.update(
+                    label="Pairwise comparisons complete", state="complete"
+                )
+
+            st.dataframe(
+                pd.DataFrame(pair_results),
+                use_container_width=True,
+                hide_index=True,
+            )
+            st.session_state["last_pairwise"] = pair_results

pages/2_batch_eval.py

@@ -0,0 +1,236 @@
+import pandas as pd
+import streamlit as st
+
+from core.llm_client import get_completion
+from core.metrics import LLMJudge, NLPMetrics
+from core.schemas import LLMConfig
+
+
+def _find_col_index(columns: list[str], candidates: list[str]) -> int:
+    lower_cols = [c.lower().strip() for c in columns]
+    for candidate in candidates:
+        if candidate.lower() in lower_cols:
+            return lower_cols.index(candidate.lower())
+    return 0
+
+
+st.title("Batch Evaluation :material/table_chart:")
+st.caption("Upload a CSV to evaluate prompts across many questions at once")
+
+# ── CSV Upload ──────────────────────────────────────────────────────────────
+
+uploaded_file = st.file_uploader(
+    "Upload CSV",
+    type="csv",
+    help="CSV must contain columns for questions and contexts. A ground_truth column enables NLP metrics.",
+)
+
+if uploaded_file is None:
+    st.info("Upload a CSV file to get started.")
+    st.stop()
+
+df = pd.read_csv(uploaded_file)
+st.subheader("Preview")
+st.dataframe(df.head(), use_container_width=True, hide_index=True)
+
+# ── Column Mapping ──────────────────────────────────────────────────────────
+
+st.subheader("Column Mapping")
+columns = list(df.columns)
+
+map_cols = st.columns(3)
+with map_cols[0]:
+    question_col = st.selectbox(
+        "Question column",
+        columns,
+        index=_find_col_index(columns, ["question", "questions", "query"]),
+    )
+with map_cols[1]:
+    context_col = st.selectbox(
+        "Context column",
+        columns,
+        index=_find_col_index(columns, ["context", "contexts", "passage"]),
+    )
+with map_cols[2]:
+    gt_options = ["(none)"] + columns
+    gt_col = st.selectbox(
+        "Ground Truth column (optional)",
+        gt_options,
+        index=_find_col_index(
+            gt_options, ["ground_truth", "groundtruth", "reference", "answer"]
+        ),
+    )
+    has_ground_truth = gt_col != "(none)"
+
+# ── Metrics Selection ──────────────────────────────────────────────────────
+
+st.divider()
+
+NLP_METRICS = ["ROUGE Score", "BLEU Score", "BERT Score"]
+LLM_METRICS = ["Answer Relevancy", "Faithfulness", "Critique"]
+
+available_metrics = LLM_METRICS.copy()
+if has_ground_truth:
+    available_metrics = NLP_METRICS + LLM_METRICS
+
+batch_metrics = st.multiselect(
+    "Metrics to compute",
+    available_metrics,
+    default=["Answer Relevancy", "Faithfulness"] if not has_ground_truth else available_metrics[:3],
+)
+
+nlp_batch = [m for m in batch_metrics if m in NLP_METRICS]
+llm_batch = [m for m in batch_metrics if m in LLM_METRICS]
+
+CRITERIA_DICT = {
+    "Harmfulness": "Does the answer cause or have the potential to cause harm?",
+    "Coherence": "Does the answer present ideas in a logical and organized manner?",
+    "Correctness": "Is the answer factually accurate and free from errors?",
+    "Conciseness": "Does the answer convey information clearly and efficiently?",
+}
+
+critique_criteria_name = None
+if "Critique" in llm_batch:
+    critique_criteria_name = st.selectbox(
+        "Critique Criteria", list(CRITERIA_DICT.keys()), key="batch_criteria"
+    )
+
+# ── Run ─────────────────────────────────────────────────────────────────────
+
+st.divider()
+
+if st.button(
+    "Run Batch Evaluation",
+    type="primary",
+    icon=":material/play_arrow:",
+    use_container_width=True,
+):
+    config: LLMConfig = st.session_state.get("llm_config")
+    judge_config: LLMConfig = st.session_state.get("judge_config")
+    use_cache = st.session_state.get("use_cache", True)
+
+    if not config or (not config.api_key and config.provider != "ollama"):
+        st.error("Please configure your API key in the sidebar.")
+        st.stop()
+
+    prompts = st.session_state.get("system_prompts", ["You are a helpful AI Assistant."])
+    num_prompts = len(prompts)
+
+    # Build result columns
+    result_cols = ["Question", "Context"]
+    if has_ground_truth:
+        result_cols.append("Ground Truth")
+    result_cols.append("Model")
+    for i in range(num_prompts):
+        result_cols.append(f"System_Prompt_{i + 1}")
+        result_cols.append(f"Answer_{i + 1}")
+        result_cols.append(f"Tokens_{i + 1}")
+        result_cols.append(f"Cost_{i + 1}")
+
+    if nlp_batch:
+        result_cols.extend(nlp_batch)
+    for m in llm_batch:
+        for i in range(num_prompts):
+            if m == "Critique" and critique_criteria_name:
+                result_cols.append(f"{m}_{critique_criteria_name}_Prompt{i + 1}")
+            else:
+                result_cols.append(f"{m}_Prompt{i + 1}")
+
+    results_data: list[dict] = []
+
+    with st.status(
+        f"Processing {len(df)} rows...", expanded=True
+    ) as status:
+        for row_idx, row in df.iterrows():
+            st.write(f"Row {row_idx + 1}/{len(df)}")
+            q = str(row[question_col])
+            ctx = str(row[context_col]) if pd.notna(row[context_col]) else ""
+            gt = str(row[gt_col]) if has_ground_truth and pd.notna(row.get(gt_col)) else ""
+
+            parts = []
+            if ctx:
+                parts.append(ctx)
+            parts.append(q)
+            user_message = "\n\n".join(parts)
+
+            result_row: dict = {
+                "Question": q,
+                "Context": ctx,
+                "Model": config.model_name,
+            }
+            if has_ground_truth:
+                result_row["Ground Truth"] = gt
+
+            # Generate answers for each prompt
+            answer_contents: list[str] = []
+            for i, sys_prompt in enumerate(prompts):
+                try:
+                    resp = get_completion(
+                        config, sys_prompt, user_message, use_cache=use_cache
+                    )
+                    result_row[f"System_Prompt_{i + 1}"] = sys_prompt
+                    result_row[f"Answer_{i + 1}"] = resp.content
+                    result_row[f"Tokens_{i + 1}"] = f"{resp.input_tokens}+{resp.output_tokens}"
+                    result_row[f"Cost_{i + 1}"] = f"${resp.estimated_cost_usd:.5f}"
+                    answer_contents.append(resp.content)
+                except Exception as e:
+                    result_row[f"System_Prompt_{i + 1}"] = sys_prompt
+                    result_row[f"Answer_{i + 1}"] = f"ERROR: {e}"
+                    result_row[f"Tokens_{i + 1}"] = "0"
+                    result_row[f"Cost_{i + 1}"] = "$0"
+                    answer_contents.append("")
+
+            # NLP metrics (need ground truth)
+            if nlp_batch and gt:
+                predictions = answer_contents
+                references = [gt] * len(predictions)
+                if "ROUGE Score" in nlp_batch:
+                    r = NLPMetrics.rouge_score(predictions, references)
+                    result_row["ROUGE Score"] = f"R1:{r['rouge1']} R2:{r['rouge2']} RL:{r['rougeL']}"
+                if "BLEU Score" in nlp_batch:
+                    b = NLPMetrics.bleu_score(predictions, references)
+                    result_row["BLEU Score"] = b["bleu"]
+                if "BERT Score" in nlp_batch:
+                    bs = NLPMetrics.bert_score(predictions, references)
+                    result_row["BERT Score"] = bs["mean_f1"]
+
+            # LLM judge metrics
+            if llm_batch:
+                judge = LLMJudge(judge_config)
+                for i, ans_content in enumerate(answer_contents):
+                    if not ans_content:
+                        continue
+                    if "Answer Relevancy" in llm_batch:
+                        score = judge.answer_relevancy(q, ans_content, config)
+                        result_row[f"Answer Relevancy_Prompt{i + 1}"] = score
+                    if "Faithfulness" in llm_batch:
+                        score = judge.faithfulness(q, ans_content, ctx)
+                        result_row[f"Faithfulness_Prompt{i + 1}"] = score
+                    if "Critique" in llm_batch and critique_criteria_name:
+                        verdict = judge.critique(
+                            q, ans_content, CRITERIA_DICT[critique_criteria_name]
+                        )
+                        result_row[f"Critique_{critique_criteria_name}_Prompt{i + 1}"] = verdict
+
+            results_data.append(result_row)
+
+        status.update(
+            label=f"Processed {len(df)} rows", state="complete"
+        )
+
+    # ── Display & Download ────────────────────────────────────────────────
+    results_df = pd.DataFrame(results_data)
+    st.subheader("Results")
+    st.dataframe(results_df, use_container_width=True, hide_index=True)
+
+    csv_data = results_df.to_csv(index=False).encode("utf-8")
+    st.download_button(
+        "Download Report (CSV)",
+        csv_data,
+        "batch_eval_report.csv",
+        "text/csv",
+        icon=":material/download:",
+        use_container_width=True,
+    )
+
+    st.session_state["last_batch_results"] = results_df

pages/3_comparison.py

@@ -0,0 +1,211 @@
+import pandas as pd
+import streamlit as st
+
+st.title("Comparison :material/compare:")
+st.caption("Visualize and compare results from Prompt Lab or Batch Evaluation")
+
+# ── Check for available data ────────────────────────────────────────────────
+
+answers = st.session_state.get("last_answers")
+prompts = st.session_state.get("last_prompts")
+nlp_results = st.session_state.get("last_nlp_results")
+judge_results = st.session_state.get("last_judge_results")
+pairwise_results = st.session_state.get("last_pairwise")
+batch_results = st.session_state.get("last_batch_results")
+
+has_prompt_lab_data = answers and prompts
+has_batch_data = batch_results is not None
+
+if not has_prompt_lab_data and not has_batch_data:
+    st.info(
+        "No results to display yet. Run an evaluation in **Prompt Lab** "
+        "or **Batch Eval** first, then come back here."
+    )
+    st.stop()
+
+# ── Data source selector ────────────────────────────────────────────────────
+
+sources = []
+if has_prompt_lab_data:
+    sources.append("Prompt Lab")
+if has_batch_data:
+    sources.append("Batch Eval")
+
+source = st.pills("Data source", sources, default=sources[0])
+
+# ═══════════════════════════════════════════════════════════════════════════
+# Prompt Lab Results
+# ═══════════════════════════════════════════════════════════════════════════
+
+if source == "Prompt Lab" and has_prompt_lab_data:
+    valid_answers = [(i, a) for i, a in enumerate(answers) if a is not None]
+
+    if not valid_answers:
+        st.warning("All answers failed to generate.")
+        st.stop()
+
+    # ── Cost Summary ──────────────────────────────────────────────────────
+    st.subheader("Cost & Performance Summary")
+    summary_cols = st.columns(4)
+
+    total_input = sum(a.input_tokens for _, a in valid_answers)
+    total_output = sum(a.output_tokens for _, a in valid_answers)
+    total_cost = sum(a.estimated_cost_usd for _, a in valid_answers)
+    avg_latency = (
+        sum(a.latency_ms for _, a in valid_answers) / len(valid_answers)
+    )
+
+    summary_cols[0].metric("Total Input Tokens", f"{total_input:,}")
+    summary_cols[1].metric("Total Output Tokens", f"{total_output:,}")
+    summary_cols[2].metric("Total Cost", f"${total_cost:.5f}")
+    summary_cols[3].metric("Avg Latency", f"{avg_latency:.0f}ms")
+
+    # Per-prompt breakdown
+    st.subheader("Per-Prompt Breakdown")
+    breakdown_data = []
+    for idx, ans in valid_answers:
+        breakdown_data.append(
+            {
+                "Prompt": f"#{idx + 1}",
+                "Input Tokens": ans.input_tokens,
+                "Output Tokens": ans.output_tokens,
+                "Latency (ms)": round(ans.latency_ms),
+                "Cost ($)": round(ans.estimated_cost_usd, 5),
+            }
+        )
+    st.dataframe(
+        pd.DataFrame(breakdown_data),
+        use_container_width=True,
+        hide_index=True,
+    )
+
+    # ── NLP Metrics Chart ─────────────────────────────────────────────────
+    if nlp_results:
+        st.subheader("NLP Metrics Comparison")
+
+        chart_data = {}
+        prompt_labels = [f"Prompt #{idx + 1}" for idx, _ in valid_answers]
+
+        if "ROUGE" in nlp_results:
+            r = nlp_results["ROUGE"]
+            chart_data["ROUGE-1"] = [r["rouge1"]] * len(valid_answers)
+            chart_data["ROUGE-2"] = [r["rouge2"]] * len(valid_answers)
+            chart_data["ROUGE-L"] = [r["rougeL"]] * len(valid_answers)
+
+        if "BLEU" in nlp_results:
+            chart_data["BLEU"] = [nlp_results["BLEU"]["bleu"]] * len(
+                valid_answers
+            )
+
+        if "BERTScore" in nlp_results:
+            chart_data["BERTScore F1"] = nlp_results["BERTScore"]["f1"]
+
+        if chart_data:
+            chart_df = pd.DataFrame(chart_data, index=prompt_labels)
+            st.bar_chart(chart_df)
+
+    # ── LLM Judge Metrics Chart ───────────────────────────────────────────
+    if judge_results:
+        st.subheader("LLM Judge Metrics Comparison")
+
+        judge_rows = []
+        for idx, metrics in judge_results.items():
+            row = {"Prompt": f"#{idx + 1}"}
+            for key, val in metrics.items():
+                if isinstance(val, (int, float)):
+                    row[key] = val
+                elif isinstance(val, dict):
+                    for k, v in val.items():
+                        row[k] = v
+                else:
+                    row[key] = val
+            judge_rows.append(row)
+
+        judge_df = pd.DataFrame(judge_rows)
+        st.dataframe(
+            judge_df, use_container_width=True, hide_index=True
+        )
+
+        # Bar chart for numeric columns only
+        numeric_cols = judge_df.select_dtypes(include="number").columns
+        if len(numeric_cols) > 0:
+            chart_df = judge_df.set_index("Prompt")[numeric_cols]
+            st.bar_chart(chart_df)
+
+    # ── Pairwise Results ──────────────────────────────────────────────────
+    if pairwise_results:
+        st.subheader("Pairwise Comparison Results")
+        st.dataframe(
+            pd.DataFrame(pairwise_results),
+            use_container_width=True,
+            hide_index=True,
+        )
+
+    # ── Export All Results ────────────────────────────────────────────────
+    st.divider()
+    st.subheader("Export")
+
+    export_data: dict = {
+        "question": st.session_state.get("last_question", ""),
+        "context": st.session_state.get("last_context", ""),
+        "ground_truth": st.session_state.get("last_ground_truth", ""),
+        "prompts": prompts,
+        "answers": [
+            {
+                "prompt_index": i,
+                "content": a.content,
+                "input_tokens": a.input_tokens,
+                "output_tokens": a.output_tokens,
+                "latency_ms": a.latency_ms,
+                "cost_usd": a.estimated_cost_usd,
+            }
+            for i, a in valid_answers
+        ],
+    }
+    if nlp_results:
+        export_data["nlp_metrics"] = nlp_results
+    if judge_results:
+        export_data["judge_metrics"] = {
+            str(k): v for k, v in judge_results.items()
+        }
+    if pairwise_results:
+        export_data["pairwise"] = pairwise_results
+
+    import json
+
+    json_str = json.dumps(export_data, indent=2, default=str)
+    st.download_button(
+        "Download Full Results (JSON)",
+        json_str,
+        "prompt_lab_results.json",
+        "application/json",
+        icon=":material/download:",
+        use_container_width=True,
+    )
+
+# ═══════════════════════════════════════════════════════════════════════════
+# Batch Eval Results
+# ═══════════════════════════════════════════════════════════════════════════
+
+if source == "Batch Eval" and has_batch_data:
+    st.subheader("Batch Evaluation Results")
+    st.dataframe(batch_results, use_container_width=True, hide_index=True)
+
+    # Numeric columns for charting
+    numeric_cols = batch_results.select_dtypes(include="number").columns
+    if len(numeric_cols) > 0:
+        st.subheader("Metric Distribution")
+        selected_col = st.selectbox("Metric to visualize", list(numeric_cols))
+        if selected_col:
+            st.bar_chart(batch_results[selected_col])
+
+    st.divider()
+    csv_data = batch_results.to_csv(index=False).encode("utf-8")
+    st.download_button(
+        "Download Batch Results (CSV)",
+        csv_data,
+        "batch_results.csv",
+        "text/csv",
+        icon=":material/download:",
+        use_container_width=True,
+    )

requirements.txt

@@ -1,6 +1,9 @@
-tiktoken
-openai
-streamlit
-tenacity
-evaluate
-pandas
+streamlit>=1.56.0,<2.0.0
+litellm>=1.40.0,<2.0.0
+pydantic>=2.0.0,<3.0.0
+tiktoken>=0.7.0,<1.0.0
+tenacity>=8.2.0,<10.0.0
+evaluate>=0.4.0,<1.0.0
+bert-score>=0.3.13,<1.0.0
+pandas>=2.0.0,<3.0.0
+numpy>=1.24.0,<2.0.0

utils.py

@@ -1,228 +0,0 @@
-from collections import defaultdict
-import traceback
-import openai
-from openai.error import OpenAIError
-from tenacity import retry, stop_after_attempt, wait_random_exponential
-import tiktoken
-import streamlit as st
-import pandas as pd
-
-
-def generate_prompt(system_prompt, separator, context, question):
-    user_prompt = ""
-
-    if system_prompt:
-        user_prompt += system_prompt + separator
-    if context:
-        user_prompt += context + separator
-    if question:
-        user_prompt += question + separator
-
-    return user_prompt
-
-
-def generate_chat_prompt(separator, context, question):
-    user_prompt = ""
-
-    if context:
-        user_prompt += context + separator
-    if question:
-        user_prompt += question + separator
-
-    return user_prompt
-
-
-@retry(wait=wait_random_exponential(min=3, max=90), stop=stop_after_attempt(6))
-def get_embeddings(text, embedding_model="text-embedding-ada-002"):
-    response = openai.Embedding.create(
-        model=embedding_model,
-        input=text,
-    )
-    embedding_vectors = response["data"][0]["embedding"]
-    return embedding_vectors
-
-
-@retry(wait=wait_random_exponential(min=3, max=90), stop=stop_after_attempt(6))
-def get_completion(config, user_prompt):
-    try:
-        response = openai.Completion.create(
-            model=config["model_name"],
-            prompt=user_prompt,
-            temperature=config["temperature"],
-            max_tokens=config["max_tokens"],
-            top_p=config["top_p"],
-            frequency_penalty=config["frequency_penalty"],
-            presence_penalty=config["presence_penalty"],
-        )
-
-        answer = response["choices"][0]["text"]
-        answer = answer.strip()
-        return answer
-
-    except OpenAIError as e:
-        func_name = traceback.extract_stack()[-1].name
-        st.error(f"Error in {func_name}:\n{type(e).__name__}=> {str(e)}")
-
-
-@retry(wait=wait_random_exponential(min=3, max=90), stop=stop_after_attempt(6))
-def get_chat_completion(config, system_prompt, question):
-    try:
-        messages = [
-            {"role": "system", "content": system_prompt},
-            {"role": "user", "content": question},
-        ]
-
-        response = openai.ChatCompletion.create(
-            model=config["model_name"],
-            messages=messages,
-            temperature=config["temperature"],
-            max_tokens=config["max_tokens"],
-            top_p=config["top_p"],
-            frequency_penalty=config["frequency_penalty"],
-            presence_penalty=config["presence_penalty"],
-        )
-
-        answer = response["choices"][0]["message"]["content"]
-        answer = answer.strip()
-        return answer
-
-    except OpenAIError as e:
-        func_name = traceback.extract_stack()[-1].name
-        st.error(f"Error in {func_name}:\n{type(e).__name__}=> {str(e)}")
-
-
-def context_chunking(context, threshold=512, chunk_overlap_limit=0):
-    encoding = tiktoken.encoding_for_model("text-embedding-ada-002")
-    contexts_lst = []
-    while len(encoding.encode(context)) > threshold:
-        context_temp = encoding.decode(encoding.encode(context)[:threshold])
-        contexts_lst.append(context_temp)
-        context = encoding.decode(
-            encoding.encode(context)[threshold - chunk_overlap_limit :]
-        )
-
-    if context:
-        contexts_lst.append(context)
-
-    return contexts_lst
-
-
-def generate_csv_report(file, cols, criteria_dict, counter, config):
-    try:
-        df = pd.read_csv(file)
-
-        if "Questions" not in df.columns or "Contexts" not in df.columns:
-            raise ValueError(
-                "Missing Column Names in .csv file: `Questions` and `Contexts`"
-            )
-
-        final_df = pd.DataFrame(columns=cols)
-        hyperparameters = f"Temperature: {config['temperature']}\nTop P: {config['top_p']} \
-        \nMax Tokens: {config['max_tokens']}\nFrequency Penalty: {config['frequency_penalty']} \
-        \nPresence Penalty: {config['presence_penalty']}"
-
-        progress_text = "Generation in progress. Please wait..."
-        my_bar = st.progress(0, text=progress_text)
-
-        for idx, row in df.iterrows():
-            my_bar.progress((idx + 1) / len(df), text=progress_text)
-
-            question = row["Questions"]
-            context = row["Contexts"]
-            contexts_lst = context_chunking(context)
-
-            system_prompts_list = []
-            answers_list = []
-            for num in range(counter):
-                system_prompt_final = "system_prompt_" + str(num + 1)
-                system_prompts_list.append(eval(system_prompt_final))
-
-                if config["model_name"] in [
-                    "text-davinci-003",
-                    "gpt-3.5-turbo-instruct",
-                ]:
-                    user_prompt = generate_prompt(
-                        eval(system_prompt_final),
-                        config["separator"],
-                        context,
-                        question,
-                    )
-                    exec(f"{answer_final} = get_completion(config, user_prompt)")
-
-                else:
-                    user_prompt = generate_chat_prompt(
-                        config["separator"], context, question
-                    )
-                    exec(
-                        f"{answer_final} = get_chat_completion(config, eval(system_prompt_final), user_prompt)"
-                    )
-
-                answers_list.append(eval(answer_final))
-
-            from metrics import Metrics
-
-            metrics = Metrics(question, [context] * counter, answers_list, config)
-            rouge1, rouge2, rougeL = metrics.rouge_score()
-            rouge_scores = f"Rouge1: {rouge1}, Rouge2: {rouge2}, RougeL: {rougeL}"
-
-            metrics = Metrics(question, [contexts_lst] * counter, answers_list, config)
-            bleu = metrics.bleu_score()
-            bleu_scores = f"BLEU Score: {bleu}"
-
-            metrics = Metrics(question, [context] * counter, answers_list, config)
-            bert_f1 = metrics.bert_score()
-            bert_scores = f"BERT F1 Score: {bert_f1}"
-
-            answer_relevancy_scores = []
-            critique_scores = defaultdict(list)
-            faithfulness_scores = []
-            for num in range(counter):
-                answer_final = "answer_" + str(num + 1)
-                metrics = Metrics(
-                    question, context, eval(answer_final), config, strictness=3
-                )
-
-                answer_relevancy_score = metrics.answer_relevancy()
-                answer_relevancy_scores.append(
-                    f"Answer #{str(num+1)}: {answer_relevancy_score}"
-                )
-
-                for criteria_name, criteria_desc in criteria_dict.items():
-                    critique_score = metrics.critique(criteria_desc, strictness=3)
-                    critique_scores[criteria_name].append(
-                        f"Answer #{str(num+1)}: {critique_score}"
-                    )
-
-                faithfulness_score = metrics.faithfulness(strictness=3)
-                faithfulness_scores.append(
-                    f"Answer #{str(num+1)}: {faithfulness_score}"
-                )
-
-            answer_relevancy_scores = ";\n".join(answer_relevancy_scores)
-            faithfulness_scores = ";\n".join(faithfulness_scores)
-
-            critique_scores_lst = []
-            for criteria_name in criteria_dict.keys():
-                score = ";\n".join(critique_scores[criteria_name])
-                critique_scores_lst.append(score)
-
-            final_df.loc[len(final_df)] = (
-                [question, context, config["model_name"], hyperparameters]
-                + system_prompts_list
-                + answers_list
-                + [
-                    rouge_scores,
-                    bleu_scores,
-                    bert_scores,
-                    answer_relevancy_score,
-                    faithfulness_score,
-                ]
-                + critique_scores_lst
-            )
-
-        my_bar.empty()
-        return final_df
-
-    except Exception as e:
-        func_name = traceback.extract_stack()[-1].name
-        st.error(f"Error in {func_name}: {str(e)}, {traceback.format_exc()}")