Evaluation Pipeline and Langfuse Monitoring

.gitignore

@@ -1,3 +0,0 @@
-assets/favicon.ico
-favicon.ico
-**/favicon.ico

evals/README.md

@@ -1,67 +0,0 @@
-# Evaluation Suite
-
-This directory contains a modular evaluation framework for the Fraud Model Explainability Assistant, built using the Strands Evaluation SDK while maintaining Langfuse + OpenTelemetry integration.
-
-## Structure
-
-```
-evals/
-├── __init__.py              # Package initialization
-├── config.py                # Shared evaluator configurations
-├── task_functions.py        # Agent wrapper functions
-├── langfuse_reporter.py     # Langfuse integration
-├── dataset_loader.py        # JSON to Case conversion
-├── generate_experiment.py   # SDK Experiment Generator
-├── run_helpfulness.py       # Helpfulness evaluation
-├── run_trajectory.py        # Trajectory evaluation
-└── run_full_suite.py        # Full evaluation suite
-```
-
-## Usage
-
-### Run Individual Evaluations
-
-```bash
-# Helpfulness only
-python -m evals.run_helpfulness
-
-# Trajectory only
-python -m evals.run_trajectory
-```
-
-### Run Full Suite
-
-```bash
-# All evaluators (Helpfulness, Faithfulness, Trajectory, Goal Success)
-python -m evals.run_full_suite
-```
-
-### Generate Synthetic Data
-
-```bash
-# Generate new test cases using SDK ExperimentGenerator
-python -m evals.generate_experiment
-```
-## Features
-
-- **SDK-Aligned**: Uses `Experiment.run_evaluations()` framework
-- **Modular**: Each evaluation type in separate file
-- **Langfuse Integration**: Automatic score logging with trace correlation
-- **OpenTelemetry**: Full observability stack integration
-- **Extensible**: Easy to add new evaluators or metrics
-
-## Evaluators
-
-1. **Helpfulness** (`OutputEvaluator`): Rates answer quality (accuracy, completeness, clarity)
-2. **Faithfulness** (`OutputEvaluator`): Verifies answer is grounded in retrieved context
-3. **Trajectory** (`TrajectoryEvaluator`): Checks tool usage sequence
-4. **Goal Success** (Heuristic): Matches expected key points in answer
-
-## Comparison with `evaluate.py`
-
-The modular suite provides the same functionality as the monolithic `evaluate.py` but with:
-- ✅ Better code organization
-- ✅ SDK-standard patterns
-- ✅ Easier to extend with new evaluators
-- ✅ Built-in reporting (`report.run_display()`, `get_summary()`)
-- ✅ Maintains Langfuse + OTel integration

evals/__init__.py

@@ -1,8 +0,0 @@
-"""
-Evaluation suite for the Fraud Model Explainability Assistant.
-
-This package provides modular evaluation capabilities using the Strands SDK
-while maintaining integration with Langfuse and OpenTelemetry.
-"""
-
-__version__ = "1.0.0"

evals/config.py

@@ -1,59 +0,0 @@
-"""
-Shared configuration for evaluators and models.
-"""
-from strands.models.openai import OpenAIModel
-from strands_evals.evaluators import OutputEvaluator, TrajectoryEvaluator, FaithfulnessEvaluator
-
-# Configure OpenAI model for evaluators
-eval_model = OpenAIModel(model_id="gpt-4o")
-
-# Helpfulness Evaluator
-helpfulness_evaluator = OutputEvaluator(
-    rubric="""
-    Evaluate the response based on:
-    1. Accuracy - Is the information factually correct?
-    2. Completeness - Does it fully answer the question?
-    3. Clarity - Is it easy to understand?
-    
-    Score 1.0 if all criteria are met excellently.
-    Score 0.5 if some criteria are partially met.
-    Score 0.0 if the response is inadequate or incorrect.
-    """,
-    include_inputs=True,
-    model=eval_model
-)
-
-# Faithfulness Evaluator (Generic for now, to enable data generation)
-faithfulness_evaluator = OutputEvaluator(
-    rubric="""
-    Evaluate if the response is faithful to the retrieved context.
-    
-    Score 1.0 if fully supported by context.
-    Score 0.5 if partially supported or context unavailable.
-    Score 0.0 if contains hallucinations or contradicts context.
-    
-    Penalize heavily for information NOT in the context.
-    """,
-    include_inputs=True,
-    model=eval_model
-)
-
-# Trajectory Evaluator
-trajectory_evaluator = TrajectoryEvaluator(
-    rubric="""
-    Evaluate the tool usage trajectory:
-    1. Correct tool selection - Were the right tools chosen?
-    2. Proper sequence - Logical order (Retrieve -> Analyze -> Explain)?
-    3. Efficiency - No unnecessary tools?
-    
-    Score 1.0 if optimal tools used correctly.
-    Score 0.5 if correct tools but suboptimal sequence.
-    SCORE 0.0 if wrong tools or major inefficiencies.
-    """,
-    include_inputs=True,
-    model=eval_model
-)
-
-# Key Points Evaluator (Custom)
-from evals.key_points_evaluator import KeyPointsEvaluator
-key_points_evaluator = KeyPointsEvaluator()

evals/dataset_loader.py

@@ -1,35 +0,0 @@
-"""
-Dataset loader that converts JSON to SDK Case objects.
-"""
-import json
-from typing import List
-from strands_evals import Case
-
-
-def load_cases_from_json(filepath: str) -> List[Case]:
-    """
-    Load test cases from JSON dataset and convert to SDK Case objects.
-    
-    Args:
-        filepath: Path to JSON dataset file
-        
-    Returns:
-        List of Case objects
-    """
-    with open(filepath) as f:
-        data = json.load(f)
-    
-    cases = []
-    for item in data:
-        case = Case(
-            name=item["id"],
-            input=item["question"],
-            expected_output=None,  # Not used for LLM-based evaluation
-            metadata={
-                "expected_key_points": item.get("expected_answer_key_points", []),
-                "expected_intent": item.get("expected_intent", "")
-            }
-        )
-        cases.append(case)
-    
-    return cases

evals/generate_experiment.py

@@ -1,86 +0,0 @@
-#!/usr/bin/env python
-"""
-Generate synthetic test cases using Strands ExperimentGenerator.
-
-This script replaces the custom generate_data.py and uses the SDK
-to generate diverse, high-quality test cases for the fraud agent.
-"""
-import os
-import json
-import asyncio
-from dotenv import load_dotenv
-
-load_dotenv()
-
-from typing import List, Dict
-from strands_evals.generators import ExperimentGenerator
-from strands_evals.evaluators import OutputEvaluator
-from evals.config import eval_model
-
-# Context description for the generator
-CONTEXT = """
-You are generating test cases for a Fraud Model Explainability Assistant for a financial services company.
-The assistant uses RAG and tools to explain fraud scores (0-1000), SHAP values, and compliance checks.
-
-Users are typically:
-1. Fraud Analysts (investigating specific cases)
-2. Data Scientists (monitoring model performance)
-3. Compliance Officers (checking for Fair Lending bias)
-4. Executives (asking for high-level summaries)
-
-Tools available:
-- get_application_summary(app_id): Returns score, risk level.
-- explain_fraud_score(app_id): Returns SHAP feature contributions.
-- compare_to_population(app_id): Returns stats vs approved/denied.
-- check_fair_lending_flags(app_id): Returns bias analysis.
-- get_identity_network(app_id): Returns linked applications.
-"""
-
-async def generate():
-    print("🚀 Starting Experiment Generation with SDK...")
-    
-    # Initialize generator with str input/output
-    generator = ExperimentGenerator[str, str](
-        input_type=str,
-        output_type=str,
-        model=eval_model
-    )
-    
-    # Generate experiment
-    print("   Generating cases (this may take a minute)...")
-    experiment = await generator.from_context_async(
-        context=CONTEXT,
-        num_cases=10,  # Generate 10 new cases
-        evaluator=OutputEvaluator, # Pass class, let generator create rubric
-        task_description="Explain fraud model decisions and risk factors.",
-        num_topics=5   # Split across different topics (High Risk, Compliance, etc.)
-    )
-    
-    print(f"✅ Generated {len(experiment.cases)} new test cases.")
-    
-    # Convert to our JSON format
-    new_cases = []
-    for i, case in enumerate(experiment.cases):
-        # Metadata might be None
-        metadata = case.metadata if case.metadata else {}
-        new_case = {
-            "id": f"synth_sdk_{i+1}",
-            "question": case.input,
-            "expected_intent": metadata.get("topic", "General"),
-            "expected_answer_key_points": [case.expected_output] if case.expected_output else []
-        }
-        new_cases.append(new_case)
-        print(f"   - [{new_case['expected_intent']}] {new_case['question'][:60]}...")
-
-    # Load existing cases to append (optional, or overwrite)
-    output_path = "evaluation/dataset_sdk.json"
-    
-    # Saving to a new file to avoid overwriting the main dataset during this test
-    with open(output_path, "w") as f:
-        json.dump(new_cases, f, indent=2)
-        
-    print(f"\n💾 Saved {len(new_cases)} cases to {output_path}")
-    print("   Review the file and merge into evaluation/dataset.json if desired.")
-
-if __name__ == "__main__":
-    asyncio.run(generate())

evals/key_points_evaluator.py

@@ -1,102 +0,0 @@
-"""
-Custom evaluator for checking if specific key points are present in the response.
-"""
-from typing import Any, List
-from strands_evals.evaluators import Evaluator
-from strands_evals.types.evaluation import EvaluationData, EvaluationOutput
-from strands_evals.types.trace import EvaluationLevel
-from typing_extensions import TypeVar
-
-InputT = TypeVar("InputT")
-OutputT = TypeVar("OutputT")
-
-class KeyPointsEvaluator(Evaluator[InputT, OutputT]):
-    """Evaluates output by checking for presence of expected key points (keywords/phrases)."""
-    
-    evaluation_level = EvaluationLevel.TRACE_LEVEL
-
-    def __init__(self, version: str = "v1"):
-        super().__init__()
-        self.version = version
-
-    def evaluate(self, evaluation_case: EvaluationData[InputT, OutputT]) -> List[EvaluationOutput]:
-        """Synchronous evaluation."""
-        return self._do_evaluation(evaluation_case)
-
-    async def evaluate_async(self, evaluation_case: EvaluationData[InputT, OutputT]) -> List[EvaluationOutput]:
-        """Asynchronous evaluation."""
-        return self._do_evaluation(evaluation_case)
-
-    def _do_evaluation(self, evaluation_case: EvaluationData[InputT, OutputT]) -> List[EvaluationOutput]:
-        """
-        Check if expected key points are present in the actual output.
-        Expects 'expected_key_points' list in case metadata.
-        """
-        # Get actual output
-        actual_output = str(evaluation_case.actual_output)
-        
-        # Get expectations from case metadata (which is attached to evaluation_case)
-        # Note: The SDK passes the whole Case object or relevant parts.
-        # However, EvaluationData typically has input/output. 
-        # Metadata is likely accessible if evaluation_case is constructed from a Case.
-        # But SDK EvaluationData doesn't strictly carry metadata field in all versions.
-        # We rely on how Experiment constructs it.
-        
-        # EXPERIMENTAL: The SDK's Experiment loop constructs EvaluationData.
-        # If it doesn't pass metadata, we need to inspect the source 'case'. 
-        # But Evaluator.evaluate receives EvaluationData, not Case.
-        # Wait, Strands SDK 1.22 might have metadata on EvaluationData?
-        # Let's check the type definition if needed. 
-        # For now, assuming we can access it or we need a workaround.
-        
-        # Workaround: For this custom evaluator to work with Experiment, 
-        # the Experiment must pass metadata. 
-        
-        # Actually, looking at the Experiment source (which we can't see right now but inferred),
-        # it might be easier to pass expected_output as the key points string?
-        # Dataset loader sets: expected_key_points in metadata.
-        
-        # Let's try to access metadata if it exists on EvaluationData, 
-        # Otherwise fall back to a safe default.
-        
-        key_points = []
-        if hasattr(evaluation_case, 'metadata') and evaluation_case.metadata:
-            key_points = evaluation_case.metadata.get("expected_key_points", [])
-        
-        # Calculate score
-        if not key_points:
-            return [EvaluationOutput(
-                score=1.0, 
-                test_pass=True, 
-                reason="No key points defined for this case.",
-                label="N/A"
-            )]
-
-        hits = 0
-        misses = []
-        
-        for point in key_points:
-            point_lower = point.lower()
-            output_lower = actual_output.lower()
-            
-            if point_lower in output_lower:
-                hits += 1
-            # partial match check (heuristic from run_full_suite)
-            elif any(word in output_lower for word in point_lower.split() if len(word) > 4):
-                 hits += 0.5
-                 misses.append(f"{point} (Partial)")
-            else:
-                 misses.append(point)
-
-        score = min(1.0, hits / len(key_points))
-        
-        reason = f"Matched {hits}/{len(key_points)} key points."
-        if misses:
-            reason += f" Missed: {', '.join(misses[:3])}..."
-            
-        return [EvaluationOutput(
-            score=score,
-            test_pass=score >= 0.7, # 70% threshold
-            reason=reason,
-            label=f"{int(score*100)}%"
-        )]

evals/langfuse_reporter.py

@@ -1,96 +0,0 @@
-"""
-Langfuse integration for logging evaluation results.
-"""
-import os
-import base64
-import httpx
-from typing import List, Dict
-from opentelemetry import trace
-
-
-class LangfuseClient:
-    """Client for logging evaluation scores to Langfuse."""
-    
-    def __init__(self):
-        self.secret_key = os.environ.get("LANGFUSE_SECRET_KEY")
-        self.public_key = os.environ.get("LANGFUSE_PUBLIC_KEY")
-        self.base_url = os.environ.get("LANGFUSE_BASE_URL", "https://cloud.langfuse.com").rstrip("/")
-        
-        if not (self.secret_key and self.public_key):
-            print("⚠ Langfuse credentials missing. Scoring disabled.")
-            self.enabled = False
-            return
-            
-        self.enabled = True
-        auth_str = f"{self.public_key}:{self.secret_key}"
-        auth_bytes = auth_str.encode("ascii")
-        base64_auth = base64.b64encode(auth_bytes).decode("ascii")
-        self.headers = {
-            "Authorization": f"Basic {base64_auth}",
-            "Content-Type": "application/json"
-        }
-
-    def score_trace(self, trace_id: str, name: str, value: float, comment: str = None):
-        """Log a single score to Langfuse."""
-        if not self.enabled:
-            return
-            
-        url = f"{self.base_url}/api/public/scores"
-        payload = {
-            "traceId": trace_id,
-            "name": name,
-            "value": value,
-            "comment": comment
-        }
-        
-        try:
-            resp = httpx.post(url, json=payload, headers=self.headers, timeout=10.0)
-            if resp.status_code not in (200, 201):
-                print(f"   ⚠ Failed to log score {name}: {resp.status_code} - {resp.text}")
-        except Exception as e:
-            print(f"   ⚠ Error logging score: {e}")
-
-
-class LangfuseReporter:
-    """Reporter that logs SDK experiment results to Langfuse."""
-    
-    def __init__(self, langfuse_client: LangfuseClient):
-        self.lf_client = langfuse_client
-        self.tracer = trace.get_tracer("evaluation_reporter")
-    
-    def log_experiment_results(self, reports, case_names: List[str], evaluator_names: List[str] = None) -> Dict[str, str]:
-        """
-        Log SDK experiment results to Langfuse with OpenTelemetry trace correlation.
-        
-        Args:
-            reports: List of evaluation reports from Experiment.run_evaluations()
-            case_names: List of case names to create trace IDs for
-            evaluator_names: Optional list of evaluator names. If not provided, generic names will be used.
-            
-        Returns:
-            Dict mapping case names to trace IDs
-        """
-        trace_ids = {}
-        
-        for i, case_name in enumerate(case_names):
-            # Create OTel span for this case
-            with self.tracer.start_as_current_span(f"Eval: {case_name}") as span:
-                # Get Trace ID
-                trace_id_int = span.get_span_context().trace_id
-                trace_id_hex = "{:032x}".format(trace_id_int)
-                trace_ids[case_name] = trace_id_hex
-                
-                # Log all evaluator scores for this case
-                for j, report in enumerate(reports):
-                    if i < len(report.scores):
-                        eval_name = evaluator_names[j] if evaluator_names and j < len(evaluator_names) else f"Evaluator_{j}"
-                        eval_name = eval_name.replace("Evaluator", "")
-                        
-                        self.lf_client.score_trace(
-                            trace_id=trace_id_hex,
-                            name=eval_name,
-                            value=report.scores[i],
-                            comment=report.reasons[i] if i < len(report.reasons) else None
-                        )
-        
-        return trace_ids

evals/run_full_suite.py

@@ -1,85 +0,0 @@
-#!/usr/bin/env python
-"""
-Run the full evaluation suite on the fraud agent.
-
-This runs all evaluators (Helpfulness, Faithfulness, Trajectory, Goal Success)
-and logs results to Langfuse with OpenTelemetry trace correlation.
-"""
-import json
-from strands_evals import Experiment
-from strands_evals.types.evaluation import EvaluationData
-from evals.config import helpfulness_evaluator, faithfulness_evaluator, trajectory_evaluator, key_points_evaluator
-from evals.task_functions import get_fraud_explanation_with_trace
-from evals.dataset_loader import load_cases_from_json
-from evals.langfuse_reporter import LangfuseClient, LangfuseReporter
-from evals.utils import get_report_summary
-
-
-def main():
-    print("🚀 Starting Full Evaluation Suite\n")
-    print("="*60)
-    
-    # Load test cases
-    cases = load_cases_from_json("evaluation/dataset.json")
-    print(f"📋 Loaded {len(cases)} test cases\n")
-    
-    # Initialize Langfuse
-    lf_client = LangfuseClient()
-    reporter = LangfuseReporter(lf_client)
-    
-    # Run evaluations with all evaluators
-    print("Running evaluations with SDK Experiment framework...")
-    print("Evaluators: Helpfulness, Faithfulness, Trajectory\n")
-    
-    experiment = Experiment(
-        cases=cases,
-        evaluators=[
-            helpfulness_evaluator,
-            faithfulness_evaluator,
-            trajectory_evaluator,
-            key_points_evaluator
-        ]
-    )
-    
-    reports = experiment.run_evaluations(get_fraud_explanation_with_trace)
-    
-    # Display results for each evaluator
-    print("\n" + "="*60)
-    print("EVALUATION RESULTS")
-    print("="*60 + "\n")
-    
-    for i, report in enumerate(reports):
-        evaluator_name = type(experiment.evaluators[i]).__name__
-        print(f"\n### {evaluator_name} ###")
-        report.display()
-        
-        summary = get_report_summary(report)
-        print(f"\n📊 {evaluator_name} Summary:")
-        print(f"   Pass Rate: {summary['pass_rate']:.1%}")
-        print(f"   Average Score: {summary['average_score']:.2f}")
-        print()
-    
-    # Log to Langfuse
-    print("\n" + "="*60)
-    print("📤 Logging to Langfuse...")
-    case_names = [case.name for case in cases]
-    evaluator_names = [type(e).__name__ for e in experiment.evaluators]
-    trace_ids = reporter.log_experiment_results(reports, case_names, evaluator_names)
-    print(f"   ✅ Logged {len(trace_ids)} traces with {len(reports)} metrics each")
-    
-    # Save experiment
-    experiment.to_file("experiment_files/full_suite_eval")
-    print("\n💾 Experiment saved to ./experiment_files/full_suite_eval.json")
-    
-    # Final summary
-    print("\n" + "="*60)
-    print("✅ EVALUATION COMPLETE")
-    print("="*60)
-    print(f"\nTotal Cases: {len(cases)}")
-    print(f"Evaluators: {len(reports)}")
-    print(f"Langfuse Traces: {len(trace_ids)}")
-    print(f"\nView results in Langfuse dashboard")
-
-
-if __name__ == "__main__":
-    main()

evals/run_helpfulness.py

@@ -1,58 +0,0 @@
-#!/usr/bin/env python
-"""
-Run helpfulness evaluation on the fraud agent.
-
-This is equivalent to basic_eval.py from the SDK quickstart,
-but customized for the fraud explainability use case.
-"""
-from strands_evals import Experiment
-from evals.config import helpfulness_evaluator
-from evals.task_functions import get_fraud_explanation
-from evals.dataset_loader import load_cases_from_json
-from evals.langfuse_reporter import LangfuseClient, LangfuseReporter
-from evals.utils import get_report_summary
-
-
-def main():
-    print("=== Helpfulness Evaluation ===\n")
-    
-    # Load test cases
-    cases = load_cases_from_json("evaluation/dataset.json")
-    print(f"Loaded {len(cases)} test cases\n")
-    
-    # Create experiment
-    experiment = Experiment(
-        cases=cases,
-        evaluators=[helpfulness_evaluator]
-    )
-    
-    # Run evaluations
-    print("Running evaluations...")
-    reports = experiment.run_evaluations(get_fraud_explanation)
-    
-    # Display SDK results
-    print("\n" + "="*60)
-    reports[0].display()
-    
-    # Get summary
-    summary = get_report_summary(reports[0])
-    print(f"\n📊 Summary:")
-    print(f"   Pass Rate: {summary['pass_rate']:.1%}")
-    print(f"   Average Score: {summary['average_score']:.2f}")
-    
-    # Log to Langfuse
-    print("\n📤 Logging to Langfuse...")
-    lf_client = LangfuseClient()
-    reporter = LangfuseReporter(lf_client)
-    case_names = [case.name for case in cases]
-    evaluator_names = [type(e).__name__ for e in experiment.evaluators]
-    trace_ids = reporter.log_experiment_results(reports, case_names, evaluator_names)
-    print(f"   Logged {len(trace_ids)} traces to Langfuse")
-    
-    # Save experiment
-    experiment.to_file("experiment_files/helpfulness_eval")
-    print("\n💾 Experiment saved to ./experiment_files/helpfulness_eval.json")
-
-
-if __name__ == "__main__":
-    main()

evals/run_trajectory.py

@@ -1,57 +0,0 @@
-#!/usr/bin/env python
-"""
-Run trajectory evaluation on the fraud agent.
-
-This evaluates whether the agent uses the correct tools in the right sequence.
-"""
-from strands_evals import Experiment
-from evals.config import trajectory_evaluator
-from evals.task_functions import get_fraud_explanation_with_trace
-from evals.dataset_loader import load_cases_from_json
-from evals.langfuse_reporter import LangfuseClient, LangfuseReporter
-from evals.utils import get_report_summary
-
-
-def main():
-    print("=== Trajectory Evaluation ===\n")
-    
-    # Load test cases
-    cases = load_cases_from_json("evaluation/dataset.json")
-    print(f"Loaded {len(cases)} test cases\n")
-    
-    # Create experiment
-    experiment = Experiment(
-        cases=cases,
-        evaluators=[trajectory_evaluator]
-    )
-    
-    # Run evaluations
-    print("Running evaluations...")
-    reports = experiment.run_evaluations(get_fraud_explanation_with_trace)
-    
-    # Display SDK results
-    print("\n" + "="*60)
-    reports[0].display()
-    
-    # Get summary
-    summary = get_report_summary(reports[0])
-    print(f"\n📊 Summary:")
-    print(f"   Pass Rate: {summary['pass_rate']:.1%}")
-    print(f"   Average Score: {summary['average_score']:.2f}")
-    
-    # Log to Langfuse
-    print("\n📤 Logging to Langfuse...")
-    lf_client = LangfuseClient()
-    reporter = LangfuseReporter(lf_client)
-    case_names = [case.name for case in cases]
-    evaluator_names = [type(e).__name__ for e in experiment.evaluators]
-    trace_ids = reporter.log_experiment_results(reports, case_names, evaluator_names)
-    print(f"   Logged {len(trace_ids)} traces to Langfuse")
-    
-    # Save experiment
-    experiment.to_file("experiment_files/trajectory_eval")
-    print("\n💾 Experiment saved to ./experiment_files/trajectory_eval.json")
-
-
-if __name__ == "__main__":
-    main()

evals/task_functions.py

@@ -1,64 +0,0 @@
-"""
-Task functions that wrap the fraud agent for evaluation.
-"""
-from typing import List, Tuple
-from strands_evals import Case
-from app import query_agent
-
-
-def extract_context_and_tools(agent_result) -> Tuple[str, List[str]]:
-    """Extracts retrieved text and tool names from AgentResult."""
-    context = []
-    tool_calls = []
-    
-    if not hasattr(agent_result, 'trace') or not agent_result.trace:
-        return "", []
-
-    for span in agent_result.trace.spans:
-        # Check for tool execution spans
-        if hasattr(span, 'span_type') and str(span.span_type) == 'tool_execution':
-             # Tool Name
-             tool_name = span.tool_call.name
-             tool_calls.append(tool_name)
-             
-             # Context from Search/Load Tools
-             if 'confluence' in tool_name or 'get_application_summary' in tool_name or 'compare' in tool_name:
-                 context.append(f"Source ({tool_name}): {span.tool_result.content}")
-
-    return "\n\n".join(context), tool_calls
-
-
-def get_fraud_explanation(case: Case) -> str:
-    """
-    Task function for basic output evaluation.
-    
-    Args:
-        case: Test case with input question
-        
-    Returns:
-        Agent's response as string
-    """
-    result = query_agent(case.input, return_full_result=False)
-    return str(result)
-
-
-def get_fraud_explanation_with_trace(case: Case) -> dict:
-    """
-    Task function for trajectory and faithfulness evaluation.
-    
-    Args:
-        case: Test case with input question
-        
-    Returns:
-        Dict with output, trajectory, and context
-    """
-    result = query_agent(case.input, return_full_result=True)
-    
-    # Extract context and tools
-    context, tools = extract_context_and_tools(result)
-    
-    return {
-        "output": str(result),
-        "trajectory": tools,
-        "context": context
-    }

evals/utils.py

@@ -1,15 +0,0 @@
-"""
-Utility functions for evaluations.
-"""
-
-def get_report_summary(report):
-    """
-    Calculate summary metrics from an EvaluationReport.
-    
-    Returns:
-        dict: A dictionary containing 'pass_rate' and 'average_score'.
-    """
-    return {
-        "pass_rate": sum(report.test_passes) / len(report.test_passes) if report.test_passes else 0,
-        "average_score": report.overall_score
-    }

evaluate.py

@@ -0,0 +1,258 @@
+
+import os
+import json
+import asyncio
+import base64
+import httpx
+from typing import List, Dict, Any
+from openai import AsyncOpenAI
+
+# OpenTelemetry
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+
+# App imports (triggers setup_telemetry)
+from app import query_agent
+
+tracer = trace.get_tracer("evaluation_script")
+
+# Initialize OpenAI Client for Evaluation
+aclient = AsyncOpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
+
+class LangfuseClient:
+    def __init__(self):
+        self.secret_key = os.environ.get("LANGFUSE_SECRET_KEY")
+        self.public_key = os.environ.get("LANGFUSE_PUBLIC_KEY")
+        self.base_url = os.environ.get("LANGFUSE_BASE_URL", "https://cloud.langfuse.com").rstrip("/")
+        
+        if not (self.secret_key and self.public_key):
+            print("⚠ Langfuse credentials missing. Scoring disabled.")
+            self.enabled = False
+            return
+            
+        self.enabled = True
+        auth_str = f"{self.public_key}:{self.secret_key}"
+        auth_bytes = auth_str.encode("ascii")
+        base64_auth = base64.b64encode(auth_bytes).decode("ascii")
+        self.headers = {
+            "Authorization": f"Basic {base64_auth}",
+            "Content-Type": "application/json"
+        }
+
+    def score_trace(self, trace_id: str, name: str, value: float, comment: str = None):
+        if not self.enabled:
+            return
+            
+        url = f"{self.base_url}/api/public/scores"
+        payload = {
+            "traceId": trace_id,
+            "name": name,
+            "value": value,
+            "comment": comment
+        }
+        
+        try:
+            # Synchronous call for simplicity in this script
+            resp = httpx.post(url, json=payload, headers=self.headers, timeout=10.0)
+            if resp.status_code not in (200, 201):
+                print(f"   ⚠ Failed to log score {name}: {resp.status_code} - {resp.text}")
+        except Exception as e:
+            print(f"   ⚠ Error logging score: {e}")
+
+async def evaluate_helpfulness(question: str, answer: str) -> dict:
+    """Evaluates if the answer is helpful using LLM."""
+    prompt = f"""
+    You are an expert evaluator. Rate the helpfulness of the AI response to the user question.
+    
+    Question: {question}
+    Response: {answer}
+    
+    Score 0.0 to 1.0 (1.0 is most helpful).
+    Provide reasoning.
+    
+    Output JSON: {{"score": float, "reason": "string"}}
+    """
+    try:
+        response = await aclient.chat.completions.create(
+            model="gpt-4o",
+            messages=[{"role": "user", "content": prompt}],
+            response_format={"type": "json_object"},
+            temperature=0
+        )
+        return json.loads(response.choices[0].message.content)
+    except Exception as e:
+        print(f"   ⚠ Eval Error: {e}")
+        return {"score": 0.0, "reason": "Evaluation failed"}
+
+async def evaluate_faithfulness(question: str, answer: str, context: str) -> dict:
+    """Evaluates if the answer is faithful to the context."""
+    if not context:
+        return {"score": 0.5, "reason": "No context available for faithfulness check."}
+        
+    prompt = f"""
+    You are an expert evaluator. Rate the faithfulness of the AI response to the retrieved context.
+    
+    Context:
+    {context[:10000]}... (truncated)
+
+    Question: {question}
+    Response: {answer}
+    
+    Score 0.0 to 1.0 (1.0 is fully supported by context).
+    If the response contains information NOT in the context (hallucination), penalize heavily.
+    
+    Output JSON: {{"score": float, "reason": "string"}}
+    """
+    try:
+        response = await aclient.chat.completions.create(
+            model="gpt-4o",
+            messages=[{"role": "user", "content": prompt}],
+            response_format={"type": "json_object"},
+            temperature=0
+        )
+        return json.loads(response.choices[0].message.content)
+    except Exception as e:
+        print(f"   ⚠ Faithfulness Eval Error: {e}")
+        return {"score": 0.0, "reason": "Evaluation failed"}
+
+async def evaluate_trajectory(question: str, tool_calls: List[str], rubric: str) -> dict:
+    """Evaluates if the tool usage followed the rubric."""
+    prompt = f"""
+    You are an expert evaluator. Rate the agent's execution trajectory against the rubric.
+    
+    Rubric: {rubric}
+    
+    Question: {question}
+    Tool Sequence: {json.dumps(tool_calls)}
+    
+    Score 0.0 to 1.0 (1.0 is perfect adherence).
+    Did it skip required steps? Did it use irrelevant tools?
+    
+    Output JSON: {{"score": float, "reason": "string"}}
+    """
+    try:
+        response = await aclient.chat.completions.create(
+            model="gpt-4o",
+            messages=[{"role": "user", "content": prompt}],
+            response_format={"type": "json_object"},
+            temperature=0
+        )
+        return json.loads(response.choices[0].message.content)
+    except Exception as e:
+        print(f"   ⚠ Trajectory Eval Error: {e}")
+        return {"score": 0.0, "reason": "Evaluation failed"}
+
+def extract_context_and_tools(agent_result) -> tuple[str, List[str]]:
+    """Extracts retrieved text and tool names from AgentResult."""
+    context = []
+    tool_calls = []
+    
+    if not hasattr(agent_result, 'trace') or not agent_result.trace:
+        return "", []
+
+    for span in agent_result.trace.spans:
+        # Check for tool execution spans (simplified check)
+        if hasattr(span, 'span_type') and str(span.span_type) == 'tool_execution':
+             # Tool Name
+             tool_name = span.tool_call.name
+             tool_calls.append(tool_name)
+             
+             # Context from Search/Load Tools
+             if 'confluence' in tool_name or 'get_application_summary' in tool_name or 'compare' in tool_name:
+                 context.append(f"Source ({tool_name}): {span.tool_result.content}")
+
+    return "\n\n".join(context), tool_calls
+
+async def run_evaluation():
+    print("🚀 Starting Evaluation Pipeline (Custom LLM Evals: Helpfulness, Faithfulness, Trajectory)...")
+    
+    # Initialize Client
+    lf_client = LangfuseClient()
+    
+    # Load dataset
+    try:
+        with open("evaluation/dataset.json", "r") as f:
+            dataset = json.load(f)
+    except FileNotFoundError:
+        print("❌ evaluation/dataset.json not found.")
+        return
+    
+    print(f"📋 Loaded {len(dataset)} test cases.")
+
+    results = []
+
+    for case in dataset:
+        case_id = case["id"]
+        question = case["question"]
+        expected_key_points = case["expected_answer_key_points"]
+        
+        print(f"\n🧪 Running Case: {case_id}")
+        
+        # Start a span for this evaluation case
+        with tracer.start_as_current_span(f"Eval: {case_id}") as span:
+            # Get Trace ID (OTel stores it as int, needs hex conversion)
+            trace_id_int = span.get_span_context().trace_id
+            trace_id_hex = "{:032x}".format(trace_id_int)
+            
+            # Add metadata
+            span.set_attribute("evaluation.case_id", case_id)
+            span.set_attribute("evaluation.question", question)
+            
+            # 1. Run Agent (Request Full Result)
+            # The agent's internal spans will be nested under this span automatically
+            result_obj = query_agent(question, return_full_result=True)
+            answer = str(result_obj)
+            
+            # Extract Internals
+            context_text, tool_sequence = extract_context_and_tools(result_obj)
+            # Log extracted data for debug
+            span.set_attribute("evaluation.tool_sequence", json.dumps(tool_sequence))
+
+            # 2. Run Evaluators & Log Scores
+            
+            # A. Helpfulness
+            help_res = await evaluate_helpfulness(question, answer)
+            lf_client.score_trace(trace_id=trace_id_hex, name="Helpfulness", value=help_res["score"], comment=help_res["reason"])
+            print(f"   ✅ Helpfulness: {help_res['score']:.2f}")
+
+            # B. Faithfulness
+            faith_res = await evaluate_faithfulness(question, answer, context_text)
+            lf_client.score_trace(trace_id=trace_id_hex, name="Faithfulness", value=faith_res["score"], comment=faith_res["reason"])
+            print(f"   📖 Faithfulness: {faith_res['score']:.2f}")
+
+            # C. Trajectory
+            rubric = "1. Retrieve data (summary). 2. Analyze specifics/compare. 3. Check compliance if relevant. 4. Explain."
+            traj_res = await evaluate_trajectory(question, tool_sequence, rubric)
+            lf_client.score_trace(trace_id=trace_id_hex, name="Trajectory", value=traj_res["score"], comment=traj_res["reason"])
+            print(f"   👣 Trajectory:  {traj_res['score']:.2f} ({len(tool_sequence)} tools)")
+
+            # D. Goal Success
+            hits = 0
+            if expected_key_points:
+                for point in expected_key_points:
+                    if point.lower() in answer.lower():
+                        hits += 1
+                    elif any(word in answer.lower() for word in point.split() if len(word) > 4):
+                         hits += 0.5 
+                success_rate = min(1.0, hits / len(expected_key_points))
+            else:
+                success_rate = 1.0
+            
+            lf_client.score_trace(trace_id=trace_id_hex, name="Goal Success", value=success_rate, comment=f"Matched {hits}/{len(expected_key_points)} key points")
+            print(f"   🎯 Goal Success: {success_rate:.2f}")
+
+            results.append({
+                "case_id": case_id,
+                "trace_id": trace_id_hex,
+                "helpfulness": help_res["score"],
+                "faithfulness": faith_res["score"],
+                "trajectory": traj_res["score"],
+                "goal_success": success_rate
+            })
+            
+    # Summary
+    print("\n📊 Evaluation Summary")
+    print(json.dumps(results, indent=2))
+
+if __name__ == "__main__":
+    asyncio.run(run_evaluation())

evaluation/dataset.json

@@ -1,314 +0,0 @@
-[
-  {
-    "id": "case_1",
-    "question": "Why was application APP-78432 flagged as high risk?",
-    "expected_intent": "Analyze why application APP-78432 was flagged",
-    "expected_answer_key_points": [
-      "Fraud Score: 449",
-      "Risk Level: MEDIUM",
-      "Model: XGBoost Fraud Ensemble v3.2",
-      "Decision: APPROVED"
-    ]
-  },
-  {
-    "id": "case_2",
-    "question": "Check fair lending compliance for APP-55555",
-    "expected_intent": "Check fair lending compliance",
-    "expected_answer_key_points": [
-      "No geographic, name-based, or age-related discrimination",
-      "Adverse Impact Ratio: 0.94",
-      "Status: COMPLIANT"
-    ]
-  },
-  {
-    "id": "case_3",
-    "question": "Explain the fraud score for APP-12345 and compare it to approved applications",
-    "expected_intent": "Explain fraud score and compare to population",
-    "expected_answer_key_points": [
-      "Fraud Score: 850",
-      "Risk Level: HIGH",
-      "Suspicious Pattern: High velocity of applications",
-      "Identity verification usage"
-    ]
-  },
-  {
-    "id": "synth_4_synth_high_risk_1",
-    "question": "Why is this account flagged as high risk for synthetic identity fraud?",
-    "expected_intent": "Understand the reasons behind the high-risk classification.",
-    "expected_answer_key_points": [
-      "Unusual account activity patterns",
-      "Mismatch between identity attributes",
-      "Use of known fraudulent identifiers"
-    ]
-  },
-  {
-    "id": "synth_5_synth_high_risk_2",
-    "question": "What specific behaviors in the transaction history indicate synthetic identity fraud?",
-    "expected_intent": "Identify behaviors that suggest synthetic identity fraud.",
-    "expected_answer_key_points": [
-      "Multiple transactions with inconsistent locations",
-      "Rapid account activity after dormancy",
-      "Attempts to access high-value services"
-    ]
-  },
-  {
-    "id": "synth_6_synth_high_risk_3",
-    "question": "How does the model differentiate between synthetic identities and legitimate customers?",
-    "expected_intent": "Understand the model's methodology for distinguishing between synthetic and legitimate identities.",
-    "expected_answer_key_points": [
-      "Advanced pattern recognition algorithms",
-      "Comparison against known legitimate customer profiles",
-      "Analysis of identity verification documentation"
-    ]
-  },
-  {
-    "id": "synth_7_synth_high_risk_4",
-    "question": "What are the common data points used by the model to detect synthetic identities?",
-    "expected_intent": "Identify key data points used by the model for detection.",
-    "expected_answer_key_points": [
-      "Social security number validation",
-      "Cross-referencing with public records",
-      "Analysis of digital footprint and device information"
-    ]
-  },
-  {
-    "id": "synth_8_low_risk_fp_1",
-    "question": "Why was this transaction flagged as fraudulent despite being low risk?",
-    "expected_intent": "Understand the reasons behind a false positive classification.",
-    "expected_answer_key_points": [
-      "Unusual transaction pattern",
-      "Customer purchase history",
-      "Changes in location or device used"
-    ]
-  },
-  {
-    "id": "synth_9_low_risk_fp_2",
-    "question": "Can you explain why this account's activity is considered suspicious?",
-    "expected_intent": "Gain insights into the factors influencing the fraud model's decision.",
-    "expected_answer_key_points": [
-      "High volume of transactions",
-      "Transaction value deviations",
-      "Comparison with typical user behavior"
-    ]
-  },
-  {
-    "id": "synth_10_low_risk_fp_3",
-    "question": "What factors led to this low-risk transaction being marked as fraud?",
-    "expected_intent": "Identify specific elements that triggered the fraud alert.",
-    "expected_answer_key_points": [
-      "Mismatch in expected transaction time",
-      "Discrepancy in user verification",
-      "Recent account changes or updates"
-    ]
-  },
-  {
-    "id": "synth_11_low_risk_fp_4",
-    "question": "Why did our system mistakenly flag this legitimate transaction?",
-    "expected_intent": "Determine the cause of the system error leading to a false positive.",
-    "expected_answer_key_points": [
-      "Error in model threshold setting",
-      "Anomaly detection misclassification",
-      "Lack of data on new customer behavior"
-    ]
-  },
-  {
-    "id": "synth_12_synth_borderline_case_1",
-    "question": "Why is this transaction flagged as suspicious when the customer has a long history of legitimate purchases?",
-    "expected_intent": "Understand mixed signals causing a fraud alert.",
-    "expected_answer_key_points": [
-      "Analyzing recent transaction behavior",
-      "Comparing with customer's purchase history",
-      "Highlighting unusual transaction patterns"
-    ]
-  },
-  {
-    "id": "synth_13_synth_borderline_case_2",
-    "question": "Can you explain why this low amount transaction is considered high-risk while the customer has been verified recently?",
-    "expected_intent": "Clarify factors contributing to risk evaluation.",
-    "expected_answer_key_points": [
-      "Risk assessment includes transaction context",
-      "Verification status vs. transaction attributes",
-      "Analysis of current vs. past behavior"
-    ]
-  },
-  {
-    "id": "synth_14_synth_borderline_case_3",
-    "question": "This merchant is reputable, so why are some of their transactions flagged for potential fraud?",
-    "expected_intent": "Investigate reasons behind inconsistent fraud signals.",
-    "expected_answer_key_points": [
-      "Merchant transaction volume vs. individual transactions",
-      "Potential changes in merchant activities",
-      "Flagged patterns specific to current transactions"
-    ]
-  },
-  {
-    "id": "synth_15_synth_borderline_case_4",
-    "question": "Why is this account flagged when the user is known for frequent travel and varied spending patterns?",
-    "expected_intent": "Identify causes of false positives in fraud detection.",
-    "expected_answer_key_points": [
-      "Account flagged due to recent activity anomalies",
-      "Analysis of travel-related spending patterns",
-      "Consideration of geographic and spending behavior"
-    ]
-  },
-  {
-    "id": "synth_16_fair_lending_compliance_1",
-    "question": "Why was the loan application of a minority applicant flagged as high risk by the model?",
-    "expected_intent": "Understand the factors leading to the high-risk categorization for minority applicants.",
-    "expected_answer_key_points": [
-      "Model's risk assessment criteria",
-      "Applicant's credit history and income",
-      "Any bias or unfair treatment in the model"
-    ]
-  },
-  {
-    "id": "synth_17_fair_lending_compliance_2",
-    "question": "Can you explain if the model's decision impacts applicants from low-income neighborhoods differently?",
-    "expected_intent": "Determine if there is any disparate impact based on the applicant's neighborhood.",
-    "expected_answer_key_points": [
-      "Impact of location on risk scoring",
-      "Comparison of approval rates by neighborhood",
-      "Fair lending compliance measures in place"
-    ]
-  },
-  {
-    "id": "synth_18_fair_lending_compliance_3",
-    "question": "What measures are in place to ensure that the model's decisions comply with fair lending regulations?",
-    "expected_intent": "Identify compliance protocols for fair lending regulations within the model.",
-    "expected_answer_key_points": [
-      "Regular bias audits",
-      "Use of fair lending algorithms",
-      "Continuous monitoring for compliance"
-    ]
-  },
-  {
-    "id": "synth_19_fair_lending_compliance_4",
-    "question": "How does the model ensure equal treatment for applicants with similar financial profiles but from different demographic groups?",
-    "expected_intent": "Verify the model's fairness and equality in decision-making across demographics.",
-    "expected_answer_key_points": [
-      "Demographic parity in decision-making",
-      "Comparative analysis of similar profiles",
-      "Adjustments made to mitigate bias"
-    ]
-  },
-  {
-    "id": "synth_20_synth_high_risk_1",
-    "question": "Why is the model flagging an unusually high number of transactions as high-risk this month?",
-    "expected_intent": "Understand the reason behind the spike in high-risk transaction flags.",
-    "expected_answer_key_points": [
-      "Change in transaction patterns",
-      "Adjustment in model thresholds",
-      "New data inputs affecting model behavior"
-    ]
-  },
-  {
-    "id": "synth_21_synth_low_accuracy_2",
-    "question": "What could be causing the recent drop in model accuracy for identifying fraudulent transactions?",
-    "expected_intent": "Identify factors leading to decreased model accuracy.",
-    "expected_answer_key_points": [
-      "Data drift or changes in data distribution",
-      "Outdated model parameters",
-      "Need for model retraining"
-    ]
-  },
-  {
-    "id": "synth_22_synth_segment_discrepancy_3",
-    "question": "Why is the model performing poorly on certain customer segments?",
-    "expected_intent": "Explain performance discrepancies across customer segments.",
-    "expected_answer_key_points": [
-      "Segment-specific behavior not well-captured",
-      "Imbalanced training data for segments",
-      "Potential need for segment-specific features"
-    ]
-  },
-  {
-    "id": "synth_23_synth_feature_importance_4",
-    "question": "What are the most influential features the model uses to determine fraud risk?",
-    "expected_intent": "Identify key features driving model predictions.",
-    "expected_answer_key_points": [
-      "List of top contributing features",
-      "Feature importance ranking",
-      "Impact of each feature on the model's decision"
-    ]
-  },
-  {
-    "id": "synth_sdk_1",
-    "question": "Application ID: 1023456",
-    "expected_intent": "General",
-    "expected_answer_key_points": [
-      "Fraud Score: 750\nRisk Level: High\nSHAP Values:\n- Transaction History Impact: +200\n- Credit Score Impact: +150\n- Recent Loan Applications Impact: -50\n- Employment History Impact: +100\nExplanation: The application has a high fraud score primarily due to the transaction history and credit score, which together contribute significantly to the high risk level. The recent loan applications positively influence the score by lowering it slightly, but not enough to offset the other risk factors. Employment history further increases the score. It's essential to focus on improving transaction patterns to lower the fraud score."
-    ]
-  },
-  {
-    "id": "synth_sdk_2",
-    "question": "Application ID: 4321XYZ\n- Request from Fraud Analyst: \"Please provide a detailed breakdown of the fraud score for this application. Include key feature contributions and provide an explanation for any features that heavily influence the score.\"\n- Tools and Capabilities Requested:\n  - use explain_fraud_score(4321XYZ) to highlight SHAP values.\n  - identify high-contributing features impacting fraud score.\n  - analyze borderline scores and explain if any apply.",
-    "expected_intent": "General",
-    "expected_answer_key_points": [
-      "Fraud Score: 680 (Moderate Risk)\nKey SHAP Contribution Analysis:\n1. \"Credit History Length\": +45 (Positive impact due to long credit history)\n2. \"Number of Recent Applications\": +100 (Negative impact due to recent application spike)\n3. \"Unusual Geolocation Activity\": +150 (High negative impact due to recent activity in a high-risk zone)\n4. \"Income Verification\": -65 (Positive impact due to verified and stable income)\n\nExplanation:\n- The fraud score is boosted to a higher risk mainly due to the unusual geolocation activity and the number of recent applications, which are significant red flags in fraud detection. While the credit history length and verified income provide a buffer reducing the risk level somewhat, the overall risk remains moderate.\n- Due to the combination and magnitude of these factors, the borderline score reflects an increased risk with possibilities of both legitimate and fraudulent activities present."
-    ]
-  },
-  {
-    "id": "synth_sdk_3",
-    "question": "Task: Analyze the performance of the fraud detection model over the past six months.\n\nData Provided: \n- List of application IDs from the last six months\n- Access to tools: compare_to_population, explain_fraud_score\n\nKey Question: What trends can be identified in the distribution of fraud scores and SHAP value importance over this period?",
-    "expected_intent": "General",
-    "expected_answer_key_points": [
-      "Output:\n1. Statistical report indicating any significant trends or shifts in fraud scores over the six-month period using compare_to_population.\n2. Identification of any changes in SHAP feature importance that may indicate shifts in model predictions.\n3. Summary of observations highlighting potential model drift or changes in population characteristics that could impact fraud detection accuracy."
-    ]
-  },
-  {
-    "id": "synth_sdk_4",
-    "question": "A data scientist is analyzing the fraud detection model's performance over the past six months for applications in a specific region. They want to understand how the model's scoring distribution has shifted over this period and identify any significant changes in feature importance that might correspond with this shift. Use the tools compare_to_population and explain_fraud_score. The focus is on applications in the Midwest region, with application IDs ranging from 1001 to 1050. The model should track score distribution shifts and highlight any significant changes in SHAP feature importance.",
-    "expected_intent": "General",
-    "expected_answer_key_points": [
-      "1. Provide a trend report showing score distribution changes over the past six months for the specified application range in the Midwest, indicating whether fraud scores have increased, decreased, or remained stable.\n2. Use compare_to_population to determine how these changes compare to national trends, noting any significant deviations.\n3. Highlight significant shifts in SHAP value contributions for key features over the same period, indicating which features have become more or less significant in model decision-making.\n4. Correlate findings with any historical applications data to suggest possible reasons for observed shifts, such as policy changes or external economic factors."
-    ]
-  },
-  {
-    "id": "synth_sdk_5",
-    "question": "The Compliance Officer wants to verify if there is any evidence of gender bias in fraud scoring for the application ID: F12345. Use the 'check_fair_lending_flags' to detect bias indicators and 'compare_to_population' to analyze demographic-specific fraud score patterns.",
-    "expected_intent": "General",
-    "expected_answer_key_points": [
-      "The 'check_fair_lending_flags' tool indicates a potential bias against female applicants due to disparate impact ratios below threshold in the fraud scoring process, indicating females are disproportionately denied compared to males. 'compare_to_population' further shows that female applicants have consistently higher average fraud scores compared to male counterparts for similar profiles, suggesting an underlying model bias."
-    ]
-  },
-  {
-    "id": "synth_sdk_6",
-    "question": "compliance_officer: \"I need to verify if application 1234 has any fair lending bias issues. Check for any bias flags and provide basic SHAP contributions for this case.\"",
-    "expected_intent": "General",
-    "expected_answer_key_points": [
-      "1. Use check_fair_lending_flags(app_id=1234) to identify if there are any bias issues. The tool will return 'No Bias Detected' or 'Potential Bias Detected'.\n2. If bias is present, explain_fraud_score(app_id=1234) to understand which features contribute most to the score.\n3. Report: 'No Bias Detected. Main factors influencing the fraud score are {feature1, feature2} contributing {value1, value2}.' OR 'Potential Bias Detected. SHAP analysis reveals major contributors: {biased_feature} with {value}.'"
-    ]
-  },
-  {
-    "id": "synth_sdk_7",
-    "question": "You are tasked with preparing an executive summary that highlights the current trends in fraud detection for the past quarter for the executive team of the financial services company. Use the following application IDs: ['A12345', 'B67890', 'C13579'].",
-    "expected_intent": "General",
-    "expected_answer_key_points": [
-      "The executive summary should include:\n1. A concise summary of the fraud scores and risk levels for each application using 'get_application_summary' for the IDs 'A12345', 'B67890', 'C13579'.\n2. A high-level analysis showing any evident trends in fraud scores over the quarter period.\n3. A brief mention of major contributing factors to the high fraud scores, utilizing SHAP values (not the specifics, just an overview of the top features contributing).\n4. A concise explanation of the application network status for each of these IDs using 'get_identity_network'.\n\nThe summary should be formatted in a report-friendly manner, prioritizing clarity and brevity for executive review."
-    ]
-  },
-  {
-    "id": "synth_sdk_8",
-    "question": "Generate a high-level summary for executives on the fraud score trends and key contributing factors for applications over the last quarter. Include how applications are linked within the identity network for potential collusion. Use the following application IDs: ['A123', 'B456', 'C789'].",
-    "expected_intent": "General",
-    "expected_answer_key_points": [
-      "The analysis for the last quarter indicates a rising trend in fraud scores, with the average score increasing by approximately 15% compared to the previous period. Key contributing factors identified through SHAP analysis include unusually high transaction volumes and discrepancies in reported income. Specific to applications ['A123', 'B456', 'C789'], notable features contributing to high fraud scores were similar across the board, indicating potential systemic fraud patterns. Moreover, the get_identity_network for these applications reveals interconnected relationships, suggesting possible collusion attempts. The network analysis indicates that several other applications are linked through shared contact information and transaction patterns, necessitating further investigation into these connections. Overall, the data suggests an evolving strategy that involves increasing sophistication in application profiles that require continued vigilance and adaptation of fraud detection measures."
-    ]
-  },
-  {
-    "id": "synth_sdk_9",
-    "question": "Application ID: 12345; Requested Action: Analyze impact of linked applications on fraud score using get_identity_network.",
-    "expected_intent": "General",
-    "expected_answer_key_points": [
-      "get_identity_network(12345) returns {\"linked_applications\": [\"11111\", \"22222\", \"33333\"]}. explain_fraud_score(12345) returns {\"SHAP_values\": {\"income\": -100, \"credit_history\": 50, \"linked_applications\": 200}}. compare_to_population(12345) shows linked applications influence fraud score to be 20% higher than similar approved applications. Risk level classified as 'Moderate Risk' due to negative impact from linked applications."
-    ]
-  },
-  {
-    "id": "synth_sdk_10",
-    "question": "Application ID: 12345\nTask: Investigate the fraud score and impact of linked applications using the get_identity_network tool. Provide an analysis of how these linked applications influence the fraud score and SHAP values for Application ID 12345. Use compare_to_population to understand how these impacts differ from the general population.",
-    "expected_intent": "General",
-    "expected_answer_key_points": [
-      "1. Identity Network Output: Application 12345 is linked with applications 56789, 98765, and 65432.\n2. Fraud Score Analysis: The fraud score for Application 12345 is 850. Linked applications have high fraud scores: Application 56789 (900), Application 98765 (875), and Application 65432 (910).\n3. SHAP Values: Key features that increased the fraud score include 'cross-application suspicious transactions' and 'irregular account activities.'\n4. Population Comparison: Compared to the approved applications' average fraud score of 500, Application 12345 and its linked applications show significantly higher risk, suggesting a potential fraud ring.\n5. Conclusion: The linkage to high-risk applications within the identity network explains the elevated fraud score for Application 12345, highlighting the influence of network associations in detecting fraud patterns."
-    ]
-  }
-]

requirements.txt

@@ -12,7 +12,6 @@ gradio>=4.0.0
 
 # HTTP client
 httpx>=0.24.0
-opentelemetry-exporter-otlp>=1.39.1
 
 # Environment variable management
 python-dotenv>=1.0.0