evals/README.md

Evaluation Suite

This directory contains a modular evaluation framework for the Fraud Model Explainability Assistant, built using the Strands Evaluation SDK while maintaining Langfuse + OpenTelemetry integration.

Structure

evals/
├── __init__.py              # Package initialization
├── config.py                # Shared evaluator configurations
├── task_functions.py        # Agent wrapper functions
├── langfuse_reporter.py     # Langfuse integration
├── dataset_loader.py        # JSON to Case conversion
├── generate_experiment.py   # SDK Experiment Generator
├── run_helpfulness.py       # Helpfulness evaluation
├── run_trajectory.py        # Trajectory evaluation
└── run_full_suite.py        # Full evaluation suite

Usage

Run Individual Evaluations

# Helpfulness only
python -m evals.run_helpfulness

# Trajectory only
python -m evals.run_trajectory

Run Full Suite

# All evaluators (Helpfulness, Faithfulness, Trajectory, Goal Success)
python -m evals.run_full_suite

Generate Synthetic Data

# Generate new test cases using SDK ExperimentGenerator
python -m evals.generate_experiment

Features

• SDK-Aligned: Uses Experiment.run_evaluations() framework
• Modular: Each evaluation type in separate file
• Langfuse Integration: Automatic score logging with trace correlation
• OpenTelemetry: Full observability stack integration
• Extensible: Easy to add new evaluators or metrics

Evaluators

1. Helpfulness (OutputEvaluator): Rates answer quality (accuracy, completeness, clarity)
2. Faithfulness (OutputEvaluator): Verifies answer is grounded in retrieved context
3. Trajectory (TrajectoryEvaluator): Checks tool usage sequence
4. Goal Success (Heuristic): Matches expected key points in answer

Comparison with evaluate.py

The modular suite provides the same functionality as the monolithic evaluate.py but with:

• ✅ Better code organization
• ✅ SDK-standard patterns
• ✅ Easier to extend with new evaluators
• ✅ Built-in reporting (report.run_display(), get_summary())
• ✅ Maintains Langfuse + OTel integration