evals/README.md

# Evaluation Suite

This directory contains a modular evaluation framework for the Fraud Model Explainability Assistant, built using the Strands Evaluation SDK while maintaining Langfuse + OpenTelemetry integration.

## Structure

```
evals/
├── __init__.py              # Package initialization
├── config.py                # Shared evaluator configurations
├── task_functions.py        # Agent wrapper functions
├── langfuse_reporter.py     # Langfuse integration
├── dataset_loader.py        # JSON to Case conversion
├── generate_experiment.py   # SDK Experiment Generator
├── run_helpfulness.py       # Helpfulness evaluation
├── run_trajectory.py        # Trajectory evaluation
└── run_full_suite.py        # Full evaluation suite
```

## Usage

### Run Individual Evaluations

```bash
# Helpfulness only
python -m evals.run_helpfulness

# Trajectory only
python -m evals.run_trajectory
```

### Run Full Suite

```bash
# All evaluators (Helpfulness, Faithfulness, Trajectory, Goal Success)
python -m evals.run_full_suite
```

### Generate Synthetic Data

```bash
# Generate new test cases using SDK ExperimentGenerator
python -m evals.generate_experiment
```
## Features

- **SDK-Aligned**: Uses `Experiment.run_evaluations()` framework
- **Modular**: Each evaluation type in separate file
- **Langfuse Integration**: Automatic score logging with trace correlation
- **OpenTelemetry**: Full observability stack integration
- **Extensible**: Easy to add new evaluators or metrics

## Evaluators

1. **Helpfulness** (`OutputEvaluator`): Rates answer quality (accuracy, completeness, clarity)
2. **Faithfulness** (`OutputEvaluator`): Verifies answer is grounded in retrieved context
3. **Trajectory** (`TrajectoryEvaluator`): Checks tool usage sequence
4. **Goal Success** (Heuristic): Matches expected key points in answer

## Comparison with `evaluate.py`

The modular suite provides the same functionality as the monolithic `evaluate.py` but with:
- ✅ Better code organization
- ✅ SDK-standard patterns
- ✅ Easier to extend with new evaluators
- ✅ Built-in reporting (`report.run_display()`, `get_summary()`)
- ✅ Maintains Langfuse + OTel integration