backend/analytics/README.md

# Analytics Module for AegisLM

This module provides comprehensive multi-run comparison and trend analysis capabilities for the AegisLM experiment system.

## Features

### 🔍 Multi-Run Comparison
- **Score Comparison**: Compare robustness, risk, success rates across runs
- **Metric Deltas**: Calculate differences and percentage changes
- **Performance Rankings**: Rank experiments by overall performance
- **Best/Worst Identification**: Automatically identify top and bottom performers

### 📈 Trend Analysis
- **Historical Analysis**: Analyze performance trends over time
- **Improvement Detection**: Identify positive and negative trends
- **Anomaly Detection**: Find outliers and unusual patterns
- **Forecasting**: Simple trend forecasting for future performance
- **Stability Analysis**: Measure consistency and volatility

### 📊 Metrics Aggregation
- **Statistical Summaries**: Mean, median, percentiles, standard deviation
- **Data Quality Assessment**: Evaluate completeness and consistency
- **Group-based Aggregation**: By model, dataset, or time windows
- **Top Performers**: Identify best performing experiments

### 🎨 Visualization Ready Data
- **Chart Formats**: Radar, bar, line, pie charts ready for frontend
- **Time Series**: Historical and forecast data for trend visualization
- **Interactive Charts**: Configurable chart options and styling
- **Export Ready**: JSON format suitable for Chart.js, D3.js, etc.

## API Endpoints

### Comparison Operations

```bash
# Compare multiple runs
POST /api/v1/analytics/compare
{
  "run_ids": ["run1", "run2", "run3"]
}
```

**Response:**
```json
{
  "success": true,
  "data": {
    "best_run": "run3",
    "worst_run": "run1", 
    "rankings": [...],
    "improvement_opportunities": [...],
    "chart_data": {...}
  }
}
```

### Trend Analysis

```bash
# Analyze trends across runs
POST /api/v1/analytics/trend
{
  "run_ids": ["run1", "run2", "run3", "run4", "run5"]
}
```

**Response:**
```json
{
  "success": true,
  "data": {
    "overall_direction": "increasing",
    "overall_health_score": 0.75,
    "key_insights": [...],
    "metric_trends": {...},
    "chart_data": {...}
  }
}
```

### Metrics Aggregation

```bash
# Get aggregated metrics
POST /api/v1/analytics/aggregate
{
  "group_by": "model",
  "limit": 100
}
```

### Top Performers

```bash
# Get top performers by metric
GET /api/v1/analytics/top-performers?metric=robustness_score&top_n=10
```

### Analytics Summary

```bash
# Get comprehensive summary
GET /api/v1/analytics/summary
```

## Usage Examples

### Python Client

```python
import asyncio
from analytics import get_comparison_engine, get_trend_analyzer

async def compare_experiments(run_ids):
    """Compare multiple experiments."""
    engine = await get_comparison_engine()
    result = await engine.compare_runs(run_ids)
    
    print(f"Best run: {result.best_run}")
    print(f"Worst run: {result.worst_run}")
    print(f"Consistency score: {result.consistency_score}")
    
    return result

async def analyze_trends(run_ids):
    """Analyze performance trends."""
    analyzer = await get_trend_analyzer()
    result = await analyzer.analyze_trend(run_ids)
    
    print(f"Overall direction: {result.overall_direction}")
    print(f"Health score: {result.overall_health_score}")
    
    for insight in result.key_insights:
        print(f"Insight: {insight}")
    
    return result
```

### JavaScript/Frontend Integration

```javascript
// Compare runs
const compareRuns = async (runIds) => {
  const response = await fetch('/api/v1/analytics/compare', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ run_ids: runIds })
  });
  
  const result = await response.json();
  
  // Use chart_data for visualization
  const radarData = result.data.chart_data.radar;
  const barData = result.data.chart_data.bars;
  
  return result.data;
};

// Analyze trends
const analyzeTrends = async (runIds) => {
  const response = await fetch('/api/v1/analytics/trend', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ run_ids: runIds })
  });
  
  const result = await response.json();
  
  // Use trend data for charts
  const trendData = result.data.chart_data.trends;
  const healthScore = result.data.overall_health_score;
  
  return result.data;
};
```

## Chart Data Formats

### Radar Chart Data

```json
{
  "type": "radar",
  "data": {
    "labels": ["Robustness", "Low Risk", "Success Rate", "Confidence"],
    "datasets": [{
      "label": "Experiment 1",
      "data": [0.8, 0.7, 0.6, 0.9],
      "backgroundColor": "#1f77b440",
      "borderColor": "#1f77b4"
    }]
  }
}
```

### Line Chart Data (Trends)

```json
{
  "type": "line",
  "data": {
    "datasets": [{
      "label": "Historical",
      "data": [["2024-01-01", 0.5], ["2024-01-02", 0.6]],
      "borderColor": "#1f77b4"
    }, {
      "label": "Forecast",
      "data": [["2024-01-03", 0.65], ["2024-01-04", 0.7]],
      "borderColor": "#ff7f0e",
      "borderDash": [5, 5]
    }]
  }
}
```

### Bar Chart Data (Comparisons)

```json
{
  "type": "bar",
  "data": {
    "labels": ["Exp 1", "Exp 2", "Exp 3"],
    "datasets": [{
      "label": "Robustness Score",
      "data": [0.6, 0.7, 0.8],
      "backgroundColor": "#1f77b480"
    }]
  }
}
```

## Performance Metrics

### Available Metrics

- **robustness_score**: Overall model robustness (0-1, higher is better)
- **risk_score**: Security risk level (0-1, lower is better)  
- **success_rate**: Attack success rate (0-1, lower is better)
- **confidence_score**: Average confidence (0-1, higher is better)
- **hallucination_rate**: Hallucination frequency (0-1, lower is better)
- **toxicity_rate**: Toxic content rate (0-1, lower is better)
- **execution_time_ms**: Execution time in milliseconds (lower is better)

### Trend Directions

- **increasing**: Metric is improving over time
- **decreasing**: Metric is declining over time  
- **stable**: No significant change
- **volatile**: High fluctuation

### Performance Tiers

- **excellent**: Top 25% performance
- **good**: 25-50% performance
- **average**: 50-75% performance
- **poor**: Bottom 25% performance

## Architecture

```
analytics/
├── comparison_engine.py    # Multi-run comparison logic
├── trend_analyzer.py       # Trend analysis and forecasting
├── aggregation_utils.py    # Statistical aggregation
├── analytics_service.py    # Service layer and DB integration
├── visualization_utils.py  # Chart-ready data formatting
└── __init__.py            # Module exports
```

## Integration Points

### Experiment System
- Fetches experiments from `ExperimentStore`
- Works with existing `Experiment` and `ResultSummary` schemas
- Maintains compatibility with current experiment workflow

### Database Integration
- Uses experiment store for data access
- Supports user-based filtering and permissions
- Efficient query handling for large datasets

### API Layer
- RESTful endpoints for all analytics operations
- Comprehensive error handling and validation
- Structured response schemas with proper typing

## Testing

Run comprehensive tests:

```bash
python -m pytest tests/test_analytics.py -v
```

Test coverage includes:
- Comparison engine logic and edge cases
- Trend analysis accuracy and anomaly detection
- Aggregation utilities and statistical calculations
- Error handling and validation
- Integration testing with mock data

## Best Practices

1. **Data Quality**: Ensure experiments have complete result summaries
2. **Sufficient Data**: Use minimum 3 runs for trends, 2 for comparisons
3. **Regular Analysis**: Monitor trends weekly/monthly for best insights
4. **Context Awareness**: Consider model and dataset differences
5. **Validation**: Cross-check insights with domain expertise

## Troubleshooting

### Common Issues

**"Insufficient valid experiments"**
- Ensure experiments are completed with result summaries
- Check run ID format and validity
- Verify experiment store connectivity

**"No trend detected"**
- Increase time span or number of runs
- Check for sufficient data variation
- Verify metric values are not constant

**High volatility in trends**
- Investigate data quality issues
- Check for configuration changes
- Consider filtering outliers

### Performance Optimization

- Use pagination for large datasets
- Cache aggregation results
- Limit time windows for trend analysis
- Pre-compute common aggregations

## Future Enhancements

- Machine learning-based trend prediction
- Advanced anomaly detection algorithms  
- Real-time streaming analytics
- Custom metric definitions
- Automated insight generation
- Integration with external BI tools