Analytics Module for AegisLM
This module provides comprehensive multi-run comparison and trend analysis capabilities for the AegisLM experiment system.
Features
🔍 Multi-Run Comparison
- Score Comparison: Compare robustness, risk, success rates across runs
- Metric Deltas: Calculate differences and percentage changes
- Performance Rankings: Rank experiments by overall performance
- Best/Worst Identification: Automatically identify top and bottom performers
📈 Trend Analysis
- Historical Analysis: Analyze performance trends over time
- Improvement Detection: Identify positive and negative trends
- Anomaly Detection: Find outliers and unusual patterns
- Forecasting: Simple trend forecasting for future performance
- Stability Analysis: Measure consistency and volatility
📊 Metrics Aggregation
- Statistical Summaries: Mean, median, percentiles, standard deviation
- Data Quality Assessment: Evaluate completeness and consistency
- Group-based Aggregation: By model, dataset, or time windows
- Top Performers: Identify best performing experiments
🎨 Visualization Ready Data
- Chart Formats: Radar, bar, line, pie charts ready for frontend
- Time Series: Historical and forecast data for trend visualization
- Interactive Charts: Configurable chart options and styling
- Export Ready: JSON format suitable for Chart.js, D3.js, etc.
API Endpoints
Comparison Operations
# Compare multiple runs
POST /api/v1/analytics/compare
{
"run_ids": ["run1", "run2", "run3"]
}
Response:
{
"success": true,
"data": {
"best_run": "run3",
"worst_run": "run1",
"rankings": [...],
"improvement_opportunities": [...],
"chart_data": {...}
}
}
Trend Analysis
# Analyze trends across runs
POST /api/v1/analytics/trend
{
"run_ids": ["run1", "run2", "run3", "run4", "run5"]
}
Response:
{
"success": true,
"data": {
"overall_direction": "increasing",
"overall_health_score": 0.75,
"key_insights": [...],
"metric_trends": {...},
"chart_data": {...}
}
}
Metrics Aggregation
# Get aggregated metrics
POST /api/v1/analytics/aggregate
{
"group_by": "model",
"limit": 100
}
Top Performers
# Get top performers by metric
GET /api/v1/analytics/top-performers?metric=robustness_score&top_n=10
Analytics Summary
# Get comprehensive summary
GET /api/v1/analytics/summary
Usage Examples
Python Client
import asyncio
from analytics import get_comparison_engine, get_trend_analyzer
async def compare_experiments(run_ids):
"""Compare multiple experiments."""
engine = await get_comparison_engine()
result = await engine.compare_runs(run_ids)
print(f"Best run: {result.best_run}")
print(f"Worst run: {result.worst_run}")
print(f"Consistency score: {result.consistency_score}")
return result
async def analyze_trends(run_ids):
"""Analyze performance trends."""
analyzer = await get_trend_analyzer()
result = await analyzer.analyze_trend(run_ids)
print(f"Overall direction: {result.overall_direction}")
print(f"Health score: {result.overall_health_score}")
for insight in result.key_insights:
print(f"Insight: {insight}")
return result
JavaScript/Frontend Integration
// Compare runs
const compareRuns = async (runIds) => {
const response = await fetch('/api/v1/analytics/compare', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ run_ids: runIds })
});
const result = await response.json();
// Use chart_data for visualization
const radarData = result.data.chart_data.radar;
const barData = result.data.chart_data.bars;
return result.data;
};
// Analyze trends
const analyzeTrends = async (runIds) => {
const response = await fetch('/api/v1/analytics/trend', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ run_ids: runIds })
});
const result = await response.json();
// Use trend data for charts
const trendData = result.data.chart_data.trends;
const healthScore = result.data.overall_health_score;
return result.data;
};
Chart Data Formats
Radar Chart Data
{
"type": "radar",
"data": {
"labels": ["Robustness", "Low Risk", "Success Rate", "Confidence"],
"datasets": [{
"label": "Experiment 1",
"data": [0.8, 0.7, 0.6, 0.9],
"backgroundColor": "#1f77b440",
"borderColor": "#1f77b4"
}]
}
}
Line Chart Data (Trends)
{
"type": "line",
"data": {
"datasets": [{
"label": "Historical",
"data": [["2024-01-01", 0.5], ["2024-01-02", 0.6]],
"borderColor": "#1f77b4"
}, {
"label": "Forecast",
"data": [["2024-01-03", 0.65], ["2024-01-04", 0.7]],
"borderColor": "#ff7f0e",
"borderDash": [5, 5]
}]
}
}
Bar Chart Data (Comparisons)
{
"type": "bar",
"data": {
"labels": ["Exp 1", "Exp 2", "Exp 3"],
"datasets": [{
"label": "Robustness Score",
"data": [0.6, 0.7, 0.8],
"backgroundColor": "#1f77b480"
}]
}
}
Performance Metrics
Available Metrics
- robustness_score: Overall model robustness (0-1, higher is better)
- risk_score: Security risk level (0-1, lower is better)
- success_rate: Attack success rate (0-1, lower is better)
- confidence_score: Average confidence (0-1, higher is better)
- hallucination_rate: Hallucination frequency (0-1, lower is better)
- toxicity_rate: Toxic content rate (0-1, lower is better)
- execution_time_ms: Execution time in milliseconds (lower is better)
Trend Directions
- increasing: Metric is improving over time
- decreasing: Metric is declining over time
- stable: No significant change
- volatile: High fluctuation
Performance Tiers
- excellent: Top 25% performance
- good: 25-50% performance
- average: 50-75% performance
- poor: Bottom 25% performance
Architecture
analytics/
├── comparison_engine.py # Multi-run comparison logic
├── trend_analyzer.py # Trend analysis and forecasting
├── aggregation_utils.py # Statistical aggregation
├── analytics_service.py # Service layer and DB integration
├── visualization_utils.py # Chart-ready data formatting
└── __init__.py # Module exports
Integration Points
Experiment System
- Fetches experiments from
ExperimentStore - Works with existing
ExperimentandResultSummaryschemas - Maintains compatibility with current experiment workflow
Database Integration
- Uses experiment store for data access
- Supports user-based filtering and permissions
- Efficient query handling for large datasets
API Layer
- RESTful endpoints for all analytics operations
- Comprehensive error handling and validation
- Structured response schemas with proper typing
Testing
Run comprehensive tests:
python -m pytest tests/test_analytics.py -v
Test coverage includes:
- Comparison engine logic and edge cases
- Trend analysis accuracy and anomaly detection
- Aggregation utilities and statistical calculations
- Error handling and validation
- Integration testing with mock data
Best Practices
- Data Quality: Ensure experiments have complete result summaries
- Sufficient Data: Use minimum 3 runs for trends, 2 for comparisons
- Regular Analysis: Monitor trends weekly/monthly for best insights
- Context Awareness: Consider model and dataset differences
- Validation: Cross-check insights with domain expertise
Troubleshooting
Common Issues
"Insufficient valid experiments"
- Ensure experiments are completed with result summaries
- Check run ID format and validity
- Verify experiment store connectivity
"No trend detected"
- Increase time span or number of runs
- Check for sufficient data variation
- Verify metric values are not constant
High volatility in trends
- Investigate data quality issues
- Check for configuration changes
- Consider filtering outliers
Performance Optimization
- Use pagination for large datasets
- Cache aggregation results
- Limit time windows for trend analysis
- Pre-compute common aggregations
Future Enhancements
- Machine learning-based trend prediction
- Advanced anomaly detection algorithms
- Real-time streaming analytics
- Custom metric definitions
- Automated insight generation
- Integration with external BI tools