Hugging Face's logo
Activity Feed

AI & ML interests

AI Model Benchmarking, LLM Evaluation, Model Drift Analysis, Leaderboards, AI Comparison Tools

Organization Card
Community About org cards

AI Stupid Level - Real-Time AI Benchmarking Platform

Website GitHub Twitter

A real-time AI benchmarking platform that measures and compares the intelligence, efficiency, and consistency of today's top AI models.

Website: aistupidlevel.info
Organization: Studio Platforms (Romania)
Focus: LLM Evaluation • Model Drift Analysis • Real-Time Benchmarks • AI Transparency

What We Do

AI Stupid Level evaluates the world's leading AI models across 9 Dimensions of AI Performance:

  • Correctness - Functional accuracy through automated testing
  • Complexity - Ability to handle sophisticated problems
  • Code Quality - Clean, maintainable, well-structured code
  • Efficiency - Algorithmic optimization and performance
  • Stability - Consistency across multiple test runs
  • Refusal Handling - Appropriate task acceptance vs over-cautious rejections
  • Recovery - Error detection and self-correction capabilities
  • Format - Proper output structure and presentation
  • Safety - Secure coding practices and vulnerability awareness

Currently benchmarking: OpenAI (GPT-4, GPT-5, O3), Anthropic (Claude Opus 4, Sonnet 4), Google (Gemini 2.5 Pro/Flash), xAI (Grok 2/4), and more.

How Our Benchmarking System Works

Multi-Layered Testing Architecture

We operate four distinct benchmark suites that provide comprehensive AI model evaluation:

1. Real-Time Benchmarks (Every 4 Hours)

  • 147 unique coding challenges across multiple programming languages
  • 5 independent trials per task with different random seeds
  • 9-axis scoring system with weighted evaluation
  • 20-30 minutes execution time per model
  • Primary ranking source for our leaderboards

2. Canary Drift Detection (Hourly)

  • 12 carefully selected tasks for rapid change detection
  • 2 trials per task for fast execution (2-3 minutes)
  • Safety sentinel tests to monitor policy changes
  • 4x faster detection of model degradation than full benchmarks
  • Drift alerts when performance drops >10%

3. Deep Reasoning Tests (Daily)

  • Complex multi-step problems requiring sustained reasoning
  • Memory retention and context management evaluation
  • Long-form analysis and problem decomposition
  • 60-90 minutes per model for comprehensive assessment

4. Tool Calling Benchmarks (Daily)

  • 15+ real-world scenarios testing agent capabilities
  • File operations, system interaction, web scraping
  • Multi-step workflows with tool chaining
  • Sandbox execution for safe testing environment

Statistical Rigor and Methodology

Our benchmarking system implements production-grade statistical analysis:

Confidence Intervals

  • T-distribution analysis for small sample sizes (n=5)
  • 95% confidence intervals on all scores
  • Standard error calculation for uncertainty quantification
  • Per-model variance calibration for fair comparison

Example Score Generation Process

Task: "Implement binary search in Python"

Trial Results:
- Trial 1 (seed=42): Score = 23.5
- Trial 2 (seed=43): Score = 25.1  
- Trial 3 (seed=44): Score = 24.2
- Trial 4 (seed=45): Score = 26.3
- Trial 5 (seed=46): Score = 24.8

Statistical Analysis:
- Mean = 24.78
- Standard Deviation = 1.02
- Standard Error = 0.46
- t-value = 2.776 (df=4, 95% CI)
- Margin of Error = 1.27

Final Result: 24.8 ± 1.3 (95% CI: 23.5-26.1)

Drift Detection Algorithm

  • CUSUM (Cumulative Sum) for sustained change detection
  • Model-specific thresholds based on historical variance
  • Multi-signal verification across different test suites
  • False positive reduction through statistical significance testing

Smart AI Router

Intelligent Model Selection System

Our Smart AI Router automatically selects the best-performing AI model for each request based on real-time benchmark data, saving users money and improving performance.

Key Features

  • OpenAI-compatible API at https://aistupidlevel.info/v1
  • Universal API keys (format: aism_xxxxx)
  • 6 routing strategies for different use cases
  • Automatic failover when models are unavailable
  • Cost optimization with performance tracking
  • AES-256 encryption for all provider keys

Routing Strategies

  1. auto - Best overall performance across all categories
  2. auto-coding - Optimized for programming and development tasks
  3. auto-reasoning - Best for complex problem-solving and analysis
  4. auto-creative - Optimized for creative writing and content generation
  5. auto-cheapest - Most cost-effective option maintaining quality
  6. auto-fastest - Lowest latency for time-sensitive applications

Usage Example

# Standard OpenAI-compatible request
curl -X POST https://aistupidlevel.info/v1/chat/completions \
  -H "Authorization: Bearer aism_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto-coding",
    "messages": [
      {"role": "user", "content": "Write a binary search function in Python"}
    ]
  }'

The router automatically:

  1. Analyzes your request type (coding task)
  2. Checks current model rankings for coding performance
  3. Selects the best available model from your configured providers
  4. Routes the request and returns the response
  5. Logs usage and cost data for analytics

Model Comparison Features

Claude vs GPT vs Gemini Analysis

Our platform provides detailed comparative analysis of major AI models:

Current Performance Leaders (2025)

  • Best for Coding: Claude Opus 4 consistently leads in code generation and debugging
  • Fastest Response: Gemini 2.5 Flash provides quickest API response times
  • Most Reliable: GPT-5 shows most consistent performance across categories
  • Best Value: Claude Sonnet 4 offers excellent performance-to-cost ratio

Comprehensive Comparison Matrix

  • Real-time performance data updated every 4 hours
  • Statistical significance testing for meaningful comparisons
  • Cost-per-token analysis across all providers
  • Latency benchmarks for response time optimization
  • Capability matrices showing strengths and weaknesses

Drift Detection and Monitoring

Production-Grade Change Detection

Our drift detection system implements recommendations from industry experts for catching model changes in production environments:

Multi-Layer Detection

  • Hourly canary tests for rapid change detection
  • Safety sentinel tasks monitoring policy boundaries
  • Statistical significance testing to avoid false positives
  • Vendor metadata tracking for correlation analysis

Alert System

  • Performance degradation alerts when scores drop >10%
  • Safety policy changes when refusal patterns shift
  • API reliability monitoring for uptime and latency
  • Cost optimization alerts when pricing changes

Technical Architecture

Backend Infrastructure

  • Language: TypeScript with Node.js
  • Framework: Fastify for high-performance APIs
  • Database: SQLite with Drizzle ORM for type safety
  • Caching: Redis for fast data access
  • Security: AES-256-GCM encryption, OAuth authentication

Frontend Platform

  • Framework: Next.js 14 with React
  • Styling: Tailwind CSS with custom vintage theme
  • Charts: Real-time data visualization
  • Authentication: NextAuth.js with multiple providers

Infrastructure

  • Deployment: PM2 process management
  • Reverse Proxy: Nginx with SSL termination
  • Monitoring: Custom health checks and alerting
  • Scaling: Horizontal scaling with load balancing

API Access and Integration

REST API Endpoints

Model Rankings

GET https://aistupidlevel.info/api/dashboard
# Returns current model rankings with confidence intervals

Historical Data

GET https://aistupidlevel.info/api/dashboard?period=7d
# Returns time series data with statistical analysis

Model Details

GET https://aistupidlevel.info/api/models/{model_id}
# Returns detailed performance breakdown by task category

Smart Router API

Chat Completions

POST https://aistupidlevel.info/v1/chat/completions
Authorization: Bearer aism_your_key

{
  "model": "auto-coding",
  "messages": [...],
  "temperature": 0.7
}

Available Models

GET https://aistupidlevel.info/v1/models
Authorization: Bearer aism_your_key

Use Cases and Applications

For Individual Developers

  • Cost optimization by automatically using the most efficient model
  • Performance maximization with task-specific model selection
  • Reliability improvement through automatic failover
  • Time savings with unified API access

For Development Teams

  • Centralized model management across projects
  • Usage analytics and cost tracking
  • Team-wide routing preferences and policies
  • Integration with existing tools (Cline, Continue, Cursor)

For Researchers and Academics

  • Transparent methodology with open-source code
  • Statistical rigor with confidence intervals
  • Reproducible results through standardized testing
  • Historical data access for trend analysis

For AI Tool Developers

  • Drop-in OpenAI replacement with intelligent routing
  • Enhanced user experience through automatic optimization
  • Reduced vendor lock-in with multi-provider support
  • Built-in analytics for usage insights

Transparency and Open Source

Methodology Transparency

  • Complete source code available on GitHub
  • Detailed documentation of all algorithms and processes
  • Statistical methodology peer-reviewed and published
  • "Test Your Keys" feature for independent verification

Community Validation

  • Open benchmark tasks for community review
  • Statistical methodology addressing academic concerns
  • Independent verification through user testing
  • Continuous improvement based on community feedback

Data Access

  • Public API for accessing benchmark results
  • Historical data available for research purposes
  • Real-time updates with confidence intervals
  • Export capabilities for further analysis

Getting Started

For Benchmarking Data

  1. Visit aistupidlevel.info
  2. View real-time model rankings and comparisons
  3. Access historical performance data
  4. Use our API for programmatic access

For Smart Router

  1. Sign up for an account at aistupidlevel.info
  2. Add your provider API keys (OpenAI, Anthropic, etc.)
  3. Generate a universal API key
  4. Replace your OpenAI endpoint with our router
  5. Start saving money with intelligent routing

For Developers

  1. Check our GitHub repository for source code
  2. Review our API documentation
  3. Integrate with your existing tools and workflows
  4. Contribute to our open-source development

Mission and Values

Our Mission

To make AI performance transparent and comparable for every developer and researcher. We believe in open evaluation, statistical rigor, and community-driven improvement of AI benchmarking standards.

Core Values

  • Transparency: All methodologies and code are open source
  • Statistical Rigor: Proper confidence intervals and significance testing
  • Community Focus: Responsive to feedback and continuous improvement
  • Practical Value: Real-world applicability over academic metrics
  • Cost Efficiency: Helping users optimize their AI spending

Contact and Community

Website: aistupidlevel.info
GitHub: Studio Platforms
Twitter/X: @aistupidlevel
Email: Contact through Studio Platforms

AI Stupid Level is an independent benchmarking project built by Ionut Visan and the Studio Platforms ecosystem, dedicated to making AI performance measurement transparent, statistically rigorous, and practically useful for the global developer community.

models 0

None public yet

datasets 0

None public yet