Spaces:
Running
Running
| license: mit | |
| title: AI Stupid Level | |
| sdk: static | |
| colorFrom: green | |
| short_description: Tracking how smart AI really is | |
| colorTo: green | |
| # AI Stupid Level - Real-Time AI Benchmarking Platform | |
| [](https://aistupidlevel.info) | |
| [](https://github.com/StudioPlatforms) | |
| [](https://x.com/aistupidlevel) | |
| **A real-time AI benchmarking platform that measures and compares the intelligence, efficiency, and consistency of today's top AI models.** | |
| **Website:** [aistupidlevel.info](https://aistupidlevel.info) | |
| **Organization:** Studio Platforms (Romania) | |
| **Focus:** LLM Evaluation • Model Drift Analysis • Real-Time Benchmarks • AI Transparency | |
| --- | |
| ## What We Do | |
| AI Stupid Level evaluates the world's leading AI models across **9 Dimensions of AI Performance**: | |
| - **Correctness** - Functional accuracy through automated testing | |
| - **Complexity** - Ability to handle sophisticated problems | |
| - **Code Quality** - Clean, maintainable, well-structured code | |
| - **Efficiency** - Algorithmic optimization and performance | |
| - **Stability** - Consistency across multiple test runs | |
| - **Refusal Handling** - Appropriate task acceptance vs over-cautious rejections | |
| - **Recovery** - Error detection and self-correction capabilities | |
| - **Format** - Proper output structure and presentation | |
| - **Safety** - Secure coding practices and vulnerability awareness | |
| **Currently benchmarking:** OpenAI (GPT-4, GPT-5, O3), Anthropic (Claude Opus 4, Sonnet 4), Google (Gemini 2.5 Pro/Flash), xAI (Grok 2/4), and more. | |
| --- | |
| ## How Our Benchmarking System Works | |
| ### Multi-Layered Testing Architecture | |
| We operate **four distinct benchmark suites** that provide comprehensive AI model evaluation: | |
| #### 1. Real-Time Benchmarks (Every 4 Hours) | |
| - **147 unique coding challenges** across multiple programming languages | |
| - **5 independent trials** per task with different random seeds | |
| - **9-axis scoring system** with weighted evaluation | |
| - **20-30 minutes** execution time per model | |
| - **Primary ranking source** for our leaderboards | |
| #### 2. Canary Drift Detection (Hourly) | |
| - **12 carefully selected tasks** for rapid change detection | |
| - **2 trials per task** for fast execution (2-3 minutes) | |
| - **Safety sentinel tests** to monitor policy changes | |
| - **4x faster detection** of model degradation than full benchmarks | |
| - **Drift alerts** when performance drops >10% | |
| #### 3. Deep Reasoning Tests (Daily) | |
| - **Complex multi-step problems** requiring sustained reasoning | |
| - **Memory retention** and context management evaluation | |
| - **Long-form analysis** and problem decomposition | |
| - **60-90 minutes** per model for comprehensive assessment | |
| #### 4. Tool Calling Benchmarks (Daily) | |
| - **15+ real-world scenarios** testing agent capabilities | |
| - **File operations, system interaction, web scraping** | |
| - **Multi-step workflows** with tool chaining | |
| - **Sandbox execution** for safe testing environment | |
| ### Statistical Rigor and Methodology | |
| Our benchmarking system implements production-grade statistical analysis: | |
| #### Confidence Intervals | |
| - **T-distribution analysis** for small sample sizes (n=5) | |
| - **95% confidence intervals** on all scores | |
| - **Standard error calculation** for uncertainty quantification | |
| - **Per-model variance calibration** for fair comparison | |
| #### Example Score Generation Process | |
| ``` | |
| Task: "Implement binary search in Python" | |
| Trial Results: | |
| - Trial 1 (seed=42): Score = 23.5 | |
| - Trial 2 (seed=43): Score = 25.1 | |
| - Trial 3 (seed=44): Score = 24.2 | |
| - Trial 4 (seed=45): Score = 26.3 | |
| - Trial 5 (seed=46): Score = 24.8 | |
| Statistical Analysis: | |
| - Mean = 24.78 | |
| - Standard Deviation = 1.02 | |
| - Standard Error = 0.46 | |
| - t-value = 2.776 (df=4, 95% CI) | |
| - Margin of Error = 1.27 | |
| Final Result: 24.8 ± 1.3 (95% CI: 23.5-26.1) | |
| ``` | |
| #### Drift Detection Algorithm | |
| - **CUSUM (Cumulative Sum)** for sustained change detection | |
| - **Model-specific thresholds** based on historical variance | |
| - **Multi-signal verification** across different test suites | |
| - **False positive reduction** through statistical significance testing | |
| --- | |
| ## Smart AI Router | |
| ### Intelligent Model Selection System | |
| Our **Smart AI Router** automatically selects the best-performing AI model for each request based on real-time benchmark data, saving users money and improving performance. | |
| #### Key Features | |
| - **OpenAI-compatible API** at `https://aistupidlevel.info/v1` | |
| - **Universal API keys** (format: `aism_xxxxx`) | |
| - **6 routing strategies** for different use cases | |
| - **Automatic failover** when models are unavailable | |
| - **Cost optimization** with performance tracking | |
| - **AES-256 encryption** for all provider keys | |
| #### Routing Strategies | |
| 1. **auto** - Best overall performance across all categories | |
| 2. **auto-coding** - Optimized for programming and development tasks | |
| 3. **auto-reasoning** - Best for complex problem-solving and analysis | |
| 4. **auto-creative** - Optimized for creative writing and content generation | |
| 5. **auto-cheapest** - Most cost-effective option maintaining quality | |
| 6. **auto-fastest** - Lowest latency for time-sensitive applications | |
| #### Usage Example | |
| ```bash | |
| # Standard OpenAI-compatible request | |
| curl -X POST https://aistupidlevel.info/v1/chat/completions \ | |
| -H "Authorization: Bearer aism_your_key_here" \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "model": "auto-coding", | |
| "messages": [ | |
| {"role": "user", "content": "Write a binary search function in Python"} | |
| ] | |
| }' | |
| ``` | |
| The router automatically: | |
| 1. Analyzes your request type (coding task) | |
| 2. Checks current model rankings for coding performance | |
| 3. Selects the best available model from your configured providers | |
| 4. Routes the request and returns the response | |
| 5. Logs usage and cost data for analytics | |
| --- | |
| ## Model Comparison Features | |
| ### Claude vs GPT vs Gemini Analysis | |
| Our platform provides detailed comparative analysis of major AI models: | |
| #### Current Performance Leaders (2025) | |
| - **Best for Coding:** Claude Opus 4 consistently leads in code generation and debugging | |
| - **Fastest Response:** Gemini 2.5 Flash provides quickest API response times | |
| - **Most Reliable:** GPT-5 shows most consistent performance across categories | |
| - **Best Value:** Claude Sonnet 4 offers excellent performance-to-cost ratio | |
| #### Comprehensive Comparison Matrix | |
| - **Real-time performance data** updated every 4 hours | |
| - **Statistical significance testing** for meaningful comparisons | |
| - **Cost-per-token analysis** across all providers | |
| - **Latency benchmarks** for response time optimization | |
| - **Capability matrices** showing strengths and weaknesses | |
| --- | |
| ## Drift Detection and Monitoring | |
| ### Production-Grade Change Detection | |
| Our drift detection system implements recommendations from industry experts for catching model changes in production environments: | |
| #### Multi-Layer Detection | |
| - **Hourly canary tests** for rapid change detection | |
| - **Safety sentinel tasks** monitoring policy boundaries | |
| - **Statistical significance testing** to avoid false positives | |
| - **Vendor metadata tracking** for correlation analysis | |
| #### Alert System | |
| - **Performance degradation alerts** when scores drop >10% | |
| - **Safety policy changes** when refusal patterns shift | |
| - **API reliability monitoring** for uptime and latency | |
| - **Cost optimization alerts** when pricing changes | |
| --- | |
| ## Technical Architecture | |
| ### Backend Infrastructure | |
| - **Language:** TypeScript with Node.js | |
| - **Framework:** Fastify for high-performance APIs | |
| - **Database:** SQLite with Drizzle ORM for type safety | |
| - **Caching:** Redis for fast data access | |
| - **Security:** AES-256-GCM encryption, OAuth authentication | |
| ### Frontend Platform | |
| - **Framework:** Next.js 14 with React | |
| - **Styling:** Tailwind CSS with custom vintage theme | |
| - **Charts:** Real-time data visualization | |
| - **Authentication:** NextAuth.js with multiple providers | |
| ### Infrastructure | |
| - **Deployment:** PM2 process management | |
| - **Reverse Proxy:** Nginx with SSL termination | |
| - **Monitoring:** Custom health checks and alerting | |
| - **Scaling:** Horizontal scaling with load balancing | |
| --- | |
| ## API Access and Integration | |
| ### REST API Endpoints | |
| #### Model Rankings | |
| ```bash | |
| GET https://aistupidlevel.info/api/dashboard | |
| # Returns current model rankings with confidence intervals | |
| ``` | |
| #### Historical Data | |
| ```bash | |
| GET https://aistupidlevel.info/api/dashboard?period=7d | |
| # Returns time series data with statistical analysis | |
| ``` | |
| #### Model Details | |
| ```bash | |
| GET https://aistupidlevel.info/api/models/{model_id} | |
| # Returns detailed performance breakdown by task category | |
| ``` | |
| ### Smart Router API | |
| #### Chat Completions | |
| ```bash | |
| POST https://aistupidlevel.info/v1/chat/completions | |
| Authorization: Bearer aism_your_key | |
| { | |
| "model": "auto-coding", | |
| "messages": [...], | |
| "temperature": 0.7 | |
| } | |
| ``` | |
| #### Available Models | |
| ```bash | |
| GET https://aistupidlevel.info/v1/models | |
| Authorization: Bearer aism_your_key | |
| ``` | |
| --- | |
| ## Use Cases and Applications | |
| ### For Individual Developers | |
| - **Cost optimization** by automatically using the most efficient model | |
| - **Performance maximization** with task-specific model selection | |
| - **Reliability improvement** through automatic failover | |
| - **Time savings** with unified API access | |
| ### For Development Teams | |
| - **Centralized model management** across projects | |
| - **Usage analytics** and cost tracking | |
| - **Team-wide routing preferences** and policies | |
| - **Integration with existing tools** (Cline, Continue, Cursor) | |
| ### For Researchers and Academics | |
| - **Transparent methodology** with open-source code | |
| - **Statistical rigor** with confidence intervals | |
| - **Reproducible results** through standardized testing | |
| - **Historical data access** for trend analysis | |
| ### For AI Tool Developers | |
| - **Drop-in OpenAI replacement** with intelligent routing | |
| - **Enhanced user experience** through automatic optimization | |
| - **Reduced vendor lock-in** with multi-provider support | |
| - **Built-in analytics** for usage insights | |
| --- | |
| ## Transparency and Open Source | |
| ### Methodology Transparency | |
| - **Complete source code** available on GitHub | |
| - **Detailed documentation** of all algorithms and processes | |
| - **Statistical methodology** peer-reviewed and published | |
| - **"Test Your Keys" feature** for independent verification | |
| ### Community Validation | |
| - **Open benchmark tasks** for community review | |
| - **Statistical methodology** addressing academic concerns | |
| - **Independent verification** through user testing | |
| - **Continuous improvement** based on community feedback | |
| ### Data Access | |
| - **Public API** for accessing benchmark results | |
| - **Historical data** available for research purposes | |
| - **Real-time updates** with confidence intervals | |
| - **Export capabilities** for further analysis | |
| --- | |
| ## Getting Started | |
| ### For Benchmarking Data | |
| 1. Visit [aistupidlevel.info](https://aistupidlevel.info) | |
| 2. View real-time model rankings and comparisons | |
| 3. Access historical performance data | |
| 4. Use our API for programmatic access | |
| ### For Smart Router | |
| 1. Sign up for an account at [aistupidlevel.info](https://aistupidlevel.info) | |
| 2. Add your provider API keys (OpenAI, Anthropic, etc.) | |
| 3. Generate a universal API key | |
| 4. Replace your OpenAI endpoint with our router | |
| 5. Start saving money with intelligent routing | |
| ### For Developers | |
| 1. Check our [GitHub repository](https://github.com/StudioPlatforms) for source code | |
| 2. Review our API documentation | |
| 3. Integrate with your existing tools and workflows | |
| 4. Contribute to our open-source development | |
| --- | |
| ## Mission and Values | |
| ### Our Mission | |
| To make AI performance transparent and comparable for every developer and researcher. We believe in open evaluation, statistical rigor, and community-driven improvement of AI benchmarking standards. | |
| ### Core Values | |
| - **Transparency:** All methodologies and code are open source | |
| - **Statistical Rigor:** Proper confidence intervals and significance testing | |
| - **Community Focus:** Responsive to feedback and continuous improvement | |
| - **Practical Value:** Real-world applicability over academic metrics | |
| - **Cost Efficiency:** Helping users optimize their AI spending | |
| --- | |
| ## Contact and Community | |
| **Website:** [aistupidlevel.info](https://aistupidlevel.info) | |
| **GitHub:** [Studio Platforms](https://github.com/StudioPlatforms) | |
| **Twitter/X:** [@aistupidlevel](https://x.com/aistupidlevel) | |
| **Email:** Contact through [Studio Platforms](https://studioplatforms.eu) | |
| --- | |
| **AI Stupid Level is an independent benchmarking project built by Ionut Visan and the Studio Platforms ecosystem, dedicated to making AI performance measurement transparent, statistically rigorous, and practically useful for the global developer community.** |