AI Stupid Level (ASL)

Continuous, independent evaluation of the language models people actually ship on.

We re-measure a fixed fleet of production models several times a day, on the same tasks, through the vendors' own APIs — and publish when their behaviour changes. The question we exist to answer is not "which model is best on a leaderboard" but "is the model you deployed last month still the model you are calling today?"

Website: aistupidlevel.info · Operated by: Studio Platforms (Romania)

---

What is measured

18 models across 6 providers, re-measured continuously:

The fleet changes as vendors ship and retire models. Retired models keep their history — nothing is deleted, because a comparable past is the entire point.

---

The measurement suites

Five suites run on independent schedules. Each has its own tasks, its own axes and its own drift baseline. They are never blended into a single number without saying so.

Coding — every 4 hours

7 Python tasks, 7 trials each. Six of the seven are repository-repair tasks: the model is given a real codebase and a bug report, and graded on whether it fixed the actual defect or silenced the symptom it was shown. Scored on 9 axes: correctness, stability, edge cases, debugging, code quality, efficiency, format, safety, complexity.

Tool calling — daily

9 sandboxed agent tasks (3 easy, 3 medium, 3 hard) covering file operations, multi-step workflows and tool chaining. Scored on 7 axes: task completion, tool selection, parameter accuracy, efficiency, error handling, context awareness, safety compliance.

Deep reasoning — daily

4 multi-turn tasks that run to completion rather than a single exchange: IDE assistance, specification following, document memory, project refactoring. Scored on 6 axes: correctness, memory retention, plan coherence, context window handling, debugging, stability.

Canary — hourly

2 tasks, 2 trials, deliberately small. A fast probe whose only job is to notice change between the heavier runs.

Calibration and known-unknowns — every 6 hours

Does the model know what it does not know? Each model is asked questions from a stratified corpus; some have a verifiable answer, some have none — a library function that does not exist, a real company with an attribute it cannot have, a fact it has no way to observe. Declining the second group is the correct behaviour and is scored as such. Reported per model: fabrication rate, accuracy, abstention rate, expected calibration error, and over/under-confidence.

These figures are published separately and never enter the composite score. Being well calibrated does not make a model better at coding, and blending the two would make both numbers mean less.

---

Drift detection

A leaderboard tells you where a model stands today. Drift detection tells you when it moved.

• Page-Hinkley change detection, run per suite on that suite's own daily series — coding, tool use and reasoning are never blended, because each has a different scale and noise floor.
• A suite needs 10 measured days on its current benchmark configuration before it may fire. The statistic is visible from day one, but a reading during warm-up is an early indication, not a verdict.
• Any scoring or corpus change mints a new configuration id and resets the baseline. Scores either side of such a change are not comparable and are not compared.
• Task-level regressions are detected separately, from execution records rather than composite scores. A single task collapsing is close to invisible in a composite — it moves the overall score by less than the alert threshold — so it is reported on its own.

---

How the numbers are kept honest

This is the part we would want to read first about someone else's benchmark.

• A provider that is not answering is skipped, not scored. A dead API key or an outage must never be published as a model getting worse.
• Refusals are recorded, not imputed. If a model declines a task, that task drops out and the run is marked partial. A refusal is not a zero.
• A missing suite drops out and the remaining weights are renormalised. No suite is ever backfilled with a placeholder value.
• Provenance is labelled. Where a figure is not backed by a live measurement, the interface says so rather than presenting it as one.
• Every task grades only what was asked. Hidden requirements that were never stated in the prompt are not graded.
• Ranks close together are statistical ties and are presented as such, with confidence intervals rather than false precision.

The full methodology, including the scoring weights and the task design, is published at aistupidlevel.info/methodology.

---

Smart router

An OpenAI-compatible endpoint that picks the model for each request from current benchmark results instead of a hard-coded name.

curl -X POST https://aistupidlevel.info/v1/chat/completions \
  -H "Authorization: Bearer aism_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto-coding",
    "messages": [{"role": "user", "content": "Write a binary search in Python"}]
  }'

Eight strategies: auto (best overall), auto-coding, auto-reasoning, auto-creative, auto-tooling, auto-agentic, auto-cheapest, auto-fastest.

You bring your own provider keys. They are encrypted at rest with AES-256-GCM, and the providers bill you directly — we never resell tokens or mark them up. A provider outage falls through to the next best model rather than failing your request. Works as a drop-in replacement in tools that accept an OpenAI base URL, such as Cline, Continue and Cursor.

---

Data API

Programmatic access to scores, rankings, history, drift signals and incidents, at https://aistupidlevel.info/api/v1 with asl_ keys.

The free tier is an evaluation allowance — enough to build against, not to run on. Documentation: aistupidlevel.info/api-docs.

---

Who this is for

• Teams running models in production — know within hours when a model you depend on starts behaving differently, instead of finding out from your users.
• Researchers — a public methodology, published per-task results, and history that survives corpus changes because configuration boundaries are recorded rather than smoothed over.
• Developers choosing a model — per-suite comparisons, cost per million tokens, and measured latency, rather than a single aggregate that hides the trade-off you care about.
• Anyone tired of vibes — every published figure traces back to a run we can point at.

Current scores, every category ranking, seven days of history, provider status and the full methodology are free and require no account. Paid plans add longer comparable history, the drift curves, calibration data, exports and the routing volume.

---

Technical notes

• API: TypeScript, Fastify, SQLite with Drizzle ORM, Redis caching
• Web: Next.js 14, React 18, Recharts, NextAuth
• Infrastructure: a single VPS under systemd, behind nginx with SSL
• Frontend source: github.com/StudioPlatforms/aistupidmeter-web

The web frontend is public. The benchmark engine is not open source — the task corpus is deliberately unpublished, because a benchmark whose answers a model has been trained on measures nothing. The methodology, the scoring weights and the per-task outcomes are public instead, which is what makes a result checkable without making it gameable.

---

Status and transparency

A live status page reports provider availability and separates three things that look identical from outside and mean different things: the provider being down, our own account or key failing, and our probe failing. When a suite could not run, the site says which and why rather than quietly publishing a lower number.

---

Contact

• Contact form: aistupidlevel.info/contact
• LinkedIn: ASL — aistupidlevel.info
• X / Twitter: @aistupidlevel
• Website: aistupidlevel.info · also reachable at asl.info

---

AI Stupid Level is an independent evaluation platform built by Ionut Adrian Visan. It is not affiliated with, endorsed by, or funded by any of the model providers it measures — which is the only reason its numbers are worth anything.