metadata
title: README
emoji: 📈
colorFrom: red
colorTo: pink
sdk: static
pinned: false
SimReal
Make the world's work checkable.
AI agents are getting good at passing tests. We build the places where they have to do the work instead: realistic environments where an agent acts, the world responds, and real outcomes decide the score.
Every benchmark we release follows the same rules. The verifier is a program, not a judge model. The scored data is never shown to the agent. And wherever possible, the bar is a real human result, not an estimate.
Benchmarks
| What the agent does | How it's scored | |
|---|---|---|
| Xitadel-QuantBench · GitHub | Researches historical order books, writes a Python trading strategy and ships it | Replayed on an unseen market day against the best human competition strategy that day: P&L, drawdown, Sharpe |
| Hard Puzzle Benchmark · GitHub | Solves 749 hard, human-written puzzles across 15 topics, many with images | Exact answers on a 707-item core, plus a separate proof-certificate track |
| MathmoBench · GitHub | Solves families of near-identical combinatorics problems where one small edit flips the answer | Machine-checkable certificates: a witness if feasible, an impossibility proof if not |
| ML Benchmark · GitHub | Runs a full ML research loop across 60 tasks, from tabular data to vision and audio | Test labels stay with the competition platform, so the scorer is not us |
Latest result
On Xitadel-QuantBench, no model has cleared the human bar overall. The best model scores 77.28 against a human reference of 80, under a shortened pilot research budget. See the leaderboard →
Work with us
- Evaluate your agent. Research teams can request an official evaluation. Results go to you first and are published only with your consent. → business@simreal.co
- Partner with us. Labs and teams can get full access to our environments. → business@simreal.co
- Join us. We work with researchers, engineers and domain experts. → simreal.co/experts