| --- |
| title: README |
| emoji: 📈 |
| colorFrom: red |
| colorTo: pink |
| sdk: static |
| pinned: false |
| --- |
| |
| # SimReal |
|
|
| **Make the world's work checkable.** |
|
|
| AI agents are getting good at passing tests. We build the places where they have to do the work instead: realistic environments where an agent acts, the world responds, and real outcomes decide the score. |
|
|
| Every benchmark we release follows the same rules. The verifier is a program, not a judge model. The scored data is never shown to the agent. And wherever possible, the bar is a real human result, not an estimate. |
|
|
| ## Benchmarks |
|
|
| | | What the agent does | How it's scored | |
| |---|---|---| |
| | **[Xitadel-QuantBench](https://huggingface.co/spaces/SimrealAI/xitadel-quantbench-leaderboard)** · [GitHub](https://github.com/Simreal-AI/Xitadel-QuantBench) | Researches historical order books, writes a Python trading strategy and ships it | Replayed on an unseen market day against the best human competition strategy that day: P&L, drawdown, Sharpe | |
| | **[Hard Puzzle Benchmark](https://huggingface.co/datasets/SimrealAI/hard-puzzle-benchmark)** · [GitHub](https://github.com/Simreal-AI/hard-puzzle-benchmark) | Solves 749 hard, human-written puzzles across 15 topics, many with images | Exact answers on a 707-item core, plus a separate proof-certificate track | |
| | **MathmoBench** · [GitHub](https://github.com/Simreal-AI/MathmoBench) | Solves families of near-identical combinatorics problems where one small edit flips the answer | Machine-checkable certificates: a witness if feasible, an impossibility proof if not | |
| | **ML Benchmark** · [GitHub](https://github.com/Simreal-AI/Simreal-MLBench) | Runs a full ML research loop across 60 tasks, from tabular data to vision and audio | Test labels stay with the competition platform, so the scorer is not us | |
|
|
| ## Latest result |
|
|
| On Xitadel-QuantBench, **no model has cleared the human bar overall**. The best model scores 77.28 against a human reference of 80, under a shortened pilot research budget. [See the leaderboard →](https://huggingface.co/spaces/SimrealAI/xitadel-quantbench-leaderboard) |
|
|
| ## Work with us |
|
|
| - **Evaluate your agent.** Research teams can request an official evaluation. Results go to you first and are published only with your consent. → [business@simreal.co](mailto:business@simreal.co?subject=Evaluate%20my%20agent) |
| - **Partner with us.** Labs and teams can get full access to our environments. → [business@simreal.co](mailto:business@simreal.co) |
| - **Join us.** We work with researchers, engineers and domain experts. → [simreal.co/experts](https://simreal.co/experts) |
|
|
| [Website](https://simreal.co) · [GitHub](https://github.com/Simreal-AI) · [X](https://x.com/simrealhq) · [LinkedIn](https://www.linkedin.com/company/simreal-ai) |
|
|
