Post
5,496
I really like the LocalLLaMA/typed-decisions benchmark, so I built a small Space to make its leaderboard easier to explore: tardellirs/typed-decisions-explorer
It reads the dataset card on every visit, so new submissions show up as soon as they are merged. It has a sortable ranking and two Pareto charts: accuracy × size and accuracy × latency.
A few things it makes easy to see in the zero-shot table today:
- Only three models are within noise of the top: Quyet-1.0-Large (0.791), meraGPT Decider 1 and Baseten Mammoth 1 (0.768).
- The smallest model above Jev 1.13 (0.727) is d3-mini, at 4B.
- The fastest local run above Jev is FastBrain-9B: 50 ms on an RTX PRO 6000.
- 15 of 41 models score above the teacher's self-agreement (0.735), so the top of the table measures agreement with the teacher more than correctness.
Congrats to @codelion and @sandeep85 , the maintainers, for a great benchmark. Feedback welcome!
It reads the dataset card on every visit, so new submissions show up as soon as they are merged. It has a sortable ranking and two Pareto charts: accuracy × size and accuracy × latency.
A few things it makes easy to see in the zero-shot table today:
- Only three models are within noise of the top: Quyet-1.0-Large (0.791), meraGPT Decider 1 and Baseten Mammoth 1 (0.768).
- The smallest model above Jev 1.13 (0.727) is d3-mini, at 4B.
- The fastest local run above Jev is FastBrain-9B: 50 ms on an RTX PRO 6000.
- 15 of 41 models score above the teacher's self-agreement (0.735), so the top of the table measures agreement with the teacher more than correctness.
Congrats to @codelion and @sandeep85 , the maintainers, for a great benchmark. Feedback welcome!
- 11 replies