| title: MBench Leaderboard | |
| emoji: 🏆 | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: gradio | |
| sdk_version: 4.36.1 | |
| python_version: '3.10' | |
| app_file: app.py | |
| pinned: false | |
| license: mit | |
| # MBench Leaderboard | |
| MBench is a benchmark for evaluating the memory capability of video world models. It focuses on whether a model can preserve a coherent world state across long-horizon video continuation and interaction. | |
| The benchmark is organized around three core memory dimensions: | |
| - **Entity Consistency:** persistent object and human identity, geometry, texture, and appearance. | |
| - **Environment Consistency:** stable spatial layout, reprojection behavior, lighting, and style. | |
| - **Causal Consistency:** reliable state evolution and interaction consequences over time. | |
| MBench uses trigger-conditioned scoring: Trigger Coverage measures whether the model actually enters the intended memory challenge, Memory Reliability measures consistency after the challenge is triggered, and M-Score balances both with a harmonic mean. | |
| The bundled seed leaderboard is transcribed from Table 2 of the MBench paper. Aggregate leaderboard columns are derived as unweighted averages over the reported sub-dimensions until official leaderboard totals are released. | |
| ## Links | |
| - **Dataset:** `studyOverflow/TempMemoryData` | |
| - **Leaderboard data repo:** `PeanutUp/membench_leaderboard_submission` | |
| - **GitHub repo:** https://github.com/study-overflow/MBench | |
| - **Project page:** https://peanutup.github.io/MBench-project/ | |