README.md

metadata

title: MBench Leaderboard
emoji: 🏆
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 4.36.1
python_version: '3.10'
app_file: app.py
pinned: false
license: mit

MBench Leaderboard

MBench is a benchmark for evaluating the memory capability of video world models. It focuses on whether a model can preserve a coherent world state across long-horizon video continuation and interaction.

The benchmark is organized around three core memory dimensions:

• Entity Consistency: persistent object and human identity, geometry, texture, and appearance.
• Environment Consistency: stable spatial layout, reprojection behavior, lighting, and style.
• Causal Consistency: reliable state evolution and interaction consequences over time.

MBench uses trigger-conditioned scoring: Trigger Coverage measures whether the model actually enters the intended memory challenge, Memory Reliability measures consistency after the challenge is triggered, and M-Score balances both with a harmonic mean.

The bundled seed leaderboard is transcribed from Table 2 of the MBench paper. Aggregate leaderboard columns are derived as unweighted averages over the reported sub-dimensions until official leaderboard totals are released.

Links

• Dataset: studyOverflow/TempMemoryData
• Leaderboard data repo: PeanutUp/membench_leaderboard_submission
• GitHub repo: https://github.com/study-overflow/MBench
• Project page: https://peanutup.github.io/MBench-project/