Spaces:
Running on CPU Upgrade
MTEB Leaderboard is really slow for loading & processing queries
Hello, i'm writting because the MTEB benchmark takes really long time to load and I don't think it's quite normal and wanted to notify it.
I don't really know what is happening under the hood
Thank you,
Alan
Yes, we know this. Some time ago leaderboard eas working fine, but overtime suddenly it became very slow. Currently we're working on new leaderboard version that would much faster. I hope it will be released in week or two
As a temporary fix we also merged this PR:
https://github.com/embeddings-benchmark/mteb/pull/4716
Doesn't solve the main underlying problems which stem from the gradio interface, but it should make it notably better until we can get in the replacement
@Alanox @devalias @axkoro You can see new demo leaderboard here: https://huggingface.co/spaces/mteb/leaderboardv2. You are welcome to provide feedback