Not able to Reproduce results in MTEB
#12
by AyushM6 - opened
Hello,
I am trying to integrate Qwen3-reranker models(0.6B, 4B, 8B) in MTEB. But we were not able to reproduce the reported results. 1 of the main reasons is that the reported results are doing reranking from Qwen3-embeddings, which we were not able to reproduce. Can any of the co-authors please check the PR in MTEB, mainly if its implementation is correct or not? Its opened from a long time, and we tagged you, but you seem to have missed it.
PR Link: https://github.com/embeddings-benchmark/mteb/pull/3958
Looks like the changes introduced 12-days back were breaking. Our code failed as well.