Update README.md
Browse files
README.md
CHANGED
|
@@ -11,17 +11,19 @@ pinned: false
|
|
| 11 |
|
| 12 |
Project page: https://mj-bench.github.io/
|
| 13 |
|
| 14 |
-

|
| 15 |
-
|
| 16 |
-
|
| 17 |
While text-to-image models like DALLE-3 and Stable Diffusion are rapidly proliferating, they often encounter challenges such as hallucination, bias, and the production of unsafe, low-quality output. To effectively address these issues, it is crucial to align these models with desired behaviors based on feedback from a multimodal judge. Despite their significance, current multimodal judges frequently undergo inadequate evaluation of their capabilities and limitations, potentially leading to misalignment and unsafe fine-tuning outcomes.
|
| 18 |
|
| 19 |
To address this issue, we introduce MJ-Bench, a novel benchmark which incorporates a comprehensive preference dataset to evaluate multimodal judges in providing feedback for image generation models across four key perspectives: **alignment**, **safety**, **image quality**, and **bias**.
|
| 20 |
|
|
|
|
|
|
|
| 21 |
Specifically, we evaluate a large variety of multimodal judges including
|
| 22 |
|
| 23 |
- 6 smaller-sized CLIP-based scoring models
|
| 24 |
- 11 open-source VLMs (e.g. LLaVA family)
|
| 25 |
- 4 and close-source VLMs (e.g. GPT-4o, Claude 3)
|
|
|
|
|
|
|
|
|
|
| 26 |
|
| 27 |
We are actively updating the [leaderboard](https://mj-bench.github.io/) and you are welcome to submit the evaluation result of your multimodal judge on [our dataset](https://huggingface.co/datasets/MJ-Bench/MJ-Bench) to [huggingface leaderboard](https://huggingface.co/spaces/MJ-Bench/MJ-Bench-Leaderboard).
|
|
|
|
| 11 |
|
| 12 |
Project page: https://mj-bench.github.io/
|
| 13 |
|
|
|
|
|
|
|
|
|
|
| 14 |
While text-to-image models like DALLE-3 and Stable Diffusion are rapidly proliferating, they often encounter challenges such as hallucination, bias, and the production of unsafe, low-quality output. To effectively address these issues, it is crucial to align these models with desired behaviors based on feedback from a multimodal judge. Despite their significance, current multimodal judges frequently undergo inadequate evaluation of their capabilities and limitations, potentially leading to misalignment and unsafe fine-tuning outcomes.
|
| 15 |
|
| 16 |
To address this issue, we introduce MJ-Bench, a novel benchmark which incorporates a comprehensive preference dataset to evaluate multimodal judges in providing feedback for image generation models across four key perspectives: **alignment**, **safety**, **image quality**, and **bias**.
|
| 17 |
|
| 18 |
+

|
| 19 |
+
|
| 20 |
Specifically, we evaluate a large variety of multimodal judges including
|
| 21 |
|
| 22 |
- 6 smaller-sized CLIP-based scoring models
|
| 23 |
- 11 open-source VLMs (e.g. LLaVA family)
|
| 24 |
- 4 and close-source VLMs (e.g. GPT-4o, Claude 3)
|
| 25 |
+
-
|
| 26 |
+

|
| 27 |
+
|
| 28 |
|
| 29 |
We are actively updating the [leaderboard](https://mj-bench.github.io/) and you are welcome to submit the evaluation result of your multimodal judge on [our dataset](https://huggingface.co/datasets/MJ-Bench/MJ-Bench) to [huggingface leaderboard](https://huggingface.co/spaces/MJ-Bench/MJ-Bench-Leaderboard).
|