Article
Live Human Feedback in the Training Loop: Aligning Diffusion Models with Real People
Rapidata
• • 9
Mohamed Sadek Saadi
yohoji
AI & ML interests
None yet
Recent Activity
upvoted an article 2 days ago
Live Human Feedback in the Training Loop: Aligning Diffusion Models with Real People reacted to jasoncorkill's post with ❤️ about 1 month ago
Most public benchmarks collapse model performance into one broad preference signal.
That makes it hard to understand which capabilities differentiate between models. It's also almost impossible to inspect the evidence behind it. So @RapidataAI is releasing Benchmark.AI.
We started with an SVG generation benchmark including 42 models, 500 prompts, 1.9M+ human judgements, 300K+ match-ups.
We evaluate models separately on Preference, Alignment and Coherence, while making the prompts, outputs, match-ups and methodology public.
Full dataset: https://huggingface.co/datasets/Rapidata/svg-benchmark
Full benchmark: https://www.benchmark.ai/svg
Methodology feedback and benchmark suggestions very welcome!
reacted to jasoncorkill's post with 🚀 about 1 month ago
Most public benchmarks collapse model performance into one broad preference signal.
That makes it hard to understand which capabilities differentiate between models. It's also almost impossible to inspect the evidence behind it. So @RapidataAI is releasing Benchmark.AI.
We started with an SVG generation benchmark including 42 models, 500 prompts, 1.9M+ human judgements, 300K+ match-ups.
We evaluate models separately on Preference, Alignment and Coherence, while making the prompts, outputs, match-ups and methodology public.
Full dataset: https://huggingface.co/datasets/Rapidata/svg-benchmark
Full benchmark: https://www.benchmark.ai/svg
Methodology feedback and benchmark suggestions very welcome!
Organizations
None yet
upvoted an article 2 days ago
reacted to jasoncorkill's post with ❤️🚀 about 1 month ago
Post
2602
Most public benchmarks collapse model performance into one broad preference signal.
That makes it hard to understand which capabilities differentiate between models. It's also almost impossible to inspect the evidence behind it. So @RapidataAI is releasing Benchmark.AI.
We started with an SVG generation benchmark including 42 models, 500 prompts, 1.9M+ human judgements, 300K+ match-ups.
We evaluate models separately on Preference, Alignment and Coherence, while making the prompts, outputs, match-ups and methodology public.
Full dataset: Rapidata/svg-benchmark
Full benchmark: https://www.benchmark.ai/svg
Methodology feedback and benchmark suggestions very welcome!
That makes it hard to understand which capabilities differentiate between models. It's also almost impossible to inspect the evidence behind it. So @RapidataAI is releasing Benchmark.AI.
We started with an SVG generation benchmark including 42 models, 500 prompts, 1.9M+ human judgements, 300K+ match-ups.
We evaluate models separately on Preference, Alignment and Coherence, while making the prompts, outputs, match-ups and methodology public.
Full dataset: Rapidata/svg-benchmark
Full benchmark: https://www.benchmark.ai/svg
Methodology feedback and benchmark suggestions very welcome!
- 13 replies
liked a
model 2 months ago
liked a
dataset 9 months ago
liked 3
datasets 10 months ago
reacted to jasoncorkill's post with 🔥😎🚀 11 months ago
Post
4751
Do you remember https://thispersondoesnotexist.com/ ? It was one of the first cases where the future of generative media really hit us. Humans are incredibly good at recognizing and analyzing faces, so they are a very good litmus test for any generative image model.
But none of the current benchmarks measure the ability of models to generate humans independently. So we built our own. We measure the models ability to generate a diverse set of human faces and using over 20'000 human annotations we ranked all of the major models on their ability to generate faces. Find the full ranking here:
https://app.rapidata.ai/mri/benchmarks/68af24ae74482280b62f7596
We have release the full underlying data publicly here on huggingface: Rapidata/Face_Generation_Benchmark
But none of the current benchmarks measure the ability of models to generate humans independently. So we built our own. We measure the models ability to generate a diverse set of human faces and using over 20'000 human annotations we ranked all of the major models on their ability to generate faces. Find the full ranking here:
https://app.rapidata.ai/mri/benchmarks/68af24ae74482280b62f7596
We have release the full underlying data publicly here on huggingface: Rapidata/Face_Generation_Benchmark
liked 3
datasets 11 months ago
liked 6
datasets about 1 year ago
Rapidata/HunyuanImage-2.1_t2i_human_preference
Viewer • Updated • 44.8k • 549 • 9
Rapidata/Recraft-v3-24-7-25_t2i_human_preference
Viewer • Updated • 65.9k • 204 • 10
Rapidata/Imagen-4-ultra-24-7-25_t2i_human_preference
Viewer • Updated • 55.9k • 309 • 8
Rapidata/text-2-video-human-preferences-moonvalley-marey
Viewer • Updated • 1.31k • 383 • 7
Rapidata/Seedream-3_t2i_human_preference
Viewer • Updated • 60k • 527 • 12
Rapidata/image-to-video-human-preference-hailuo-02-marey
Viewer • Updated • 98 • 144 • 11