Spaces:
Running
Running
refactor: update about tab with experimentation details
Browse files
ui.py
CHANGED
|
@@ -50,12 +50,13 @@ there is no single score across InferBench.
|
|
| 50 |
|
| 51 |
### OneIG Alignment
|
| 52 |
Prompt-image **alignment** on anime / stylization, human / portrait, and
|
| 53 |
-
general object prompts. This is the alignment slice of
|
| 54 |
-
suite. Default rank is the mean of the alignment
|
| 55 |
-
that row.
|
| 56 |
|
| 57 |
### Qwen Image Bench
|
| 58 |
-
|
|
|
|
| 59 |
"""
|
| 60 |
|
| 61 |
ABOUT_DETAILS_CONTENT = """
|
|
@@ -69,8 +70,9 @@ ABOUT_DETAILS_CONTENT = """
|
|
| 69 |
not a pairwise Elo. It is not a text-to-image model on the leaderboard. Use
|
| 70 |
it alongside Datapoint and Rapidata Elo, not instead of them.
|
| 71 |
- **Datapoint Elo**: human-preference Elo from Datapoint pairwise comparisons.
|
| 72 |
-
- **Rapidata Elo**: human-preference Elo from
|
| 73 |
-
|
|
|
|
| 74 |
- **Generation time**: median and minimum generation time in seconds, as
|
| 75 |
reported in the evaluation table. This is not a p95, and we do not state
|
| 76 |
warm vs cold or concurrent load.
|
|
@@ -83,17 +85,30 @@ models *within* a column.
|
|
| 83 |
|
| 84 |
## Setup
|
| 85 |
|
| 86 |
-
- **Evaluation window:**
|
| 87 |
- **Update policy:** numbers come from evaluation snapshots in the tables,
|
| 88 |
not a live API poll.
|
| 89 |
-
- **Prompt counts:** OneIG
|
|
|
|
|
|
|
|
|
|
| 90 |
- **Generation:** one image per prompt per endpoint when the run exists.
|
| 91 |
-
|
| 92 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
|
| 94 |
## Limits
|
| 95 |
|
| 96 |
- Empty cells mean that track was not run or not reported for that model.
|
|
|
|
|
|
|
| 97 |
- Elo ratings can shift when the comparison pool changes: treat them as
|
| 98 |
relative rankings for the snapshot, not absolute constants.
|
| 99 |
- Close scores can be a tie in practice; the table does not show confidence
|
|
|
|
| 50 |
|
| 51 |
### OneIG Alignment
|
| 52 |
Prompt-image **alignment** on anime / stylization, human / portrait, and
|
| 53 |
+
general object prompts (100 prompts each). This is the alignment slice of
|
| 54 |
+
OneIG, not the full suite. Default rank is the mean of the alignment
|
| 55 |
+
categories that exist for that row.
|
| 56 |
|
| 57 |
### Qwen Image Bench
|
| 58 |
+
100 prompts from the 1,000-prompt Qwen Image Bench set, sampled for coverage
|
| 59 |
+
across its fine-grained (L3) categories.
|
| 60 |
"""
|
| 61 |
|
| 62 |
ABOUT_DETAILS_CONTENT = """
|
|
|
|
| 70 |
not a pairwise Elo. It is not a text-to-image model on the leaderboard. Use
|
| 71 |
it alongside Datapoint and Rapidata Elo, not instead of them.
|
| 72 |
- **Datapoint Elo**: human-preference Elo from Datapoint pairwise comparisons.
|
| 73 |
+
- **Rapidata Elo**: human-preference Elo from Rapidata pairwise comparisons.
|
| 74 |
+
Rapidata rejects prompts over 400 characters, so this Elo is on a subset
|
| 75 |
+
of each suite (see Setup).
|
| 76 |
- **Generation time**: median and minimum generation time in seconds, as
|
| 77 |
reported in the evaluation table. This is not a p95, and we do not state
|
| 78 |
warm vs cold or concurrent load.
|
|
|
|
| 85 |
|
| 86 |
## Setup
|
| 87 |
|
| 88 |
+
- **Evaluation window:** July–August 2026.
|
| 89 |
- **Update policy:** numbers come from evaluation snapshots in the tables,
|
| 90 |
not a live API poll.
|
| 91 |
+
- **Prompt counts:** OneIG Alignment uses the first 100 prompts from each of
|
| 92 |
+
the three categories (300 total). Qwen Image Bench uses 100 prompts sampled
|
| 93 |
+
from the 1,000-prompt pool for roughly even coverage of its fine-grained
|
| 94 |
+
(L3) categories.
|
| 95 |
- **Generation:** one image per prompt per endpoint when the run exists.
|
| 96 |
+
Default resolution is 1024×1024. Exceptions: FLUX 1.1 Pro Ultra at 2K,
|
| 97 |
+
FLUX 2 Flex at 1008×1008, and any endpoint labeled 2K. The seed is derived
|
| 98 |
+
from the prompt, so every model gets the same seed for the same prompt.
|
| 99 |
+
Steps, CFG, prompt rewrite, and safety filters follow each endpoint's
|
| 100 |
+
default.
|
| 101 |
+
- **Datapoint:** every model pair is compared on every prompt, with 10 votes
|
| 102 |
+
per battle.
|
| 103 |
+
- **Rapidata:** prompts longer than 400 characters are dropped, leaving 212
|
| 104 |
+
OneIG prompts and 85 Qwen Image Bench prompts. 4 votes per pair; about
|
| 105 |
+
26,000 votes on OneIG and 35,000 on Qwen Image Bench.
|
| 106 |
|
| 107 |
## Limits
|
| 108 |
|
| 109 |
- Empty cells mean that track was not run or not reported for that model.
|
| 110 |
+
- Rapidata Elo is not on the full prompt suite, so it is not directly
|
| 111 |
+
comparable to Datapoint Elo even on the same benchmark.
|
| 112 |
- Elo ratings can shift when the comparison pool changes: treat them as
|
| 113 |
relative rankings for the snapshot, not absolute constants.
|
| 114 |
- Close scores can be a tie in practice; the table does not show confidence
|