begumcig commited on
Commit
3473c50
·
1 Parent(s): 517e668

refactor: update about tab with experimentation details

Browse files
Files changed (1) hide show
  1. ui.py +25 -10
ui.py CHANGED
@@ -50,12 +50,13 @@ there is no single score across InferBench.
50
 
51
  ### OneIG Alignment
52
  Prompt-image **alignment** on anime / stylization, human / portrait, and
53
- general object prompts. This is the alignment slice of OneIG, not the full
54
- suite. Default rank is the mean of the alignment categories that exist for
55
- that row.
56
 
57
  ### Qwen Image Bench
58
- A shared prompt set used to compare endpoints on the same prompts.
 
59
  """
60
 
61
  ABOUT_DETAILS_CONTENT = """
@@ -69,8 +70,9 @@ ABOUT_DETAILS_CONTENT = """
69
  not a pairwise Elo. It is not a text-to-image model on the leaderboard. Use
70
  it alongside Datapoint and Rapidata Elo, not instead of them.
71
  - **Datapoint Elo**: human-preference Elo from Datapoint pairwise comparisons.
72
- - **Rapidata Elo**: human-preference Elo from the Rapidata evaluation on the
73
- same suite.
 
74
  - **Generation time**: median and minimum generation time in seconds, as
75
  reported in the evaluation table. This is not a p95, and we do not state
76
  warm vs cold or concurrent load.
@@ -83,17 +85,30 @@ models *within* a column.
83
 
84
  ## Setup
85
 
86
- - **Evaluation window:**
87
  - **Update policy:** numbers come from evaluation snapshots in the tables,
88
  not a live API poll.
89
- - **Prompt counts:** OneIG —, Qwen —.
 
 
 
90
  - **Generation:** one image per prompt per endpoint when the run exists.
91
- Resolution, steps, CFG, seed policy, prompt rewrite, and safety filters: —.
92
- - **Pairwise comparisons (Datapoint / Rapidata):**
 
 
 
 
 
 
 
 
93
 
94
  ## Limits
95
 
96
  - Empty cells mean that track was not run or not reported for that model.
 
 
97
  - Elo ratings can shift when the comparison pool changes: treat them as
98
  relative rankings for the snapshot, not absolute constants.
99
  - Close scores can be a tie in practice; the table does not show confidence
 
50
 
51
  ### OneIG Alignment
52
  Prompt-image **alignment** on anime / stylization, human / portrait, and
53
+ general object prompts (100 prompts each). This is the alignment slice of
54
+ OneIG, not the full suite. Default rank is the mean of the alignment
55
+ categories that exist for that row.
56
 
57
  ### Qwen Image Bench
58
+ 100 prompts from the 1,000-prompt Qwen Image Bench set, sampled for coverage
59
+ across its fine-grained (L3) categories.
60
  """
61
 
62
  ABOUT_DETAILS_CONTENT = """
 
70
  not a pairwise Elo. It is not a text-to-image model on the leaderboard. Use
71
  it alongside Datapoint and Rapidata Elo, not instead of them.
72
  - **Datapoint Elo**: human-preference Elo from Datapoint pairwise comparisons.
73
+ - **Rapidata Elo**: human-preference Elo from Rapidata pairwise comparisons.
74
+ Rapidata rejects prompts over 400 characters, so this Elo is on a subset
75
+ of each suite (see Setup).
76
  - **Generation time**: median and minimum generation time in seconds, as
77
  reported in the evaluation table. This is not a p95, and we do not state
78
  warm vs cold or concurrent load.
 
85
 
86
  ## Setup
87
 
88
+ - **Evaluation window:** July–August 2026.
89
  - **Update policy:** numbers come from evaluation snapshots in the tables,
90
  not a live API poll.
91
+ - **Prompt counts:** OneIG Alignment uses the first 100 prompts from each of
92
+ the three categories (300 total). Qwen Image Bench uses 100 prompts sampled
93
+ from the 1,000-prompt pool for roughly even coverage of its fine-grained
94
+ (L3) categories.
95
  - **Generation:** one image per prompt per endpoint when the run exists.
96
+ Default resolution is 1024×1024. Exceptions: FLUX 1.1 Pro Ultra at 2K,
97
+ FLUX 2 Flex at 1008×1008, and any endpoint labeled 2K. The seed is derived
98
+ from the prompt, so every model gets the same seed for the same prompt.
99
+ Steps, CFG, prompt rewrite, and safety filters follow each endpoint's
100
+ default.
101
+ - **Datapoint:** every model pair is compared on every prompt, with 10 votes
102
+ per battle.
103
+ - **Rapidata:** prompts longer than 400 characters are dropped, leaving 212
104
+ OneIG prompts and 85 Qwen Image Bench prompts. 4 votes per pair; about
105
+ 26,000 votes on OneIG and 35,000 on Qwen Image Bench.
106
 
107
  ## Limits
108
 
109
  - Empty cells mean that track was not run or not reported for that model.
110
+ - Rapidata Elo is not on the full prompt suite, so it is not directly
111
+ comparable to Datapoint Elo even on the same benchmark.
112
  - Elo ratings can shift when the comparison pool changes: treat them as
113
  relative rankings for the snapshot, not absolute constants.
114
  - Close scores can be a tie in practice; the table does not show confidence