Update README.md
Browse files
README.md
CHANGED
|
@@ -143,29 +143,19 @@ first three improvements and the first drop in a random draw from those images.
|
|
| 143 |
| COCO 300233 | mixing bowl corn food | Ingredients and a food processor are arranged on a gray surface. (0.40) | **Ingredients and a food processor are laid out on a table. (0.47)** |
|
| 144 |
| COCO 419228 (failure) | ice bear polar | **A white polar bear walks on rocks in an enclosure. (2.54)** | A large white polar bear walks on a rocky surface. (1.85) |
|
| 145 |
|
| 146 |
-
**Summary: gain over each captioner's own top beam (CIDEr).**
|
| 147 |
|
| 148 |
| Captioner | Top beam | + This model | Gain [95% CI] |
|
| 149 |
| --- | ---: | ---: | --- |
|
| 150 |
| BLIP-base (training captioner) | 1.0632 | 1.1470 | +0.0838 [+0.073, +0.095] |
|
| 151 |
| ViLBERT (paper's candidates) | 1.1150 | 1.1475 | +0.0325 [+0.021, +0.044] |
|
| 152 |
| Gemma 4 12B | 0.9624 | 0.9783 | +0.0159 [+0.009, +0.023] |
|
|
|
|
| 153 |
|
| 154 |
-
## Examples (ViLBERT candidates, CIDEr per caption)
|
| 155 |
-
|
| 156 |
-
| Image | Best Beam | This model |
|
| 157 |
-
| --- | --- | --- |
|
| 158 |
-
| COCO 460621 | a white toilet sitting next to a white bath tub (0.39) | **a bathroom with a sink and a toilet (0.83)** |
|
| 159 |
-
| COCO 43680 | a group of cows that are standing in the dirt (0.64) | **a herd of cattle walking across a dry field (1.32)** |
|
| 160 |
-
| COCO 424432 | a white and brown dog laying on a bed (0.73) | **a dog laying on a bed with a blanket (0.86)** |
|
| 161 |
-
| COCO 365685 (failure) | **a woman is taking a bite out of a hot dog (0.92)** | a woman sitting at a table eating a hotdog (0.11) |
|
| 162 |
-
|
| 163 |
-
When the model changes the caption (1,948 of 5,000 images), the change improves
|
| 164 |
-
CIDEr 1,110 times and lowers it 838 times.
|
| 165 |
|
| 166 |
## Limitations
|
| 167 |
|
| 168 |
-
- **Use it as a reranker, not a relatedness classifier.** Reward training shifts
|
| 169 |
the scores upward. On the paper's 41,849 held-out relatedness pairs, accuracy at
|
| 170 |
the 0.5 cutoff falls from 0.893 to 0.631, and ROC-AUC from 0.962 to 0.937. For
|
| 171 |
relatedness probabilities, use the stage I BERT-CNN.
|
|
|
|
| 143 |
| COCO 300233 | mixing bowl corn food | Ingredients and a food processor are arranged on a gray surface. (0.40) | **Ingredients and a food processor are laid out on a table. (0.47)** |
|
| 144 |
| COCO 419228 (failure) | ice bear polar | **A white polar bear walks on rocks in an enclosure. (2.54)** | A large white polar bear walks on a rocky surface. (1.85) |
|
| 145 |
|
| 146 |
+
<!-- **Summary: gain over each captioner's own top beam (CIDEr).**
|
| 147 |
|
| 148 |
| Captioner | Top beam | + This model | Gain [95% CI] |
|
| 149 |
| --- | ---: | ---: | --- |
|
| 150 |
| BLIP-base (training captioner) | 1.0632 | 1.1470 | +0.0838 [+0.073, +0.095] |
|
| 151 |
| ViLBERT (paper's candidates) | 1.1150 | 1.1475 | +0.0325 [+0.021, +0.044] |
|
| 152 |
| Gemma 4 12B | 0.9624 | 0.9783 | +0.0159 [+0.009, +0.023] |
|
| 153 |
+
-->
|
| 154 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 155 |
|
| 156 |
## Limitations
|
| 157 |
|
| 158 |
+
- **Use it as a reranker, not a (relatedness classifier)[https://huggingface.co/AhmedSSabir/ModernBERT-CNN-Caption-Reranker]. ** Reward training shifts
|
| 159 |
the scores upward. On the paper's 41,849 held-out relatedness pairs, accuracy at
|
| 160 |
the 0.5 cutoff falls from 0.893 to 0.631, and ROC-AUC from 0.962 to 0.937. For
|
| 161 |
relatedness probabilities, use the stage I BERT-CNN.
|