| ============================================================================== |
| EVAL SUMMARY | refcoco_2ep_native |
| ============================================================================== |
|
|
| Metric: mean-over-IoU (fastevaluate). BPS = boxes/second, batch 1, 1 GPU. |
| fallback% = switch_to_ar / forward_steps (fraction of block-decisions; paper implies ~7-8%) |
|
|
| profile mode | F1 P R F1@.5 | BPS steps boxes fallb% |
| --------------------------------------------------------------------------------- |
| paper_seed0 slow | 47.0 49.7 44.5 68.5 | 2.70 67.5 7.06 0.0 |
| paper_seed0 fast | 45.8 54.5 39.5 64.1 | 8.82 15.3 6.91 0.0 |
| paper_seed0 hybrid | 46.7 55.0 40.6 65.3 | 7.84 19.1 7.00 5.5 |
|
|
| paper_seed1 slow | 46.8 49.6 44.3 68.3 | 2.73 67.9 7.11 0.0 |
| paper_seed1 fast | 45.8 54.4 39.5 64.1 | 8.85 15.3 6.90 0.0 |
| paper_seed1 hybrid | 46.5 54.9 40.4 65.3 | 7.79 19.0 6.97 5.5 |
|
|
| paper_greedy slow | 49.3 51.7 47.0 70.0 | 2.81 69.4 7.27 0.0 |
| paper_greedy fast | 46.9 51.7 43.0 66.8 | 9.20 15.1 7.01 0.0 |
| paper_greedy hybrid | 47.9 52.4 44.2 68.0 | 8.08 19.4 7.16 6.1 |
|
|
| legacy slow | 49.3 51.7 47.0 70.0 | 2.77 69.4 7.27 0.0 |
| legacy fast | 46.9 51.7 43.0 66.8 | 9.29 15.1 7.01 0.0 |
| legacy hybrid | 47.9 52.4 44.2 68.0 | 8.17 19.4 7.16 6.1 |
|
|
| paper is STOCHASTIC (nucleus sampling). Mean +/- half-spread over 2 seeds: |
| --------------------------------------------------------------------------------- |
| paper (mean) slow | F1 46.9+/-0.1 BPS 2.7+/-0.0 fallback% 0.0+/-0.0 |
| paper (mean) fast | F1 45.8+/-0.0 BPS 8.8+/-0.0 fallback% 0.0+/-0.0 |
| paper (mean) hybrid | F1 46.6+/-0.1 BPS 7.8+/-0.0 fallback% 5.5+/-0.0 |
|
|
| -> a gap smaller than the spread is NOT evidence. Use paper_greedy |
| (deterministic) for tight comparisons. |
|
|
| INTEGRITY legacy/slow (49.27) vs paper_greedy/slow (49.27) -> OK (identical, as required) |
|
|
| B1 READ-OUT hybrid fallback rate (switch_to_ar/forward_steps): legacy 6.1% vs paper 5.5% (paper ~7-8%; the RELATIVE legacy>paper gap is the primary signal) |
| The 0.9 abnormality threshold is calibrated on the SHARPENED (T=0.7 + |
| top-p) distribution. Greedy's flatter softmax trips it far more often, so |
| legacy should be markedly higher. If they match, B1 has not taken effect. |
|
|
| ============================================================================== |
| PROMPT-SHAPE (n_cats) OOD -- profile: paper_greedy |
| ============================================================================== |
| Training chunks each image's categories into queries of 1..3. The eval set |
| asks for the image's FULL list, and 28.8% of eval queries have >3 categories |
| -- a prompt shape the model never saw. Caveat: images with more categories |
| also have more objects, so a drop may be difficulty, not novelty. |
|
|
| --- paper_greedy/slow --- |
| predictions: 4952 | eval queries: 4952 | unbucketed: 0 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| 1 | 1025 | 71.4 | 75.1 | 68.2 | 86.3 |
| 2 | 1522 | 61.6 | 63.3 | 59.9 | 80.1 |
| 3 | 980 | 52.0 | 55.4 | 49.0 | 72.3 |
| 4-5 | 935 | 48.0 | 51.6 | 44.8 | 69.1 |
| 6+ | 490 | 43.2 | 50.3 | 37.8 | 64.5 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| <=3 (IN) | 3527 | 56.3 | 58.0 | 54.7 | 75.7 |
| >3 (OOD) | 1425 | 44.3 | 48.6 | 40.7 | 65.6 |
|
|
| If '>3 (OOD)' is materially below '<=3 (IN)', the prompt-shape gap |
| is real: training never showed the model a 4+-category query. |
| Careful: bigger images have more categories AND more objects, so a |
| small drop may just be task difficulty, not prompt-shape OOD. |
|
|
| --- paper_greedy/fast --- |
| predictions: 4952 | eval queries: 4952 | unbucketed: 0 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| 1 | 1025 | 68.5 | 75.8 | 62.6 | 83.1 |
| 2 | 1522 | 59.3 | 62.3 | 56.6 | 78.0 |
| 3 | 980 | 49.4 | 54.5 | 45.1 | 68.5 |
| 4-5 | 935 | 46.3 | 53.3 | 41.0 | 66.1 |
| 6+ | 490 | 40.5 | 52.0 | 33.1 | 59.8 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| <=3 (IN) | 3527 | 53.8 | 57.2 | 50.8 | 72.6 |
| >3 (OOD) | 1425 | 42.3 | 49.9 | 36.7 | 62.0 |
|
|
| If '>3 (OOD)' is materially below '<=3 (IN)', the prompt-shape gap |
| is real: training never showed the model a 4+-category query. |
| Careful: bigger images have more categories AND more objects, so a |
| small drop may just be task difficulty, not prompt-shape OOD. |
|
|
| --- paper_greedy/hybrid --- |
| predictions: 4952 | eval queries: 4952 | unbucketed: 0 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| 1 | 1025 | 70.3 | 74.7 | 66.4 | 86.4 |
| 2 | 1522 | 60.3 | 62.8 | 58.0 | 79.1 |
| 3 | 980 | 50.7 | 55.7 | 46.6 | 70.2 |
| 4-5 | 935 | 46.7 | 53.5 | 41.4 | 67.0 |
| 6+ | 490 | 40.8 | 52.0 | 33.6 | 60.5 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| <=3 (IN) | 3527 | 55.2 | 58.0 | 52.6 | 74.3 |
| >3 (OOD) | 1425 | 42.7 | 50.2 | 37.2 | 62.8 |
|
|
| If '>3 (OOD)' is materially below '<=3 (IN)', the prompt-shape gap |
| is real: training never showed the model a 4+-category query. |
| Careful: bigger images have more categories AND more objects, so a |
| small drop may just be task difficulty, not prompt-shape OOD. |
|
|
|
|
