| ============================================================================== |
| EVAL SUMMARY | coco_2ep_native |
| ============================================================================== |
|
|
| Metric: mean-over-IoU (fastevaluate). BPS = boxes/second, batch 1, 1 GPU. |
| fallback% = switch_to_ar / forward_steps (fraction of block-decisions; paper implies ~7-8%) |
|
|
| profile mode | F1 P R F1@.5 | BPS steps boxes fallb% |
| --------------------------------------------------------------------------------- |
| paper_seed0 slow | 48.7 51.6 46.2 69.4 | 2.72 67.4 7.06 0.0 |
| paper_seed0 fast | 47.5 57.6 40.3 64.9 | 8.72 15.1 6.77 0.0 |
| paper_seed0 hybrid | 48.2 57.7 41.4 65.9 | 8.08 17.7 6.89 4.1 |
|
|
| paper_seed1 slow | 48.9 51.9 46.3 69.6 | 2.71 67.3 7.05 0.0 |
| paper_seed1 fast | 47.4 57.6 40.3 64.8 | 8.78 15.1 6.77 0.0 |
| paper_seed1 hybrid | 48.2 57.8 41.3 66.0 | 8.10 17.7 6.89 4.1 |
|
|
| paper_greedy slow | 51.2 54.0 48.7 71.0 | 2.78 68.4 7.16 0.0 |
| paper_greedy fast | 49.0 54.5 44.5 67.8 | 9.23 14.8 6.88 0.0 |
| paper_greedy hybrid | 50.1 55.3 45.8 69.4 | 8.42 17.7 7.01 4.5 |
|
|
| legacy slow | 51.2 54.0 48.7 71.0 | 2.78 68.4 7.16 0.0 |
| legacy fast | 49.0 54.6 44.5 67.9 | 9.25 14.8 6.88 0.0 |
| legacy hybrid | 50.1 55.3 45.8 69.4 | 8.44 17.7 7.01 4.5 |
|
|
| paper is STOCHASTIC (nucleus sampling). Mean +/- half-spread over 2 seeds: |
| --------------------------------------------------------------------------------- |
| paper (mean) slow | F1 48.8+/-0.1 BPS 2.7+/-0.0 fallback% 0.0+/-0.0 |
| paper (mean) fast | F1 47.4+/-0.0 BPS 8.7+/-0.0 fallback% 0.0+/-0.0 |
| paper (mean) hybrid | F1 48.2+/-0.0 BPS 8.1+/-0.0 fallback% 4.1+/-0.0 |
|
|
| -> a gap smaller than the spread is NOT evidence. Use paper_greedy |
| (deterministic) for tight comparisons. |
|
|
| INTEGRITY legacy/slow (51.23) vs paper_greedy/slow (51.23) -> OK (identical, as required) |
|
|
| B1 READ-OUT hybrid fallback rate (switch_to_ar/forward_steps): legacy 4.5% vs paper 4.1% (paper ~7-8%; the RELATIVE legacy>paper gap is the primary signal) |
| The 0.9 abnormality threshold is calibrated on the SHARPENED (T=0.7 + |
| top-p) distribution. Greedy's flatter softmax trips it far more often, so |
| legacy should be markedly higher. If they match, B1 has not taken effect. |
|
|
| ============================================================================== |
| PROMPT-SHAPE (n_cats) OOD -- profile: paper_greedy |
| ============================================================================== |
| Training chunks each image's categories into queries of 1..3. The eval set |
| asks for the image's FULL list, and 28.8% of eval queries have >3 categories |
| -- a prompt shape the model never saw. Caveat: images with more categories |
| also have more objects, so a drop may be difficulty, not novelty. |
|
|
| --- paper_greedy/slow --- |
| predictions: 4952 | eval queries: 4952 | unbucketed: 0 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| 1 | 1025 | 72.2 | 76.2 | 68.7 | 86.8 |
| 2 | 1522 | 63.2 | 64.8 | 61.8 | 81.0 |
| 3 | 980 | 54.1 | 57.1 | 51.3 | 73.6 |
| 4-5 | 935 | 49.9 | 54.6 | 45.9 | 70.4 |
| 6+ | 490 | 45.3 | 52.8 | 39.7 | 64.7 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| <=3 (IN) | 3527 | 58.4 | 60.0 | 56.9 | 77.0 |
| >3 (OOD) | 1425 | 46.3 | 51.2 | 42.3 | 66.7 |
|
|
| If '>3 (OOD)' is materially below '<=3 (IN)', the prompt-shape gap |
| is real: training never showed the model a 4+-category query. |
| Careful: bigger images have more categories AND more objects, so a |
| small drop may just be task difficulty, not prompt-shape OOD. |
|
|
| --- paper_greedy/fast --- |
| predictions: 4952 | eval queries: 4952 | unbucketed: 0 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| 1 | 1025 | 70.1 | 77.3 | 64.1 | 83.7 |
| 2 | 1522 | 61.2 | 64.7 | 58.1 | 78.5 |
| 3 | 980 | 52.3 | 57.9 | 47.8 | 70.8 |
| 4-5 | 935 | 48.0 | 56.2 | 41.9 | 67.2 |
| 6+ | 490 | 42.7 | 54.0 | 35.3 | 60.7 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| <=3 (IN) | 3527 | 56.3 | 60.1 | 53.0 | 74.1 |
| >3 (OOD) | 1425 | 44.5 | 53.5 | 38.1 | 63.4 |
|
|
| If '>3 (OOD)' is materially below '<=3 (IN)', the prompt-shape gap |
| is real: training never showed the model a 4+-category query. |
| Careful: bigger images have more categories AND more objects, so a |
| small drop may just be task difficulty, not prompt-shape OOD. |
|
|
| --- paper_greedy/hybrid --- |
| predictions: 4952 | eval queries: 4952 | unbucketed: 0 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| 1 | 1025 | 72.4 | 77.3 | 68.0 | 86.6 |
| 2 | 1522 | 62.3 | 65.1 | 59.7 | 80.1 |
| 3 | 980 | 53.5 | 58.9 | 49.0 | 72.5 |
| 4-5 | 935 | 48.8 | 56.8 | 42.8 | 68.4 |
| 6+ | 490 | 43.1 | 55.3 | 35.3 | 60.7 |
|
|
| n_cats | N | F1 | P | R | F1@.5 |
| -------------------------------------------------- |
| <=3 (IN) | 3527 | 57.7 | 61.0 | 54.8 | 76.1 |
| >3 (OOD) | 1425 | 45.1 | 54.0 | 38.7 | 64.2 |
|
|
| If '>3 (OOD)' is materially below '<=3 (IN)', the prompt-shape gap |
| is real: training never showed the model a 4+-category query. |
| Careful: bigger images have more categories AND more objects, so a |
| small drop may just be task difficulty, not prompt-shape OOD. |
|
|
|
|
