| # robopulsepp-v1: 7720 items |
|
|
| | model | answered | acc (95% CI) | balanced acc | macro-F1 | F1 pos / neg | AUROC | NLL | ECE | pred pos rate | |
| |---|---|---|---|---|---|---|---|---|---| |
| | rj4b-v02-final | 7720/7720 | 60.4 [57.9, 62.9] | 65.9 | 55.8 | 70.0 / 41.6 | 0.728 | 0.649 | 0.095 | 51.0 | |
| | JevAny-4B | 7720/7720 | 69.9 [67.7, 72.0] | 59.1 | 57.5 | 80.5 / 34.4 | 0.641 | 0.580 | 0.050 | 73.0 | |
|
|
| ## Timing |
|
|
| | model | GPUs | load s | wall s (loop) | mean / median / p95 ms per item | items/s | median input tokens | GPU-hours | |
| |---|---|---|---|---|---|---|---| |
| | rj4b-v02-final | 1x NVIDIA A100-SXM4-40GB | 4 | 2022 | 262 / 207 / 520 | 3.82 | 790 | 0.56 | |
| | JevAny-4B | 1x NVIDIA A100-SXM4-40GB | 7 | 2034 | 263 / 210 / 515 | 3.80 | 790 | 0.57 | |
|
|
| ## Interval level (majority of kept pairs) |
|
|
| | model | intervals | acc (95% CI) | macro-F1 | |
| |---|---|---|---| |
| | rj4b-v02-final | 1062 | 67.6 [64.6, 70.4] | 65.6 | |
| | JevAny-4B | 1062 | 67.3 [64.8, 70.0] | 58.7 | |
|
|
| ## Slices (accuracy / macro-F1, %) |
|
|
| | slice | n | rj4b-v02-final | JevAny-4B | |
| |---|---|---|---| |
| | n_views=1 | 5097 | 65.9 / 60.2 | 70.0 / 58.6 | |
| | n_views=2 | 1693 | 45.4 / 44.8 | 66.6 / 52.3 | |
| | n_views=3 | 930 | 57.2 / 49.3 | 75.5 / 59.9 | |
| |
| ## Paired: rj4b-v02-final − JevAny-4B |
| |
| - accuracy: -9.6 pts, 95% CI [-12.6, -6.2] |
| - macro-F1: -1.7 pts, 95% CI [-4.8, 1.8] |
| - McNemar: rj4b-v02-final only correct 995, JevAny-4B only correct 1733, exact p = 9.384e-46 |
| |
