robopulsepp-v1: 7720 items
| model | answered | acc (95% CI) | balanced acc | macro-F1 | F1 pos / neg | AUROC | NLL | ECE | pred pos rate |
|---|---|---|---|---|---|---|---|---|---|
| rj4b-v02-final | 7720/7720 | 60.4 [57.9, 62.9] | 65.9 | 55.8 | 70.0 / 41.6 | 0.728 | 0.649 | 0.095 | 51.0 |
| JevAny-4B | 7720/7720 | 69.9 [67.7, 72.0] | 59.1 | 57.5 | 80.5 / 34.4 | 0.641 | 0.580 | 0.050 | 73.0 |
Timing
| model | GPUs | load s | wall s (loop) | mean / median / p95 ms per item | items/s | median input tokens | GPU-hours |
|---|---|---|---|---|---|---|---|
| rj4b-v02-final | 1x NVIDIA A100-SXM4-40GB | 4 | 2022 | 262 / 207 / 520 | 3.82 | 790 | 0.56 |
| JevAny-4B | 1x NVIDIA A100-SXM4-40GB | 7 | 2034 | 263 / 210 / 515 | 3.80 | 790 | 0.57 |
Interval level (majority of kept pairs)
| model | intervals | acc (95% CI) | macro-F1 |
|---|---|---|---|
| rj4b-v02-final | 1062 | 67.6 [64.6, 70.4] | 65.6 |
| JevAny-4B | 1062 | 67.3 [64.8, 70.0] | 58.7 |
Slices (accuracy / macro-F1, %)
| slice | n | rj4b-v02-final | JevAny-4B |
|---|---|---|---|
| n_views=1 | 5097 | 65.9 / 60.2 | 70.0 / 58.6 |
| n_views=2 | 1693 | 45.4 / 44.8 | 66.6 / 52.3 |
| n_views=3 | 930 | 57.2 / 49.3 | 75.5 / 59.9 |
Paired: rj4b-v02-final − JevAny-4B
- accuracy: -9.6 pts, 95% CI [-12.6, -6.2]
- macro-F1: -1.7 pts, 95% CI [-4.8, 1.8]
- McNemar: rj4b-v02-final only correct 995, JevAny-4B only correct 1733, exact p = 9.384e-46