adriandj3's picture
Benchmark results: ScreenSpot-v2, ScreenSpot-Pro, GUI-360 grounding and actions
a3e7c27 verified
|
Raw History Blame Contribute Delete
3.15 kB

v2 glance

benchmark cases base model TernaryPilot v2 gain
ScreenSpot-v2 1,272 46.5% 60.7% +14.2
ScreenSpot-v2, desktop + web only 771 37.0% 71.5% +34.5
ScreenSpot-Pro 423 1.7% 11.1% +9.4
GUI-360 grounding 2,000 16.9% 66.4% +49.5
GUI-360 actions 1,192 0.5% 42.2% +41.7

v1 page

benchmark cases base model v1 v2
ScreenSpot-v2 1,272 46.5% 80.5% 60.7%
ScreenSpot-Pro 423 1.7% 6.4% 11.1%
GUI-360 grounding 2,000 16.9% 42.2% 66.4%
GUI-360 actions 1,192 0.5% 17.3% 42.2%

ss2 by platform and type

platform / type cases base v2
mobile-text 290 68.3% 52.8%
web-text 234 40.6% 82.9%
mobile-icon 211 51.7% 32.2%
web-icon 203 30.5% 66.5%
desktop-text 194 39.2% 73.7%
desktop-icon 140 37.1% 56.4%

sspro by application group

application group cases base v2
Creative 195 1.0% 9.7%
Dev 80 1.2% 7.5%
Office 64 1.6% 14.1%
Scientific 50 6.0% 26.0%
CAD 34 0.0% 0.0%

sspro by target type

target cases base v2
text 269 2.2% 14.9%
icon 154 0.6% 4.5%

g360g by app

app cases base v2
word 848 24.4% 71.0%
excel 620 9.7% 63.2%
ppt 532 13.2% 62.8%

g360a by action

action cases base v2
click 927 0.0% 43.4%
type 243 0.0% 37.9%
wheel_mouse_input 20 30.0% 45.0%
drag 2 0.0% 0.0%

g360a by app

app cases base v2
word 491 0.2% 44.2%
excel 371 0.3% 38.8%
ppt 330 1.2% 43.0%

decider public

set language questions base TernaryDecider v1 gain
BoolQ (validation) en 1,000 83.2% 89.8% +6.6
Civil Comments, toxic? (test) en 1,000 80.6% 91.3% +10.7
MASSIVE intent, 8 options (test) en 1,000 91.2% 97.5% +6.3
MASSIVE intent, 8 options (test) it 1,000 88.5% 96.7% +8.2
PAWS-X (test) de 1,000 73.1% 89.8% +16.7
PAWS-X (test) en 1,000 78.9% 93.5% +14.6
PAWS-X (test) es 1,000 74.8% 89.6% +14.8
PAWS-X (test) fr 1,000 73.4% 89.8% +16.4
SQuAD 2.0, answerable? (validation) en 1,000 83.1% 90.2% +7.1
XNLI (test) de 1,000 72.2% 82.3% +10.1
XNLI (test) en 1,000 79.5% 88.3% +8.8
XNLI (test) es 1,000 74.8% 83.6% +8.8
XNLI (test) fr 1,000 71.3% 82.5% +11.2
all 13 sets 5 languages 13,000 78.8% 89.6% +10.8

public overall base 78.8 lora 89.6 n 13000