File size: 3,154 Bytes
a3e7c27
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
## v2 glance
| benchmark | cases | base model | TernaryPilot v2 | gain |
|---|---:|---:|---:|---:|
| ScreenSpot-v2 | 1,272 | 46.5% | **60.7%** | +14.2 |
| ScreenSpot-v2, desktop + web only | 771 | 37.0% | **71.5%** | +34.5 |
| ScreenSpot-Pro | 423 | 1.7% | **11.1%** | +9.4 |
| GUI-360 grounding | 2,000 | 16.9% | **66.4%** | +49.5 |
| GUI-360 actions | 1,192 | 0.5% | **42.2%** | +41.7 |

## v1 page
| benchmark | cases | base model | v1 | v2 |
|---|---:|---:|---:|---:|
| ScreenSpot-v2 | 1,272 | 46.5% | **80.5%** | 60.7% |
| ScreenSpot-Pro | 423 | 1.7% | 6.4% | **11.1%** |
| GUI-360 grounding | 2,000 | 16.9% | 42.2% | **66.4%** |
| GUI-360 actions | 1,192 | 0.5% | 17.3% | **42.2%** |

## ss2 by platform and type
| platform / type | cases | base | v2 |
|---|---:|---:|---:|
| mobile-text | 290 | **68.3%** | 52.8% |
| web-text | 234 | 40.6% | **82.9%** |
| mobile-icon | 211 | **51.7%** | 32.2% |
| web-icon | 203 | 30.5% | **66.5%** |
| desktop-text | 194 | 39.2% | **73.7%** |
| desktop-icon | 140 | 37.1% | **56.4%** |

## sspro by application group
| application group | cases | base | v2 |
|---|---:|---:|---:|
| Creative | 195 | 1.0% | **9.7%** |
| Dev | 80 | 1.2% | **7.5%** |
| Office | 64 | 1.6% | **14.1%** |
| Scientific | 50 | 6.0% | **26.0%** |
| CAD | 34 | **0.0%** | **0.0%** |

## sspro by target type
| target | cases | base | v2 |
|---|---:|---:|---:|
| text | 269 | 2.2% | **14.9%** |
| icon | 154 | 0.6% | **4.5%** |

## g360g by app
| app | cases | base | v2 |
|---|---:|---:|---:|
| word | 848 | 24.4% | **71.0%** |
| excel | 620 | 9.7% | **63.2%** |
| ppt | 532 | 13.2% | **62.8%** |

## g360a by action
| action | cases | base | v2 |
|---|---:|---:|---:|
| click | 927 | 0.0% | **43.4%** |
| type | 243 | 0.0% | **37.9%** |
| wheel_mouse_input | 20 | 30.0% | **45.0%** |
| drag | 2 | **0.0%** | **0.0%** |

## g360a by app
| app | cases | base | v2 |
|---|---:|---:|---:|
| word | 491 | 0.2% | **44.2%** |
| excel | 371 | 0.3% | **38.8%** |
| ppt | 330 | 1.2% | **43.0%** |

## decider public
| set | language | questions | base | **TernaryDecider v1** | gain |
|---|---|---:|---:|---:|---:|
| BoolQ (validation) | en | 1,000 | 83.2% | **89.8%** | +6.6 |
| Civil Comments, toxic? (test) | en | 1,000 | 80.6% | **91.3%** | +10.7 |
| MASSIVE intent, 8 options (test) | en | 1,000 | 91.2% | **97.5%** | +6.3 |
| MASSIVE intent, 8 options (test) | it | 1,000 | 88.5% | **96.7%** | +8.2 |
| PAWS-X (test) | de | 1,000 | 73.1% | **89.8%** | +16.7 |
| PAWS-X (test) | en | 1,000 | 78.9% | **93.5%** | +14.6 |
| PAWS-X (test) | es | 1,000 | 74.8% | **89.6%** | +14.8 |
| PAWS-X (test) | fr | 1,000 | 73.4% | **89.8%** | +16.4 |
| SQuAD 2.0, answerable? (validation) | en | 1,000 | 83.1% | **90.2%** | +7.1 |
| XNLI (test) | de | 1,000 | 72.2% | **82.3%** | +10.1 |
| XNLI (test) | en | 1,000 | 79.5% | **88.3%** | +8.8 |
| XNLI (test) | es | 1,000 | 74.8% | **83.6%** | +8.8 |
| XNLI (test) | fr | 1,000 | 71.3% | **82.5%** | +11.2 |
| **all 13 sets** | 5 languages | 13,000 | 78.8% | **89.6%** | +10.8 |

public overall base 78.8 lora 89.6 n 13000