File size: 3,154 Bytes
a3e7c27 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | ## v2 glance
| benchmark | cases | base model | TernaryPilot v2 | gain |
|---|---:|---:|---:|---:|
| ScreenSpot-v2 | 1,272 | 46.5% | **60.7%** | +14.2 |
| ScreenSpot-v2, desktop + web only | 771 | 37.0% | **71.5%** | +34.5 |
| ScreenSpot-Pro | 423 | 1.7% | **11.1%** | +9.4 |
| GUI-360 grounding | 2,000 | 16.9% | **66.4%** | +49.5 |
| GUI-360 actions | 1,192 | 0.5% | **42.2%** | +41.7 |
## v1 page
| benchmark | cases | base model | v1 | v2 |
|---|---:|---:|---:|---:|
| ScreenSpot-v2 | 1,272 | 46.5% | **80.5%** | 60.7% |
| ScreenSpot-Pro | 423 | 1.7% | 6.4% | **11.1%** |
| GUI-360 grounding | 2,000 | 16.9% | 42.2% | **66.4%** |
| GUI-360 actions | 1,192 | 0.5% | 17.3% | **42.2%** |
## ss2 by platform and type
| platform / type | cases | base | v2 |
|---|---:|---:|---:|
| mobile-text | 290 | **68.3%** | 52.8% |
| web-text | 234 | 40.6% | **82.9%** |
| mobile-icon | 211 | **51.7%** | 32.2% |
| web-icon | 203 | 30.5% | **66.5%** |
| desktop-text | 194 | 39.2% | **73.7%** |
| desktop-icon | 140 | 37.1% | **56.4%** |
## sspro by application group
| application group | cases | base | v2 |
|---|---:|---:|---:|
| Creative | 195 | 1.0% | **9.7%** |
| Dev | 80 | 1.2% | **7.5%** |
| Office | 64 | 1.6% | **14.1%** |
| Scientific | 50 | 6.0% | **26.0%** |
| CAD | 34 | **0.0%** | **0.0%** |
## sspro by target type
| target | cases | base | v2 |
|---|---:|---:|---:|
| text | 269 | 2.2% | **14.9%** |
| icon | 154 | 0.6% | **4.5%** |
## g360g by app
| app | cases | base | v2 |
|---|---:|---:|---:|
| word | 848 | 24.4% | **71.0%** |
| excel | 620 | 9.7% | **63.2%** |
| ppt | 532 | 13.2% | **62.8%** |
## g360a by action
| action | cases | base | v2 |
|---|---:|---:|---:|
| click | 927 | 0.0% | **43.4%** |
| type | 243 | 0.0% | **37.9%** |
| wheel_mouse_input | 20 | 30.0% | **45.0%** |
| drag | 2 | **0.0%** | **0.0%** |
## g360a by app
| app | cases | base | v2 |
|---|---:|---:|---:|
| word | 491 | 0.2% | **44.2%** |
| excel | 371 | 0.3% | **38.8%** |
| ppt | 330 | 1.2% | **43.0%** |
## decider public
| set | language | questions | base | **TernaryDecider v1** | gain |
|---|---|---:|---:|---:|---:|
| BoolQ (validation) | en | 1,000 | 83.2% | **89.8%** | +6.6 |
| Civil Comments, toxic? (test) | en | 1,000 | 80.6% | **91.3%** | +10.7 |
| MASSIVE intent, 8 options (test) | en | 1,000 | 91.2% | **97.5%** | +6.3 |
| MASSIVE intent, 8 options (test) | it | 1,000 | 88.5% | **96.7%** | +8.2 |
| PAWS-X (test) | de | 1,000 | 73.1% | **89.8%** | +16.7 |
| PAWS-X (test) | en | 1,000 | 78.9% | **93.5%** | +14.6 |
| PAWS-X (test) | es | 1,000 | 74.8% | **89.6%** | +14.8 |
| PAWS-X (test) | fr | 1,000 | 73.4% | **89.8%** | +16.4 |
| SQuAD 2.0, answerable? (validation) | en | 1,000 | 83.1% | **90.2%** | +7.1 |
| XNLI (test) | de | 1,000 | 72.2% | **82.3%** | +10.1 |
| XNLI (test) | en | 1,000 | 79.5% | **88.3%** | +8.8 |
| XNLI (test) | es | 1,000 | 74.8% | **83.6%** | +8.8 |
| XNLI (test) | fr | 1,000 | 71.3% | **82.5%** | +11.2 |
| **all 13 sets** | 5 languages | 13,000 | 78.8% | **89.6%** | +10.8 |
public overall base 78.8 lora 89.6 n 13000
|