## v2 glance | benchmark | cases | base model | TernaryPilot v2 | gain | |---|---:|---:|---:|---:| | ScreenSpot-v2 | 1,272 | 46.5% | **60.7%** | +14.2 | | ScreenSpot-v2, desktop + web only | 771 | 37.0% | **71.5%** | +34.5 | | ScreenSpot-Pro | 423 | 1.7% | **11.1%** | +9.4 | | GUI-360 grounding | 2,000 | 16.9% | **66.4%** | +49.5 | | GUI-360 actions | 1,192 | 0.5% | **42.2%** | +41.7 | ## v1 page | benchmark | cases | base model | v1 | v2 | |---|---:|---:|---:|---:| | ScreenSpot-v2 | 1,272 | 46.5% | **80.5%** | 60.7% | | ScreenSpot-Pro | 423 | 1.7% | 6.4% | **11.1%** | | GUI-360 grounding | 2,000 | 16.9% | 42.2% | **66.4%** | | GUI-360 actions | 1,192 | 0.5% | 17.3% | **42.2%** | ## ss2 by platform and type | platform / type | cases | base | v2 | |---|---:|---:|---:| | mobile-text | 290 | **68.3%** | 52.8% | | web-text | 234 | 40.6% | **82.9%** | | mobile-icon | 211 | **51.7%** | 32.2% | | web-icon | 203 | 30.5% | **66.5%** | | desktop-text | 194 | 39.2% | **73.7%** | | desktop-icon | 140 | 37.1% | **56.4%** | ## sspro by application group | application group | cases | base | v2 | |---|---:|---:|---:| | Creative | 195 | 1.0% | **9.7%** | | Dev | 80 | 1.2% | **7.5%** | | Office | 64 | 1.6% | **14.1%** | | Scientific | 50 | 6.0% | **26.0%** | | CAD | 34 | **0.0%** | **0.0%** | ## sspro by target type | target | cases | base | v2 | |---|---:|---:|---:| | text | 269 | 2.2% | **14.9%** | | icon | 154 | 0.6% | **4.5%** | ## g360g by app | app | cases | base | v2 | |---|---:|---:|---:| | word | 848 | 24.4% | **71.0%** | | excel | 620 | 9.7% | **63.2%** | | ppt | 532 | 13.2% | **62.8%** | ## g360a by action | action | cases | base | v2 | |---|---:|---:|---:| | click | 927 | 0.0% | **43.4%** | | type | 243 | 0.0% | **37.9%** | | wheel_mouse_input | 20 | 30.0% | **45.0%** | | drag | 2 | **0.0%** | **0.0%** | ## g360a by app | app | cases | base | v2 | |---|---:|---:|---:| | word | 491 | 0.2% | **44.2%** | | excel | 371 | 0.3% | **38.8%** | | ppt | 330 | 1.2% | **43.0%** | ## decider public | set | language | questions | base | **TernaryDecider v1** | gain | |---|---|---:|---:|---:|---:| | BoolQ (validation) | en | 1,000 | 83.2% | **89.8%** | +6.6 | | Civil Comments, toxic? (test) | en | 1,000 | 80.6% | **91.3%** | +10.7 | | MASSIVE intent, 8 options (test) | en | 1,000 | 91.2% | **97.5%** | +6.3 | | MASSIVE intent, 8 options (test) | it | 1,000 | 88.5% | **96.7%** | +8.2 | | PAWS-X (test) | de | 1,000 | 73.1% | **89.8%** | +16.7 | | PAWS-X (test) | en | 1,000 | 78.9% | **93.5%** | +14.6 | | PAWS-X (test) | es | 1,000 | 74.8% | **89.6%** | +14.8 | | PAWS-X (test) | fr | 1,000 | 73.4% | **89.8%** | +16.4 | | SQuAD 2.0, answerable? (validation) | en | 1,000 | 83.1% | **90.2%** | +7.1 | | XNLI (test) | de | 1,000 | 72.2% | **82.3%** | +10.1 | | XNLI (test) | en | 1,000 | 79.5% | **88.3%** | +8.8 | | XNLI (test) | es | 1,000 | 74.8% | **83.6%** | +8.8 | | XNLI (test) | fr | 1,000 | 71.3% | **82.5%** | +11.2 | | **all 13 sets** | 5 languages | 13,000 | 78.8% | **89.6%** | +10.8 | public overall base 78.8 lora 89.6 n 13000