|
Download eval/tables.md from KeytechSRL/TernaryPilot-v2-Computer-Browser-Use-LoRA: direct link, hf CLI and curl.
- Browser
- Download file 3.15 kB
-
https://huggingface.co/KeytechSRL/TernaryPilot-v2-Computer-Browser-Use-LoRA/resolve/main/eval/tables.md
- Command line
-
hf download hf://KeytechSRL/TernaryPilot-v2-Computer-Browser-Use-LoRA/eval/tables.md
-
curl -L -o tables.md https://huggingface.co/KeytechSRL/TernaryPilot-v2-Computer-Browser-Use-LoRA/resolve/main/eval/tables.md
3.15 kB
| ## v2 glance | |
| | benchmark | cases | base model | TernaryPilot v2 | gain | | |
| |---|---:|---:|---:|---:| | |
| | ScreenSpot-v2 | 1,272 | 46.5% | **60.7%** | +14.2 | | |
| | ScreenSpot-v2, desktop + web only | 771 | 37.0% | **71.5%** | +34.5 | | |
| | ScreenSpot-Pro | 423 | 1.7% | **11.1%** | +9.4 | | |
| | GUI-360 grounding | 2,000 | 16.9% | **66.4%** | +49.5 | | |
| | GUI-360 actions | 1,192 | 0.5% | **42.2%** | +41.7 | | |
| ## v1 page | |
| | benchmark | cases | base model | v1 | v2 | | |
| |---|---:|---:|---:|---:| | |
| | ScreenSpot-v2 | 1,272 | 46.5% | **80.5%** | 60.7% | | |
| | ScreenSpot-Pro | 423 | 1.7% | 6.4% | **11.1%** | | |
| | GUI-360 grounding | 2,000 | 16.9% | 42.2% | **66.4%** | | |
| | GUI-360 actions | 1,192 | 0.5% | 17.3% | **42.2%** | | |
| ## ss2 by platform and type | |
| | platform / type | cases | base | v2 | | |
| |---|---:|---:|---:| | |
| | mobile-text | 290 | **68.3%** | 52.8% | | |
| | web-text | 234 | 40.6% | **82.9%** | | |
| | mobile-icon | 211 | **51.7%** | 32.2% | | |
| | web-icon | 203 | 30.5% | **66.5%** | | |
| | desktop-text | 194 | 39.2% | **73.7%** | | |
| | desktop-icon | 140 | 37.1% | **56.4%** | | |
| ## sspro by application group | |
| | application group | cases | base | v2 | | |
| |---|---:|---:|---:| | |
| | Creative | 195 | 1.0% | **9.7%** | | |
| | Dev | 80 | 1.2% | **7.5%** | | |
| | Office | 64 | 1.6% | **14.1%** | | |
| | Scientific | 50 | 6.0% | **26.0%** | | |
| | CAD | 34 | **0.0%** | **0.0%** | | |
| ## sspro by target type | |
| | target | cases | base | v2 | | |
| |---|---:|---:|---:| | |
| | text | 269 | 2.2% | **14.9%** | | |
| | icon | 154 | 0.6% | **4.5%** | | |
| ## g360g by app | |
| | app | cases | base | v2 | | |
| |---|---:|---:|---:| | |
| | word | 848 | 24.4% | **71.0%** | | |
| | excel | 620 | 9.7% | **63.2%** | | |
| | ppt | 532 | 13.2% | **62.8%** | | |
| ## g360a by action | |
| | action | cases | base | v2 | | |
| |---|---:|---:|---:| | |
| | click | 927 | 0.0% | **43.4%** | | |
| | type | 243 | 0.0% | **37.9%** | | |
| | wheel_mouse_input | 20 | 30.0% | **45.0%** | | |
| | drag | 2 | **0.0%** | **0.0%** | | |
| ## g360a by app | |
| | app | cases | base | v2 | | |
| |---|---:|---:|---:| | |
| | word | 491 | 0.2% | **44.2%** | | |
| | excel | 371 | 0.3% | **38.8%** | | |
| | ppt | 330 | 1.2% | **43.0%** | | |
| ## decider public | |
| | set | language | questions | base | **TernaryDecider v1** | gain | | |
| |---|---|---:|---:|---:|---:| | |
| | BoolQ (validation) | en | 1,000 | 83.2% | **89.8%** | +6.6 | | |
| | Civil Comments, toxic? (test) | en | 1,000 | 80.6% | **91.3%** | +10.7 | | |
| | MASSIVE intent, 8 options (test) | en | 1,000 | 91.2% | **97.5%** | +6.3 | | |
| | MASSIVE intent, 8 options (test) | it | 1,000 | 88.5% | **96.7%** | +8.2 | | |
| | PAWS-X (test) | de | 1,000 | 73.1% | **89.8%** | +16.7 | | |
| | PAWS-X (test) | en | 1,000 | 78.9% | **93.5%** | +14.6 | | |
| | PAWS-X (test) | es | 1,000 | 74.8% | **89.6%** | +14.8 | | |
| | PAWS-X (test) | fr | 1,000 | 73.4% | **89.8%** | +16.4 | | |
| | SQuAD 2.0, answerable? (validation) | en | 1,000 | 83.1% | **90.2%** | +7.1 | | |
| | XNLI (test) | de | 1,000 | 72.2% | **82.3%** | +10.1 | | |
| | XNLI (test) | en | 1,000 | 79.5% | **88.3%** | +8.8 | | |
| | XNLI (test) | es | 1,000 | 74.8% | **83.6%** | +8.8 | | |
| | XNLI (test) | fr | 1,000 | 71.3% | **82.5%** | +11.2 | | |
| | **all 13 sets** | 5 languages | 13,000 | 78.8% | **89.6%** | +10.8 | | |
| public overall base 78.8 lora 89.6 n 13000 | |