[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1199,668)
[1] Button "Sound" center=(1831,534)
[2] Button "Displays" center=(1386,953)
[3] Button "Network" center=(565,655)
[4] CheckBox "Do Not Disturb" center=(1701,870)
[5] Button "Privacy & Security" center=(1431,101)
[6] CheckBox "Bluetooth" center=(1585,258)
[7] Slider "Brightness" center=(1348,834)
[8] CheckBox "Wi-Fi" center=(156,307)

Instruction: click network

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(618,915)
[1] Button "Commit" center=(1120,164)
[2] Button "Format" center=(255,991)
[3] Button "Open" center=(1432,236)
[4] Button "Find" center=(161,135)

Instruction: press format

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1652,167)
[1] CheckBox "Wi-Fi" center=(380,514)
[2] Slider "Brightness" center=(792,648)
[3] PopUpButton "Appearance" center=(205,959)
[4] Button "Sound" center=(1712,526)
[5] CheckBox "Do Not Disturb" center=(156,62)
[6] Button "Privacy & Security" center=(543,317)
[7] CheckBox "Bluetooth" center=(535,965)
[8] Button "Network" center=(1594,511)

Instruction: select brightness

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(227,936)
[1] Button "Commit" center=(1369,727)
[2] Button "Run" center=(357,250)
[3] Button "Open" center=(292,114)
[4] Button "New File" center=(1722,384)
[5] Button "Format" center=(568,151)
[6] Button "Save" center=(1344,108)

Instruction: click save

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1549,451)
[1] Slider "Volume" center=(1370,637)
[2] Button "Fullscreen" center=(457,94)
[3] Button "Shuffle" center=(1645,73)
[4] Button "Previous" center=(778,932)

Instruction: select next

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1711,244)
[1] TextField "Email" center=(1187,311)
[2] Button "Sign in" center=(1411,179)
[3] Link "Forgot password" center=(687,833)
[4] Button "Create account" center=(909,162)
[5] TextField "Password" center=(293,892)

Instruction: type into password

[Assistant] {"action":"type","id":5,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1662,758)
[1] TextField "Email" center=(214,586)
[2] TextField "Password" center=(603,911)
[3] Button "Sign in" center=(1733,377)
[4] Link "Forgot password" center=(684,254)
[5] CheckBox "Remember me" center=(938,821)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1133,149)
[1] Link "Account" center=(592,911)
[2] Button "Checkout" center=(1293,634)
[3] Button "Filters" center=(1713,912)
[4] SearchField "Search products" center=(1385,358)

Instruction: press account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1411,794)
[1] CheckBox "Wi-Fi" center=(1327,381)
[2] PopUpButton "Appearance" center=(1833,700)
[3] CheckBox "Bluetooth" center=(391,560)
[4] Button "Privacy & Security" center=(1100,70)
[5] CheckBox "Do Not Disturb" center=(728,68)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1654,848)
[1] Button "Pause" center=(1724,341)
[2] Button "Fullscreen" center=(309,406)
[3] Button "Previous" center=(410,327)
[4] Button "Play" center=(1481,758)
[5] Slider "Volume" center=(1009,544)

Instruction: make it full screen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(618,866)
[1] Button "Filters" center=(148,89)
[2] Button "Add to cart" center=(930,48)
[3] Button "Buy now" center=(831,621)
[4] SearchField "Search products" center=(1129,934)
[5] Button "Checkout" center=(1086,986)

Instruction: add this to my cart

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1255,273)
[1] Button "Pause" center=(1648,947)
[2] Button "Play" center=(1710,615)
[3] Slider "Volume" center=(760,964)
[4] Button "Next" center=(1374,926)
[5] Button "Previous" center=(634,708)

Instruction: go fullscreen

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(129,134)
[1] Button "Run inference" center=(1587,484)
[2] Button "Bookmark" center=(163,627)
[3] SearchField "Search or enter address" center=(1747,669)
[4] Button "Downloads" center=(1789,873)
[5] Link "Sign in" center=(495,184)
[6] Link "Docs" center=(1765,687)
[7] Button "Reload" center=(1562,298)
[8] Link "Pricing" center=(999,71)
[9] Button "Extensions" center=(941,617)
[10] Button "New tab" center=(333,446)

Instruction: log in

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(485,174)
[1] Slider "Volume" center=(801,100)
[2] Button "Shuffle" center=(1369,885)
[3] Button "Play" center=(513,807)
[4] Button "Pause" center=(1433,749)

Instruction: press shuffle

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1180,626)
[1] Button "Play" center=(772,199)
[2] Button "Pause" center=(1444,986)
[3] Button "Next" center=(537,477)
[4] Button "Fullscreen" center=(225,845)
[5] Slider "Volume" center=(1783,551)

Instruction: tap pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1541,559)
[1] Button "Pause" center=(1833,158)
[2] Button "Previous" center=(394,499)
[3] Slider "Volume" center=(1525,237)
[4] Button "Shuffle" center=(1526,151)
[5] Button "Fullscreen" center=(1049,766)

Instruction: press fullscreen

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(467,443)
[1] Button "Find" center=(1806,721)
[2] Button "Terminal" center=(304,527)
[3] Button "Save" center=(833,144)
[4] Button "New File" center=(985,458)
[5] Button "Commit" center=(1710,317)
[6] TextField "Search files" center=(1495,71)
[7] Button "Run" center=(1609,722)
[8] Button "Open" center=(606,81)

Instruction: type into search files

[Assistant] {"action":"type","id":6,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(513,556)
[1] Link "Forgot password" center=(202,1019)
[2] Button "Create account" center=(291,1027)
[3] TextField "Email" center=(72,398)
[4] Button "Sign in" center=(53,1000)
[5] CheckBox "Remember me" center=(318,764)

Instruction: i forgot my password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1667,319)
[1] Button "Reload" center=(413,251)
[2] Button "Extensions" center=(1319,255)
[3] Button "Bookmark" center=(691,755)
[4] Link "Docs" center=(582,262)
[5] Button "Run inference" center=(1580,274)
[6] Button "Go forward" center=(1719,694)
[7] Link "Sign in" center=(1842,507)
[8] Button "Downloads" center=(436,788)
[9] Button "New tab" center=(1652,176)
[10] SearchField "Search or enter address" center=(1505,491)
[11] Link "Pricing" center=(1105,1026)

Instruction: press go forward

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(436,955)
[1] Link "Pricing" center=(1326,681)
[2] Button "Extensions" center=(554,1016)
[3] Button "New tab" center=(1467,574)
[4] Link "Docs" center=(829,1019)
[5] Link "Sign in" center=(1836,622)
[6] Button "Bookmark" center=(1404,846)
[7] Button "Reload" center=(823,1034)
[8] Button "Run inference" center=(1658,988)

Instruction: tap new tab

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(1178,504)
[1] Link "Sign in" center=(494,71)
[2] SearchField "Search or enter address" center=(1402,869)
[3] Link "Docs" center=(1203,410)
[4] Button "Run inference" center=(145,527)
[5] Button "Reload" center=(1315,254)

Instruction: open the docs

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1411,974)
[1] Link "Sign in" center=(804,661)
[2] Link "Docs" center=(1348,889)
[3] SearchField "Search or enter address" center=(1873,753)
[4] Button "Extensions" center=(94,578)
[5] Button "Downloads" center=(345,461)
[6] Button "Go back" center=(1154,773)
[7] Button "New tab" center=(77,40)
[8] Button "Bookmark" center=(1284,100)
[9] Link "Pricing" center=(1006,194)
[10] Button "Run inference" center=(393,68)
[11] Button "Reload" center=(1188,710)

Instruction: click sign in

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(886,830)
[1] Button "Wishlist" center=(117,174)
[2] Button "Add to cart" center=(520,254)
[3] Button "Buy now" center=(194,872)
[4] Link "Account" center=(711,870)
[5] Button "Checkout" center=(1311,968)
[6] SearchField "Search products" center=(1636,955)

Instruction: go to checkout

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(177,516)
[1] Button "Run inference" center=(728,52)
[2] SearchField "Search or enter address" center=(386,1017)
[3] Button "New tab" center=(1668,791)
[4] Link "Sign in" center=(612,164)
[5] Link "Docs" center=(492,995)
[6] Button "Go forward" center=(89,88)
[7] Button "Bookmark" center=(948,565)
[8] Button "Go back" center=(1320,599)

Instruction: show documentation

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(661,383)
[1] TextField "Password" center=(128,343)
[2] TextField "Email" center=(1306,888)
[3] CheckBox "Remember me" center=(936,220)
[4] Link "Forgot password" center=(548,440)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(253,593)
[1] Slider "Volume" center=(295,334)
[2] Button "Next" center=(1611,746)
[3] Button "Shuffle" center=(315,80)
[4] Button "Fullscreen" center=(671,852)
[5] Button "Pause" center=(502,706)

Instruction: select pause

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(258,475)
[1] Button "Go back" center=(1645,939)
[2] Button "Downloads" center=(1504,810)
[3] Link "Pricing" center=(1855,426)
[4] Link "Sign in" center=(959,302)
[5] Button "Run inference" center=(108,492)
[6] Button "Bookmark" center=(1178,955)
[7] Button "Go forward" center=(1430,613)
[8] Button "Reload" center=(1416,938)

Instruction: select reload

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1362,870)
[1] SearchField "Search products" center=(934,508)
[2] Link "Account" center=(1352,194)
[3] Button "Add to cart" center=(918,336)
[4] Button "Filters" center=(227,556)
[5] Button "Wishlist" center=(956,679)

Instruction: tap filters

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(367,325)
[1] Button "Add to cart" center=(1217,257)
[2] Button "Filters" center=(1839,903)
[3] Button "Wishlist" center=(264,481)
[4] Button "Buy now" center=(1556,312)

Instruction: press buy now

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(639,80)
[1] Button "Sign in" center=(1028,683)
[2] CheckBox "Remember me" center=(1246,245)
[3] TextField "Email" center=(836,392)
[4] Button "Create account" center=(179,861)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1747,1034)
[1] TextField "Email" center=(1624,836)
[2] CheckBox "Remember me" center=(491,724)
[3] TextField "Password" center=(1525,202)
[4] Link "Forgot password" center=(1001,1015)

Instruction: tap remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1391,558)
[1] Button "New tab" center=(1806,629)
[2] Button "Bookmark" center=(754,846)
[3] Button "Run inference" center=(1242,157)
[4] SearchField "Search or enter address" center=(792,279)
[5] Button "Go forward" center=(330,762)
[6] Button "Go back" center=(664,225)
[7] Button "Extensions" center=(737,834)
[8] Link "Pricing" center=(835,429)
[9] Link "Docs" center=(80,47)
[10] Button "Downloads" center=(1228,592)
[11] Button "Reload" center=(1646,781)

Instruction: tap go forward

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(454,963)
[1] Link "Account" center=(590,114)
[2] Button "Checkout" center=(1721,972)
[3] Button "Add to cart" center=(543,881)
[4] SearchField "Search products" center=(1344,455)

Instruction: press account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1683,491)
[1] Button "Buy now" center=(1711,1028)
[2] Button "Filters" center=(96,794)
[3] Link "Account" center=(1060,860)
[4] Button "Wishlist" center=(129,476)
[5] Button "Checkout" center=(130,65)
[6] SearchField "Search products" center=(1061,318)

Instruction: press add to cart

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1866,1012)
[1] CheckBox "Bluetooth" center=(621,241)
[2] Button "Sound" center=(833,613)
[3] Button "Privacy & Security" center=(936,124)
[4] Button "Displays" center=(1392,785)
[5] CheckBox "Do Not Disturb" center=(1344,885)
[6] PopUpButton "Appearance" center=(1775,964)

Instruction: tap displays

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(807,893)
[1] Button "Terminal" center=(1067,487)
[2] Button "Save" center=(918,164)
[3] Button "Commit" center=(1112,192)
[4] Button "Format" center=(634,818)
[5] Button "Run" center=(1144,781)
[6] TextField "Search files" center=(1062,354)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(896,252)
[1] Button "Commit" center=(1491,304)
[2] Button "Find" center=(1445,625)
[3] Button "New File" center=(392,124)
[4] Button "Save" center=(1336,204)
[5] Button "Terminal" center=(319,249)

Instruction: press save

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1511,319)
[1] Link "Docs" center=(1087,304)
[2] Button "Reload" center=(1638,399)
[3] Button "New tab" center=(899,504)
[4] Button "Go forward" center=(1668,534)
[5] Link "Sign in" center=(185,412)
[6] Button "Extensions" center=(1739,887)
[7] Link "Pricing" center=(1405,312)
[8] SearchField "Search or enter address" center=(266,791)

Instruction: tap extensions

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(941,708)
[1] Button "Go forward" center=(860,543)
[2] Button "Reload" center=(1651,878)
[3] Link "Pricing" center=(1406,1008)
[4] SearchField "Search or enter address" center=(961,809)
[5] Button "Go back" center=(1708,449)
[6] Button "Extensions" center=(802,876)
[7] Button "Bookmark" center=(1010,776)
[8] Link "Sign in" center=(926,434)
[9] Link "Docs" center=(860,921)

Instruction: click extensions

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(932,358)
[1] SearchField "Search products" center=(714,941)
[2] Button "Wishlist" center=(696,988)
[3] Button "Filters" center=(418,112)
[4] Button "Add to cart" center=(1045,412)
[5] Button "Buy now" center=(167,334)
[6] Button "Checkout" center=(1213,678)

Instruction: press filters

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1556,322)
[1] Button "Checkout" center=(330,210)
[2] Link "Account" center=(50,481)
[3] Button "Buy now" center=(197,743)
[4] Button "Wishlist" center=(1473,607)
[5] Button "Filters" center=(780,748)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(166,429)
[1] Button "Displays" center=(1679,530)
[2] CheckBox "Do Not Disturb" center=(326,352)
[3] Button "Network" center=(889,921)
[4] CheckBox "Bluetooth" center=(924,247)

Instruction: press displays

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(262,983)
[1] Button "Reload" center=(269,860)
[2] Button "Run inference" center=(1819,560)
[3] Link "Docs" center=(578,724)
[4] Link "Pricing" center=(83,450)
[5] Button "Downloads" center=(764,297)

Instruction: select run inference

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(354,623)
[1] Button "Filters" center=(844,146)
[2] SearchField "Search products" center=(743,116)
[3] Button "Buy now" center=(606,237)
[4] Link "Account" center=(883,663)
[5] Button "Checkout" center=(976,998)

Instruction: press buy now

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1384,287)
[1] Link "Forgot password" center=(712,397)
[2] Button "Sign in" center=(853,1018)
[3] TextField "Password" center=(908,975)
[4] Button "Create account" center=(1304,482)

Instruction: i forgot my password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(990,709)
[1] Button "Buy now" center=(1133,985)
[2] Button "Filters" center=(328,374)
[3] Button "Wishlist" center=(1437,436)
[4] Button "Checkout" center=(134,749)

Instruction: click checkout

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1003,471)
[1] Button "Shuffle" center=(1481,1011)
[2] Slider "Volume" center=(384,646)
[3] Button "Fullscreen" center=(441,442)
[4] Button "Next" center=(1681,946)
[5] Button "Pause" center=(1440,998)
[6] Button "Previous" center=(1139,330)

Instruction: click next

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1381,992)
[1] Button "New File" center=(1712,909)
[2] Button "Format" center=(1052,872)
[3] Button "Save" center=(197,661)
[4] Button "Find" center=(1594,99)
[5] Button "Commit" center=(1431,845)

Instruction: press format

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(760,788)
[1] Button "Extensions" center=(1671,431)
[2] Button "New tab" center=(391,886)
[3] Button "Downloads" center=(664,875)
[4] SearchField "Search or enter address" center=(1520,317)
[5] Button "Go back" center=(894,745)
[6] Button "Run inference" center=(414,194)
[7] Button "Go forward" center=(113,136)
[8] Button "Reload" center=(840,384)

Instruction: press downloads

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(426,789)
[1] Button "Wishlist" center=(309,432)
[2] SearchField "Search products" center=(302,370)
[3] Button "Add to cart" center=(532,729)
[4] Button "Checkout" center=(1142,133)
[5] Link "Account" center=(606,54)

Instruction: tap account

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(165,184)
[1] TextField "Password" center=(1850,329)
[2] TextField "Email" center=(1123,344)
[3] CheckBox "Remember me" center=(428,811)
[4] Link "Forgot password" center=(749,90)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(592,44)
[1] Link "Pricing" center=(659,990)
[2] Button "Downloads" center=(1653,247)
[3] Link "Sign in" center=(216,64)
[4] Link "Docs" center=(1309,52)
[5] Button "Go back" center=(998,535)
[6] Button "New tab" center=(237,725)
[7] Button "Reload" center=(1181,218)
[8] SearchField "Search or enter address" center=(522,733)

Instruction: press sign in

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(172,488)
[1] Button "Format" center=(300,494)
[2] Button "Open" center=(771,527)
[3] Button "Find" center=(501,385)
[4] Button "New File" center=(696,357)
[5] Button "Commit" center=(1181,988)
[6] TextField "Search files" center=(258,335)
[7] Button "Run" center=(998,1000)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(585,223)
[1] Button "Save" center=(987,764)
[2] TextField "Search files" center=(144,892)
[3] Button "Find" center=(324,1024)
[4] Button "Terminal" center=(181,394)
[5] Button "Format" center=(160,996)
[6] Button "New File" center=(1354,140)
[7] Button "Open" center=(465,945)
[8] Button "Commit" center=(561,890)

Instruction: search

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1483,945)
[1] CheckBox "Wi-Fi" center=(1744,618)
[2] PopUpButton "Appearance" center=(183,891)
[3] Button "Network" center=(309,657)
[4] CheckBox "Do Not Disturb" center=(1811,658)
[5] CheckBox "Bluetooth" center=(1810,84)

Instruction: open privacy settings

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(629,861)
[1] Button "Network" center=(410,126)
[2] PopUpButton "Appearance" center=(1657,797)
[3] CheckBox "Do Not Disturb" center=(223,471)
[4] Button "Privacy & Security" center=(1419,231)

Instruction: tap wi-fi

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(605,894)
[1] Slider "Volume" center=(554,234)
[2] Button "Previous" center=(846,297)
[3] Button "Play" center=(1835,260)
[4] Button "Next" center=(843,313)
[5] Button "Fullscreen" center=(634,297)
[6] Button "Shuffle" center=(470,883)

Instruction: tap shuffle

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(155,514)
[1] Button "Filters" center=(1328,193)
[2] Link "Account" center=(378,981)
[3] SearchField "Search products" center=(1861,1020)
[4] Button "Buy now" center=(944,447)
[5] Button "Checkout" center=(1510,552)
[6] Button "Add to cart" center=(529,366)

Instruction: click buy now

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(155,656)
[1] Button "Pause" center=(487,232)
[2] Button "Shuffle" center=(1146,808)
[3] Button "Next" center=(802,263)
[4] Button "Fullscreen" center=(120,54)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(864,418)
[1] Link "Sign in" center=(1169,463)
[2] Button "Go forward" center=(286,530)
[3] Button "Extensions" center=(379,606)
[4] Button "Run inference" center=(1387,356)
[5] Link "Pricing" center=(987,709)
[6] Button "New tab" center=(96,287)
[7] Button "Downloads" center=(632,320)
[8] Button "Reload" center=(1685,471)

Instruction: press extensions

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(724,343)
[1] Button "Pause" center=(1573,460)
[2] Slider "Volume" center=(1544,245)
[3] Button "Shuffle" center=(721,163)
[4] Button "Fullscreen" center=(993,360)

Instruction: click volume

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1481,876)
[1] Button "Buy now" center=(1199,409)
[2] Button "Filters" center=(1227,458)
[3] SearchField "Search products" center=(710,848)
[4] Button "Wishlist" center=(883,52)
[5] Button "Add to cart" center=(125,844)

Instruction: press buy now

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(220,48)
[1] Button "Commit" center=(1194,203)
[2] TextField "Search files" center=(416,1034)
[3] Button "Terminal" center=(1857,170)
[4] Button "Run" center=(707,134)
[5] Button "Find" center=(1539,311)

Instruction: select terminal

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(132,319)
[1] Button "Next" center=(183,331)
[2] Button "Shuffle" center=(334,998)
[3] Slider "Volume" center=(919,354)
[4] Button "Fullscreen" center=(478,98)
[5] Button "Previous" center=(418,673)

Instruction: play

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1347,815)
[1] Button "Terminal" center=(344,358)
[2] Button "Run" center=(1867,727)
[3] Button "Save" center=(429,660)
[4] Button "Open" center=(352,210)
[5] Button "Commit" center=(1283,693)
[6] Button "Find" center=(796,73)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1426,701)
[1] Link "Account" center=(1407,645)
[2] Button "Add to cart" center=(1698,376)
[3] Button "Checkout" center=(445,657)
[4] SearchField "Search products" center=(151,983)
[5] Button "Buy now" center=(1157,519)

Instruction: select account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(661,448)
[1] Button "Run" center=(922,441)
[2] Button "Open" center=(812,662)
[3] TextField "Search files" center=(672,744)
[4] Button "Save" center=(1190,267)
[5] Button "Find" center=(923,991)
[6] Button "Format" center=(1441,879)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1562,404)
[1] Button "Play" center=(1532,56)
[2] Button "Fullscreen" center=(1124,374)
[3] Button "Previous" center=(726,633)
[4] Slider "Volume" center=(410,382)
[5] Button "Pause" center=(1652,688)
[6] Button "Next" center=(914,209)

Instruction: next song

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1708,202)
[1] Button "Checkout" center=(1039,290)
[2] Button "Wishlist" center=(1334,1008)
[3] SearchField "Search products" center=(662,775)
[4] Link "Account" center=(1735,943)
[5] Button "Buy now" center=(885,983)
[6] Button "Filters" center=(1146,879)

Instruction: press buy now

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(892,447)
[1] Button "New File" center=(255,87)
[2] Button "Commit" center=(763,142)
[3] Button "Terminal" center=(650,653)
[4] Button "Format" center=(1532,801)
[5] Button "Open" center=(831,129)
[6] Button "Run" center=(1159,89)
[7] Button "Find" center=(815,930)

Instruction: make a commit

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(553,658)
[1] Button "Play" center=(1871,300)
[2] Button "Next" center=(1861,59)
[3] Button "Previous" center=(1584,291)
[4] Button "Fullscreen" center=(1433,997)
[5] Slider "Volume" center=(1670,817)

Instruction: make it full screen

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(136,910)
[1] Button "Buy now" center=(1189,1038)
[2] SearchField "Search products" center=(1146,273)
[3] Button "Add to cart" center=(1464,711)
[4] Button "Wishlist" center=(521,929)
[5] Button "Checkout" center=(247,597)

Instruction: click filters

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1051,656)
[1] Button "New File" center=(1647,293)
[2] Button "Format" center=(1333,1033)
[3] Button "Save" center=(1289,604)
[4] Button "Terminal" center=(1692,42)
[5] Button "Commit" center=(226,834)
[6] TextField "Search files" center=(1419,937)
[7] Button "Find" center=(363,203)

Instruction: click open

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1227,141)
[1] Button "Format" center=(785,195)
[2] TextField "Search files" center=(482,114)
[3] Button "Save" center=(439,288)
[4] Button "Terminal" center=(1706,885)
[5] Button "New File" center=(848,692)

Instruction: click format

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(133,227)
[1] Button "Sign in" center=(540,165)
[2] CheckBox "Remember me" center=(1164,649)
[3] TextField "Password" center=(913,535)
[4] Link "Forgot password" center=(423,75)
[5] TextField "Email" center=(1747,155)

Instruction: click remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1471,45)
[1] CheckBox "Wi-Fi" center=(1453,147)
[2] Button "Displays" center=(1383,635)
[3] Button "Privacy & Security" center=(1348,809)
[4] CheckBox "Do Not Disturb" center=(1573,121)

Instruction: press privacy & security

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(803,975)
[1] Button "Network" center=(188,360)
[2] Button "Privacy & Security" center=(623,776)
[3] Button "Sound" center=(1049,351)
[4] CheckBox "Do Not Disturb" center=(931,673)

Instruction: click sound

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1131,1030)
[1] Button "Pause" center=(421,967)
[2] Button "Fullscreen" center=(107,904)
[3] Button "Play" center=(561,94)
[4] Button "Previous" center=(135,678)

Instruction: click previous

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(271,429)
[1] SearchField "Search products" center=(1404,737)
[2] Button "Wishlist" center=(65,65)
[3] Button "Add to cart" center=(807,921)
[4] Button "Checkout" center=(1269,931)

Instruction: search for something

[Assistant] {"action":"type","id":1,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(368,409)
[1] Button "Wishlist" center=(342,711)
[2] SearchField "Search products" center=(1819,83)
[3] Link "Account" center=(1850,207)
[4] Button "Checkout" center=(220,265)
[5] Button "Filters" center=(268,141)

Instruction: go to checkout

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(953,557)
[1] CheckBox "Do Not Disturb" center=(266,155)
[2] Button "Displays" center=(1684,483)
[3] CheckBox "Wi-Fi" center=(999,853)
[4] Button "Sound" center=(1738,799)
[5] Slider "Brightness" center=(1139,461)
[6] CheckBox "Bluetooth" center=(1549,178)
[7] Button "Network" center=(1264,47)

Instruction: select do not disturb

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(389,215)
[1] CheckBox "Wi-Fi" center=(123,669)
[2] Button "Displays" center=(1214,346)
[3] Button "Network" center=(211,534)
[4] CheckBox "Bluetooth" center=(1444,866)

Instruction: press displays

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1285,574)
[1] Button "Previous" center=(615,838)
[2] Button "Pause" center=(192,102)
[3] Button "Shuffle" center=(1401,390)
[4] Button "Next" center=(96,498)

Instruction: click pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1042,497)
[1] Button "Wishlist" center=(167,278)
[2] Link "Account" center=(1640,548)
[3] Button "Buy now" center=(1072,296)
[4] Button "Filters" center=(933,502)
[5] SearchField "Search products" center=(1289,543)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(598,715)
[1] SearchField "Search products" center=(642,890)
[2] Button "Checkout" center=(122,387)
[3] Button "Add to cart" center=(1084,277)
[4] Button "Buy now" center=(699,375)
[5] Button "Filters" center=(1510,957)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(423,618)
[1] PopUpButton "Appearance" center=(92,160)
[2] CheckBox "Wi-Fi" center=(1774,746)
[3] Button "Privacy & Security" center=(1160,969)
[4] Button "Displays" center=(523,173)
[5] Button "Sound" center=(62,409)
[6] Button "Network" center=(261,132)
[7] CheckBox "Do Not Disturb" center=(1578,201)
[8] Slider "Brightness" center=(169,886)

Instruction: turn off bluetooth

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(660,82)
[1] Button "Run" center=(861,362)
[2] Button "Open" center=(171,107)
[3] Button "Commit" center=(1421,520)
[4] Button "Format" center=(1346,770)
[5] Button "Find" center=(1743,319)

Instruction: click commit

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1178,455)
[1] Button "Open" center=(439,685)
[2] Button "Format" center=(1386,131)
[3] Button "Run" center=(374,955)
[4] Button "Terminal" center=(1648,852)
[5] Button "Find" center=(561,889)
[6] Button "New File" center=(154,498)
[7] Button "Commit" center=(1551,716)
[8] Button "Save" center=(1277,65)

Instruction: click terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1063,854)
[1] CheckBox "Do Not Disturb" center=(1463,451)
[2] Slider "Brightness" center=(991,489)
[3] Button "Privacy & Security" center=(892,963)
[4] Button "Sound" center=(185,638)
[5] CheckBox "Wi-Fi" center=(1802,386)
[6] Button "Displays" center=(304,314)
[7] CheckBox "Bluetooth" center=(1196,448)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1851,196)
[1] Button "New tab" center=(1402,250)
[2] Button "Extensions" center=(1176,44)
[3] Button "Bookmark" center=(555,760)
[4] Button "Go back" center=(756,1020)
[5] Link "Sign in" center=(511,352)
[6] SearchField "Search or enter address" center=(965,556)

Instruction: tap bookmark

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1244,699)
[1] Button "Previous" center=(1452,226)
[2] Button "Play" center=(1129,314)
[3] Slider "Volume" center=(1113,617)
[4] Button "Fullscreen" center=(1177,525)
[5] Button "Next" center=(1645,281)
[6] Button "Pause" center=(778,765)

Instruction: tap previous

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1473,962)
[1] CheckBox "Wi-Fi" center=(562,70)
[2] Button "Displays" center=(1871,875)
[3] Slider "Brightness" center=(1294,288)
[4] CheckBox "Do Not Disturb" center=(1218,555)
[5] Button "Privacy & Security" center=(861,721)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1520,817)
[1] Button "Save" center=(844,855)
[2] Button "Commit" center=(1426,395)
[3] Button "Format" center=(308,767)
[4] Button "Run" center=(531,657)
[5] Button "Open" center=(198,606)
[6] Button "New File" center=(1541,606)

Instruction: tap format

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1331,201)
[1] CheckBox "Wi-Fi" center=(155,912)
[2] Button "Network" center=(253,460)
[3] Button "Displays" center=(908,239)
[4] Button "Privacy & Security" center=(543,316)
[5] PopUpButton "Appearance" center=(1636,100)
[6] CheckBox "Bluetooth" center=(278,308)
[7] Slider "Brightness" center=(1455,938)

Instruction: tap appearance

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(390,858)
[1] Button "Play" center=(1819,580)
[2] Button "Previous" center=(1179,469)
[3] Button "Fullscreen" center=(412,948)
[4] Button "Shuffle" center=(484,239)
[5] Button "Next" center=(786,510)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(594,932)
[1] TextField "Password" center=(634,996)
[2] Link "Forgot password" center=(341,126)
[3] TextField "Email" center=(198,147)
[4] CheckBox "Remember me" center=(996,387)

Instruction: select remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1607,872)
[1] TextField "Password" center=(205,686)
[2] CheckBox "Remember me" center=(1175,584)
[3] Button "Create account" center=(1511,346)
[4] Button "Sign in" center=(407,916)

Instruction: tap remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1580,542)
[1] Button "Displays" center=(1354,665)
[2] Button "Network" center=(395,963)
[3] Button "Sound" center=(555,716)
[4] Slider "Brightness" center=(833,335)
[5] CheckBox "Wi-Fi" center=(404,640)
[6] CheckBox "Do Not Disturb" center=(1261,939)

Instruction: press sound

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1606,700)
[1] Link "Forgot password" center=(1207,871)
[2] TextField "Email" center=(282,408)
[3] Button "Sign in" center=(531,914)
[4] TextField "Password" center=(44,102)
[5] Button "Create account" center=(991,363)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1447,959)
[1] Button "Sign in" center=(520,89)
[2] Button "Create account" center=(735,1020)
[3] TextField "Password" center=(993,872)
[4] TextField "Email" center=(1859,611)

Instruction: click create account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(671,250)
[1] Link "Pricing" center=(1565,88)
[2] Button "Downloads" center=(1715,690)
[3] Link "Docs" center=(185,983)
[4] Button "Run inference" center=(590,687)
[5] SearchField "Search or enter address" center=(891,53)
[6] Button "Bookmark" center=(911,470)

Instruction: tap run inference

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1264,1018)
[1] Button "New tab" center=(1473,683)
[2] Link "Sign in" center=(115,435)
[3] SearchField "Search or enter address" center=(732,49)
[4] Button "Reload" center=(909,119)
[5] Link "Pricing" center=(1779,831)
[6] Button "Downloads" center=(585,488)
[7] Button "Bookmark" center=(211,398)
[8] Button "Run inference" center=(1230,49)
[9] Link "Docs" center=(249,150)
[10] Button "Go back" center=(377,366)
[11] Button "Extensions" center=(44,182)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":3,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(271,987)
[1] Link "Pricing" center=(393,101)
[2] Button "Reload" center=(1174,440)
[3] Button "Downloads" center=(1859,121)
[4] Button "Extensions" center=(581,805)
[5] Button "Bookmark" center=(1477,671)
[6] Button "Go forward" center=(1162,342)

Instruction: tap sign in

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1756,316)
[1] Button "Run" center=(319,303)
[2] Button "Commit" center=(843,54)
[3] TextField "Search files" center=(1354,527)
[4] Button "Format" center=(496,902)
[5] Button "Save" center=(1040,213)
[6] Button "Open" center=(545,762)

Instruction: type into search files

[Assistant] {"action":"type","id":3,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1468,487)
[1] Button "Bookmark" center=(476,956)
[2] Button "Go forward" center=(1583,959)
[3] Link "Docs" center=(1043,759)
[4] Link "Pricing" center=(1606,663)
[5] Link "Sign in" center=(1071,214)
[6] Button "New tab" center=(1254,275)
[7] Button "Reload" center=(98,388)
[8] SearchField "Search or enter address" center=(683,550)
[9] Button "Run inference" center=(1018,628)
[10] Button "Downloads" center=(1000,715)
[11] Button "Extensions" center=(1720,655)

Instruction: show documentation

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(49,830)
[1] Link "Sign in" center=(614,876)
[2] Button "Bookmark" center=(1774,275)
[3] Button "Reload" center=(1206,97)
[4] Link "Docs" center=(524,389)
[5] Button "Run inference" center=(1695,484)
[6] SearchField "Search or enter address" center=(1559,66)
[7] Button "Downloads" center=(1831,176)
[8] Button "Go back" center=(1126,404)
[9] Button "Go forward" center=(1611,604)

Instruction: tap go forward

[Assistant] {"action":"click","id":9}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(708,905)
[1] TextField "Search files" center=(207,856)
[2] Button "Terminal" center=(391,129)
[3] Button "Run" center=(156,130)
[4] Button "Format" center=(1064,337)
[5] Button "New File" center=(305,438)
[6] Button "Open" center=(378,719)
[7] Button "Commit" center=(320,428)

Instruction: click format

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1067,397)
[1] Link "Account" center=(1650,661)
[2] Button "Wishlist" center=(1336,279)
[3] SearchField "Search products" center=(698,455)
[4] Button "Checkout" center=(804,286)

Instruction: press wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1035,936)
[1] Button "Find" center=(1800,298)
[2] Button "Open" center=(1187,871)
[3] Button "New File" center=(1574,858)
[4] TextField "Search files" center=(1528,401)
[5] Button "Run" center=(1529,982)

Instruction: click open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1771,695)
[1] Button "Network" center=(145,124)
[2] CheckBox "Bluetooth" center=(1391,543)
[3] PopUpButton "Appearance" center=(1689,372)
[4] Button "Displays" center=(1622,213)
[5] Slider "Brightness" center=(1785,201)
[6] Button "Privacy & Security" center=(1784,531)
[7] Button "Sound" center=(1246,809)
[8] CheckBox "Do Not Disturb" center=(1339,969)

Instruction: tap displays

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(331,101)
[1] Button "Sign in" center=(1719,269)
[2] Link "Forgot password" center=(1401,569)
[3] Button "Create account" center=(735,315)
[4] CheckBox "Remember me" center=(117,488)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(889,617)
[1] TextField "Password" center=(1555,367)
[2] Button "Create account" center=(1002,950)
[3] CheckBox "Remember me" center=(695,266)
[4] Button "Sign in" center=(1754,751)

Instruction: press forgot password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(622,467)
[1] Button "Play" center=(184,359)
[2] Button "Fullscreen" center=(307,780)
[3] Button "Pause" center=(929,805)
[4] Button "Shuffle" center=(383,62)
[5] Button "Previous" center=(595,100)
[6] Slider "Volume" center=(1086,402)

Instruction: press pause

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(959,187)
[1] CheckBox "Bluetooth" center=(1107,223)
[2] CheckBox "Do Not Disturb" center=(1372,431)
[3] Button "Network" center=(755,657)
[4] PopUpButton "Appearance" center=(1869,890)
[5] Slider "Brightness" center=(426,714)

Instruction: click appearance

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(138,291)
[1] Button "Create account" center=(454,888)
[2] TextField "Password" center=(1551,862)
[3] CheckBox "Remember me" center=(1427,276)
[4] Link "Forgot password" center=(165,556)
[5] TextField "Email" center=(1709,422)

Instruction: submit the login

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(495,522)
[1] Button "Network" center=(1173,593)
[2] CheckBox "Do Not Disturb" center=(138,788)
[3] Slider "Brightness" center=(364,740)
[4] CheckBox "Bluetooth" center=(176,72)

Instruction: tap brightness

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1787,466)
[1] Button "Next" center=(740,823)
[2] Button "Pause" center=(974,426)
[3] Slider "Volume" center=(677,405)
[4] Button "Fullscreen" center=(753,345)

Instruction: play

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(143,183)
[1] Button "Add to cart" center=(821,679)
[2] Button "Wishlist" center=(458,627)
[3] Button "Buy now" center=(685,420)
[4] Button "Filters" center=(460,280)

Instruction: press filters

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1381,572)
[1] Button "Privacy & Security" center=(684,384)
[2] Slider "Brightness" center=(1569,844)
[3] CheckBox "Do Not Disturb" center=(98,515)
[4] CheckBox "Bluetooth" center=(86,186)
[5] CheckBox "Wi-Fi" center=(811,296)
[6] Button "Sound" center=(75,482)

Instruction: tap sound

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(361,149)
[1] Link "Sign in" center=(900,1022)
[2] Button "Bookmark" center=(625,615)
[3] Button "Run inference" center=(541,792)
[4] Button "Reload" center=(562,716)
[5] Button "Go back" center=(318,504)
[6] SearchField "Search or enter address" center=(1467,939)
[7] Button "Downloads" center=(1750,952)
[8] Button "Go forward" center=(1225,504)
[9] Button "New tab" center=(550,95)
[10] Link "Pricing" center=(396,637)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":6,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(508,798)
[1] Link "Account" center=(1673,833)
[2] Button "Add to cart" center=(779,559)
[3] Button "Filters" center=(1309,251)
[4] Button "Buy now" center=(969,513)

Instruction: select account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(773,1014)
[1] Button "Filters" center=(209,433)
[2] Link "Account" center=(1429,782)
[3] Button "Add to cart" center=(723,679)
[4] Button "Checkout" center=(1232,108)
[5] SearchField "Search products" center=(1574,763)
[6] Button "Buy now" center=(1474,637)

Instruction: tap buy now

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1728,638)
[1] Button "Play" center=(52,70)
[2] Button "Pause" center=(568,114)
[3] Button "Shuffle" center=(798,213)
[4] Button "Previous" center=(660,933)

Instruction: tap previous

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1266,743)
[1] Button "Open" center=(749,822)
[2] Button "Find" center=(132,167)
[3] Button "Save" center=(1869,741)
[4] Button "Run" center=(656,424)
[5] Button "Commit" center=(964,1001)
[6] Button "New File" center=(129,806)
[7] Button "Format" center=(1301,735)

Instruction: tap new file

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(985,535)
[1] Button "Privacy & Security" center=(1496,73)
[2] Button "Network" center=(1737,690)
[3] Slider "Brightness" center=(1054,287)
[4] Button "Displays" center=(592,757)

Instruction: press do not disturb

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1852,314)
[1] Link "Pricing" center=(756,532)
[2] Button "Reload" center=(702,903)
[3] SearchField "Search or enter address" center=(403,539)
[4] Link "Sign in" center=(747,335)
[5] Button "Run inference" center=(1366,223)
[6] Button "Downloads" center=(1509,554)
[7] Button "Go back" center=(1497,983)
[8] Button "New tab" center=(1405,934)
[9] Button "Bookmark" center=(452,930)
[10] Button "Go forward" center=(297,818)

Instruction: tap downloads

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1851,440)
[1] Button "Fullscreen" center=(1107,997)
[2] Button "Previous" center=(1080,111)
[3] Button "Shuffle" center=(751,407)
[4] Button "Play" center=(1561,519)
[5] Button "Next" center=(264,749)

Instruction: tap shuffle

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(1349,54)
[1] Button "Reload" center=(739,622)
[2] SearchField "Search or enter address" center=(1847,419)
[3] Button "Downloads" center=(682,162)
[4] Button "Bookmark" center=(1464,463)
[5] Button "Go forward" center=(278,924)
[6] Button "Run inference" center=(186,264)
[7] Button "Go back" center=(464,641)
[8] Button "New tab" center=(724,465)
[9] Link "Sign in" center=(424,231)
[10] Button "Extensions" center=(309,307)

Instruction: reload

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(963,905)
[1] Button "Run" center=(1553,953)
[2] Button "New File" center=(678,611)
[3] Button "Format" center=(1759,818)
[4] Button "Save" center=(1473,938)
[5] Button "Terminal" center=(834,633)
[6] Button "Find" center=(1035,606)
[7] Button "Open" center=(662,578)
[8] Button "Commit" center=(1026,624)

Instruction: make a commit

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(371,546)
[1] Button "Next" center=(1324,92)
[2] Button "Shuffle" center=(572,765)
[3] Button "Previous" center=(459,438)
[4] Slider "Volume" center=(1208,936)
[5] Button "Play" center=(1741,227)

Instruction: click shuffle

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1240,978)
[1] Button "Terminal" center=(1511,824)
[2] Button "Run" center=(60,153)
[3] Button "Commit" center=(1039,439)
[4] Button "Format" center=(457,1036)
[5] Button "Save" center=(261,811)
[6] Button "Open" center=(1280,132)
[7] Button "New File" center=(1208,885)

Instruction: press open

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(395,237)
[1] Button "Find" center=(64,463)
[2] TextField "Search files" center=(426,396)
[3] Button "Run" center=(894,521)
[4] Button "Open" center=(462,696)
[5] Button "New File" center=(1549,401)

Instruction: type into search files

[Assistant] {"action":"type","id":2,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1841,48)
[1] Button "Run" center=(1756,1037)
[2] Button "Save" center=(1493,76)
[3] Button "Terminal" center=(484,915)
[4] Button "Format" center=(1564,203)
[5] TextField "Search files" center=(1000,248)
[6] Button "Commit" center=(77,263)

Instruction: press save

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1439,782)
[1] Slider "Volume" center=(509,845)
[2] Button "Shuffle" center=(739,899)
[3] Button "Pause" center=(886,122)
[4] Button "Fullscreen" center=(277,610)

Instruction: click volume

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1394,108)
[1] CheckBox "Bluetooth" center=(1191,382)
[2] Button "Privacy & Security" center=(449,980)
[3] CheckBox "Do Not Disturb" center=(1370,1030)
[4] Button "Displays" center=(1401,935)
[5] Button "Network" center=(171,767)

Instruction: press network

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1345,440)
[1] Link "Account" center=(908,434)
[2] Button "Checkout" center=(845,274)
[3] SearchField "Search products" center=(843,706)
[4] Button "Wishlist" center=(1514,1001)

Instruction: search for something

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(60,237)
[1] Button "Shuffle" center=(451,1040)
[2] Slider "Volume" center=(1440,543)
[3] Button "Next" center=(1291,90)
[4] Button "Previous" center=(1508,217)
[5] Button "Fullscreen" center=(1407,719)

Instruction: tap play

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(358,767)
[1] TextField "Email" center=(553,797)
[2] Link "Forgot password" center=(742,169)
[3] CheckBox "Remember me" center=(1771,745)
[4] Button "Sign in" center=(957,1015)

Instruction: tap remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1657,108)
[1] Link "Docs" center=(297,757)
[2] Button "Extensions" center=(1797,831)
[3] Button "New tab" center=(855,377)
[4] Link "Pricing" center=(350,330)
[5] Button "Reload" center=(1779,914)
[6] Link "Sign in" center=(1315,339)
[7] Button "Bookmark" center=(1669,594)
[8] Button "Go back" center=(1276,113)
[9] Button "Downloads" center=(1065,404)
[10] SearchField "Search or enter address" center=(367,858)
[11] Button "Run inference" center=(1521,653)

Instruction: press new tab

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(543,449)
[1] Button "Reload" center=(1526,126)
[2] Link "Sign in" center=(876,917)
[3] Link "Docs" center=(760,491)
[4] Button "Run inference" center=(887,899)
[5] Button "Bookmark" center=(749,180)

Instruction: tap reload

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(428,727)
[1] Button "Extensions" center=(1785,802)
[2] Button "Bookmark" center=(1707,302)
[3] Button "Reload" center=(1556,918)
[4] Link "Sign in" center=(695,509)
[5] Button "Go forward" center=(1023,893)
[6] Link "Docs" center=(1696,459)
[7] Button "Go back" center=(1808,444)
[8] Button "New tab" center=(197,779)

Instruction: tap new tab

[Assistant] {"action":"click","id":8}