[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1782,684)
[1] Button "Previous" center=(379,830)
[2] Button "Pause" center=(1805,559)
[3] Button "Play" center=(1847,970)
[4] Button "Shuffle" center=(64,384)
[5] Slider "Volume" center=(182,462)
[6] Button "Next" center=(864,481)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1677,238)
[1] Button "Go back" center=(892,124)
[2] Button "Go forward" center=(708,183)
[3] Button "Downloads" center=(1010,440)
[4] Link "Sign in" center=(1292,370)
[5] Link "Docs" center=(1738,604)
[6] Button "New tab" center=(1699,871)
[7] Link "Pricing" center=(1792,918)
[8] Button "Extensions" center=(617,350)
[9] Button "Run inference" center=(660,199)
[10] Button "Bookmark" center=(70,188)

Instruction: log in

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1818,512)
[1] Button "Sign in" center=(520,513)
[2] TextField "Email" center=(197,347)
[3] Button "Create account" center=(791,82)
[4] CheckBox "Remember me" center=(503,1005)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1242,871)
[1] SearchField "Search products" center=(289,982)
[2] Button "Checkout" center=(579,846)
[3] Button "Wishlist" center=(288,658)
[4] Button "Buy now" center=(1601,294)
[5] Link "Account" center=(1411,627)
[6] Button "Add to cart" center=(877,46)

Instruction: select filters

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1387,732)
[1] Button "Run" center=(1267,186)
[2] Button "New File" center=(1859,351)
[3] Button "Terminal" center=(1527,537)
[4] Button "Find" center=(878,613)
[5] Button "Save" center=(1170,725)
[6] Button "Commit" center=(1631,256)
[7] TextField "Search files" center=(632,104)

Instruction: select terminal

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(1468,760)
[1] Link "Pricing" center=(514,200)
[2] Button "Downloads" center=(684,691)
[3] Button "Reload" center=(277,495)
[4] Link "Docs" center=(1829,641)
[5] Button "Extensions" center=(1409,42)

Instruction: reload

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(979,579)
[1] Button "Open" center=(475,110)
[2] Button "Format" center=(84,242)
[3] Button "Terminal" center=(254,308)
[4] Button "Run" center=(1697,866)
[5] TextField "Search files" center=(764,821)
[6] Button "Commit" center=(838,318)
[7] Button "Find" center=(684,525)
[8] Button "Save" center=(296,1009)

Instruction: search

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1796,999)
[1] SearchField "Search products" center=(961,311)
[2] Link "Account" center=(568,673)
[3] Button "Filters" center=(1324,634)
[4] Button "Wishlist" center=(1296,983)

Instruction: click filters

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1571,1032)
[1] Button "Wishlist" center=(1299,335)
[2] Button "Checkout" center=(222,230)
[3] Button "Add to cart" center=(1710,685)
[4] SearchField "Search products" center=(868,1026)
[5] Button "Filters" center=(655,112)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(534,129)
[1] SearchField "Search or enter address" center=(1443,896)
[2] Button "Run inference" center=(114,123)
[3] Link "Sign in" center=(94,586)
[4] Button "New tab" center=(1274,455)
[5] Button "Go back" center=(1358,380)
[6] Button "Go forward" center=(644,456)
[7] Button "Downloads" center=(1651,136)
[8] Link "Pricing" center=(379,369)
[9] Link "Docs" center=(502,445)
[10] Button "Reload" center=(116,437)

Instruction: click extensions

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1626,690)
[1] Link "Forgot password" center=(1342,637)
[2] TextField "Password" center=(1711,571)
[3] CheckBox "Remember me" center=(1250,318)
[4] Button "Sign in" center=(1081,298)
[5] Button "Create account" center=(1098,111)

Instruction: make a new account

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1696,246)
[1] Button "Reload" center=(1710,996)
[2] Button "New tab" center=(115,494)
[3] Button "Downloads" center=(1120,145)
[4] Button "Run inference" center=(266,993)
[5] Link "Docs" center=(1852,115)
[6] Button "Bookmark" center=(945,149)
[7] Button "Go forward" center=(1215,760)

Instruction: select sign in

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1699,405)
[1] Button "Wishlist" center=(724,909)
[2] Link "Account" center=(1252,473)
[3] Button "Checkout" center=(956,326)
[4] Button "Filters" center=(505,277)

Instruction: select filters

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1202,151)
[1] Button "Play" center=(1023,834)
[2] Button "Pause" center=(1432,67)
[3] Button "Next" center=(1716,256)
[4] Button "Previous" center=(1528,279)

Instruction: click fullscreen

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(479,776)
[1] Button "Sign in" center=(830,774)
[2] Button "Create account" center=(268,1020)
[3] TextField "Password" center=(1554,668)
[4] CheckBox "Remember me" center=(1018,703)
[5] Link "Forgot password" center=(1742,507)

Instruction: type into password

[Assistant] {"action":"type","id":3,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1663,804)
[1] CheckBox "Wi-Fi" center=(1702,613)
[2] PopUpButton "Appearance" center=(1724,397)
[3] Button "Network" center=(1772,578)
[4] CheckBox "Do Not Disturb" center=(810,918)
[5] CheckBox "Bluetooth" center=(714,1033)

Instruction: press do not disturb

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(451,850)
[1] Button "Open" center=(294,956)
[2] Button "Run" center=(1606,533)
[3] TextField "Search files" center=(1287,439)
[4] Button "New File" center=(941,999)
[5] Button "Save" center=(108,480)
[6] Button "Format" center=(52,414)
[7] Button "Terminal" center=(785,612)
[8] Button "Commit" center=(1352,143)

Instruction: type into search files

[Assistant] {"action":"type","id":3,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(319,387)
[1] Button "Sign in" center=(90,442)
[2] TextField "Email" center=(1468,870)
[3] TextField "Password" center=(1243,610)
[4] Link "Forgot password" center=(1294,232)
[5] Button "Create account" center=(1835,419)

Instruction: type into password

[Assistant] {"action":"type","id":3,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1397,1040)
[1] Button "Sign in" center=(1326,701)
[2] TextField "Email" center=(711,409)
[3] CheckBox "Remember me" center=(1035,45)
[4] TextField "Password" center=(1774,934)
[5] Button "Create account" center=(512,653)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(699,169)
[1] Slider "Volume" center=(1277,893)
[2] Button "Fullscreen" center=(1785,704)
[3] Button "Shuffle" center=(152,377)
[4] Button "Play" center=(88,676)
[5] Button "Previous" center=(395,438)

Instruction: skip to next track

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(1619,804)
[1] Button "Run inference" center=(1316,681)
[2] Button "New tab" center=(1093,123)
[3] Link "Sign in" center=(325,1010)
[4] Link "Pricing" center=(219,915)
[5] Button "Bookmark" center=(513,41)
[6] Button "Extensions" center=(136,728)
[7] Link "Docs" center=(1555,609)
[8] Button "Downloads" center=(1088,183)

Instruction: sign in

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(301,241)
[1] Button "Pause" center=(1420,994)
[2] Button "Next" center=(486,115)
[3] Button "Play" center=(165,310)
[4] Button "Shuffle" center=(1402,308)
[5] Slider "Volume" center=(1123,167)

Instruction: press play

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(187,830)
[1] Slider "Volume" center=(1126,600)
[2] Button "Play" center=(318,939)
[3] Button "Fullscreen" center=(326,61)
[4] Button "Shuffle" center=(881,258)

Instruction: tap shuffle

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(358,234)
[1] Button "Sign in" center=(886,308)
[2] CheckBox "Remember me" center=(1605,159)
[3] Link "Forgot password" center=(1453,682)
[4] Button "Create account" center=(933,662)

Instruction: make a new account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1543,273)
[1] TextField "Search files" center=(1388,223)
[2] Button "Format" center=(743,183)
[3] Button "Find" center=(1654,961)
[4] Button "Open" center=(1132,192)
[5] Button "Save" center=(1853,948)
[6] Button "New File" center=(1340,864)
[7] Button "Commit" center=(863,864)
[8] Button "Terminal" center=(970,973)

Instruction: press new file

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(568,348)
[1] TextField "Search files" center=(1440,572)
[2] Button "Save" center=(610,818)
[3] Button "New File" center=(548,860)
[4] Button "Terminal" center=(1569,684)
[5] Button "Open" center=(919,825)
[6] Button "Commit" center=(1480,787)

Instruction: click terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(214,901)
[1] Button "Downloads" center=(723,364)
[2] Button "Extensions" center=(1478,396)
[3] SearchField "Search or enter address" center=(1256,731)
[4] Link "Pricing" center=(1391,840)
[5] Button "Reload" center=(1461,97)
[6] Link "Sign in" center=(1618,695)
[7] Button "Go back" center=(835,755)
[8] Button "Run inference" center=(1860,210)
[9] Button "Bookmark" center=(1637,533)
[10] Button "New tab" center=(1126,530)
[11] Button "Go forward" center=(1288,93)

Instruction: refresh the page

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1524,63)
[1] SearchField "Search or enter address" center=(483,189)
[2] Button "Reload" center=(429,371)
[3] Button "Run inference" center=(1081,344)
[4] Link "Sign in" center=(188,588)
[5] Button "Go back" center=(1670,46)
[6] Button "Go forward" center=(398,677)
[7] Link "Pricing" center=(694,392)
[8] Button "Bookmark" center=(1387,674)
[9] Link "Docs" center=(1247,621)
[10] Button "New tab" center=(1042,519)

Instruction: press pricing

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1357,875)
[1] Button "Add to cart" center=(339,300)
[2] Button "Buy now" center=(1514,502)
[3] Link "Account" center=(591,738)
[4] Button "Wishlist" center=(214,115)
[5] Button "Checkout" center=(171,530)
[6] Button "Filters" center=(1423,261)

Instruction: click account

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(368,413)
[1] CheckBox "Wi-Fi" center=(1177,338)
[2] Slider "Brightness" center=(1535,910)
[3] Button "Displays" center=(368,250)
[4] Button "Network" center=(465,71)

Instruction: click network

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1326,298)
[1] CheckBox "Remember me" center=(1345,612)
[2] Button "Sign in" center=(325,563)
[3] TextField "Email" center=(995,268)
[4] TextField "Password" center=(1757,925)

Instruction: reset password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(943,275)
[1] Button "Next" center=(1506,477)
[2] Button "Play" center=(763,820)
[3] Button "Pause" center=(635,597)
[4] Button "Fullscreen" center=(754,192)
[5] Slider "Volume" center=(416,42)
[6] Button "Shuffle" center=(1006,176)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1436,773)
[1] TextField "Search files" center=(368,541)
[2] Button "Format" center=(1295,754)
[3] Button "New File" center=(920,820)
[4] Button "Run" center=(1858,254)
[5] Button "Terminal" center=(1328,445)
[6] Button "Open" center=(708,107)
[7] Button "Find" center=(361,679)

Instruction: select terminal

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1091,157)
[1] Button "Create account" center=(1306,350)
[2] TextField "Password" center=(1450,590)
[3] Link "Forgot password" center=(1614,302)
[4] CheckBox "Remember me" center=(1494,171)

Instruction: make a new account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1484,966)
[1] Button "New tab" center=(429,725)
[2] Button "Extensions" center=(596,739)
[3] Button "Run inference" center=(331,938)
[4] Link "Sign in" center=(1717,695)

Instruction: select run inference

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1492,445)
[1] Button "Network" center=(492,166)
[2] CheckBox "Do Not Disturb" center=(904,900)
[3] CheckBox "Wi-Fi" center=(1646,476)
[4] PopUpButton "Appearance" center=(1293,434)
[5] Button "Displays" center=(1752,611)
[6] Slider "Brightness" center=(1005,784)
[7] Button "Privacy & Security" center=(1424,352)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(48,71)
[1] Button "Reload" center=(1444,599)
[2] Link "Pricing" center=(628,337)
[3] SearchField "Search or enter address" center=(1655,792)
[4] Link "Docs" center=(718,605)
[5] Button "Bookmark" center=(96,142)
[6] Button "Go back" center=(714,786)
[7] Button "Go forward" center=(1383,137)
[8] Button "Run inference" center=(1043,456)
[9] Button "Extensions" center=(653,186)
[10] Button "Downloads" center=(1228,1008)
[11] Button "New tab" center=(885,890)

Instruction: click run inference

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1715,58)
[1] Button "Wishlist" center=(1379,664)
[2] Button "Filters" center=(1690,457)
[3] SearchField "Search products" center=(1300,626)
[4] Button "Add to cart" center=(1484,465)
[5] Button "Checkout" center=(242,788)

Instruction: click wishlist

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(312,716)
[1] Button "Create account" center=(1664,605)
[2] TextField "Email" center=(138,372)
[3] Link "Forgot password" center=(1070,897)
[4] Button "Sign in" center=(1162,237)

Instruction: i forgot my password

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(736,804)
[1] Button "Create account" center=(1822,732)
[2] TextField "Password" center=(446,650)
[3] CheckBox "Remember me" center=(785,87)
[4] TextField "Email" center=(1620,149)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1170,591)
[1] CheckBox "Do Not Disturb" center=(402,451)
[2] Button "Privacy & Security" center=(122,266)
[3] CheckBox "Bluetooth" center=(208,495)
[4] Button "Network" center=(605,584)
[5] Slider "Brightness" center=(1243,59)
[6] Button "Displays" center=(527,60)
[7] PopUpButton "Appearance" center=(561,970)
[8] Button "Sound" center=(774,861)

Instruction: click sound

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1676,950)
[1] Button "Pause" center=(1779,903)
[2] Button "Next" center=(673,886)
[3] Button "Play" center=(308,222)
[4] Button "Previous" center=(1287,954)

Instruction: select next

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1664,401)
[1] Button "Previous" center=(1194,456)
[2] Button "Play" center=(947,368)
[3] Button "Pause" center=(951,547)
[4] Slider "Volume" center=(1267,661)
[5] Button "Next" center=(1786,130)
[6] Button "Shuffle" center=(259,736)

Instruction: press pause

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(836,40)
[1] Button "Go back" center=(1165,623)
[2] Button "Run inference" center=(1867,909)
[3] SearchField "Search or enter address" center=(770,1031)
[4] Button "Reload" center=(1126,624)
[5] Button "Bookmark" center=(991,973)

Instruction: press sign in

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(440,318)
[1] CheckBox "Do Not Disturb" center=(979,483)
[2] CheckBox "Wi-Fi" center=(1434,806)
[3] Button "Sound" center=(184,173)
[4] Button "Displays" center=(1415,559)
[5] Slider "Brightness" center=(1341,631)
[6] PopUpButton "Appearance" center=(1858,568)
[7] Button "Network" center=(1241,672)
[8] CheckBox "Bluetooth" center=(1517,272)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1222,945)
[1] CheckBox "Remember me" center=(1801,905)
[2] Button "Sign in" center=(1257,169)
[3] Link "Forgot password" center=(995,147)
[4] TextField "Email" center=(1337,968)
[5] Button "Create account" center=(712,752)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1802,669)
[1] CheckBox "Bluetooth" center=(1088,446)
[2] CheckBox "Wi-Fi" center=(1089,707)
[3] CheckBox "Do Not Disturb" center=(254,599)
[4] Button "Privacy & Security" center=(1623,825)
[5] Button "Displays" center=(568,834)
[6] Button "Sound" center=(520,1032)
[7] Slider "Brightness" center=(1218,518)
[8] Button "Network" center=(850,753)

Instruction: click appearance

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1440,962)
[1] TextField "Search files" center=(1355,715)
[2] Button "Open" center=(557,930)
[3] Button "Format" center=(1678,357)
[4] Button "Commit" center=(607,152)
[5] Button "New File" center=(1780,1026)
[6] Button "Find" center=(128,502)
[7] Button "Terminal" center=(865,635)

Instruction: tap open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(263,745)
[1] Button "Network" center=(840,71)
[2] Button "Displays" center=(1679,791)
[3] PopUpButton "Appearance" center=(570,295)
[4] CheckBox "Bluetooth" center=(738,633)
[5] Slider "Brightness" center=(1375,981)
[6] CheckBox "Do Not Disturb" center=(1095,254)
[7] Button "Sound" center=(1291,568)
[8] Button "Privacy & Security" center=(714,1017)

Instruction: enable wifi

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1079,814)
[1] Button "Pause" center=(496,984)
[2] Button "Shuffle" center=(1416,548)
[3] Button "Next" center=(715,466)
[4] Button "Previous" center=(1403,856)
[5] Slider "Volume" center=(1170,113)

Instruction: select previous

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(993,1020)
[1] Button "Privacy & Security" center=(834,136)
[2] PopUpButton "Appearance" center=(1842,983)
[3] Slider "Brightness" center=(1445,692)
[4] CheckBox "Bluetooth" center=(1801,237)
[5] Button "Sound" center=(687,713)
[6] Button "Network" center=(92,697)
[7] CheckBox "Wi-Fi" center=(1322,42)

Instruction: tap brightness

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(67,281)
[1] Button "Wishlist" center=(427,458)
[2] Button "Checkout" center=(1231,641)
[3] SearchField "Search products" center=(1277,388)
[4] Link "Account" center=(306,1016)
[5] Button "Filters" center=(1175,381)

Instruction: press add to cart

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(623,678)
[1] Button "New tab" center=(962,199)
[2] Link "Sign in" center=(577,648)
[3] Button "Extensions" center=(1453,94)
[4] Button "Run inference" center=(425,183)
[5] Link "Pricing" center=(302,291)

Instruction: click extensions

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(682,416)
[1] CheckBox "Remember me" center=(745,751)
[2] TextField "Email" center=(964,82)
[3] Button "Sign in" center=(344,77)
[4] Link "Forgot password" center=(123,460)
[5] Button "Create account" center=(272,521)

Instruction: tap remember me

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1608,203)
[1] Button "Create account" center=(260,604)
[2] TextField "Password" center=(1194,295)
[3] Link "Forgot password" center=(1397,651)
[4] CheckBox "Remember me" center=(230,1019)

Instruction: i forgot my password

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1148,704)
[1] Button "New File" center=(525,711)
[2] Button "Find" center=(1494,687)
[3] Button "Terminal" center=(94,413)
[4] Button "Commit" center=(830,343)
[5] TextField "Search files" center=(620,469)

Instruction: find in file

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(298,1032)
[1] Button "Create account" center=(875,257)
[2] TextField "Password" center=(1485,380)
[3] Link "Forgot password" center=(917,46)
[4] CheckBox "Remember me" center=(1063,550)

Instruction: select remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(99,572)
[1] Slider "Brightness" center=(404,750)
[2] Button "Displays" center=(1559,884)
[3] CheckBox "Wi-Fi" center=(540,449)
[4] CheckBox "Bluetooth" center=(1198,426)

Instruction: tap brightness

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(730,732)
[1] Button "Wishlist" center=(87,450)
[2] Button "Filters" center=(253,508)
[3] SearchField "Search products" center=(1623,660)
[4] Link "Account" center=(116,482)
[5] Button "Buy now" center=(1357,1001)

Instruction: click checkout

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1236,908)
[1] Button "Sign in" center=(467,298)
[2] Link "Forgot password" center=(673,666)
[3] TextField "Password" center=(107,508)
[4] TextField "Email" center=(933,653)

Instruction: tap sign in

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(813,841)
[1] Slider "Brightness" center=(1368,664)
[2] CheckBox "Do Not Disturb" center=(340,155)
[3] CheckBox "Wi-Fi" center=(45,910)
[4] Button "Privacy & Security" center=(730,269)
[5] PopUpButton "Appearance" center=(90,531)
[6] Button "Sound" center=(214,203)
[7] Button "Network" center=(394,981)
[8] CheckBox "Bluetooth" center=(895,83)

Instruction: turn off bluetooth

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(969,838)
[1] Button "Filters" center=(150,363)
[2] Button "Checkout" center=(625,675)
[3] SearchField "Search products" center=(1186,371)
[4] Button "Wishlist" center=(1045,792)

Instruction: type into search products

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1392,570)
[1] Button "Checkout" center=(1741,750)
[2] Link "Account" center=(1570,800)
[3] Button "Filters" center=(92,1023)
[4] Button "Wishlist" center=(202,211)
[5] Button "Add to cart" center=(903,1039)
[6] Button "Buy now" center=(1221,606)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1556,942)
[1] TextField "Password" center=(594,542)
[2] Link "Forgot password" center=(876,53)
[3] Button "Sign in" center=(1769,621)
[4] CheckBox "Remember me" center=(893,878)

Instruction: type into email

[Assistant] {"action":"type","id":0,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(402,410)
[1] Slider "Volume" center=(984,262)
[2] Button "Shuffle" center=(967,603)
[3] Button "Pause" center=(993,751)
[4] Button "Fullscreen" center=(553,520)
[5] Button "Next" center=(1053,600)

Instruction: click shuffle

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(216,185)
[1] TextField "Search files" center=(1299,566)
[2] Button "Open" center=(1064,661)
[3] Button "Terminal" center=(969,1016)
[4] Button "Commit" center=(353,598)
[5] Button "New File" center=(667,198)
[6] Button "Run" center=(856,574)
[7] Button "Find" center=(165,854)

Instruction: tap open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(42,389)
[1] Button "Pause" center=(441,926)
[2] Button "Fullscreen" center=(1049,116)
[3] Button "Next" center=(1361,568)
[4] Button "Previous" center=(908,74)

Instruction: skip to next track

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(994,636)
[1] CheckBox "Wi-Fi" center=(1112,235)
[2] Button "Displays" center=(1878,166)
[3] Button "Sound" center=(1842,218)
[4] Slider "Brightness" center=(1598,820)
[5] CheckBox "Do Not Disturb" center=(577,176)

Instruction: press displays

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1221,658)
[1] Button "Create account" center=(215,1040)
[2] TextField "Password" center=(1303,609)
[3] TextField "Email" center=(1434,307)
[4] Button "Sign in" center=(1179,60)

Instruction: click forgot password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1368,145)
[1] Button "Terminal" center=(1523,1011)
[2] Button "Format" center=(152,522)
[3] Button "Open" center=(1733,332)
[4] Button "New File" center=(159,331)
[5] TextField "Search files" center=(503,270)
[6] Button "Run" center=(909,1038)
[7] Button "Commit" center=(1162,632)

Instruction: make a commit

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(177,133)
[1] Slider "Volume" center=(111,988)
[2] Button "Play" center=(1737,624)
[3] Button "Previous" center=(41,92)
[4] Button "Next" center=(1425,520)
[5] Button "Shuffle" center=(144,874)

Instruction: click next

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(631,618)
[1] Button "Fullscreen" center=(1009,207)
[2] Button "Previous" center=(1445,820)
[3] Button "Shuffle" center=(1323,275)
[4] Button "Next" center=(1480,739)
[5] Button "Play" center=(1305,219)
[6] Button "Pause" center=(107,123)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1458,709)
[1] Link "Account" center=(1399,920)
[2] Button "Buy now" center=(723,695)
[3] Button "Filters" center=(511,928)
[4] SearchField "Search products" center=(1267,472)
[5] Button "Checkout" center=(698,649)
[6] Button "Wishlist" center=(1815,806)

Instruction: check out

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(616,125)
[1] TextField "Email" center=(868,331)
[2] Link "Forgot password" center=(271,397)
[3] Button "Sign in" center=(1634,776)
[4] Button "Create account" center=(472,491)

Instruction: i forgot my password

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1132,157)
[1] CheckBox "Bluetooth" center=(1562,732)
[2] CheckBox "Do Not Disturb" center=(811,825)
[3] Button "Displays" center=(49,897)
[4] Button "Sound" center=(1125,367)
[5] PopUpButton "Appearance" center=(743,548)
[6] Button "Privacy & Security" center=(1610,130)
[7] Button "Network" center=(1695,854)

Instruction: select wi-fi

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1293,75)
[1] Button "Checkout" center=(1095,99)
[2] Button "Buy now" center=(473,1018)
[3] Button "Add to cart" center=(1718,856)
[4] SearchField "Search products" center=(731,555)
[5] Button "Wishlist" center=(860,215)
[6] Link "Account" center=(620,792)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(805,823)
[1] Button "Commit" center=(538,364)
[2] Button "Format" center=(337,1037)
[3] Button "Run" center=(943,794)
[4] Button "Terminal" center=(1250,151)
[5] Button "Save" center=(239,962)
[6] Button "New File" center=(278,132)

Instruction: select open

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1288,607)
[1] Button "Previous" center=(1408,784)
[2] Button "Pause" center=(698,650)
[3] Button "Next" center=(495,878)
[4] Slider "Volume" center=(379,118)
[5] Button "Play" center=(537,405)
[6] Button "Fullscreen" center=(1733,630)

Instruction: press volume

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(357,394)
[1] Button "Run inference" center=(1301,1031)
[2] Button "Reload" center=(1054,718)
[3] Link "Sign in" center=(314,48)
[4] Button "Go forward" center=(725,911)
[5] Button "Extensions" center=(719,343)
[6] Link "Docs" center=(1159,886)
[7] Button "Go back" center=(375,882)
[8] Button "New tab" center=(1216,569)

Instruction: click go back

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1532,592)
[1] Button "Previous" center=(613,497)
[2] Button "Shuffle" center=(664,435)
[3] Button "Next" center=(50,341)
[4] Button "Play" center=(397,637)
[5] Button "Pause" center=(547,997)
[6] Slider "Volume" center=(973,1021)

Instruction: select pause

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(415,551)
[1] Slider "Volume" center=(728,588)
[2] Button "Shuffle" center=(137,868)
[3] Button "Next" center=(595,392)
[4] Button "Play" center=(1679,187)
[5] Button "Pause" center=(1312,892)

Instruction: next song

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(717,340)
[1] Button "Buy now" center=(610,377)
[2] Button "Checkout" center=(1090,304)
[3] Button "Wishlist" center=(1849,502)
[4] Link "Account" center=(1593,341)
[5] Button "Filters" center=(181,281)

Instruction: press add to cart

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(930,226)
[1] Button "Fullscreen" center=(1464,504)
[2] Button "Play" center=(721,719)
[3] Button "Previous" center=(171,755)
[4] Button "Pause" center=(1230,253)
[5] Button "Shuffle" center=(554,425)
[6] Slider "Volume" center=(1874,295)

Instruction: next song

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(283,983)
[1] Link "Account" center=(1856,111)
[2] SearchField "Search products" center=(777,459)
[3] Button "Buy now" center=(1175,862)
[4] Button "Wishlist" center=(863,179)
[5] Button "Filters" center=(1484,175)

Instruction: click account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(815,75)
[1] Button "Run" center=(308,763)
[2] Button "Open" center=(307,500)
[3] TextField "Search files" center=(1341,985)
[4] Button "Terminal" center=(653,287)

Instruction: press run

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(650,507)
[1] Slider "Volume" center=(1826,935)
[2] Button "Next" center=(761,468)
[3] Button "Fullscreen" center=(425,922)
[4] Button "Play" center=(168,858)
[5] Button "Previous" center=(1691,147)

Instruction: press pause

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1592,857)
[1] SearchField "Search products" center=(137,967)
[2] Button "Checkout" center=(849,466)
[3] Button "Buy now" center=(198,366)
[4] Button "Wishlist" center=(1467,943)
[5] Button "Add to cart" center=(651,395)
[6] Link "Account" center=(1315,922)

Instruction: tap wishlist

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1327,915)
[1] Button "Commit" center=(1168,736)
[2] Button "Find" center=(1109,220)
[3] Button "Save" center=(733,182)
[4] Button "Terminal" center=(195,1000)
[5] Button "New File" center=(390,976)

Instruction: tap new file

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1139,553)
[1] Button "Open" center=(1863,200)
[2] Button "Terminal" center=(1458,900)
[3] Button "Run" center=(1130,333)
[4] Button "New File" center=(959,944)
[5] Button "Find" center=(113,376)

Instruction: click open

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1567,203)
[1] Button "Next" center=(74,252)
[2] Button "Play" center=(1816,482)
[3] Button "Previous" center=(821,575)
[4] Button "Fullscreen" center=(933,340)

Instruction: tap previous

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1123,513)
[1] Link "Account" center=(1553,427)
[2] Button "Add to cart" center=(99,779)
[3] Button "Checkout" center=(538,193)
[4] Button "Filters" center=(52,557)

Instruction: check out

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1727,267)
[1] CheckBox "Remember me" center=(584,693)
[2] TextField "Password" center=(1336,395)
[3] Link "Forgot password" center=(1705,930)
[4] TextField "Email" center=(1584,814)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(154,300)
[1] Button "New File" center=(717,258)
[2] Button "Commit" center=(735,583)
[3] Button "Find" center=(770,762)
[4] Button "Terminal" center=(1236,521)

Instruction: tap new file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(125,821)
[1] Button "Privacy & Security" center=(1528,1034)
[2] CheckBox "Bluetooth" center=(352,379)
[3] CheckBox "Wi-Fi" center=(740,385)
[4] Button "Network" center=(634,970)
[5] Slider "Brightness" center=(880,573)
[6] Button "Displays" center=(1276,783)

Instruction: press brightness

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(891,528)
[1] SearchField "Search products" center=(1182,590)
[2] Button "Buy now" center=(1788,1017)
[3] Button "Add to cart" center=(181,408)
[4] Link "Account" center=(1655,637)

Instruction: tap buy now

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(60,344)
[1] Button "Next" center=(610,567)
[2] Button "Shuffle" center=(1414,989)
[3] Button "Fullscreen" center=(1878,310)
[4] Slider "Volume" center=(1568,727)
[5] Button "Pause" center=(923,551)
[6] Button "Previous" center=(1568,273)

Instruction: press pause

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1134,516)
[1] SearchField "Search products" center=(1452,488)
[2] Button "Wishlist" center=(99,50)
[3] Button "Checkout" center=(1390,909)
[4] Button "Buy now" center=(906,952)
[5] Button "Add to cart" center=(371,284)

Instruction: select wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1039,462)
[1] Link "Account" center=(89,691)
[2] Button "Wishlist" center=(1176,648)
[3] Button "Buy now" center=(701,251)
[4] Button "Checkout" center=(161,127)

Instruction: add this to my cart

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1815,817)
[1] Button "Wishlist" center=(809,187)
[2] Button "Checkout" center=(1154,269)
[3] Button "Buy now" center=(547,404)
[4] Link "Account" center=(908,602)

Instruction: click checkout

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(866,509)
[1] Button "Run inference" center=(83,172)
[2] Button "New tab" center=(1342,711)
[3] Button "Bookmark" center=(961,542)
[4] Link "Pricing" center=(379,563)
[5] Button "Extensions" center=(1610,511)
[6] Link "Docs" center=(185,366)
[7] Button "Downloads" center=(820,414)
[8] Button "Go forward" center=(1285,481)
[9] Button "Go back" center=(1391,675)

Instruction: take me back

[Assistant] {"action":"click","id":9}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(92,952)
[1] TextField "Password" center=(741,371)
[2] CheckBox "Remember me" center=(1876,704)
[3] Link "Forgot password" center=(531,744)
[4] Button "Sign in" center=(1328,436)
[5] Button "Create account" center=(314,616)

Instruction: type into email

[Assistant] {"action":"type","id":0,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(746,1004)
[1] Button "Run" center=(368,746)
[2] Button "Commit" center=(680,174)
[3] Button "Save" center=(1553,913)
[4] Button "Format" center=(1397,958)
[5] Button "New File" center=(791,186)
[6] Button "Terminal" center=(1696,995)
[7] Button "Find" center=(166,1005)
[8] Button "Open" center=(968,955)

Instruction: press open

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(644,767)
[1] Button "Reload" center=(645,367)
[2] Button "Extensions" center=(345,101)
[3] Button "Go back" center=(1686,249)
[4] Link "Pricing" center=(122,848)
[5] Button "Run inference" center=(1692,720)
[6] Button "New tab" center=(317,524)
[7] Link "Sign in" center=(1331,434)
[8] Button "Go forward" center=(1606,677)
[9] Button "Downloads" center=(1845,308)
[10] SearchField "Search or enter address" center=(988,247)
[11] Link "Docs" center=(92,565)

Instruction: press go forward

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1521,999)
[1] CheckBox "Remember me" center=(227,844)
[2] Button "Create account" center=(385,795)
[3] Button "Sign in" center=(622,939)
[4] TextField "Password" center=(625,49)

Instruction: press remember me

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1724,604)
[1] TextField "Search files" center=(119,410)
[2] Button "Commit" center=(1511,890)
[3] Button "New File" center=(1391,208)
[4] Button "Terminal" center=(764,529)

Instruction: click terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(524,965)
[1] Slider "Volume" center=(355,372)
[2] Button "Fullscreen" center=(659,456)
[3] Button "Shuffle" center=(56,128)
[4] Button "Previous" center=(1173,961)
[5] Button "Next" center=(1257,129)
[6] Button "Play" center=(1640,156)

Instruction: press fullscreen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1270,825)
[1] CheckBox "Remember me" center=(1444,678)
[2] TextField "Email" center=(925,748)
[3] Button "Create account" center=(400,953)
[4] Button "Sign in" center=(985,477)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1308,128)
[1] Button "Privacy & Security" center=(1848,752)
[2] PopUpButton "Appearance" center=(1489,248)
[3] Button "Displays" center=(1069,125)
[4] CheckBox "Wi-Fi" center=(498,80)
[5] CheckBox "Bluetooth" center=(228,336)

Instruction: click appearance

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(658,868)
[1] CheckBox "Bluetooth" center=(778,201)
[2] CheckBox "Do Not Disturb" center=(425,159)
[3] Slider "Brightness" center=(1728,626)
[4] Button "Sound" center=(260,732)
[5] Button "Privacy & Security" center=(1219,929)
[6] Button "Displays" center=(1248,76)
[7] CheckBox "Wi-Fi" center=(1090,197)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1597,481)
[1] Button "Network" center=(1844,923)
[2] Slider "Brightness" center=(1322,448)
[3] CheckBox "Wi-Fi" center=(1041,95)
[4] CheckBox "Bluetooth" center=(655,1037)
[5] PopUpButton "Appearance" center=(1497,941)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(1705,927)
[1] Button "Downloads" center=(101,154)
[2] Link "Pricing" center=(529,779)
[3] Button "Run inference" center=(461,840)
[4] Link "Docs" center=(365,63)
[5] Button "Extensions" center=(1603,948)
[6] Button "New tab" center=(549,318)
[7] Button "Go forward" center=(1676,309)
[8] Link "Sign in" center=(1253,416)
[9] Button "Bookmark" center=(920,68)

Instruction: select downloads

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(750,809)
[1] Button "Buy now" center=(1480,197)
[2] SearchField "Search products" center=(773,764)
[3] Button "Add to cart" center=(1800,708)
[4] Button "Checkout" center=(980,195)
[5] Button "Filters" center=(503,117)

Instruction: press filters

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(424,817)
[1] Link "Sign in" center=(1794,627)
[2] Button "Run inference" center=(1293,428)
[3] Button "Reload" center=(1800,830)
[4] Button "Extensions" center=(461,101)

Instruction: reload

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(320,824)
[1] Button "Add to cart" center=(547,263)
[2] Button "Buy now" center=(1453,512)
[3] Button "Filters" center=(161,69)
[4] Button "Wishlist" center=(1049,150)
[5] Button "Checkout" center=(1560,737)
[6] SearchField "Search products" center=(691,575)

Instruction: find a product

[Assistant] {"action":"type","id":6,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(1676,810)
[1] Button "Run inference" center=(871,385)
[2] Button "Bookmark" center=(1861,587)
[3] Button "Extensions" center=(1633,542)
[4] Button "Downloads" center=(640,332)

Instruction: press downloads

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1049,230)
[1] Button "Add to cart" center=(331,577)
[2] SearchField "Search products" center=(1782,950)
[3] Button "Wishlist" center=(705,975)
[4] Button "Filters" center=(1149,898)
[5] Button "Buy now" center=(1458,184)
[6] Link "Account" center=(1444,422)

Instruction: type into search products

[Assistant] {"action":"type","id":2,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1640,64)
[1] Link "Forgot password" center=(479,981)
[2] CheckBox "Remember me" center=(676,571)
[3] TextField "Password" center=(1805,209)
[4] TextField "Email" center=(1633,809)
[5] Button "Sign in" center=(521,478)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(74,73)
[1] CheckBox "Do Not Disturb" center=(568,751)
[2] CheckBox "Wi-Fi" center=(522,461)
[3] Button "Privacy & Security" center=(562,489)
[4] Button "Displays" center=(785,571)
[5] Button "Network" center=(677,463)
[6] CheckBox "Bluetooth" center=(902,960)
[7] Slider "Brightness" center=(1677,320)
[8] PopUpButton "Appearance" center=(1099,481)

Instruction: click displays

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(976,380)
[1] TextField "Search files" center=(746,951)
[2] Button "Find" center=(385,628)
[3] Button "Format" center=(945,532)
[4] Button "Open" center=(858,78)

Instruction: find in file

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(405,597)
[1] Button "Buy now" center=(270,763)
[2] Link "Account" center=(678,768)
[3] SearchField "Search products" center=(478,759)
[4] Button "Checkout" center=(816,953)
[5] Button "Filters" center=(180,114)

Instruction: find a product

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(739,387)
[1] Button "Find" center=(1492,685)
[2] Button "Commit" center=(592,428)
[3] Button "Open" center=(758,971)
[4] Button "Format" center=(568,219)
[5] TextField "Search files" center=(1558,447)
[6] Button "Terminal" center=(1065,1033)

Instruction: select format

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(615,386)
[1] Button "Sound" center=(1775,620)
[2] Button "Network" center=(765,730)
[3] CheckBox "Bluetooth" center=(880,846)
[4] Button "Privacy & Security" center=(908,69)
[5] CheckBox "Do Not Disturb" center=(58,276)
[6] CheckBox "Wi-Fi" center=(54,911)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1446,168)
[1] Button "Find" center=(1454,141)
[2] Button "Terminal" center=(1374,917)
[3] Button "Commit" center=(1528,1029)
[4] Button "Format" center=(1608,495)
[5] Button "Open" center=(1563,933)
[6] Button "Run" center=(1684,252)

Instruction: press format

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1657,639)
[1] Button "Buy now" center=(436,304)
[2] Button "Wishlist" center=(100,839)
[3] Button "Filters" center=(1506,659)
[4] Button "Add to cart" center=(1214,127)

Instruction: click wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(532,321)
[1] SearchField "Search products" center=(1711,580)
[2] Button "Checkout" center=(1326,297)
[3] Button "Wishlist" center=(1190,318)
[4] Button "Buy now" center=(426,932)

Instruction: tap wishlist

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1701,225)
[1] Button "Extensions" center=(1256,1017)
[2] Button "Bookmark" center=(767,301)
[3] Button "Run inference" center=(623,667)
[4] Button "New tab" center=(1573,436)
[5] SearchField "Search or enter address" center=(916,629)
[6] Link "Sign in" center=(714,340)
[7] Button "Downloads" center=(323,895)
[8] Link "Pricing" center=(213,799)

Instruction: select downloads

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1560,950)
[1] Button "Commit" center=(1858,993)
[2] Button "Run" center=(1511,925)
[3] Button "Save" center=(176,703)
[4] Button "Find" center=(955,249)
[5] Button "Open" center=(594,353)
[6] TextField "Search files" center=(984,633)
[7] Button "Terminal" center=(1396,841)

Instruction: press open

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1539,566)
[1] PopUpButton "Appearance" center=(1281,538)
[2] CheckBox "Wi-Fi" center=(1511,133)
[3] Button "Displays" center=(1446,1018)
[4] Slider "Brightness" center=(1439,912)
[5] Button "Network" center=(134,94)
[6] Button "Privacy & Security" center=(225,495)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1185,328)
[1] Link "Pricing" center=(224,755)
[2] Link "Sign in" center=(1787,335)
[3] Button "Go forward" center=(395,680)
[4] Button "Go back" center=(1485,952)
[5] Link "Docs" center=(1344,908)
[6] Button "Bookmark" center=(1861,859)
[7] Button "Downloads" center=(542,360)
[8] Button "New tab" center=(1501,399)

Instruction: press new tab

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1388,543)
[1] Slider "Volume" center=(1215,608)
[2] Button "Next" center=(295,715)
[3] Button "Shuffle" center=(466,89)
[4] Button "Fullscreen" center=(323,464)
[5] Button "Previous" center=(1853,373)

Instruction: tap previous

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1532,106)
[1] Slider "Volume" center=(1751,577)
[2] Button "Fullscreen" center=(1812,1023)
[3] Button "Next" center=(999,1022)
[4] Button "Shuffle" center=(116,731)
[5] Button "Play" center=(1862,574)
[6] Button "Previous" center=(1658,648)

Instruction: tap play

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(861,303)
[1] Button "Next" center=(561,430)
[2] Button "Fullscreen" center=(1499,705)
[3] Button "Previous" center=(207,1031)
[4] Button "Shuffle" center=(251,636)
[5] Button "Play" center=(1793,556)

Instruction: tap shuffle

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1494,229)
[1] TextField "Email" center=(533,548)
[2] Link "Forgot password" center=(1517,725)
[3] CheckBox "Remember me" center=(1028,1033)
[4] TextField "Password" center=(638,612)
[5] Button "Sign in" center=(1421,898)

Instruction: tap remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(361,128)
[1] Button "Bookmark" center=(1577,705)
[2] Button "Extensions" center=(1529,829)
[3] Link "Docs" center=(1176,888)
[4] Button "Reload" center=(103,603)
[5] SearchField "Search or enter address" center=(374,813)

Instruction: go to docs

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1737,624)
[1] Link "Pricing" center=(1364,409)
[2] Button "Downloads" center=(980,104)
[3] Button "Reload" center=(879,354)
[4] Button "Bookmark" center=(864,816)
[5] Button "New tab" center=(1730,724)
[6] Button "Go forward" center=(1124,176)
[7] Button "Go back" center=(262,826)

Instruction: i want to log in

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1333,564)
[1] Button "Format" center=(1579,790)
[2] Button "New File" center=(893,981)
[3] Button "Commit" center=(1501,761)
[4] Button "Open" center=(1604,485)
[5] Button "Terminal" center=(1578,866)
[6] Button "Run" center=(1227,270)
[7] TextField "Search files" center=(376,77)
[8] Button "Save" center=(1061,859)

Instruction: find in file

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1519,351)
[1] Button "Create account" center=(1147,490)
[2] CheckBox "Remember me" center=(1811,459)
[3] TextField "Password" center=(1441,220)
[4] Button "Sign in" center=(1300,459)
[5] Link "Forgot password" center=(751,100)

Instruction: select remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(849,409)
[1] TextField "Password" center=(1426,322)
[2] TextField "Email" center=(223,831)
[3] CheckBox "Remember me" center=(1264,562)
[4] Button "Sign in" center=(368,554)
[5] Button "Create account" center=(250,497)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(164,230)
[1] TextField "Search files" center=(935,1038)
[2] Button "Find" center=(407,446)
[3] Button "New File" center=(1292,1040)
[4] Button "Open" center=(502,99)
[5] Button "Commit" center=(483,168)

Instruction: select new file

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1292,972)
[1] Link "Forgot password" center=(1505,817)
[2] TextField "Email" center=(107,512)
[3] Button "Create account" center=(1583,833)
[4] Button "Sign in" center=(1719,304)

Instruction: tap remember me

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(992,449)
[1] CheckBox "Wi-Fi" center=(1565,530)
[2] PopUpButton "Appearance" center=(793,165)
[3] Slider "Brightness" center=(296,390)
[4] CheckBox "Do Not Disturb" center=(1490,611)
[5] Button "Displays" center=(449,533)
[6] Button "Network" center=(1596,688)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(484,304)
[1] CheckBox "Do Not Disturb" center=(305,193)
[2] Button "Network" center=(1131,1006)
[3] Slider "Brightness" center=(766,753)
[4] Button "Sound" center=(859,203)
[5] CheckBox "Bluetooth" center=(1623,1009)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(112,278)
[1] CheckBox "Wi-Fi" center=(1698,274)
[2] PopUpButton "Appearance" center=(1509,522)
[3] CheckBox "Do Not Disturb" center=(1728,825)
[4] Button "Privacy & Security" center=(1541,755)

Instruction: click displays

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1875,74)
[1] Button "Add to cart" center=(458,800)
[2] Button "Filters" center=(952,51)
[3] Button "Buy now" center=(587,242)
[4] Button "Checkout" center=(431,278)
[5] SearchField "Search products" center=(1856,819)

Instruction: press wishlist

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1037,952)
[1] Button "Buy now" center=(1161,680)
[2] SearchField "Search products" center=(65,315)
[3] Button "Add to cart" center=(991,264)
[4] Button "Wishlist" center=(940,143)
[5] Button "Filters" center=(1253,576)
[6] Button "Checkout" center=(1680,767)

Instruction: type into search products

[Assistant] {"action":"type","id":2,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(73,955)
[1] Button "Pause" center=(1376,682)
[2] Button "Next" center=(830,342)
[3] Slider "Volume" center=(796,904)
[4] Button "Previous" center=(1485,405)
[5] Button "Play" center=(669,349)
[6] Button "Shuffle" center=(1530,891)

Instruction: go fullscreen

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(445,703)
[1] Button "Format" center=(1273,478)
[2] Button "Find" center=(1109,582)
[3] Button "Open" center=(782,671)
[4] Button "Terminal" center=(458,309)
[5] Button "Save" center=(1526,1024)
[6] TextField "Search files" center=(311,578)
[7] Button "Commit" center=(1178,258)

Instruction: execute

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1460,421)
[1] Button "Add to cart" center=(86,283)
[2] Button "Checkout" center=(377,136)
[3] Button "Buy now" center=(1256,562)
[4] Button "Filters" center=(806,815)

Instruction: select buy now

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(932,484)
[1] Link "Forgot password" center=(1835,973)
[2] Button "Create account" center=(1548,704)
[3] TextField "Password" center=(1113,454)
[4] CheckBox "Remember me" center=(1346,727)
[5] TextField "Email" center=(1611,77)

Instruction: type into password

[Assistant] {"action":"type","id":3,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(331,612)
[1] CheckBox "Bluetooth" center=(1235,406)
[2] Button "Sound" center=(1523,670)
[3] CheckBox "Do Not Disturb" center=(222,951)
[4] CheckBox "Wi-Fi" center=(1536,40)
[5] Button "Privacy & Security" center=(513,580)

Instruction: press sound

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1426,492)
[1] TextField "Search files" center=(644,250)
[2] Button "Run" center=(1213,732)
[3] Button "Find" center=(889,273)
[4] Button "Terminal" center=(1261,658)
[5] Button "Save" center=(566,669)

Instruction: select terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(446,116)
[1] Button "New tab" center=(1002,99)
[2] Button "Go forward" center=(1091,542)
[3] Button "Extensions" center=(484,72)
[4] Button "Reload" center=(149,193)
[5] Link "Pricing" center=(1552,912)
[6] Link "Docs" center=(226,56)
[7] SearchField "Search or enter address" center=(1396,189)
[8] Button "Downloads" center=(1547,866)
[9] Button "Run inference" center=(997,1002)

Instruction: go to docs

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1074,502)
[1] CheckBox "Do Not Disturb" center=(588,617)
[2] CheckBox "Wi-Fi" center=(477,652)
[3] Button "Displays" center=(173,476)
[4] CheckBox "Bluetooth" center=(1407,440)
[5] Button "Network" center=(50,856)
[6] PopUpButton "Appearance" center=(236,634)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1624,827)
[1] Button "New File" center=(1182,947)
[2] Button "Find" center=(1209,683)
[3] Button "Run" center=(1454,242)
[4] Button "Commit" center=(1400,520)
[5] Button "Format" center=(1092,405)
[6] Button "Open" center=(1297,411)
[7] Button "Terminal" center=(1330,1001)

Instruction: tap terminal

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(624,398)
[1] Link "Docs" center=(1578,542)
[2] Button "Downloads" center=(1708,292)
[3] Button "New tab" center=(1505,318)
[4] Button "Reload" center=(897,485)

Instruction: select downloads

[Assistant] {"action":"click","id":2}