[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(134,523)
[1] Button "Next" center=(785,432)
[2] Button "Previous" center=(347,223)
[3] Button "Fullscreen" center=(1796,76)
[4] Button "Pause" center=(1172,862)

Instruction: go fullscreen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(497,863)
[1] Button "Run" center=(1096,87)
[2] TextField "Search files" center=(861,110)
[3] Button "New File" center=(1836,516)
[4] Button "Terminal" center=(1813,329)
[5] Button "Save" center=(1624,905)

Instruction: tap open

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(53,994)
[1] Button "Extensions" center=(879,64)
[2] Button "Go back" center=(517,594)
[3] Link "Docs" center=(749,691)
[4] Link "Sign in" center=(1759,750)
[5] Button "Reload" center=(1836,49)
[6] Link "Pricing" center=(726,41)

Instruction: select docs

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1475,957)
[1] Button "New tab" center=(802,509)
[2] Button "Run inference" center=(1571,126)
[3] SearchField "Search or enter address" center=(1229,137)
[4] Button "Reload" center=(1075,176)
[5] Button "Go forward" center=(1345,879)
[6] Button "Go back" center=(847,116)
[7] Link "Pricing" center=(1252,621)
[8] Link "Docs" center=(167,485)
[9] Button "Extensions" center=(1396,1027)
[10] Link "Sign in" center=(307,282)

Instruction: tap go forward

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(192,1006)
[1] Button "Previous" center=(455,179)
[2] Button "Pause" center=(1637,828)
[3] Slider "Volume" center=(1237,151)
[4] Button "Next" center=(358,148)

Instruction: select volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(225,201)
[1] Button "Find" center=(1693,742)
[2] Button "Terminal" center=(663,886)
[3] Button "New File" center=(1483,753)
[4] Button "Run" center=(1275,81)

Instruction: press terminal

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(1178,806)
[1] SearchField "Search or enter address" center=(1166,295)
[2] Link "Pricing" center=(233,513)
[3] Button "Go forward" center=(248,863)
[4] Link "Sign in" center=(1665,183)
[5] Button "Reload" center=(288,52)
[6] Button "Run inference" center=(166,890)
[7] Button "Go back" center=(1639,267)
[8] Link "Docs" center=(307,242)
[9] Button "Downloads" center=(1726,450)
[10] Button "Bookmark" center=(802,742)
[11] Button "Extensions" center=(1330,1000)

Instruction: select downloads

[Assistant] {"action":"click","id":9}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1215,483)
[1] Button "Run inference" center=(1722,781)
[2] SearchField "Search or enter address" center=(848,548)
[3] Button "New tab" center=(100,431)
[4] Link "Docs" center=(1439,476)
[5] Link "Sign in" center=(393,402)
[6] Button "Extensions" center=(479,970)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":2,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(883,943)
[1] Button "Run" center=(1167,530)
[2] TextField "Search files" center=(164,129)
[3] Button "Open" center=(612,432)
[4] Button "Format" center=(321,469)
[5] Button "Commit" center=(448,698)
[6] Button "Find" center=(1851,856)
[7] Button "Save" center=(1116,295)

Instruction: tap terminal

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(593,224)
[1] Button "Fullscreen" center=(93,366)
[2] Button "Pause" center=(1262,264)
[3] Button "Shuffle" center=(100,957)
[4] Slider "Volume" center=(1700,837)
[5] Button "Previous" center=(615,99)

Instruction: select shuffle

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1132,875)
[1] SearchField "Search products" center=(1660,979)
[2] Button "Checkout" center=(1549,97)
[3] Link "Account" center=(1604,1010)
[4] Button "Add to cart" center=(489,629)
[5] Button "Filters" center=(1773,856)

Instruction: find a product

[Assistant] {"action":"type","id":1,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1194,185)
[1] Button "Pause" center=(1193,1011)
[2] Button "Fullscreen" center=(425,896)
[3] Button "Play" center=(1699,942)
[4] Slider "Volume" center=(1048,595)
[5] Button "Next" center=(672,219)
[6] Button "Shuffle" center=(1044,78)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(452,865)
[1] Button "Filters" center=(307,388)
[2] Link "Account" center=(1508,221)
[3] Button "Wishlist" center=(1683,904)
[4] SearchField "Search products" center=(1818,371)
[5] Button "Add to cart" center=(161,62)
[6] Button "Buy now" center=(342,640)

Instruction: click filters

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(246,384)
[1] Button "Buy now" center=(658,370)
[2] Button "Wishlist" center=(319,1039)
[3] Link "Account" center=(362,789)
[4] Button "Add to cart" center=(934,850)
[5] Button "Checkout" center=(1349,1023)
[6] Button "Filters" center=(1037,704)

Instruction: select checkout

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(841,911)
[1] Button "Displays" center=(1618,1028)
[2] CheckBox "Wi-Fi" center=(1202,73)
[3] CheckBox "Bluetooth" center=(412,656)
[4] Slider "Brightness" center=(691,983)
[5] Button "Sound" center=(1696,1030)
[6] PopUpButton "Appearance" center=(1292,266)
[7] Button "Privacy & Security" center=(1349,619)
[8] CheckBox "Do Not Disturb" center=(253,999)

Instruction: click sound

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(469,295)
[1] Button "Displays" center=(1459,109)
[2] Button "Privacy & Security" center=(757,302)
[3] CheckBox "Do Not Disturb" center=(1651,150)
[4] CheckBox "Wi-Fi" center=(1515,41)
[5] Slider "Brightness" center=(138,435)
[6] CheckBox "Bluetooth" center=(937,790)

Instruction: tap displays

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(964,115)
[1] TextField "Search files" center=(1676,906)
[2] Button "New File" center=(108,341)
[3] Button "Terminal" center=(81,366)
[4] Button "Save" center=(574,1014)
[5] Button "Run" center=(254,115)

Instruction: press terminal

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(103,481)
[1] Button "Pause" center=(1378,208)
[2] Button "Previous" center=(1283,951)
[3] Button "Next" center=(929,200)
[4] Button "Fullscreen" center=(149,143)
[5] Slider "Volume" center=(718,250)
[6] Button "Play" center=(429,459)

Instruction: select volume

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1165,900)
[1] Slider "Volume" center=(1045,197)
[2] Button "Previous" center=(151,409)
[3] Button "Play" center=(47,477)
[4] Button "Pause" center=(219,341)
[5] Button "Fullscreen" center=(1727,717)
[6] Button "Next" center=(1320,654)

Instruction: click shuffle

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1872,716)
[1] Button "Sound" center=(582,133)
[2] CheckBox "Bluetooth" center=(1354,597)
[3] PopUpButton "Appearance" center=(833,229)
[4] Button "Privacy & Security" center=(799,946)
[5] Button "Network" center=(821,418)
[6] CheckBox "Wi-Fi" center=(422,496)
[7] Button "Displays" center=(131,309)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(887,1034)
[1] Link "Account" center=(1589,643)
[2] Button "Checkout" center=(505,606)
[3] SearchField "Search products" center=(164,900)
[4] Button "Filters" center=(832,878)
[5] Button "Buy now" center=(1110,468)
[6] Button "Add to cart" center=(1136,743)

Instruction: search for something

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(382,1035)
[1] Button "Commit" center=(473,867)
[2] TextField "Search files" center=(434,169)
[3] Button "Terminal" center=(127,473)
[4] Button "Save" center=(200,739)
[5] Button "Format" center=(931,246)

Instruction: press format

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(707,917)
[1] Link "Account" center=(605,572)
[2] Button "Buy now" center=(978,979)
[3] SearchField "Search products" center=(1828,52)
[4] Button "Add to cart" center=(1650,757)
[5] Button "Filters" center=(1316,641)
[6] Button "Checkout" center=(1110,840)

Instruction: search for something

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(392,322)
[1] SearchField "Search or enter address" center=(865,187)
[2] Link "Pricing" center=(1306,663)
[3] Button "Extensions" center=(1434,750)
[4] Link "Docs" center=(862,79)

Instruction: press pricing

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(315,347)
[1] SearchField "Search or enter address" center=(1563,493)
[2] Button "Go forward" center=(1625,885)
[3] Button "Run inference" center=(697,315)
[4] Link "Sign in" center=(1532,466)
[5] Link "Docs" center=(1278,701)
[6] Button "Reload" center=(222,232)
[7] Button "Go back" center=(944,256)

Instruction: select reload

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(1754,99)
[1] SearchField "Search or enter address" center=(550,73)
[2] Button "Bookmark" center=(974,91)
[3] Button "Go back" center=(778,761)
[4] Button "Downloads" center=(447,320)
[5] Button "Extensions" center=(794,883)
[6] Button "Run inference" center=(990,557)
[7] Link "Docs" center=(852,690)

Instruction: press pricing

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(745,207)
[1] TextField "Password" center=(1755,955)
[2] TextField "Email" center=(569,806)
[3] CheckBox "Remember me" center=(1741,745)
[4] Link "Forgot password" center=(225,331)
[5] Button "Create account" center=(96,435)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1010,973)
[1] CheckBox "Wi-Fi" center=(310,702)
[2] Button "Displays" center=(394,655)
[3] Slider "Brightness" center=(1680,48)
[4] CheckBox "Do Not Disturb" center=(498,298)

Instruction: select displays

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(249,47)
[1] Button "Terminal" center=(1087,718)
[2] Button "Find" center=(1767,899)
[3] TextField "Search files" center=(806,614)
[4] Button "Open" center=(1365,648)
[5] Button "New File" center=(1287,329)
[6] Button "Run" center=(651,740)
[7] Button "Format" center=(261,533)
[8] Button "Commit" center=(170,384)

Instruction: select new file

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1689,917)
[1] Link "Forgot password" center=(1683,869)
[2] TextField "Email" center=(1268,453)
[3] TextField "Password" center=(1392,367)
[4] Button "Sign in" center=(1434,183)

Instruction: type into password

[Assistant] {"action":"type","id":3,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(742,783)
[1] TextField "Email" center=(1500,267)
[2] TextField "Password" center=(1445,587)
[3] Link "Forgot password" center=(1347,843)
[4] Button "Create account" center=(1014,968)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(302,798)
[1] Button "Add to cart" center=(569,782)
[2] Button "Wishlist" center=(1695,612)
[3] Link "Account" center=(1602,974)
[4] SearchField "Search products" center=(414,1009)
[5] Button "Buy now" center=(1432,961)

Instruction: press filters

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1239,341)
[1] Link "Forgot password" center=(989,911)
[2] CheckBox "Remember me" center=(281,521)
[3] Button "Create account" center=(149,927)
[4] Button "Sign in" center=(1339,637)
[5] TextField "Password" center=(1228,1006)

Instruction: click create account

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(450,390)
[1] Button "Shuffle" center=(878,623)
[2] Button "Play" center=(1868,968)
[3] Button "Fullscreen" center=(977,186)
[4] Button "Next" center=(687,923)

Instruction: tap pause

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(965,320)
[1] TextField "Email" center=(468,819)
[2] Button "Create account" center=(1624,212)
[3] Button "Sign in" center=(1220,822)
[4] TextField "Password" center=(364,541)

Instruction: reset password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1133,244)
[1] Button "Play" center=(1536,422)
[2] Button "Pause" center=(68,953)
[3] Slider "Volume" center=(699,815)
[4] Button "Shuffle" center=(1746,583)

Instruction: skip to next track

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(650,1011)
[1] Slider "Brightness" center=(635,506)
[2] PopUpButton "Appearance" center=(1530,572)
[3] CheckBox "Do Not Disturb" center=(1261,576)
[4] Button "Network" center=(731,484)

Instruction: select brightness

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1543,62)
[1] Button "New File" center=(1247,556)
[2] Button "Find" center=(386,166)
[3] Button "Commit" center=(790,768)
[4] Button "Save" center=(1372,851)
[5] Button "Terminal" center=(116,424)

Instruction: click find

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(949,429)
[1] Button "Checkout" center=(293,600)
[2] Button "Wishlist" center=(557,525)
[3] SearchField "Search products" center=(1477,196)
[4] Button "Filters" center=(460,751)
[5] Link "Account" center=(1444,685)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1741,960)
[1] Button "Fullscreen" center=(254,563)
[2] Button "Previous" center=(1512,672)
[3] Button "Next" center=(281,329)
[4] Slider "Volume" center=(273,843)
[5] Button "Shuffle" center=(257,549)

Instruction: press fullscreen

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(465,744)
[1] Link "Forgot password" center=(1024,378)
[2] TextField "Email" center=(520,42)
[3] Button "Sign in" center=(66,727)
[4] Button "Create account" center=(222,968)
[5] CheckBox "Remember me" center=(266,895)

Instruction: log in

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(646,310)
[1] SearchField "Search products" center=(990,91)
[2] Button "Wishlist" center=(244,872)
[3] Button "Checkout" center=(397,935)
[4] Button "Filters" center=(120,912)
[5] Link "Account" center=(635,408)

Instruction: tap wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(772,955)
[1] Button "Next" center=(936,1018)
[2] Button "Previous" center=(1313,314)
[3] Slider "Volume" center=(1839,790)
[4] Button "Shuffle" center=(230,421)
[5] Button "Fullscreen" center=(1709,390)

Instruction: tap volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1769,203)
[1] Button "New File" center=(252,466)
[2] Button "Save" center=(1249,534)
[3] Button "Open" center=(1683,622)
[4] Button "Terminal" center=(1429,488)

Instruction: select open

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(942,87)
[1] Button "Filters" center=(169,307)
[2] SearchField "Search products" center=(1381,361)
[3] Button "Wishlist" center=(65,761)
[4] Button "Buy now" center=(1818,734)
[5] Link "Account" center=(1114,825)

Instruction: press account

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(748,719)
[1] Button "Previous" center=(1425,696)
[2] Button "Pause" center=(828,192)
[3] Button "Play" center=(1758,886)
[4] Slider "Volume" center=(1183,953)
[5] Button "Next" center=(1295,95)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(533,378)
[1] SearchField "Search products" center=(609,91)
[2] Button "Add to cart" center=(1078,268)
[3] Button "Buy now" center=(1167,698)
[4] Link "Account" center=(815,452)
[5] Button "Wishlist" center=(1713,280)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1566,709)
[1] Link "Account" center=(76,145)
[2] Button "Buy now" center=(866,462)
[3] SearchField "Search products" center=(113,610)
[4] Button "Filters" center=(1164,40)
[5] Button "Add to cart" center=(1731,133)

Instruction: select filters

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1769,505)
[1] Button "Add to cart" center=(719,613)
[2] Link "Account" center=(1611,674)
[3] Button "Checkout" center=(1268,524)
[4] Button "Wishlist" center=(1507,712)

Instruction: proceed to payment

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1876,208)
[1] PopUpButton "Appearance" center=(1048,183)
[2] Button "Network" center=(1723,774)
[3] Button "Displays" center=(1560,778)
[4] Button "Privacy & Security" center=(878,149)

Instruction: press appearance

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(1061,44)
[1] Button "Reload" center=(1126,793)
[2] Button "Go forward" center=(139,606)
[3] Button "New tab" center=(926,898)
[4] Button "Extensions" center=(1238,534)
[5] Link "Sign in" center=(1081,839)
[6] Button "Go back" center=(402,748)

Instruction: click sign in

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(967,864)
[1] Button "Terminal" center=(313,181)
[2] TextField "Search files" center=(885,656)
[3] Button "New File" center=(1473,947)
[4] Button "Open" center=(1572,564)
[5] Button "Find" center=(652,903)
[6] Button "Run" center=(1476,600)

Instruction: press find

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1598,400)
[1] PopUpButton "Appearance" center=(1191,531)
[2] Button "Privacy & Security" center=(1233,624)
[3] Slider "Brightness" center=(1406,767)
[4] CheckBox "Bluetooth" center=(894,730)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(1439,296)
[1] Button "New tab" center=(339,899)
[2] Button "Bookmark" center=(167,747)
[3] SearchField "Search or enter address" center=(1646,342)
[4] Button "Extensions" center=(457,811)
[5] Button "Downloads" center=(1148,73)
[6] Button "Run inference" center=(774,873)
[7] Button "Reload" center=(966,144)
[8] Link "Docs" center=(1301,928)

Instruction: tap docs

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1170,95)
[1] CheckBox "Do Not Disturb" center=(66,801)
[2] CheckBox "Bluetooth" center=(554,247)
[3] Button "Network" center=(1310,929)
[4] CheckBox "Wi-Fi" center=(1639,921)
[5] Button "Privacy & Security" center=(1822,493)
[6] Button "Sound" center=(1739,270)

Instruction: select privacy & security

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(959,221)
[1] Button "Add to cart" center=(1438,116)
[2] Button "Checkout" center=(422,1025)
[3] Button "Buy now" center=(395,792)
[4] Button "Filters" center=(1092,162)
[5] Link "Account" center=(812,81)

Instruction: select filters

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1262,92)
[1] Button "Sound" center=(1624,994)
[2] Button "Privacy & Security" center=(1320,670)
[3] CheckBox "Bluetooth" center=(1766,717)
[4] Button "Network" center=(656,542)
[5] PopUpButton "Appearance" center=(1657,626)
[6] CheckBox "Wi-Fi" center=(1166,406)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1152,589)
[1] Button "Wishlist" center=(866,134)
[2] Button "Checkout" center=(755,266)
[3] Button "Add to cart" center=(1849,258)
[4] SearchField "Search products" center=(1398,379)
[5] Button "Filters" center=(718,409)

Instruction: search for something

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(1146,781)
[1] Button "Go forward" center=(619,65)
[2] Link "Docs" center=(1098,881)
[3] Button "Run inference" center=(1617,423)
[4] Button "Reload" center=(1093,862)
[5] Button "Extensions" center=(1081,496)
[6] Button "Go back" center=(1036,84)
[7] Link "Sign in" center=(647,959)
[8] Button "Bookmark" center=(1374,237)

Instruction: click go forward

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(461,590)
[1] Button "Network" center=(1737,759)
[2] CheckBox "Do Not Disturb" center=(1839,1039)
[3] Button "Displays" center=(1558,833)
[4] CheckBox "Bluetooth" center=(931,702)

Instruction: press privacy & security

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(874,437)
[1] Button "Bookmark" center=(963,685)
[2] Button "New tab" center=(868,483)
[3] Link "Docs" center=(687,57)
[4] Button "Reload" center=(837,729)
[5] Button "Extensions" center=(282,587)
[6] Button "Run inference" center=(52,696)
[7] Link "Sign in" center=(1515,673)
[8] Button "Go forward" center=(169,814)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":0,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(217,328)
[1] Slider "Volume" center=(1607,988)
[2] Button "Pause" center=(936,430)
[3] Button "Shuffle" center=(849,881)
[4] Button "Next" center=(1393,71)
[5] Button "Play" center=(1010,200)
[6] Button "Fullscreen" center=(1132,265)

Instruction: select shuffle

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(125,45)
[1] Button "Previous" center=(1212,650)
[2] Slider "Volume" center=(847,590)
[3] Button "Play" center=(1161,1009)
[4] Button "Shuffle" center=(214,352)

Instruction: press volume

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1414,858)
[1] Button "Run" center=(1201,912)
[2] Button "Save" center=(151,911)
[3] Button "Terminal" center=(1578,994)
[4] Button "Format" center=(1164,512)
[5] Button "Commit" center=(1360,602)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1414,711)
[1] Button "Sign in" center=(520,142)
[2] Link "Forgot password" center=(623,674)
[3] Button "Create account" center=(182,961)
[4] TextField "Password" center=(436,365)

Instruction: reset password

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(49,360)
[1] Button "Previous" center=(544,844)
[2] Button "Shuffle" center=(493,769)
[3] Button "Next" center=(1555,696)
[4] Button "Fullscreen" center=(933,323)

Instruction: select previous

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1798,886)
[1] Button "Format" center=(326,952)
[2] Button "Find" center=(1143,241)
[3] Button "Save" center=(579,791)
[4] Button "Run" center=(1855,681)
[5] Button "Commit" center=(196,784)
[6] Button "New File" center=(194,953)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1169,306)
[1] Button "Fullscreen" center=(1153,955)
[2] Button "Next" center=(1145,523)
[3] Button "Previous" center=(1379,833)
[4] Button "Shuffle" center=(344,414)

Instruction: select shuffle

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1441,209)
[1] Button "Privacy & Security" center=(1590,566)
[2] Button "Sound" center=(327,496)
[3] CheckBox "Wi-Fi" center=(366,223)
[4] Button "Displays" center=(1199,1011)
[5] CheckBox "Do Not Disturb" center=(1198,715)
[6] Button "Network" center=(1826,1030)
[7] CheckBox "Bluetooth" center=(339,793)
[8] Slider "Brightness" center=(1539,535)

Instruction: press appearance

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1660,409)
[1] TextField "Search files" center=(834,520)
[2] Button "Format" center=(971,942)
[3] Button "Save" center=(135,80)
[4] Button "Terminal" center=(1368,371)
[5] Button "Commit" center=(251,760)
[6] Button "New File" center=(1107,688)
[7] Button "Open" center=(1739,320)
[8] Button "Run" center=(1749,799)

Instruction: execute

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(496,568)
[1] Button "Pause" center=(871,401)
[2] Button "Shuffle" center=(1713,928)
[3] Button "Play" center=(746,538)
[4] Slider "Volume" center=(1663,533)
[5] Button "Next" center=(1856,658)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(781,656)
[1] Button "Format" center=(1509,876)
[2] Button "Commit" center=(282,716)
[3] Button "Open" center=(511,464)
[4] TextField "Search files" center=(1701,378)
[5] Button "Find" center=(1515,156)

Instruction: tap open

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1329,564)
[1] Button "Displays" center=(120,952)
[2] PopUpButton "Appearance" center=(1212,190)
[3] Button "Sound" center=(154,210)
[4] Slider "Brightness" center=(568,984)
[5] CheckBox "Bluetooth" center=(408,569)

Instruction: click brightness

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1005,928)
[1] Button "Pause" center=(1026,214)
[2] Button "Previous" center=(489,890)
[3] Button "Play" center=(881,655)
[4] Button "Next" center=(332,853)
[5] Button "Fullscreen" center=(1233,880)
[6] Button "Shuffle" center=(1611,226)

Instruction: next song

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1800,971)
[1] Button "New File" center=(1072,740)
[2] Button "Format" center=(899,886)
[3] Button "Save" center=(346,660)
[4] TextField "Search files" center=(237,221)
[5] Button "Commit" center=(1014,963)
[6] Button "Terminal" center=(1299,654)
[7] Button "Run" center=(119,938)
[8] Button "Open" center=(1645,584)

Instruction: click new file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(918,274)
[1] Button "Terminal" center=(1683,533)
[2] Button "Open" center=(1657,827)
[3] Button "Find" center=(820,398)
[4] Button "Run" center=(1246,863)
[5] Button "Format" center=(1558,969)
[6] Button "Commit" center=(1599,897)
[7] Button "Save" center=(1466,629)
[8] Button "New File" center=(240,475)

Instruction: click save

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1492,801)
[1] TextField "Email" center=(1656,47)
[2] TextField "Password" center=(1223,575)
[3] Button "Create account" center=(281,572)
[4] CheckBox "Remember me" center=(54,424)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(788,666)
[1] TextField "Search files" center=(604,736)
[2] Button "Commit" center=(853,813)
[3] Button "New File" center=(457,463)
[4] Button "Find" center=(1170,842)
[5] Button "Run" center=(1181,534)
[6] Button "Terminal" center=(406,419)
[7] Button "Open" center=(1185,608)
[8] Button "Format" center=(1030,299)

Instruction: tap run

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(479,560)
[1] Slider "Volume" center=(860,844)
[2] Button "Previous" center=(733,860)
[3] Button "Fullscreen" center=(1167,668)
[4] Button "Play" center=(1467,740)

Instruction: tap volume

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(171,309)
[1] Slider "Volume" center=(494,419)
[2] Button "Next" center=(203,663)
[3] Button "Previous" center=(117,781)
[4] Button "Pause" center=(827,993)

Instruction: select pause

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1781,712)
[1] Button "Shuffle" center=(1686,570)
[2] Button "Fullscreen" center=(625,239)
[3] Button "Previous" center=(108,842)
[4] Button "Pause" center=(1267,562)

Instruction: press shuffle

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1636,449)
[1] Button "Shuffle" center=(763,725)
[2] Button "Previous" center=(849,455)
[3] Button "Fullscreen" center=(1240,245)
[4] Button "Pause" center=(1100,55)
[5] Button "Next" center=(1549,760)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1281,1026)
[1] Button "Wishlist" center=(438,505)
[2] Button "Buy now" center=(867,628)
[3] Button "Filters" center=(49,307)
[4] SearchField "Search products" center=(398,456)

Instruction: find a product

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(812,616)
[1] TextField "Password" center=(1030,828)
[2] Button "Create account" center=(538,815)
[3] Button "Sign in" center=(735,1024)
[4] Link "Forgot password" center=(1628,509)
[5] TextField "Email" center=(1435,511)

Instruction: tap remember me

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(548,334)
[1] Button "Go back" center=(230,448)
[2] Button "Extensions" center=(1419,806)
[3] Button "Downloads" center=(1721,390)
[4] Link "Pricing" center=(195,607)
[5] Button "Reload" center=(405,527)
[6] Button "Run inference" center=(682,402)
[7] Link "Sign in" center=(487,955)
[8] Button "Bookmark" center=(1214,315)

Instruction: tap downloads

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1749,295)
[1] Button "Previous" center=(1752,673)
[2] Button "Play" center=(769,199)
[3] Button "Next" center=(1635,149)
[4] Button "Shuffle" center=(285,982)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1879,942)
[1] Button "Checkout" center=(1877,628)
[2] Button "Buy now" center=(1819,321)
[3] Button "Wishlist" center=(1045,227)
[4] Button "Filters" center=(1605,977)
[5] SearchField "Search products" center=(1086,326)

Instruction: press filters

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(686,745)
[1] Button "Run inference" center=(57,52)
[2] SearchField "Search or enter address" center=(1420,255)
[3] Button "Go forward" center=(817,634)
[4] Button "Bookmark" center=(246,373)
[5] Button "Reload" center=(780,667)
[6] Button "New tab" center=(1805,305)
[7] Button "Go back" center=(423,601)
[8] Button "Extensions" center=(578,282)
[9] Link "Pricing" center=(591,522)
[10] Link "Docs" center=(1295,391)
[11] Link "Sign in" center=(789,228)

Instruction: tap go forward

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1210,174)
[1] Slider "Volume" center=(1219,742)
[2] Button "Pause" center=(508,873)
[3] Button "Previous" center=(244,778)
[4] Button "Shuffle" center=(592,863)

Instruction: tap previous

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1133,777)
[1] Button "Sound" center=(686,844)
[2] Button "Displays" center=(1036,347)
[3] Button "Privacy & Security" center=(553,615)
[4] CheckBox "Bluetooth" center=(1798,621)
[5] CheckBox "Wi-Fi" center=(137,447)
[6] Button "Network" center=(242,747)
[7] PopUpButton "Appearance" center=(374,101)
[8] CheckBox "Do Not Disturb" center=(481,873)

Instruction: tap sound

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(80,461)
[1] Button "Pause" center=(288,399)
[2] Button "Previous" center=(1816,735)
[3] Slider "Volume" center=(1280,243)
[4] Button "Shuffle" center=(1221,629)
[5] Button "Fullscreen" center=(1370,704)

Instruction: select previous

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(820,668)
[1] TextField "Password" center=(626,514)
[2] Button "Sign in" center=(1333,1019)
[3] Button "Create account" center=(336,929)
[4] Link "Forgot password" center=(383,695)

Instruction: sign in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(688,708)
[1] SearchField "Search products" center=(636,767)
[2] Button "Filters" center=(587,659)
[3] Button "Buy now" center=(1360,444)
[4] Link "Account" center=(595,948)
[5] Button "Add to cart" center=(951,839)

Instruction: tap filters

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(471,210)
[1] Button "Run" center=(853,145)
[2] Button "Terminal" center=(235,105)
[3] Button "Find" center=(1839,524)
[4] Button "Open" center=(1163,95)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1381,993)
[1] SearchField "Search products" center=(1853,439)
[2] Button "Wishlist" center=(673,274)
[3] Button "Checkout" center=(546,795)
[4] Link "Account" center=(621,493)

Instruction: search for something

[Assistant] {"action":"type","id":1,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1332,806)
[1] Link "Sign in" center=(1342,938)
[2] Button "Extensions" center=(1195,485)
[3] Link "Docs" center=(1112,695)
[4] Button "Reload" center=(1349,586)
[5] Button "New tab" center=(674,970)
[6] Button "Bookmark" center=(751,533)
[7] Link "Pricing" center=(1145,272)
[8] Button "Run inference" center=(1688,131)
[9] Button "Downloads" center=(944,355)
[10] SearchField "Search or enter address" center=(789,462)
[11] Button "Go back" center=(572,199)

Instruction: press go forward

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(653,49)
[1] TextField "Password" center=(1608,480)
[2] CheckBox "Remember me" center=(1674,339)
[3] TextField "Email" center=(1262,276)
[4] Button "Create account" center=(1671,47)
[5] Button "Sign in" center=(1162,422)

Instruction: make a new account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1605,859)
[1] Button "Displays" center=(1295,866)
[2] Button "Sound" center=(1461,741)
[3] Button "Network" center=(596,331)
[4] CheckBox "Wi-Fi" center=(1368,800)
[5] CheckBox "Do Not Disturb" center=(1129,868)
[6] CheckBox "Bluetooth" center=(590,462)
[7] PopUpButton "Appearance" center=(425,913)
[8] Button "Privacy & Security" center=(180,226)

Instruction: tap displays

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(351,90)
[1] Button "Buy now" center=(1135,62)
[2] Button "Add to cart" center=(420,1035)
[3] Button "Filters" center=(249,328)
[4] SearchField "Search products" center=(591,168)

Instruction: press account

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(204,903)
[1] SearchField "Search or enter address" center=(1577,1014)
[2] Button "Go forward" center=(339,97)
[3] Button "Run inference" center=(585,834)
[4] Link "Docs" center=(843,113)
[5] Button "Extensions" center=(1152,521)
[6] Link "Sign in" center=(411,236)
[7] Button "Downloads" center=(1874,901)
[8] Button "Bookmark" center=(552,876)
[9] Link "Pricing" center=(821,888)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1403,253)
[1] Button "Commit" center=(1008,438)
[2] Button "New File" center=(1435,1024)
[3] TextField "Search files" center=(679,998)
[4] Button "Open" center=(701,202)
[5] Button "Save" center=(1328,785)
[6] Button "Format" center=(126,802)
[7] Button "Run" center=(1165,573)
[8] Button "Terminal" center=(290,299)

Instruction: press open

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(70,211)
[1] Link "Forgot password" center=(1666,322)
[2] Button "Create account" center=(1237,162)
[3] CheckBox "Remember me" center=(850,848)
[4] TextField "Password" center=(537,49)
[5] TextField "Email" center=(126,635)

Instruction: press remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1595,680)
[1] CheckBox "Remember me" center=(162,101)
[2] Button "Create account" center=(768,700)
[3] Link "Forgot password" center=(355,758)
[4] TextField "Password" center=(1138,850)
[5] TextField "Email" center=(259,168)

Instruction: select create account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1487,488)
[1] Button "Terminal" center=(1184,341)
[2] Button "Commit" center=(1151,753)
[3] Button "Open" center=(1043,234)
[4] Button "New File" center=(1200,231)
[5] TextField "Search files" center=(1153,396)
[6] Button "Format" center=(702,789)
[7] Button "Save" center=(999,321)

Instruction: commit my changes

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1848,59)
[1] Button "Open" center=(1309,917)
[2] Button "Format" center=(785,834)
[3] Button "Save" center=(1310,284)
[4] Button "Run" center=(696,778)
[5] Button "New File" center=(169,714)

Instruction: save

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(237,575)
[1] Button "Wishlist" center=(1389,108)
[2] Button "Filters" center=(129,895)
[3] Link "Account" center=(305,222)
[4] Button "Checkout" center=(1828,72)

Instruction: press account

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1862,892)
[1] Button "Downloads" center=(203,519)
[2] Button "Bookmark" center=(1680,694)
[3] SearchField "Search or enter address" center=(164,409)
[4] Button "Go forward" center=(1182,801)
[5] Button "Reload" center=(1863,465)
[6] Button "Extensions" center=(1723,417)
[7] Button "New tab" center=(1666,765)
[8] Link "Pricing" center=(159,883)
[9] Link "Docs" center=(1303,312)
[10] Button "Run inference" center=(1123,81)
[11] Button "Go back" center=(1543,112)

Instruction: press downloads

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1385,294)
[1] Button "Checkout" center=(1207,680)
[2] Link "Account" center=(406,611)
[3] Button "Wishlist" center=(1574,611)
[4] Button "Buy now" center=(890,804)
[5] Button "Filters" center=(256,824)
[6] Button "Add to cart" center=(1347,630)

Instruction: tap wishlist

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1770,420)
[1] SearchField "Search or enter address" center=(884,822)
[2] Button "Run inference" center=(1336,480)
[3] Button "Bookmark" center=(1731,783)
[4] Button "New tab" center=(702,663)
[5] Button "Go forward" center=(532,616)
[6] Link "Pricing" center=(1157,259)
[7] Button "Downloads" center=(808,824)
[8] Link "Docs" center=(1166,988)
[9] Link "Sign in" center=(1564,300)

Instruction: tap downloads

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(494,259)
[1] Button "Format" center=(1792,362)
[2] Button "Run" center=(84,830)
[3] Button "Open" center=(1723,1005)
[4] Button "Terminal" center=(1452,882)
[5] Button "Find" center=(605,981)
[6] Button "New File" center=(1173,563)

Instruction: press terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(228,624)
[1] Button "Bookmark" center=(858,622)
[2] Button "Extensions" center=(485,791)
[3] Button "Go forward" center=(198,780)
[4] Button "New tab" center=(704,851)
[5] Link "Docs" center=(1028,217)
[6] SearchField "Search or enter address" center=(1039,885)
[7] Link "Sign in" center=(1452,759)
[8] Button "Reload" center=(1220,879)

Instruction: click new tab

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1841,876)
[1] Button "Pause" center=(929,466)
[2] Button "Shuffle" center=(1817,163)
[3] Button "Play" center=(1821,828)
[4] Button "Previous" center=(504,805)
[5] Slider "Volume" center=(1376,909)

Instruction: next song

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1581,649)
[1] Button "Bookmark" center=(1218,556)
[2] Button "Run inference" center=(224,267)
[3] SearchField "Search or enter address" center=(924,336)
[4] Button "Go forward" center=(1650,443)
[5] Button "Extensions" center=(343,175)
[6] Link "Pricing" center=(728,124)
[7] Button "Reload" center=(1014,625)
[8] Button "Downloads" center=(1566,90)
[9] Button "Go back" center=(1162,448)
[10] Link "Sign in" center=(577,127)
[11] Button "New tab" center=(856,553)

Instruction: press new tab

[Assistant] {"action":"click","id":11}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(186,312)
[1] CheckBox "Wi-Fi" center=(1630,480)
[2] CheckBox "Do Not Disturb" center=(702,741)
[3] CheckBox "Bluetooth" center=(1219,131)
[4] Button "Displays" center=(270,727)
[5] Button "Privacy & Security" center=(887,473)

Instruction: enable wifi

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(338,491)
[1] Slider "Volume" center=(131,986)
[2] Button "Previous" center=(1715,279)
[3] Button "Shuffle" center=(448,741)
[4] Button "Pause" center=(805,199)
[5] Button "Next" center=(1591,785)
[6] Button "Play" center=(1786,127)

Instruction: go fullscreen

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(86,379)
[1] Link "Forgot password" center=(468,797)
[2] TextField "Password" center=(1141,256)
[3] CheckBox "Remember me" center=(457,599)
[4] Button "Sign in" center=(1111,808)
[5] TextField "Email" center=(569,601)

Instruction: type into email

[Assistant] {"action":"type","id":5,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(621,245)
[1] Button "Network" center=(1656,753)
[2] CheckBox "Bluetooth" center=(1305,317)
[3] PopUpButton "Appearance" center=(148,516)
[4] Slider "Brightness" center=(1380,1007)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(831,276)
[1] Button "Pause" center=(819,560)
[2] Button "Previous" center=(1493,141)
[3] Button "Play" center=(686,536)
[4] Button "Fullscreen" center=(1815,651)
[5] Button "Shuffle" center=(1128,130)
[6] Button "Next" center=(421,243)

Instruction: play

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1320,349)
[1] Button "Reload" center=(523,553)
[2] Button "Run inference" center=(604,871)
[3] Button "Go back" center=(166,680)
[4] Button "Go forward" center=(567,939)
[5] SearchField "Search or enter address" center=(832,951)
[6] Button "Extensions" center=(1708,70)
[7] Link "Pricing" center=(1176,1011)
[8] Button "Downloads" center=(613,270)
[9] Button "New tab" center=(350,389)

Instruction: click run inference

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(352,476)
[1] Button "Play" center=(679,971)
[2] Button "Fullscreen" center=(1279,377)
[3] Button "Pause" center=(959,595)
[4] Button "Shuffle" center=(309,251)
[5] Slider "Volume" center=(1376,181)
[6] Button "Previous" center=(1229,594)

Instruction: select play

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(178,610)
[1] Button "Next" center=(1131,1011)
[2] Button "Play" center=(1513,161)
[3] Button "Previous" center=(417,365)
[4] Button "Pause" center=(1759,940)
[5] Slider "Volume" center=(338,784)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(418,66)
[1] Button "Terminal" center=(1601,666)
[2] Button "Save" center=(1303,313)
[3] Button "Open" center=(1280,620)
[4] Button "New File" center=(1294,276)
[5] Button "Format" center=(1261,277)

Instruction: type into search files

[Assistant] {"action":"type","id":0,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(416,912)
[1] Link "Forgot password" center=(833,543)
[2] Button "Sign in" center=(1010,180)
[3] TextField "Password" center=(375,237)
[4] Button "Create account" center=(503,73)
[5] TextField "Email" center=(876,822)

Instruction: press sign in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1043,744)
[1] Slider "Volume" center=(705,927)
[2] Button "Previous" center=(977,948)
[3] Button "Play" center=(545,477)
[4] Button "Fullscreen" center=(869,834)
[5] Button "Pause" center=(125,662)
[6] Button "Shuffle" center=(1545,884)

Instruction: tap previous

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(975,886)
[1] Button "Wishlist" center=(1837,83)
[2] Button "Checkout" center=(1837,291)
[3] Button "Add to cart" center=(1102,590)
[4] Button "Filters" center=(850,792)
[5] SearchField "Search products" center=(120,611)

Instruction: type into search products

[Assistant] {"action":"type","id":5,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1295,981)
[1] CheckBox "Bluetooth" center=(151,746)
[2] CheckBox "Wi-Fi" center=(1098,965)
[3] Button "Privacy & Security" center=(331,514)
[4] Button "Sound" center=(1739,701)
[5] CheckBox "Do Not Disturb" center=(965,132)
[6] Button "Network" center=(1519,552)

Instruction: press network

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(224,296)
[1] Button "Terminal" center=(703,748)
[2] Button "New File" center=(1368,968)
[3] TextField "Search files" center=(1497,855)
[4] Button "Open" center=(375,775)
[5] Button "Save" center=(1541,166)
[6] Button "Run" center=(226,156)
[7] Button "Format" center=(1486,500)

Instruction: tap terminal

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1613,214)
[1] Button "Next" center=(726,619)
[2] Button "Previous" center=(1824,715)
[3] Button "Pause" center=(1806,193)
[4] Button "Shuffle" center=(372,929)

Instruction: start playback

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(47,677)
[1] Slider "Volume" center=(1607,299)
[2] Button "Play" center=(65,509)
[3] Button "Pause" center=(1059,915)
[4] Button "Next" center=(1061,596)

Instruction: press next

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1327,610)
[1] CheckBox "Bluetooth" center=(1719,885)
[2] Button "Sound" center=(465,90)
[3] Button "Displays" center=(418,191)
[4] CheckBox "Wi-Fi" center=(152,708)
[5] Button "Privacy & Security" center=(754,77)
[6] Button "Network" center=(337,914)
[7] PopUpButton "Appearance" center=(106,723)
[8] Slider "Brightness" center=(1223,898)

Instruction: click privacy & security

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(701,417)
[1] Button "Extensions" center=(1878,428)
[2] Button "Go back" center=(1719,109)
[3] Link "Pricing" center=(1593,477)
[4] Link "Sign in" center=(1767,603)
[5] Button "Downloads" center=(172,339)
[6] Button "Bookmark" center=(1004,177)
[7] Button "Run inference" center=(289,751)
[8] Button "New tab" center=(1800,321)

Instruction: press bookmark

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1522,745)
[1] Button "Previous" center=(1278,996)
[2] Button "Next" center=(1654,763)
[3] Button "Shuffle" center=(628,477)
[4] Button "Play" center=(1286,908)
[5] Slider "Volume" center=(688,305)

Instruction: play

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(516,765)
[1] Button "Add to cart" center=(634,238)
[2] Button "Buy now" center=(43,161)
[3] Link "Account" center=(89,516)
[4] Button "Filters" center=(1320,855)
[5] Button "Wishlist" center=(281,795)
[6] SearchField "Search products" center=(1776,252)

Instruction: tap account

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(1077,603)
[1] Link "Docs" center=(703,825)
[2] Link "Pricing" center=(52,156)
[3] Button "New tab" center=(1444,381)
[4] Button "Reload" center=(739,445)
[5] Button "Go forward" center=(1523,622)
[6] SearchField "Search or enter address" center=(1301,1023)
[7] Button "Go back" center=(99,892)
[8] Button "Bookmark" center=(399,238)

Instruction: tap go forward

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1750,573)
[1] Button "Find" center=(653,618)
[2] Button "Format" center=(1524,734)
[3] Button "Terminal" center=(686,608)
[4] Button "Commit" center=(1137,59)

Instruction: select format

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(1484,918)
[1] Button "Extensions" center=(684,766)
[2] Button "Run inference" center=(104,318)
[3] Button "Bookmark" center=(887,128)
[4] Button "Go forward" center=(1057,938)
[5] SearchField "Search or enter address" center=(526,420)
[6] Link "Sign in" center=(965,889)
[7] Link "Docs" center=(659,261)
[8] Button "Downloads" center=(1115,349)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":5,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(1540,934)
[1] Button "Downloads" center=(1112,357)
[2] Button "Run inference" center=(1628,375)
[3] SearchField "Search or enter address" center=(333,827)
[4] Button "New tab" center=(1825,140)
[5] Button "Extensions" center=(1723,1000)
[6] Link "Docs" center=(1097,732)
[7] Button "Go back" center=(311,250)
[8] Link "Sign in" center=(1523,77)
[9] Link "Pricing" center=(980,436)
[10] Button "Go forward" center=(1501,925)
[11] Button "Bookmark" center=(1109,158)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1106,824)
[1] Button "Shuffle" center=(1389,803)
[2] Button "Pause" center=(679,63)
[3] Button "Next" center=(1222,816)
[4] Button "Play" center=(679,694)

Instruction: press pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(600,157)
[1] Button "Privacy & Security" center=(71,1021)
[2] CheckBox "Bluetooth" center=(1315,613)
[3] CheckBox "Wi-Fi" center=(1092,144)
[4] Button "Network" center=(463,595)
[5] CheckBox "Do Not Disturb" center=(757,602)

Instruction: select privacy & security

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(948,91)
[1] Link "Account" center=(306,69)
[2] Button "Wishlist" center=(1004,964)
[3] Button "Checkout" center=(1423,91)
[4] Button "Add to cart" center=(256,432)
[5] Button "Filters" center=(150,128)
[6] Button "Buy now" center=(913,441)

Instruction: tap add to cart

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1202,174)
[1] Button "Add to cart" center=(230,872)
[2] SearchField "Search products" center=(1683,981)
[3] Button "Buy now" center=(1470,519)
[4] Link "Account" center=(239,506)
[5] Button "Filters" center=(675,740)

Instruction: tap buy now

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1655,921)
[1] CheckBox "Wi-Fi" center=(1847,372)
[2] CheckBox "Bluetooth" center=(968,539)
[3] Button "Sound" center=(101,106)
[4] Slider "Brightness" center=(1013,155)
[5] Button "Displays" center=(759,530)
[6] Button "Privacy & Security" center=(984,906)

Instruction: open privacy settings

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1131,512)
[1] SearchField "Search products" center=(860,644)
[2] Link "Account" center=(886,284)
[3] Button "Checkout" center=(1311,289)
[4] Button "Filters" center=(1162,202)
[5] Button "Wishlist" center=(1027,433)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1397,725)
[1] Button "Play" center=(199,797)
[2] Button "Fullscreen" center=(1497,627)
[3] Button "Next" center=(1753,538)
[4] Button "Shuffle" center=(274,646)

Instruction: make it full screen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(225,262)
[1] CheckBox "Do Not Disturb" center=(1713,182)
[2] CheckBox "Bluetooth" center=(1857,634)
[3] Button "Network" center=(1586,691)
[4] Slider "Brightness" center=(1698,40)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1836,364)
[1] Button "Filters" center=(653,623)
[2] Button "Add to cart" center=(1666,975)
[3] Link "Account" center=(1431,351)
[4] SearchField "Search products" center=(930,775)
[5] Button "Wishlist" center=(745,131)
[6] Button "Checkout" center=(1233,48)

Instruction: search for something

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1803,328)
[1] TextField "Password" center=(1812,71)
[2] Button "Create account" center=(521,482)
[3] CheckBox "Remember me" center=(1478,644)
[4] Button "Sign in" center=(847,680)

Instruction: tap remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1760,467)
[1] Button "Find" center=(355,335)
[2] Button "Save" center=(1827,726)
[3] TextField "Search files" center=(297,177)
[4] Button "Terminal" center=(63,113)
[5] Button "Open" center=(1367,56)
[6] Button "Format" center=(334,778)

Instruction: tap open

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1083,474)
[1] CheckBox "Wi-Fi" center=(1170,879)
[2] Slider "Brightness" center=(961,433)
[3] Button "Displays" center=(767,271)
[4] Button "Privacy & Security" center=(420,474)
[5] Button "Network" center=(1278,47)
[6] Button "Sound" center=(1223,550)
[7] CheckBox "Do Not Disturb" center=(1774,788)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1098,120)
[1] Link "Sign in" center=(1111,951)
[2] Button "Reload" center=(1289,130)
[3] SearchField "Search or enter address" center=(542,523)
[4] Button "Extensions" center=(1331,799)
[5] Button "Bookmark" center=(1399,1023)
[6] Button "New tab" center=(1409,119)
[7] Button "Go forward" center=(1440,688)
[8] Button "Downloads" center=(1144,857)
[9] Link "Pricing" center=(1227,359)
[10] Button "Run inference" center=(1295,94)
[11] Button "Go back" center=(743,587)

Instruction: tap go forward

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(383,76)
[1] Link "Account" center=(1463,985)
[2] Button "Buy now" center=(1263,807)
[3] Button "Filters" center=(662,784)
[4] Button "Checkout" center=(1647,617)
[5] SearchField "Search products" center=(573,102)

Instruction: check out

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(299,557)
[1] Button "Fullscreen" center=(197,911)
[2] Button "Pause" center=(1645,324)
[3] Button "Play" center=(556,175)
[4] Slider "Volume" center=(1301,327)

Instruction: click pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1694,219)
[1] Button "Previous" center=(575,841)
[2] Button "Next" center=(1475,281)
[3] Button "Shuffle" center=(1033,225)
[4] Button "Pause" center=(1677,580)
[5] Slider "Volume" center=(1805,73)

Instruction: press fullscreen

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1355,310)
[1] Button "Find" center=(1587,879)
[2] Button "Save" center=(277,93)
[3] Button "Terminal" center=(383,515)
[4] Button "Run" center=(856,527)

Instruction: execute

[Assistant] {"action":"click","id":4}