[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1066,1006)
[1] Button "Open" center=(1212,70)
[2] Button "New File" center=(1390,379)
[3] Button "Save" center=(836,487)
[4] Button "Commit" center=(746,980)
[5] Button "Terminal" center=(219,606)

Instruction: select commit

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1709,310)
[1] Button "Next" center=(766,84)
[2] Button "Shuffle" center=(1281,879)
[3] Button "Pause" center=(865,560)
[4] Button "Play" center=(693,673)

Instruction: tap volume

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1350,209)
[1] CheckBox "Do Not Disturb" center=(583,992)
[2] PopUpButton "Appearance" center=(495,345)
[3] Button "Network" center=(1626,935)
[4] Slider "Brightness" center=(515,406)
[5] Button "Privacy & Security" center=(554,581)
[6] CheckBox "Bluetooth" center=(870,649)
[7] CheckBox "Wi-Fi" center=(715,221)
[8] Button "Sound" center=(269,231)

Instruction: open privacy settings

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(899,676)
[1] Button "Buy now" center=(635,430)
[2] Button "Filters" center=(914,40)
[3] SearchField "Search products" center=(653,641)
[4] Link "Account" center=(1683,100)
[5] Button "Add to cart" center=(474,528)

Instruction: find a product

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(277,871)
[1] Button "Next" center=(732,85)
[2] Button "Fullscreen" center=(1585,283)
[3] Button "Previous" center=(1528,628)
[4] Button "Shuffle" center=(1711,736)

Instruction: tap fullscreen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(682,573)
[1] Button "Pause" center=(1273,266)
[2] Button "Previous" center=(480,261)
[3] Button "Fullscreen" center=(803,174)
[4] Button "Play" center=(1770,596)

Instruction: go fullscreen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(435,848)
[1] Button "Buy now" center=(1102,51)
[2] Button "Wishlist" center=(256,936)
[3] SearchField "Search products" center=(1851,71)
[4] Button "Add to cart" center=(180,228)
[5] Button "Filters" center=(1792,341)

Instruction: search for something

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1284,621)
[1] CheckBox "Remember me" center=(1618,632)
[2] Button "Sign in" center=(277,234)
[3] Button "Create account" center=(861,680)
[4] TextField "Email" center=(1646,781)
[5] Link "Forgot password" center=(805,77)

Instruction: log in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(937,123)
[1] Button "Previous" center=(1067,483)
[2] Button "Next" center=(1668,716)
[3] Button "Pause" center=(1048,118)
[4] Slider "Volume" center=(1029,47)
[5] Button "Shuffle" center=(1138,51)
[6] Button "Fullscreen" center=(1354,358)

Instruction: press previous

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(905,125)
[1] Button "Previous" center=(354,1025)
[2] Button "Fullscreen" center=(1250,265)
[3] Button "Play" center=(465,872)
[4] Button "Next" center=(327,116)
[5] Slider "Volume" center=(1601,239)
[6] Button "Pause" center=(612,630)

Instruction: click previous

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(189,998)
[1] Button "Buy now" center=(1004,547)
[2] Button "Wishlist" center=(783,574)
[3] SearchField "Search products" center=(1178,397)
[4] Button "Add to cart" center=(829,198)
[5] Button "Checkout" center=(1507,499)
[6] Button "Filters" center=(562,593)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(919,365)
[1] Button "Run inference" center=(1044,729)
[2] Link "Sign in" center=(231,212)
[3] Button "Bookmark" center=(265,160)
[4] Link "Pricing" center=(1175,481)
[5] Button "Extensions" center=(138,891)

Instruction: click pricing

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(161,479)
[1] Button "Downloads" center=(1653,194)
[2] Link "Pricing" center=(492,157)
[3] Button "New tab" center=(1659,331)
[4] Button "Extensions" center=(1711,718)
[5] Button "Go forward" center=(738,654)
[6] Button "Go back" center=(921,608)

Instruction: click extensions

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(328,812)
[1] Button "Bookmark" center=(140,902)
[2] Button "New tab" center=(484,851)
[3] Button "Downloads" center=(904,46)
[4] Link "Pricing" center=(1102,165)
[5] Link "Sign in" center=(869,844)

Instruction: click extensions

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(249,302)
[1] Link "Forgot password" center=(118,863)
[2] CheckBox "Remember me" center=(1626,605)
[3] TextField "Email" center=(1761,281)
[4] Button "Create account" center=(599,475)
[5] Button "Sign in" center=(1194,373)

Instruction: click create account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(737,447)
[1] Button "Open" center=(925,648)
[2] Button "Terminal" center=(1181,265)
[3] Button "Run" center=(1441,673)
[4] TextField "Search files" center=(1345,599)
[5] Button "Find" center=(1746,396)
[6] Button "Commit" center=(1210,709)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1800,706)
[1] Button "Privacy & Security" center=(947,969)
[2] Button "Sound" center=(922,223)
[3] CheckBox "Wi-Fi" center=(687,908)
[4] PopUpButton "Appearance" center=(430,730)
[5] Slider "Brightness" center=(1598,682)
[6] CheckBox "Do Not Disturb" center=(1132,334)
[7] CheckBox "Bluetooth" center=(970,455)

Instruction: select privacy & security

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1001,340)
[1] Button "Displays" center=(1416,769)
[2] Button "Sound" center=(808,951)
[3] Slider "Brightness" center=(1102,367)
[4] Button "Privacy & Security" center=(1232,968)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(56,826)
[1] CheckBox "Bluetooth" center=(428,355)
[2] Button "Sound" center=(1262,574)
[3] Button "Displays" center=(301,316)
[4] Button "Network" center=(1694,1003)
[5] PopUpButton "Appearance" center=(859,937)
[6] CheckBox "Wi-Fi" center=(1163,648)
[7] Button "Privacy & Security" center=(1332,844)

Instruction: open privacy settings

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1876,412)
[1] Button "Find" center=(1723,997)
[2] Button "Save" center=(1058,1014)
[3] Button "New File" center=(310,251)
[4] TextField "Search files" center=(217,774)

Instruction: save the file

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1755,674)
[1] CheckBox "Bluetooth" center=(672,654)
[2] Button "Displays" center=(839,939)
[3] CheckBox "Wi-Fi" center=(168,503)
[4] Button "Network" center=(408,150)
[5] PopUpButton "Appearance" center=(781,1018)
[6] Slider "Brightness" center=(1779,97)
[7] CheckBox "Do Not Disturb" center=(735,831)
[8] Button "Sound" center=(347,48)

Instruction: toggle wifi

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1399,192)
[1] Slider "Brightness" center=(536,193)
[2] Button "Sound" center=(299,783)
[3] Button "Privacy & Security" center=(1510,768)
[4] CheckBox "Wi-Fi" center=(293,46)
[5] Button "Network" center=(1278,127)
[6] CheckBox "Do Not Disturb" center=(1848,190)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1805,71)
[1] Slider "Volume" center=(1693,965)
[2] Button "Previous" center=(1142,988)
[3] Button "Next" center=(1657,537)
[4] Button "Play" center=(1088,267)
[5] Button "Shuffle" center=(583,751)

Instruction: tap volume

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1127,190)
[1] Button "Commit" center=(507,134)
[2] Button "Find" center=(484,543)
[3] Button "Open" center=(1315,1002)
[4] Button "Save" center=(1198,926)
[5] Button "Terminal" center=(1435,962)
[6] Button "Format" center=(1567,569)
[7] Button "New File" center=(205,554)

Instruction: tap run

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1072,524)
[1] CheckBox "Bluetooth" center=(964,1035)
[2] CheckBox "Wi-Fi" center=(618,834)
[3] PopUpButton "Appearance" center=(1823,395)
[4] Button "Privacy & Security" center=(1815,170)
[5] Slider "Brightness" center=(894,258)
[6] Button "Network" center=(1611,759)

Instruction: press privacy & security

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(709,159)
[1] Button "Network" center=(178,960)
[2] CheckBox "Wi-Fi" center=(994,1033)
[3] CheckBox "Bluetooth" center=(1118,735)
[4] Button "Sound" center=(1180,815)
[5] Button "Privacy & Security" center=(422,606)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(54,308)
[1] Button "Network" center=(943,446)
[2] Button "Privacy & Security" center=(1008,485)
[3] Slider "Brightness" center=(817,977)
[4] PopUpButton "Appearance" center=(815,765)
[5] CheckBox "Bluetooth" center=(130,585)

Instruction: tap appearance

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1321,523)
[1] CheckBox "Remember me" center=(681,252)
[2] TextField "Email" center=(187,740)
[3] Button "Create account" center=(1764,881)
[4] Link "Forgot password" center=(480,773)

Instruction: i forgot my password

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1658,1017)
[1] Button "New tab" center=(579,265)
[2] Link "Pricing" center=(47,690)
[3] Button "Bookmark" center=(1717,164)
[4] Button "Go back" center=(71,864)
[5] Button "Run inference" center=(833,123)
[6] Button "Extensions" center=(608,984)
[7] Button "Go forward" center=(626,404)
[8] SearchField "Search or enter address" center=(331,645)

Instruction: select extensions

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(181,649)
[1] TextField "Email" center=(1669,667)
[2] TextField "Password" center=(1781,530)
[3] Button "Sign in" center=(238,870)
[4] Button "Create account" center=(290,739)

Instruction: type into email

[Assistant] {"action":"type","id":1,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(997,69)
[1] Button "Sound" center=(228,65)
[2] CheckBox "Do Not Disturb" center=(1576,86)
[3] CheckBox "Bluetooth" center=(1568,250)
[4] PopUpButton "Appearance" center=(1661,688)
[5] Slider "Brightness" center=(659,271)
[6] Button "Privacy & Security" center=(643,348)
[7] Button "Displays" center=(1268,654)
[8] Button "Network" center=(1056,413)

Instruction: select network

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1495,992)
[1] Button "Checkout" center=(1696,943)
[2] Button "Buy now" center=(307,158)
[3] SearchField "Search products" center=(790,626)
[4] Button "Add to cart" center=(1249,197)

Instruction: press buy now

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(479,160)
[1] Button "Save" center=(368,211)
[2] TextField "Search files" center=(421,274)
[3] Button "New File" center=(1823,656)
[4] Button "Terminal" center=(698,310)
[5] Button "Open" center=(1180,81)

Instruction: click new file

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1177,514)
[1] CheckBox "Remember me" center=(674,510)
[2] Link "Forgot password" center=(1005,780)
[3] TextField "Email" center=(1798,548)
[4] TextField "Password" center=(563,46)
[5] Button "Create account" center=(980,58)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1790,602)
[1] TextField "Search files" center=(1785,198)
[2] Button "New File" center=(286,926)
[3] Button "Find" center=(410,846)
[4] Button "Run" center=(1447,700)
[5] Button "Terminal" center=(1143,170)

Instruction: type into search files

[Assistant] {"action":"type","id":1,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(313,476)
[1] Link "Account" center=(575,765)
[2] Button "Checkout" center=(1658,260)
[3] Button "Filters" center=(1266,250)
[4] Button "Buy now" center=(444,910)
[5] SearchField "Search products" center=(62,968)

Instruction: go to checkout

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1857,319)
[1] Button "Checkout" center=(1152,400)
[2] Button "Filters" center=(514,136)
[3] Button "Wishlist" center=(412,247)
[4] Button "Add to cart" center=(556,709)

Instruction: select checkout

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(355,1030)
[1] Button "Wishlist" center=(575,225)
[2] Button "Buy now" center=(282,69)
[3] Button "Checkout" center=(1827,438)
[4] Link "Account" center=(1574,280)

Instruction: tap filters

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1123,137)
[1] Button "Shuffle" center=(210,745)
[2] Button "Previous" center=(881,784)
[3] Button "Pause" center=(732,633)
[4] Slider "Volume" center=(756,987)
[5] Button "Next" center=(780,795)
[6] Button "Play" center=(1719,605)

Instruction: next song

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1451,443)
[1] Button "Sound" center=(729,630)
[2] Button "Privacy & Security" center=(1056,221)
[3] CheckBox "Wi-Fi" center=(1041,449)
[4] PopUpButton "Appearance" center=(984,477)
[5] Button "Displays" center=(1104,544)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1259,114)
[1] SearchField "Search products" center=(1237,394)
[2] Link "Account" center=(490,833)
[3] Button "Checkout" center=(1707,608)
[4] Button "Buy now" center=(88,1022)
[5] Button "Add to cart" center=(1161,708)

Instruction: press account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(275,412)
[1] Button "Format" center=(1813,68)
[2] TextField "Search files" center=(107,136)
[3] Button "Save" center=(193,123)
[4] Button "Find" center=(498,664)
[5] Button "Commit" center=(257,382)
[6] Button "Open" center=(670,578)
[7] Button "New File" center=(40,355)

Instruction: save

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(846,139)
[1] Button "Commit" center=(1678,996)
[2] Button "Run" center=(827,835)
[3] Button "Save" center=(1717,969)
[4] Button "Format" center=(1726,296)
[5] Button "Find" center=(1730,511)
[6] TextField "Search files" center=(1481,660)
[7] Button "Open" center=(77,359)
[8] Button "New File" center=(1758,683)

Instruction: make a commit

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(965,225)
[1] Slider "Brightness" center=(114,638)
[2] CheckBox "Bluetooth" center=(1423,443)
[3] Button "Sound" center=(282,86)
[4] Button "Displays" center=(1680,225)
[5] Button "Privacy & Security" center=(1599,238)
[6] Button "Network" center=(978,225)
[7] CheckBox "Wi-Fi" center=(925,352)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1401,904)
[1] Link "Account" center=(1581,701)
[2] Button "Filters" center=(785,410)
[3] Button "Checkout" center=(1436,494)
[4] Button "Buy now" center=(1305,240)

Instruction: check out

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1713,1029)
[1] Link "Docs" center=(1240,474)
[2] Button "Reload" center=(1080,388)
[3] Button "Run inference" center=(872,560)
[4] Link "Sign in" center=(1170,744)

Instruction: log in

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1467,977)
[1] Button "Go forward" center=(438,915)
[2] Link "Docs" center=(881,320)
[3] SearchField "Search or enter address" center=(596,944)
[4] Button "Extensions" center=(1056,92)
[5] Button "New tab" center=(384,431)
[6] Button "Reload" center=(451,193)
[7] Button "Bookmark" center=(1043,445)

Instruction: select reload

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1307,305)
[1] Button "Buy now" center=(1717,637)
[2] SearchField "Search products" center=(1336,311)
[3] Button "Filters" center=(946,802)
[4] Link "Account" center=(200,397)
[5] Button "Checkout" center=(764,902)

Instruction: go to checkout

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(534,493)
[1] Link "Forgot password" center=(1054,213)
[2] Button "Create account" center=(1269,702)
[3] TextField "Password" center=(51,612)
[4] TextField "Email" center=(1524,319)
[5] Button "Sign in" center=(1431,891)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1688,874)
[1] Button "Terminal" center=(755,326)
[2] Button "Save" center=(1174,927)
[3] Button "Commit" center=(106,831)
[4] Button "New File" center=(1027,577)
[5] Button "Find" center=(54,423)
[6] TextField "Search files" center=(510,584)
[7] Button "Run" center=(482,243)

Instruction: press find

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1701,839)
[1] CheckBox "Wi-Fi" center=(99,526)
[2] Button "Sound" center=(493,509)
[3] CheckBox "Do Not Disturb" center=(468,96)
[4] Button "Privacy & Security" center=(1684,936)
[5] Button "Displays" center=(602,129)
[6] Slider "Brightness" center=(760,41)

Instruction: press do not disturb

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(511,930)
[1] Button "Privacy & Security" center=(1131,502)
[2] Button "Network" center=(1333,635)
[3] CheckBox "Bluetooth" center=(924,250)
[4] Button "Sound" center=(453,952)
[5] Slider "Brightness" center=(636,986)
[6] CheckBox "Do Not Disturb" center=(848,518)
[7] CheckBox "Wi-Fi" center=(615,703)
[8] Button "Displays" center=(914,312)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1308,525)
[1] Button "Fullscreen" center=(581,154)
[2] Button "Previous" center=(727,648)
[3] Slider "Volume" center=(340,841)
[4] Button "Shuffle" center=(149,765)
[5] Button "Pause" center=(1592,629)
[6] Button "Play" center=(620,877)

Instruction: click next

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(842,346)
[1] CheckBox "Bluetooth" center=(1100,308)
[2] CheckBox "Do Not Disturb" center=(1252,942)
[3] PopUpButton "Appearance" center=(985,527)
[4] Button "Displays" center=(1198,689)

Instruction: press displays

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(592,762)
[1] Button "Displays" center=(1008,580)
[2] Button "Network" center=(1248,255)
[3] CheckBox "Bluetooth" center=(1791,389)
[4] Button "Sound" center=(86,1008)
[5] CheckBox "Do Not Disturb" center=(1695,973)
[6] PopUpButton "Appearance" center=(744,724)

Instruction: press do not disturb

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(926,814)
[1] Button "Open" center=(1003,101)
[2] Button "Terminal" center=(1024,127)
[3] Button "Format" center=(1343,906)
[4] TextField "Search files" center=(1472,594)
[5] Button "Run" center=(1433,1004)
[6] Button "Commit" center=(285,1004)

Instruction: press open

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(985,299)
[1] Button "Next" center=(785,652)
[2] Button "Fullscreen" center=(1360,642)
[3] Button "Shuffle" center=(1106,1028)
[4] Button "Pause" center=(649,1022)
[5] Button "Play" center=(1416,144)

Instruction: press next

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1326,861)
[1] Button "Run" center=(1583,454)
[2] Button "Open" center=(1770,696)
[3] TextField "Search files" center=(784,161)
[4] Button "Find" center=(1855,70)

Instruction: click open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(441,864)
[1] Button "Run" center=(61,273)
[2] Button "Format" center=(560,569)
[3] Button "New File" center=(1763,827)
[4] Button "Commit" center=(759,513)
[5] Button "Open" center=(476,483)
[6] Button "Find" center=(1813,376)
[7] Button "Terminal" center=(1391,331)
[8] Button "Save" center=(1453,734)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(427,160)
[1] Button "Sign in" center=(1815,502)
[2] TextField "Email" center=(1174,929)
[3] Button "Create account" center=(1717,558)
[4] TextField "Password" center=(541,850)

Instruction: select create account

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(587,950)
[1] PopUpButton "Appearance" center=(125,604)
[2] Slider "Brightness" center=(1190,714)
[3] CheckBox "Do Not Disturb" center=(1212,962)
[4] Button "Displays" center=(686,800)
[5] Button "Sound" center=(1243,260)
[6] Button "Network" center=(89,244)

Instruction: click sound

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1562,410)
[1] Button "Terminal" center=(365,251)
[2] Button "Run" center=(1746,973)
[3] Button "Open" center=(1369,78)
[4] Button "New File" center=(306,661)

Instruction: tap new file

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(582,1032)
[1] Button "New File" center=(46,290)
[2] Button "Terminal" center=(100,490)
[3] Button "Format" center=(1263,698)
[4] Button "Open" center=(1254,160)
[5] Button "Find" center=(1685,401)
[6] Button "Save" center=(603,864)

Instruction: tap commit

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(708,44)
[1] Button "Network" center=(85,339)
[2] Button "Privacy & Security" center=(302,245)
[3] PopUpButton "Appearance" center=(200,720)
[4] Button "Displays" center=(1128,183)
[5] CheckBox "Bluetooth" center=(1473,79)
[6] Slider "Brightness" center=(1592,749)
[7] CheckBox "Wi-Fi" center=(1789,433)
[8] CheckBox "Do Not Disturb" center=(1099,169)

Instruction: select sound

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1126,334)
[1] Button "Wishlist" center=(1491,499)
[2] SearchField "Search products" center=(485,977)
[3] Button "Buy now" center=(1097,407)
[4] Link "Account" center=(948,814)
[5] Button "Add to cart" center=(1728,537)

Instruction: press wishlist

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1797,126)
[1] Button "Create account" center=(1137,200)
[2] TextField "Email" center=(1035,188)
[3] Link "Forgot password" center=(1800,734)
[4] Button "Sign in" center=(424,823)
[5] CheckBox "Remember me" center=(1095,931)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1625,580)
[1] Button "Network" center=(1644,578)
[2] Button "Privacy & Security" center=(1394,781)
[3] PopUpButton "Appearance" center=(202,384)
[4] Slider "Brightness" center=(733,645)
[5] CheckBox "Do Not Disturb" center=(124,289)
[6] Button "Displays" center=(149,999)
[7] CheckBox "Bluetooth" center=(1091,166)

Instruction: select do not disturb

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(70,884)
[1] TextField "Email" center=(1491,820)
[2] TextField "Password" center=(642,564)
[3] Button "Create account" center=(1175,328)
[4] Link "Forgot password" center=(1238,797)
[5] CheckBox "Remember me" center=(275,946)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(873,768)
[1] Button "Previous" center=(634,82)
[2] Button "Next" center=(526,979)
[3] Button "Pause" center=(720,67)
[4] Button "Shuffle" center=(948,995)
[5] Slider "Volume" center=(1782,462)

Instruction: next song

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1079,814)
[1] Button "Add to cart" center=(1497,177)
[2] Button "Wishlist" center=(1399,196)
[3] SearchField "Search products" center=(954,489)
[4] Button "Checkout" center=(1629,377)

Instruction: find a product

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(298,262)
[1] CheckBox "Remember me" center=(1093,498)
[2] Button "Create account" center=(343,911)
[3] TextField "Password" center=(630,794)
[4] Link "Forgot password" center=(1241,871)

Instruction: type into password

[Assistant] {"action":"type","id":3,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(623,935)
[1] Button "Filters" center=(717,954)
[2] Button "Buy now" center=(785,654)
[3] Link "Account" center=(622,600)
[4] Button "Add to cart" center=(1744,357)
[5] SearchField "Search products" center=(1205,744)
[6] Button "Wishlist" center=(1549,466)

Instruction: click buy now

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(355,377)
[1] Button "Run" center=(785,640)
[2] TextField "Search files" center=(492,391)
[3] Button "Terminal" center=(101,992)
[4] Button "Find" center=(385,835)

Instruction: select terminal

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1045,191)
[1] SearchField "Search products" center=(1431,534)
[2] Button "Filters" center=(853,409)
[3] Button "Checkout" center=(444,864)
[4] Button "Wishlist" center=(168,1004)

Instruction: click filters

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(734,329)
[1] Button "Previous" center=(786,384)
[2] Button "Fullscreen" center=(1178,286)
[3] Button "Play" center=(1658,336)
[4] Button "Next" center=(763,404)

Instruction: select fullscreen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(889,561)
[1] TextField "Search files" center=(150,613)
[2] Button "Commit" center=(1720,841)
[3] Button "Format" center=(1294,404)
[4] Button "Terminal" center=(1593,47)
[5] Button "Find" center=(657,862)
[6] Button "Run" center=(1563,634)
[7] Button "Open" center=(227,70)

Instruction: save the file

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(975,637)
[1] Button "Find" center=(514,268)
[2] Button "Save" center=(840,609)
[3] Button "Commit" center=(825,948)
[4] Button "Run" center=(134,837)
[5] Button "Terminal" center=(1301,738)
[6] Button "New File" center=(536,931)
[7] Button "Format" center=(180,724)
[8] Button "Open" center=(1432,470)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(801,849)
[1] Link "Account" center=(1601,605)
[2] Button "Wishlist" center=(443,651)
[3] Button "Add to cart" center=(200,135)
[4] Button "Checkout" center=(261,418)
[5] Button "Filters" center=(1762,249)

Instruction: tap account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1208,483)
[1] Link "Forgot password" center=(537,551)
[2] Button "Sign in" center=(1243,83)
[3] TextField "Password" center=(418,971)
[4] CheckBox "Remember me" center=(44,721)

Instruction: reset password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(961,572)
[1] Button "Wishlist" center=(243,810)
[2] Button "Checkout" center=(1588,462)
[3] Button "Filters" center=(1622,503)
[4] Button "Add to cart" center=(925,453)
[5] Link "Account" center=(548,805)

Instruction: click wishlist

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1561,824)
[1] Button "Filters" center=(868,485)
[2] Button "Wishlist" center=(638,532)
[3] Button "Add to cart" center=(609,192)
[4] SearchField "Search products" center=(1530,875)

Instruction: click wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1267,880)
[1] TextField "Email" center=(929,456)
[2] Button "Sign in" center=(530,492)
[3] TextField "Password" center=(1256,843)
[4] Link "Forgot password" center=(1373,375)
[5] Button "Create account" center=(1101,135)

Instruction: reset password

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1399,130)
[1] Button "Sign in" center=(666,994)
[2] Button "Create account" center=(566,433)
[3] TextField "Password" center=(1710,664)
[4] CheckBox "Remember me" center=(484,852)
[5] Link "Forgot password" center=(1205,671)

Instruction: submit the login

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1299,325)
[1] TextField "Email" center=(60,1004)
[2] Link "Forgot password" center=(352,681)
[3] Button "Sign in" center=(815,851)
[4] CheckBox "Remember me" center=(1665,637)

Instruction: tap remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1394,590)
[1] TextField "Email" center=(412,899)
[2] TextField "Password" center=(324,630)
[3] CheckBox "Remember me" center=(1640,849)
[4] Button "Create account" center=(338,288)

Instruction: type into email

[Assistant] {"action":"type","id":1,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(503,518)
[1] Button "Sign in" center=(465,275)
[2] TextField "Email" center=(1035,763)
[3] CheckBox "Remember me" center=(1182,832)
[4] Link "Forgot password" center=(592,94)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(583,846)
[1] CheckBox "Wi-Fi" center=(715,375)
[2] CheckBox "Bluetooth" center=(1236,434)
[3] Button "Sound" center=(1445,859)
[4] Button "Displays" center=(1861,373)

Instruction: tap sound

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1426,42)
[1] Button "Pause" center=(150,947)
[2] Slider "Volume" center=(762,385)
[3] Button "Previous" center=(177,910)
[4] Button "Next" center=(1544,174)
[5] Button "Shuffle" center=(850,692)
[6] Button "Fullscreen" center=(1573,530)

Instruction: select volume

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(792,460)
[1] SearchField "Search products" center=(259,301)
[2] Button "Add to cart" center=(929,403)
[3] Link "Account" center=(435,943)
[4] Button "Wishlist" center=(554,858)

Instruction: click filters

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1505,142)
[1] Button "Sign in" center=(1812,417)
[2] CheckBox "Remember me" center=(321,91)
[3] Button "Create account" center=(551,373)
[4] TextField "Email" center=(209,686)
[5] TextField "Password" center=(1216,134)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(522,269)
[1] Button "Create account" center=(688,940)
[2] TextField "Password" center=(1785,600)
[3] Button "Sign in" center=(694,90)
[4] TextField "Email" center=(1152,129)

Instruction: press create account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1789,541)
[1] Button "Play" center=(1415,645)
[2] Button "Pause" center=(1456,773)
[3] Button "Shuffle" center=(137,838)
[4] Button "Previous" center=(611,171)

Instruction: click previous

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(755,510)
[1] Button "New tab" center=(1757,710)
[2] Button "Downloads" center=(1634,893)
[3] Button "Run inference" center=(645,634)
[4] Button "Reload" center=(1743,825)
[5] Button "Extensions" center=(872,410)
[6] SearchField "Search or enter address" center=(1305,652)
[7] Link "Pricing" center=(82,381)
[8] Button "Go forward" center=(1775,175)

Instruction: select downloads

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1071,881)
[1] Link "Account" center=(436,85)
[2] Button "Buy now" center=(679,1004)
[3] Button "Filters" center=(1816,46)
[4] Button "Add to cart" center=(1325,535)
[5] Button "Wishlist" center=(1021,789)

Instruction: select wishlist

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(734,862)
[1] Button "Reload" center=(451,882)
[2] Button "Run inference" center=(981,152)
[3] Link "Pricing" center=(618,497)
[4] Button "Downloads" center=(529,1029)
[5] Link "Sign in" center=(1748,198)
[6] Button "New tab" center=(129,330)
[7] SearchField "Search or enter address" center=(543,124)
[8] Button "Go forward" center=(1288,169)

Instruction: click pricing

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(862,128)
[1] Button "Sound" center=(887,1031)
[2] Button "Displays" center=(768,387)
[3] CheckBox "Wi-Fi" center=(1084,62)
[4] Slider "Brightness" center=(897,821)
[5] CheckBox "Do Not Disturb" center=(338,296)

Instruction: select displays

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(334,505)
[1] Button "Displays" center=(1522,361)
[2] CheckBox "Do Not Disturb" center=(270,47)
[3] Button "Network" center=(55,950)
[4] PopUpButton "Appearance" center=(769,965)

Instruction: tap network

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(969,930)
[1] Button "Bookmark" center=(1449,653)
[2] Button "Reload" center=(454,869)
[3] Link "Sign in" center=(409,1035)
[4] Button "Go forward" center=(533,413)
[5] Button "Extensions" center=(1677,854)
[6] SearchField "Search or enter address" center=(424,628)
[7] Link "Pricing" center=(1685,275)
[8] Button "Downloads" center=(1596,752)
[9] Button "Go back" center=(513,57)
[10] Button "Run inference" center=(1329,925)

Instruction: select downloads

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1714,97)
[1] Button "Fullscreen" center=(1875,492)
[2] Button "Pause" center=(1231,343)
[3] Button "Play" center=(425,188)
[4] Button "Next" center=(1802,150)
[5] Button "Shuffle" center=(1486,580)

Instruction: make it full screen

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(266,627)
[1] Button "Pause" center=(90,186)
[2] Button "Shuffle" center=(1516,75)
[3] Button "Fullscreen" center=(663,110)
[4] Slider "Volume" center=(1793,833)
[5] Button "Previous" center=(1376,967)
[6] Button "Next" center=(172,370)

Instruction: click volume

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1626,413)
[1] Button "Create account" center=(1521,293)
[2] Link "Forgot password" center=(499,466)
[3] TextField "Password" center=(455,822)
[4] CheckBox "Remember me" center=(527,46)
[5] TextField "Email" center=(686,533)

Instruction: i forgot my password

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(963,991)
[1] Button "Format" center=(959,73)
[2] Button "Open" center=(1350,702)
[3] Button "Terminal" center=(340,138)
[4] Button "New File" center=(1650,513)
[5] Button "Commit" center=(61,888)

Instruction: tap open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(819,581)
[1] Button "Run inference" center=(262,504)
[2] Button "Go back" center=(815,659)
[3] Link "Pricing" center=(950,423)
[4] Button "New tab" center=(115,403)
[5] Button "Extensions" center=(1277,136)
[6] Link "Docs" center=(1050,1014)
[7] Button "Bookmark" center=(1554,282)
[8] SearchField "Search or enter address" center=(1451,832)
[9] Button "Downloads" center=(1192,284)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1245,256)
[1] Button "Go back" center=(1861,424)
[2] Button "Reload" center=(628,699)
[3] Button "Run inference" center=(1063,68)
[4] Link "Sign in" center=(1103,656)
[5] Link "Pricing" center=(1501,876)
[6] Button "New tab" center=(567,780)
[7] Link "Docs" center=(654,84)
[8] Button "Go forward" center=(829,398)
[9] Button "Extensions" center=(1482,774)
[10] Button "Downloads" center=(1681,684)
[11] SearchField "Search or enter address" center=(646,990)

Instruction: press pricing

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1058,912)
[1] Button "Play" center=(822,925)
[2] Button "Shuffle" center=(795,46)
[3] Button "Fullscreen" center=(1220,307)
[4] Button "Pause" center=(842,224)
[5] Slider "Volume" center=(1820,52)
[6] Button "Next" center=(1000,590)

Instruction: tap shuffle

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1653,797)
[1] PopUpButton "Appearance" center=(1717,1022)
[2] Button "Privacy & Security" center=(172,164)
[3] Button "Network" center=(1189,486)
[4] CheckBox "Bluetooth" center=(1089,921)
[5] Button "Displays" center=(1811,60)
[6] CheckBox "Wi-Fi" center=(1554,160)
[7] Slider "Brightness" center=(94,660)

Instruction: tap network

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(318,551)
[1] Button "Reload" center=(1453,455)
[2] Button "Downloads" center=(1405,212)
[3] Button "Bookmark" center=(1120,64)
[4] Link "Pricing" center=(745,991)
[5] Link "Sign in" center=(1768,861)
[6] Link "Docs" center=(1408,722)
[7] Button "Go back" center=(1453,576)

Instruction: click go back

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(914,703)
[1] Link "Sign in" center=(1383,831)
[2] Button "Go back" center=(1545,654)
[3] Button "Go forward" center=(861,353)
[4] Button "Extensions" center=(928,68)
[5] Button "Bookmark" center=(1045,500)
[6] Button "Downloads" center=(454,690)
[7] Button "Reload" center=(691,1024)
[8] Link "Pricing" center=(1257,410)
[9] Button "Run inference" center=(839,752)

Instruction: press sign in

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1552,718)
[1] Slider "Brightness" center=(981,856)
[2] Button "Displays" center=(618,788)
[3] CheckBox "Do Not Disturb" center=(1126,841)
[4] CheckBox "Bluetooth" center=(1412,664)
[5] PopUpButton "Appearance" center=(397,400)
[6] Button "Privacy & Security" center=(131,89)
[7] Button "Sound" center=(1055,405)
[8] CheckBox "Wi-Fi" center=(256,474)

Instruction: tap bluetooth

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(271,541)
[1] Button "Fullscreen" center=(1528,548)
[2] Slider "Volume" center=(576,561)
[3] Button "Play" center=(960,1006)
[4] Button "Next" center=(617,551)
[5] Button "Pause" center=(963,661)
[6] Button "Previous" center=(528,422)

Instruction: select next

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1225,644)
[1] Button "Shuffle" center=(1001,809)
[2] Button "Previous" center=(294,221)
[3] Button "Fullscreen" center=(758,188)
[4] Button "Pause" center=(1623,76)
[5] Button "Next" center=(1765,122)
[6] Slider "Volume" center=(1292,244)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(269,631)
[1] TextField "Password" center=(222,622)
[2] CheckBox "Remember me" center=(900,1025)
[3] TextField "Email" center=(1238,629)
[4] Link "Forgot password" center=(427,255)
[5] Button "Create account" center=(796,284)

Instruction: type into password

[Assistant] {"action":"type","id":1,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(697,696)
[1] Button "Fullscreen" center=(845,583)
[2] Button "Next" center=(311,567)
[3] Button "Previous" center=(1673,795)
[4] Slider "Volume" center=(1522,841)

Instruction: make it full screen

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1023,556)
[1] Button "Sound" center=(293,156)
[2] Button "Network" center=(421,611)
[3] Slider "Brightness" center=(362,452)
[4] Button "Displays" center=(1250,915)
[5] CheckBox "Bluetooth" center=(1400,643)
[6] CheckBox "Wi-Fi" center=(443,337)

Instruction: tap appearance

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(923,383)
[1] Slider "Brightness" center=(56,852)
[2] Button "Sound" center=(540,621)
[3] CheckBox "Wi-Fi" center=(1373,323)
[4] Button "Privacy & Security" center=(1444,778)
[5] Button "Displays" center=(1701,904)
[6] CheckBox "Bluetooth" center=(939,422)
[7] PopUpButton "Appearance" center=(1753,816)
[8] CheckBox "Do Not Disturb" center=(1359,703)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(297,725)
[1] Button "Format" center=(656,256)
[2] Button "Open" center=(658,696)
[3] Button "Run" center=(693,251)
[4] Button "Save" center=(756,293)

Instruction: click new file

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(363,91)
[1] Button "Shuffle" center=(1307,318)
[2] Slider "Volume" center=(1095,75)
[3] Button "Play" center=(1541,846)
[4] Button "Fullscreen" center=(425,429)
[5] Button "Previous" center=(1251,758)

Instruction: start playback

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(49,865)
[1] Button "Privacy & Security" center=(1106,589)
[2] CheckBox "Do Not Disturb" center=(685,372)
[3] Slider "Brightness" center=(73,449)
[4] CheckBox "Wi-Fi" center=(1212,817)
[5] PopUpButton "Appearance" center=(597,273)
[6] CheckBox "Bluetooth" center=(1625,984)
[7] Button "Network" center=(1226,698)

Instruction: click wi-fi

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1314,237)
[1] Slider "Volume" center=(754,325)
[2] Button "Previous" center=(1763,638)
[3] Button "Pause" center=(347,157)
[4] Button "Next" center=(42,149)
[5] Button "Fullscreen" center=(1782,577)

Instruction: next song

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(958,1034)
[1] Button "Checkout" center=(697,255)
[2] Button "Filters" center=(1122,354)
[3] Button "Buy now" center=(1671,554)
[4] Button "Add to cart" center=(1858,689)
[5] Link "Account" center=(198,143)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(423,349)
[1] SearchField "Search or enter address" center=(1744,1003)
[2] Button "Go forward" center=(724,970)
[3] Button "Extensions" center=(702,330)
[4] Button "Run inference" center=(892,206)
[5] Link "Sign in" center=(46,973)
[6] Button "Go back" center=(404,1024)
[7] Link "Docs" center=(292,890)
[8] Button "Downloads" center=(512,389)
[9] Link "Pricing" center=(1492,296)
[10] Button "New tab" center=(1664,438)

Instruction: press go forward

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(544,722)
[1] Button "Previous" center=(644,274)
[2] Button "Fullscreen" center=(605,434)
[3] Button "Play" center=(824,974)
[4] Button "Next" center=(167,980)
[5] Slider "Volume" center=(1681,679)

Instruction: go fullscreen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(944,631)
[1] Button "Next" center=(1624,406)
[2] Slider "Volume" center=(261,440)
[3] Button "Fullscreen" center=(428,385)
[4] Button "Previous" center=(313,116)

Instruction: press pause

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(933,299)
[1] Button "Run inference" center=(1400,889)
[2] Link "Docs" center=(215,581)
[3] Button "Downloads" center=(1538,1017)
[4] Button "Go back" center=(727,316)
[5] Button "Bookmark" center=(289,236)
[6] SearchField "Search or enter address" center=(487,730)
[7] Link "Pricing" center=(443,485)
[8] Button "Go forward" center=(842,408)
[9] Button "New tab" center=(1536,862)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(939,702)
[1] Button "Create account" center=(1294,141)
[2] TextField "Password" center=(1405,422)
[3] Button "Sign in" center=(1528,524)
[4] TextField "Email" center=(1163,214)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(367,624)
[1] Button "Network" center=(1233,972)
[2] Slider "Brightness" center=(1284,316)
[3] Button "Sound" center=(1071,155)
[4] PopUpButton "Appearance" center=(634,152)
[5] CheckBox "Wi-Fi" center=(622,47)
[6] CheckBox "Bluetooth" center=(1578,126)

Instruction: select appearance

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1790,212)
[1] Button "Play" center=(884,369)
[2] Button "Next" center=(355,45)
[3] Button "Pause" center=(287,888)
[4] Slider "Volume" center=(372,880)
[5] Button "Previous" center=(1495,71)

Instruction: press next

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1153,115)
[1] Button "Reload" center=(1455,595)
[2] Button "Go forward" center=(340,514)
[3] Button "Bookmark" center=(422,1002)
[4] Button "Downloads" center=(1676,654)
[5] Link "Pricing" center=(295,293)
[6] SearchField "Search or enter address" center=(1395,434)
[7] Link "Docs" center=(976,948)
[8] Button "Extensions" center=(1408,374)

Instruction: select reload

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1513,156)
[1] Button "Create account" center=(1186,710)
[2] TextField "Password" center=(92,953)
[3] TextField "Email" center=(234,140)
[4] Link "Forgot password" center=(430,103)

Instruction: sign up

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(342,942)
[1] Button "Save" center=(1527,937)
[2] Button "Run" center=(1544,983)
[3] Button "Format" center=(555,444)
[4] Button "Commit" center=(396,1005)
[5] Button "Terminal" center=(844,528)

Instruction: press terminal

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1678,80)
[1] Button "Network" center=(1706,498)
[2] PopUpButton "Appearance" center=(1010,434)
[3] Button "Sound" center=(100,358)
[4] Button "Privacy & Security" center=(1622,177)
[5] CheckBox "Bluetooth" center=(1307,980)
[6] Slider "Brightness" center=(1538,242)
[7] CheckBox "Do Not Disturb" center=(1374,689)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1807,177)
[1] Button "Pause" center=(193,80)
[2] Slider "Volume" center=(959,705)
[3] Button "Play" center=(732,222)
[4] Button "Next" center=(1665,339)
[5] Button "Fullscreen" center=(1766,744)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1687,1038)
[1] CheckBox "Bluetooth" center=(321,392)
[2] Button "Displays" center=(606,227)
[3] Button "Network" center=(925,394)
[4] Slider "Brightness" center=(632,946)
[5] PopUpButton "Appearance" center=(172,487)
[6] Button "Privacy & Security" center=(1565,870)
[7] Button "Sound" center=(1404,584)

Instruction: press network

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(386,854)
[1] Button "Format" center=(1087,630)
[2] Button "Save" center=(430,969)
[3] Button "Open" center=(820,703)
[4] Button "New File" center=(432,1033)
[5] TextField "Search files" center=(1413,681)
[6] Button "Find" center=(380,835)
[7] Button "Run" center=(1748,510)
[8] Button "Commit" center=(1164,706)

Instruction: tap terminal

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(513,120)
[1] Button "Shuffle" center=(1107,668)
[2] Button "Previous" center=(1273,416)
[3] Slider "Volume" center=(1225,280)
[4] Button "Pause" center=(184,880)
[5] Button "Next" center=(205,782)
[6] Button "Play" center=(913,738)

Instruction: click shuffle

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(156,394)
[1] Slider "Volume" center=(54,874)
[2] Button "Previous" center=(1001,915)
[3] Button "Shuffle" center=(1382,43)
[4] Button "Fullscreen" center=(1055,530)
[5] Button "Next" center=(807,46)
[6] Button "Pause" center=(1521,611)

Instruction: select pause

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(813,556)
[1] Button "Buy now" center=(892,761)
[2] Button "Add to cart" center=(889,168)
[3] Button "Wishlist" center=(1614,112)
[4] SearchField "Search products" center=(526,599)
[5] Link "Account" center=(63,871)

Instruction: find a product

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(799,636)
[1] Button "Add to cart" center=(721,384)
[2] Button "Wishlist" center=(246,240)
[3] SearchField "Search products" center=(1713,906)
[4] Button "Filters" center=(1728,233)

Instruction: click wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1571,760)
[1] TextField "Password" center=(243,66)
[2] Link "Forgot password" center=(1261,460)
[3] Button "Create account" center=(1059,356)
[4] TextField "Email" center=(741,149)

Instruction: sign up

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(526,548)
[1] Button "Previous" center=(1848,983)
[2] Button "Pause" center=(331,184)
[3] Button "Shuffle" center=(525,307)
[4] Button "Next" center=(315,469)
[5] Button "Play" center=(1813,758)

Instruction: tap next

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1802,354)
[1] Button "Run inference" center=(1421,493)
[2] Button "Downloads" center=(1559,738)
[3] Link "Sign in" center=(1784,807)
[4] Button "Go back" center=(695,938)
[5] Button "New tab" center=(324,1017)
[6] Link "Pricing" center=(1721,162)
[7] Button "Go forward" center=(184,964)
[8] SearchField "Search or enter address" center=(370,491)
[9] Button "Reload" center=(1574,364)

Instruction: press pricing

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1135,555)
[1] Button "Format" center=(864,205)
[2] Button "Commit" center=(533,95)
[3] Button "Terminal" center=(1712,482)
[4] Button "Find" center=(1539,825)

Instruction: click find

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(1427,78)
[1] Link "Docs" center=(131,172)
[2] Button "New tab" center=(178,552)
[3] Button "Reload" center=(353,351)
[4] Button "Extensions" center=(195,763)
[5] Button "Downloads" center=(557,64)

Instruction: reload

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1595,105)
[1] Button "Save" center=(616,621)
[2] Button "New File" center=(1282,627)
[3] Button "Terminal" center=(1019,922)
[4] Button "Run" center=(1789,51)
[5] Button "Find" center=(469,720)

Instruction: type into search files

[Assistant] {"action":"type","id":0,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(668,352)
[1] TextField "Email" center=(305,146)
[2] CheckBox "Remember me" center=(128,461)
[3] Button "Create account" center=(779,107)
[4] Button "Sign in" center=(1823,86)

Instruction: tap remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(312,88)
[1] Link "Pricing" center=(1516,523)
[2] Button "Downloads" center=(1596,1039)
[3] Button "Run inference" center=(340,741)
[4] Button "Reload" center=(1434,170)
[5] Link "Sign in" center=(1430,277)
[6] SearchField "Search or enter address" center=(1262,1023)
[7] Button "Bookmark" center=(771,473)
[8] Button "Extensions" center=(1586,921)

Instruction: press bookmark

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1174,967)
[1] Button "Pause" center=(1778,882)
[2] Button "Next" center=(969,542)
[3] Button "Fullscreen" center=(907,527)
[4] Button "Shuffle" center=(1546,488)
[5] Button "Previous" center=(799,925)
[6] Button "Play" center=(170,262)

Instruction: go fullscreen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(608,558)
[1] CheckBox "Wi-Fi" center=(1015,270)
[2] CheckBox "Do Not Disturb" center=(119,321)
[3] Button "Network" center=(779,159)
[4] Slider "Brightness" center=(1242,298)

Instruction: tap brightness

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1177,960)
[1] Link "Account" center=(1444,71)
[2] SearchField "Search products" center=(1061,1024)
[3] Button "Buy now" center=(1225,713)
[4] Button "Wishlist" center=(864,837)
[5] Button "Add to cart" center=(678,785)
[6] Button "Filters" center=(1427,81)

Instruction: select add to cart

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1395,440)
[1] Button "Terminal" center=(1515,183)
[2] Button "New File" center=(107,410)
[3] Button "Save" center=(611,887)
[4] Button "Commit" center=(666,655)
[5] Button "Open" center=(431,825)

Instruction: tap terminal

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(327,1040)
[1] Link "Docs" center=(151,796)
[2] Button "Extensions" center=(998,1026)
[3] Button "Reload" center=(1340,116)
[4] Link "Sign in" center=(1612,403)

Instruction: log in

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1594,1001)
[1] Button "Go back" center=(1097,765)
[2] Link "Pricing" center=(1255,733)
[3] Button "New tab" center=(886,506)
[4] Button "Reload" center=(998,755)
[5] Button "Go forward" center=(1070,935)
[6] Link "Docs" center=(1116,590)
[7] Link "Sign in" center=(1683,641)
[8] Button "Run inference" center=(1835,208)
[9] Button "Extensions" center=(1359,772)
[10] Button "Downloads" center=(1878,787)
[11] SearchField "Search or enter address" center=(317,1038)

Instruction: click go back

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(675,546)
[1] Button "Shuffle" center=(1596,1033)
[2] Slider "Volume" center=(179,167)
[3] Button "Fullscreen" center=(1642,217)
[4] Button "Pause" center=(644,452)
[5] Button "Previous" center=(256,744)
[6] Button "Play" center=(948,415)

Instruction: click pause

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1030,123)
[1] TextField "Password" center=(1108,468)
[2] Button "Create account" center=(566,1013)
[3] Button "Sign in" center=(599,192)
[4] CheckBox "Remember me" center=(927,169)
[5] TextField "Email" center=(1616,339)

Instruction: submit the login

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1068,514)
[1] Button "Checkout" center=(994,837)
[2] Button "Filters" center=(68,276)
[3] Button "Add to cart" center=(536,200)
[4] Link "Account" center=(708,514)

Instruction: select filters

[Assistant] {"action":"click","id":2}