[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(679,745)
[1] Button "Pause" center=(1867,927)
[2] Button "Shuffle" center=(224,375)
[3] Button "Next" center=(736,642)
[4] Button "Previous" center=(574,361)
[5] Button "Fullscreen" center=(1685,715)
[6] Slider "Volume" center=(140,153)

Instruction: press pause

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(905,739)
[1] Button "Go forward" center=(1531,103)
[2] Link "Sign in" center=(505,892)
[3] Button "Extensions" center=(1260,590)
[4] Button "Reload" center=(1581,829)
[5] Button "Go back" center=(1177,866)

Instruction: select extensions

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(154,428)
[1] Button "Network" center=(1034,47)
[2] CheckBox "Bluetooth" center=(140,65)
[3] Slider "Brightness" center=(1256,711)
[4] Button "Sound" center=(1373,626)
[5] Button "Displays" center=(656,866)
[6] CheckBox "Wi-Fi" center=(318,345)
[7] CheckBox "Do Not Disturb" center=(1126,1034)
[8] Button "Privacy & Security" center=(1228,296)

Instruction: tap sound

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(371,54)
[1] Button "New File" center=(1719,849)
[2] Button "Open" center=(676,362)
[3] Button "Save" center=(907,925)
[4] Button "Format" center=(785,541)
[5] Button "Terminal" center=(1588,493)
[6] Button "Run" center=(736,590)

Instruction: tap format

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(307,288)
[1] TextField "Email" center=(1795,841)
[2] CheckBox "Remember me" center=(796,304)
[3] Link "Forgot password" center=(1676,454)
[4] TextField "Password" center=(1833,430)
[5] Button "Sign in" center=(1644,173)

Instruction: type into email

[Assistant] {"action":"type","id":1,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(149,49)
[1] Button "Open" center=(1462,994)
[2] Button "Terminal" center=(1422,475)
[3] Button "New File" center=(368,916)
[4] Button "Run" center=(422,161)
[5] Button "Commit" center=(398,195)

Instruction: tap new file

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(201,221)
[1] Button "Commit" center=(615,766)
[2] Button "Format" center=(244,134)
[3] Button "Save" center=(802,972)
[4] Button "New File" center=(1784,555)

Instruction: tap format

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(441,935)
[1] SearchField "Search or enter address" center=(202,217)
[2] Button "Reload" center=(1223,649)
[3] Button "Downloads" center=(1847,1004)
[4] Button "Bookmark" center=(773,494)
[5] Link "Docs" center=(1648,958)
[6] Button "New tab" center=(67,993)

Instruction: press downloads

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(267,981)
[1] Button "Shuffle" center=(377,94)
[2] Button "Play" center=(457,581)
[3] Button "Fullscreen" center=(554,274)
[4] Button "Pause" center=(1546,817)
[5] Button "Next" center=(1851,530)
[6] Button "Previous" center=(740,273)

Instruction: next song

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(582,1010)
[1] Button "Filters" center=(1092,219)
[2] Link "Account" center=(674,170)
[3] Button "Checkout" center=(292,727)
[4] SearchField "Search products" center=(1181,751)
[5] Button "Wishlist" center=(1578,278)

Instruction: select account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(624,707)
[1] Button "Sign in" center=(996,386)
[2] TextField "Password" center=(1555,189)
[3] Button "Create account" center=(1821,241)
[4] CheckBox "Remember me" center=(1771,941)
[5] TextField "Email" center=(197,154)

Instruction: sign in

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(537,720)
[1] Button "New File" center=(641,700)
[2] Button "Open" center=(1353,734)
[3] Button "Run" center=(1560,210)
[4] Button "Save" center=(1042,213)
[5] TextField "Search files" center=(102,236)

Instruction: save the file

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(713,771)
[1] Button "Reload" center=(1341,239)
[2] SearchField "Search or enter address" center=(107,435)
[3] Button "Bookmark" center=(1286,851)
[4] Button "New tab" center=(510,372)
[5] Button "Extensions" center=(1512,598)
[6] Link "Docs" center=(1070,889)
[7] Button "Run inference" center=(336,313)
[8] Link "Sign in" center=(1318,955)

Instruction: select bookmark

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1765,475)
[1] Link "Pricing" center=(747,216)
[2] Button "Reload" center=(611,300)
[3] Button "New tab" center=(560,978)
[4] Link "Docs" center=(1412,1018)
[5] Button "Bookmark" center=(776,307)

Instruction: press extensions

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1482,843)
[1] SearchField "Search products" center=(1171,149)
[2] Button "Filters" center=(71,697)
[3] Button "Wishlist" center=(1445,706)
[4] Button "Add to cart" center=(1587,972)
[5] Link "Account" center=(1679,783)

Instruction: search for something

[Assistant] {"action":"type","id":1,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1120,566)
[1] CheckBox "Bluetooth" center=(1630,198)
[2] Button "Sound" center=(1852,948)
[3] Button "Network" center=(1605,747)
[4] CheckBox "Do Not Disturb" center=(1266,102)
[5] Button "Privacy & Security" center=(666,362)
[6] Button "Displays" center=(1025,632)
[7] Slider "Brightness" center=(1479,176)
[8] PopUpButton "Appearance" center=(1755,530)

Instruction: press brightness

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(398,314)
[1] Link "Forgot password" center=(310,966)
[2] CheckBox "Remember me" center=(457,891)
[3] TextField "Email" center=(1180,704)
[4] TextField "Password" center=(503,646)
[5] Button "Create account" center=(960,206)

Instruction: i forgot my password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1180,986)
[1] CheckBox "Do Not Disturb" center=(474,211)
[2] Button "Privacy & Security" center=(338,164)
[3] Button "Displays" center=(131,875)
[4] PopUpButton "Appearance" center=(279,55)
[5] Button "Network" center=(419,275)
[6] CheckBox "Wi-Fi" center=(399,950)
[7] Slider "Brightness" center=(44,56)
[8] CheckBox "Bluetooth" center=(169,748)

Instruction: tap brightness

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(62,600)
[1] TextField "Search files" center=(211,877)
[2] Button "New File" center=(645,233)
[3] Button "Run" center=(803,432)
[4] Button "Format" center=(1671,135)

Instruction: type into search files

[Assistant] {"action":"type","id":1,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(605,471)
[1] Button "Wishlist" center=(1643,809)
[2] Link "Account" center=(1609,916)
[3] Button "Add to cart" center=(1167,696)
[4] Button "Buy now" center=(323,239)
[5] Button "Filters" center=(684,356)

Instruction: select buy now

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1750,809)
[1] Button "Next" center=(867,977)
[2] Button "Previous" center=(413,43)
[3] Slider "Volume" center=(797,876)
[4] Button "Shuffle" center=(1710,325)
[5] Button "Play" center=(714,1015)

Instruction: tap volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1365,442)
[1] Button "New tab" center=(799,346)
[2] Button "Run inference" center=(1816,128)
[3] Button "Reload" center=(1491,132)
[4] Link "Pricing" center=(1481,900)

Instruction: select new tab

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1623,615)
[1] Button "Run inference" center=(1596,239)
[2] Button "Go back" center=(920,469)
[3] Button "Bookmark" center=(1162,247)
[4] Button "Reload" center=(642,607)
[5] Button "Extensions" center=(328,426)
[6] Button "New tab" center=(1134,783)
[7] Link "Docs" center=(1202,618)
[8] Button "Go forward" center=(1313,454)
[9] Button "Downloads" center=(366,437)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":0,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(188,412)
[1] Button "Checkout" center=(102,419)
[2] Button "Filters" center=(1684,645)
[3] Button "Add to cart" center=(918,224)
[4] Button "Wishlist" center=(783,504)
[5] SearchField "Search products" center=(1768,119)
[6] Button "Buy now" center=(1206,505)

Instruction: press account

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(70,349)
[1] PopUpButton "Appearance" center=(201,606)
[2] Button "Displays" center=(1790,45)
[3] Button "Sound" center=(1598,1006)
[4] CheckBox "Do Not Disturb" center=(1877,625)
[5] Button "Network" center=(794,996)
[6] Slider "Brightness" center=(325,1003)
[7] CheckBox "Bluetooth" center=(189,842)

Instruction: select network

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1645,116)
[1] Button "Reload" center=(237,594)
[2] Link "Sign in" center=(1418,856)
[3] Button "Bookmark" center=(1749,700)
[4] Button "Run inference" center=(1169,188)
[5] Link "Docs" center=(614,134)
[6] Button "Downloads" center=(1286,769)

Instruction: refresh the page

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1516,788)
[1] Button "Network" center=(790,312)
[2] CheckBox "Bluetooth" center=(310,147)
[3] CheckBox "Wi-Fi" center=(1726,606)
[4] Button "Sound" center=(749,737)
[5] Button "Displays" center=(641,421)
[6] PopUpButton "Appearance" center=(1035,893)

Instruction: turn off bluetooth

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1334,448)
[1] Button "Bookmark" center=(489,987)
[2] Button "Extensions" center=(1482,960)
[3] Link "Sign in" center=(740,459)
[4] Button "New tab" center=(603,1013)
[5] Button "Go back" center=(148,510)

Instruction: select bookmark

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1034,572)
[1] Button "Run" center=(1375,1008)
[2] Button "Save" center=(1712,569)
[3] Button "Format" center=(414,204)
[4] Button "Terminal" center=(1450,308)

Instruction: execute

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(42,665)
[1] Button "New tab" center=(404,904)
[2] Button "Bookmark" center=(989,947)
[3] Button "Run inference" center=(1693,151)
[4] Button "Reload" center=(1711,712)
[5] Button "Extensions" center=(216,362)
[6] SearchField "Search or enter address" center=(420,477)
[7] Link "Pricing" center=(205,805)
[8] Button "Go back" center=(1842,84)
[9] Button "Downloads" center=(1563,411)

Instruction: click pricing

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1255,361)
[1] Button "Wishlist" center=(1433,55)
[2] Link "Account" center=(49,1019)
[3] SearchField "Search products" center=(1030,302)
[4] Button "Add to cart" center=(1474,889)
[5] Button "Filters" center=(1378,128)

Instruction: select add to cart

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(745,174)
[1] Button "Play" center=(1258,303)
[2] Slider "Volume" center=(611,1019)
[3] Button "Previous" center=(1007,824)
[4] Button "Shuffle" center=(276,463)
[5] Button "Pause" center=(1752,790)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1618,782)
[1] Button "Play" center=(1624,112)
[2] Button "Fullscreen" center=(112,304)
[3] Button "Shuffle" center=(720,86)
[4] Button "Previous" center=(354,400)

Instruction: tap previous

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1604,942)
[1] Button "Privacy & Security" center=(1727,287)
[2] Button "Displays" center=(730,932)
[3] Button "Sound" center=(350,470)
[4] Slider "Brightness" center=(1125,101)
[5] PopUpButton "Appearance" center=(870,922)
[6] Button "Network" center=(599,598)
[7] CheckBox "Bluetooth" center=(931,79)
[8] CheckBox "Do Not Disturb" center=(1566,479)

Instruction: press do not disturb

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(691,859)
[1] Button "Previous" center=(506,431)
[2] Button "Shuffle" center=(1673,969)
[3] Button "Pause" center=(1331,1032)
[4] Slider "Volume" center=(1780,345)

Instruction: click previous

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(845,408)
[1] CheckBox "Remember me" center=(959,674)
[2] TextField "Password" center=(727,400)
[3] Button "Create account" center=(203,412)
[4] TextField "Email" center=(853,203)

Instruction: click remember me

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1328,756)
[1] Button "Wishlist" center=(673,788)
[2] Button "Filters" center=(96,722)
[3] Link "Account" center=(1268,144)
[4] Button "Checkout" center=(1861,279)
[5] SearchField "Search products" center=(244,1013)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1436,438)
[1] Button "New File" center=(150,743)
[2] Button "Run" center=(1479,221)
[3] Button "Open" center=(384,689)
[4] Button "Terminal" center=(88,455)
[5] Button "Find" center=(1508,675)
[6] Button "Save" center=(799,981)
[7] Button "Commit" center=(518,362)

Instruction: click terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1647,852)
[1] Button "Previous" center=(1198,405)
[2] Slider "Volume" center=(1825,1025)
[3] Button "Fullscreen" center=(161,50)
[4] Button "Shuffle" center=(363,920)
[5] Button "Play" center=(895,817)

Instruction: skip to next track

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1337,951)
[1] Link "Forgot password" center=(1304,387)
[2] Button "Create account" center=(105,488)
[3] Button "Sign in" center=(618,84)
[4] CheckBox "Remember me" center=(1749,544)
[5] TextField "Password" center=(244,823)

Instruction: tap forgot password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1404,809)
[1] Link "Account" center=(1738,107)
[2] Button "Wishlist" center=(138,49)
[3] Button "Filters" center=(844,334)
[4] Button "Checkout" center=(1427,676)

Instruction: click wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1077,666)
[1] Button "Displays" center=(1470,1035)
[2] Button "Network" center=(1597,331)
[3] Slider "Brightness" center=(171,113)
[4] PopUpButton "Appearance" center=(1151,347)

Instruction: press sound

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1596,808)
[1] Button "Previous" center=(538,971)
[2] Button "Fullscreen" center=(805,742)
[3] Button "Pause" center=(1201,388)
[4] Button "Shuffle" center=(340,86)
[5] Slider "Volume" center=(780,437)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1159,436)
[1] Button "Commit" center=(1508,489)
[2] Button "New File" center=(1017,309)
[3] Button "Find" center=(130,836)
[4] Button "Run" center=(1464,956)

Instruction: find in file

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(82,42)
[1] Button "Previous" center=(844,542)
[2] Slider "Volume" center=(978,426)
[3] Button "Next" center=(1593,908)
[4] Button "Fullscreen" center=(1773,200)

Instruction: start playback

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1140,300)
[1] Button "Go back" center=(416,636)
[2] Button "New tab" center=(165,682)
[3] SearchField "Search or enter address" center=(422,717)
[4] Button "Run inference" center=(1212,759)
[5] Button "Extensions" center=(1635,193)
[6] Button "Reload" center=(488,848)
[7] Button "Downloads" center=(493,355)
[8] Link "Docs" center=(717,916)
[9] Button "Bookmark" center=(235,111)
[10] Link "Pricing" center=(71,794)
[11] Button "Go forward" center=(1652,72)

Instruction: tap pricing

[Assistant] {"action":"click","id":10}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(558,288)
[1] Slider "Volume" center=(68,195)
[2] Button "Shuffle" center=(1855,964)
[3] Button "Fullscreen" center=(278,130)
[4] Button "Pause" center=(826,442)
[5] Button "Next" center=(949,265)
[6] Button "Play" center=(1667,179)

Instruction: select previous

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(390,647)
[1] Button "Play" center=(1549,647)
[2] Button "Pause" center=(1619,473)
[3] Button "Shuffle" center=(678,211)
[4] Button "Fullscreen" center=(849,87)

Instruction: play

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1463,650)
[1] Button "Bookmark" center=(298,41)
[2] Button "Run inference" center=(951,157)
[3] SearchField "Search or enter address" center=(226,860)
[4] Button "Go back" center=(745,902)

Instruction: previous page

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1213,971)
[1] Button "Run" center=(1041,780)
[2] Button "Commit" center=(48,742)
[3] Button "Save" center=(789,619)
[4] Button "Terminal" center=(233,484)
[5] Button "Find" center=(257,831)
[6] Button "Format" center=(269,555)
[7] Button "Open" center=(934,256)
[8] Button "New File" center=(1136,420)

Instruction: click new file

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1481,747)
[1] Button "Pause" center=(232,430)
[2] Button "Shuffle" center=(1178,569)
[3] Button "Previous" center=(131,765)
[4] Button "Fullscreen" center=(143,808)

Instruction: press shuffle

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1685,561)
[1] Button "Format" center=(605,379)
[2] Button "New File" center=(1846,358)
[3] Button "Open" center=(1814,94)
[4] Button "Terminal" center=(1386,138)
[5] Button "Find" center=(1092,181)

Instruction: press open

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(556,130)
[1] Button "Extensions" center=(608,325)
[2] SearchField "Search or enter address" center=(1037,325)
[3] Link "Docs" center=(1381,256)
[4] Button "Downloads" center=(1537,174)

Instruction: click downloads

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(607,999)
[1] Button "Reload" center=(1503,752)
[2] Button "New tab" center=(345,264)
[3] Button "Extensions" center=(1785,174)
[4] Link "Pricing" center=(307,774)

Instruction: press new tab

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(231,467)
[1] Slider "Volume" center=(727,947)
[2] Button "Fullscreen" center=(136,456)
[3] Button "Shuffle" center=(559,445)
[4] Button "Next" center=(443,42)
[5] Button "Pause" center=(1437,884)
[6] Button "Play" center=(157,392)

Instruction: next song

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1374,205)
[1] Button "New File" center=(1265,131)
[2] TextField "Search files" center=(891,739)
[3] Button "Open" center=(1849,872)
[4] Button "Save" center=(1444,800)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(547,902)
[1] Button "Terminal" center=(1559,594)
[2] TextField "Search files" center=(1721,322)
[3] Button "Open" center=(176,674)
[4] Button "Commit" center=(832,441)
[5] Button "Format" center=(372,194)
[6] Button "Find" center=(1202,123)

Instruction: click terminal

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1763,1032)
[1] Button "Open" center=(845,511)
[2] Button "Commit" center=(1582,938)
[3] Button "Run" center=(1864,81)
[4] Button "New File" center=(824,279)
[5] TextField "Search files" center=(1474,443)
[6] Button "Find" center=(353,126)
[7] Button "Save" center=(1208,564)

Instruction: type into search files

[Assistant] {"action":"type","id":5,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(943,1028)
[1] Button "Create account" center=(923,440)
[2] TextField "Password" center=(442,741)
[3] Link "Forgot password" center=(667,659)
[4] Button "Sign in" center=(1617,233)

Instruction: log in

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(305,906)
[1] TextField "Password" center=(385,434)
[2] TextField "Email" center=(315,484)
[3] Button "Create account" center=(1846,894)
[4] Link "Forgot password" center=(280,130)

Instruction: type into password

[Assistant] {"action":"type","id":1,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(585,729)
[1] Link "Forgot password" center=(349,143)
[2] Button "Sign in" center=(1430,567)
[3] CheckBox "Remember me" center=(1781,565)
[4] Button "Create account" center=(1343,947)
[5] TextField "Password" center=(120,586)

Instruction: click create account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(213,713)
[1] Button "Shuffle" center=(1270,624)
[2] Button "Next" center=(1438,429)
[3] Button "Fullscreen" center=(579,265)
[4] Button "Pause" center=(451,613)
[5] Button "Previous" center=(733,864)
[6] Slider "Volume" center=(530,948)

Instruction: press next

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(542,306)
[1] Link "Forgot password" center=(454,1012)
[2] Button "Create account" center=(1709,75)
[3] Button "Sign in" center=(1696,746)
[4] TextField "Email" center=(533,119)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1808,518)
[1] Button "Next" center=(1634,785)
[2] Button "Pause" center=(839,978)
[3] Button "Play" center=(1452,617)
[4] Button "Shuffle" center=(1422,315)
[5] Button "Previous" center=(1062,378)

Instruction: start playback

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(799,617)
[1] Button "Play" center=(1007,917)
[2] Button "Previous" center=(594,442)
[3] Button "Pause" center=(831,993)
[4] Button "Next" center=(1531,648)
[5] Button "Shuffle" center=(1631,554)

Instruction: tap volume

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1468,970)
[1] Button "Privacy & Security" center=(56,626)
[2] Button "Sound" center=(720,987)
[3] PopUpButton "Appearance" center=(1757,999)
[4] Slider "Brightness" center=(1306,979)
[5] CheckBox "Bluetooth" center=(667,560)
[6] CheckBox "Wi-Fi" center=(219,874)
[7] CheckBox "Do Not Disturb" center=(739,229)
[8] Button "Displays" center=(1597,448)

Instruction: select bluetooth

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(67,504)
[1] SearchField "Search products" center=(1821,322)
[2] Button "Add to cart" center=(319,285)
[3] Link "Account" center=(793,835)
[4] Button "Checkout" center=(1509,642)

Instruction: add this to my cart

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1712,159)
[1] Link "Forgot password" center=(656,835)
[2] CheckBox "Remember me" center=(688,947)
[3] TextField "Password" center=(672,258)
[4] Button "Sign in" center=(1511,869)
[5] TextField "Email" center=(154,503)

Instruction: reset password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(330,994)
[1] PopUpButton "Appearance" center=(286,121)
[2] CheckBox "Do Not Disturb" center=(794,787)
[3] Button "Network" center=(1164,497)
[4] Button "Privacy & Security" center=(229,978)
[5] Button "Displays" center=(1180,327)

Instruction: select appearance

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(586,904)
[1] Slider "Volume" center=(1433,706)
[2] Button "Fullscreen" center=(1863,735)
[3] Button "Previous" center=(1060,89)
[4] Button "Pause" center=(270,549)
[5] Button "Shuffle" center=(437,698)

Instruction: click shuffle

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(967,175)
[1] Button "Shuffle" center=(1222,480)
[2] Button "Play" center=(859,738)
[3] Button "Pause" center=(703,456)
[4] Button "Next" center=(425,239)
[5] Button "Previous" center=(998,43)

Instruction: go fullscreen

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(792,373)
[1] Button "Play" center=(1369,571)
[2] Button "Fullscreen" center=(1388,86)
[3] Button "Previous" center=(114,65)
[4] Button "Pause" center=(351,535)

Instruction: click play

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(911,67)
[1] TextField "Email" center=(1277,348)
[2] Link "Forgot password" center=(504,1030)
[3] Button "Create account" center=(1214,748)
[4] Button "Sign in" center=(1661,166)

Instruction: type into email

[Assistant] {"action":"type","id":1,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1314,394)
[1] Button "New File" center=(758,766)
[2] Button "Find" center=(355,570)
[3] Button "Format" center=(1628,440)
[4] Button "Save" center=(400,968)
[5] Button "Terminal" center=(1516,67)
[6] Button "Run" center=(1270,700)
[7] Button "Open" center=(421,629)

Instruction: press open

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(343,213)
[1] TextField "Email" center=(348,695)
[2] Button "Create account" center=(1459,829)
[3] Link "Forgot password" center=(1708,493)
[4] Button "Sign in" center=(504,756)

Instruction: log in

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(111,520)
[1] Button "Go back" center=(853,133)
[2] Button "Extensions" center=(140,604)
[3] Link "Sign in" center=(980,86)
[4] Button "Run inference" center=(1087,231)
[5] Link "Docs" center=(836,661)
[6] Button "Go forward" center=(1059,846)
[7] SearchField "Search or enter address" center=(287,822)
[8] Button "Bookmark" center=(1075,866)
[9] Link "Pricing" center=(364,889)

Instruction: click go forward

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1799,544)
[1] SearchField "Search or enter address" center=(1404,852)
[2] Button "Downloads" center=(1764,685)
[3] Button "Run inference" center=(930,985)
[4] Button "New tab" center=(1488,820)
[5] Link "Sign in" center=(576,586)

Instruction: tap run inference

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(330,1015)
[1] PopUpButton "Appearance" center=(370,981)
[2] CheckBox "Wi-Fi" center=(1498,560)
[3] Button "Displays" center=(74,689)
[4] CheckBox "Bluetooth" center=(257,812)
[5] CheckBox "Do Not Disturb" center=(1284,339)

Instruction: select appearance

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1543,1021)
[1] Button "Add to cart" center=(764,454)
[2] Button "Checkout" center=(157,426)
[3] Button "Buy now" center=(414,880)
[4] SearchField "Search products" center=(1859,672)

Instruction: proceed to payment

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(604,147)
[1] Button "Reload" center=(1735,281)
[2] Button "Run inference" center=(52,537)
[3] Button "New tab" center=(1410,362)
[4] Button "Go back" center=(509,917)
[5] Button "Downloads" center=(1184,1012)
[6] Button "Extensions" center=(664,830)
[7] Button "Go forward" center=(289,708)
[8] SearchField "Search or enter address" center=(1190,371)
[9] Link "Pricing" center=(344,53)
[10] Button "Bookmark" center=(1859,715)
[11] Link "Docs" center=(186,265)

Instruction: tap run inference

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(788,435)
[1] Button "Pause" center=(169,361)
[2] Button "Fullscreen" center=(1536,323)
[3] Button "Previous" center=(1659,527)
[4] Button "Next" center=(132,410)
[5] Slider "Volume" center=(692,297)
[6] Button "Play" center=(1204,114)

Instruction: click volume

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(715,478)
[1] Link "Forgot password" center=(596,57)
[2] CheckBox "Remember me" center=(1206,995)
[3] Button "Sign in" center=(1413,133)
[4] Button "Create account" center=(1220,778)
[5] TextField "Email" center=(809,646)

Instruction: sign up

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(559,212)
[1] Button "Sign in" center=(930,1039)
[2] Link "Forgot password" center=(364,722)
[3] Button "Create account" center=(70,112)
[4] TextField "Email" center=(1315,927)

Instruction: press remember me

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(538,809)
[1] Button "Previous" center=(1507,480)
[2] Button "Fullscreen" center=(1310,453)
[3] Button "Next" center=(987,584)
[4] Button "Pause" center=(1200,702)
[5] Button "Play" center=(969,455)
[6] Button "Shuffle" center=(114,619)

Instruction: skip to next track

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1422,822)
[1] Button "Run" center=(97,511)
[2] Button "Format" center=(734,752)
[3] Button "Commit" center=(1221,1026)
[4] Button "Find" center=(1126,587)
[5] Button "Save" center=(1313,526)
[6] Button "Open" center=(1368,597)
[7] TextField "Search files" center=(900,400)

Instruction: select commit

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1416,782)
[1] Button "Privacy & Security" center=(1648,632)
[2] CheckBox "Wi-Fi" center=(1344,382)
[3] Button "Displays" center=(412,278)
[4] CheckBox "Do Not Disturb" center=(295,188)
[5] Button "Network" center=(133,202)
[6] Button "Sound" center=(287,640)
[7] PopUpButton "Appearance" center=(181,400)
[8] CheckBox "Bluetooth" center=(1764,64)

Instruction: tap appearance

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1496,91)
[1] CheckBox "Remember me" center=(42,183)
[2] TextField "Password" center=(901,1031)
[3] Button "Create account" center=(1189,515)
[4] Link "Forgot password" center=(850,353)

Instruction: sign up

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(829,951)
[1] Button "Buy now" center=(751,476)
[2] Button "Add to cart" center=(1454,74)
[3] Button "Checkout" center=(566,918)
[4] Button "Filters" center=(717,73)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(597,618)
[1] CheckBox "Remember me" center=(913,671)
[2] TextField "Email" center=(421,1025)
[3] Button "Create account" center=(991,491)
[4] TextField "Password" center=(201,152)
[5] Button "Sign in" center=(1847,690)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(407,416)
[1] Button "Sound" center=(756,152)
[2] Slider "Brightness" center=(370,581)
[3] PopUpButton "Appearance" center=(369,513)
[4] CheckBox "Wi-Fi" center=(886,730)

Instruction: press brightness

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(651,1035)
[1] Link "Pricing" center=(716,811)
[2] Button "Go forward" center=(576,935)
[3] Link "Docs" center=(638,803)
[4] Button "Reload" center=(493,354)
[5] Button "Bookmark" center=(1492,303)
[6] SearchField "Search or enter address" center=(1648,114)
[7] Button "Extensions" center=(1661,958)

Instruction: tap pricing

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1224,501)
[1] Button "Wishlist" center=(623,271)
[2] Button "Checkout" center=(1217,942)
[3] Link "Account" center=(592,938)
[4] Button "Buy now" center=(879,310)
[5] Button "Filters" center=(75,578)

Instruction: press filters

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1218,725)
[1] CheckBox "Bluetooth" center=(1714,585)
[2] Slider "Brightness" center=(1355,829)
[3] PopUpButton "Appearance" center=(1847,574)
[4] Button "Privacy & Security" center=(1233,918)
[5] CheckBox "Do Not Disturb" center=(592,142)

Instruction: click sound

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1020,374)
[1] Button "Filters" center=(657,960)
[2] SearchField "Search products" center=(1784,571)
[3] Link "Account" center=(752,237)
[4] Button "Add to cart" center=(627,685)

Instruction: select account

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(60,83)
[1] Button "Sign in" center=(658,324)
[2] TextField "Email" center=(1200,483)
[3] TextField "Password" center=(1777,351)
[4] Link "Forgot password" center=(821,991)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1542,924)
[1] Button "Displays" center=(985,381)
[2] Button "Network" center=(1798,442)
[3] Slider "Brightness" center=(1810,404)
[4] CheckBox "Do Not Disturb" center=(1122,763)
[5] Button "Sound" center=(1258,441)

Instruction: select do not disturb

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(574,762)
[1] Button "Next" center=(778,605)
[2] Slider "Volume" center=(651,233)
[3] Button "Fullscreen" center=(571,752)
[4] Button "Play" center=(710,446)

Instruction: click pause

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1413,798)
[1] Button "Network" center=(317,74)
[2] Button "Privacy & Security" center=(1797,294)
[3] Slider "Brightness" center=(439,357)
[4] CheckBox "Do Not Disturb" center=(976,339)
[5] CheckBox "Bluetooth" center=(415,311)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1205,921)
[1] CheckBox "Bluetooth" center=(614,430)
[2] PopUpButton "Appearance" center=(1582,636)
[3] CheckBox "Wi-Fi" center=(142,196)
[4] Button "Network" center=(1854,864)
[5] Button "Sound" center=(1379,131)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(374,205)
[1] Button "Run inference" center=(1709,612)
[2] Button "Extensions" center=(1288,1006)
[3] Link "Docs" center=(614,472)
[4] Link "Sign in" center=(840,386)

Instruction: press extensions

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(315,378)
[1] Slider "Brightness" center=(861,627)
[2] CheckBox "Wi-Fi" center=(260,260)
[3] Button "Sound" center=(458,938)
[4] Button "Privacy & Security" center=(60,275)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1644,507)
[1] Button "Sign in" center=(787,995)
[2] TextField "Email" center=(1710,50)
[3] Link "Forgot password" center=(936,916)
[4] CheckBox "Remember me" center=(914,185)
[5] Button "Create account" center=(1629,529)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(154,120)
[1] Button "Sign in" center=(394,972)
[2] Button "Create account" center=(512,792)
[3] CheckBox "Remember me" center=(1231,609)
[4] TextField "Email" center=(1616,304)

Instruction: log in

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(179,1009)
[1] Link "Forgot password" center=(1677,635)
[2] Button "Create account" center=(1697,799)
[3] TextField "Email" center=(1672,979)
[4] TextField "Password" center=(1618,56)
[5] CheckBox "Remember me" center=(226,505)

Instruction: i forgot my password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1392,944)
[1] Link "Forgot password" center=(1159,513)
[2] TextField "Password" center=(1534,834)
[3] CheckBox "Remember me" center=(248,891)
[4] Button "Sign in" center=(1015,697)
[5] TextField "Email" center=(1342,747)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1826,982)
[1] Button "Format" center=(942,197)
[2] Button "Find" center=(1655,733)
[3] Button "Commit" center=(1000,684)
[4] Button "Run" center=(343,387)
[5] Button "Save" center=(1075,933)
[6] Button "Terminal" center=(171,224)
[7] Button "New File" center=(358,364)
[8] Button "Open" center=(1266,81)

Instruction: save

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(424,502)
[1] Button "Play" center=(1214,827)
[2] Button "Previous" center=(196,853)
[3] Button "Shuffle" center=(421,142)
[4] Button "Pause" center=(306,126)
[5] Slider "Volume" center=(216,88)
[6] Button "Next" center=(284,583)

Instruction: click pause

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1791,90)
[1] Button "Network" center=(1807,884)
[2] CheckBox "Do Not Disturb" center=(1032,338)
[3] CheckBox "Wi-Fi" center=(1460,76)
[4] Button "Displays" center=(1674,684)
[5] PopUpButton "Appearance" center=(501,652)
[6] Slider "Brightness" center=(349,400)
[7] Button "Privacy & Security" center=(252,641)

Instruction: select displays

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1572,300)
[1] Button "Privacy & Security" center=(551,265)
[2] PopUpButton "Appearance" center=(1181,505)
[3] Button "Displays" center=(412,207)
[4] Slider "Brightness" center=(83,409)
[5] CheckBox "Wi-Fi" center=(1795,336)
[6] Button "Network" center=(346,187)
[7] CheckBox "Bluetooth" center=(271,424)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1145,876)
[1] Button "Add to cart" center=(440,247)
[2] Button "Checkout" center=(633,341)
[3] SearchField "Search products" center=(658,493)
[4] Button "Buy now" center=(1836,966)
[5] Button "Wishlist" center=(1747,58)
[6] Link "Account" center=(664,554)

Instruction: find a product

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1293,824)
[1] Button "Go forward" center=(1831,360)
[2] Button "Downloads" center=(428,846)
[3] Button "Bookmark" center=(1436,498)
[4] Button "Extensions" center=(1057,303)
[5] Link "Sign in" center=(1317,758)

Instruction: tap bookmark

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(418,503)
[1] Button "Play" center=(1059,525)
[2] Slider "Volume" center=(693,853)
[3] Button "Next" center=(389,203)
[4] Button "Fullscreen" center=(774,620)
[5] Button "Pause" center=(354,57)

Instruction: skip to next track

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(186,370)
[1] Button "Network" center=(1810,1017)
[2] Slider "Brightness" center=(1614,460)
[3] CheckBox "Bluetooth" center=(487,83)
[4] Button "Displays" center=(350,645)

Instruction: tap displays

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(127,481)
[1] Button "Displays" center=(812,117)
[2] CheckBox "Wi-Fi" center=(726,719)
[3] PopUpButton "Appearance" center=(570,678)
[4] CheckBox "Bluetooth" center=(1093,67)
[5] Button "Sound" center=(1363,724)

Instruction: press do not disturb

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(171,878)
[1] Slider "Brightness" center=(1355,125)
[2] PopUpButton "Appearance" center=(402,910)
[3] Button "Privacy & Security" center=(350,136)
[4] CheckBox "Wi-Fi" center=(1737,217)
[5] Button "Displays" center=(1641,118)

Instruction: click appearance

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1062,451)
[1] Button "Shuffle" center=(650,955)
[2] Button "Fullscreen" center=(57,146)
[3] Button "Previous" center=(1603,636)
[4] Button "Pause" center=(264,583)
[5] Slider "Volume" center=(709,308)
[6] Button "Next" center=(1179,590)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(945,693)
[1] Button "Create account" center=(1114,531)
[2] CheckBox "Remember me" center=(540,891)
[3] Link "Forgot password" center=(682,573)
[4] TextField "Email" center=(710,834)
[5] TextField "Password" center=(1548,1022)

Instruction: reset password

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(646,714)
[1] Slider "Volume" center=(409,154)
[2] Button "Fullscreen" center=(1290,184)
[3] Button "Previous" center=(1504,992)
[4] Button "Play" center=(933,868)

Instruction: press pause

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(484,854)
[1] CheckBox "Bluetooth" center=(270,328)
[2] Slider "Brightness" center=(1540,537)
[3] CheckBox "Do Not Disturb" center=(1086,979)
[4] Button "Privacy & Security" center=(1842,621)
[5] PopUpButton "Appearance" center=(1622,922)
[6] Button "Sound" center=(1473,926)
[7] Button "Displays" center=(269,132)
[8] CheckBox "Wi-Fi" center=(1004,923)

Instruction: tap brightness

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(369,566)
[1] Button "Add to cart" center=(1470,243)
[2] Button "Wishlist" center=(267,793)
[3] Button "Checkout" center=(1165,60)
[4] Link "Account" center=(156,264)
[5] Button "Filters" center=(1832,482)
[6] SearchField "Search products" center=(904,459)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(729,238)
[1] Button "Previous" center=(1154,445)
[2] Slider "Volume" center=(426,933)
[3] Button "Next" center=(237,43)
[4] Button "Shuffle" center=(698,686)
[5] Button "Fullscreen" center=(135,374)
[6] Button "Play" center=(1698,223)

Instruction: press shuffle

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1634,618)
[1] Button "Filters" center=(1611,666)
[2] Button "Wishlist" center=(167,101)
[3] Button "Buy now" center=(963,447)
[4] Button "Checkout" center=(1133,754)
[5] Button "Add to cart" center=(925,207)
[6] Link "Account" center=(1724,508)

Instruction: press filters

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1708,535)
[1] CheckBox "Wi-Fi" center=(874,961)
[2] CheckBox "Do Not Disturb" center=(1569,125)
[3] Slider "Brightness" center=(1075,384)
[4] Button "Sound" center=(1340,156)
[5] Button "Displays" center=(1848,686)

Instruction: press brightness

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(428,198)
[1] Button "Format" center=(1399,557)
[2] Button "Save" center=(1660,200)
[3] Button "New File" center=(451,961)
[4] Button "Find" center=(502,774)

Instruction: commit my changes

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1490,1000)
[1] CheckBox "Remember me" center=(1605,793)
[2] Button "Sign in" center=(1474,60)
[3] TextField "Email" center=(871,1015)
[4] Button "Create account" center=(1874,533)
[5] TextField "Password" center=(1049,434)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1617,116)
[1] Button "Open" center=(1312,170)
[2] Button "Terminal" center=(996,661)
[3] Button "Commit" center=(950,1019)
[4] Button "Find" center=(879,159)
[5] TextField "Search files" center=(983,980)

Instruction: execute

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1544,731)
[1] Button "Wishlist" center=(74,773)
[2] Link "Account" center=(1723,789)
[3] Button "Checkout" center=(348,758)
[4] Button "Buy now" center=(840,425)

Instruction: check out

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(458,355)
[1] Button "Buy now" center=(1303,559)
[2] Button "Wishlist" center=(1254,619)
[3] Button "Add to cart" center=(1864,1015)
[4] SearchField "Search products" center=(372,391)
[5] Button "Checkout" center=(707,268)

Instruction: proceed to payment

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1248,351)
[1] SearchField "Search or enter address" center=(168,822)
[2] Link "Docs" center=(936,627)
[3] Button "Go forward" center=(1307,445)
[4] Button "Bookmark" center=(1032,62)
[5] Button "Downloads" center=(1191,43)
[6] Button "Run inference" center=(1173,840)
[7] Button "Extensions" center=(1259,508)
[8] Button "Go back" center=(668,742)
[9] Button "Reload" center=(1432,549)
[10] Button "New tab" center=(562,612)

Instruction: select reload

[Assistant] {"action":"click","id":9}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(536,335)
[1] Button "Format" center=(1758,509)
[2] Button "Save" center=(1102,1006)
[3] Button "New File" center=(1330,391)
[4] Button "Commit" center=(462,448)
[5] Button "Terminal" center=(1541,635)
[6] TextField "Search files" center=(1138,867)
[7] Button "Open" center=(1132,149)

Instruction: click commit

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1833,50)
[1] Button "Create account" center=(709,1028)
[2] Link "Forgot password" center=(390,161)
[3] TextField "Email" center=(1751,948)
[4] CheckBox "Remember me" center=(104,207)
[5] Button "Sign in" center=(856,182)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1134,893)
[1] Button "Run" center=(557,407)
[2] Button "New File" center=(1105,146)
[3] Button "Format" center=(1617,266)
[4] Button "Save" center=(41,609)
[5] TextField "Search files" center=(1836,592)
[6] Button "Terminal" center=(1213,608)

Instruction: type into search files

[Assistant] {"action":"type","id":5,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(891,926)
[1] Button "Go back" center=(524,54)
[2] Button "Extensions" center=(345,987)
[3] Button "Reload" center=(955,847)
[4] SearchField "Search or enter address" center=(798,977)
[5] Link "Docs" center=(66,88)
[6] Button "Run inference" center=(41,257)
[7] Button "Bookmark" center=(134,735)
[8] Link "Sign in" center=(746,777)
[9] Button "New tab" center=(839,818)

Instruction: take me back

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(574,825)
[1] Button "Shuffle" center=(1197,236)
[2] Button "Fullscreen" center=(1730,488)
[3] Button "Previous" center=(1148,720)
[4] Button "Next" center=(1769,958)
[5] Button "Play" center=(642,482)
[6] Button "Pause" center=(959,623)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1685,75)
[1] Button "Pause" center=(103,155)
[2] Slider "Volume" center=(1332,808)
[3] Button "Previous" center=(457,477)
[4] Button "Fullscreen" center=(802,949)
[5] Button "Shuffle" center=(1448,165)
[6] Button "Play" center=(137,276)

Instruction: select volume

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1402,391)
[1] Button "Sign in" center=(904,612)
[2] CheckBox "Remember me" center=(860,878)
[3] TextField "Email" center=(806,208)
[4] Link "Forgot password" center=(656,1002)
[5] TextField "Password" center=(119,426)

Instruction: submit the login

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(417,1001)
[1] Button "Checkout" center=(1791,225)
[2] Button "Filters" center=(295,663)
[3] Button "Buy now" center=(56,379)
[4] Link "Account" center=(523,260)

Instruction: tap filters

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(708,214)
[1] Button "Go back" center=(237,908)
[2] Button "Go forward" center=(1727,877)
[3] Button "Downloads" center=(514,587)
[4] Link "Sign in" center=(501,360)
[5] Button "Run inference" center=(52,540)
[6] Button "Reload" center=(1740,123)
[7] Link "Docs" center=(320,334)
[8] SearchField "Search or enter address" center=(722,786)
[9] Button "Extensions" center=(752,263)
[10] Link "Pricing" center=(248,210)

Instruction: click extensions

[Assistant] {"action":"click","id":9}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(1053,200)
[1] Button "Downloads" center=(911,830)
[2] Link "Pricing" center=(594,588)
[3] Button "Go forward" center=(1355,598)
[4] SearchField "Search or enter address" center=(1538,292)
[5] Button "Run inference" center=(1304,394)
[6] Button "Go back" center=(1178,368)
[7] Link "Sign in" center=(1498,731)
[8] Button "Reload" center=(697,607)
[9] Link "Docs" center=(705,215)
[10] Button "Bookmark" center=(410,638)
[11] Button "Extensions" center=(1409,352)

Instruction: press run inference

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(794,669)
[1] Button "Pause" center=(212,499)
[2] Slider "Volume" center=(176,96)
[3] Button "Previous" center=(728,978)
[4] Button "Play" center=(146,581)
[5] Button "Next" center=(1784,692)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(511,823)
[1] Link "Account" center=(789,693)
[2] Button "Buy now" center=(479,405)
[3] SearchField "Search products" center=(1621,184)
[4] Button "Wishlist" center=(649,488)
[5] Button "Checkout" center=(1829,581)

Instruction: go to checkout

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1040,562)
[1] Link "Sign in" center=(1134,71)
[2] Button "Go back" center=(495,239)
[3] Button "Go forward" center=(295,582)
[4] SearchField "Search or enter address" center=(485,471)
[5] Link "Docs" center=(210,934)
[6] Button "Reload" center=(1652,110)
[7] Button "Bookmark" center=(1449,245)

Instruction: go back

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1666,218)
[1] Button "Add to cart" center=(1022,596)
[2] Button "Checkout" center=(1013,241)
[3] Button "Buy now" center=(59,332)
[4] Link "Account" center=(1734,608)
[5] SearchField "Search products" center=(930,228)

Instruction: select filters

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1532,271)
[1] CheckBox "Wi-Fi" center=(1028,871)
[2] Button "Displays" center=(1690,843)
[3] Button "Privacy & Security" center=(1088,91)
[4] Button "Network" center=(247,564)
[5] CheckBox "Do Not Disturb" center=(1196,924)
[6] Button "Sound" center=(259,404)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(173,357)
[1] TextField "Search files" center=(499,254)
[2] Button "Save" center=(1706,445)
[3] Button "Find" center=(319,641)
[4] Button "Commit" center=(1278,629)
[5] Button "Run" center=(566,949)
[6] Button "Format" center=(810,684)
[7] Button "Open" center=(1665,480)
[8] Button "Terminal" center=(576,1034)

Instruction: click new file

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(74,584)
[1] Slider "Volume" center=(519,892)
[2] Button "Pause" center=(1440,765)
[3] Button "Next" center=(737,207)
[4] Button "Shuffle" center=(234,761)
[5] Button "Fullscreen" center=(214,593)
[6] Button "Play" center=(1257,459)

Instruction: select pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(799,488)
[1] CheckBox "Bluetooth" center=(389,614)
[2] Button "Network" center=(1835,419)
[3] Slider "Brightness" center=(1727,833)
[4] PopUpButton "Appearance" center=(1862,542)
[5] Button "Sound" center=(712,690)
[6] CheckBox "Do Not Disturb" center=(1494,624)
[7] Button "Privacy & Security" center=(753,812)
[8] Button "Displays" center=(1104,169)

Instruction: click bluetooth

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(652,581)
[1] CheckBox "Wi-Fi" center=(1146,232)
[2] PopUpButton "Appearance" center=(1349,663)
[3] Button "Privacy & Security" center=(1801,502)
[4] Button "Displays" center=(1880,123)
[5] Button "Sound" center=(986,54)

Instruction: select displays

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1597,934)
[1] TextField "Password" center=(367,575)
[2] Link "Forgot password" center=(841,575)
[3] CheckBox "Remember me" center=(254,124)
[4] TextField "Email" center=(413,783)
[5] Button "Sign in" center=(485,693)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1046,894)
[1] Button "Filters" center=(1143,121)
[2] Link "Account" center=(603,1007)
[3] Button "Add to cart" center=(729,630)
[4] SearchField "Search products" center=(63,948)
[5] Button "Wishlist" center=(1721,63)
[6] Button "Checkout" center=(1551,854)

Instruction: press checkout

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1725,848)
[1] Link "Sign in" center=(805,131)
[2] SearchField "Search or enter address" center=(1326,888)
[3] Button "Bookmark" center=(1385,360)
[4] Button "New tab" center=(1564,556)
[5] Button "Go forward" center=(1142,808)
[6] Button "Reload" center=(82,812)
[7] Button "Downloads" center=(1518,515)

Instruction: select docs

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(203,50)
[1] Button "Network" center=(549,1021)
[2] PopUpButton "Appearance" center=(1796,585)
[3] Button "Displays" center=(297,913)
[4] Slider "Brightness" center=(1214,525)
[5] Button "Sound" center=(669,177)
[6] CheckBox "Wi-Fi" center=(1363,926)
[7] Button "Privacy & Security" center=(685,397)
[8] CheckBox "Do Not Disturb" center=(288,289)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(164,940)
[1] Button "Buy now" center=(317,346)
[2] Button "Wishlist" center=(474,771)
[3] Link "Account" center=(984,251)
[4] SearchField "Search products" center=(860,111)
[5] Button "Add to cart" center=(742,995)

Instruction: find a product

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(763,287)
[1] Button "Commit" center=(1675,673)
[2] Button "Find" center=(163,761)
[3] Button "Run" center=(54,681)
[4] TextField "Search files" center=(723,597)
[5] Button "Open" center=(698,1038)

Instruction: execute

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(420,405)
[1] Button "Terminal" center=(750,848)
[2] Button "Save" center=(1836,158)
[3] Button "Find" center=(247,428)
[4] TextField "Search files" center=(634,292)
[5] Button "New File" center=(844,1019)
[6] Button "Format" center=(1430,668)
[7] Button "Open" center=(127,715)
[8] Button "Run" center=(314,898)

Instruction: find in file

[Assistant] {"action":"click","id":3}