[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(391,865)
[1] SearchField "Search products" center=(1729,432)
[2] Button "Filters" center=(456,594)
[3] Button "Add to cart" center=(1302,57)
[4] Link "Account" center=(368,133)

Instruction: select account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(491,326)
[1] Button "Save" center=(259,43)
[2] Button "Format" center=(529,727)
[3] Button "Find" center=(671,485)
[4] Button "Commit" center=(1227,921)
[5] Button "Run" center=(1268,909)

Instruction: save the file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1845,729)
[1] Button "Commit" center=(1046,487)
[2] Button "Save" center=(1657,936)
[3] Button "Find" center=(1113,390)
[4] Button "Run" center=(699,61)
[5] Button "Format" center=(1062,559)

Instruction: search

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(601,294)
[1] CheckBox "Do Not Disturb" center=(1504,337)
[2] Button "Sound" center=(1880,276)
[3] Button "Displays" center=(817,654)
[4] Slider "Brightness" center=(1837,520)
[5] Button "Network" center=(89,742)
[6] PopUpButton "Appearance" center=(1193,289)
[7] Button "Privacy & Security" center=(1361,754)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(223,487)
[1] Button "Open" center=(562,396)
[2] Button "New File" center=(508,315)
[3] Button "Find" center=(790,973)
[4] Button "Commit" center=(572,530)
[5] Button "Save" center=(363,212)
[6] Button "Run" center=(488,914)
[7] TextField "Search files" center=(1239,700)

Instruction: tap find

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1300,752)
[1] Button "Create account" center=(1172,91)
[2] Button "Sign in" center=(552,121)
[3] CheckBox "Remember me" center=(1115,994)
[4] TextField "Password" center=(562,503)

Instruction: select create account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(146,626)
[1] Slider "Brightness" center=(1380,841)
[2] PopUpButton "Appearance" center=(1646,932)
[3] CheckBox "Wi-Fi" center=(1815,1004)
[4] Button "Displays" center=(1557,220)
[5] CheckBox "Bluetooth" center=(630,601)
[6] CheckBox "Do Not Disturb" center=(1003,1004)
[7] Button "Privacy & Security" center=(906,956)
[8] Button "Network" center=(383,281)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(294,837)
[1] Button "Wishlist" center=(987,606)
[2] Button "Buy now" center=(1598,417)
[3] SearchField "Search products" center=(1521,223)
[4] Button "Filters" center=(352,586)
[5] Button "Add to cart" center=(422,291)
[6] Link "Account" center=(1651,48)

Instruction: select wishlist

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(139,520)
[1] Button "Wishlist" center=(359,906)
[2] Button "Buy now" center=(310,287)
[3] Link "Account" center=(1804,663)
[4] SearchField "Search products" center=(1748,416)

Instruction: tap buy now

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(812,179)
[1] Button "Privacy & Security" center=(41,334)
[2] PopUpButton "Appearance" center=(983,581)
[3] Button "Displays" center=(1676,541)
[4] CheckBox "Bluetooth" center=(1427,812)

Instruction: tap bluetooth

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1070,918)
[1] Button "Shuffle" center=(1063,169)
[2] Button "Pause" center=(1688,77)
[3] Button "Previous" center=(463,115)
[4] Button "Fullscreen" center=(1268,416)

Instruction: start playback

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(931,833)
[1] Button "Terminal" center=(1347,808)
[2] Button "Format" center=(341,500)
[3] Button "Run" center=(1811,644)
[4] TextField "Search files" center=(1658,816)
[5] Button "Save" center=(1574,183)

Instruction: tap format

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(800,943)
[1] Button "Displays" center=(459,514)
[2] Button "Privacy & Security" center=(307,465)
[3] Button "Sound" center=(145,291)
[4] PopUpButton "Appearance" center=(338,958)
[5] Button "Network" center=(1350,984)
[6] CheckBox "Bluetooth" center=(247,342)
[7] CheckBox "Wi-Fi" center=(806,94)

Instruction: tap bluetooth

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1077,209)
[1] SearchField "Search or enter address" center=(62,143)
[2] Button "Go forward" center=(338,802)
[3] Button "Reload" center=(1496,615)
[4] Link "Docs" center=(592,101)
[5] Button "Bookmark" center=(90,561)
[6] Button "Go back" center=(487,685)
[7] Link "Pricing" center=(1107,420)
[8] Link "Sign in" center=(89,499)

Instruction: select pricing

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1593,539)
[1] Button "Wishlist" center=(927,821)
[2] SearchField "Search products" center=(1092,77)
[3] Button "Add to cart" center=(1404,838)
[4] Link "Account" center=(1162,786)
[5] Button "Filters" center=(1573,619)

Instruction: tap wishlist

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(230,671)
[1] Link "Forgot password" center=(826,322)
[2] TextField "Email" center=(847,265)
[3] Button "Sign in" center=(125,85)
[4] Button "Create account" center=(660,52)

Instruction: i forgot my password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1798,78)
[1] Button "Format" center=(583,784)
[2] Button "Save" center=(1028,845)
[3] Button "Open" center=(175,264)
[4] TextField "Search files" center=(999,977)
[5] Button "Terminal" center=(379,987)

Instruction: execute

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1118,199)
[1] CheckBox "Remember me" center=(1407,738)
[2] TextField "Password" center=(1750,773)
[3] Button "Create account" center=(1247,572)
[4] Button "Sign in" center=(1437,325)
[5] Link "Forgot password" center=(1691,648)

Instruction: press remember me

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(258,883)
[1] Button "Go forward" center=(343,608)
[2] Button "Reload" center=(497,904)
[3] Button "Go back" center=(1644,264)
[4] Button "Extensions" center=(1843,1019)
[5] Link "Sign in" center=(549,515)
[6] Button "Bookmark" center=(353,988)
[7] Button "Run inference" center=(241,754)
[8] Button "Downloads" center=(1150,557)

Instruction: tap bookmark

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(175,833)
[1] Button "Wishlist" center=(682,988)
[2] SearchField "Search products" center=(1861,98)
[3] Button "Buy now" center=(932,455)
[4] Button "Add to cart" center=(1709,482)
[5] Button "Filters" center=(355,464)
[6] Button "Checkout" center=(1009,71)

Instruction: search for something

[Assistant] {"action":"type","id":2,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1290,863)
[1] Button "Format" center=(377,384)
[2] Button "Save" center=(366,652)
[3] TextField "Search files" center=(709,647)
[4] Button "Run" center=(811,913)
[5] Button "Open" center=(209,143)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(106,176)
[1] Button "Displays" center=(636,903)
[2] Slider "Brightness" center=(791,735)
[3] Button "Sound" center=(1444,980)
[4] CheckBox "Bluetooth" center=(1733,756)
[5] CheckBox "Wi-Fi" center=(771,722)

Instruction: press do not disturb

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1298,40)
[1] Button "Create account" center=(1722,984)
[2] CheckBox "Remember me" center=(433,848)
[3] Link "Forgot password" center=(388,958)
[4] TextField "Email" center=(840,883)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(173,513)
[1] TextField "Password" center=(1281,912)
[2] CheckBox "Remember me" center=(572,201)
[3] TextField "Email" center=(1357,834)
[4] Button "Sign in" center=(1473,567)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(667,340)
[1] Button "Sound" center=(1378,762)
[2] PopUpButton "Appearance" center=(1708,450)
[3] Button "Privacy & Security" center=(896,866)
[4] CheckBox "Bluetooth" center=(521,853)
[5] CheckBox "Do Not Disturb" center=(1774,356)
[6] Button "Network" center=(1404,374)
[7] CheckBox "Wi-Fi" center=(471,286)
[8] Button "Displays" center=(984,192)

Instruction: click brightness

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(1511,533)
[1] Button "Bookmark" center=(931,459)
[2] Button "Downloads" center=(291,992)
[3] Button "Extensions" center=(1359,724)
[4] Button "Go back" center=(1415,605)
[5] Link "Docs" center=(1533,641)
[6] Link "Sign in" center=(659,999)
[7] Link "Pricing" center=(1492,254)

Instruction: select new tab

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1704,549)
[1] PopUpButton "Appearance" center=(989,714)
[2] CheckBox "Bluetooth" center=(854,842)
[3] Button "Displays" center=(1492,923)
[4] Button "Sound" center=(702,500)
[5] CheckBox "Wi-Fi" center=(312,875)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(230,538)
[1] SearchField "Search products" center=(1833,382)
[2] Button "Wishlist" center=(343,83)
[3] Button "Add to cart" center=(1850,183)
[4] Button "Buy now" center=(531,565)
[5] Link "Account" center=(1276,547)

Instruction: type into search products

[Assistant] {"action":"type","id":1,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1028,830)
[1] Button "Terminal" center=(1728,1032)
[2] Button "Open" center=(1066,246)
[3] Button "Find" center=(472,191)
[4] Button "Commit" center=(720,736)

Instruction: make a commit

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1512,645)
[1] Button "Add to cart" center=(1160,337)
[2] Button "Wishlist" center=(1143,813)
[3] Button "Buy now" center=(974,85)
[4] Link "Account" center=(622,380)
[5] Button "Filters" center=(76,457)

Instruction: go to checkout

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1712,274)
[1] Button "New File" center=(1408,966)
[2] Button "Terminal" center=(1226,300)
[3] TextField "Search files" center=(848,311)
[4] Button "Open" center=(48,583)

Instruction: tap new file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1067,929)
[1] CheckBox "Bluetooth" center=(1540,853)
[2] PopUpButton "Appearance" center=(1610,695)
[3] CheckBox "Do Not Disturb" center=(1653,353)
[4] Button "Sound" center=(1614,654)

Instruction: press do not disturb

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1526,376)
[1] Button "Buy now" center=(402,598)
[2] Button "Wishlist" center=(1383,245)
[3] Button "Add to cart" center=(69,741)
[4] Button "Checkout" center=(919,121)
[5] SearchField "Search products" center=(713,702)
[6] Link "Account" center=(542,1009)

Instruction: press filters

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(371,967)
[1] Slider "Volume" center=(913,248)
[2] Button "Previous" center=(1533,207)
[3] Button "Fullscreen" center=(780,139)
[4] Button "Pause" center=(1606,994)
[5] Button "Next" center=(631,356)

Instruction: skip to next track

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1064,824)
[1] Button "Downloads" center=(1750,838)
[2] Button "New tab" center=(1134,516)
[3] Button "Run inference" center=(1086,165)
[4] Link "Pricing" center=(1025,724)
[5] Button "Extensions" center=(1298,324)

Instruction: click new tab

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1310,863)
[1] Link "Forgot password" center=(1664,824)
[2] Button "Sign in" center=(216,104)
[3] CheckBox "Remember me" center=(372,189)
[4] Button "Create account" center=(740,204)

Instruction: sign up

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(1749,712)
[1] Button "Go back" center=(1639,546)
[2] Button "Reload" center=(1181,829)
[3] Button "Bookmark" center=(977,436)
[4] Button "Extensions" center=(699,697)
[5] Link "Pricing" center=(795,243)
[6] Link "Sign in" center=(125,77)
[7] Button "Run inference" center=(967,232)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(1009,546)
[1] Link "Sign in" center=(1173,604)
[2] Button "New tab" center=(1526,819)
[3] Button "Go forward" center=(737,162)
[4] Link "Pricing" center=(1319,870)
[5] Button "Bookmark" center=(106,845)
[6] Button "Reload" center=(1750,693)

Instruction: tap new tab

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1710,428)
[1] Button "Create account" center=(353,327)
[2] TextField "Password" center=(736,910)
[3] Link "Forgot password" center=(883,116)
[4] CheckBox "Remember me" center=(1127,412)
[5] TextField "Email" center=(193,550)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(272,137)
[1] Button "Format" center=(581,826)
[2] Button "Run" center=(723,365)
[3] Button "Terminal" center=(1859,166)
[4] Button "Open" center=(414,630)
[5] Button "Commit" center=(474,636)
[6] Button "New File" center=(1679,120)

Instruction: run the code

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1873,453)
[1] Button "Pause" center=(1580,993)
[2] Button "Previous" center=(123,359)
[3] Button "Next" center=(1838,470)
[4] Button "Shuffle" center=(230,713)

Instruction: play

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1149,149)
[1] Button "Checkout" center=(581,202)
[2] Button "Buy now" center=(1250,1002)
[3] Link "Account" center=(1189,956)
[4] Button "Wishlist" center=(1666,293)
[5] Button "Filters" center=(207,382)

Instruction: click filters

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1551,1020)
[1] Button "Shuffle" center=(1723,924)
[2] Button "Next" center=(1437,561)
[3] Button "Pause" center=(528,457)
[4] Button "Previous" center=(50,737)

Instruction: skip to next track

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(318,454)
[1] Button "Find" center=(1249,461)
[2] Button "Commit" center=(1224,334)
[3] Button "Format" center=(591,295)
[4] Button "Open" center=(1302,916)
[5] Button "Terminal" center=(1861,167)
[6] Button "Save" center=(1487,470)

Instruction: tap open

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(311,768)
[1] Button "Go back" center=(122,276)
[2] Button "Reload" center=(551,802)
[3] SearchField "Search or enter address" center=(135,788)
[4] Link "Pricing" center=(187,740)
[5] Button "Extensions" center=(848,176)
[6] Button "Downloads" center=(486,323)
[7] Button "New tab" center=(270,628)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(955,187)
[1] Button "Pause" center=(341,781)
[2] Slider "Volume" center=(1367,844)
[3] Button "Previous" center=(292,929)
[4] Button "Fullscreen" center=(795,372)

Instruction: select previous

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1041,1011)
[1] Button "Save" center=(600,487)
[2] Button "Find" center=(140,298)
[3] Button "Commit" center=(46,483)
[4] Button "Terminal" center=(989,202)
[5] TextField "Search files" center=(165,840)
[6] Button "Open" center=(1860,893)
[7] Button "New File" center=(162,806)

Instruction: press terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1649,820)
[1] Button "Save" center=(1124,106)
[2] Button "Run" center=(1241,505)
[3] Button "Commit" center=(752,350)
[4] Button "Format" center=(1359,269)
[5] Button "New File" center=(960,245)
[6] TextField "Search files" center=(1449,924)
[7] Button "Find" center=(604,899)

Instruction: type into search files

[Assistant] {"action":"type","id":6,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1639,280)
[1] TextField "Email" center=(1361,537)
[2] Link "Forgot password" center=(827,264)
[3] CheckBox "Remember me" center=(1169,561)
[4] Button "Create account" center=(1354,996)

Instruction: select forgot password

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(168,620)
[1] Button "Pause" center=(1852,149)
[2] Button "Previous" center=(534,960)
[3] Button "Shuffle" center=(619,298)
[4] Slider "Volume" center=(1698,290)

Instruction: press volume

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1616,607)
[1] Button "Fullscreen" center=(899,621)
[2] Button "Shuffle" center=(738,41)
[3] Button "Pause" center=(1025,238)
[4] Button "Play" center=(423,151)
[5] Slider "Volume" center=(1540,353)
[6] Button "Previous" center=(1516,822)

Instruction: select volume

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1319,240)
[1] Button "Filters" center=(1863,290)
[2] Button "Add to cart" center=(705,86)
[3] Button "Checkout" center=(1796,384)
[4] Button "Wishlist" center=(770,522)
[5] Link "Account" center=(1594,286)

Instruction: proceed to payment

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1434,1039)
[1] Button "Downloads" center=(1163,491)
[2] Button "New tab" center=(1511,178)
[3] Link "Docs" center=(492,795)
[4] Button "Run inference" center=(479,263)
[5] Button "Go back" center=(799,139)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(524,666)
[1] SearchField "Search products" center=(283,135)
[2] Link "Account" center=(1689,296)
[3] Button "Filters" center=(1660,652)
[4] Button "Checkout" center=(1139,804)
[5] Button "Add to cart" center=(1328,1000)

Instruction: tap buy now

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1857,378)
[1] Button "Previous" center=(244,130)
[2] Button "Shuffle" center=(1071,636)
[3] Button "Play" center=(509,987)
[4] Button "Pause" center=(1218,324)
[5] Slider "Volume" center=(40,477)

Instruction: tap shuffle

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(603,237)
[1] Button "Wishlist" center=(392,882)
[2] Button "Checkout" center=(361,279)
[3] SearchField "Search products" center=(1550,86)
[4] Button "Add to cart" center=(1540,223)
[5] Button "Buy now" center=(441,887)
[6] Link "Account" center=(468,242)

Instruction: find a product

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(355,744)
[1] Button "Displays" center=(169,893)
[2] Button "Network" center=(365,525)
[3] PopUpButton "Appearance" center=(814,47)
[4] CheckBox "Bluetooth" center=(390,423)
[5] Button "Sound" center=(111,585)
[6] Slider "Brightness" center=(398,740)
[7] CheckBox "Wi-Fi" center=(1766,954)

Instruction: press displays

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1501,572)
[1] Button "Sound" center=(1501,608)
[2] Slider "Brightness" center=(1850,733)
[3] PopUpButton "Appearance" center=(1127,1012)
[4] Button "Privacy & Security" center=(1251,960)
[5] CheckBox "Wi-Fi" center=(1239,377)
[6] Button "Network" center=(1855,503)

Instruction: open privacy settings

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(441,455)
[1] Button "Save" center=(645,424)
[2] Button "Open" center=(1622,588)
[3] TextField "Search files" center=(992,989)
[4] Button "Terminal" center=(226,935)

Instruction: press save

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1100,224)
[1] Button "Fullscreen" center=(1143,936)
[2] Button "Pause" center=(514,858)
[3] Button "Previous" center=(298,286)
[4] Button "Play" center=(143,560)

Instruction: press play

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1164,179)
[1] Button "Filters" center=(1735,222)
[2] Button "Buy now" center=(130,477)
[3] Link "Account" center=(1044,910)
[4] Button "Checkout" center=(1756,250)
[5] Button "Add to cart" center=(326,862)

Instruction: add this to my cart

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(377,737)
[1] Button "Buy now" center=(605,907)
[2] Button "Checkout" center=(731,70)
[3] Link "Account" center=(107,823)
[4] Button "Filters" center=(63,838)
[5] Button "Add to cart" center=(95,833)

Instruction: type into search products

[Assistant] {"action":"type","id":0,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1448,234)
[1] Button "Shuffle" center=(1024,1015)
[2] Slider "Volume" center=(1689,140)
[3] Button "Pause" center=(939,842)
[4] Button "Next" center=(1871,591)

Instruction: click previous

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1362,336)
[1] Button "Play" center=(1581,339)
[2] Slider "Volume" center=(574,203)
[3] Button "Previous" center=(214,598)
[4] Button "Shuffle" center=(1704,191)
[5] Button "Next" center=(1269,724)
[6] Button "Pause" center=(1220,592)

Instruction: click previous

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1078,485)
[1] Button "Network" center=(997,66)
[2] CheckBox "Bluetooth" center=(130,653)
[3] PopUpButton "Appearance" center=(140,42)
[4] CheckBox "Wi-Fi" center=(1395,307)

Instruction: select appearance

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(813,616)
[1] Button "Shuffle" center=(291,281)
[2] Button "Previous" center=(885,192)
[3] Slider "Volume" center=(1673,315)
[4] Button "Fullscreen" center=(1087,273)
[5] Button "Play" center=(698,418)

Instruction: select previous

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(532,460)
[1] CheckBox "Bluetooth" center=(1273,916)
[2] PopUpButton "Appearance" center=(1297,611)
[3] Button "Network" center=(982,672)
[4] Slider "Brightness" center=(767,845)
[5] Button "Sound" center=(74,830)

Instruction: press sound

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1123,796)
[1] Button "Displays" center=(50,307)
[2] PopUpButton "Appearance" center=(1321,790)
[3] Button "Network" center=(569,1025)
[4] Button "Sound" center=(1795,699)
[5] Slider "Brightness" center=(403,600)
[6] CheckBox "Do Not Disturb" center=(1010,550)
[7] CheckBox "Wi-Fi" center=(1843,993)
[8] CheckBox "Bluetooth" center=(549,568)

Instruction: click network

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1428,342)
[1] Button "Terminal" center=(1526,214)
[2] Button "New File" center=(1108,625)
[3] Button "Commit" center=(1361,725)
[4] Button "Save" center=(1405,788)
[5] Button "Format" center=(669,457)
[6] Button "Open" center=(649,202)
[7] Button "Run" center=(82,288)
[8] Button "Find" center=(1841,469)

Instruction: select new file

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(208,513)
[1] Button "Fullscreen" center=(1828,81)
[2] Button "Play" center=(1723,439)
[3] Button "Pause" center=(637,1008)
[4] Button "Previous" center=(1333,439)
[5] Slider "Volume" center=(360,403)

Instruction: click play

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(432,1022)
[1] CheckBox "Wi-Fi" center=(1812,973)
[2] Slider "Brightness" center=(1533,63)
[3] Button "Displays" center=(1409,126)
[4] CheckBox "Do Not Disturb" center=(1002,426)
[5] PopUpButton "Appearance" center=(670,461)
[6] Button "Sound" center=(1662,326)

Instruction: click do not disturb

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(993,664)
[1] CheckBox "Do Not Disturb" center=(1743,753)
[2] Button "Network" center=(166,729)
[3] Button "Privacy & Security" center=(1449,388)
[4] Slider "Brightness" center=(116,336)
[5] Button "Displays" center=(1446,694)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(848,785)
[1] Button "Displays" center=(494,866)
[2] Slider "Brightness" center=(329,865)
[3] Button "Sound" center=(249,132)
[4] Button "Privacy & Security" center=(233,768)

Instruction: open privacy settings

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1430,242)
[1] Button "Filters" center=(1048,261)
[2] Button "Checkout" center=(315,454)
[3] Link "Account" center=(1358,370)
[4] Button "Buy now" center=(502,80)
[5] SearchField "Search products" center=(1847,456)
[6] Button "Add to cart" center=(1790,75)

Instruction: tap wishlist

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(61,973)
[1] TextField "Password" center=(418,253)
[2] Button "Sign in" center=(1720,67)
[3] CheckBox "Remember me" center=(1700,70)
[4] TextField "Email" center=(1240,149)

Instruction: click remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1007,692)
[1] Button "Filters" center=(1479,549)
[2] Button "Wishlist" center=(580,786)
[3] Button "Add to cart" center=(620,882)
[4] Link "Account" center=(1260,162)
[5] SearchField "Search products" center=(1524,969)

Instruction: press checkout

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1823,578)
[1] Button "Find" center=(1349,830)
[2] Button "Run" center=(1217,333)
[3] Button "New File" center=(262,592)
[4] Button "Terminal" center=(638,271)

Instruction: tap terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1709,780)
[1] CheckBox "Remember me" center=(362,1004)
[2] Button "Create account" center=(481,661)
[3] TextField "Password" center=(1272,557)
[4] Link "Forgot password" center=(342,115)

Instruction: sign up

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(526,544)
[1] Button "Downloads" center=(849,478)
[2] Button "Extensions" center=(931,1029)
[3] Button "Run inference" center=(1659,801)
[4] Button "Bookmark" center=(134,571)
[5] Link "Docs" center=(1460,138)
[6] Link "Sign in" center=(278,804)
[7] Button "Reload" center=(303,476)
[8] SearchField "Search or enter address" center=(1638,906)
[9] Button "Go back" center=(76,1001)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(696,1002)
[1] Button "Go forward" center=(1566,257)
[2] Link "Pricing" center=(244,1015)
[3] SearchField "Search or enter address" center=(399,826)
[4] Button "Run inference" center=(487,300)
[5] Button "Go back" center=(1878,388)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":3,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1602,614)
[1] Button "Checkout" center=(1359,964)
[2] Link "Account" center=(332,676)
[3] Button "Add to cart" center=(471,577)
[4] Button "Wishlist" center=(1742,310)
[5] Button "Buy now" center=(634,807)

Instruction: press filters

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(526,528)
[1] TextField "Password" center=(1807,723)
[2] CheckBox "Remember me" center=(1166,794)
[3] Button "Create account" center=(1870,581)
[4] TextField "Email" center=(319,457)
[5] Link "Forgot password" center=(134,257)

Instruction: click remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(625,627)
[1] Button "Run inference" center=(412,997)
[2] Button "Bookmark" center=(72,771)
[3] Button "Reload" center=(967,301)
[4] Link "Docs" center=(1233,866)
[5] Link "Sign in" center=(379,452)
[6] Button "New tab" center=(1588,652)
[7] Button "Go back" center=(1486,626)
[8] Link "Pricing" center=(1317,1005)

Instruction: select new tab

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1136,842)
[1] Button "Buy now" center=(789,633)
[2] Button "Add to cart" center=(826,692)
[3] Button "Wishlist" center=(1204,294)
[4] Button "Filters" center=(340,378)

Instruction: add to cart

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(642,839)
[1] Button "Checkout" center=(329,834)
[2] Button "Add to cart" center=(920,453)
[3] Button "Filters" center=(1194,613)
[4] Link "Account" center=(152,457)

Instruction: tap wishlist

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1190,1005)
[1] CheckBox "Do Not Disturb" center=(1523,273)
[2] Button "Displays" center=(1104,661)
[3] Button "Sound" center=(491,373)
[4] CheckBox "Wi-Fi" center=(1412,718)
[5] PopUpButton "Appearance" center=(1273,969)
[6] Button "Network" center=(780,705)

Instruction: enable wifi

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(871,644)
[1] SearchField "Search or enter address" center=(1602,407)
[2] Link "Sign in" center=(1155,732)
[3] Button "Bookmark" center=(340,484)
[4] Button "Extensions" center=(1393,312)
[5] Button "Run inference" center=(231,842)
[6] Link "Pricing" center=(1167,431)

Instruction: log in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(729,985)
[1] Button "Sound" center=(1576,163)
[2] Button "Displays" center=(555,903)
[3] CheckBox "Bluetooth" center=(433,835)
[4] PopUpButton "Appearance" center=(586,448)
[5] Button "Network" center=(1148,192)
[6] Button "Privacy & Security" center=(532,641)
[7] Slider "Brightness" center=(1670,831)
[8] CheckBox "Wi-Fi" center=(676,910)

Instruction: click do not disturb

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(168,96)
[1] CheckBox "Bluetooth" center=(279,893)
[2] CheckBox "Do Not Disturb" center=(1603,436)
[3] PopUpButton "Appearance" center=(1071,246)
[4] Button "Displays" center=(193,123)
[5] Button "Privacy & Security" center=(1408,237)
[6] CheckBox "Wi-Fi" center=(1707,666)
[7] Button "Network" center=(725,883)

Instruction: click sound

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(968,457)
[1] Link "Pricing" center=(1624,221)
[2] Button "Extensions" center=(1807,684)
[3] Button "Run inference" center=(1108,240)
[4] SearchField "Search or enter address" center=(1825,636)

Instruction: press run inference

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(372,99)
[1] Button "Go forward" center=(1502,1005)
[2] Button "Reload" center=(959,946)
[3] Link "Pricing" center=(326,783)
[4] SearchField "Search or enter address" center=(865,821)
[5] Button "Run inference" center=(1807,461)
[6] Button "Go back" center=(1415,173)
[7] Link "Sign in" center=(1261,180)
[8] Button "New tab" center=(745,59)
[9] Button "Bookmark" center=(1177,857)
[10] Button "Downloads" center=(228,803)

Instruction: tap downloads

[Assistant] {"action":"click","id":10}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1113,730)
[1] Button "Run" center=(1132,309)
[2] Button "Open" center=(177,1033)
[3] TextField "Search files" center=(439,243)
[4] Button "Commit" center=(455,504)
[5] Button "Format" center=(515,131)
[6] Button "Find" center=(311,1017)
[7] Button "New File" center=(683,406)

Instruction: press open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(61,249)
[1] Link "Forgot password" center=(1779,960)
[2] Button "Sign in" center=(376,734)
[3] CheckBox "Remember me" center=(248,120)
[4] TextField "Email" center=(1274,207)
[5] Button "Create account" center=(40,428)

Instruction: press remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(288,234)
[1] Button "Filters" center=(486,498)
[2] SearchField "Search products" center=(192,952)
[3] Button "Wishlist" center=(1407,598)
[4] Button "Buy now" center=(647,354)
[5] Button "Checkout" center=(581,646)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(923,322)
[1] Link "Account" center=(623,94)
[2] SearchField "Search products" center=(524,502)
[3] Button "Wishlist" center=(459,738)
[4] Button "Buy now" center=(552,891)

Instruction: click wishlist

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1348,134)
[1] CheckBox "Do Not Disturb" center=(1838,543)
[2] CheckBox "Wi-Fi" center=(1871,880)
[3] Button "Privacy & Security" center=(1246,589)
[4] CheckBox "Bluetooth" center=(1734,226)
[5] Button "Displays" center=(249,683)

Instruction: tap bluetooth

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1831,882)
[1] CheckBox "Bluetooth" center=(1144,288)
[2] CheckBox "Wi-Fi" center=(1835,157)
[3] Button "Privacy & Security" center=(1776,567)
[4] Button "Displays" center=(402,596)
[5] Button "Network" center=(1435,366)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1019,756)
[1] Button "Fullscreen" center=(165,452)
[2] Button "Pause" center=(1803,1000)
[3] Button "Shuffle" center=(1056,281)
[4] Slider "Volume" center=(869,570)
[5] Button "Previous" center=(1637,866)

Instruction: skip to next track

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1420,203)
[1] Button "Run" center=(216,442)
[2] Button "Commit" center=(1430,934)
[3] Button "Find" center=(1390,414)
[4] Button "Open" center=(70,835)
[5] Button "Format" center=(726,592)
[6] Button "Save" center=(956,999)

Instruction: commit my changes

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1202,447)
[1] Button "Previous" center=(975,813)
[2] Button "Next" center=(1256,1017)
[3] Button "Fullscreen" center=(120,610)
[4] Button "Pause" center=(1249,587)
[5] Slider "Volume" center=(1622,738)

Instruction: make it full screen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1279,748)
[1] Button "Wishlist" center=(1534,275)
[2] Link "Account" center=(1396,901)
[3] SearchField "Search products" center=(1150,483)
[4] Button "Buy now" center=(1512,652)

Instruction: select account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(212,330)
[1] Button "Play" center=(750,502)
[2] Button "Previous" center=(1322,598)
[3] Button "Shuffle" center=(1079,508)
[4] Slider "Volume" center=(1699,809)
[5] Button "Next" center=(113,751)
[6] Button "Pause" center=(1381,491)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1300,502)
[1] CheckBox "Wi-Fi" center=(602,429)
[2] CheckBox "Do Not Disturb" center=(1637,119)
[3] CheckBox "Bluetooth" center=(1297,105)
[4] Button "Sound" center=(1513,682)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1754,894)
[1] Button "Open" center=(305,367)
[2] Button "Terminal" center=(1097,630)
[3] Button "Run" center=(441,545)
[4] Button "Format" center=(1575,79)
[5] Button "Find" center=(1659,698)

Instruction: search

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1398,956)
[1] Slider "Volume" center=(1817,409)
[2] Button "Play" center=(1587,439)
[3] Button "Next" center=(820,785)
[4] Button "Fullscreen" center=(1308,379)

Instruction: go fullscreen

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1112,442)
[1] Button "Network" center=(983,647)
[2] CheckBox "Bluetooth" center=(847,240)
[3] PopUpButton "Appearance" center=(327,401)
[4] Slider "Brightness" center=(1732,874)
[5] Button "Sound" center=(1292,341)

Instruction: press network

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1158,1023)
[1] Button "Play" center=(1212,996)
[2] Button "Pause" center=(251,388)
[3] Button "Next" center=(869,208)
[4] Button "Fullscreen" center=(573,121)

Instruction: press pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1218,215)
[1] Link "Account" center=(866,1008)
[2] SearchField "Search products" center=(1246,266)
[3] Button "Buy now" center=(1818,751)
[4] Button "Wishlist" center=(495,391)

Instruction: search for something

[Assistant] {"action":"type","id":2,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(578,1019)
[1] Button "Buy now" center=(831,1012)
[2] Button "Add to cart" center=(1781,612)
[3] SearchField "Search products" center=(704,56)
[4] Button "Checkout" center=(1289,368)
[5] Button "Wishlist" center=(1588,539)
[6] Link "Account" center=(1623,736)

Instruction: add this to my cart

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1025,653)
[1] Button "Create account" center=(1031,208)
[2] Button "Sign in" center=(136,243)
[3] TextField "Email" center=(1108,893)
[4] Link "Forgot password" center=(1483,381)

Instruction: click sign in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(969,150)
[1] CheckBox "Remember me" center=(709,887)
[2] TextField "Email" center=(711,247)
[3] Button "Sign in" center=(259,261)
[4] Button "Create account" center=(1532,217)
[5] TextField "Password" center=(789,674)

Instruction: sign in

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1616,851)
[1] SearchField "Search or enter address" center=(218,76)
[2] Link "Sign in" center=(1629,402)
[3] Button "Go forward" center=(403,507)
[4] Button "Reload" center=(824,670)
[5] Button "New tab" center=(1123,1010)
[6] Button "Run inference" center=(289,752)
[7] Button "Go back" center=(1251,576)
[8] Button "Downloads" center=(1008,614)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(717,610)
[1] TextField "Search files" center=(1697,974)
[2] Button "Terminal" center=(1697,834)
[3] Button "Commit" center=(916,466)
[4] Button "Format" center=(159,1018)
[5] Button "Open" center=(704,924)

Instruction: tap format

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(730,798)
[1] Button "Format" center=(1869,725)
[2] Button "Run" center=(612,594)
[3] Button "Find" center=(1334,387)
[4] Button "Open" center=(1540,642)
[5] Button "Commit" center=(904,1027)
[6] Button "New File" center=(784,855)
[7] Button "Terminal" center=(1815,364)
[8] TextField "Search files" center=(1653,769)

Instruction: press open

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(92,179)
[1] Button "Filters" center=(1782,46)
[2] Button "Wishlist" center=(1847,80)
[3] Button "Add to cart" center=(127,199)
[4] SearchField "Search products" center=(1681,354)
[5] Button "Checkout" center=(802,694)

Instruction: click checkout

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(736,532)
[1] Button "Displays" center=(109,574)
[2] Button "Network" center=(457,688)
[3] CheckBox "Wi-Fi" center=(1336,475)
[4] Button "Sound" center=(1177,795)
[5] PopUpButton "Appearance" center=(413,738)
[6] Button "Privacy & Security" center=(245,330)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1357,463)
[1] TextField "Search files" center=(346,804)
[2] Button "Find" center=(53,172)
[3] Button "Commit" center=(790,906)
[4] Button "Open" center=(1095,796)
[5] Button "Format" center=(1481,211)
[6] Button "Run" center=(795,334)

Instruction: save the file

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1282,186)
[1] Button "Next" center=(1839,901)
[2] Button "Shuffle" center=(1361,47)
[3] Button "Fullscreen" center=(1323,352)
[4] Button "Pause" center=(1318,134)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(1610,522)
[1] Button "Go back" center=(867,571)
[2] Button "Downloads" center=(1486,129)
[3] SearchField "Search or enter address" center=(1402,506)
[4] Button "Extensions" center=(402,845)
[5] Button "Run inference" center=(410,961)
[6] Button "Go forward" center=(756,531)

Instruction: select downloads

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(106,373)
[1] Button "Add to cart" center=(331,673)
[2] SearchField "Search products" center=(224,563)
[3] Button "Buy now" center=(1712,834)
[4] Button "Wishlist" center=(1790,725)
[5] Button "Checkout" center=(1559,504)
[6] Link "Account" center=(823,961)

Instruction: tap add to cart

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1196,711)
[1] Button "New tab" center=(1233,1029)
[2] Button "Downloads" center=(421,1019)
[3] Link "Docs" center=(1755,988)
[4] Button "Bookmark" center=(99,373)
[5] Button "Go back" center=(412,216)
[6] Button "Extensions" center=(357,630)
[7] Button "Run inference" center=(1200,971)

Instruction: press extensions

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1377,412)
[1] Button "Reload" center=(1478,1027)
[2] Button "Go back" center=(540,848)
[3] SearchField "Search or enter address" center=(1222,916)
[4] Button "Run inference" center=(1871,593)
[5] Link "Pricing" center=(456,367)
[6] Link "Sign in" center=(1178,1002)
[7] Button "New tab" center=(670,313)
[8] Button "Go forward" center=(895,165)
[9] Button "Downloads" center=(1041,782)

Instruction: press new tab

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1700,1015)
[1] Button "Network" center=(1627,550)
[2] PopUpButton "Appearance" center=(1291,772)
[3] CheckBox "Do Not Disturb" center=(1524,159)
[4] Slider "Brightness" center=(409,954)
[5] CheckBox "Wi-Fi" center=(955,658)
[6] CheckBox "Bluetooth" center=(1729,908)
[7] Button "Displays" center=(1802,636)
[8] Button "Privacy & Security" center=(938,42)

Instruction: tap appearance

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(323,633)
[1] Button "Go forward" center=(1183,1008)
[2] Button "Run inference" center=(1102,288)
[3] SearchField "Search or enter address" center=(307,192)
[4] Link "Sign in" center=(97,871)
[5] Button "New tab" center=(1422,810)
[6] Link "Docs" center=(241,177)
[7] Link "Pricing" center=(1664,413)
[8] Button "Bookmark" center=(853,786)
[9] Button "Downloads" center=(1659,783)

Instruction: press bookmark

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1440,255)
[1] CheckBox "Bluetooth" center=(1459,797)
[2] Button "Privacy & Security" center=(1518,761)
[3] Button "Sound" center=(1839,421)
[4] Button "Network" center=(1406,566)
[5] Button "Displays" center=(771,229)
[6] CheckBox "Do Not Disturb" center=(524,692)

Instruction: click brightness

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(982,108)
[1] Button "Sound" center=(750,895)
[2] Button "Privacy & Security" center=(1825,206)
[3] PopUpButton "Appearance" center=(470,932)
[4] Button "Network" center=(1530,624)

Instruction: click brightness

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1378,928)
[1] CheckBox "Bluetooth" center=(1545,563)
[2] Button "Displays" center=(434,98)
[3] PopUpButton "Appearance" center=(1293,883)
[4] Button "Privacy & Security" center=(442,973)
[5] CheckBox "Wi-Fi" center=(50,630)
[6] Slider "Brightness" center=(891,521)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(464,201)
[1] Button "Find" center=(281,927)
[2] Button "New File" center=(1446,986)
[3] Button "Commit" center=(1474,1023)
[4] Button "Save" center=(982,459)

Instruction: find in file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(1395,800)
[1] Button "Bookmark" center=(1123,818)
[2] Button "Run inference" center=(936,994)
[3] Button "New tab" center=(549,242)
[4] Button "Downloads" center=(1315,804)
[5] Button "Go back" center=(1534,259)
[6] Link "Sign in" center=(115,340)
[7] Button "Reload" center=(1211,237)
[8] Button "Extensions" center=(923,429)
[9] SearchField "Search or enter address" center=(730,253)

Instruction: tap run inference

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(245,132)
[1] Button "Pause" center=(622,63)
[2] Button "Previous" center=(1657,611)
[3] Button "Play" center=(1438,444)
[4] Slider "Volume" center=(1320,577)

Instruction: click previous

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(303,57)
[1] Button "Wishlist" center=(1760,882)
[2] Button "Filters" center=(1251,238)
[3] SearchField "Search products" center=(1474,907)
[4] Button "Buy now" center=(791,1030)
[5] Button "Checkout" center=(756,778)

Instruction: click add to cart

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(969,161)
[1] Slider "Brightness" center=(330,322)
[2] CheckBox "Wi-Fi" center=(430,474)
[3] Button "Network" center=(775,602)
[4] Button "Displays" center=(708,592)
[5] PopUpButton "Appearance" center=(790,436)

Instruction: press brightness

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(512,610)
[1] Button "Extensions" center=(1053,324)
[2] SearchField "Search or enter address" center=(163,210)
[3] Button "Downloads" center=(1266,674)
[4] Link "Docs" center=(165,493)
[5] Link "Sign in" center=(1512,778)
[6] Button "Bookmark" center=(214,481)
[7] Button "Run inference" center=(912,922)
[8] Button "Go back" center=(1249,62)
[9] Button "Reload" center=(884,273)
[10] Button "Go forward" center=(306,169)
[11] Button "New tab" center=(1801,328)

Instruction: select extensions

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(551,100)
[1] Link "Forgot password" center=(295,57)
[2] Button "Create account" center=(1301,411)
[3] TextField "Email" center=(870,1019)
[4] TextField "Password" center=(1056,617)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(801,331)
[1] Link "Forgot password" center=(910,903)
[2] TextField "Password" center=(393,1008)
[3] TextField "Email" center=(544,297)
[4] Button "Sign in" center=(1022,901)
[5] Button "Create account" center=(257,180)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(464,67)
[1] Button "New File" center=(1690,973)
[2] Button "Run" center=(1548,851)
[3] Button "Format" center=(884,607)
[4] Button "Find" center=(510,142)

Instruction: click run

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(957,149)
[1] Button "Run" center=(343,646)
[2] TextField "Search files" center=(102,929)
[3] Button "Format" center=(1514,150)
[4] Button "Terminal" center=(741,761)
[5] Button "Save" center=(164,514)
[6] Button "Find" center=(212,240)

Instruction: tap format

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(654,244)
[1] Button "Go forward" center=(1680,450)
[2] Link "Sign in" center=(398,156)
[3] Button "New tab" center=(1481,241)
[4] Button "Reload" center=(90,57)
[5] Link "Docs" center=(1197,485)
[6] SearchField "Search or enter address" center=(412,397)
[7] Button "Downloads" center=(1128,920)
[8] Button "Extensions" center=(1121,732)
[9] Link "Pricing" center=(767,497)
[10] Button "Go back" center=(912,760)
[11] Button "Bookmark" center=(919,650)

Instruction: press run inference

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(934,882)
[1] Button "Reload" center=(344,120)
[2] Link "Docs" center=(763,639)
[3] Button "Run inference" center=(171,577)
[4] Button "Go forward" center=(532,890)
[5] Link "Pricing" center=(323,756)
[6] Button "New tab" center=(1788,526)
[7] Button "Extensions" center=(78,72)
[8] Button "Bookmark" center=(157,312)
[9] Button "Downloads" center=(865,464)
[10] Button "Go back" center=(911,142)

Instruction: click pricing

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1153,826)
[1] Button "Extensions" center=(1170,109)
[2] Button "Downloads" center=(1720,772)
[3] Button "New tab" center=(1192,815)
[4] Link "Docs" center=(1473,918)
[5] Button "Reload" center=(1488,686)
[6] SearchField "Search or enter address" center=(1466,316)
[7] Link "Sign in" center=(1215,43)

Instruction: select extensions

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1722,548)
[1] CheckBox "Bluetooth" center=(848,842)
[2] CheckBox "Do Not Disturb" center=(1281,581)
[3] PopUpButton "Appearance" center=(1049,694)
[4] Button "Sound" center=(498,703)
[5] Button "Privacy & Security" center=(173,243)
[6] Slider "Brightness" center=(988,644)
[7] Button "Displays" center=(939,741)
[8] CheckBox "Wi-Fi" center=(286,873)

Instruction: click wi-fi

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1209,312)
[1] Button "Format" center=(548,278)
[2] Button "Save" center=(1741,218)
[3] Button "New File" center=(1600,370)
[4] Button "Run" center=(934,227)
[5] TextField "Search files" center=(471,794)
[6] Button "Terminal" center=(1450,1030)
[7] Button "Open" center=(1095,725)

Instruction: run the code

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(325,66)
[1] Button "Buy now" center=(478,518)
[2] SearchField "Search products" center=(144,567)
[3] Button "Checkout" center=(983,879)
[4] Link "Account" center=(1147,904)
[5] Button "Add to cart" center=(391,148)
[6] Button "Wishlist" center=(222,460)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(796,740)
[1] Button "Previous" center=(1477,808)
[2] Slider "Volume" center=(1299,625)
[3] Button "Next" center=(1090,649)
[4] Button "Play" center=(1073,946)

Instruction: play

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(662,587)
[1] Button "Extensions" center=(323,204)
[2] Button "Reload" center=(1592,163)
[3] Link "Docs" center=(964,1009)
[4] Button "Run inference" center=(1842,944)
[5] SearchField "Search or enter address" center=(769,574)
[6] Button "Downloads" center=(1090,795)
[7] Button "New tab" center=(517,624)
[8] Link "Sign in" center=(941,1035)

Instruction: click new tab

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1011,367)
[1] SearchField "Search products" center=(1009,722)
[2] Button "Wishlist" center=(231,888)
[3] Button "Checkout" center=(1765,224)
[4] Link "Account" center=(1764,233)
[5] Button "Add to cart" center=(818,616)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1073,194)
[1] Button "Commit" center=(988,783)
[2] Button "Save" center=(1580,441)
[3] Button "New File" center=(1224,1005)
[4] Button "Run" center=(1412,992)
[5] Button "Find" center=(1776,528)
[6] Button "Terminal" center=(1401,690)
[7] TextField "Search files" center=(142,313)
[8] Button "Open" center=(128,642)

Instruction: commit my changes

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1355,294)
[1] Button "Buy now" center=(402,539)
[2] Link "Account" center=(621,730)
[3] Button "Filters" center=(1870,867)
[4] Button "Checkout" center=(1455,606)
[5] Button "Add to cart" center=(1183,269)
[6] Button "Wishlist" center=(722,780)

Instruction: type into search products

[Assistant] {"action":"type","id":0,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(810,432)
[1] Button "Open" center=(1159,294)
[2] TextField "Search files" center=(1515,311)
[3] Button "Run" center=(1292,479)
[4] Button "Save" center=(92,312)
[5] Button "Format" center=(967,563)
[6] Button "Terminal" center=(679,195)
[7] Button "New File" center=(172,494)
[8] Button "Commit" center=(63,196)

Instruction: tap open

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1069,1030)
[1] CheckBox "Wi-Fi" center=(1215,1023)
[2] Button "Privacy & Security" center=(1656,837)
[3] PopUpButton "Appearance" center=(494,312)
[4] Button "Network" center=(1134,449)

Instruction: press network

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1581,579)
[1] Link "Account" center=(951,656)
[2] SearchField "Search products" center=(1219,109)
[3] Button "Checkout" center=(910,62)
[4] Button "Buy now" center=(1316,893)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(516,258)
[1] Link "Sign in" center=(1013,776)
[2] Button "Reload" center=(874,955)
[3] Link "Docs" center=(40,389)
[4] SearchField "Search or enter address" center=(422,294)
[5] Button "New tab" center=(129,980)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":4,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1576,349)
[1] Button "Buy now" center=(1054,188)
[2] Link "Account" center=(576,167)
[3] Button "Filters" center=(1302,139)
[4] Button "Checkout" center=(918,365)
[5] Button "Wishlist" center=(1371,72)

Instruction: select wishlist

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1587,798)
[1] CheckBox "Remember me" center=(1031,958)
[2] TextField "Password" center=(1570,106)
[3] Button "Create account" center=(1621,568)
[4] Link "Forgot password" center=(299,756)
[5] TextField "Email" center=(569,648)

Instruction: i forgot my password

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1776,147)
[1] Button "Format" center=(1652,271)
[2] Button "Open" center=(1759,374)
[3] Button "Run" center=(1314,229)
[4] Button "Commit" center=(847,681)
[5] TextField "Search files" center=(811,1009)
[6] Button "Save" center=(582,599)
[7] Button "New File" center=(904,671)
[8] Button "Terminal" center=(1398,591)

Instruction: execute

[Assistant] {"action":"click","id":3}