[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(425,802)
[1] CheckBox "Bluetooth" center=(277,626)
[2] PopUpButton "Appearance" center=(793,170)
[3] Slider "Brightness" center=(1787,360)
[4] Button "Privacy & Security" center=(818,630)
[5] CheckBox "Do Not Disturb" center=(1549,41)
[6] CheckBox "Wi-Fi" center=(676,103)
[7] Button "Displays" center=(1248,588)
[8] Button "Network" center=(428,907)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1142,912)
[1] Button "Previous" center=(983,363)
[2] Button "Play" center=(336,875)
[3] Button "Shuffle" center=(647,897)
[4] Slider "Volume" center=(488,551)

Instruction: select previous

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1053,338)
[1] CheckBox "Bluetooth" center=(1482,1029)
[2] Button "Network" center=(811,100)
[3] Button "Privacy & Security" center=(1699,381)
[4] PopUpButton "Appearance" center=(352,887)
[5] Button "Displays" center=(1507,255)

Instruction: click bluetooth

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(905,734)
[1] Button "Displays" center=(1009,396)
[2] PopUpButton "Appearance" center=(1591,440)
[3] CheckBox "Wi-Fi" center=(840,113)
[4] CheckBox "Bluetooth" center=(1121,784)
[5] Button "Sound" center=(619,329)
[6] CheckBox "Do Not Disturb" center=(1466,985)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1877,188)
[1] Button "Extensions" center=(469,263)
[2] Link "Docs" center=(1687,806)
[3] Link "Pricing" center=(440,792)
[4] Button "Go forward" center=(1373,49)
[5] SearchField "Search or enter address" center=(781,783)
[6] Button "Bookmark" center=(1793,629)
[7] Button "New tab" center=(248,734)
[8] Button "Go back" center=(1648,919)

Instruction: press bookmark

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(109,161)
[1] Button "Extensions" center=(533,446)
[2] Link "Docs" center=(1040,349)
[3] Button "Bookmark" center=(1448,650)
[4] Link "Sign in" center=(393,708)
[5] Button "New tab" center=(1566,424)
[6] Button "Run inference" center=(1628,748)
[7] Button "Go back" center=(1668,726)

Instruction: tap go back

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(209,164)
[1] Link "Forgot password" center=(1041,696)
[2] TextField "Email" center=(1409,970)
[3] Button "Sign in" center=(1083,270)
[4] Button "Create account" center=(87,640)

Instruction: make a new account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(706,105)
[1] Button "Add to cart" center=(1501,970)
[2] Button "Filters" center=(1805,854)
[3] SearchField "Search products" center=(1140,229)
[4] Link "Account" center=(870,872)

Instruction: find a product

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(291,80)
[1] Link "Pricing" center=(303,628)
[2] Link "Docs" center=(462,708)
[3] Button "New tab" center=(370,129)
[4] SearchField "Search or enter address" center=(1191,302)
[5] Button "Go back" center=(1783,811)
[6] Button "Downloads" center=(111,126)
[7] Button "Bookmark" center=(295,558)
[8] Button "Go forward" center=(651,412)
[9] Button "Extensions" center=(1154,215)

Instruction: refresh the page

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(680,575)
[1] Button "Buy now" center=(761,660)
[2] Button "Add to cart" center=(1142,562)
[3] Button "Wishlist" center=(469,744)
[4] SearchField "Search products" center=(242,718)
[5] Button "Checkout" center=(1718,85)
[6] Button "Filters" center=(525,77)

Instruction: press add to cart

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1166,301)
[1] Slider "Brightness" center=(573,430)
[2] Button "Privacy & Security" center=(1370,965)
[3] CheckBox "Bluetooth" center=(374,449)
[4] Button "Sound" center=(974,125)

Instruction: press bluetooth

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1386,703)
[1] Button "Wishlist" center=(1720,563)
[2] SearchField "Search products" center=(1278,207)
[3] Button "Filters" center=(899,881)
[4] Button "Checkout" center=(354,363)
[5] Button "Add to cart" center=(1653,484)
[6] Button "Buy now" center=(816,66)

Instruction: click wishlist

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(714,695)
[1] TextField "Password" center=(222,411)
[2] Button "Sign in" center=(723,836)
[3] CheckBox "Remember me" center=(599,189)
[4] Link "Forgot password" center=(1819,857)

Instruction: make a new account

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1812,328)
[1] Button "Commit" center=(1169,317)
[2] Button "Save" center=(1284,141)
[3] Button "Run" center=(1876,89)
[4] Button "New File" center=(389,365)
[5] Button "Find" center=(1579,438)
[6] Button "Terminal" center=(410,393)
[7] TextField "Search files" center=(1690,799)
[8] Button "Open" center=(1428,1031)

Instruction: tap open

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(815,621)
[1] Button "Commit" center=(1546,313)
[2] TextField "Search files" center=(148,980)
[3] Button "Open" center=(1525,694)
[4] Button "Terminal" center=(798,764)
[5] Button "Format" center=(1632,295)

Instruction: tap open

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1511,942)
[1] PopUpButton "Appearance" center=(1707,274)
[2] CheckBox "Bluetooth" center=(665,690)
[3] CheckBox "Do Not Disturb" center=(1797,129)
[4] Button "Displays" center=(665,456)
[5] Button "Network" center=(1618,360)
[6] Button "Privacy & Security" center=(1771,132)
[7] Button "Sound" center=(1339,121)
[8] Slider "Brightness" center=(197,787)

Instruction: press appearance

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1081,994)
[1] Button "Buy now" center=(1536,179)
[2] Link "Account" center=(1022,549)
[3] Button "Wishlist" center=(1094,741)
[4] Button "Checkout" center=(1426,949)
[5] Button "Filters" center=(1130,754)
[6] Button "Add to cart" center=(295,455)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(958,1027)
[1] Slider "Volume" center=(385,730)
[2] Button "Fullscreen" center=(1846,822)
[3] Button "Next" center=(1112,965)
[4] Button "Previous" center=(500,219)
[5] Button "Shuffle" center=(193,415)
[6] Button "Play" center=(1759,176)

Instruction: tap pause

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(652,534)
[1] Button "Format" center=(641,311)
[2] Button "Find" center=(476,42)
[3] Button "Open" center=(1551,686)
[4] TextField "Search files" center=(1204,443)
[5] Button "Save" center=(485,985)
[6] Button "Terminal" center=(664,991)

Instruction: save the file

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(259,45)
[1] Button "Commit" center=(756,951)
[2] Button "Open" center=(1519,230)
[3] Button "Run" center=(863,403)
[4] Button "Find" center=(58,143)
[5] TextField "Search files" center=(922,543)
[6] Button "New File" center=(444,126)

Instruction: make a commit

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(262,489)
[1] Button "Go back" center=(1384,899)
[2] Button "Go forward" center=(939,205)
[3] Button "Run inference" center=(326,402)
[4] Button "Downloads" center=(1570,484)
[5] Button "Extensions" center=(51,407)
[6] Link "Sign in" center=(1849,931)
[7] Link "Pricing" center=(1492,243)
[8] Button "Bookmark" center=(266,234)

Instruction: tap go forward

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(902,540)
[1] Button "Find" center=(1166,547)
[2] Button "New File" center=(1678,738)
[3] Button "Run" center=(1502,170)
[4] Button "Commit" center=(705,809)

Instruction: execute

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(146,734)
[1] Button "Sound" center=(1739,390)
[2] Button "Network" center=(919,1039)
[3] Button "Displays" center=(1262,795)
[4] Slider "Brightness" center=(56,484)
[5] CheckBox "Wi-Fi" center=(1023,336)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(446,525)
[1] Button "Displays" center=(1486,841)
[2] Button "Network" center=(1698,145)
[3] Button "Sound" center=(1475,787)
[4] CheckBox "Wi-Fi" center=(1068,665)
[5] Slider "Brightness" center=(411,233)
[6] CheckBox "Bluetooth" center=(1790,645)
[7] PopUpButton "Appearance" center=(1328,689)

Instruction: press displays

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1810,92)
[1] Button "Terminal" center=(818,457)
[2] Button "New File" center=(1115,353)
[3] TextField "Search files" center=(1473,688)
[4] Button "Run" center=(513,267)

Instruction: type into search files

[Assistant] {"action":"type","id":3,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(695,956)
[1] Link "Pricing" center=(137,226)
[2] Button "Bookmark" center=(123,40)
[3] Button "Extensions" center=(146,716)
[4] Button "Downloads" center=(1565,369)
[5] SearchField "Search or enter address" center=(474,332)
[6] Button "Reload" center=(1006,326)
[7] Button "Go forward" center=(1282,143)

Instruction: tap downloads

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(718,409)
[1] Link "Forgot password" center=(1558,869)
[2] Button "Create account" center=(780,800)
[3] TextField "Email" center=(1634,643)
[4] CheckBox "Remember me" center=(1123,659)

Instruction: sign up

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(193,1004)
[1] Button "Bookmark" center=(1159,673)
[2] SearchField "Search or enter address" center=(742,348)
[3] Link "Docs" center=(1525,525)
[4] Link "Sign in" center=(1598,246)
[5] Button "Downloads" center=(1132,778)

Instruction: press bookmark

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(178,534)
[1] Button "Save" center=(898,576)
[2] Button "Open" center=(1071,849)
[3] Button "Format" center=(84,210)
[4] Button "New File" center=(1267,320)
[5] Button "Terminal" center=(1466,1030)
[6] Button "Commit" center=(304,336)
[7] Button "Find" center=(133,1018)

Instruction: tap format

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1283,318)
[1] Button "Commit" center=(463,228)
[2] Button "New File" center=(47,203)
[3] Button "Format" center=(1211,263)
[4] Button "Find" center=(1252,753)
[5] Button "Run" center=(58,940)
[6] Button "Terminal" center=(1659,565)
[7] Button "Open" center=(972,555)

Instruction: find in file

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1132,283)
[1] Link "Forgot password" center=(637,80)
[2] Button "Create account" center=(1349,191)
[3] TextField "Email" center=(1849,324)
[4] Button "Sign in" center=(312,316)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1279,1035)
[1] Button "Shuffle" center=(795,372)
[2] Button "Previous" center=(210,1009)
[3] Button "Play" center=(1484,945)
[4] Slider "Volume" center=(1164,310)
[5] Button "Fullscreen" center=(1518,176)
[6] Button "Pause" center=(876,959)

Instruction: click previous

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(564,1034)
[1] Button "Previous" center=(72,590)
[2] Slider "Volume" center=(1386,928)
[3] Button "Pause" center=(1633,1001)
[4] Button "Next" center=(1638,637)

Instruction: select pause

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(590,248)
[1] Button "Play" center=(1231,684)
[2] Button "Previous" center=(1694,918)
[3] Button "Pause" center=(1581,444)
[4] Button "Fullscreen" center=(358,496)
[5] Slider "Volume" center=(514,787)

Instruction: press previous

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1366,328)
[1] Button "Format" center=(83,357)
[2] Button "Save" center=(373,946)
[3] Button "Find" center=(403,877)
[4] Button "Commit" center=(847,743)
[5] Button "Run" center=(1284,232)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(244,426)
[1] Button "Find" center=(826,61)
[2] Button "New File" center=(265,886)
[3] Button "Open" center=(67,821)
[4] Button "Save" center=(1740,699)
[5] TextField "Search files" center=(129,175)
[6] Button "Format" center=(548,828)
[7] Button "Terminal" center=(211,314)
[8] Button "Run" center=(332,1001)

Instruction: click commit

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(356,289)
[1] Link "Pricing" center=(154,596)
[2] SearchField "Search or enter address" center=(1457,313)
[3] Button "Run inference" center=(1325,596)
[4] Link "Docs" center=(292,61)
[5] Button "New tab" center=(1729,85)
[6] Button "Extensions" center=(250,597)
[7] Button "Reload" center=(647,145)
[8] Button "Bookmark" center=(1087,947)
[9] Button "Go forward" center=(1753,465)
[10] Button "Downloads" center=(330,41)
[11] Link "Sign in" center=(499,64)

Instruction: press extensions

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1047,213)
[1] Button "Create account" center=(265,269)
[2] Link "Forgot password" center=(248,1019)
[3] TextField "Password" center=(617,405)
[4] Button "Sign in" center=(196,670)

Instruction: click forgot password

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(753,936)
[1] Link "Sign in" center=(252,317)
[2] Button "Extensions" center=(1062,717)
[3] Link "Pricing" center=(795,995)
[4] Button "Bookmark" center=(583,870)
[5] Button "Go forward" center=(1496,857)
[6] Button "New tab" center=(196,847)
[7] Button "Downloads" center=(44,922)
[8] Link "Docs" center=(344,836)

Instruction: press go forward

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(225,370)
[1] Button "Open" center=(301,226)
[2] Button "New File" center=(1611,434)
[3] Button "Run" center=(762,366)
[4] Button "Format" center=(1052,474)
[5] Button "Save" center=(1025,750)

Instruction: select format

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(211,634)
[1] CheckBox "Wi-Fi" center=(1504,471)
[2] CheckBox "Do Not Disturb" center=(1119,631)
[3] Button "Sound" center=(1630,750)
[4] Slider "Brightness" center=(1701,806)
[5] Button "Displays" center=(489,759)
[6] PopUpButton "Appearance" center=(218,280)
[7] Button "Privacy & Security" center=(780,42)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(902,288)
[1] Slider "Brightness" center=(471,887)
[2] Button "Displays" center=(499,97)
[3] Button "Sound" center=(1569,181)
[4] PopUpButton "Appearance" center=(280,1025)
[5] Button "Privacy & Security" center=(1464,760)
[6] CheckBox "Do Not Disturb" center=(333,426)
[7] CheckBox "Wi-Fi" center=(1541,66)
[8] CheckBox "Bluetooth" center=(1465,1028)

Instruction: click network

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(329,120)
[1] Button "Next" center=(541,118)
[2] Button "Shuffle" center=(65,622)
[3] Button "Pause" center=(1742,873)
[4] Button "Previous" center=(892,193)
[5] Button "Play" center=(1607,226)

Instruction: select next

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1844,697)
[1] TextField "Search files" center=(105,658)
[2] Button "Find" center=(1578,217)
[3] Button "New File" center=(1070,204)
[4] Button "Save" center=(62,445)

Instruction: select new file

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1108,426)
[1] Button "Filters" center=(296,800)
[2] Button "Add to cart" center=(473,450)
[3] Button "Wishlist" center=(1307,97)
[4] Button "Buy now" center=(488,441)

Instruction: click buy now

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1685,420)
[1] Button "Play" center=(1364,474)
[2] Button "Pause" center=(438,316)
[3] Button "Shuffle" center=(262,146)
[4] Button "Next" center=(199,411)

Instruction: click pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(424,604)
[1] Button "Privacy & Security" center=(453,693)
[2] Slider "Brightness" center=(106,286)
[3] CheckBox "Bluetooth" center=(591,127)
[4] Button "Sound" center=(1207,618)
[5] CheckBox "Wi-Fi" center=(131,617)
[6] PopUpButton "Appearance" center=(1658,535)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1479,730)
[1] Button "Run inference" center=(239,379)
[2] Link "Pricing" center=(874,958)
[3] Button "Bookmark" center=(1558,555)
[4] Button "Reload" center=(246,554)
[5] Button "Go forward" center=(1693,833)
[6] Link "Sign in" center=(347,64)

Instruction: tap bookmark

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1852,680)
[1] Button "Wishlist" center=(231,1015)
[2] Link "Account" center=(1511,277)
[3] Button "Buy now" center=(1851,260)
[4] Button "Checkout" center=(1693,737)

Instruction: select wishlist

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(467,665)
[1] Button "Open" center=(1282,116)
[2] Button "Save" center=(352,620)
[3] Button "Run" center=(1325,549)
[4] Button "Find" center=(239,864)
[5] Button "Commit" center=(70,186)

Instruction: save the file

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(784,401)
[1] Button "Run" center=(222,112)
[2] Button "Save" center=(406,767)
[3] Button "Format" center=(1620,511)
[4] Button "Commit" center=(568,524)
[5] TextField "Search files" center=(564,821)
[6] Button "New File" center=(1835,492)
[7] Button "Terminal" center=(1371,986)
[8] Button "Open" center=(327,1002)

Instruction: click terminal

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(791,564)
[1] TextField "Email" center=(1770,679)
[2] CheckBox "Remember me" center=(682,422)
[3] Link "Forgot password" center=(210,738)
[4] Button "Create account" center=(886,558)
[5] Button "Sign in" center=(961,309)

Instruction: sign in

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(496,437)
[1] Button "Go forward" center=(1830,513)
[2] Button "Downloads" center=(1410,135)
[3] SearchField "Search or enter address" center=(754,517)
[4] Link "Pricing" center=(999,697)
[5] Button "Run inference" center=(1135,961)

Instruction: click new tab

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(741,658)
[1] CheckBox "Wi-Fi" center=(1035,257)
[2] Button "Displays" center=(1478,799)
[3] Button "Sound" center=(1015,157)
[4] PopUpButton "Appearance" center=(610,505)
[5] Button "Network" center=(118,598)
[6] Slider "Brightness" center=(358,1011)

Instruction: press network

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1009,258)
[1] TextField "Password" center=(150,256)
[2] CheckBox "Remember me" center=(1817,63)
[3] Button "Create account" center=(1593,740)
[4] TextField "Email" center=(906,316)
[5] Link "Forgot password" center=(556,690)

Instruction: type into password

[Assistant] {"action":"type","id":1,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1740,95)
[1] Button "Downloads" center=(229,956)
[2] Button "Reload" center=(917,390)
[3] Button "Run inference" center=(1582,344)
[4] Link "Sign in" center=(376,243)
[5] Button "Bookmark" center=(755,1004)
[6] SearchField "Search or enter address" center=(898,533)

Instruction: press bookmark

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1520,723)
[1] Button "Shuffle" center=(1224,671)
[2] Button "Play" center=(336,371)
[3] Button "Fullscreen" center=(1400,883)
[4] Slider "Volume" center=(1543,800)

Instruction: start playback

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1744,917)
[1] Button "Privacy & Security" center=(297,689)
[2] Button "Displays" center=(960,694)
[3] CheckBox "Do Not Disturb" center=(1224,139)
[4] Button "Network" center=(114,263)
[5] PopUpButton "Appearance" center=(1567,232)
[6] CheckBox "Wi-Fi" center=(900,753)
[7] Button "Sound" center=(1158,832)

Instruction: toggle wifi

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1368,506)
[1] Button "Displays" center=(887,460)
[2] CheckBox "Bluetooth" center=(760,400)
[3] Slider "Brightness" center=(1736,595)
[4] PopUpButton "Appearance" center=(912,575)
[5] CheckBox "Do Not Disturb" center=(1606,383)
[6] Button "Privacy & Security" center=(272,988)
[7] Button "Sound" center=(400,509)

Instruction: turn off bluetooth

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1310,1000)
[1] Button "Buy now" center=(1795,141)
[2] SearchField "Search products" center=(219,865)
[3] Link "Account" center=(1754,146)
[4] Button "Checkout" center=(223,705)
[5] Button "Filters" center=(73,671)
[6] Button "Wishlist" center=(882,146)

Instruction: click wishlist

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(473,268)
[1] Button "Pause" center=(1631,505)
[2] Button "Next" center=(1169,430)
[3] Button "Play" center=(274,240)
[4] Button "Shuffle" center=(1793,520)
[5] Slider "Volume" center=(294,399)

Instruction: start playback

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(889,73)
[1] Button "Go back" center=(71,66)
[2] Link "Pricing" center=(354,849)
[3] SearchField "Search or enter address" center=(806,900)
[4] Link "Docs" center=(1712,571)

Instruction: open the docs

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1681,994)
[1] Link "Account" center=(1223,479)
[2] Button "Add to cart" center=(303,1001)
[3] Button "Filters" center=(1343,439)
[4] SearchField "Search products" center=(1531,217)

Instruction: add to cart

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(365,1011)
[1] PopUpButton "Appearance" center=(700,248)
[2] Slider "Brightness" center=(230,135)
[3] Button "Network" center=(92,487)
[4] Button "Privacy & Security" center=(63,683)
[5] CheckBox "Wi-Fi" center=(1617,647)
[6] Button "Displays" center=(353,68)
[7] CheckBox "Do Not Disturb" center=(1305,764)
[8] CheckBox "Bluetooth" center=(1153,982)

Instruction: click displays

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1878,267)
[1] Slider "Volume" center=(648,238)
[2] Button "Play" center=(1136,900)
[3] Button "Next" center=(542,783)
[4] Button "Previous" center=(1637,866)
[5] Button "Pause" center=(1649,427)
[6] Button "Shuffle" center=(1617,306)

Instruction: play

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(291,149)
[1] Button "Pause" center=(330,329)
[2] Button "Previous" center=(576,956)
[3] Button "Fullscreen" center=(409,135)
[4] Slider "Volume" center=(198,546)
[5] Button "Shuffle" center=(1769,410)
[6] Button "Play" center=(1756,252)

Instruction: tap fullscreen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(405,715)
[1] Button "New tab" center=(1201,686)
[2] Button "Go back" center=(1282,915)
[3] Button "Run inference" center=(104,160)
[4] Button "Extensions" center=(742,395)

Instruction: click extensions

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1088,819)
[1] Button "Network" center=(352,261)
[2] CheckBox "Do Not Disturb" center=(1815,413)
[3] Button "Privacy & Security" center=(275,960)
[4] Slider "Brightness" center=(484,70)
[5] CheckBox "Bluetooth" center=(710,181)
[6] CheckBox "Wi-Fi" center=(490,125)
[7] Button "Displays" center=(1530,748)
[8] PopUpButton "Appearance" center=(299,314)

Instruction: select privacy & security

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1089,1009)
[1] Button "Pause" center=(859,984)
[2] Button "Fullscreen" center=(435,296)
[3] Button "Previous" center=(394,242)
[4] Button "Next" center=(1447,906)
[5] Button "Shuffle" center=(1857,774)

Instruction: click previous

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1848,496)
[1] Button "Wishlist" center=(907,650)
[2] Button "Add to cart" center=(1570,318)
[3] Button "Buy now" center=(1710,366)
[4] Button "Checkout" center=(1315,150)
[5] SearchField "Search products" center=(1283,432)

Instruction: proceed to payment

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(408,868)
[1] Button "Run inference" center=(700,479)
[2] Button "Go forward" center=(1659,230)
[3] Link "Docs" center=(772,291)
[4] SearchField "Search or enter address" center=(188,260)
[5] Button "Bookmark" center=(1385,420)

Instruction: click go forward

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1119,262)
[1] Button "Run inference" center=(250,614)
[2] Link "Docs" center=(609,773)
[3] SearchField "Search or enter address" center=(1284,1008)
[4] Button "Go forward" center=(1849,866)
[5] Button "Reload" center=(1801,727)
[6] Button "New tab" center=(193,627)
[7] Link "Pricing" center=(1868,701)

Instruction: select new tab

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(602,471)
[1] Button "Sign in" center=(284,167)
[2] Link "Forgot password" center=(967,215)
[3] TextField "Email" center=(1271,374)
[4] CheckBox "Remember me" center=(1250,432)

Instruction: make a new account

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1002,137)
[1] Button "Format" center=(1739,861)
[2] Button "Open" center=(890,780)
[3] TextField "Search files" center=(656,321)
[4] Button "Save" center=(689,811)

Instruction: select terminal

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1305,145)
[1] TextField "Search files" center=(99,269)
[2] Button "Terminal" center=(137,885)
[3] Button "Find" center=(61,182)
[4] Button "Run" center=(255,370)
[5] Button "Format" center=(1804,744)
[6] Button "New File" center=(1523,486)

Instruction: press open

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1849,843)
[1] Button "Previous" center=(128,153)
[2] Button "Shuffle" center=(138,1009)
[3] Button "Play" center=(663,116)
[4] Slider "Volume" center=(580,377)
[5] Button "Next" center=(1448,667)

Instruction: play

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1108,580)
[1] Button "Displays" center=(532,631)
[2] Button "Sound" center=(1018,717)
[3] Button "Network" center=(1043,1039)
[4] Button "Privacy & Security" center=(614,369)

Instruction: press network

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(475,53)
[1] SearchField "Search or enter address" center=(1413,261)
[2] Link "Pricing" center=(998,878)
[3] Button "Go back" center=(828,232)
[4] Button "Run inference" center=(1038,147)

Instruction: select run inference

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(488,978)
[1] TextField "Email" center=(784,434)
[2] Button "Sign in" center=(966,928)
[3] TextField "Password" center=(1310,565)
[4] Button "Create account" center=(455,838)

Instruction: i forgot my password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(265,286)
[1] Button "Shuffle" center=(73,955)
[2] Button "Previous" center=(983,231)
[3] Button "Pause" center=(586,69)
[4] Button "Play" center=(1015,717)
[5] Slider "Volume" center=(1127,503)
[6] Button "Next" center=(760,311)

Instruction: start playback

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(797,140)
[1] CheckBox "Bluetooth" center=(140,704)
[2] CheckBox "Wi-Fi" center=(823,789)
[3] PopUpButton "Appearance" center=(905,238)
[4] Slider "Brightness" center=(642,233)
[5] Button "Network" center=(864,220)
[6] Button "Displays" center=(1256,322)

Instruction: tap appearance

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(604,271)
[1] Link "Pricing" center=(1847,675)
[2] Button "New tab" center=(1472,773)
[3] Button "Reload" center=(868,441)
[4] Button "Downloads" center=(844,143)
[5] Link "Docs" center=(490,791)

Instruction: open the docs

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(419,555)
[1] CheckBox "Wi-Fi" center=(1474,677)
[2] PopUpButton "Appearance" center=(1016,796)
[3] Button "Privacy & Security" center=(665,913)
[4] Button "Displays" center=(1448,284)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1844,118)
[1] Button "Find" center=(389,735)
[2] Button "Terminal" center=(1597,913)
[3] TextField "Search files" center=(59,648)
[4] Button "Commit" center=(761,1032)
[5] Button "Run" center=(1010,637)
[6] Button "Format" center=(42,540)

Instruction: type into search files

[Assistant] {"action":"type","id":3,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(1686,793)
[1] Button "Reload" center=(727,503)
[2] SearchField "Search or enter address" center=(1044,792)
[3] Button "Downloads" center=(758,598)
[4] Button "Go forward" center=(567,719)
[5] Link "Docs" center=(1036,299)
[6] Link "Sign in" center=(786,509)
[7] Button "Bookmark" center=(67,887)
[8] Button "New tab" center=(762,348)

Instruction: tap new tab

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1222,798)
[1] Button "Run inference" center=(1465,1036)
[2] Button "Extensions" center=(922,408)
[3] Link "Pricing" center=(462,802)
[4] Button "Downloads" center=(222,238)
[5] Button "Go forward" center=(813,694)
[6] Button "Bookmark" center=(261,1028)
[7] Button "Reload" center=(600,938)

Instruction: press reload

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(673,235)
[1] Button "Create account" center=(124,633)
[2] TextField "Password" center=(1105,758)
[3] Link "Forgot password" center=(743,894)
[4] Button "Sign in" center=(1294,165)
[5] CheckBox "Remember me" center=(927,854)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1434,193)
[1] Link "Forgot password" center=(1111,1029)
[2] Button "Sign in" center=(633,678)
[3] Button "Create account" center=(1183,720)
[4] CheckBox "Remember me" center=(373,350)
[5] TextField "Email" center=(1037,335)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(142,297)
[1] Link "Sign in" center=(322,398)
[2] Button "Extensions" center=(1839,330)
[3] Link "Docs" center=(1648,919)
[4] Button "Reload" center=(1068,721)

Instruction: refresh the page

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(488,72)
[1] Button "Play" center=(1009,995)
[2] Button "Fullscreen" center=(766,294)
[3] Button "Previous" center=(1702,734)
[4] Button "Shuffle" center=(1344,435)
[5] Button "Pause" center=(905,448)

Instruction: select volume

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1687,632)
[1] Button "Create account" center=(325,739)
[2] CheckBox "Remember me" center=(1612,244)
[3] TextField "Password" center=(552,178)
[4] Button "Sign in" center=(892,607)
[5] Link "Forgot password" center=(448,348)

Instruction: tap remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(681,555)
[1] Button "Filters" center=(1443,769)
[2] SearchField "Search products" center=(772,519)
[3] Button "Add to cart" center=(765,477)
[4] Link "Account" center=(995,470)
[5] Button "Wishlist" center=(880,512)
[6] Button "Buy now" center=(96,974)

Instruction: tap buy now

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(865,726)
[1] Link "Docs" center=(52,513)
[2] Button "Go back" center=(1693,821)
[3] Button "Extensions" center=(1831,570)
[4] Link "Pricing" center=(982,899)
[5] SearchField "Search or enter address" center=(151,126)
[6] Button "Go forward" center=(152,316)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":5,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1097,187)
[1] Button "Bookmark" center=(1191,798)
[2] Link "Docs" center=(1054,179)
[3] Link "Sign in" center=(497,956)
[4] Button "Go back" center=(1423,695)
[5] Button "Reload" center=(1785,198)
[6] Button "Extensions" center=(656,683)
[7] Link "Pricing" center=(676,776)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":0,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(458,146)
[1] TextField "Search files" center=(1010,65)
[2] Button "New File" center=(1745,557)
[3] Button "Format" center=(948,440)
[4] Button "Terminal" center=(153,988)
[5] Button "Commit" center=(289,347)

Instruction: type into search files

[Assistant] {"action":"type","id":1,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(418,185)
[1] Button "Sound" center=(1716,802)
[2] CheckBox "Bluetooth" center=(1386,286)
[3] CheckBox "Wi-Fi" center=(389,563)
[4] Button "Network" center=(1558,756)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(830,754)
[1] Button "Pause" center=(1678,149)
[2] Button "Shuffle" center=(990,466)
[3] Button "Next" center=(1706,475)
[4] Button "Previous" center=(343,68)
[5] Slider "Volume" center=(620,508)

Instruction: next song

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(772,552)
[1] Link "Sign in" center=(1792,921)
[2] Button "Downloads" center=(1070,255)
[3] Button "New tab" center=(754,964)
[4] Button "Bookmark" center=(1486,577)
[5] Button "Go forward" center=(839,565)
[6] Button "Reload" center=(168,218)

Instruction: select downloads

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1006,660)
[1] Button "Open" center=(1389,224)
[2] Button "Save" center=(1219,765)
[3] Button "Run" center=(1078,834)
[4] TextField "Search files" center=(1389,578)
[5] Button "Format" center=(570,172)
[6] Button "Commit" center=(1469,371)
[7] Button "Find" center=(1245,1018)

Instruction: commit my changes

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1596,277)
[1] Button "Next" center=(1620,867)
[2] Button "Shuffle" center=(1540,854)
[3] Slider "Volume" center=(524,147)
[4] Button "Play" center=(1843,218)
[5] Button "Fullscreen" center=(1531,953)

Instruction: click volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1470,500)
[1] Link "Pricing" center=(813,176)
[2] Link "Docs" center=(1345,627)
[3] Link "Sign in" center=(1179,244)
[4] SearchField "Search or enter address" center=(440,298)
[5] Button "Reload" center=(1045,488)

Instruction: press pricing

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1409,138)
[1] Button "Next" center=(456,74)
[2] Button "Play" center=(1698,183)
[3] Button "Fullscreen" center=(1869,45)
[4] Slider "Volume" center=(776,832)
[5] Button "Previous" center=(875,621)
[6] Button "Pause" center=(1159,925)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1734,525)
[1] Button "Next" center=(1065,400)
[2] Button "Play" center=(770,673)
[3] Button "Previous" center=(1184,913)
[4] Slider "Volume" center=(322,860)
[5] Button "Fullscreen" center=(177,489)
[6] Button "Shuffle" center=(915,680)

Instruction: click pause

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(498,881)
[1] Slider "Brightness" center=(633,984)
[2] Button "Sound" center=(1736,297)
[3] CheckBox "Bluetooth" center=(839,760)
[4] Button "Network" center=(772,109)

Instruction: select brightness

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1377,70)
[1] Button "Go forward" center=(106,386)
[2] Button "Reload" center=(903,307)
[3] Button "Downloads" center=(450,382)
[4] Button "Go back" center=(1858,823)
[5] Link "Pricing" center=(161,182)
[6] SearchField "Search or enter address" center=(458,600)
[7] Button "New tab" center=(1712,791)
[8] Button "Extensions" center=(715,986)

Instruction: open the docs

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1554,667)
[1] Button "Buy now" center=(205,948)
[2] Button "Filters" center=(1169,530)
[3] Button "Checkout" center=(305,317)
[4] Button "Add to cart" center=(1521,82)
[5] Link "Account" center=(253,76)
[6] SearchField "Search products" center=(1057,559)

Instruction: tap filters

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1171,533)
[1] Button "Terminal" center=(146,436)
[2] Button "New File" center=(1608,426)
[3] Button "Open" center=(1464,168)
[4] Button "Format" center=(920,758)

Instruction: click terminal

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(228,642)
[1] Button "Find" center=(60,272)
[2] Button "Save" center=(885,805)
[3] Button "Run" center=(1581,949)
[4] Button "Commit" center=(1694,852)
[5] TextField "Search files" center=(1138,621)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(618,84)
[1] Button "Fullscreen" center=(683,398)
[2] Button "Previous" center=(1273,710)
[3] Slider "Volume" center=(1863,290)
[4] Button "Shuffle" center=(638,172)

Instruction: click volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(948,770)
[1] Slider "Volume" center=(659,1004)
[2] Button "Previous" center=(791,838)
[3] Button "Pause" center=(1169,339)
[4] Button "Next" center=(1819,709)

Instruction: select previous

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1313,537)
[1] Button "Next" center=(612,120)
[2] Button "Pause" center=(456,973)
[3] Button "Shuffle" center=(320,901)
[4] Button "Play" center=(1106,276)
[5] Button "Fullscreen" center=(664,799)
[6] Slider "Volume" center=(1172,255)

Instruction: press fullscreen

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1734,1028)
[1] Link "Sign in" center=(213,101)
[2] Link "Docs" center=(611,915)
[3] Button "New tab" center=(1288,433)
[4] Button "Reload" center=(1054,59)
[5] Button "Run inference" center=(1806,431)
[6] Button "Extensions" center=(827,494)
[7] Button "Go back" center=(130,269)
[8] Button "Go forward" center=(461,428)

Instruction: click extensions

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1699,811)
[1] Button "Pause" center=(1250,261)
[2] Button "Shuffle" center=(1590,68)
[3] Button "Fullscreen" center=(738,783)
[4] Slider "Volume" center=(1738,676)
[5] Button "Play" center=(928,926)

Instruction: click play

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1567,986)
[1] Button "Go forward" center=(527,175)
[2] Button "Go back" center=(71,327)
[3] Button "Reload" center=(1677,738)
[4] Button "Downloads" center=(601,127)
[5] Button "Run inference" center=(551,829)
[6] Button "Bookmark" center=(1151,342)
[7] Link "Pricing" center=(731,624)
[8] Button "Extensions" center=(848,79)
[9] Button "New tab" center=(1300,320)
[10] Link "Docs" center=(780,315)
[11] Link "Sign in" center=(1390,296)

Instruction: click new tab

[Assistant] {"action":"click","id":9}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(774,531)
[1] Button "Go forward" center=(1346,904)
[2] Link "Sign in" center=(1179,263)
[3] SearchField "Search or enter address" center=(876,99)
[4] Button "Go back" center=(1226,1001)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1819,546)
[1] Button "Filters" center=(300,272)
[2] SearchField "Search products" center=(1330,218)
[3] Button "Buy now" center=(340,932)
[4] Button "Wishlist" center=(1305,711)
[5] Button "Add to cart" center=(141,278)

Instruction: press account

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1503,305)
[1] Button "Shuffle" center=(567,56)
[2] Button "Play" center=(1429,905)
[3] Button "Next" center=(1796,909)
[4] Button "Fullscreen" center=(763,593)

Instruction: go fullscreen

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1354,476)
[1] Button "Downloads" center=(1833,329)
[2] Button "Bookmark" center=(333,65)
[3] Button "New tab" center=(708,661)
[4] Button "Reload" center=(1654,408)
[5] Link "Docs" center=(726,1038)

Instruction: show documentation

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(92,148)
[1] Button "Wishlist" center=(1287,256)
[2] Link "Account" center=(1694,360)
[3] Button "Filters" center=(1086,772)
[4] Button "Checkout" center=(1639,799)
[5] Button "Buy now" center=(1080,842)
[6] Button "Add to cart" center=(1230,769)

Instruction: tap wishlist

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(499,618)
[1] Button "Filters" center=(1071,628)
[2] Button "Checkout" center=(1330,601)
[3] Button "Wishlist" center=(1543,468)
[4] Button "Buy now" center=(1143,790)

Instruction: click buy now

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1752,629)
[1] Button "Checkout" center=(640,972)
[2] Button "Buy now" center=(539,376)
[3] Button "Filters" center=(537,858)
[4] SearchField "Search products" center=(1612,1040)

Instruction: select filters

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(203,400)
[1] Slider "Volume" center=(1187,557)
[2] Button "Next" center=(196,976)
[3] Button "Previous" center=(781,709)
[4] Button "Pause" center=(1031,659)
[5] Button "Shuffle" center=(511,499)

Instruction: press volume

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(739,361)
[1] Link "Account" center=(281,123)
[2] Button "Checkout" center=(876,471)
[3] Button "Filters" center=(1074,191)
[4] SearchField "Search products" center=(1219,546)
[5] Button "Wishlist" center=(1335,942)

Instruction: tap account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(189,175)
[1] Button "Bookmark" center=(1094,927)
[2] Link "Sign in" center=(1124,980)
[3] Button "Run inference" center=(834,965)
[4] SearchField "Search or enter address" center=(904,402)
[5] Button "Go back" center=(301,451)
[6] Button "Downloads" center=(1467,610)

Instruction: sign in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(877,685)
[1] Button "Buy now" center=(961,798)
[2] Button "Wishlist" center=(1749,486)
[3] Link "Account" center=(632,815)
[4] Button "Add to cart" center=(1017,756)
[5] Button "Filters" center=(315,791)

Instruction: click add to cart

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1081,779)
[1] CheckBox "Do Not Disturb" center=(1244,326)
[2] CheckBox "Bluetooth" center=(1623,139)
[3] Button "Displays" center=(515,344)
[4] Button "Network" center=(1050,389)
[5] Button "Privacy & Security" center=(138,617)

Instruction: tap network

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1707,1007)
[1] Link "Forgot password" center=(1544,943)
[2] TextField "Email" center=(1537,255)
[3] CheckBox "Remember me" center=(921,571)
[4] Button "Create account" center=(609,628)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1281,811)
[1] Link "Pricing" center=(257,986)
[2] Button "Go back" center=(514,722)
[3] Link "Sign in" center=(610,184)
[4] Button "Extensions" center=(236,251)
[5] Button "Bookmark" center=(1170,898)
[6] Button "Run inference" center=(1081,812)
[7] SearchField "Search or enter address" center=(385,68)
[8] Button "Go forward" center=(1265,964)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(617,1004)
[1] Link "Account" center=(1209,476)
[2] SearchField "Search products" center=(1539,495)
[3] Button "Wishlist" center=(1106,169)
[4] Button "Add to cart" center=(1741,504)

Instruction: click checkout

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1685,120)
[1] Button "Format" center=(1564,852)
[2] Button "Run" center=(495,784)
[3] TextField "Search files" center=(192,374)
[4] Button "Open" center=(360,559)
[5] Button "Save" center=(726,322)
[6] Button "New File" center=(968,685)

Instruction: select save

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1826,1005)
[1] Button "Reload" center=(490,729)
[2] Link "Pricing" center=(440,311)
[3] Button "Run inference" center=(1002,195)
[4] Button "Go back" center=(1032,873)
[5] Button "New tab" center=(423,791)
[6] Button "Extensions" center=(1057,329)
[7] Button "Bookmark" center=(309,243)
[8] Button "Downloads" center=(1340,936)
[9] Link "Sign in" center=(221,1013)
[10] Button "Go forward" center=(1097,917)

Instruction: click run inference

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1296,429)
[1] Button "Run" center=(1332,637)
[2] Button "Save" center=(520,693)
[3] Button "Format" center=(754,535)
[4] TextField "Search files" center=(1126,712)
[5] Button "Commit" center=(408,938)
[6] Button "Find" center=(654,900)

Instruction: save

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(270,1026)
[1] Button "Play" center=(1844,958)
[2] Button "Pause" center=(1586,822)
[3] Button "Fullscreen" center=(628,469)
[4] Button "Next" center=(955,373)

Instruction: make it full screen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1624,943)
[1] TextField "Password" center=(1030,900)
[2] Button "Create account" center=(1688,924)
[3] Button "Sign in" center=(1041,655)
[4] TextField "Email" center=(883,743)

Instruction: make a new account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(856,306)
[1] Button "Find" center=(151,314)
[2] Button "Open" center=(1034,785)
[3] Button "Run" center=(488,387)
[4] Button "Commit" center=(1343,878)
[5] Button "Terminal" center=(133,355)
[6] Button "New File" center=(956,459)
[7] Button "Format" center=(493,626)

Instruction: click run

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(224,839)
[1] Button "Wishlist" center=(1483,381)
[2] Link "Account" center=(762,825)
[3] Button "Add to cart" center=(1608,606)
[4] Button "Buy now" center=(169,710)
[5] Button "Checkout" center=(140,571)

Instruction: press buy now

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(886,260)
[1] Button "Displays" center=(1172,790)
[2] CheckBox "Wi-Fi" center=(549,1002)
[3] CheckBox "Do Not Disturb" center=(67,76)
[4] CheckBox "Bluetooth" center=(1771,60)
[5] Button "Sound" center=(1586,1002)
[6] Slider "Brightness" center=(1118,144)

Instruction: tap brightness

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1382,450)
[1] Link "Pricing" center=(308,877)
[2] Button "Run inference" center=(1556,937)
[3] Link "Sign in" center=(592,922)
[4] Button "Go forward" center=(213,118)
[5] Link "Docs" center=(191,556)
[6] Button "Downloads" center=(1518,760)
[7] Button "Bookmark" center=(1601,790)
[8] Button "Go back" center=(890,60)
[9] SearchField "Search or enter address" center=(70,800)
[10] Button "New tab" center=(1740,594)

Instruction: click run inference

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1562,63)
[1] Button "Shuffle" center=(658,320)
[2] Button "Fullscreen" center=(979,247)
[3] Button "Next" center=(244,310)
[4] Button "Pause" center=(1383,630)
[5] Button "Previous" center=(1005,425)
[6] Slider "Volume" center=(636,479)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1530,443)
[1] Button "Add to cart" center=(803,821)
[2] Link "Account" center=(1159,771)
[3] Button "Buy now" center=(261,503)
[4] Button "Filters" center=(498,631)
[5] Button "Wishlist" center=(90,446)

Instruction: select buy now

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1811,1004)
[1] Button "Run" center=(1043,647)
[2] Button "Commit" center=(551,125)
[3] Button "Terminal" center=(329,452)
[4] Button "Save" center=(1302,898)
[5] TextField "Search files" center=(1009,575)

Instruction: click save

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1266,456)
[1] Button "Run" center=(1824,45)
[2] Button "Save" center=(589,746)
[3] Button "Open" center=(468,813)
[4] Button "Terminal" center=(1191,853)
[5] Button "Format" center=(1040,598)
[6] Button "New File" center=(1063,360)
[7] Button "Commit" center=(1375,281)
[8] Button "Find" center=(699,512)

Instruction: find in file

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(119,946)
[1] Button "Bookmark" center=(908,547)
[2] Button "Downloads" center=(389,681)
[3] Button "Run inference" center=(1684,577)
[4] Link "Pricing" center=(1037,183)
[5] Button "New tab" center=(1573,935)

Instruction: click run inference

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1299,950)
[1] TextField "Password" center=(801,617)
[2] Link "Forgot password" center=(1371,209)
[3] Button "Sign in" center=(329,337)
[4] Button "Create account" center=(1111,741)
[5] TextField "Email" center=(458,862)

Instruction: type into email

[Assistant] {"action":"type","id":5,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(716,989)
[1] Button "Create account" center=(1569,777)
[2] TextField "Password" center=(1785,990)
[3] Button "Sign in" center=(317,208)
[4] CheckBox "Remember me" center=(1113,772)

Instruction: type into email

[Assistant] {"action":"type","id":0,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(678,758)
[1] TextField "Email" center=(424,244)
[2] CheckBox "Remember me" center=(1461,136)
[3] Link "Forgot password" center=(1141,509)
[4] TextField "Password" center=(717,136)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(602,331)
[1] Link "Account" center=(1720,973)
[2] Button "Wishlist" center=(1817,476)
[3] Button "Filters" center=(1236,786)
[4] Button "Checkout" center=(666,416)

Instruction: select filters

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1536,261)
[1] Button "Wishlist" center=(613,222)
[2] Button "Checkout" center=(578,617)
[3] Button "Add to cart" center=(249,169)
[4] Button "Buy now" center=(1672,574)

Instruction: add to cart

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(483,133)
[1] Button "Next" center=(1685,669)
[2] Button "Play" center=(340,186)
[3] Button "Pause" center=(377,526)
[4] Button "Previous" center=(1831,40)

Instruction: press shuffle

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1596,160)
[1] Button "Checkout" center=(533,613)
[2] Button "Buy now" center=(845,426)
[3] SearchField "Search products" center=(1624,434)
[4] Link "Account" center=(1486,525)
[5] Button "Wishlist" center=(767,359)
[6] Button "Filters" center=(1757,574)

Instruction: press add to cart

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1182,961)
[1] Slider "Brightness" center=(931,707)
[2] CheckBox "Bluetooth" center=(753,222)
[3] Button "Sound" center=(91,926)
[4] PopUpButton "Appearance" center=(788,915)

Instruction: press bluetooth

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(945,409)
[1] Button "Filters" center=(1117,239)
[2] Button "Add to cart" center=(51,640)
[3] Button "Checkout" center=(675,376)
[4] Button "Wishlist" center=(232,972)

Instruction: add this to my cart

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1482,299)
[1] Link "Account" center=(601,680)
[2] Button "Buy now" center=(157,1034)
[3] Button "Wishlist" center=(1211,883)
[4] Button "Filters" center=(102,923)
[5] Button "Add to cart" center=(1510,989)
[6] Button "Checkout" center=(1004,662)

Instruction: press wishlist

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(209,469)
[1] Button "Find" center=(1748,906)
[2] Button "Format" center=(284,97)
[3] Button "Terminal" center=(948,692)
[4] Button "Commit" center=(1495,103)
[5] Button "New File" center=(683,139)

Instruction: press format

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(386,967)
[1] Button "Pause" center=(1241,779)
[2] Button "Previous" center=(790,568)
[3] Button "Next" center=(69,1003)
[4] Button "Play" center=(1400,452)
[5] Button "Shuffle" center=(892,897)
[6] Slider "Volume" center=(1407,725)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(604,663)
[1] Button "Filters" center=(446,745)
[2] Button "Wishlist" center=(1075,459)
[3] Button "Add to cart" center=(394,690)
[4] Button "Checkout" center=(1569,902)

Instruction: find a product

[Assistant] {"action":"type","id":0,"text":"wireless headphones"}