[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(550,342)
[1] TextField "Search files" center=(402,573)
[2] Button "Open" center=(208,426)
[3] Button "Format" center=(1693,301)
[4] Button "Run" center=(565,110)
[5] Button "Commit" center=(1713,761)
[6] Button "Find" center=(1438,462)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1736,311)
[1] SearchField "Search products" center=(127,1038)
[2] Button "Filters" center=(804,1026)
[3] Button "Buy now" center=(294,78)
[4] Button "Checkout" center=(426,725)
[5] Button "Wishlist" center=(1187,599)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1826,144)
[1] PopUpButton "Appearance" center=(579,688)
[2] Slider "Brightness" center=(570,732)
[3] Button "Privacy & Security" center=(594,504)
[4] Button "Sound" center=(277,581)
[5] Button "Network" center=(1586,632)
[6] CheckBox "Bluetooth" center=(510,646)
[7] Button "Displays" center=(689,853)
[8] CheckBox "Wi-Fi" center=(472,988)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1588,921)
[1] Button "Checkout" center=(1390,920)
[2] Button "Buy now" center=(1282,681)
[3] SearchField "Search products" center=(1457,981)
[4] Button "Wishlist" center=(47,623)
[5] Button "Add to cart" center=(395,217)
[6] Button "Filters" center=(1075,589)

Instruction: tap buy now

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1141,806)
[1] Button "Create account" center=(1868,504)
[2] Link "Forgot password" center=(318,906)
[3] TextField "Email" center=(1045,166)
[4] CheckBox "Remember me" center=(968,312)
[5] Button "Sign in" center=(1859,241)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(664,1009)
[1] Button "Filters" center=(466,644)
[2] Button "Buy now" center=(1071,818)
[3] Button "Wishlist" center=(1110,962)
[4] Link "Account" center=(173,559)

Instruction: tap account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(68,889)
[1] Button "New File" center=(554,136)
[2] Button "Find" center=(516,774)
[3] Button "Terminal" center=(57,659)
[4] Button "Commit" center=(65,677)
[5] Button "Open" center=(270,1004)
[6] TextField "Search files" center=(754,237)
[7] Button "Run" center=(1089,885)

Instruction: type into search files

[Assistant] {"action":"type","id":6,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1806,930)
[1] Button "Bookmark" center=(516,955)
[2] Button "New tab" center=(214,665)
[3] Button "Go back" center=(111,781)
[4] Button "Go forward" center=(1376,404)
[5] SearchField "Search or enter address" center=(962,512)
[6] Link "Pricing" center=(314,47)

Instruction: click new tab

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(161,676)
[1] Button "Extensions" center=(1028,771)
[2] Link "Docs" center=(1301,356)
[3] Button "Reload" center=(242,1017)
[4] Button "Downloads" center=(1339,985)
[5] SearchField "Search or enter address" center=(335,187)

Instruction: tap downloads

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(200,547)
[1] Button "Extensions" center=(1764,345)
[2] Button "Go back" center=(1664,187)
[3] Button "Go forward" center=(419,336)
[4] Button "Bookmark" center=(68,311)
[5] Button "New tab" center=(1744,951)
[6] SearchField "Search or enter address" center=(1461,701)
[7] Link "Docs" center=(1419,259)
[8] Button "Reload" center=(393,469)
[9] Link "Pricing" center=(244,150)

Instruction: click pricing

[Assistant] {"action":"click","id":9}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(345,599)
[1] SearchField "Search or enter address" center=(857,499)
[2] Link "Pricing" center=(1462,708)
[3] Button "Go forward" center=(387,97)
[4] Button "Extensions" center=(350,837)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1209,409)
[1] TextField "Email" center=(442,459)
[2] Button "Sign in" center=(1153,637)
[3] Link "Forgot password" center=(103,50)
[4] Button "Create account" center=(734,516)

Instruction: log in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(985,131)
[1] TextField "Search files" center=(428,810)
[2] Button "Open" center=(1634,75)
[3] Button "Format" center=(1380,502)
[4] Button "Save" center=(391,204)
[5] Button "Find" center=(231,151)
[6] Button "Terminal" center=(458,593)
[7] Button "New File" center=(1705,429)
[8] Button "Run" center=(1199,82)

Instruction: click new file

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(687,931)
[1] Button "Next" center=(1493,832)
[2] Button "Play" center=(612,801)
[3] Slider "Volume" center=(874,851)
[4] Button "Pause" center=(269,547)
[5] Button "Fullscreen" center=(267,623)

Instruction: tap shuffle

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(778,187)
[1] Button "Extensions" center=(230,703)
[2] Button "Run inference" center=(853,55)
[3] Button "Reload" center=(335,390)
[4] Button "Downloads" center=(410,259)
[5] Button "Go forward" center=(1243,316)
[6] Button "Go back" center=(1586,124)

Instruction: click go forward

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(86,253)
[1] Button "Buy now" center=(1421,227)
[2] Button "Add to cart" center=(901,696)
[3] SearchField "Search products" center=(1321,401)
[4] Button "Wishlist" center=(686,136)
[5] Button "Filters" center=(500,985)
[6] Button "Checkout" center=(307,521)

Instruction: tap filters

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(746,887)
[1] Button "Wishlist" center=(1457,407)
[2] Link "Account" center=(1236,593)
[3] Button "Add to cart" center=(758,154)
[4] SearchField "Search products" center=(822,77)
[5] Button "Checkout" center=(309,148)

Instruction: proceed to payment

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(624,790)
[1] SearchField "Search products" center=(1282,861)
[2] Button "Checkout" center=(1137,946)
[3] Button "Wishlist" center=(887,301)
[4] Button "Buy now" center=(1020,609)
[5] Button "Add to cart" center=(916,850)
[6] Button "Filters" center=(1839,970)

Instruction: add this to my cart

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(88,563)
[1] Button "Run" center=(1574,276)
[2] Button "Open" center=(937,880)
[3] Button "New File" center=(1386,610)
[4] Button "Format" center=(221,1040)
[5] Button "Find" center=(1196,152)
[6] Button "Terminal" center=(1211,73)

Instruction: press format

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(1648,373)
[1] Link "Sign in" center=(511,325)
[2] Button "Bookmark" center=(1146,315)
[3] Button "Go forward" center=(1084,487)
[4] Link "Pricing" center=(593,499)
[5] Button "Downloads" center=(120,556)
[6] SearchField "Search or enter address" center=(363,160)
[7] Button "Extensions" center=(498,627)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":6,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(175,210)
[1] CheckBox "Remember me" center=(448,801)
[2] TextField "Email" center=(1528,543)
[3] TextField "Password" center=(1445,1001)
[4] Button "Create account" center=(1560,191)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(77,601)
[1] Button "Sign in" center=(1102,425)
[2] CheckBox "Remember me" center=(549,852)
[3] Link "Forgot password" center=(637,801)
[4] TextField "Password" center=(1502,452)

Instruction: type into password

[Assistant] {"action":"type","id":4,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(827,621)
[1] Button "Downloads" center=(1582,799)
[2] Button "New tab" center=(1325,195)
[3] Button "Bookmark" center=(1720,280)
[4] Link "Docs" center=(324,741)
[5] Button "Reload" center=(1052,246)

Instruction: tap reload

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1152,902)
[1] Button "Pause" center=(208,1005)
[2] Button "Fullscreen" center=(1790,927)
[3] Button "Previous" center=(1161,475)
[4] Button "Play" center=(1023,304)
[5] Slider "Volume" center=(835,290)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(929,161)
[1] Button "Add to cart" center=(1863,139)
[2] Button "Wishlist" center=(1520,895)
[3] Button "Checkout" center=(540,1015)
[4] Link "Account" center=(952,124)

Instruction: click account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1066,184)
[1] TextField "Password" center=(1594,627)
[2] Button "Create account" center=(1688,356)
[3] Button "Sign in" center=(347,1036)
[4] Link "Forgot password" center=(77,233)
[5] TextField "Email" center=(1342,306)

Instruction: type into email

[Assistant] {"action":"type","id":5,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(1217,661)
[1] Link "Docs" center=(1263,296)
[2] Button "Extensions" center=(620,635)
[3] Button "Go back" center=(1863,760)
[4] Link "Sign in" center=(599,923)
[5] Link "Pricing" center=(868,482)
[6] Button "Reload" center=(944,191)
[7] Button "Run inference" center=(1506,94)
[8] SearchField "Search or enter address" center=(235,99)
[9] Button "Bookmark" center=(1167,181)
[10] Button "Downloads" center=(586,942)

Instruction: press new tab

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(954,650)
[1] Button "Next" center=(1610,335)
[2] Button "Shuffle" center=(42,192)
[3] Button "Pause" center=(1864,440)
[4] Button "Play" center=(328,74)
[5] Slider "Volume" center=(1127,394)
[6] Button "Fullscreen" center=(1065,948)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(678,203)
[1] Button "Play" center=(495,392)
[2] Slider "Volume" center=(970,57)
[3] Button "Shuffle" center=(57,341)
[4] Button "Previous" center=(218,203)
[5] Button "Pause" center=(1309,136)

Instruction: press previous

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1326,220)
[1] Button "Buy now" center=(1650,686)
[2] Button "Filters" center=(710,394)
[3] SearchField "Search products" center=(1196,58)
[4] Button "Add to cart" center=(627,721)

Instruction: search for something

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1190,288)
[1] Link "Docs" center=(1109,186)
[2] Button "New tab" center=(978,128)
[3] Button "Extensions" center=(100,131)
[4] Button "Downloads" center=(1521,940)
[5] Link "Sign in" center=(1567,969)
[6] SearchField "Search or enter address" center=(399,245)

Instruction: go to docs

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1059,793)
[1] Button "Terminal" center=(919,485)
[2] Button "Run" center=(1578,372)
[3] Button "New File" center=(499,1018)
[4] Button "Commit" center=(1708,830)

Instruction: run the code

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1785,689)
[1] Button "Sign in" center=(654,432)
[2] Button "Create account" center=(1730,1029)
[3] Link "Forgot password" center=(91,149)
[4] TextField "Password" center=(1313,130)

Instruction: type into email

[Assistant] {"action":"type","id":0,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(521,398)
[1] TextField "Email" center=(1183,1024)
[2] Link "Forgot password" center=(903,125)
[3] Button "Sign in" center=(1045,459)
[4] TextField "Password" center=(1135,213)
[5] Button "Create account" center=(374,1002)

Instruction: press remember me

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1471,841)
[1] TextField "Search files" center=(859,1015)
[2] Button "Open" center=(352,916)
[3] Button "Terminal" center=(306,322)
[4] Button "Find" center=(196,1019)
[5] Button "Save" center=(1312,940)
[6] Button "Run" center=(945,945)

Instruction: execute

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(163,694)
[1] SearchField "Search products" center=(465,410)
[2] Button "Filters" center=(70,77)
[3] Button "Add to cart" center=(885,1038)
[4] Link "Account" center=(262,865)
[5] Button "Wishlist" center=(1085,350)
[6] Button "Checkout" center=(1065,619)

Instruction: press filters

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1839,796)
[1] Button "Displays" center=(1583,427)
[2] CheckBox "Wi-Fi" center=(1161,196)
[3] Slider "Brightness" center=(266,194)
[4] CheckBox "Bluetooth" center=(217,384)
[5] CheckBox "Do Not Disturb" center=(1659,692)
[6] Button "Privacy & Security" center=(926,903)
[7] Button "Sound" center=(557,659)

Instruction: press do not disturb

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1680,329)
[1] Button "Create account" center=(1356,878)
[2] TextField "Email" center=(220,824)
[3] TextField "Password" center=(293,143)
[4] Link "Forgot password" center=(1413,767)
[5] Button "Sign in" center=(1294,544)

Instruction: i forgot my password

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1426,640)
[1] Button "Terminal" center=(1655,334)
[2] Button "Find" center=(1064,935)
[3] Button "Open" center=(1809,554)
[4] TextField "Search files" center=(1691,223)
[5] Button "Format" center=(1522,106)
[6] Button "Commit" center=(1121,569)

Instruction: commit my changes

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(226,204)
[1] Button "Terminal" center=(1709,939)
[2] Button "Run" center=(799,358)
[3] Button "Format" center=(441,712)
[4] Button "New File" center=(1575,325)
[5] Button "Save" center=(208,861)

Instruction: click terminal

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(159,247)
[1] Link "Sign in" center=(374,103)
[2] Link "Pricing" center=(1471,246)
[3] Button "Downloads" center=(255,648)
[4] Button "Bookmark" center=(954,603)

Instruction: click bookmark

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1247,882)
[1] Button "Downloads" center=(995,820)
[2] SearchField "Search or enter address" center=(1375,608)
[3] Button "Extensions" center=(889,86)
[4] Link "Sign in" center=(1638,807)
[5] Link "Pricing" center=(254,945)

Instruction: press extensions

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1231,703)
[1] TextField "Password" center=(623,905)
[2] Button "Sign in" center=(1342,517)
[3] TextField "Email" center=(1722,576)
[4] Link "Forgot password" center=(713,928)

Instruction: make a new account

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1007,984)
[1] CheckBox "Wi-Fi" center=(380,611)
[2] PopUpButton "Appearance" center=(1122,732)
[3] Button "Privacy & Security" center=(1816,897)
[4] Button "Displays" center=(948,575)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(913,68)
[1] SearchField "Search products" center=(1171,746)
[2] Link "Account" center=(1088,153)
[3] Button "Wishlist" center=(1635,670)
[4] Button "Buy now" center=(1002,1039)

Instruction: type into search products

[Assistant] {"action":"type","id":1,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1136,242)
[1] Button "Shuffle" center=(1033,386)
[2] Button "Pause" center=(1755,272)
[3] Button "Next" center=(122,498)
[4] Button "Previous" center=(1838,469)

Instruction: click play

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(104,55)
[1] Button "Network" center=(387,706)
[2] CheckBox "Bluetooth" center=(566,725)
[3] CheckBox "Do Not Disturb" center=(1817,340)
[4] PopUpButton "Appearance" center=(437,857)
[5] Button "Privacy & Security" center=(1612,178)
[6] Button "Sound" center=(476,937)
[7] CheckBox "Wi-Fi" center=(849,1027)
[8] Button "Displays" center=(421,206)

Instruction: tap network

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(571,447)
[1] Button "Buy now" center=(1404,523)
[2] Button "Checkout" center=(80,438)
[3] Link "Account" center=(1680,493)
[4] SearchField "Search products" center=(1437,739)
[5] Button "Wishlist" center=(770,107)

Instruction: click wishlist

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(835,114)
[1] PopUpButton "Appearance" center=(49,531)
[2] CheckBox "Bluetooth" center=(59,262)
[3] Button "Displays" center=(444,412)
[4] Button "Privacy & Security" center=(135,1003)
[5] Slider "Brightness" center=(1705,704)

Instruction: press bluetooth

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1149,915)
[1] Link "Forgot password" center=(639,555)
[2] Button "Sign in" center=(175,265)
[3] CheckBox "Remember me" center=(88,50)
[4] TextField "Email" center=(657,974)
[5] TextField "Password" center=(259,796)

Instruction: select create account

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(834,755)
[1] Button "Sound" center=(635,453)
[2] CheckBox "Bluetooth" center=(1500,853)
[3] Button "Network" center=(541,276)
[4] Button "Privacy & Security" center=(1407,928)
[5] Slider "Brightness" center=(1298,923)
[6] PopUpButton "Appearance" center=(715,278)
[7] CheckBox "Do Not Disturb" center=(674,603)
[8] CheckBox "Wi-Fi" center=(225,135)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(545,605)
[1] Button "Go forward" center=(730,368)
[2] Button "Go back" center=(1564,823)
[3] Button "Run inference" center=(1809,153)
[4] Button "Downloads" center=(320,599)
[5] Link "Sign in" center=(1617,118)
[6] Button "Bookmark" center=(959,969)
[7] Button "Reload" center=(1049,377)
[8] Button "Extensions" center=(1693,140)
[9] Link "Docs" center=(314,279)
[10] Link "Pricing" center=(1252,421)
[11] Button "New tab" center=(1606,616)

Instruction: select go forward

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(369,272)
[1] TextField "Email" center=(1543,432)
[2] CheckBox "Remember me" center=(78,197)
[3] Button "Create account" center=(645,845)
[4] TextField "Password" center=(639,485)
[5] Button "Sign in" center=(1593,795)

Instruction: select remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(983,248)
[1] Button "Run" center=(165,886)
[2] Button "Commit" center=(1064,514)
[3] Button "Format" center=(290,410)
[4] Button "New File" center=(826,864)

Instruction: press format

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1859,492)
[1] Button "Format" center=(647,881)
[2] Button "Commit" center=(189,900)
[3] Button "Find" center=(657,555)
[4] Button "New File" center=(1773,372)
[5] Button "Terminal" center=(1485,510)

Instruction: find in file

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1176,568)
[1] TextField "Password" center=(1747,347)
[2] TextField "Email" center=(133,499)
[3] Link "Forgot password" center=(893,478)
[4] Button "Create account" center=(553,179)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(466,784)
[1] Slider "Volume" center=(1163,1030)
[2] Button "Next" center=(861,527)
[3] Button "Play" center=(895,748)
[4] Button "Previous" center=(323,406)
[5] Button "Fullscreen" center=(1276,168)

Instruction: tap previous

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1098,980)
[1] Button "Shuffle" center=(1111,442)
[2] Button "Pause" center=(1251,735)
[3] Button "Play" center=(518,220)
[4] Button "Fullscreen" center=(157,750)
[5] Button "Previous" center=(725,838)

Instruction: select shuffle

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(567,835)
[1] Button "Run inference" center=(58,361)
[2] Link "Docs" center=(432,738)
[3] SearchField "Search or enter address" center=(1383,795)
[4] Button "Bookmark" center=(698,461)
[5] Button "New tab" center=(962,99)
[6] Button "Go forward" center=(1548,525)
[7] Link "Pricing" center=(1064,649)
[8] Button "Reload" center=(1845,514)
[9] Link "Sign in" center=(1063,767)
[10] Button "Go back" center=(734,392)

Instruction: click go forward

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(470,920)
[1] Button "Wishlist" center=(864,807)
[2] Button "Filters" center=(1754,173)
[3] Button "Checkout" center=(972,786)
[4] Button "Buy now" center=(462,224)
[5] Button "Add to cart" center=(162,763)

Instruction: add this to my cart

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(1636,341)
[1] Link "Sign in" center=(362,671)
[2] Button "Extensions" center=(1848,596)
[3] Button "Run inference" center=(801,156)
[4] Button "Bookmark" center=(631,314)

Instruction: press run inference

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1204,853)
[1] SearchField "Search or enter address" center=(1165,54)
[2] Link "Pricing" center=(235,951)
[3] Button "Extensions" center=(585,228)
[4] Link "Sign in" center=(1429,447)
[5] Button "New tab" center=(590,547)
[6] Button "Run inference" center=(1737,119)
[7] Button "Go back" center=(164,472)
[8] Link "Docs" center=(1158,65)
[9] Button "Go forward" center=(1818,113)
[10] Button "Reload" center=(962,515)

Instruction: press bookmark

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(418,463)
[1] Button "New tab" center=(1750,509)
[2] SearchField "Search or enter address" center=(415,299)
[3] Button "Reload" center=(612,425)
[4] Button "Extensions" center=(1222,269)
[5] Link "Sign in" center=(1786,155)
[6] Button "Run inference" center=(1807,867)
[7] Button "Downloads" center=(942,203)

Instruction: tap downloads

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(572,102)
[1] Button "Sound" center=(360,942)
[2] CheckBox "Do Not Disturb" center=(756,670)
[3] Slider "Brightness" center=(1532,667)
[4] Button "Privacy & Security" center=(1714,603)
[5] CheckBox "Bluetooth" center=(705,151)
[6] Button "Displays" center=(1471,658)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(495,396)
[1] Button "Next" center=(1375,88)
[2] Button "Shuffle" center=(651,1038)
[3] Button "Play" center=(1621,616)
[4] Button "Fullscreen" center=(1713,479)
[5] Button "Previous" center=(161,168)

Instruction: tap previous

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(460,642)
[1] Button "Find" center=(1041,311)
[2] Button "New File" center=(1316,806)
[3] Button "Terminal" center=(1323,640)
[4] Button "Run" center=(805,63)
[5] Button "Open" center=(967,780)
[6] TextField "Search files" center=(41,557)

Instruction: select terminal

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1258,87)
[1] Button "Bookmark" center=(379,230)
[2] Button "Reload" center=(1620,119)
[3] Button "Run inference" center=(800,835)
[4] Button "New tab" center=(1730,1007)
[5] SearchField "Search or enter address" center=(652,943)
[6] Button "Go forward" center=(1706,283)

Instruction: refresh the page

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(394,46)
[1] Button "Wishlist" center=(383,290)
[2] Button "Add to cart" center=(772,108)
[3] Button "Checkout" center=(123,197)
[4] Link "Account" center=(1698,423)

Instruction: add this to my cart

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1317,238)
[1] Button "Reload" center=(1586,834)
[2] SearchField "Search or enter address" center=(542,237)
[3] Button "Go forward" center=(1832,206)
[4] Button "Go back" center=(643,910)
[5] Link "Pricing" center=(544,443)
[6] Link "Docs" center=(374,646)
[7] Button "Extensions" center=(913,175)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1637,381)
[1] Button "New tab" center=(1578,652)
[2] SearchField "Search or enter address" center=(844,360)
[3] Button "Go back" center=(1831,666)
[4] Button "Extensions" center=(1271,669)
[5] Button "Reload" center=(314,1014)
[6] Link "Pricing" center=(155,160)
[7] Button "Go forward" center=(332,288)
[8] Button "Bookmark" center=(1056,451)
[9] Link "Sign in" center=(445,117)
[10] Button "Run inference" center=(1025,510)
[11] Button "Downloads" center=(672,642)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":2,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(410,445)
[1] Button "Extensions" center=(954,117)
[2] SearchField "Search or enter address" center=(1479,274)
[3] Button "Run inference" center=(244,533)
[4] Button "Bookmark" center=(802,277)
[5] Link "Pricing" center=(1519,450)
[6] Link "Sign in" center=(302,1006)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":2,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1476,66)
[1] Button "Save" center=(1863,675)
[2] Button "Open" center=(911,163)
[3] Button "Terminal" center=(958,868)
[4] Button "Run" center=(1609,274)
[5] Button "Find" center=(94,44)

Instruction: save

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(168,402)
[1] Link "Forgot password" center=(1455,275)
[2] Button "Create account" center=(1801,379)
[3] TextField "Email" center=(766,50)
[4] Button "Sign in" center=(690,136)

Instruction: i forgot my password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(574,517)
[1] Button "Sign in" center=(1279,1037)
[2] CheckBox "Remember me" center=(299,423)
[3] Button "Create account" center=(1458,799)
[4] TextField "Email" center=(554,920)

Instruction: i forgot my password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(55,349)
[1] Button "Previous" center=(1620,595)
[2] Slider "Volume" center=(1029,544)
[3] Button "Play" center=(124,481)
[4] Button "Shuffle" center=(1049,689)
[5] Button "Next" center=(822,889)
[6] Button "Fullscreen" center=(1216,243)

Instruction: press previous

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(157,209)
[1] Button "Buy now" center=(736,518)
[2] Button "Add to cart" center=(1507,595)
[3] Button "Checkout" center=(1402,336)
[4] Button "Wishlist" center=(927,321)
[5] Button "Filters" center=(157,944)
[6] SearchField "Search products" center=(1104,381)

Instruction: type into search products

[Assistant] {"action":"type","id":6,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1665,529)
[1] Button "Next" center=(316,454)
[2] Button "Previous" center=(1654,216)
[3] Button "Shuffle" center=(76,797)
[4] Button "Pause" center=(1080,312)
[5] Button "Fullscreen" center=(854,533)
[6] Slider "Volume" center=(296,714)

Instruction: skip to next track

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1177,202)
[1] Button "Pause" center=(393,227)
[2] Slider "Volume" center=(1820,398)
[3] Button "Play" center=(57,813)
[4] Button "Shuffle" center=(257,586)

Instruction: start playback

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1783,1008)
[1] Button "Save" center=(387,851)
[2] Button "Format" center=(372,291)
[3] Button "Terminal" center=(600,129)
[4] TextField "Search files" center=(628,623)

Instruction: type into search files

[Assistant] {"action":"type","id":4,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(40,308)
[1] Button "Go forward" center=(1741,953)
[2] Button "New tab" center=(741,887)
[3] Button "Extensions" center=(1758,93)
[4] Button "Bookmark" center=(1229,409)
[5] Button "Downloads" center=(921,238)
[6] Link "Sign in" center=(419,108)
[7] Button "Reload" center=(1017,896)
[8] SearchField "Search or enter address" center=(623,702)
[9] Link "Docs" center=(120,59)
[10] Button "Run inference" center=(1508,277)

Instruction: click pricing

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(368,345)
[1] Button "Add to cart" center=(1752,514)
[2] SearchField "Search products" center=(231,128)
[3] Link "Account" center=(1252,267)
[4] Button "Checkout" center=(822,912)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(267,985)
[1] Button "Format" center=(1443,806)
[2] Button "Open" center=(896,890)
[3] Button "Terminal" center=(1761,96)
[4] TextField "Search files" center=(1010,1039)
[5] Button "Commit" center=(932,931)
[6] Button "Run" center=(305,440)
[7] Button "New File" center=(616,263)

Instruction: make a commit

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1795,268)
[1] Button "Sound" center=(1418,153)
[2] Button "Privacy & Security" center=(461,928)
[3] CheckBox "Wi-Fi" center=(1854,242)
[4] Button "Network" center=(1650,460)
[5] PopUpButton "Appearance" center=(274,52)

Instruction: select network

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1505,141)
[1] Link "Forgot password" center=(836,397)
[2] TextField "Password" center=(605,961)
[3] Button "Create account" center=(1713,334)
[4] TextField "Email" center=(278,977)

Instruction: sign up

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1798,749)
[1] Button "Play" center=(466,849)
[2] Button "Pause" center=(554,658)
[3] Button "Fullscreen" center=(638,224)
[4] Button "Next" center=(1317,1038)
[5] Slider "Volume" center=(132,878)

Instruction: tap fullscreen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1309,670)
[1] CheckBox "Remember me" center=(1196,233)
[2] Link "Forgot password" center=(393,346)
[3] Button "Create account" center=(1468,227)
[4] TextField "Email" center=(124,943)
[5] Button "Sign in" center=(385,333)

Instruction: sign up

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1207,819)
[1] Link "Sign in" center=(62,209)
[2] Button "Extensions" center=(775,935)
[3] Link "Docs" center=(640,831)
[4] Button "Go forward" center=(1137,418)
[5] SearchField "Search or enter address" center=(1566,270)
[6] Button "Run inference" center=(815,902)
[7] Button "Downloads" center=(1598,180)
[8] Button "New tab" center=(896,612)
[9] Button "Bookmark" center=(1202,82)
[10] Button "Reload" center=(615,744)
[11] Link "Pricing" center=(997,217)

Instruction: select run inference

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(227,440)
[1] Button "Go forward" center=(1527,325)
[2] Link "Docs" center=(520,368)
[3] Link "Pricing" center=(1637,644)
[4] Link "Sign in" center=(1275,56)
[5] Button "Reload" center=(1750,857)
[6] Button "New tab" center=(106,635)

Instruction: click pricing

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1549,136)
[1] Button "Sign in" center=(641,267)
[2] TextField "Password" center=(973,557)
[3] TextField "Email" center=(1110,308)
[4] Button "Create account" center=(604,257)
[5] CheckBox "Remember me" center=(976,92)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(793,714)
[1] SearchField "Search or enter address" center=(648,943)
[2] Button "Go back" center=(1461,129)
[3] Button "Run inference" center=(1103,421)
[4] Button "Go forward" center=(1712,569)
[5] Button "Extensions" center=(1568,663)

Instruction: reload

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(123,754)
[1] Button "Fullscreen" center=(970,705)
[2] Button "Next" center=(1762,897)
[3] Button "Shuffle" center=(1191,434)
[4] Button "Play" center=(81,337)
[5] Slider "Volume" center=(523,412)

Instruction: click shuffle

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1252,561)
[1] Button "Sign in" center=(543,256)
[2] Button "Create account" center=(95,445)
[3] Link "Forgot password" center=(1029,700)
[4] TextField "Password" center=(941,721)
[5] TextField "Email" center=(472,62)

Instruction: click forgot password

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1355,327)
[1] Button "Checkout" center=(1629,123)
[2] SearchField "Search products" center=(1298,56)
[3] Button "Wishlist" center=(1619,564)
[4] Link "Account" center=(965,560)
[5] Button "Filters" center=(1365,559)
[6] Button "Buy now" center=(922,376)

Instruction: select buy now

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(663,736)
[1] Button "Pause" center=(1705,213)
[2] Button "Shuffle" center=(1466,567)
[3] Button "Fullscreen" center=(1460,672)
[4] Button "Previous" center=(1818,908)

Instruction: click fullscreen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(322,182)
[1] Button "Pause" center=(1736,160)
[2] Slider "Volume" center=(1462,506)
[3] Button "Play" center=(160,308)
[4] Button "Shuffle" center=(1366,917)
[5] Button "Next" center=(1188,472)
[6] Button "Fullscreen" center=(1606,756)

Instruction: tap shuffle

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1758,102)
[1] Slider "Volume" center=(1601,111)
[2] Button "Pause" center=(863,821)
[3] Button "Fullscreen" center=(373,631)
[4] Button "Play" center=(1424,957)

Instruction: go fullscreen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(355,468)
[1] Link "Account" center=(1478,370)
[2] Button "Wishlist" center=(1849,827)
[3] Button "Buy now" center=(457,831)
[4] Button "Add to cart" center=(1687,278)
[5] SearchField "Search products" center=(989,132)
[6] Button "Filters" center=(977,795)

Instruction: go to checkout

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1510,752)
[1] SearchField "Search products" center=(1251,588)
[2] Button "Add to cart" center=(1446,1016)
[3] Button "Filters" center=(1401,280)
[4] Button "Wishlist" center=(1850,129)

Instruction: type into search products

[Assistant] {"action":"type","id":1,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1627,619)
[1] TextField "Password" center=(1290,468)
[2] Button "Create account" center=(960,67)
[3] TextField "Email" center=(819,653)
[4] Link "Forgot password" center=(407,105)
[5] CheckBox "Remember me" center=(1303,388)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1089,800)
[1] PopUpButton "Appearance" center=(1628,324)
[2] Button "Network" center=(1388,845)
[3] CheckBox "Bluetooth" center=(542,871)
[4] Slider "Brightness" center=(151,754)
[5] Button "Displays" center=(340,841)
[6] CheckBox "Do Not Disturb" center=(1824,750)
[7] Button "Sound" center=(1429,881)
[8] CheckBox "Wi-Fi" center=(1556,117)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(494,734)
[1] Button "Buy now" center=(273,391)
[2] Button "Checkout" center=(1848,700)
[3] SearchField "Search products" center=(1120,633)
[4] Button "Wishlist" center=(1808,813)
[5] Button "Add to cart" center=(1593,964)
[6] Button "Filters" center=(146,145)

Instruction: select filters

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(959,833)
[1] Button "Wishlist" center=(134,157)
[2] SearchField "Search products" center=(1014,95)
[3] Button "Add to cart" center=(1266,1040)
[4] Button "Buy now" center=(359,462)

Instruction: check out

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1122,209)
[1] CheckBox "Bluetooth" center=(1270,1008)
[2] Button "Privacy & Security" center=(889,603)
[3] Button "Network" center=(396,706)
[4] Slider "Brightness" center=(1822,828)
[5] Button "Sound" center=(646,892)

Instruction: select brightness

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(1145,42)
[1] Link "Docs" center=(906,999)
[2] Button "Extensions" center=(1253,741)
[3] Link "Sign in" center=(270,776)
[4] Button "Bookmark" center=(1268,945)
[5] Button "Go forward" center=(1871,434)
[6] Button "New tab" center=(1798,152)
[7] Button "Downloads" center=(391,859)
[8] Button "Reload" center=(1007,455)
[9] Link "Pricing" center=(842,175)
[10] Button "Go back" center=(71,610)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(591,263)
[1] Button "New tab" center=(707,862)
[2] Button "Reload" center=(1258,430)
[3] Link "Docs" center=(1330,706)
[4] Button "Extensions" center=(416,697)
[5] SearchField "Search or enter address" center=(558,382)
[6] Button "Downloads" center=(1108,736)
[7] Button "Go back" center=(1722,936)
[8] Link "Pricing" center=(1032,961)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(359,1019)
[1] Button "Go back" center=(496,509)
[2] Link "Docs" center=(785,294)
[3] Button "Bookmark" center=(58,227)
[4] Button "Go forward" center=(1169,971)
[5] Link "Pricing" center=(585,623)

Instruction: click pricing

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1769,823)
[1] Button "Displays" center=(1865,746)
[2] Button "Network" center=(671,83)
[3] Button "Sound" center=(760,403)
[4] CheckBox "Do Not Disturb" center=(495,755)
[5] CheckBox "Bluetooth" center=(1604,475)

Instruction: tap appearance

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(283,400)
[1] CheckBox "Remember me" center=(1830,774)
[2] Button "Create account" center=(1207,502)
[3] TextField "Password" center=(1060,294)
[4] TextField "Email" center=(1100,418)

Instruction: type into password

[Assistant] {"action":"type","id":3,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1277,834)
[1] TextField "Search files" center=(1319,1033)
[2] Button "Format" center=(1123,457)
[3] Button "Save" center=(621,639)
[4] Button "Terminal" center=(190,172)
[5] Button "Commit" center=(803,635)
[6] Button "Open" center=(675,620)
[7] Button "Run" center=(1712,536)

Instruction: press format

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1194,674)
[1] Button "Shuffle" center=(303,51)
[2] Button "Fullscreen" center=(826,466)
[3] Button "Previous" center=(855,114)
[4] Button "Pause" center=(969,1011)
[5] Button "Next" center=(1205,637)

Instruction: select shuffle

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(963,665)
[1] Button "Format" center=(423,421)
[2] Button "Commit" center=(1639,484)
[3] Button "Terminal" center=(65,348)
[4] Button "Find" center=(288,292)
[5] TextField "Search files" center=(561,577)
[6] Button "Open" center=(393,90)
[7] Button "Run" center=(1111,1010)

Instruction: press find

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(100,44)
[1] Link "Pricing" center=(1876,108)
[2] Button "Extensions" center=(633,344)
[3] Button "Go forward" center=(141,708)
[4] Button "Go back" center=(1388,755)

Instruction: press go forward

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1823,331)
[1] SearchField "Search products" center=(1537,998)
[2] Button "Wishlist" center=(1195,112)
[3] Button "Checkout" center=(485,347)
[4] Button "Add to cart" center=(282,57)

Instruction: add this to my cart

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(710,892)
[1] Button "Shuffle" center=(1414,432)
[2] Button "Play" center=(648,203)
[3] Button "Pause" center=(1560,844)
[4] Button "Previous" center=(1406,1002)
[5] Slider "Volume" center=(1468,178)
[6] Button "Fullscreen" center=(647,970)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(103,623)
[1] Button "Previous" center=(1032,476)
[2] Slider "Volume" center=(916,663)
[3] Button "Fullscreen" center=(1679,144)
[4] Button "Play" center=(1314,1005)

Instruction: play

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1389,413)
[1] Button "Shuffle" center=(543,683)
[2] Button "Play" center=(45,499)
[3] Button "Pause" center=(1300,458)
[4] Button "Previous" center=(877,142)
[5] Button "Next" center=(1117,480)

Instruction: press pause

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1045,303)
[1] Button "Previous" center=(286,767)
[2] Button "Pause" center=(374,365)
[3] Button "Play" center=(501,1027)
[4] Button "Next" center=(188,712)
[5] Button "Fullscreen" center=(437,533)

Instruction: tap pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(502,469)
[1] Button "Reload" center=(216,815)
[2] Button "New tab" center=(1218,820)
[3] SearchField "Search or enter address" center=(1398,223)
[4] Link "Docs" center=(132,190)

Instruction: click docs

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(428,345)
[1] Link "Account" center=(1538,228)
[2] Button "Add to cart" center=(1699,340)
[3] Button "Filters" center=(1784,357)
[4] Button "Wishlist" center=(603,906)

Instruction: click wishlist

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1626,270)
[1] Button "Sign in" center=(685,107)
[2] TextField "Password" center=(1852,45)
[3] TextField "Email" center=(1456,897)
[4] CheckBox "Remember me" center=(1003,463)

Instruction: click remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(588,400)
[1] Button "Terminal" center=(825,237)
[2] Button "New File" center=(563,77)
[3] Button "Find" center=(1279,873)
[4] Button "Open" center=(190,611)

Instruction: click open

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1873,895)
[1] Button "Create account" center=(965,306)
[2] CheckBox "Remember me" center=(595,775)
[3] TextField "Email" center=(1679,674)
[4] Link "Forgot password" center=(843,725)
[5] Button "Sign in" center=(789,294)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(543,953)
[1] Button "Format" center=(1737,860)
[2] Button "Open" center=(65,774)
[3] Button "Find" center=(677,366)
[4] TextField "Search files" center=(1868,359)
[5] Button "Run" center=(280,592)
[6] Button "Commit" center=(605,58)
[7] Button "Terminal" center=(1598,294)
[8] Button "Save" center=(1277,985)

Instruction: save the file

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1537,80)
[1] Button "Filters" center=(528,566)
[2] SearchField "Search products" center=(801,977)
[3] Button "Wishlist" center=(1720,597)
[4] Button "Buy now" center=(1175,448)
[5] Link "Account" center=(1441,1011)
[6] Button "Add to cart" center=(610,683)

Instruction: click account

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(362,662)
[1] Button "Pause" center=(1469,386)
[2] Button "Fullscreen" center=(225,1034)
[3] Button "Play" center=(822,329)
[4] Button "Shuffle" center=(1457,487)
[5] Slider "Volume" center=(1492,654)

Instruction: press pause

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1278,519)
[1] Button "Sign in" center=(654,309)
[2] TextField "Email" center=(1780,61)
[3] TextField "Password" center=(1070,969)
[4] Button "Create account" center=(1078,961)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1472,669)
[1] Button "Run" center=(229,593)
[2] Button "New File" center=(1758,780)
[3] Button "Commit" center=(1724,749)
[4] Button "Open" center=(1328,946)
[5] Button "Find" center=(617,926)
[6] TextField "Search files" center=(495,120)
[7] Button "Terminal" center=(228,244)
[8] Button "Save" center=(1434,916)

Instruction: type into search files

[Assistant] {"action":"type","id":6,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(373,859)
[1] Button "Save" center=(857,286)
[2] Button "Run" center=(421,713)
[3] Button "Format" center=(284,154)
[4] Button "Commit" center=(1256,711)
[5] Button "Find" center=(865,73)
[6] Button "New File" center=(644,505)

Instruction: press find

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1023,783)
[1] Button "Go forward" center=(1469,1021)
[2] Button "Extensions" center=(1657,98)
[3] Button "New tab" center=(1808,584)
[4] Button "Reload" center=(1430,953)
[5] Link "Docs" center=(1430,831)
[6] SearchField "Search or enter address" center=(321,980)
[7] Button "Go back" center=(1351,956)
[8] Link "Pricing" center=(442,121)

Instruction: select go forward

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(875,175)
[1] Button "Run" center=(1827,457)
[2] Button "Commit" center=(408,294)
[3] Button "New File" center=(709,759)
[4] Button "Find" center=(1594,630)
[5] TextField "Search files" center=(809,984)
[6] Button "Terminal" center=(45,412)
[7] Button "Format" center=(515,1039)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1098,450)
[1] Link "Docs" center=(139,378)
[2] SearchField "Search or enter address" center=(1704,93)
[3] Button "Downloads" center=(845,282)
[4] Button "Run inference" center=(125,445)
[5] Button "Bookmark" center=(850,284)
[6] Link "Pricing" center=(928,889)
[7] Button "Reload" center=(57,419)
[8] Button "Go back" center=(130,172)
[9] Button "New tab" center=(1536,583)
[10] Link "Sign in" center=(1856,696)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1253,161)
[1] Button "Sign in" center=(377,100)
[2] Button "Create account" center=(1689,798)
[3] CheckBox "Remember me" center=(525,694)
[4] Link "Forgot password" center=(99,186)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(997,752)
[1] Link "Account" center=(1263,415)
[2] Button "Wishlist" center=(513,918)
[3] SearchField "Search products" center=(269,628)
[4] Button "Buy now" center=(1369,884)
[5] Button "Filters" center=(422,1032)
[6] Button "Checkout" center=(843,82)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1701,319)
[1] Button "Find" center=(1330,644)
[2] Button "Open" center=(320,84)
[3] Button "Format" center=(693,332)
[4] Button "Terminal" center=(1570,610)
[5] Button "Run" center=(1278,328)
[6] Button "Commit" center=(1460,332)

Instruction: find in file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1328,398)
[1] Button "Buy now" center=(1124,256)
[2] Button "Checkout" center=(620,947)
[3] Button "Filters" center=(1758,905)
[4] SearchField "Search products" center=(619,820)
[5] Link "Account" center=(716,447)

Instruction: press buy now

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1053,93)
[1] Button "Wishlist" center=(684,756)
[2] Button "Add to cart" center=(874,429)
[3] Button "Buy now" center=(64,280)
[4] Button "Checkout" center=(1702,458)
[5] Link "Account" center=(250,870)
[6] Button "Filters" center=(900,900)

Instruction: go to checkout

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(95,112)
[1] Button "Extensions" center=(806,1017)
[2] SearchField "Search or enter address" center=(447,737)
[3] Button "Run inference" center=(48,109)
[4] Button "Downloads" center=(943,275)
[5] Button "Go forward" center=(1431,783)
[6] Button "Go back" center=(1340,1013)
[7] Link "Sign in" center=(224,473)
[8] Link "Docs" center=(1057,868)
[9] Button "Reload" center=(232,363)

Instruction: tap go forward

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(468,291)
[1] Link "Account" center=(1249,374)
[2] Button "Wishlist" center=(669,492)
[3] SearchField "Search products" center=(1201,311)
[4] Button "Buy now" center=(155,766)

Instruction: select wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1212,995)
[1] CheckBox "Do Not Disturb" center=(1545,121)
[2] PopUpButton "Appearance" center=(1593,698)
[3] CheckBox "Wi-Fi" center=(1114,692)
[4] Button "Displays" center=(1218,584)

Instruction: tap appearance

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1844,370)
[1] SearchField "Search products" center=(1501,661)
[2] Button "Add to cart" center=(1053,310)
[3] Link "Account" center=(1324,726)
[4] Button "Checkout" center=(1185,511)

Instruction: select filters

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1681,788)
[1] Button "Fullscreen" center=(221,595)
[2] Button "Next" center=(583,290)
[3] Slider "Volume" center=(1792,46)
[4] Button "Pause" center=(1690,697)
[5] Button "Play" center=(245,617)

Instruction: select shuffle

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1198,921)
[1] Button "New File" center=(869,336)
[2] Button "Save" center=(570,80)
[3] TextField "Search files" center=(421,929)
[4] Button "Format" center=(1188,983)
[5] Button "Commit" center=(313,369)

Instruction: tap new file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1516,587)
[1] TextField "Search files" center=(1816,386)
[2] Button "Save" center=(1374,184)
[3] Button "Find" center=(135,179)
[4] Button "Run" center=(528,742)
[5] Button "Open" center=(413,62)

Instruction: save the file

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1391,112)
[1] Link "Sign in" center=(1278,645)
[2] Button "Go back" center=(1115,192)
[3] Button "Go forward" center=(1861,662)
[4] Button "Downloads" center=(58,858)
[5] Link "Pricing" center=(1417,534)
[6] Button "New tab" center=(718,879)
[7] Button "Reload" center=(697,922)
[8] SearchField "Search or enter address" center=(212,376)
[9] Button "Bookmark" center=(1765,315)
[10] Button "Run inference" center=(1453,390)
[11] Link "Docs" center=(299,958)

Instruction: refresh the page

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1419,198)
[1] Button "Displays" center=(417,832)
[2] CheckBox "Bluetooth" center=(1593,482)
[3] Slider "Brightness" center=(1320,782)
[4] Button "Sound" center=(1635,194)
[5] Button "Privacy & Security" center=(302,544)

Instruction: press brightness

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(97,987)
[1] CheckBox "Do Not Disturb" center=(1683,493)
[2] PopUpButton "Appearance" center=(47,252)
[3] Button "Network" center=(1183,410)
[4] CheckBox "Wi-Fi" center=(147,283)
[5] Button "Displays" center=(815,856)
[6] Button "Sound" center=(110,383)
[7] Button "Privacy & Security" center=(1298,360)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1640,816)
[1] Link "Docs" center=(791,558)
[2] Link "Sign in" center=(382,478)
[3] Button "Run inference" center=(968,470)
[4] Button "Reload" center=(322,420)
[5] Button "Bookmark" center=(1205,988)
[6] Button "Go forward" center=(44,902)
[7] SearchField "Search or enter address" center=(920,890)
[8] Button "New tab" center=(367,54)
[9] Link "Pricing" center=(1765,80)
[10] Button "Go back" center=(541,763)
[11] Button "Extensions" center=(869,235)

Instruction: tap go back

[Assistant] {"action":"click","id":10}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(685,649)
[1] TextField "Password" center=(1034,701)
[2] CheckBox "Remember me" center=(1240,253)
[3] Button "Sign in" center=(253,575)
[4] TextField "Email" center=(485,641)
[5] Button "Create account" center=(1471,828)

Instruction: select sign in

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(69,927)
[1] Button "Buy now" center=(617,651)
[2] Link "Account" center=(1478,859)
[3] Button "Wishlist" center=(1395,252)
[4] Button "Checkout" center=(332,883)
[5] Button "Add to cart" center=(893,833)

Instruction: press account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1511,601)
[1] Button "Sound" center=(1239,760)
[2] Button "Privacy & Security" center=(1843,586)
[3] CheckBox "Wi-Fi" center=(1689,706)
[4] Slider "Brightness" center=(192,716)
[5] PopUpButton "Appearance" center=(1815,972)

Instruction: select brightness

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1442,501)
[1] SearchField "Search products" center=(934,799)
[2] Button "Wishlist" center=(1471,981)
[3] Button "Add to cart" center=(1833,207)
[4] Button "Filters" center=(321,971)

Instruction: select filters

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1465,543)
[1] Button "Run" center=(171,907)
[2] TextField "Search files" center=(1091,832)
[3] Button "Terminal" center=(1066,578)
[4] Button "Open" center=(329,970)
[5] Button "Format" center=(303,249)
[6] Button "Commit" center=(962,578)
[7] Button "New File" center=(1356,599)
[8] Button "Find" center=(1369,534)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1469,780)
[1] Link "Forgot password" center=(563,629)
[2] TextField "Password" center=(1355,418)
[3] CheckBox "Remember me" center=(860,353)
[4] Button "Create account" center=(997,805)

Instruction: sign up

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1849,734)
[1] Button "Open" center=(1149,287)
[2] Button "Commit" center=(47,916)
[3] Button "New File" center=(1311,918)
[4] Button "Terminal" center=(950,197)
[5] Button "Run" center=(452,444)

Instruction: type into search files

[Assistant] {"action":"type","id":0,"text":"main.swift"}