[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(690,645)
[1] Button "New tab" center=(1323,1028)
[2] Button "Reload" center=(81,825)
[3] Link "Sign in" center=(386,847)
[4] Button "Extensions" center=(61,262)
[5] Link "Docs" center=(1346,423)
[6] Button "Go back" center=(982,123)

Instruction: select new tab

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(627,222)
[1] Slider "Volume" center=(611,183)
[2] Button "Play" center=(1197,363)
[3] Button "Next" center=(887,271)
[4] Button "Fullscreen" center=(1750,232)
[5] Button "Pause" center=(598,944)
[6] Button "Shuffle" center=(448,221)

Instruction: start playback

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1749,639)
[1] Button "Go back" center=(829,199)
[2] Button "Reload" center=(1141,622)
[3] Button "Bookmark" center=(471,759)
[4] Button "New tab" center=(928,395)

Instruction: reload

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1241,370)
[1] Button "Run inference" center=(802,617)
[2] Button "Extensions" center=(813,714)
[3] Button "Go forward" center=(1245,962)
[4] Button "Downloads" center=(109,370)
[5] Button "Go back" center=(1476,147)
[6] Button "Reload" center=(1329,407)
[7] Link "Pricing" center=(1820,962)
[8] SearchField "Search or enter address" center=(56,202)
[9] Button "New tab" center=(903,526)
[10] Button "Bookmark" center=(725,384)
[11] Link "Sign in" center=(1366,964)

Instruction: navigate back

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(857,524)
[1] Button "Filters" center=(1748,289)
[2] Button "Checkout" center=(557,780)
[3] SearchField "Search products" center=(804,655)
[4] Button "Wishlist" center=(731,534)
[5] Button "Add to cart" center=(1429,450)
[6] Button "Buy now" center=(654,81)

Instruction: click filters

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1855,651)
[1] Button "New File" center=(1468,1005)
[2] Button "Format" center=(1113,243)
[3] Button "Save" center=(1203,809)
[4] TextField "Search files" center=(325,173)
[5] Button "Run" center=(1398,381)
[6] Button "Terminal" center=(1654,1022)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(41,714)
[1] TextField "Email" center=(342,245)
[2] CheckBox "Remember me" center=(1848,216)
[3] Button "Create account" center=(657,354)
[4] Button "Sign in" center=(204,229)
[5] TextField "Password" center=(523,918)

Instruction: press forgot password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1004,603)
[1] Button "Pause" center=(1446,443)
[2] Button "Fullscreen" center=(1471,257)
[3] Button "Previous" center=(1007,724)
[4] Button "Play" center=(810,891)

Instruction: click pause

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(846,586)
[1] Button "Find" center=(186,395)
[2] Button "Terminal" center=(1040,725)
[3] TextField "Search files" center=(1556,1033)
[4] Button "Run" center=(1241,67)
[5] Button "Open" center=(494,946)
[6] Button "Commit" center=(1286,222)
[7] Button "Format" center=(1612,280)
[8] Button "New File" center=(795,410)

Instruction: save

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(1379,224)
[1] Button "Downloads" center=(911,194)
[2] Link "Docs" center=(623,358)
[3] Link "Sign in" center=(832,731)
[4] Button "Run inference" center=(1721,532)
[5] Button "Go forward" center=(111,307)
[6] Button "Go back" center=(903,223)
[7] Button "Extensions" center=(374,447)
[8] Link "Pricing" center=(1437,569)
[9] Button "Reload" center=(1748,148)
[10] Button "Bookmark" center=(992,865)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1078,433)
[1] Slider "Volume" center=(1445,505)
[2] Button "Shuffle" center=(63,537)
[3] Button "Pause" center=(1379,1022)
[4] Button "Play" center=(1412,451)
[5] Button "Previous" center=(905,829)
[6] Button "Fullscreen" center=(151,350)

Instruction: click previous

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(787,154)
[1] Link "Account" center=(1648,223)
[2] Button "Buy now" center=(908,584)
[3] Button "Checkout" center=(1595,432)
[4] SearchField "Search products" center=(1822,730)
[5] Button "Wishlist" center=(203,193)
[6] Button "Filters" center=(54,385)

Instruction: select wishlist

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(982,415)
[1] Button "Displays" center=(142,950)
[2] Button "Sound" center=(1267,742)
[3] CheckBox "Do Not Disturb" center=(796,298)
[4] Button "Privacy & Security" center=(911,385)
[5] CheckBox "Wi-Fi" center=(1422,343)
[6] Button "Network" center=(1610,878)
[7] Slider "Brightness" center=(88,595)

Instruction: open privacy settings

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1552,445)
[1] Button "Run" center=(549,219)
[2] Button "Open" center=(212,587)
[3] Button "Commit" center=(233,169)
[4] Button "Terminal" center=(1528,188)
[5] Button "Save" center=(1336,545)
[6] TextField "Search files" center=(922,954)
[7] Button "Find" center=(602,393)
[8] Button "Format" center=(978,745)

Instruction: tap format

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1344,807)
[1] Link "Docs" center=(658,472)
[2] Link "Pricing" center=(860,62)
[3] Button "Bookmark" center=(416,1013)
[4] Link "Sign in" center=(1569,303)
[5] Button "Run inference" center=(1517,708)
[6] Button "New tab" center=(1561,944)
[7] Button "Reload" center=(202,896)
[8] Button "Go back" center=(927,1028)
[9] SearchField "Search or enter address" center=(278,852)

Instruction: press sign in

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1569,745)
[1] Button "Pause" center=(1633,230)
[2] Button "Shuffle" center=(606,553)
[3] Button "Previous" center=(1415,86)
[4] Button "Next" center=(1278,448)
[5] Button "Play" center=(411,734)

Instruction: tap volume

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(493,270)
[1] Button "Network" center=(77,766)
[2] Slider "Brightness" center=(704,266)
[3] Button "Displays" center=(170,666)
[4] CheckBox "Do Not Disturb" center=(629,785)

Instruction: tap displays

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(737,367)
[1] Button "Play" center=(749,882)
[2] Button "Pause" center=(950,195)
[3] Button "Previous" center=(986,914)
[4] Button "Fullscreen" center=(1560,550)
[5] Button "Next" center=(471,585)

Instruction: next song

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(964,730)
[1] Button "Bookmark" center=(997,70)
[2] Link "Sign in" center=(359,679)
[3] Button "Reload" center=(1237,210)
[4] Button "New tab" center=(1318,400)
[5] Button "Extensions" center=(1196,481)
[6] Button "Go back" center=(890,823)
[7] Button "Downloads" center=(87,809)
[8] Link "Pricing" center=(621,609)
[9] Button "Go forward" center=(1563,542)

Instruction: press go back

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1811,202)
[1] Button "Find" center=(1610,836)
[2] Button "New File" center=(1544,450)
[3] Button "Save" center=(1166,458)
[4] Button "Format" center=(760,809)
[5] Button "Terminal" center=(485,468)

Instruction: select new file

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1799,453)
[1] TextField "Password" center=(1053,814)
[2] Button "Create account" center=(990,333)
[3] Button "Sign in" center=(1731,986)
[4] TextField "Email" center=(960,200)

Instruction: type into password

[Assistant] {"action":"type","id":1,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1228,708)
[1] Button "Terminal" center=(896,1037)
[2] Button "Format" center=(896,460)
[3] Button "Open" center=(426,691)
[4] Button "Run" center=(99,673)
[5] Button "New File" center=(499,181)
[6] TextField "Search files" center=(1288,674)
[7] Button "Save" center=(1070,122)

Instruction: type into search files

[Assistant] {"action":"type","id":6,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1781,975)
[1] Button "Go forward" center=(370,508)
[2] Button "New tab" center=(795,780)
[3] Link "Docs" center=(1831,285)
[4] Link "Sign in" center=(398,523)
[5] Button "Go back" center=(1499,983)
[6] Button "Extensions" center=(449,370)
[7] Link "Pricing" center=(311,340)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1064,696)
[1] CheckBox "Do Not Disturb" center=(267,862)
[2] CheckBox "Wi-Fi" center=(1277,897)
[3] Button "Network" center=(718,933)
[4] Button "Displays" center=(366,290)
[5] Button "Sound" center=(986,779)
[6] CheckBox "Bluetooth" center=(183,47)
[7] PopUpButton "Appearance" center=(1573,73)

Instruction: tap bluetooth

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1024,887)
[1] Button "Run" center=(1526,101)
[2] Button "Terminal" center=(639,871)
[3] Button "Save" center=(855,888)
[4] Button "Find" center=(1724,782)
[5] Button "Format" center=(1614,80)
[6] Button "New File" center=(784,449)
[7] Button "Commit" center=(1507,989)

Instruction: commit my changes

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(581,139)
[1] Button "Downloads" center=(715,697)
[2] Button "Go back" center=(1460,115)
[3] Link "Sign in" center=(978,960)
[4] Link "Docs" center=(1492,543)
[5] Button "Bookmark" center=(1581,290)
[6] Link "Pricing" center=(1590,306)
[7] Button "Extensions" center=(741,342)

Instruction: tap bookmark

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(630,66)
[1] Button "Shuffle" center=(492,941)
[2] Button "Previous" center=(799,617)
[3] Button "Play" center=(782,770)
[4] Button "Fullscreen" center=(636,354)
[5] Slider "Volume" center=(1506,202)
[6] Button "Next" center=(1202,689)

Instruction: tap volume

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(566,692)
[1] Button "Previous" center=(506,68)
[2] Button "Fullscreen" center=(1782,256)
[3] Slider "Volume" center=(1234,350)
[4] Button "Shuffle" center=(343,638)
[5] Button "Pause" center=(64,305)

Instruction: tap shuffle

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1812,988)
[1] Button "Commit" center=(1033,703)
[2] Button "Run" center=(386,931)
[3] TextField "Search files" center=(1115,533)
[4] Button "Find" center=(399,535)
[5] Button "Open" center=(1877,428)
[6] Button "Terminal" center=(69,724)
[7] Button "Format" center=(172,390)

Instruction: type into search files

[Assistant] {"action":"type","id":3,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1206,974)
[1] PopUpButton "Appearance" center=(1347,968)
[2] Button "Network" center=(64,900)
[3] CheckBox "Bluetooth" center=(1568,906)
[4] CheckBox "Wi-Fi" center=(1717,841)
[5] Button "Sound" center=(1653,576)
[6] Slider "Brightness" center=(1879,273)
[7] Button "Displays" center=(1558,247)

Instruction: open privacy settings

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(954,620)
[1] CheckBox "Remember me" center=(1110,293)
[2] Button "Create account" center=(541,302)
[3] Button "Sign in" center=(1672,797)
[4] TextField "Email" center=(414,746)

Instruction: press remember me

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(608,133)
[1] Slider "Volume" center=(1561,234)
[2] Button "Next" center=(1541,890)
[3] Button "Play" center=(170,967)
[4] Button "Pause" center=(311,575)
[5] Button "Previous" center=(660,102)
[6] Button "Shuffle" center=(465,273)

Instruction: tap shuffle

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(210,527)
[1] Button "Find" center=(444,142)
[2] Button "Run" center=(1150,643)
[3] TextField "Search files" center=(480,652)
[4] Button "Terminal" center=(1037,347)
[5] Button "Format" center=(1205,49)
[6] Button "New File" center=(1231,480)
[7] Button "Save" center=(1803,966)

Instruction: press save

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1271,317)
[1] Button "Buy now" center=(1048,686)
[2] Button "Add to cart" center=(401,376)
[3] Button "Wishlist" center=(1078,534)
[4] SearchField "Search products" center=(1093,330)
[5] Link "Account" center=(1361,346)

Instruction: find a product

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(53,366)
[1] Button "Shuffle" center=(1869,1037)
[2] Button "Play" center=(1083,521)
[3] Button "Previous" center=(1055,207)
[4] Button "Next" center=(406,107)
[5] Button "Fullscreen" center=(1597,482)

Instruction: tap pause

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(476,354)
[1] Button "Sign in" center=(1067,352)
[2] TextField "Email" center=(695,824)
[3] TextField "Password" center=(1126,523)
[4] Link "Forgot password" center=(250,741)

Instruction: i forgot my password

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(706,1019)
[1] Button "Sign in" center=(1097,525)
[2] TextField "Email" center=(1378,649)
[3] Button "Create account" center=(1655,466)
[4] CheckBox "Remember me" center=(1636,783)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(670,475)
[1] Button "Commit" center=(1193,442)
[2] Button "Run" center=(1601,467)
[3] TextField "Search files" center=(51,820)
[4] Button "Terminal" center=(867,96)
[5] Button "New File" center=(877,402)
[6] Button "Open" center=(1362,864)
[7] Button "Format" center=(326,1038)

Instruction: click terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(915,849)
[1] Button "Play" center=(823,373)
[2] Button "Shuffle" center=(1182,613)
[3] Slider "Volume" center=(1553,244)
[4] Button "Next" center=(1673,260)
[5] Button "Fullscreen" center=(1304,555)

Instruction: click shuffle

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(161,143)
[1] Button "Go forward" center=(154,694)
[2] Link "Docs" center=(741,322)
[3] Button "New tab" center=(1141,418)
[4] Button "Extensions" center=(937,985)
[5] Button "Run inference" center=(622,153)
[6] Button "Go back" center=(1006,177)
[7] Link "Sign in" center=(1843,807)
[8] Link "Pricing" center=(191,372)
[9] Button "Downloads" center=(1627,790)
[10] Button "Reload" center=(887,884)

Instruction: go to docs

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(327,830)
[1] Button "Sound" center=(1616,109)
[2] Button "Privacy & Security" center=(1823,844)
[3] Button "Displays" center=(1472,768)
[4] CheckBox "Do Not Disturb" center=(1553,361)

Instruction: tap privacy & security

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(722,243)
[1] Button "Shuffle" center=(1427,1034)
[2] Button "Play" center=(1481,305)
[3] Button "Previous" center=(562,658)
[4] Button "Next" center=(1571,995)
[5] Button "Fullscreen" center=(87,183)

Instruction: tap previous

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1072,934)
[1] Button "Open" center=(1674,374)
[2] Button "Format" center=(1658,591)
[3] Button "New File" center=(1047,329)
[4] TextField "Search files" center=(1217,381)

Instruction: run the code

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1611,740)
[1] Link "Account" center=(922,877)
[2] Button "Checkout" center=(607,516)
[3] Button "Wishlist" center=(964,950)
[4] Button "Add to cart" center=(1843,796)

Instruction: proceed to payment

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(641,46)
[1] Button "Buy now" center=(1317,888)
[2] Button "Add to cart" center=(838,59)
[3] Link "Account" center=(57,270)
[4] SearchField "Search products" center=(700,477)
[5] Button "Wishlist" center=(348,305)
[6] Button "Checkout" center=(1792,268)

Instruction: go to checkout

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1623,191)
[1] Button "Play" center=(462,355)
[2] Button "Previous" center=(411,143)
[3] Button "Next" center=(93,586)
[4] Slider "Volume" center=(1272,1007)

Instruction: play

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(908,317)
[1] Button "Create account" center=(517,998)
[2] TextField "Email" center=(484,830)
[3] Link "Forgot password" center=(767,67)
[4] TextField "Password" center=(321,938)

Instruction: reset password

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(398,558)
[1] Link "Sign in" center=(1224,781)
[2] Button "Downloads" center=(1097,164)
[3] Button "New tab" center=(373,814)
[4] Button "Go forward" center=(469,163)
[5] Button "Run inference" center=(1041,798)

Instruction: select run inference

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(73,47)
[1] Button "Commit" center=(818,862)
[2] Button "Open" center=(900,449)
[3] Button "Terminal" center=(1796,142)
[4] Button "New File" center=(766,409)
[5] Button "Find" center=(674,896)
[6] Button "Save" center=(173,484)
[7] TextField "Search files" center=(1776,707)

Instruction: make a commit

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1196,49)
[1] Button "Terminal" center=(302,571)
[2] Button "Run" center=(358,183)
[3] TextField "Search files" center=(1463,521)
[4] Button "New File" center=(1601,79)
[5] Button "Commit" center=(165,650)

Instruction: make a commit

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1697,869)
[1] Button "Checkout" center=(1409,225)
[2] Button "Add to cart" center=(787,940)
[3] Link "Account" center=(264,845)
[4] Button "Filters" center=(372,70)
[5] Button "Buy now" center=(1058,248)
[6] Button "Wishlist" center=(666,453)

Instruction: select wishlist

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1545,932)
[1] Button "Buy now" center=(335,975)
[2] SearchField "Search products" center=(874,87)
[3] Button "Filters" center=(1144,426)
[4] Link "Account" center=(1601,652)

Instruction: press buy now

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1794,657)
[1] Button "Reload" center=(1697,692)
[2] Link "Sign in" center=(41,358)
[3] Button "New tab" center=(1015,520)
[4] Button "Run inference" center=(566,171)
[5] Link "Pricing" center=(701,712)
[6] Button "Extensions" center=(1145,414)

Instruction: click pricing

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1562,137)
[1] Button "New File" center=(1508,453)
[2] Button "Save" center=(618,948)
[3] Button "Format" center=(1299,147)
[4] Button "Find" center=(551,271)

Instruction: save

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(807,717)
[1] Button "Pause" center=(337,633)
[2] Button "Play" center=(1548,997)
[3] Button "Next" center=(1782,75)
[4] Button "Shuffle" center=(1571,967)
[5] Button "Fullscreen" center=(1677,434)

Instruction: press pause

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(1327,930)
[1] Button "Downloads" center=(1713,1034)
[2] Button "Go forward" center=(1222,455)
[3] Link "Docs" center=(965,259)
[4] Button "Run inference" center=(968,676)
[5] Link "Pricing" center=(578,598)
[6] Link "Sign in" center=(940,160)
[7] Button "Extensions" center=(1565,326)

Instruction: select pricing

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1603,571)
[1] CheckBox "Wi-Fi" center=(578,104)
[2] CheckBox "Do Not Disturb" center=(1641,414)
[3] Slider "Brightness" center=(913,697)
[4] PopUpButton "Appearance" center=(92,1027)
[5] Button "Network" center=(1152,169)
[6] Button "Privacy & Security" center=(233,272)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1719,1003)
[1] PopUpButton "Appearance" center=(1853,563)
[2] CheckBox "Do Not Disturb" center=(439,185)
[3] CheckBox "Wi-Fi" center=(1760,736)
[4] Slider "Brightness" center=(494,834)
[5] Button "Privacy & Security" center=(74,243)
[6] Button "Network" center=(1542,802)
[7] Button "Displays" center=(764,578)

Instruction: click do not disturb

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1433,209)
[1] Button "Add to cart" center=(1340,689)
[2] Button "Wishlist" center=(1799,673)
[3] Link "Account" center=(503,903)
[4] Button "Buy now" center=(1578,773)

Instruction: tap buy now

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(318,778)
[1] Button "Sound" center=(654,40)
[2] CheckBox "Bluetooth" center=(193,297)
[3] PopUpButton "Appearance" center=(1490,609)
[4] Button "Privacy & Security" center=(1408,359)
[5] CheckBox "Do Not Disturb" center=(467,985)
[6] Button "Network" center=(1542,667)

Instruction: click brightness

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1251,238)
[1] Button "Find" center=(549,894)
[2] Button "Save" center=(757,112)
[3] Button "Commit" center=(503,1026)
[4] Button "Format" center=(605,183)
[5] Button "Open" center=(1868,451)
[6] Button "Run" center=(292,783)
[7] TextField "Search files" center=(1554,639)

Instruction: save the file

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(250,830)
[1] Button "Pause" center=(1327,652)
[2] Button "Shuffle" center=(1004,277)
[3] Slider "Volume" center=(1392,488)
[4] Button "Play" center=(651,881)
[5] Button "Fullscreen" center=(295,594)
[6] Button "Next" center=(1561,664)

Instruction: select previous

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1238,212)
[1] TextField "Password" center=(1834,766)
[2] CheckBox "Remember me" center=(1456,926)
[3] Button "Sign in" center=(1667,538)
[4] Button "Create account" center=(1593,338)
[5] TextField "Email" center=(500,575)

Instruction: type into email

[Assistant] {"action":"type","id":5,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(156,693)
[1] Button "Filters" center=(255,925)
[2] Link "Account" center=(1616,326)
[3] Button "Wishlist" center=(1765,781)
[4] Button "Buy now" center=(1540,322)
[5] Button "Checkout" center=(555,695)

Instruction: select wishlist

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1239,546)
[1] TextField "Password" center=(493,645)
[2] Button "Sign in" center=(928,721)
[3] Link "Forgot password" center=(584,146)
[4] CheckBox "Remember me" center=(1639,886)

Instruction: tap remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1756,852)
[1] Button "Filters" center=(1356,1016)
[2] Button "Checkout" center=(241,629)
[3] Button "Add to cart" center=(234,160)
[4] Button "Wishlist" center=(1604,284)
[5] Link "Account" center=(778,634)
[6] SearchField "Search products" center=(1417,766)

Instruction: click add to cart

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1705,864)
[1] TextField "Email" center=(1326,802)
[2] Button "Sign in" center=(1632,482)
[3] Link "Forgot password" center=(1410,969)
[4] Button "Create account" center=(92,261)
[5] CheckBox "Remember me" center=(1201,225)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1645,88)
[1] Button "Go forward" center=(704,509)
[2] Button "Run inference" center=(1471,690)
[3] Link "Sign in" center=(578,827)
[4] SearchField "Search or enter address" center=(1665,204)
[5] Button "Downloads" center=(468,611)
[6] Button "Reload" center=(1618,924)
[7] Button "Extensions" center=(1160,703)
[8] Link "Pricing" center=(699,798)
[9] Button "Go back" center=(861,811)
[10] Link "Docs" center=(753,839)
[11] Button "New tab" center=(1710,216)

Instruction: select go forward

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1518,132)
[1] Button "Sign in" center=(1349,455)
[2] Button "Create account" center=(680,676)
[3] TextField "Password" center=(1058,694)
[4] CheckBox "Remember me" center=(421,814)

Instruction: tap create account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(72,154)
[1] Link "Forgot password" center=(428,635)
[2] TextField "Email" center=(675,970)
[3] CheckBox "Remember me" center=(1076,757)
[4] Button "Create account" center=(1355,613)
[5] Button "Sign in" center=(646,814)

Instruction: i forgot my password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(764,446)
[1] Slider "Brightness" center=(312,983)
[2] CheckBox "Wi-Fi" center=(828,146)
[3] CheckBox "Do Not Disturb" center=(1770,1019)
[4] Button "Network" center=(1547,874)
[5] Button "Displays" center=(1834,948)
[6] Button "Sound" center=(1675,340)
[7] PopUpButton "Appearance" center=(1296,623)
[8] CheckBox "Bluetooth" center=(208,371)

Instruction: click appearance

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1832,271)
[1] Button "Privacy & Security" center=(941,703)
[2] Button "Displays" center=(1702,261)
[3] Button "Network" center=(337,52)
[4] CheckBox "Wi-Fi" center=(647,289)
[5] Slider "Brightness" center=(82,918)

Instruction: tap network

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1218,949)
[1] Link "Account" center=(832,804)
[2] Button "Add to cart" center=(575,763)
[3] Button "Wishlist" center=(1671,412)
[4] Button "Buy now" center=(1685,860)
[5] Button "Filters" center=(579,481)
[6] SearchField "Search products" center=(798,413)

Instruction: select account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1225,171)
[1] Link "Account" center=(237,979)
[2] Button "Wishlist" center=(1646,559)
[3] Button "Filters" center=(1490,527)
[4] Button "Buy now" center=(1136,361)
[5] Button "Add to cart" center=(254,866)
[6] Button "Checkout" center=(1732,658)

Instruction: press wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(390,62)
[1] Button "Open" center=(1186,499)
[2] Button "New File" center=(1184,705)
[3] Button "Format" center=(157,957)
[4] Button "Commit" center=(751,528)

Instruction: click open

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1264,542)
[1] Button "Shuffle" center=(922,761)
[2] Button "Previous" center=(1605,1005)
[3] Slider "Volume" center=(313,445)
[4] Button "Fullscreen" center=(480,717)
[5] Button "Play" center=(965,357)
[6] Button "Next" center=(1142,753)

Instruction: press previous

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1288,94)
[1] Link "Account" center=(1498,812)
[2] Button "Checkout" center=(773,641)
[3] Button "Buy now" center=(285,833)
[4] Button "Wishlist" center=(185,668)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(100,704)
[1] Button "Play" center=(1053,786)
[2] Button "Next" center=(284,379)
[3] Slider "Volume" center=(222,136)
[4] Button "Fullscreen" center=(866,627)

Instruction: tap volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1488,480)
[1] Button "Format" center=(1840,298)
[2] Button "Find" center=(1063,132)
[3] Button "Save" center=(417,306)
[4] Button "Open" center=(862,245)
[5] TextField "Search files" center=(504,399)

Instruction: type into search files

[Assistant] {"action":"type","id":5,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(385,41)
[1] Button "Run inference" center=(1123,52)
[2] Button "Reload" center=(49,374)
[3] Button "Extensions" center=(827,353)
[4] Link "Docs" center=(96,859)
[5] SearchField "Search or enter address" center=(1203,616)

Instruction: press bookmark

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(806,627)
[1] Button "Network" center=(402,952)
[2] Button "Privacy & Security" center=(942,763)
[3] CheckBox "Wi-Fi" center=(1772,940)
[4] PopUpButton "Appearance" center=(1621,221)
[5] CheckBox "Bluetooth" center=(1079,192)
[6] Button "Displays" center=(1707,878)
[7] CheckBox "Do Not Disturb" center=(951,71)
[8] Slider "Brightness" center=(1314,79)

Instruction: select network

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1059,237)
[1] CheckBox "Remember me" center=(1275,198)
[2] Link "Forgot password" center=(1295,982)
[3] TextField "Password" center=(730,1000)
[4] Button "Sign in" center=(971,990)

Instruction: type into password

[Assistant] {"action":"type","id":3,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1666,453)
[1] Button "Run" center=(125,406)
[2] Button "Save" center=(160,978)
[3] Button "Format" center=(1023,870)
[4] Button "Commit" center=(1767,971)
[5] Button "Terminal" center=(1390,764)

Instruction: commit my changes

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(696,178)
[1] Button "Play" center=(957,821)
[2] Button "Fullscreen" center=(445,339)
[3] Button "Previous" center=(1322,476)
[4] Slider "Volume" center=(867,485)
[5] Button "Next" center=(812,124)

Instruction: click previous

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1618,740)
[1] Button "Sign in" center=(1471,96)
[2] TextField "Password" center=(1120,891)
[3] CheckBox "Remember me" center=(125,232)
[4] Button "Create account" center=(222,354)
[5] Link "Forgot password" center=(1867,641)

Instruction: sign up

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(760,1027)
[1] Button "Next" center=(1141,93)
[2] Button "Pause" center=(1727,885)
[3] Button "Shuffle" center=(1441,934)
[4] Button "Fullscreen" center=(928,796)

Instruction: skip to next track

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(463,624)
[1] Button "Network" center=(1351,667)
[2] Button "Displays" center=(71,86)
[3] CheckBox "Wi-Fi" center=(821,250)
[4] CheckBox "Do Not Disturb" center=(1451,159)
[5] Button "Privacy & Security" center=(634,44)
[6] PopUpButton "Appearance" center=(532,715)
[7] Slider "Brightness" center=(1706,977)

Instruction: tap displays

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1433,624)
[1] Button "Network" center=(1494,924)
[2] CheckBox "Do Not Disturb" center=(1322,357)
[3] Button "Displays" center=(459,1002)
[4] PopUpButton "Appearance" center=(1839,709)
[5] CheckBox "Bluetooth" center=(1548,975)
[6] CheckBox "Wi-Fi" center=(598,459)
[7] Button "Privacy & Security" center=(1453,50)

Instruction: toggle wifi

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1397,746)
[1] Button "Open" center=(496,112)
[2] Button "New File" center=(1345,157)
[3] Button "Terminal" center=(276,202)
[4] Button "Run" center=(1660,540)
[5] Button "Commit" center=(176,227)

Instruction: select open

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1792,623)
[1] Button "Pause" center=(575,151)
[2] Slider "Volume" center=(1598,162)
[3] Button "Next" center=(848,882)
[4] Button "Previous" center=(1853,945)
[5] Button "Fullscreen" center=(453,1013)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(417,125)
[1] PopUpButton "Appearance" center=(1601,750)
[2] Button "Sound" center=(1691,109)
[3] Button "Displays" center=(535,486)
[4] Slider "Brightness" center=(1300,419)
[5] Button "Privacy & Security" center=(937,796)
[6] Button "Network" center=(332,492)
[7] CheckBox "Wi-Fi" center=(1262,827)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(757,513)
[1] CheckBox "Do Not Disturb" center=(450,963)
[2] PopUpButton "Appearance" center=(1359,97)
[3] CheckBox "Bluetooth" center=(841,429)
[4] Button "Privacy & Security" center=(353,1009)
[5] Button "Displays" center=(637,557)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1272,581)
[1] Slider "Volume" center=(629,610)
[2] Button "Fullscreen" center=(369,186)
[3] Button "Play" center=(1160,1027)
[4] Button "Previous" center=(690,98)

Instruction: tap shuffle

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(118,186)
[1] Button "Wishlist" center=(252,740)
[2] Button "Checkout" center=(518,46)
[3] Button "Buy now" center=(820,43)
[4] Button "Filters" center=(1192,99)
[5] Button "Add to cart" center=(1706,436)

Instruction: press account

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1360,435)
[1] Button "Pause" center=(618,742)
[2] Button "Previous" center=(1301,436)
[3] Button "Fullscreen" center=(960,972)
[4] Button "Shuffle" center=(1572,283)
[5] Slider "Volume" center=(1438,542)
[6] Button "Play" center=(1294,296)

Instruction: tap volume

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(566,943)
[1] Button "Save" center=(368,416)
[2] Button "Run" center=(1555,61)
[3] Button "Commit" center=(924,864)
[4] Button "Open" center=(654,187)
[5] TextField "Search files" center=(127,510)
[6] Button "Find" center=(880,767)
[7] Button "Terminal" center=(1841,594)

Instruction: search

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(731,373)
[1] CheckBox "Do Not Disturb" center=(1402,448)
[2] CheckBox "Bluetooth" center=(312,780)
[3] Button "Displays" center=(1125,250)
[4] Button "Sound" center=(543,262)
[5] Slider "Brightness" center=(1499,235)
[6] Button "Network" center=(1204,239)

Instruction: select brightness

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(262,595)
[1] Button "Save" center=(1375,503)
[2] Button "Run" center=(655,966)
[3] Button "Open" center=(1745,1018)
[4] Button "Terminal" center=(1626,930)
[5] Button "Format" center=(1317,462)
[6] Button "New File" center=(144,244)
[7] Button "Commit" center=(484,385)

Instruction: execute

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1217,840)
[1] PopUpButton "Appearance" center=(1680,242)
[2] Button "Displays" center=(348,461)
[3] Button "Privacy & Security" center=(688,519)
[4] Button "Network" center=(543,928)
[5] CheckBox "Bluetooth" center=(1552,173)
[6] Button "Sound" center=(1601,993)
[7] CheckBox "Wi-Fi" center=(1036,363)

Instruction: click do not disturb

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(669,900)
[1] Button "Bookmark" center=(175,411)
[2] Link "Pricing" center=(1316,971)
[3] Button "New tab" center=(686,637)
[4] Button "Reload" center=(1736,497)
[5] Link "Sign in" center=(1012,386)
[6] Link "Docs" center=(1480,219)
[7] Button "Downloads" center=(633,350)

Instruction: press pricing

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1453,908)
[1] Button "Sound" center=(550,1039)
[2] CheckBox "Do Not Disturb" center=(272,719)
[3] Slider "Brightness" center=(340,971)
[4] Button "Network" center=(951,139)
[5] PopUpButton "Appearance" center=(1587,80)
[6] Button "Displays" center=(317,941)

Instruction: tap brightness

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1415,620)
[1] PopUpButton "Appearance" center=(744,946)
[2] Button "Privacy & Security" center=(735,127)
[3] CheckBox "Bluetooth" center=(1855,418)
[4] Slider "Brightness" center=(181,221)
[5] Button "Displays" center=(316,679)
[6] Button "Network" center=(654,733)
[7] Button "Sound" center=(846,978)

Instruction: select bluetooth

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1313,952)
[1] TextField "Password" center=(1248,467)
[2] Button "Create account" center=(386,413)
[3] Button "Sign in" center=(1783,298)
[4] TextField "Email" center=(433,565)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(598,807)
[1] Button "New tab" center=(1595,712)
[2] Button "Go forward" center=(978,715)
[3] Button "Extensions" center=(204,380)
[4] Link "Docs" center=(1169,72)

Instruction: go to docs

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(873,764)
[1] Button "Extensions" center=(740,175)
[2] Button "Go forward" center=(749,776)
[3] Link "Sign in" center=(575,605)
[4] Button "Reload" center=(435,540)
[5] Button "Go back" center=(616,313)
[6] Button "Run inference" center=(210,975)
[7] SearchField "Search or enter address" center=(961,980)
[8] Link "Pricing" center=(1415,126)
[9] Button "Bookmark" center=(1689,722)
[10] Link "Docs" center=(692,922)

Instruction: click extensions

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1808,117)
[1] Button "Displays" center=(59,780)
[2] Slider "Brightness" center=(67,794)
[3] Button "Privacy & Security" center=(1692,389)
[4] CheckBox "Do Not Disturb" center=(1822,970)

Instruction: tap displays

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1567,988)
[1] Button "Shuffle" center=(206,230)
[2] Button "Previous" center=(227,775)
[3] Slider "Volume" center=(227,43)
[4] Button "Play" center=(926,850)

Instruction: click shuffle

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1054,659)
[1] Button "Fullscreen" center=(1486,230)
[2] Button "Shuffle" center=(1000,334)
[3] Button "Pause" center=(1538,732)
[4] Button "Next" center=(1252,925)
[5] Button "Previous" center=(1803,102)
[6] Button "Play" center=(946,229)

Instruction: press volume

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(423,908)
[1] Button "Network" center=(698,814)
[2] Slider "Brightness" center=(506,206)
[3] CheckBox "Wi-Fi" center=(848,690)
[4] CheckBox "Bluetooth" center=(1752,646)
[5] Button "Privacy & Security" center=(647,655)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1584,205)
[1] Link "Forgot password" center=(172,103)
[2] CheckBox "Remember me" center=(168,545)
[3] TextField "Email" center=(744,964)
[4] TextField "Password" center=(1541,101)
[5] Button "Sign in" center=(1309,813)

Instruction: sign up

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(963,406)
[1] Button "Run inference" center=(1560,870)
[2] Button "Reload" center=(1023,419)
[3] Button "Go back" center=(1063,347)
[4] Button "New tab" center=(1603,540)
[5] Link "Docs" center=(681,599)
[6] Link "Pricing" center=(1565,234)
[7] Link "Sign in" center=(507,641)
[8] Button "Downloads" center=(389,83)
[9] Button "Go forward" center=(1454,401)

Instruction: sign in

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(821,931)
[1] Button "New File" center=(1334,192)
[2] Button "Save" center=(858,414)
[3] Button "Find" center=(68,484)
[4] Button "Run" center=(1660,877)
[5] Button "Commit" center=(1395,967)
[6] Button "Open" center=(393,194)

Instruction: press new file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1739,1018)
[1] Button "Run inference" center=(1788,602)
[2] Button "Downloads" center=(608,273)
[3] Button "Bookmark" center=(429,538)
[4] Button "Extensions" center=(1282,781)
[5] Button "Go forward" center=(1062,483)
[6] Link "Sign in" center=(931,1010)
[7] Button "New tab" center=(1784,203)
[8] Button "Reload" center=(1038,1032)

Instruction: click go forward

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(670,686)
[1] SearchField "Search or enter address" center=(364,386)
[2] Button "Extensions" center=(1331,358)
[3] Link "Docs" center=(1440,673)
[4] Button "Bookmark" center=(918,666)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":1,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(76,497)
[1] SearchField "Search products" center=(797,467)
[2] Button "Wishlist" center=(544,254)
[3] Button "Buy now" center=(708,744)
[4] Link "Account" center=(518,971)

Instruction: tap wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(699,840)
[1] TextField "Email" center=(1335,626)
[2] Button "Create account" center=(508,765)
[3] CheckBox "Remember me" center=(1254,664)
[4] Button "Sign in" center=(525,872)

Instruction: tap remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(960,438)
[1] Button "Extensions" center=(1632,237)
[2] Button "Run inference" center=(1660,828)
[3] Button "Bookmark" center=(605,768)
[4] Button "Go back" center=(333,412)
[5] Link "Pricing" center=(1776,836)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(683,659)
[1] Button "Privacy & Security" center=(1848,589)
[2] Slider "Brightness" center=(449,883)
[3] CheckBox "Bluetooth" center=(1700,701)
[4] Button "Network" center=(425,1020)
[5] Button "Displays" center=(1263,659)
[6] CheckBox "Do Not Disturb" center=(286,342)
[7] Button "Sound" center=(1542,102)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(91,69)
[1] Button "Network" center=(1374,699)
[2] Slider "Brightness" center=(1601,471)
[3] Button "Displays" center=(201,417)
[4] CheckBox "Do Not Disturb" center=(992,834)
[5] CheckBox "Wi-Fi" center=(342,497)
[6] Button "Sound" center=(1321,431)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(623,206)
[1] CheckBox "Remember me" center=(1241,171)
[2] TextField "Password" center=(701,1019)
[3] Button "Create account" center=(74,166)
[4] Button "Sign in" center=(147,928)
[5] TextField "Email" center=(708,605)

Instruction: select remember me

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(404,374)
[1] Button "Pause" center=(1295,896)
[2] Button "Next" center=(1597,75)
[3] Button "Fullscreen" center=(1358,329)
[4] Button "Play" center=(703,619)
[5] Slider "Volume" center=(1589,622)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1585,408)
[1] Button "New tab" center=(771,386)
[2] SearchField "Search or enter address" center=(206,462)
[3] Button "Downloads" center=(743,808)
[4] Button "Go back" center=(182,544)
[5] Link "Docs" center=(860,598)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(276,821)
[1] Slider "Volume" center=(1819,307)
[2] Button "Shuffle" center=(505,285)
[3] Button "Play" center=(155,700)
[4] Button "Pause" center=(1439,611)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(833,814)
[1] Button "Run" center=(1339,726)
[2] Button "New File" center=(850,59)
[3] Button "Find" center=(1084,823)
[4] Button "Format" center=(633,544)
[5] Button "Commit" center=(88,725)
[6] Button "Open" center=(1503,753)

Instruction: click terminal

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1845,416)
[1] Button "Sound" center=(1082,960)
[2] Button "Privacy & Security" center=(1737,208)
[3] PopUpButton "Appearance" center=(853,257)
[4] Slider "Brightness" center=(351,589)
[5] Button "Displays" center=(55,977)
[6] CheckBox "Bluetooth" center=(1325,802)
[7] Button "Network" center=(1315,328)
[8] CheckBox "Wi-Fi" center=(584,609)

Instruction: turn off bluetooth

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1372,332)
[1] Button "Wishlist" center=(964,69)
[2] SearchField "Search products" center=(952,657)
[3] Link "Account" center=(1374,727)
[4] Button "Buy now" center=(1088,828)

Instruction: click wishlist

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1366,681)
[1] Link "Sign in" center=(510,620)
[2] Link "Pricing" center=(1863,945)
[3] Button "Reload" center=(1767,298)
[4] SearchField "Search or enter address" center=(484,563)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1583,209)
[1] Button "Displays" center=(1046,531)
[2] Button "Privacy & Security" center=(1188,651)
[3] PopUpButton "Appearance" center=(624,854)
[4] Slider "Brightness" center=(1406,917)
[5] Button "Network" center=(548,119)
[6] CheckBox "Bluetooth" center=(1302,149)
[7] Button "Sound" center=(1851,40)
[8] CheckBox "Do Not Disturb" center=(1786,71)

Instruction: click wi-fi

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(480,701)
[1] Button "Sign in" center=(430,1034)
[2] TextField "Email" center=(906,799)
[3] TextField "Password" center=(1834,421)
[4] CheckBox "Remember me" center=(195,326)
[5] Button "Create account" center=(895,709)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(216,782)
[1] SearchField "Search or enter address" center=(520,506)
[2] Button "Run inference" center=(1821,947)
[3] Link "Sign in" center=(1227,751)
[4] Link "Docs" center=(1181,319)
[5] Button "Downloads" center=(593,63)

Instruction: i want to log in

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1740,516)
[1] Slider "Brightness" center=(884,219)
[2] Button "Displays" center=(67,630)
[3] CheckBox "Wi-Fi" center=(1877,85)
[4] Button "Privacy & Security" center=(1384,104)
[5] Button "Sound" center=(1616,937)
[6] PopUpButton "Appearance" center=(615,766)
[7] CheckBox "Bluetooth" center=(616,776)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1505,908)
[1] CheckBox "Bluetooth" center=(1560,88)
[2] CheckBox "Do Not Disturb" center=(626,412)
[3] PopUpButton "Appearance" center=(221,652)
[4] Button "Sound" center=(792,507)
[5] CheckBox "Wi-Fi" center=(1720,523)

Instruction: select do not disturb

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(396,634)
[1] Button "Shuffle" center=(862,421)
[2] Button "Previous" center=(997,395)
[3] Button "Next" center=(1611,339)
[4] Button "Pause" center=(342,873)
[5] Button "Fullscreen" center=(1417,809)

Instruction: make it full screen

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(743,558)
[1] Button "Save" center=(1800,680)
[2] Button "Commit" center=(337,653)
[3] Button "Run" center=(739,593)
[4] Button "Format" center=(1335,891)

Instruction: press format

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(48,300)
[1] Button "Sound" center=(1167,720)
[2] CheckBox "Do Not Disturb" center=(377,797)
[3] Button "Displays" center=(504,740)
[4] CheckBox "Wi-Fi" center=(1493,846)
[5] Button "Network" center=(725,540)
[6] PopUpButton "Appearance" center=(279,373)
[7] CheckBox "Bluetooth" center=(1538,896)
[8] Button "Privacy & Security" center=(1672,422)

Instruction: tap appearance

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1869,852)
[1] CheckBox "Do Not Disturb" center=(1073,859)
[2] Slider "Brightness" center=(260,547)
[3] Button "Network" center=(335,977)
[4] CheckBox "Bluetooth" center=(187,751)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(103,342)
[1] Button "Sign in" center=(287,252)
[2] Button "Create account" center=(420,887)
[3] TextField "Email" center=(1629,736)
[4] Link "Forgot password" center=(186,108)

Instruction: tap remember me

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(834,423)
[1] TextField "Password" center=(466,351)
[2] Button "Sign in" center=(917,674)
[3] TextField "Email" center=(1247,981)
[4] Button "Create account" center=(1019,690)
[5] CheckBox "Remember me" center=(1299,647)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(95,429)
[1] Button "Reload" center=(119,1025)
[2] Button "Go forward" center=(1038,817)
[3] Link "Sign in" center=(217,526)
[4] Link "Docs" center=(556,596)
[5] Button "Bookmark" center=(228,124)
[6] Button "Run inference" center=(542,781)
[7] Button "New tab" center=(1741,772)
[8] Button "Go back" center=(1332,984)
[9] SearchField "Search or enter address" center=(232,505)
[10] Button "Downloads" center=(817,807)
[11] Link "Pricing" center=(730,287)

Instruction: click new tab

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(61,740)
[1] TextField "Password" center=(302,647)
[2] TextField "Email" center=(1806,731)
[3] Button "Create account" center=(1639,68)
[4] Link "Forgot password" center=(1076,495)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1045,218)
[1] Button "Format" center=(965,340)
[2] Button "Run" center=(1788,206)
[3] Button "New File" center=(121,337)
[4] Button "Save" center=(1271,379)
[5] TextField "Search files" center=(913,152)
[6] Button "Terminal" center=(1446,198)
[7] Button "Commit" center=(1754,118)
[8] Button "Find" center=(447,533)

Instruction: tap open

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(497,388)
[1] Slider "Volume" center=(1812,739)
[2] Button "Play" center=(1235,905)
[3] Button "Previous" center=(1050,274)
[4] Button "Pause" center=(1501,495)

Instruction: skip to next track

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(257,468)
[1] CheckBox "Bluetooth" center=(1010,74)
[2] Button "Sound" center=(611,486)
[3] PopUpButton "Appearance" center=(1181,758)
[4] Button "Privacy & Security" center=(904,108)
[5] CheckBox "Do Not Disturb" center=(173,129)
[6] Button "Network" center=(578,938)
[7] Slider "Brightness" center=(1401,406)
[8] CheckBox "Wi-Fi" center=(1012,450)

Instruction: press appearance

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(176,1034)
[1] Button "Run inference" center=(1657,712)
[2] Link "Docs" center=(419,636)
[3] Link "Sign in" center=(886,158)
[4] Button "New tab" center=(1186,80)
[5] Button "Downloads" center=(515,247)
[6] Link "Pricing" center=(341,347)
[7] Button "Bookmark" center=(312,804)
[8] SearchField "Search or enter address" center=(1476,440)
[9] Button "Go forward" center=(62,420)
[10] Button "Reload" center=(162,562)
[11] Button "Go back" center=(1654,52)

Instruction: refresh the page

[Assistant] {"action":"click","id":10}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1399,526)
[1] TextField "Email" center=(1019,530)
[2] Button "Create account" center=(478,141)
[3] CheckBox "Remember me" center=(393,437)
[4] Button "Sign in" center=(1172,688)

Instruction: i forgot my password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1323,811)
[1] SearchField "Search products" center=(886,899)
[2] Button "Checkout" center=(1321,410)
[3] Link "Account" center=(501,887)
[4] Button "Wishlist" center=(577,456)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(1299,1012)
[1] Button "Run inference" center=(1658,561)
[2] Button "Reload" center=(1860,456)
[3] Button "Go forward" center=(1363,932)
[4] Button "New tab" center=(1587,690)
[5] SearchField "Search or enter address" center=(362,287)
[6] Link "Sign in" center=(1266,573)
[7] Button "Extensions" center=(629,549)
[8] Button "Bookmark" center=(1623,116)
[9] Button "Go back" center=(926,654)
[10] Link "Docs" center=(103,467)
[11] Button "Downloads" center=(175,615)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1692,979)
[1] Slider "Volume" center=(739,350)
[2] Button "Pause" center=(1752,588)
[3] Button "Previous" center=(1279,374)
[4] Button "Shuffle" center=(671,915)
[5] Button "Next" center=(1002,563)
[6] Button "Play" center=(1038,435)

Instruction: play

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1831,335)
[1] Button "Find" center=(1257,565)
[2] Button "Terminal" center=(506,360)
[3] Button "Save" center=(703,968)
[4] Button "New File" center=(530,619)
[5] Button "Run" center=(431,396)
[6] TextField "Search files" center=(667,236)

Instruction: tap new file

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1682,828)
[1] TextField "Email" center=(1449,342)
[2] CheckBox "Remember me" center=(429,743)
[3] Button "Create account" center=(1444,410)
[4] Link "Forgot password" center=(1499,595)

Instruction: type into email

[Assistant] {"action":"type","id":1,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1857,703)
[1] Button "Run" center=(1698,548)
[2] Button "Commit" center=(807,291)
[3] Button "Open" center=(842,535)
[4] Button "Terminal" center=(1508,317)
[5] Button "Format" center=(1203,510)

Instruction: run the code

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1702,996)
[1] Button "Sign in" center=(561,990)
[2] CheckBox "Remember me" center=(1053,505)
[3] Link "Forgot password" center=(1167,637)
[4] Button "Create account" center=(1721,323)

Instruction: i forgot my password

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(291,410)
[1] Button "Find" center=(1596,860)
[2] Button "Save" center=(151,82)
[3] Button "New File" center=(1775,111)
[4] Button "Format" center=(608,150)
[5] Button "Commit" center=(1163,588)

Instruction: type into search files

[Assistant] {"action":"type","id":0,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(268,385)
[1] Button "Fullscreen" center=(1318,776)
[2] Button "Next" center=(607,331)
[3] Button "Play" center=(920,286)
[4] Button "Pause" center=(512,265)

Instruction: select pause

[Assistant] {"action":"click","id":4}