[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(311,47)
[1] Link "Account" center=(242,468)
[2] Button "Filters" center=(700,404)
[3] Button "Wishlist" center=(692,790)
[4] Button "Buy now" center=(960,740)
[5] SearchField "Search products" center=(1077,456)

Instruction: tap filters

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(244,1029)
[1] Button "Shuffle" center=(392,300)
[2] Button "Pause" center=(1003,481)
[3] Button "Fullscreen" center=(602,45)
[4] Button "Next" center=(54,93)
[5] Button "Previous" center=(831,576)

Instruction: tap next

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(823,401)
[1] Button "Wishlist" center=(562,372)
[2] Link "Account" center=(258,197)
[3] Button "Add to cart" center=(398,953)
[4] Button "Checkout" center=(565,922)
[5] Button "Buy now" center=(659,306)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1123,456)
[1] Button "Filters" center=(1134,476)
[2] Link "Account" center=(74,564)
[3] Button "Add to cart" center=(1628,278)
[4] Button "Checkout" center=(1753,365)

Instruction: click buy now

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1089,698)
[1] Button "Find" center=(882,102)
[2] Button "Format" center=(395,86)
[3] Button "New File" center=(1672,750)
[4] TextField "Search files" center=(721,788)
[5] Button "Open" center=(139,593)

Instruction: type into search files

[Assistant] {"action":"type","id":4,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1617,483)
[1] Button "Privacy & Security" center=(1346,107)
[2] Button "Network" center=(1726,661)
[3] CheckBox "Do Not Disturb" center=(1014,650)
[4] CheckBox "Bluetooth" center=(1032,928)
[5] PopUpButton "Appearance" center=(1769,981)
[6] Button "Sound" center=(1721,956)
[7] Button "Displays" center=(1219,938)
[8] CheckBox "Wi-Fi" center=(1037,178)

Instruction: press appearance

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(66,345)
[1] Button "Wishlist" center=(690,976)
[2] SearchField "Search products" center=(126,515)
[3] Button "Filters" center=(459,402)
[4] Button "Buy now" center=(1583,828)
[5] Button "Add to cart" center=(813,809)
[6] Button "Checkout" center=(1455,543)

Instruction: click buy now

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1237,888)
[1] Button "Add to cart" center=(1326,819)
[2] Link "Account" center=(1163,86)
[3] Button "Filters" center=(327,713)
[4] Button "Buy now" center=(925,41)
[5] Button "Wishlist" center=(385,774)

Instruction: press account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1106,327)
[1] Button "Previous" center=(1290,901)
[2] Slider "Volume" center=(1823,57)
[3] Button "Shuffle" center=(999,235)
[4] Button "Pause" center=(1769,359)
[5] Button "Play" center=(354,81)

Instruction: tap pause

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(788,1018)
[1] Slider "Volume" center=(272,712)
[2] Button "Pause" center=(672,108)
[3] Button "Shuffle" center=(1091,960)
[4] Button "Next" center=(911,795)
[5] Button "Fullscreen" center=(587,970)
[6] Button "Previous" center=(1446,575)

Instruction: click volume

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1093,865)
[1] Button "Filters" center=(606,148)
[2] Button "Wishlist" center=(1208,470)
[3] Button "Checkout" center=(365,290)
[4] Button "Buy now" center=(1364,824)
[5] Link "Account" center=(490,212)
[6] SearchField "Search products" center=(1865,479)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1757,532)
[1] Button "Pause" center=(639,990)
[2] Button "Play" center=(1097,254)
[3] Button "Next" center=(242,909)
[4] Button "Fullscreen" center=(948,237)
[5] Button "Previous" center=(1473,42)

Instruction: press pause

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1147,552)
[1] Button "Previous" center=(346,341)
[2] Slider "Volume" center=(1635,190)
[3] Button "Next" center=(1855,430)
[4] Button "Play" center=(1726,410)
[5] Button "Fullscreen" center=(1722,388)

Instruction: skip to next track

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(962,679)
[1] Button "Format" center=(1069,1008)
[2] TextField "Search files" center=(1226,67)
[3] Button "Find" center=(350,945)
[4] Button "Commit" center=(390,273)

Instruction: make a commit

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(954,634)
[1] TextField "Password" center=(701,760)
[2] TextField "Email" center=(571,112)
[3] Button "Create account" center=(1324,850)
[4] Link "Forgot password" center=(951,470)
[5] Button "Sign in" center=(1056,361)

Instruction: i forgot my password

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(158,611)
[1] Button "Commit" center=(1229,349)
[2] Button "Open" center=(796,562)
[3] Button "Format" center=(1286,820)
[4] Button "New File" center=(1498,682)
[5] Button "Save" center=(839,211)
[6] Button "Find" center=(869,886)
[7] Button "Terminal" center=(1166,255)
[8] Button "Run" center=(1075,309)

Instruction: tap terminal

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(527,647)
[1] Button "Displays" center=(857,737)
[2] Button "Sound" center=(262,224)
[3] Button "Network" center=(274,998)
[4] Slider "Brightness" center=(1333,845)
[5] CheckBox "Bluetooth" center=(1407,657)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1645,820)
[1] Button "Pause" center=(241,563)
[2] Button "Previous" center=(1610,177)
[3] Button "Fullscreen" center=(1710,867)
[4] Button "Shuffle" center=(94,704)
[5] Slider "Volume" center=(1734,540)
[6] Button "Play" center=(54,977)

Instruction: click volume

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(602,967)
[1] Button "Extensions" center=(1486,563)
[2] Link "Docs" center=(1397,992)
[3] Button "Downloads" center=(1779,487)
[4] Button "New tab" center=(1394,351)
[5] Button "Go forward" center=(1284,963)
[6] SearchField "Search or enter address" center=(970,983)
[7] Button "Bookmark" center=(134,288)

Instruction: sign in

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(67,859)
[1] Button "New File" center=(1037,492)
[2] Button "Run" center=(523,797)
[3] Button "Format" center=(247,824)
[4] Button "Open" center=(389,900)
[5] Button "Find" center=(1689,713)
[6] Button "Terminal" center=(1880,940)
[7] Button "Commit" center=(486,142)

Instruction: search

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(600,767)
[1] Button "Find" center=(1723,326)
[2] Button "New File" center=(579,85)
[3] Button "Run" center=(677,416)
[4] TextField "Search files" center=(1082,590)
[5] Button "Terminal" center=(468,147)
[6] Button "Open" center=(196,358)
[7] Button "Format" center=(170,355)
[8] Button "Save" center=(1858,534)

Instruction: press find

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(704,682)
[1] Button "Shuffle" center=(1576,442)
[2] Button "Previous" center=(1717,996)
[3] Button "Pause" center=(1630,586)
[4] Button "Play" center=(1272,116)

Instruction: start playback

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(586,405)
[1] Button "Filters" center=(913,564)
[2] Link "Account" center=(637,309)
[3] Button "Buy now" center=(883,1039)
[4] Button "Add to cart" center=(1008,628)
[5] SearchField "Search products" center=(1699,340)
[6] Button "Wishlist" center=(90,976)

Instruction: tap filters

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(315,667)
[1] Button "Pause" center=(693,531)
[2] Button "Shuffle" center=(487,816)
[3] Button "Play" center=(1061,558)
[4] Button "Next" center=(1395,253)

Instruction: tap shuffle

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(253,561)
[1] CheckBox "Remember me" center=(845,1024)
[2] Button "Sign in" center=(1185,832)
[3] Link "Forgot password" center=(1561,253)
[4] TextField "Password" center=(1495,78)

Instruction: type into email

[Assistant] {"action":"type","id":0,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(504,219)
[1] TextField "Email" center=(654,132)
[2] Button "Create account" center=(809,915)
[3] CheckBox "Remember me" center=(1548,421)
[4] Button "Sign in" center=(181,603)

Instruction: sign up

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(401,784)
[1] Button "Find" center=(290,393)
[2] Button "Commit" center=(1561,687)
[3] Button "Run" center=(1212,405)
[4] Button "Format" center=(1155,330)
[5] Button "New File" center=(1371,124)
[6] Button "Open" center=(1324,538)
[7] Button "Terminal" center=(588,798)

Instruction: search

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(1865,141)
[1] Link "Docs" center=(506,846)
[2] Link "Sign in" center=(1603,347)
[3] Button "Bookmark" center=(1755,320)
[4] SearchField "Search or enter address" center=(1407,531)
[5] Button "Extensions" center=(831,755)
[6] Button "New tab" center=(1861,99)
[7] Button "Go forward" center=(1484,644)
[8] Button "Run inference" center=(562,876)

Instruction: tap go forward

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1400,692)
[1] Button "Create account" center=(534,1034)
[2] Button "Sign in" center=(452,829)
[3] Link "Forgot password" center=(98,871)
[4] TextField "Password" center=(350,474)
[5] TextField "Email" center=(1786,689)

Instruction: type into email

[Assistant] {"action":"type","id":5,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(292,654)
[1] Link "Pricing" center=(1568,426)
[2] Button "Downloads" center=(890,463)
[3] Button "Bookmark" center=(931,1015)
[4] Button "Go forward" center=(199,41)
[5] Button "New tab" center=(325,360)
[6] Button "Go back" center=(1569,443)
[7] Button "Reload" center=(532,324)
[8] Link "Docs" center=(396,870)
[9] Button "Extensions" center=(105,177)
[10] SearchField "Search or enter address" center=(1664,909)
[11] Button "Run inference" center=(1756,171)

Instruction: press run inference

[Assistant] {"action":"click","id":11}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(907,607)
[1] Button "Sign in" center=(492,952)
[2] Button "Create account" center=(902,539)
[3] TextField "Email" center=(773,672)
[4] CheckBox "Remember me" center=(1556,56)
[5] Link "Forgot password" center=(232,525)

Instruction: click remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(641,67)
[1] TextField "Email" center=(1303,208)
[2] CheckBox "Remember me" center=(472,291)
[3] Link "Forgot password" center=(1631,918)
[4] Button "Sign in" center=(757,104)
[5] TextField "Password" center=(977,351)

Instruction: tap remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1042,809)
[1] CheckBox "Remember me" center=(302,428)
[2] Link "Forgot password" center=(486,108)
[3] TextField "Email" center=(1202,638)
[4] Button "Create account" center=(1642,808)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1443,489)
[1] Button "Add to cart" center=(809,911)
[2] SearchField "Search products" center=(1843,309)
[3] Button "Buy now" center=(1702,134)
[4] Button "Filters" center=(651,662)
[5] Button "Checkout" center=(132,66)

Instruction: select filters

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(328,224)
[1] Link "Sign in" center=(1132,972)
[2] Link "Pricing" center=(1137,274)
[3] Button "New tab" center=(1212,40)
[4] Button "Run inference" center=(655,520)
[5] Button "Extensions" center=(1586,286)
[6] Link "Docs" center=(518,107)
[7] Button "Bookmark" center=(557,987)
[8] Button "Go forward" center=(779,322)
[9] SearchField "Search or enter address" center=(814,788)
[10] Button "Downloads" center=(1553,752)
[11] Button "Reload" center=(211,422)

Instruction: press run inference

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1727,364)
[1] Slider "Brightness" center=(477,105)
[2] PopUpButton "Appearance" center=(314,935)
[3] Button "Network" center=(1449,48)
[4] Button "Displays" center=(632,54)
[5] CheckBox "Do Not Disturb" center=(1234,369)
[6] CheckBox "Wi-Fi" center=(624,289)
[7] CheckBox "Bluetooth" center=(942,547)
[8] Button "Sound" center=(1372,485)

Instruction: press sound

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(462,636)
[1] Button "Reload" center=(212,372)
[2] Button "Run inference" center=(228,1000)
[3] Button "Bookmark" center=(172,296)
[4] Button "New tab" center=(508,483)
[5] Link "Docs" center=(1025,587)
[6] Link "Pricing" center=(884,214)
[7] Button "Go back" center=(1332,146)
[8] SearchField "Search or enter address" center=(1678,55)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":8,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1598,505)
[1] Button "Extensions" center=(1758,94)
[2] Button "Run inference" center=(872,697)
[3] Button "Bookmark" center=(1162,707)
[4] Button "Reload" center=(1755,42)
[5] Button "New tab" center=(332,924)
[6] Button "Downloads" center=(916,881)
[7] Link "Docs" center=(1224,712)
[8] Button "Go back" center=(445,290)
[9] Button "Go forward" center=(1696,778)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1068,482)
[1] CheckBox "Do Not Disturb" center=(1830,383)
[2] Button "Displays" center=(1365,747)
[3] CheckBox "Bluetooth" center=(726,971)
[4] Button "Sound" center=(1464,59)
[5] Slider "Brightness" center=(1422,374)

Instruction: press do not disturb

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(866,458)
[1] Button "Create account" center=(1023,77)
[2] TextField "Password" center=(251,362)
[3] Link "Forgot password" center=(194,302)
[4] TextField "Email" center=(694,796)

Instruction: sign up

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(609,78)
[1] Button "New File" center=(104,367)
[2] Button "Save" center=(515,929)
[3] Button "Open" center=(1242,173)
[4] Button "Find" center=(417,720)
[5] Button "Format" center=(400,88)

Instruction: click new file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(434,273)
[1] Button "Bookmark" center=(1354,218)
[2] Button "Extensions" center=(784,225)
[3] Button "New tab" center=(1098,427)
[4] Button "Reload" center=(279,855)
[5] Link "Pricing" center=(479,202)

Instruction: reload

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(858,345)
[1] Button "Play" center=(1622,863)
[2] Button "Previous" center=(650,664)
[3] Slider "Volume" center=(946,478)
[4] Button "Pause" center=(1664,45)
[5] Button "Fullscreen" center=(394,310)
[6] Button "Next" center=(1235,564)

Instruction: play

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1412,524)
[1] Button "Shuffle" center=(1320,966)
[2] Button "Previous" center=(1815,880)
[3] Button "Pause" center=(536,323)
[4] Button "Next" center=(1395,626)
[5] Slider "Volume" center=(1166,692)
[6] Button "Fullscreen" center=(1320,409)

Instruction: press pause

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(867,419)
[1] Link "Forgot password" center=(1707,380)
[2] CheckBox "Remember me" center=(1192,907)
[3] Button "Sign in" center=(175,313)
[4] TextField "Email" center=(1039,822)
[5] Button "Create account" center=(1877,439)

Instruction: press remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(585,524)
[1] Button "Next" center=(1643,418)
[2] Button "Pause" center=(488,295)
[3] Button "Previous" center=(868,909)
[4] Button "Fullscreen" center=(1231,754)

Instruction: tap pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(466,614)
[1] TextField "Email" center=(1228,755)
[2] Link "Forgot password" center=(1451,756)
[3] Button "Sign in" center=(875,170)
[4] CheckBox "Remember me" center=(1600,689)
[5] Button "Create account" center=(1843,824)

Instruction: press forgot password

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(251,386)
[1] TextField "Email" center=(1012,123)
[2] Button "Sign in" center=(360,696)
[3] Link "Forgot password" center=(977,924)
[4] Button "Create account" center=(1360,754)
[5] TextField "Password" center=(896,1013)

Instruction: type into password

[Assistant] {"action":"type","id":5,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1163,376)
[1] Button "Wishlist" center=(1064,807)
[2] SearchField "Search products" center=(918,292)
[3] Button "Filters" center=(1409,631)
[4] Link "Account" center=(93,462)
[5] Button "Checkout" center=(1647,143)

Instruction: press account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(236,402)
[1] TextField "Email" center=(1802,269)
[2] Button "Sign in" center=(1419,1009)
[3] TextField "Password" center=(1064,125)
[4] Button "Create account" center=(247,443)
[5] Link "Forgot password" center=(1876,840)

Instruction: type into email

[Assistant] {"action":"type","id":1,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1010,173)
[1] Button "Fullscreen" center=(817,100)
[2] Slider "Volume" center=(1005,747)
[3] Button "Pause" center=(380,833)
[4] Button "Shuffle" center=(1740,831)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(831,789)
[1] TextField "Email" center=(603,627)
[2] TextField "Password" center=(465,160)
[3] Button "Create account" center=(309,165)
[4] CheckBox "Remember me" center=(624,447)

Instruction: select remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1765,759)
[1] Button "Save" center=(647,392)
[2] Button "Find" center=(1577,455)
[3] Button "Open" center=(460,979)
[4] TextField "Search files" center=(1711,235)
[5] Button "Format" center=(1264,151)
[6] Button "Terminal" center=(982,182)
[7] Button "New File" center=(1572,993)

Instruction: type into search files

[Assistant] {"action":"type","id":4,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1201,511)
[1] Link "Sign in" center=(221,675)
[2] Button "Reload" center=(879,502)
[3] Button "Extensions" center=(1416,66)
[4] Link "Pricing" center=(883,738)
[5] Button "Go back" center=(1860,603)
[6] Button "Downloads" center=(499,263)
[7] Button "Go forward" center=(713,265)
[8] Button "New tab" center=(1268,153)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(438,271)
[1] Link "Sign in" center=(514,318)
[2] Link "Docs" center=(596,260)
[3] Button "Extensions" center=(1015,979)
[4] Link "Pricing" center=(1403,469)
[5] SearchField "Search or enter address" center=(1746,533)

Instruction: select sign in

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1531,103)
[1] Button "Buy now" center=(185,925)
[2] SearchField "Search products" center=(929,545)
[3] Link "Account" center=(1269,261)
[4] Button "Checkout" center=(117,115)

Instruction: go to checkout

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(283,453)
[1] Button "Extensions" center=(1154,73)
[2] Button "Bookmark" center=(1571,1027)
[3] Link "Docs" center=(188,937)
[4] Link "Sign in" center=(1311,656)
[5] Button "Reload" center=(121,292)
[6] Link "Pricing" center=(637,187)
[7] Button "Go back" center=(289,1017)
[8] Button "Go forward" center=(1771,828)
[9] SearchField "Search or enter address" center=(450,304)
[10] Button "Run inference" center=(1067,956)

Instruction: reload

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1414,138)
[1] Button "Sound" center=(1034,897)
[2] Button "Displays" center=(1251,142)
[3] Slider "Brightness" center=(460,982)
[4] PopUpButton "Appearance" center=(1153,431)
[5] CheckBox "Bluetooth" center=(160,121)
[6] CheckBox "Do Not Disturb" center=(703,846)
[7] Button "Network" center=(1024,141)
[8] CheckBox "Wi-Fi" center=(1791,758)

Instruction: click network

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1410,928)
[1] Slider "Brightness" center=(48,205)
[2] Button "Network" center=(1106,608)
[3] CheckBox "Wi-Fi" center=(1684,710)
[4] PopUpButton "Appearance" center=(220,336)
[5] Button "Sound" center=(228,162)
[6] Button "Displays" center=(311,900)
[7] CheckBox "Bluetooth" center=(1132,510)

Instruction: select appearance

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(819,341)
[1] CheckBox "Do Not Disturb" center=(1801,478)
[2] Button "Sound" center=(424,659)
[3] Button "Network" center=(186,284)
[4] PopUpButton "Appearance" center=(1174,847)
[5] Button "Displays" center=(203,78)
[6] Button "Privacy & Security" center=(1344,551)
[7] CheckBox "Bluetooth" center=(657,669)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(727,263)
[1] Button "Displays" center=(718,92)
[2] PopUpButton "Appearance" center=(881,651)
[3] Button "Privacy & Security" center=(1398,483)
[4] Button "Sound" center=(1395,229)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1869,116)
[1] Button "Create account" center=(44,182)
[2] Link "Forgot password" center=(1336,412)
[3] CheckBox "Remember me" center=(619,538)
[4] Button "Sign in" center=(1515,868)
[5] TextField "Password" center=(173,996)

Instruction: sign in

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(618,558)
[1] Button "Wishlist" center=(1207,1012)
[2] Button "Buy now" center=(1264,266)
[3] Button "Filters" center=(159,71)
[4] Link "Account" center=(796,148)
[5] SearchField "Search products" center=(1250,683)
[6] Button "Checkout" center=(1475,788)

Instruction: select filters

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1784,228)
[1] Button "Shuffle" center=(958,693)
[2] Button "Fullscreen" center=(1523,128)
[3] Slider "Volume" center=(973,199)
[4] Button "Next" center=(838,453)

Instruction: go fullscreen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(642,919)
[1] Button "Sign in" center=(1379,983)
[2] Link "Forgot password" center=(602,153)
[3] CheckBox "Remember me" center=(1757,267)
[4] TextField "Email" center=(50,878)

Instruction: press remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1346,872)
[1] Button "Sign in" center=(883,458)
[2] CheckBox "Remember me" center=(1148,653)
[3] Button "Create account" center=(160,728)
[4] Link "Forgot password" center=(596,1035)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(759,931)
[1] Button "Terminal" center=(1663,715)
[2] Button "Commit" center=(370,183)
[3] Button "Open" center=(509,186)
[4] Button "Format" center=(1170,191)

Instruction: press format

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1117,237)
[1] Button "Create account" center=(773,660)
[2] TextField "Password" center=(1250,649)
[3] Button "Sign in" center=(648,72)
[4] Link "Forgot password" center=(42,319)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1482,158)
[1] TextField "Password" center=(544,395)
[2] CheckBox "Remember me" center=(1406,70)
[3] TextField "Email" center=(1731,913)
[4] Link "Forgot password" center=(1625,361)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1179,81)
[1] CheckBox "Bluetooth" center=(1266,817)
[2] Button "Sound" center=(991,576)
[3] Slider "Brightness" center=(532,69)
[4] Button "Network" center=(301,988)
[5] CheckBox "Wi-Fi" center=(1146,57)
[6] Button "Privacy & Security" center=(1783,114)
[7] PopUpButton "Appearance" center=(88,956)

Instruction: select network

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1001,609)
[1] Button "Privacy & Security" center=(879,859)
[2] Button "Sound" center=(970,334)
[3] CheckBox "Wi-Fi" center=(1540,944)
[4] Slider "Brightness" center=(1327,171)
[5] CheckBox "Do Not Disturb" center=(1167,77)
[6] PopUpButton "Appearance" center=(398,127)

Instruction: press do not disturb

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(494,883)
[1] Button "New tab" center=(1393,675)
[2] Button "Reload" center=(1525,966)
[3] Link "Sign in" center=(1511,495)
[4] SearchField "Search or enter address" center=(553,993)
[5] Button "Go forward" center=(530,881)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":4,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1253,625)
[1] Button "Wishlist" center=(666,754)
[2] Button "Buy now" center=(1613,61)
[3] SearchField "Search products" center=(1155,817)
[4] Link "Account" center=(609,231)
[5] Button "Checkout" center=(1219,956)

Instruction: find a product

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(812,108)
[1] Button "Buy now" center=(1530,89)
[2] Button "Filters" center=(835,379)
[3] SearchField "Search products" center=(900,510)
[4] Button "Checkout" center=(532,279)

Instruction: select filters

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(896,719)
[1] TextField "Search files" center=(1187,110)
[2] Button "Find" center=(652,641)
[3] Button "Commit" center=(1880,605)
[4] Button "Terminal" center=(159,183)
[5] Button "New File" center=(772,265)

Instruction: select terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(981,832)
[1] CheckBox "Do Not Disturb" center=(1754,550)
[2] Button "Sound" center=(1800,971)
[3] Button "Network" center=(409,370)
[4] CheckBox "Bluetooth" center=(1683,937)
[5] PopUpButton "Appearance" center=(1702,49)
[6] Button "Displays" center=(1128,328)

Instruction: press do not disturb

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(600,121)
[1] TextField "Email" center=(970,766)
[2] Button "Create account" center=(298,355)
[3] Button "Sign in" center=(266,777)
[4] CheckBox "Remember me" center=(717,390)

Instruction: submit the login

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(396,894)
[1] Button "Add to cart" center=(1061,616)
[2] SearchField "Search products" center=(1696,427)
[3] Link "Account" center=(70,206)
[4] Button "Wishlist" center=(608,208)
[5] Button "Filters" center=(795,503)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(699,695)
[1] Button "Network" center=(1141,983)
[2] CheckBox "Wi-Fi" center=(1699,182)
[3] Button "Privacy & Security" center=(1115,436)
[4] PopUpButton "Appearance" center=(503,87)

Instruction: tap network

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(324,889)
[1] Button "Find" center=(308,67)
[2] Button "Format" center=(658,129)
[3] Button "Commit" center=(874,515)
[4] Button "Save" center=(1149,989)
[5] Button "Terminal" center=(1615,221)
[6] Button "Run" center=(284,407)
[7] Button "Open" center=(1441,286)
[8] Button "New File" center=(1249,220)

Instruction: execute

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1128,838)
[1] Link "Sign in" center=(1722,406)
[2] Button "Run inference" center=(55,229)
[3] Button "New tab" center=(349,86)
[4] Button "Extensions" center=(896,137)
[5] Link "Pricing" center=(1415,639)
[6] Button "Go forward" center=(139,148)
[7] Button "Go back" center=(1220,285)
[8] Button "Downloads" center=(1356,289)

Instruction: tap new tab

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(84,228)
[1] Button "Checkout" center=(1491,635)
[2] Button "Filters" center=(1367,389)
[3] Link "Account" center=(921,141)
[4] SearchField "Search products" center=(811,189)
[5] Button "Add to cart" center=(417,901)
[6] Button "Wishlist" center=(737,425)

Instruction: add to cart

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1665,838)
[1] Button "Reload" center=(744,784)
[2] Link "Pricing" center=(1686,1040)
[3] Button "Go forward" center=(484,179)
[4] Button "Extensions" center=(1561,279)
[5] Button "Bookmark" center=(209,1025)
[6] Button "Run inference" center=(828,706)
[7] Button "Go back" center=(1325,519)
[8] Button "New tab" center=(993,978)

Instruction: reload

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1299,342)
[1] TextField "Email" center=(1590,401)
[2] CheckBox "Remember me" center=(441,733)
[3] TextField "Password" center=(1298,756)
[4] Button "Sign in" center=(253,982)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1367,957)
[1] Button "Terminal" center=(1595,468)
[2] Button "Open" center=(177,168)
[3] Button "Save" center=(94,842)
[4] Button "New File" center=(1218,203)
[5] Button "Run" center=(1174,128)
[6] Button "Format" center=(1722,757)
[7] TextField "Search files" center=(1367,368)

Instruction: find in file

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(273,440)
[1] Button "Next" center=(1019,681)
[2] Button "Shuffle" center=(1160,736)
[3] Button "Fullscreen" center=(944,64)
[4] Button "Pause" center=(831,684)
[5] Button "Play" center=(1731,529)
[6] Slider "Volume" center=(1335,477)

Instruction: skip to next track

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1819,181)
[1] Button "Shuffle" center=(815,878)
[2] Button "Fullscreen" center=(1342,569)
[3] Button "Next" center=(1515,331)
[4] Slider "Volume" center=(1365,564)

Instruction: tap shuffle

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1206,810)
[1] Button "Run inference" center=(1141,618)
[2] Link "Pricing" center=(1467,192)
[3] Button "Extensions" center=(1829,289)
[4] Button "Downloads" center=(1479,754)
[5] Link "Docs" center=(189,252)
[6] Button "Go back" center=(333,305)
[7] Button "Bookmark" center=(935,392)

Instruction: previous page

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(741,623)
[1] Button "Sound" center=(186,943)
[2] CheckBox "Do Not Disturb" center=(738,957)
[3] CheckBox "Bluetooth" center=(470,981)
[4] PopUpButton "Appearance" center=(1112,182)
[5] Button "Network" center=(124,136)
[6] CheckBox "Wi-Fi" center=(241,837)
[7] Button "Displays" center=(602,723)

Instruction: press wi-fi

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1824,948)
[1] Slider "Volume" center=(795,105)
[2] Button "Shuffle" center=(1689,187)
[3] Button "Pause" center=(1039,227)
[4] Button "Next" center=(612,363)

Instruction: press volume

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(943,563)
[1] Button "Terminal" center=(641,642)
[2] Button "Open" center=(912,203)
[3] TextField "Search files" center=(377,325)
[4] Button "Find" center=(1707,861)
[5] Button "Run" center=(1223,334)
[6] Button "New File" center=(1490,416)
[7] Button "Commit" center=(1740,163)
[8] Button "Format" center=(84,354)

Instruction: select open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1608,169)
[1] Link "Pricing" center=(1577,498)
[2] Link "Sign in" center=(1833,467)
[3] Button "Go back" center=(1496,544)
[4] SearchField "Search or enter address" center=(407,651)
[5] Button "New tab" center=(513,797)
[6] Button "Bookmark" center=(694,759)
[7] Button "Downloads" center=(84,668)
[8] Button "Extensions" center=(942,638)
[9] Button "Reload" center=(1632,260)
[10] Button "Run inference" center=(567,741)

Instruction: i want to log in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(746,1035)
[1] Link "Sign in" center=(1049,780)
[2] Button "Run inference" center=(1201,514)
[3] Button "Bookmark" center=(1797,938)
[4] Link "Pricing" center=(320,422)
[5] SearchField "Search or enter address" center=(278,924)
[6] Button "Go forward" center=(1104,338)
[7] Button "Reload" center=(1469,711)
[8] Button "Go back" center=(146,509)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1433,901)
[1] Button "Next" center=(334,832)
[2] Button "Shuffle" center=(537,237)
[3] Button "Play" center=(950,700)
[4] Button "Pause" center=(1537,941)
[5] Button "Fullscreen" center=(359,750)
[6] Slider "Volume" center=(154,988)

Instruction: go fullscreen

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(873,685)
[1] Button "Pause" center=(489,727)
[2] Button "Play" center=(327,909)
[3] Slider "Volume" center=(1807,842)
[4] Button "Shuffle" center=(1694,572)

Instruction: skip to next track

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(834,876)
[1] CheckBox "Bluetooth" center=(558,831)
[2] Button "Network" center=(1821,596)
[3] CheckBox "Do Not Disturb" center=(456,249)
[4] Button "Displays" center=(1581,257)
[5] Button "Privacy & Security" center=(906,909)

Instruction: tap displays

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(1350,733)
[1] Button "New tab" center=(558,661)
[2] Button "Bookmark" center=(625,952)
[3] Button "Go forward" center=(930,937)
[4] Button "Go back" center=(1267,176)
[5] Button "Run inference" center=(89,615)
[6] Button "Downloads" center=(798,979)
[7] Link "Pricing" center=(898,326)
[8] Link "Docs" center=(1203,772)
[9] Button "Extensions" center=(1747,93)

Instruction: press go forward

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(142,884)
[1] Button "Create account" center=(1377,1032)
[2] CheckBox "Remember me" center=(1862,935)
[3] TextField "Email" center=(41,119)
[4] Button "Sign in" center=(261,960)
[5] Link "Forgot password" center=(719,338)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(562,693)
[1] Button "Sign in" center=(179,102)
[2] Button "Create account" center=(617,102)
[3] CheckBox "Remember me" center=(596,287)
[4] TextField "Email" center=(170,396)
[5] Link "Forgot password" center=(870,434)

Instruction: make a new account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(675,509)
[1] Button "Create account" center=(694,400)
[2] CheckBox "Remember me" center=(1075,91)
[3] TextField "Email" center=(443,83)
[4] Link "Forgot password" center=(1318,694)
[5] TextField "Password" center=(64,145)

Instruction: tap forgot password

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(226,1025)
[1] Button "Sign in" center=(1285,78)
[2] TextField "Password" center=(356,949)
[3] CheckBox "Remember me" center=(987,849)
[4] TextField "Email" center=(1385,145)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(231,149)
[1] Button "Add to cart" center=(789,568)
[2] Button "Checkout" center=(837,474)
[3] Button "Wishlist" center=(1676,569)
[4] Button "Buy now" center=(905,155)
[5] SearchField "Search products" center=(214,622)

Instruction: tap wishlist

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(887,107)
[1] TextField "Email" center=(1015,930)
[2] Button "Sign in" center=(125,776)
[3] TextField "Password" center=(697,925)
[4] CheckBox "Remember me" center=(97,380)

Instruction: press remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(904,375)
[1] TextField "Email" center=(940,762)
[2] Button "Create account" center=(347,598)
[3] Link "Forgot password" center=(796,292)
[4] Button "Sign in" center=(786,1014)
[5] TextField "Password" center=(530,293)

Instruction: press sign in

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(998,967)
[1] TextField "Email" center=(1630,757)
[2] Button "Sign in" center=(1027,383)
[3] Link "Forgot password" center=(1185,84)
[4] Button "Create account" center=(176,737)

Instruction: tap sign in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1655,567)
[1] Button "Checkout" center=(738,249)
[2] Button "Filters" center=(362,962)
[3] Button "Add to cart" center=(1731,513)
[4] Button "Buy now" center=(406,530)

Instruction: add this to my cart

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(695,168)
[1] Link "Forgot password" center=(1831,570)
[2] Button "Create account" center=(799,1006)
[3] CheckBox "Remember me" center=(654,476)
[4] TextField "Email" center=(258,1031)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1607,888)
[1] Button "Terminal" center=(498,608)
[2] Button "New File" center=(1811,772)
[3] Button "Open" center=(1769,210)
[4] TextField "Search files" center=(1030,852)
[5] Button "Commit" center=(1013,520)
[6] Button "Find" center=(973,981)
[7] Button "Save" center=(1569,1012)

Instruction: select terminal

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(165,48)
[1] Slider "Brightness" center=(946,969)
[2] CheckBox "Wi-Fi" center=(209,189)
[3] Button "Displays" center=(189,89)
[4] Button "Network" center=(1734,184)
[5] Button "Privacy & Security" center=(1507,918)
[6] PopUpButton "Appearance" center=(610,184)
[7] CheckBox "Do Not Disturb" center=(436,342)
[8] Button "Sound" center=(1422,337)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1428,633)
[1] Button "Pause" center=(1233,686)
[2] Button "Next" center=(1012,150)
[3] Button "Play" center=(810,413)
[4] Button "Fullscreen" center=(1493,664)

Instruction: start playback

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(72,57)
[1] Button "Format" center=(1299,770)
[2] Button "Terminal" center=(790,195)
[3] Button "Open" center=(370,998)
[4] Button "Run" center=(1727,899)
[5] Button "Commit" center=(616,1007)
[6] TextField "Search files" center=(323,730)

Instruction: make a commit

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(539,678)
[1] Button "Checkout" center=(1480,53)
[2] Button "Add to cart" center=(979,61)
[3] SearchField "Search products" center=(781,749)
[4] Button "Filters" center=(622,529)
[5] Link "Account" center=(1415,289)

Instruction: select wishlist

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(898,49)
[1] Button "Terminal" center=(102,556)
[2] Button "Run" center=(106,177)
[3] Button "Find" center=(861,397)
[4] Button "Format" center=(1187,625)
[5] Button "Save" center=(1509,253)

Instruction: tap find

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1047,955)
[1] Button "Add to cart" center=(222,654)
[2] SearchField "Search products" center=(422,150)
[3] Button "Checkout" center=(620,624)
[4] Link "Account" center=(865,523)
[5] Button "Buy now" center=(1207,133)

Instruction: proceed to payment

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(86,937)
[1] Button "Add to cart" center=(465,133)
[2] SearchField "Search products" center=(1053,749)
[3] Button "Checkout" center=(324,876)
[4] Button "Filters" center=(1386,457)

Instruction: press add to cart

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1411,171)
[1] TextField "Email" center=(634,42)
[2] TextField "Password" center=(120,870)
[3] Button "Sign in" center=(110,745)
[4] Link "Forgot password" center=(1843,792)
[5] Button "Create account" center=(1423,713)

Instruction: i forgot my password

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1054,938)
[1] Link "Pricing" center=(274,450)
[2] Button "Go forward" center=(1374,247)
[3] Button "Reload" center=(1534,98)
[4] Button "Run inference" center=(1840,700)
[5] Button "Extensions" center=(1465,782)
[6] Link "Sign in" center=(445,280)

Instruction: press reload

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1720,649)
[1] TextField "Search files" center=(1245,280)
[2] Button "New File" center=(175,606)
[3] Button "Find" center=(948,573)
[4] Button "Format" center=(273,784)
[5] Button "Commit" center=(47,868)
[6] Button "Terminal" center=(674,914)
[7] Button "Save" center=(1727,584)
[8] Button "Run" center=(131,217)

Instruction: search

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(894,474)
[1] Button "Bookmark" center=(221,288)
[2] Link "Docs" center=(1617,467)
[3] Link "Pricing" center=(846,214)
[4] Button "Go forward" center=(1133,978)
[5] Button "Extensions" center=(1356,65)
[6] Button "New tab" center=(1088,847)
[7] Button "Go back" center=(1700,124)
[8] Button "Reload" center=(327,180)
[9] Button "Run inference" center=(1421,690)
[10] Link "Sign in" center=(918,572)
[11] Button "Downloads" center=(176,397)

Instruction: select run inference

[Assistant] {"action":"click","id":9}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(754,737)
[1] TextField "Search files" center=(1163,1009)
[2] Button "Format" center=(474,730)
[3] Button "Open" center=(1636,799)
[4] Button "New File" center=(540,48)

Instruction: tap open

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1406,472)
[1] Button "Privacy & Security" center=(1281,278)
[2] CheckBox "Do Not Disturb" center=(1011,457)
[3] Button "Sound" center=(1542,924)
[4] PopUpButton "Appearance" center=(769,703)
[5] CheckBox "Wi-Fi" center=(98,815)

Instruction: tap sound

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(523,482)
[1] Button "New File" center=(1155,578)
[2] Button "Save" center=(1401,722)
[3] Button "Commit" center=(772,65)
[4] Button "Format" center=(219,786)
[5] Button "Run" center=(1380,446)
[6] Button "Find" center=(1840,834)
[7] Button "Open" center=(1642,695)

Instruction: find in file

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1358,446)
[1] Button "Sound" center=(215,1012)
[2] CheckBox "Do Not Disturb" center=(849,876)
[3] CheckBox "Wi-Fi" center=(705,213)
[4] Button "Displays" center=(423,603)
[5] Slider "Brightness" center=(1352,656)
[6] Button "Network" center=(1424,953)
[7] PopUpButton "Appearance" center=(1502,590)

Instruction: turn off bluetooth

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1314,528)
[1] TextField "Password" center=(1023,516)
[2] Button "Create account" center=(351,484)
[3] TextField "Email" center=(700,1037)
[4] CheckBox "Remember me" center=(1275,896)
[5] Link "Forgot password" center=(1055,429)

Instruction: press remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1495,1003)
[1] Button "Wishlist" center=(635,167)
[2] Button "Buy now" center=(1623,491)
[3] Button "Add to cart" center=(263,953)
[4] SearchField "Search products" center=(1638,436)

Instruction: search for something

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1116,478)
[1] Slider "Brightness" center=(120,97)
[2] PopUpButton "Appearance" center=(1827,707)
[3] Button "Network" center=(779,832)
[4] Button "Displays" center=(129,44)
[5] CheckBox "Wi-Fi" center=(1511,657)
[6] CheckBox "Do Not Disturb" center=(416,308)

Instruction: enable wifi

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1625,184)
[1] Button "Find" center=(1247,240)
[2] Button "Commit" center=(76,166)
[3] Button "Save" center=(248,788)
[4] Button "Run" center=(1836,193)

Instruction: click open

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1584,1032)
[1] Button "Fullscreen" center=(87,451)
[2] Button "Pause" center=(1394,698)
[3] Button "Shuffle" center=(1270,392)
[4] Button "Play" center=(345,225)
[5] Button "Previous" center=(371,272)
[6] Slider "Volume" center=(1782,233)

Instruction: click fullscreen

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(828,706)
[1] Button "Find" center=(1495,473)
[2] Button "Format" center=(324,305)
[3] Button "Commit" center=(1477,864)
[4] Button "Open" center=(1215,502)
[5] Button "Run" center=(784,284)

Instruction: run the code

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(840,1008)
[1] Button "Pause" center=(1398,1024)
[2] Button "Shuffle" center=(1263,416)
[3] Button "Play" center=(875,734)
[4] Button "Previous" center=(1631,970)
[5] Button "Next" center=(1055,301)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(560,378)
[1] Button "Filters" center=(1523,107)
[2] Link "Account" center=(614,429)
[3] SearchField "Search products" center=(1382,479)
[4] Button "Checkout" center=(1875,602)

Instruction: search for something

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(763,710)
[1] Button "Play" center=(1160,1007)
[2] Button "Previous" center=(918,685)
[3] Button "Pause" center=(367,915)
[4] Slider "Volume" center=(892,892)
[5] Button "Next" center=(1052,261)
[6] Button "Shuffle" center=(929,441)

Instruction: select volume

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1080,962)
[1] PopUpButton "Appearance" center=(1287,337)
[2] CheckBox "Bluetooth" center=(539,202)
[3] Button "Displays" center=(1623,945)
[4] Button "Network" center=(1372,458)

Instruction: press network

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1454,385)
[1] Button "Pause" center=(1814,588)
[2] Button "Play" center=(104,404)
[3] Slider "Volume" center=(88,821)
[4] Button "Next" center=(1522,849)
[5] Button "Shuffle" center=(665,366)
[6] Button "Fullscreen" center=(1677,101)

Instruction: click shuffle

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1724,48)
[1] Button "Add to cart" center=(144,508)
[2] Button "Filters" center=(443,327)
[3] Button "Wishlist" center=(392,979)
[4] Button "Checkout" center=(1592,657)
[5] SearchField "Search products" center=(1593,240)
[6] Button "Buy now" center=(1201,86)

Instruction: press wishlist

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1327,1032)
[1] Button "Fullscreen" center=(1444,44)
[2] Button "Shuffle" center=(534,615)
[3] Button "Next" center=(98,104)
[4] Slider "Volume" center=(903,181)
[5] Button "Pause" center=(726,612)
[6] Button "Previous" center=(759,152)

Instruction: press pause

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(418,1022)
[1] Slider "Volume" center=(1191,136)
[2] Button "Pause" center=(678,254)
[3] Button "Next" center=(743,272)
[4] Button "Play" center=(1851,106)
[5] Button "Fullscreen" center=(325,970)
[6] Button "Shuffle" center=(990,650)

Instruction: go fullscreen

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1219,69)
[1] CheckBox "Do Not Disturb" center=(602,929)
[2] Button "Network" center=(381,450)
[3] Button "Displays" center=(773,229)
[4] PopUpButton "Appearance" center=(1852,511)
[5] CheckBox "Wi-Fi" center=(1713,310)
[6] CheckBox "Bluetooth" center=(345,604)

Instruction: toggle wifi

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(434,607)
[1] Slider "Volume" center=(1504,733)
[2] Button "Pause" center=(756,396)
[3] Button "Fullscreen" center=(639,390)
[4] Button "Shuffle" center=(204,305)
[5] Button "Play" center=(1334,814)
[6] Button "Next" center=(923,1040)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1403,963)
[1] Button "Previous" center=(663,392)
[2] Button "Fullscreen" center=(1737,406)
[3] Button "Next" center=(680,813)
[4] Button "Pause" center=(1441,265)
[5] Slider "Volume" center=(1450,504)

Instruction: skip to next track

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(552,434)
[1] Button "Checkout" center=(235,903)
[2] Link "Account" center=(327,906)
[3] SearchField "Search products" center=(217,723)
[4] Button "Wishlist" center=(690,890)
[5] Button "Add to cart" center=(1014,423)

Instruction: press add to cart

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1644,710)
[1] TextField "Search files" center=(945,441)
[2] Button "Format" center=(367,54)
[3] Button "New File" center=(1246,676)
[4] Button "Save" center=(161,474)
[5] Button "Run" center=(1209,279)
[6] Button "Terminal" center=(211,657)
[7] Button "Open" center=(1744,481)
[8] Button "Find" center=(532,60)

Instruction: select new file

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(506,732)
[1] Button "Bookmark" center=(1686,142)
[2] Button "Extensions" center=(394,1015)
[3] Button "Go forward" center=(1597,260)
[4] Button "Reload" center=(614,840)
[5] SearchField "Search or enter address" center=(509,887)
[6] Button "Run inference" center=(1628,583)
[7] Button "Downloads" center=(1178,583)
[8] Button "New tab" center=(118,355)
[9] Button "Go back" center=(1068,392)
[10] Link "Pricing" center=(1781,998)
[11] Link "Docs" center=(225,713)

Instruction: click extensions

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(399,844)
[1] Button "Privacy & Security" center=(1519,283)
[2] Button "Displays" center=(259,292)
[3] CheckBox "Wi-Fi" center=(562,175)
[4] PopUpButton "Appearance" center=(1005,410)
[5] CheckBox "Do Not Disturb" center=(1611,419)
[6] Slider "Brightness" center=(944,427)
[7] Button "Network" center=(995,229)
[8] Button "Sound" center=(1281,664)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1873,127)
[1] TextField "Password" center=(1107,911)
[2] Button "Create account" center=(1488,247)
[3] Button "Sign in" center=(1364,346)
[4] TextField "Email" center=(549,486)

Instruction: make a new account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1439,638)
[1] Button "Go forward" center=(274,1011)
[2] Button "New tab" center=(1418,998)
[3] Button "Run inference" center=(997,659)
[4] Link "Docs" center=(299,439)
[5] Button "Downloads" center=(1642,1022)
[6] Button "Go back" center=(735,868)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1041,274)
[1] Button "Sound" center=(1719,673)
[2] PopUpButton "Appearance" center=(358,200)
[3] CheckBox "Wi-Fi" center=(240,647)
[4] Slider "Brightness" center=(1166,292)
[5] Button "Network" center=(853,620)
[6] Button "Privacy & Security" center=(194,845)
[7] CheckBox "Bluetooth" center=(699,493)
[8] CheckBox "Do Not Disturb" center=(486,735)

Instruction: turn off bluetooth

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1357,339)
[1] TextField "Email" center=(749,910)
[2] CheckBox "Remember me" center=(1258,798)
[3] Button "Create account" center=(1307,900)
[4] Link "Forgot password" center=(1532,122)
[5] TextField "Password" center=(804,924)

Instruction: select remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1560,866)
[1] Button "Sign in" center=(1106,289)
[2] Link "Forgot password" center=(1187,546)
[3] TextField "Password" center=(1176,667)
[4] CheckBox "Remember me" center=(110,948)
[5] TextField "Email" center=(476,802)

Instruction: type into email

[Assistant] {"action":"type","id":5,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1599,446)
[1] Link "Forgot password" center=(136,283)
[2] Button "Create account" center=(1368,837)
[3] Button "Sign in" center=(137,318)
[4] TextField "Password" center=(57,659)
[5] TextField "Email" center=(1639,776)

Instruction: sign up

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1074,138)
[1] Button "Previous" center=(131,117)
[2] Slider "Volume" center=(1859,289)
[3] Button "Pause" center=(1740,602)
[4] Button "Play" center=(1401,175)
[5] Button "Fullscreen" center=(954,216)
[6] Button "Next" center=(186,89)

Instruction: click volume

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(241,280)
[1] Button "Privacy & Security" center=(1136,713)
[2] Slider "Brightness" center=(1160,213)
[3] CheckBox "Do Not Disturb" center=(780,855)
[4] Button "Sound" center=(523,217)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1636,692)
[1] Slider "Volume" center=(881,267)
[2] Button "Next" center=(890,901)
[3] Button "Previous" center=(58,283)
[4] Button "Play" center=(1140,618)
[5] Button "Shuffle" center=(1425,723)
[6] Button "Pause" center=(1433,128)

Instruction: tap shuffle

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1027,228)
[1] CheckBox "Bluetooth" center=(134,83)
[2] PopUpButton "Appearance" center=(87,402)
[3] CheckBox "Do Not Disturb" center=(305,122)
[4] Button "Displays" center=(566,755)
[5] Button "Sound" center=(1508,142)
[6] Button "Network" center=(1638,91)
[7] CheckBox "Wi-Fi" center=(1010,975)

Instruction: delete my account permanently

[Assistant] {"action":"none"}