[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(979,46)
[1] TextField "Search files" center=(1123,420)
[2] Button "Commit" center=(205,823)
[3] Button "Format" center=(265,943)
[4] Button "Save" center=(1639,638)
[5] Button "New File" center=(1583,122)

Instruction: select new file

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1303,444)
[1] TextField "Password" center=(744,386)
[2] Link "Forgot password" center=(837,882)
[3] Button "Create account" center=(275,654)
[4] CheckBox "Remember me" center=(421,504)

Instruction: type into email

[Assistant] {"action":"type","id":0,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1237,402)
[1] CheckBox "Bluetooth" center=(860,149)
[2] Button "Displays" center=(315,799)
[3] Button "Network" center=(862,823)
[4] CheckBox "Wi-Fi" center=(1738,430)
[5] Button "Privacy & Security" center=(1629,491)

Instruction: select do not disturb

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1570,333)
[1] Button "Sign in" center=(1725,87)
[2] TextField "Email" center=(379,786)
[3] CheckBox "Remember me" center=(326,262)
[4] TextField "Password" center=(1447,711)
[5] Button "Create account" center=(1509,819)

Instruction: make a new account

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(459,784)
[1] TextField "Password" center=(487,40)
[2] TextField "Email" center=(647,1014)
[3] Link "Forgot password" center=(1432,153)
[4] Button "Create account" center=(186,864)
[5] Button "Sign in" center=(976,466)

Instruction: sign in

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1270,446)
[1] Button "Fullscreen" center=(1129,794)
[2] Button "Previous" center=(903,252)
[3] Button "Shuffle" center=(1856,985)
[4] Button "Pause" center=(891,308)

Instruction: tap previous

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(483,426)
[1] TextField "Password" center=(1168,612)
[2] Button "Sign in" center=(1636,1024)
[3] CheckBox "Remember me" center=(513,43)
[4] Link "Forgot password" center=(97,324)

Instruction: press remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(272,717)
[1] Button "Save" center=(1358,211)
[2] Button "Format" center=(1028,997)
[3] Button "Terminal" center=(158,432)
[4] Button "Find" center=(715,652)
[5] Button "New File" center=(496,326)
[6] Button "Commit" center=(171,84)
[7] Button "Open" center=(581,91)
[8] Button "Run" center=(812,800)

Instruction: click format

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(954,753)
[1] Link "Docs" center=(1234,831)
[2] Button "New tab" center=(860,707)
[3] Button "Go forward" center=(149,908)
[4] Button "Go back" center=(1734,531)
[5] Button "Bookmark" center=(725,733)

Instruction: select go back

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(370,992)
[1] Button "Add to cart" center=(1818,178)
[2] Link "Account" center=(258,542)
[3] Button "Filters" center=(294,153)
[4] SearchField "Search products" center=(611,599)
[5] Button "Checkout" center=(1788,459)

Instruction: select filters

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1048,256)
[1] Button "Buy now" center=(1334,860)
[2] Button "Wishlist" center=(1442,914)
[3] Button "Filters" center=(1812,763)
[4] SearchField "Search products" center=(681,456)

Instruction: press checkout

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(486,539)
[1] Button "Extensions" center=(1188,559)
[2] SearchField "Search or enter address" center=(185,130)
[3] Button "Go back" center=(1215,899)
[4] Link "Docs" center=(1866,256)
[5] Button "Go forward" center=(1116,913)
[6] Button "Bookmark" center=(1452,767)
[7] Link "Sign in" center=(1248,703)
[8] Button "Downloads" center=(1229,948)
[9] Link "Pricing" center=(1238,94)
[10] Button "Run inference" center=(842,583)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":2,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1051,396)
[1] Button "Previous" center=(426,854)
[2] Slider "Volume" center=(1439,752)
[3] Button "Shuffle" center=(391,157)
[4] Button "Pause" center=(836,267)
[5] Button "Fullscreen" center=(1683,747)

Instruction: press next

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1015,728)
[1] Link "Docs" center=(374,655)
[2] Button "Bookmark" center=(771,807)
[3] Button "Reload" center=(645,420)
[4] SearchField "Search or enter address" center=(1096,835)
[5] Button "Go back" center=(721,555)
[6] Button "Extensions" center=(500,389)
[7] Button "New tab" center=(1463,471)

Instruction: refresh the page

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(256,488)
[1] Button "Network" center=(1696,318)
[2] CheckBox "Wi-Fi" center=(182,893)
[3] Button "Privacy & Security" center=(1269,757)
[4] PopUpButton "Appearance" center=(1555,228)
[5] CheckBox "Bluetooth" center=(700,611)
[6] Button "Sound" center=(1843,939)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1465,595)
[1] TextField "Search files" center=(849,278)
[2] Button "Run" center=(1210,652)
[3] Button "Find" center=(574,406)
[4] Button "Open" center=(568,140)

Instruction: select open

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1275,288)
[1] Link "Forgot password" center=(57,956)
[2] CheckBox "Remember me" center=(1620,860)
[3] Button "Sign in" center=(1003,652)
[4] TextField "Email" center=(859,573)

Instruction: click remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1329,324)
[1] Button "Pause" center=(1533,72)
[2] Button "Shuffle" center=(1191,807)
[3] Button "Fullscreen" center=(1291,659)
[4] Button "Next" center=(1814,220)

Instruction: go fullscreen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1836,251)
[1] Link "Forgot password" center=(481,1026)
[2] TextField "Password" center=(1313,521)
[3] Button "Create account" center=(433,996)
[4] TextField "Email" center=(918,503)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(1545,911)
[1] Button "Extensions" center=(593,136)
[2] Link "Sign in" center=(1449,671)
[3] Link "Docs" center=(1115,940)
[4] Button "Reload" center=(1661,899)
[5] Button "Go forward" center=(1518,275)
[6] Button "Downloads" center=(1700,868)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(634,314)
[1] PopUpButton "Appearance" center=(384,343)
[2] Button "Sound" center=(1598,1035)
[3] Button "Displays" center=(1172,933)
[4] CheckBox "Wi-Fi" center=(1786,317)
[5] Slider "Brightness" center=(358,725)
[6] CheckBox "Do Not Disturb" center=(1408,71)

Instruction: press sound

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1476,955)
[1] Button "Create account" center=(226,160)
[2] TextField "Password" center=(1408,793)
[3] Button "Sign in" center=(1094,476)
[4] Link "Forgot password" center=(1386,106)
[5] TextField "Email" center=(728,466)

Instruction: select remember me

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(1757,372)
[1] Button "Extensions" center=(231,740)
[2] Link "Docs" center=(1337,678)
[3] Button "Reload" center=(154,560)
[4] Button "Go back" center=(1728,223)
[5] Link "Sign in" center=(829,920)
[6] SearchField "Search or enter address" center=(195,723)
[7] Button "Go forward" center=(1723,838)
[8] Button "Bookmark" center=(184,67)

Instruction: i want to log in

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1023,431)
[1] Button "Sign in" center=(1496,430)
[2] TextField "Email" center=(926,637)
[3] CheckBox "Remember me" center=(1399,999)
[4] Link "Forgot password" center=(907,654)

Instruction: log in

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1426,56)
[1] Button "Sound" center=(1751,676)
[2] Button "Network" center=(1664,276)
[3] Slider "Brightness" center=(1644,853)
[4] CheckBox "Do Not Disturb" center=(665,169)
[5] Button "Displays" center=(614,709)
[6] CheckBox "Wi-Fi" center=(1332,919)
[7] Button "Privacy & Security" center=(344,341)
[8] PopUpButton "Appearance" center=(44,427)

Instruction: press network

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(567,253)
[1] TextField "Email" center=(1007,419)
[2] CheckBox "Remember me" center=(249,1001)
[3] Link "Forgot password" center=(1550,579)
[4] TextField "Password" center=(820,486)

Instruction: click remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1191,220)
[1] Button "Add to cart" center=(825,1006)
[2] Button "Filters" center=(1310,1007)
[3] Button "Buy now" center=(313,607)
[4] Link "Account" center=(715,391)
[5] Button "Checkout" center=(82,500)
[6] Button "Wishlist" center=(590,249)

Instruction: click wishlist

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1767,686)
[1] SearchField "Search products" center=(1812,48)
[2] Button "Wishlist" center=(549,622)
[3] Link "Account" center=(914,866)
[4] Button "Buy now" center=(1279,934)
[5] Button "Checkout" center=(904,330)

Instruction: click wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(117,820)
[1] Button "Find" center=(1252,857)
[2] Button "Terminal" center=(1245,388)
[3] TextField "Search files" center=(601,179)
[4] Button "Run" center=(1167,560)

Instruction: select find

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(961,643)
[1] Button "Pause" center=(1143,665)
[2] Button "Next" center=(1226,304)
[3] Button "Fullscreen" center=(975,56)
[4] Button "Previous" center=(876,757)
[5] Slider "Volume" center=(258,508)

Instruction: click volume

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1593,260)
[1] Button "Format" center=(1675,576)
[2] TextField "Search files" center=(573,855)
[3] Button "Run" center=(1506,994)
[4] Button "Find" center=(957,145)

Instruction: tap run

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1046,718)
[1] Button "Commit" center=(639,307)
[2] Button "Find" center=(1192,948)
[3] Button "Run" center=(1521,739)
[4] Button "Terminal" center=(410,249)
[5] Button "New File" center=(621,750)
[6] TextField "Search files" center=(464,1032)
[7] Button "Save" center=(793,665)

Instruction: type into search files

[Assistant] {"action":"type","id":6,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(726,840)
[1] Button "Filters" center=(877,458)
[2] Button "Buy now" center=(1794,764)
[3] Link "Account" center=(1220,849)
[4] Button "Add to cart" center=(509,536)
[5] Button "Checkout" center=(742,358)
[6] SearchField "Search products" center=(133,178)

Instruction: press checkout

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(655,1021)
[1] Button "Checkout" center=(108,904)
[2] SearchField "Search products" center=(306,378)
[3] Button "Buy now" center=(154,922)
[4] Button "Filters" center=(1437,919)

Instruction: type into search products

[Assistant] {"action":"type","id":2,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1099,702)
[1] Button "Open" center=(628,626)
[2] Button "Find" center=(540,816)
[3] Button "Save" center=(1573,456)
[4] Button "Terminal" center=(1800,612)
[5] Button "Run" center=(1818,93)

Instruction: select open

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(939,602)
[1] Button "Filters" center=(1223,426)
[2] Button "Wishlist" center=(1271,381)
[3] SearchField "Search products" center=(212,1030)
[4] Button "Add to cart" center=(1691,665)

Instruction: select account

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(810,817)
[1] Button "Shuffle" center=(1012,89)
[2] Button "Fullscreen" center=(1865,261)
[3] Button "Previous" center=(833,375)
[4] Button "Pause" center=(889,190)
[5] Button "Next" center=(145,1008)
[6] Slider "Volume" center=(437,805)

Instruction: make it full screen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(187,730)
[1] CheckBox "Do Not Disturb" center=(348,240)
[2] CheckBox "Wi-Fi" center=(495,419)
[3] CheckBox "Bluetooth" center=(940,735)
[4] Button "Network" center=(903,130)
[5] Slider "Brightness" center=(1016,771)
[6] PopUpButton "Appearance" center=(1764,800)
[7] Button "Privacy & Security" center=(137,499)
[8] Button "Displays" center=(1847,265)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(950,591)
[1] Button "Save" center=(681,753)
[2] Button "Find" center=(124,561)
[3] TextField "Search files" center=(1297,963)
[4] Button "Run" center=(1175,976)
[5] Button "Format" center=(1855,659)
[6] Button "Open" center=(1040,873)
[7] Button "Commit" center=(1055,311)

Instruction: select format

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1243,402)
[1] Slider "Volume" center=(1777,234)
[2] Button "Previous" center=(953,807)
[3] Button "Fullscreen" center=(981,846)
[4] Button "Pause" center=(559,939)

Instruction: press play

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(86,1021)
[1] Button "Open" center=(1814,531)
[2] Button "Save" center=(902,518)
[3] Button "New File" center=(766,122)
[4] TextField "Search files" center=(671,651)
[5] Button "Find" center=(951,399)

Instruction: click format

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1357,774)
[1] Button "Sign in" center=(135,624)
[2] TextField "Email" center=(1783,324)
[3] Link "Forgot password" center=(1552,74)
[4] TextField "Password" center=(1670,458)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(540,577)
[1] Button "Fullscreen" center=(1679,381)
[2] Button "Previous" center=(1380,712)
[3] Button "Next" center=(905,414)
[4] Button "Shuffle" center=(543,390)
[5] Button "Pause" center=(409,975)

Instruction: press pause

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1791,624)
[1] Button "New File" center=(1549,120)
[2] Button "Open" center=(183,101)
[3] Button "Find" center=(1546,291)
[4] Button "Terminal" center=(1708,355)

Instruction: tap open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1068,260)
[1] Button "Bookmark" center=(1573,1024)
[2] Button "Downloads" center=(957,846)
[3] Button "New tab" center=(606,451)
[4] Button "Go back" center=(913,118)
[5] Button "Reload" center=(678,787)
[6] Link "Pricing" center=(667,521)
[7] Link "Docs" center=(958,475)
[8] Link "Sign in" center=(230,562)
[9] SearchField "Search or enter address" center=(1356,822)

Instruction: log in

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1490,668)
[1] Button "Sign in" center=(72,798)
[2] TextField "Password" center=(803,769)
[3] Link "Forgot password" center=(312,417)
[4] Button "Create account" center=(867,391)

Instruction: make a new account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1535,224)
[1] TextField "Email" center=(1216,761)
[2] Button "Sign in" center=(420,548)
[3] Link "Forgot password" center=(905,970)
[4] TextField "Password" center=(625,915)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(482,936)
[1] Button "Format" center=(220,584)
[2] Button "Find" center=(835,425)
[3] Button "New File" center=(106,355)
[4] Button "Save" center=(1032,354)

Instruction: press format

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(574,506)
[1] Button "Buy now" center=(652,925)
[2] Button "Filters" center=(659,887)
[3] Button "Wishlist" center=(302,42)
[4] Button "Checkout" center=(424,375)

Instruction: click filters

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(675,987)
[1] Link "Docs" center=(1815,925)
[2] Button "New tab" center=(1057,530)
[3] Button "Go back" center=(659,137)
[4] Button "Run inference" center=(457,132)
[5] Button "Go forward" center=(456,980)
[6] Button "Downloads" center=(790,118)
[7] Link "Pricing" center=(1075,730)
[8] Button "Bookmark" center=(424,48)
[9] Button "Reload" center=(329,403)
[10] SearchField "Search or enter address" center=(1092,422)

Instruction: select downloads

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(383,750)
[1] Button "Wishlist" center=(283,540)
[2] SearchField "Search products" center=(1167,656)
[3] Button "Buy now" center=(711,391)
[4] Button "Checkout" center=(1835,677)
[5] Button "Add to cart" center=(1692,674)
[6] Button "Filters" center=(158,868)

Instruction: type into search products

[Assistant] {"action":"type","id":2,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1759,948)
[1] Button "Next" center=(1812,153)
[2] Slider "Volume" center=(811,511)
[3] Button "Previous" center=(746,816)
[4] Button "Pause" center=(1311,222)
[5] Button "Play" center=(480,590)
[6] Button "Fullscreen" center=(460,1005)

Instruction: press pause

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(441,1023)
[1] Button "Create account" center=(1131,106)
[2] Button "Sign in" center=(1717,304)
[3] TextField "Email" center=(1813,434)
[4] CheckBox "Remember me" center=(427,305)

Instruction: press remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1602,907)
[1] CheckBox "Remember me" center=(146,900)
[2] Button "Create account" center=(1243,793)
[3] Button "Sign in" center=(248,217)
[4] TextField "Email" center=(169,567)

Instruction: select sign in

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(168,368)
[1] Button "Downloads" center=(497,79)
[2] Button "Run inference" center=(428,100)
[3] SearchField "Search or enter address" center=(1695,354)
[4] Button "Reload" center=(647,50)

Instruction: refresh the page

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1574,859)
[1] Link "Pricing" center=(665,373)
[2] Button "Go forward" center=(1855,350)
[3] Button "Bookmark" center=(285,841)
[4] SearchField "Search or enter address" center=(1183,799)
[5] Button "Reload" center=(416,651)
[6] Button "Extensions" center=(1250,140)
[7] Button "New tab" center=(329,456)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":4,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(86,469)
[1] Link "Forgot password" center=(254,979)
[2] CheckBox "Remember me" center=(1462,806)
[3] Button "Create account" center=(823,696)
[4] TextField "Email" center=(819,746)

Instruction: sign in

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1678,797)
[1] Button "Previous" center=(292,454)
[2] Button "Next" center=(1136,184)
[3] Button "Fullscreen" center=(894,884)
[4] Button "Pause" center=(1071,987)

Instruction: start playback

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(339,71)
[1] Link "Forgot password" center=(1123,238)
[2] Button "Sign in" center=(1333,317)
[3] TextField "Password" center=(1056,626)
[4] Button "Create account" center=(1677,994)
[5] CheckBox "Remember me" center=(1490,128)

Instruction: type into email

[Assistant] {"action":"type","id":0,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(305,818)
[1] Button "Create account" center=(311,604)
[2] Button "Sign in" center=(1324,506)
[3] CheckBox "Remember me" center=(140,760)
[4] TextField "Password" center=(452,201)
[5] Link "Forgot password" center=(338,71)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1688,235)
[1] TextField "Email" center=(1179,514)
[2] CheckBox "Remember me" center=(290,188)
[3] Button "Create account" center=(1289,398)
[4] Button "Sign in" center=(220,342)
[5] Link "Forgot password" center=(725,230)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(72,865)
[1] Button "Create account" center=(818,84)
[2] Button "Sign in" center=(629,605)
[3] TextField "Email" center=(98,217)
[4] Link "Forgot password" center=(1660,997)
[5] CheckBox "Remember me" center=(1743,226)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(125,775)
[1] Link "Forgot password" center=(1226,63)
[2] TextField "Password" center=(566,931)
[3] CheckBox "Remember me" center=(210,689)
[4] Button "Create account" center=(1069,1035)
[5] Button "Sign in" center=(1532,638)

Instruction: tap remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1762,402)
[1] Button "Fullscreen" center=(923,864)
[2] Button "Shuffle" center=(315,265)
[3] Slider "Volume" center=(1094,612)
[4] Button "Play" center=(1047,583)
[5] Button "Pause" center=(1081,988)
[6] Button "Next" center=(1085,493)

Instruction: select volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1767,645)
[1] Button "Find" center=(87,945)
[2] Button "New File" center=(220,140)
[3] Button "Format" center=(1677,784)
[4] Button "Save" center=(1817,238)
[5] TextField "Search files" center=(1178,431)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(714,164)
[1] Button "New File" center=(1596,778)
[2] Button "Run" center=(1535,302)
[3] Button "Commit" center=(1794,573)
[4] Button "Format" center=(1241,856)
[5] TextField "Search files" center=(1607,370)

Instruction: press format

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(981,868)
[1] Button "Previous" center=(1193,788)
[2] Button "Fullscreen" center=(137,1020)
[3] Button "Play" center=(780,278)
[4] Button "Shuffle" center=(670,491)
[5] Button "Next" center=(302,614)

Instruction: tap fullscreen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(56,867)
[1] TextField "Search files" center=(1179,301)
[2] Button "Terminal" center=(1810,562)
[3] Button "Open" center=(1594,569)
[4] Button "New File" center=(867,874)
[5] Button "Find" center=(1323,581)
[6] Button "Commit" center=(1572,828)
[7] Button "Save" center=(887,894)

Instruction: commit my changes

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1138,962)
[1] Link "Account" center=(43,818)
[2] Button "Add to cart" center=(1604,153)
[3] Button "Filters" center=(1119,263)
[4] SearchField "Search products" center=(355,291)
[5] Button "Checkout" center=(804,254)
[6] Button "Buy now" center=(125,487)

Instruction: find a product

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(77,665)
[1] Button "Pause" center=(857,325)
[2] Button "Next" center=(703,860)
[3] Slider "Volume" center=(1169,264)
[4] Button "Shuffle" center=(1022,1036)
[5] Button "Previous" center=(590,791)
[6] Button "Fullscreen" center=(614,628)

Instruction: tap shuffle

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1018,370)
[1] Link "Forgot password" center=(1877,392)
[2] Button "Create account" center=(1440,846)
[3] CheckBox "Remember me" center=(1492,377)
[4] TextField "Email" center=(1268,683)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(171,762)
[1] Button "Play" center=(1342,338)
[2] Button "Pause" center=(1118,435)
[3] Button "Shuffle" center=(1440,948)
[4] Button "Next" center=(360,478)
[5] Button "Previous" center=(948,370)
[6] Slider "Volume" center=(1597,163)

Instruction: click previous

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1295,456)
[1] Button "Add to cart" center=(768,846)
[2] Button "Buy now" center=(691,113)
[3] Button "Checkout" center=(1611,406)
[4] Link "Account" center=(1634,696)
[5] Button "Wishlist" center=(932,358)

Instruction: tap account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1548,556)
[1] Button "Run" center=(1715,228)
[2] Button "New File" center=(164,895)
[3] Button "Find" center=(265,445)
[4] Button "Save" center=(1646,236)
[5] Button "Format" center=(1754,78)
[6] Button "Open" center=(215,816)

Instruction: save the file

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(821,460)
[1] CheckBox "Wi-Fi" center=(1173,89)
[2] Button "Network" center=(1202,683)
[3] Button "Privacy & Security" center=(1104,824)
[4] Slider "Brightness" center=(560,486)
[5] PopUpButton "Appearance" center=(1381,750)
[6] CheckBox "Bluetooth" center=(140,1017)
[7] CheckBox "Do Not Disturb" center=(209,863)

Instruction: select sound

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(1667,942)
[1] Link "Docs" center=(1119,448)
[2] Button "Downloads" center=(859,804)
[3] SearchField "Search or enter address" center=(1035,146)
[4] Button "Go forward" center=(748,374)
[5] Button "Reload" center=(574,319)
[6] Link "Pricing" center=(893,941)
[7] Button "Go back" center=(72,393)
[8] Link "Sign in" center=(541,639)
[9] Button "New tab" center=(355,953)

Instruction: tap new tab

[Assistant] {"action":"click","id":9}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1574,251)
[1] Button "New File" center=(849,987)
[2] Button "Save" center=(1466,913)
[3] Button "Run" center=(1664,346)
[4] Button "Open" center=(1318,627)
[5] Button "Terminal" center=(168,149)
[6] Button "Format" center=(1267,747)

Instruction: select open

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(697,779)
[1] Button "Format" center=(1854,910)
[2] Button "Run" center=(1646,704)
[3] Button "Save" center=(1757,712)
[4] TextField "Search files" center=(485,610)
[5] Button "Find" center=(543,921)
[6] Button "Commit" center=(80,729)

Instruction: press run

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1572,444)
[1] Button "Downloads" center=(45,496)
[2] Link "Sign in" center=(1189,664)
[3] Button "Bookmark" center=(1630,892)
[4] Button "Run inference" center=(85,891)
[5] Link "Pricing" center=(1024,890)
[6] Button "Go forward" center=(743,124)
[7] Button "New tab" center=(1404,840)
[8] SearchField "Search or enter address" center=(1870,616)

Instruction: click downloads

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1277,839)
[1] TextField "Search files" center=(378,387)
[2] Button "Find" center=(461,728)
[3] Button "Commit" center=(245,479)
[4] Button "Open" center=(604,674)
[5] Button "Save" center=(1148,222)
[6] Button "New File" center=(1104,817)

Instruction: type into search files

[Assistant] {"action":"type","id":1,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(929,398)
[1] SearchField "Search products" center=(816,838)
[2] Button "Buy now" center=(693,636)
[3] Button "Wishlist" center=(1878,77)
[4] Button "Checkout" center=(1025,59)
[5] Link "Account" center=(780,373)

Instruction: proceed to payment

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(728,920)
[1] CheckBox "Bluetooth" center=(880,859)
[2] CheckBox "Do Not Disturb" center=(1194,218)
[3] Button "Sound" center=(328,783)
[4] PopUpButton "Appearance" center=(45,889)

Instruction: click appearance

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(295,92)
[1] Button "Find" center=(1399,88)
[2] Button "Terminal" center=(264,166)
[3] Button "Open" center=(722,698)
[4] TextField "Search files" center=(556,513)
[5] Button "Save" center=(971,919)
[6] Button "New File" center=(508,824)
[7] Button "Run" center=(629,669)

Instruction: click open

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(788,519)
[1] Button "Fullscreen" center=(1064,384)
[2] Button "Play" center=(861,300)
[3] Button "Previous" center=(252,268)
[4] Slider "Volume" center=(741,316)
[5] Button "Shuffle" center=(1367,1029)

Instruction: select previous

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(803,397)
[1] SearchField "Search or enter address" center=(1171,58)
[2] Button "Go back" center=(1748,833)
[3] Button "Extensions" center=(1420,807)
[4] Button "Reload" center=(1232,844)
[5] Button "New tab" center=(1753,392)
[6] Button "Go forward" center=(1035,804)
[7] Link "Pricing" center=(1264,348)
[8] Button "Run inference" center=(1745,539)
[9] Button "Downloads" center=(781,990)

Instruction: tap go forward

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(43,347)
[1] Link "Docs" center=(77,516)
[2] Button "Extensions" center=(248,297)
[3] Button "Run inference" center=(860,866)
[4] Button "Downloads" center=(1254,1039)
[5] SearchField "Search or enter address" center=(950,688)
[6] Link "Pricing" center=(1507,138)
[7] Button "Bookmark" center=(561,787)
[8] Button "Go forward" center=(1049,725)
[9] Button "New tab" center=(1248,540)
[10] Link "Sign in" center=(1515,868)
[11] Button "Reload" center=(1691,880)

Instruction: click downloads

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(620,421)
[1] Button "Wishlist" center=(1502,824)
[2] Button "Add to cart" center=(864,77)
[3] SearchField "Search products" center=(291,512)
[4] Button "Filters" center=(92,235)
[5] Link "Account" center=(1286,884)

Instruction: click buy now

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(982,626)
[1] Button "Find" center=(1659,701)
[2] Button "New File" center=(592,628)
[3] Button "Run" center=(108,540)
[4] Button "Commit" center=(1152,482)
[5] Button "Terminal" center=(1475,651)
[6] TextField "Search files" center=(404,1006)
[7] Button "Save" center=(1123,360)

Instruction: find in file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(531,416)
[1] Button "New tab" center=(1840,729)
[2] Button "Go back" center=(1669,404)
[3] Button "Reload" center=(1480,579)
[4] Button "Go forward" center=(1257,457)
[5] Link "Pricing" center=(133,540)
[6] Link "Sign in" center=(158,505)
[7] Link "Docs" center=(302,176)
[8] Button "Bookmark" center=(1133,591)
[9] Button "Run inference" center=(479,316)
[10] Button "Downloads" center=(1635,127)
[11] Button "Extensions" center=(477,276)

Instruction: press new tab

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(377,544)
[1] Button "Wishlist" center=(1340,357)
[2] Link "Account" center=(1298,272)
[3] Button "Buy now" center=(925,454)
[4] Button "Checkout" center=(1782,728)
[5] Button "Filters" center=(500,361)

Instruction: tap filters

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1419,778)
[1] Button "Run" center=(507,863)
[2] Button "New File" center=(785,832)
[3] Button "Format" center=(1327,487)
[4] TextField "Search files" center=(1767,413)
[5] Button "Save" center=(1188,873)
[6] Button "Terminal" center=(354,439)
[7] Button "Find" center=(1728,65)

Instruction: save

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1699,169)
[1] Button "Privacy & Security" center=(1416,171)
[2] PopUpButton "Appearance" center=(441,855)
[3] Button "Network" center=(1683,210)
[4] CheckBox "Bluetooth" center=(194,310)

Instruction: open privacy settings

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1592,816)
[1] Button "New File" center=(692,502)
[2] Button "Format" center=(580,684)
[3] Button "Terminal" center=(1328,319)
[4] Button "Commit" center=(813,976)
[5] Button "Run" center=(802,756)
[6] Button "Find" center=(479,225)
[7] TextField "Search files" center=(807,742)

Instruction: commit my changes

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(335,99)
[1] Button "Go back" center=(462,605)
[2] Button "Reload" center=(1730,703)
[3] Link "Docs" center=(1357,592)
[4] Button "Run inference" center=(245,746)
[5] Button "Bookmark" center=(1254,261)
[6] Link "Pricing" center=(950,897)
[7] SearchField "Search or enter address" center=(1519,479)
[8] Button "Downloads" center=(1090,348)
[9] Button "Extensions" center=(1638,142)

Instruction: tap run inference

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(586,1038)
[1] Button "Downloads" center=(162,960)
[2] SearchField "Search or enter address" center=(239,153)
[3] Button "Go forward" center=(418,493)
[4] Button "Extensions" center=(386,594)
[5] Link "Sign in" center=(464,973)
[6] Link "Pricing" center=(837,51)
[7] Button "Go back" center=(440,41)
[8] Button "Run inference" center=(1080,533)
[9] Button "Reload" center=(1839,836)

Instruction: tap extensions

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(812,187)
[1] Button "New File" center=(1556,658)
[2] Button "Terminal" center=(804,42)
[3] Button "Find" center=(1029,471)
[4] Button "Commit" center=(1379,534)
[5] Button "Format" center=(1451,898)
[6] Button "Run" center=(174,691)
[7] TextField "Search files" center=(892,229)

Instruction: type into search files

[Assistant] {"action":"type","id":7,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(250,553)
[1] PopUpButton "Appearance" center=(838,873)
[2] CheckBox "Do Not Disturb" center=(1216,267)
[3] Button "Network" center=(131,271)
[4] Button "Sound" center=(1854,886)
[5] Button "Privacy & Security" center=(1414,205)
[6] Slider "Brightness" center=(832,905)

Instruction: tap do not disturb

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(861,77)
[1] TextField "Email" center=(701,775)
[2] TextField "Password" center=(1442,810)
[3] CheckBox "Remember me" center=(512,663)
[4] Button "Create account" center=(1436,60)

Instruction: type into password

[Assistant] {"action":"type","id":2,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(728,873)
[1] Button "Buy now" center=(1395,157)
[2] Link "Account" center=(1484,1024)
[3] Button "Filters" center=(1260,501)
[4] Button "Wishlist" center=(1438,94)

Instruction: select checkout

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(841,515)
[1] Link "Account" center=(62,513)
[2] Button "Checkout" center=(607,219)
[3] Button "Wishlist" center=(281,1023)
[4] Button "Buy now" center=(720,984)
[5] Button "Add to cart" center=(545,108)
[6] SearchField "Search products" center=(1265,294)

Instruction: press wishlist

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(907,625)
[1] Button "Sign in" center=(1545,991)
[2] Link "Forgot password" center=(1589,65)
[3] TextField "Email" center=(1167,764)
[4] Button "Create account" center=(1084,506)

Instruction: i forgot my password

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1437,899)
[1] Button "Wishlist" center=(1312,694)
[2] SearchField "Search products" center=(209,44)
[3] Link "Account" center=(876,410)
[4] Button "Checkout" center=(958,285)

Instruction: type into search products

[Assistant] {"action":"type","id":2,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(93,937)
[1] Button "Terminal" center=(373,792)
[2] Button "Commit" center=(206,351)
[3] Button "Open" center=(136,969)
[4] TextField "Search files" center=(1694,823)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(734,867)
[1] CheckBox "Wi-Fi" center=(1593,923)
[2] Button "Privacy & Security" center=(1274,504)
[3] PopUpButton "Appearance" center=(244,544)
[4] Button "Sound" center=(1049,123)
[5] CheckBox "Do Not Disturb" center=(1795,537)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1470,431)
[1] Button "Filters" center=(49,377)
[2] Link "Account" center=(230,1026)
[3] Button "Checkout" center=(843,299)
[4] SearchField "Search products" center=(1147,868)
[5] Button "Add to cart" center=(514,819)

Instruction: search for something

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1812,723)
[1] Button "Sound" center=(932,918)
[2] CheckBox "Bluetooth" center=(164,49)
[3] Button "Network" center=(834,326)
[4] CheckBox "Wi-Fi" center=(499,357)
[5] CheckBox "Do Not Disturb" center=(704,1004)
[6] Button "Displays" center=(1383,903)
[7] Button "Privacy & Security" center=(1503,993)

Instruction: open privacy settings

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1586,551)
[1] Button "Downloads" center=(1875,310)
[2] Button "New tab" center=(1818,388)
[3] Button "Extensions" center=(278,838)
[4] Button "Reload" center=(40,349)
[5] Link "Docs" center=(1151,327)
[6] Button "Go forward" center=(1876,586)
[7] Button "Run inference" center=(1798,84)
[8] Button "Go back" center=(1241,350)
[9] Button "Bookmark" center=(1475,934)

Instruction: click extensions

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1030,92)
[1] Link "Account" center=(971,648)
[2] Button "Buy now" center=(1636,909)
[3] Button "Filters" center=(1235,952)
[4] Button "Wishlist" center=(777,877)

Instruction: tap buy now

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(781,336)
[1] Button "New File" center=(1419,801)
[2] Button "Terminal" center=(759,468)
[3] Button "Find" center=(739,512)
[4] Button "Run" center=(346,564)
[5] Button "Commit" center=(1197,849)
[6] Button "Save" center=(973,195)
[7] TextField "Search files" center=(380,436)

Instruction: search

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1443,429)
[1] Link "Forgot password" center=(1122,918)
[2] Button "Sign in" center=(754,799)
[3] TextField "Password" center=(893,51)
[4] TextField "Email" center=(1766,484)
[5] CheckBox "Remember me" center=(816,780)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1319,825)
[1] Button "Sound" center=(1805,493)
[2] CheckBox "Do Not Disturb" center=(1848,491)
[3] Button "Network" center=(1863,507)
[4] PopUpButton "Appearance" center=(939,650)

Instruction: press wi-fi

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1246,228)
[1] Button "Filters" center=(1840,803)
[2] Button "Add to cart" center=(1694,817)
[3] Link "Account" center=(949,111)
[4] Button "Buy now" center=(1412,844)
[5] Button "Wishlist" center=(214,96)

Instruction: select filters

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(453,207)
[1] Button "New tab" center=(514,237)
[2] Link "Pricing" center=(1441,1004)
[3] Button "Bookmark" center=(1629,287)
[4] Button "Go back" center=(146,846)
[5] Button "Downloads" center=(333,260)
[6] Button "Go forward" center=(1303,643)
[7] Button "Extensions" center=(510,298)
[8] Link "Docs" center=(613,428)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(44,448)
[1] Button "Fullscreen" center=(1293,392)
[2] Button "Shuffle" center=(1380,173)
[3] Button "Next" center=(430,389)
[4] Slider "Volume" center=(936,590)
[5] Button "Previous" center=(1842,441)
[6] Button "Play" center=(701,837)

Instruction: click shuffle

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(1127,947)
[1] CheckBox "Do Not Disturb" center=(1735,609)
[2] Button "Privacy & Security" center=(728,334)
[3] Button "Displays" center=(1128,530)
[4] CheckBox "Wi-Fi" center=(1637,1009)
[5] Button "Network" center=(737,291)
[6] CheckBox "Bluetooth" center=(258,159)

Instruction: click network

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(703,293)
[1] TextField "Email" center=(949,460)
[2] CheckBox "Remember me" center=(1596,349)
[3] Link "Forgot password" center=(310,413)
[4] Button "Sign in" center=(722,859)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(306,212)
[1] Button "New File" center=(290,295)
[2] Button "Commit" center=(1858,116)
[3] Button "Open" center=(1024,75)
[4] Button "Save" center=(230,676)
[5] Button "Run" center=(1508,965)
[6] Button "Find" center=(707,613)
[7] Button "Terminal" center=(861,1027)
[8] TextField "Search files" center=(889,984)

Instruction: make a commit

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1681,201)
[1] Button "Checkout" center=(1689,357)
[2] Button "Add to cart" center=(1202,410)
[3] Link "Account" center=(1672,458)
[4] Button "Buy now" center=(395,630)
[5] Button "Wishlist" center=(1714,155)

Instruction: tap wishlist

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1596,227)
[1] CheckBox "Remember me" center=(1149,610)
[2] Link "Forgot password" center=(1077,46)
[3] Button "Create account" center=(1004,44)
[4] TextField "Email" center=(79,371)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1536,760)
[1] Button "Sound" center=(1383,810)
[2] Button "Displays" center=(47,124)
[3] CheckBox "Bluetooth" center=(1521,948)
[4] CheckBox "Wi-Fi" center=(1041,409)
[5] Button "Privacy & Security" center=(649,415)

Instruction: tap network

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1818,949)
[1] TextField "Email" center=(591,455)
[2] Button "Sign in" center=(890,174)
[3] TextField "Password" center=(1830,517)
[4] Button "Create account" center=(694,705)

Instruction: select remember me

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1025,587)
[1] Button "Displays" center=(1610,389)
[2] Button "Network" center=(638,792)
[3] CheckBox "Wi-Fi" center=(1151,994)
[4] CheckBox "Bluetooth" center=(1688,233)

Instruction: tap bluetooth

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(727,496)
[1] CheckBox "Do Not Disturb" center=(1257,197)
[2] Button "Sound" center=(509,177)
[3] Button "Privacy & Security" center=(861,537)
[4] CheckBox "Wi-Fi" center=(194,314)
[5] Button "Displays" center=(1558,719)
[6] Slider "Brightness" center=(1128,390)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(190,625)
[1] Button "Reload" center=(307,87)
[2] Button "Run inference" center=(284,908)
[3] Link "Sign in" center=(1191,198)
[4] Link "Pricing" center=(1165,781)
[5] Button "New tab" center=(1516,401)
[6] Button "Bookmark" center=(298,283)
[7] Link "Docs" center=(504,761)
[8] Button "Extensions" center=(345,1032)
[9] Button "Go back" center=(1197,500)

Instruction: press pricing

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(919,902)
[1] TextField "Search files" center=(472,861)
[2] Button "Open" center=(1277,701)
[3] Button "Commit" center=(948,125)
[4] Button "Terminal" center=(618,655)

Instruction: tap run

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1189,870)
[1] CheckBox "Remember me" center=(997,886)
[2] TextField "Password" center=(527,636)
[3] TextField "Email" center=(883,202)
[4] Button "Sign in" center=(829,173)
[5] Button "Create account" center=(748,655)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1595,986)
[1] PopUpButton "Appearance" center=(1362,484)
[2] Button "Privacy & Security" center=(1490,848)
[3] Button "Network" center=(984,531)
[4] Button "Displays" center=(391,943)
[5] Button "Sound" center=(1579,304)

Instruction: select sound

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(1015,152)
[1] CheckBox "Bluetooth" center=(731,965)
[2] CheckBox "Wi-Fi" center=(871,810)
[3] PopUpButton "Appearance" center=(1496,911)
[4] Button "Privacy & Security" center=(669,268)
[5] Button "Network" center=(1871,570)
[6] Slider "Brightness" center=(1814,732)

Instruction: press brightness

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(647,168)
[1] Link "Account" center=(640,522)
[2] Button "Wishlist" center=(1230,924)
[3] Button "Checkout" center=(897,1024)
[4] Button "Buy now" center=(143,681)
[5] Button "Add to cart" center=(102,778)
[6] SearchField "Search products" center=(142,621)

Instruction: click buy now

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(822,751)
[1] Button "Save" center=(1635,1032)
[2] Button "Find" center=(1461,893)
[3] Button "Commit" center=(323,168)
[4] TextField "Search files" center=(789,307)
[5] Button "Run" center=(1256,197)
[6] Button "Open" center=(107,266)
[7] Button "Terminal" center=(884,840)

Instruction: click new file

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(995,287)
[1] Button "Pause" center=(262,1006)
[2] Button "Fullscreen" center=(1495,156)
[3] Slider "Volume" center=(1580,511)
[4] Button "Play" center=(417,879)

Instruction: select fullscreen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(1290,856)
[1] Button "Open" center=(496,262)
[2] Button "New File" center=(715,665)
[3] Button "Terminal" center=(1120,794)
[4] TextField "Search files" center=(598,499)
[5] Button "Save" center=(1680,353)

Instruction: type into search files

[Assistant] {"action":"type","id":4,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1025,846)
[1] Button "Filters" center=(1016,856)
[2] SearchField "Search products" center=(208,401)
[3] Button "Wishlist" center=(550,923)
[4] Button "Checkout" center=(1258,60)
[5] Button "Buy now" center=(1657,212)
[6] Link "Account" center=(380,481)

Instruction: select filters

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(117,760)
[1] Button "Terminal" center=(1342,96)
[2] Button "Find" center=(925,347)
[3] Button "Open" center=(1765,504)
[4] Button "Format" center=(872,109)
[5] Button "Save" center=(1810,900)
[6] Button "New File" center=(1360,359)
[7] Button "Commit" center=(656,189)
[8] Button "Run" center=(964,555)

Instruction: save the file

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(334,781)
[1] Button "Fullscreen" center=(1484,807)
[2] Slider "Volume" center=(165,792)
[3] Button "Play" center=(926,493)
[4] Button "Shuffle" center=(784,870)
[5] Button "Pause" center=(1807,314)
[6] Button "Next" center=(1209,270)

Instruction: press volume

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1554,836)
[1] Button "Next" center=(1415,644)
[2] Button "Pause" center=(1811,227)
[3] Button "Play" center=(1660,745)
[4] Button "Fullscreen" center=(750,218)

Instruction: make it full screen

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(516,323)
[1] Button "New File" center=(737,294)
[2] Button "Format" center=(1328,959)
[3] Button "Open" center=(1756,735)
[4] Button "Save" center=(466,91)
[5] Button "Find" center=(410,275)
[6] Button "Commit" center=(906,357)
[7] TextField "Search files" center=(247,905)

Instruction: select save

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(131,371)
[1] Button "Reload" center=(188,449)
[2] Link "Docs" center=(138,168)
[3] Button "Go back" center=(549,881)
[4] Button "Go forward" center=(310,374)
[5] Link "Pricing" center=(236,878)
[6] Button "Extensions" center=(98,539)
[7] Button "Bookmark" center=(1491,131)
[8] Button "Downloads" center=(788,1009)
[9] Link "Sign in" center=(770,372)

Instruction: click bookmark

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1714,743)
[1] Button "Wishlist" center=(1422,346)
[2] SearchField "Search products" center=(738,442)
[3] Button "Filters" center=(665,112)
[4] Button "Add to cart" center=(201,647)
[5] Link "Account" center=(1311,200)
[6] Button "Checkout" center=(800,872)

Instruction: press buy now

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1769,295)
[1] Button "Create account" center=(1724,1040)
[2] CheckBox "Remember me" center=(1540,1037)
[3] TextField "Email" center=(1854,393)
[4] Link "Forgot password" center=(83,397)

Instruction: select create account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(1058,918)
[1] Button "Buy now" center=(1589,361)
[2] Link "Account" center=(773,548)
[3] Button "Wishlist" center=(288,955)
[4] SearchField "Search products" center=(843,482)
[5] Button "Add to cart" center=(1673,276)
[6] Button "Checkout" center=(395,1021)

Instruction: add to cart

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1385,413)
[1] Button "New File" center=(1866,1011)
[2] Button "Find" center=(1506,482)
[3] Button "Format" center=(526,363)
[4] Button "Terminal" center=(309,49)
[5] TextField "Search files" center=(1829,837)
[6] Button "Commit" center=(41,509)
[7] Button "Save" center=(932,1002)
[8] Button "Run" center=(189,360)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(428,742)
[1] Button "Go back" center=(481,749)
[2] Button "Go forward" center=(905,64)
[3] Button "New tab" center=(1074,83)
[4] Button "Bookmark" center=(1566,823)
[5] SearchField "Search or enter address" center=(1254,78)
[6] Link "Pricing" center=(263,85)
[7] Button "Extensions" center=(201,978)
[8] Link "Docs" center=(622,823)

Instruction: press go back

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(615,858)
[1] Link "Account" center=(1874,440)
[2] Button "Checkout" center=(1384,157)
[3] Button "Add to cart" center=(830,1000)
[4] SearchField "Search products" center=(537,608)

Instruction: tap add to cart

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1260,86)
[1] Button "Pause" center=(757,682)
[2] Button "Fullscreen" center=(112,716)
[3] Button "Play" center=(1250,430)
[4] Button "Next" center=(724,449)
[5] Button "Shuffle" center=(1593,610)

Instruction: tap pause

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(269,729)
[1] TextField "Search files" center=(1409,1006)
[2] Button "Commit" center=(1811,337)
[3] Button "Format" center=(956,198)
[4] Button "Open" center=(1334,237)
[5] Button "Run" center=(978,255)
[6] Button "Terminal" center=(760,470)
[7] Button "Save" center=(619,58)
[8] Button "New File" center=(994,568)

Instruction: press find

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(985,953)
[1] Button "Extensions" center=(202,306)
[2] Button "Go forward" center=(833,532)
[3] Button "New tab" center=(579,617)
[4] Button "Reload" center=(112,1006)
[5] SearchField "Search or enter address" center=(930,428)
[6] Button "Go back" center=(1118,567)
[7] Button "Downloads" center=(1521,467)
[8] Link "Sign in" center=(960,421)
[9] Link "Pricing" center=(180,929)
[10] Button "Run inference" center=(1189,385)

Instruction: click extensions

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(549,988)
[1] Button "Terminal" center=(1577,131)
[2] Button "Open" center=(1071,89)
[3] TextField "Search files" center=(1321,711)
[4] Button "New File" center=(938,254)
[5] Button "Save" center=(1245,994)
[6] Button "Run" center=(340,56)

Instruction: select terminal

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1480,528)
[1] SearchField "Search products" center=(1027,853)
[2] Button "Add to cart" center=(535,58)
[3] Link "Account" center=(1426,244)
[4] Button "Wishlist" center=(243,427)
[5] Button "Checkout" center=(1054,744)

Instruction: press buy now

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(577,720)
[1] Button "Commit" center=(157,181)
[2] Button "Save" center=(807,1018)
[3] TextField "Search files" center=(1715,337)
[4] Button "Run" center=(730,368)
[5] Button "Find" center=(88,713)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(862,686)
[1] Button "Create account" center=(978,282)
[2] Button "Sign in" center=(1105,562)
[3] CheckBox "Remember me" center=(1019,178)
[4] TextField "Email" center=(674,426)
[5] Link "Forgot password" center=(239,441)

Instruction: select create account

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(497,360)
[1] CheckBox "Do Not Disturb" center=(362,669)
[2] Slider "Brightness" center=(669,673)
[3] Button "Displays" center=(502,58)
[4] CheckBox "Wi-Fi" center=(1047,748)
[5] Button "Sound" center=(1502,324)
[6] Button "Privacy & Security" center=(257,488)

Instruction: tap wi-fi

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(775,142)
[1] TextField "Password" center=(1108,660)
[2] TextField "Email" center=(1675,426)
[3] Link "Forgot password" center=(361,188)
[4] Button "Create account" center=(729,198)

Instruction: type into email

[Assistant] {"action":"type","id":2,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(305,269)
[1] Button "Next" center=(1876,1016)
[2] Slider "Volume" center=(627,79)
[3] Button "Pause" center=(802,431)
[4] Button "Shuffle" center=(1562,519)

Instruction: go fullscreen

[Assistant] {"action":"click","id":0}