[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1125,739)
[1] Button "Format" center=(124,139)
[2] Button "Open" center=(1128,692)
[3] Button "Terminal" center=(764,1019)
[4] Button "Save" center=(1769,482)
[5] Button "Run" center=(1290,980)
[6] TextField "Search files" center=(1375,918)

Instruction: select open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(681,600)
[1] Button "Bookmark" center=(61,728)
[2] Button "Reload" center=(57,891)
[3] Button "Extensions" center=(179,52)
[4] Button "Go forward" center=(882,140)
[5] Button "Downloads" center=(1115,146)
[6] Link "Sign in" center=(1443,496)

Instruction: select downloads

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1256,712)
[1] Button "Create account" center=(1870,895)
[2] Button "Sign in" center=(484,1013)
[3] Link "Forgot password" center=(1267,693)
[4] TextField "Email" center=(289,765)

Instruction: press remember me

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(606,827)
[1] Button "Save" center=(914,748)
[2] Button "Commit" center=(1407,828)
[3] Button "Find" center=(1617,371)
[4] TextField "Search files" center=(567,205)
[5] Button "Terminal" center=(1638,115)
[6] Button "Format" center=(607,835)
[7] Button "Open" center=(167,431)

Instruction: execute

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(334,593)
[1] Button "Sound" center=(1614,481)
[2] CheckBox "Wi-Fi" center=(487,668)
[3] Button "Privacy & Security" center=(1357,148)
[4] Button "Displays" center=(944,772)

Instruction: click do not disturb

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(350,756)
[1] Link "Docs" center=(715,966)
[2] Button "Reload" center=(841,628)
[3] SearchField "Search or enter address" center=(47,567)
[4] Link "Sign in" center=(1489,1002)
[5] Button "Downloads" center=(1618,542)
[6] Button "Go forward" center=(1019,486)
[7] Button "Extensions" center=(867,904)
[8] Button "Go back" center=(1530,839)

Instruction: open the docs

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(153,687)
[1] TextField "Search files" center=(1365,132)
[2] Button "Run" center=(788,587)
[3] Button "Format" center=(1787,1023)
[4] Button "New File" center=(736,122)
[5] Button "Commit" center=(1762,510)
[6] Button "Find" center=(1597,824)
[7] Button "Open" center=(633,703)

Instruction: press save

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(213,946)
[1] TextField "Password" center=(1335,488)
[2] Button "Sign in" center=(1545,234)
[3] TextField "Email" center=(326,76)
[4] Button "Create account" center=(435,65)
[5] CheckBox "Remember me" center=(1514,647)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(282,256)
[1] Button "Bookmark" center=(1692,709)
[2] Button "Go forward" center=(1406,575)
[3] Button "Downloads" center=(978,1002)
[4] Button "Reload" center=(1428,931)
[5] Link "Docs" center=(1830,589)
[6] Button "Go back" center=(345,798)
[7] Button "Extensions" center=(1226,278)
[8] Link "Pricing" center=(187,77)
[9] Button "New tab" center=(1546,184)
[10] SearchField "Search or enter address" center=(1041,709)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(1714,877)
[1] Button "Extensions" center=(257,156)
[2] Button "Go forward" center=(1058,649)
[3] Link "Docs" center=(1676,670)
[4] Link "Pricing" center=(491,104)
[5] Link "Sign in" center=(1687,247)

Instruction: sign in

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(600,184)
[1] Button "New tab" center=(1591,465)
[2] Button "Bookmark" center=(384,753)
[3] Link "Pricing" center=(797,606)
[4] Button "Go back" center=(377,489)
[5] Button "Go forward" center=(538,987)

Instruction: click new tab

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(54,223)
[1] Button "Shuffle" center=(222,763)
[2] Button "Fullscreen" center=(1374,647)
[3] Button "Play" center=(1856,330)
[4] Button "Pause" center=(892,750)

Instruction: play

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1293,1013)
[1] Button "Privacy & Security" center=(220,682)
[2] CheckBox "Do Not Disturb" center=(1732,841)
[3] CheckBox "Wi-Fi" center=(656,396)
[4] Button "Sound" center=(897,44)
[5] CheckBox "Bluetooth" center=(1182,281)
[6] Button "Network" center=(433,160)
[7] Slider "Brightness" center=(1125,1040)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(270,855)
[1] Button "Add to cart" center=(345,741)
[2] Button "Filters" center=(1768,1031)
[3] Link "Account" center=(1011,549)
[4] Button "Wishlist" center=(401,203)

Instruction: go to checkout

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(253,798)
[1] Link "Forgot password" center=(147,987)
[2] CheckBox "Remember me" center=(1140,220)
[3] Button "Sign in" center=(363,844)
[4] Button "Create account" center=(99,536)

Instruction: type into email

[Assistant] {"action":"type","id":0,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(240,282)
[1] Button "Privacy & Security" center=(1550,462)
[2] CheckBox "Wi-Fi" center=(1821,244)
[3] PopUpButton "Appearance" center=(809,952)
[4] CheckBox "Do Not Disturb" center=(491,115)
[5] CheckBox "Bluetooth" center=(1004,639)

Instruction: press appearance

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(592,247)
[1] CheckBox "Wi-Fi" center=(727,170)
[2] Button "Sound" center=(105,748)
[3] Button "Privacy & Security" center=(1204,101)
[4] CheckBox "Bluetooth" center=(614,179)
[5] PopUpButton "Appearance" center=(267,912)
[6] Button "Network" center=(993,509)

Instruction: select network

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(883,917)
[1] CheckBox "Wi-Fi" center=(1108,540)
[2] Slider "Brightness" center=(1805,261)
[3] CheckBox "Bluetooth" center=(1251,524)
[4] PopUpButton "Appearance" center=(1754,513)

Instruction: press bluetooth

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(130,81)
[1] CheckBox "Wi-Fi" center=(1003,791)
[2] CheckBox "Do Not Disturb" center=(1614,556)
[3] PopUpButton "Appearance" center=(108,475)
[4] Slider "Brightness" center=(1127,443)
[5] CheckBox "Bluetooth" center=(1021,869)
[6] Button "Displays" center=(1674,753)
[7] Button "Privacy & Security" center=(1722,140)

Instruction: click appearance

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(529,703)
[1] SearchField "Search products" center=(225,56)
[2] Link "Account" center=(1022,725)
[3] Button "Buy now" center=(664,205)
[4] Button "Wishlist" center=(1480,510)
[5] Button "Add to cart" center=(1797,624)

Instruction: tap checkout

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1869,747)
[1] Button "Displays" center=(721,583)
[2] Button "Sound" center=(106,343)
[3] CheckBox "Wi-Fi" center=(912,810)
[4] Button "Network" center=(592,754)
[5] Slider "Brightness" center=(1294,595)
[6] PopUpButton "Appearance" center=(1218,853)
[7] CheckBox "Do Not Disturb" center=(464,846)

Instruction: click brightness

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1307,81)
[1] Button "Filters" center=(788,415)
[2] Button "Buy now" center=(398,703)
[3] SearchField "Search products" center=(1184,761)
[4] Button "Add to cart" center=(177,614)
[5] Button "Checkout" center=(1492,832)

Instruction: press buy now

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(762,959)
[1] Button "Filters" center=(587,555)
[2] Button "Checkout" center=(1433,473)
[3] Button "Buy now" center=(607,95)
[4] Button "Wishlist" center=(169,421)
[5] Link "Account" center=(747,503)

Instruction: click wishlist

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(204,440)
[1] Button "Open" center=(1878,223)
[2] Button "Commit" center=(630,1021)
[3] Button "Save" center=(197,151)
[4] Button "New File" center=(377,322)

Instruction: tap open

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(949,795)
[1] Button "Privacy & Security" center=(1360,61)
[2] Button "Sound" center=(637,233)
[3] CheckBox "Bluetooth" center=(168,62)
[4] Button "Displays" center=(1825,861)
[5] PopUpButton "Appearance" center=(515,894)
[6] CheckBox "Do Not Disturb" center=(1104,700)
[7] CheckBox "Wi-Fi" center=(727,461)

Instruction: tap appearance

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(310,959)
[1] Button "Checkout" center=(1656,353)
[2] Button "Buy now" center=(391,848)
[3] Button "Wishlist" center=(572,272)
[4] SearchField "Search products" center=(1031,420)
[5] Button "Filters" center=(630,828)
[6] Link "Account" center=(1771,88)

Instruction: tap buy now

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(688,628)
[1] PopUpButton "Appearance" center=(299,491)
[2] CheckBox "Wi-Fi" center=(573,418)
[3] Slider "Brightness" center=(417,250)
[4] Button "Sound" center=(1191,764)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(288,982)
[1] TextField "Email" center=(91,454)
[2] Link "Forgot password" center=(1062,826)
[3] Button "Create account" center=(337,1010)
[4] Button "Sign in" center=(978,990)
[5] TextField "Password" center=(695,199)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1532,968)
[1] Button "Add to cart" center=(945,386)
[2] Button "Buy now" center=(1451,155)
[3] Button "Wishlist" center=(1538,530)
[4] Button "Filters" center=(122,182)
[5] Link "Account" center=(1428,604)
[6] Button "Checkout" center=(279,355)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(440,63)
[1] Button "Displays" center=(211,591)
[2] PopUpButton "Appearance" center=(1779,496)
[3] CheckBox "Bluetooth" center=(1426,727)
[4] Button "Privacy & Security" center=(218,1039)
[5] CheckBox "Do Not Disturb" center=(1088,443)
[6] CheckBox "Wi-Fi" center=(438,542)

Instruction: open privacy settings

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(64,331)
[1] Button "Fullscreen" center=(746,399)
[2] Slider "Volume" center=(670,1040)
[3] Button "Shuffle" center=(952,935)
[4] Button "Pause" center=(1188,129)
[5] Button "Previous" center=(481,785)

Instruction: tap previous

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(465,350)
[1] Button "Terminal" center=(1012,928)
[2] Button "Save" center=(1115,383)
[3] TextField "Search files" center=(175,995)
[4] Button "Run" center=(1249,814)
[5] Button "Format" center=(809,130)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1174,542)
[1] Button "Run" center=(126,882)
[2] Button "Save" center=(1732,349)
[3] Button "Commit" center=(1749,639)
[4] Button "Open" center=(662,545)
[5] Button "Find" center=(1185,797)
[6] Button "Format" center=(989,269)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(407,543)
[1] CheckBox "Bluetooth" center=(1448,700)
[2] CheckBox "Do Not Disturb" center=(541,69)
[3] CheckBox "Wi-Fi" center=(284,823)
[4] Button "Network" center=(607,544)
[5] Button "Privacy & Security" center=(176,653)
[6] Button "Sound" center=(1566,410)
[7] PopUpButton "Appearance" center=(1230,532)

Instruction: tap network

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(159,189)
[1] Button "Previous" center=(694,512)
[2] Button "Pause" center=(1838,749)
[3] Button "Shuffle" center=(317,405)
[4] Button "Next" center=(374,94)
[5] Button "Play" center=(598,409)
[6] Button "Fullscreen" center=(899,691)

Instruction: click pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1606,136)
[1] Button "Create account" center=(1266,810)
[2] CheckBox "Remember me" center=(382,596)
[3] TextField "Email" center=(700,933)
[4] Link "Forgot password" center=(687,886)

Instruction: select remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(66,180)
[1] Button "Pause" center=(1101,328)
[2] Slider "Volume" center=(803,837)
[3] Button "Play" center=(1033,461)
[4] Button "Next" center=(736,983)

Instruction: press previous

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(278,878)
[1] CheckBox "Do Not Disturb" center=(308,985)
[2] Button "Sound" center=(75,531)
[3] CheckBox "Wi-Fi" center=(753,359)
[4] Button "Displays" center=(1455,411)

Instruction: select wi-fi

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(669,168)
[1] Button "Privacy & Security" center=(469,173)
[2] CheckBox "Do Not Disturb" center=(200,475)
[3] PopUpButton "Appearance" center=(1454,821)
[4] CheckBox "Bluetooth" center=(1645,154)
[5] Slider "Brightness" center=(199,59)

Instruction: click appearance

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(128,107)
[1] Button "Network" center=(657,723)
[2] PopUpButton "Appearance" center=(61,812)
[3] Slider "Brightness" center=(237,841)
[4] Button "Displays" center=(293,982)
[5] Button "Sound" center=(1272,250)
[6] Button "Privacy & Security" center=(1243,962)

Instruction: select sound

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1718,632)
[1] Button "Go back" center=(1603,54)
[2] SearchField "Search or enter address" center=(1123,262)
[3] Link "Docs" center=(1505,236)
[4] Link "Pricing" center=(617,215)
[5] Button "Run inference" center=(1584,590)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1749,403)
[1] CheckBox "Wi-Fi" center=(817,474)
[2] Button "Network" center=(478,566)
[3] CheckBox "Bluetooth" center=(506,457)
[4] Button "Displays" center=(271,819)
[5] Button "Sound" center=(683,166)

Instruction: press displays

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(828,330)
[1] Button "New File" center=(975,654)
[2] Button "Open" center=(1547,359)
[3] Button "Commit" center=(1447,1039)
[4] Button "Save" center=(1762,494)

Instruction: make a commit

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(851,488)
[1] Button "Displays" center=(1110,714)
[2] Button "Privacy & Security" center=(1425,127)
[3] CheckBox "Wi-Fi" center=(284,186)
[4] PopUpButton "Appearance" center=(848,450)

Instruction: press network

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(769,527)
[1] Button "Reload" center=(1405,657)
[2] Button "Run inference" center=(868,536)
[3] Button "New tab" center=(1087,325)
[4] Link "Sign in" center=(341,826)
[5] Button "Go forward" center=(971,150)
[6] Button "Go back" center=(290,296)
[7] Link "Docs" center=(1042,549)
[8] SearchField "Search or enter address" center=(512,683)
[9] Button "Extensions" center=(315,70)
[10] Button "Bookmark" center=(243,795)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1393,809)
[1] TextField "Search files" center=(1104,825)
[2] Button "Commit" center=(1048,851)
[3] Button "Open" center=(994,246)
[4] Button "Terminal" center=(641,717)

Instruction: select terminal

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(825,500)
[1] Button "Buy now" center=(684,913)
[2] Button "Wishlist" center=(810,493)
[3] Button "Add to cart" center=(981,76)
[4] Link "Account" center=(107,221)

Instruction: press wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1443,592)
[1] Button "Open" center=(1211,521)
[2] Button "Save" center=(1014,615)
[3] Button "Find" center=(1789,200)
[4] Button "Run" center=(1077,97)
[5] Button "Format" center=(841,309)
[6] Button "Terminal" center=(1053,645)
[7] Button "New File" center=(813,671)
[8] Button "Commit" center=(1425,401)

Instruction: run the code

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(738,429)
[1] Button "Filters" center=(239,478)
[2] Button "Wishlist" center=(746,405)
[3] SearchField "Search products" center=(883,399)
[4] Button "Checkout" center=(155,717)
[5] Button "Buy now" center=(540,293)
[6] Button "Add to cart" center=(520,703)

Instruction: click account

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(983,713)
[1] Button "Create account" center=(1808,472)
[2] CheckBox "Remember me" center=(1203,567)
[3] TextField "Password" center=(902,313)
[4] Button "Sign in" center=(81,752)

Instruction: tap forgot password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1422,250)
[1] Button "Filters" center=(728,548)
[2] Button "Buy now" center=(1069,870)
[3] Button "Add to cart" center=(1606,723)
[4] SearchField "Search products" center=(735,210)
[5] Button "Checkout" center=(1299,346)
[6] Button "Wishlist" center=(784,1032)

Instruction: add this to my cart

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1319,800)
[1] SearchField "Search products" center=(228,774)
[2] Button "Filters" center=(1768,632)
[3] Button "Add to cart" center=(234,646)
[4] Link "Account" center=(729,319)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(888,451)
[1] Button "Terminal" center=(554,679)
[2] Button "Open" center=(1095,504)
[3] Button "Find" center=(1415,821)
[4] TextField "Search files" center=(1161,224)
[5] Button "Save" center=(1844,59)

Instruction: type into search files

[Assistant] {"action":"type","id":4,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1528,681)
[1] CheckBox "Bluetooth" center=(162,574)
[2] Button "Displays" center=(450,413)
[3] CheckBox "Do Not Disturb" center=(987,191)
[4] PopUpButton "Appearance" center=(532,194)
[5] Button "Privacy & Security" center=(1568,741)
[6] Button "Sound" center=(1607,347)
[7] Button "Network" center=(1768,485)
[8] Slider "Brightness" center=(880,366)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(528,156)
[1] TextField "Email" center=(347,236)
[2] Button "Create account" center=(96,291)
[3] CheckBox "Remember me" center=(1672,887)
[4] TextField "Password" center=(675,515)
[5] Button "Sign in" center=(1034,613)

Instruction: click remember me

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1519,537)
[1] Button "Sound" center=(1642,60)
[2] Button "Network" center=(153,920)
[3] Slider "Brightness" center=(1876,125)
[4] PopUpButton "Appearance" center=(682,656)

Instruction: tap privacy & security

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(269,624)
[1] Link "Forgot password" center=(1544,552)
[2] Button "Sign in" center=(287,574)
[3] Button "Create account" center=(1400,888)
[4] TextField "Email" center=(1228,1012)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(714,892)
[1] Link "Forgot password" center=(1236,794)
[2] Button "Sign in" center=(838,865)
[3] TextField "Password" center=(1572,801)
[4] TextField "Email" center=(523,997)
[5] Button "Create account" center=(906,263)

Instruction: reset password

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(688,598)
[1] Button "Open" center=(907,526)
[2] Button "New File" center=(1438,292)
[3] Button "Run" center=(608,940)
[4] Button "Save" center=(1646,999)

Instruction: select run

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1135,927)
[1] Button "Go forward" center=(524,918)
[2] Button "Downloads" center=(946,437)
[3] Link "Docs" center=(56,971)
[4] Link "Pricing" center=(844,989)
[5] Button "New tab" center=(563,348)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1656,74)
[1] Link "Forgot password" center=(601,927)
[2] Button "Sign in" center=(204,882)
[3] Button "Create account" center=(1636,1003)
[4] CheckBox "Remember me" center=(1046,522)
[5] TextField "Password" center=(1549,462)

Instruction: click sign in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(1353,696)
[1] CheckBox "Wi-Fi" center=(416,685)
[2] Slider "Brightness" center=(948,850)
[3] Button "Privacy & Security" center=(1352,179)
[4] PopUpButton "Appearance" center=(46,450)

Instruction: select brightness

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1444,460)
[1] Link "Account" center=(1663,1001)
[2] Button "Checkout" center=(694,502)
[3] SearchField "Search products" center=(1019,934)
[4] Button "Buy now" center=(1545,176)
[5] Button "Filters" center=(573,217)
[6] Button "Add to cart" center=(1447,647)

Instruction: add this to my cart

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1365,666)
[1] Button "Terminal" center=(1323,274)
[2] Button "Save" center=(222,1037)
[3] Button "Open" center=(987,243)
[4] Button "Find" center=(576,758)
[5] Button "Commit" center=(1784,212)
[6] Button "Run" center=(141,450)
[7] Button "New File" center=(1305,479)
[8] Button "Format" center=(1269,306)

Instruction: commit my changes

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(822,173)
[1] Button "Reload" center=(923,916)
[2] Button "Bookmark" center=(463,654)
[3] Button "New tab" center=(481,883)
[4] Button "Downloads" center=(619,784)
[5] Button "Go forward" center=(1426,233)
[6] Button "Go back" center=(896,224)
[7] SearchField "Search or enter address" center=(876,851)
[8] Link "Pricing" center=(338,69)
[9] Link "Docs" center=(634,1039)
[10] Link "Sign in" center=(576,281)
[11] Button "Extensions" center=(1305,501)

Instruction: tap go forward

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(235,610)
[1] SearchField "Search or enter address" center=(160,93)
[2] Button "Run inference" center=(721,993)
[3] Button "New tab" center=(520,522)
[4] Button "Bookmark" center=(765,660)
[5] Button "Go forward" center=(1413,1030)

Instruction: click go forward

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(573,213)
[1] Button "Go back" center=(1071,544)
[2] Link "Pricing" center=(734,938)
[3] Button "Go forward" center=(998,175)
[4] Button "Reload" center=(223,364)
[5] Button "Downloads" center=(1494,583)
[6] Button "Bookmark" center=(926,185)

Instruction: click pricing

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(681,486)
[1] PopUpButton "Appearance" center=(1868,208)
[2] CheckBox "Bluetooth" center=(686,794)
[3] Button "Privacy & Security" center=(1046,822)
[4] Button "Network" center=(349,609)
[5] CheckBox "Do Not Disturb" center=(1787,985)
[6] Button "Sound" center=(1386,691)
[7] Button "Displays" center=(1674,490)
[8] Slider "Brightness" center=(148,233)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(944,405)
[1] CheckBox "Remember me" center=(1492,614)
[2] TextField "Email" center=(1033,508)
[3] Link "Forgot password" center=(1049,155)
[4] Button "Create account" center=(1772,777)

Instruction: log in

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1117,825)
[1] TextField "Email" center=(1208,224)
[2] Button "Create account" center=(892,982)
[3] Button "Sign in" center=(197,292)
[4] CheckBox "Remember me" center=(1613,550)

Instruction: make a new account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(969,683)
[1] Button "Find" center=(591,648)
[2] Button "Run" center=(1387,45)
[3] Button "Save" center=(714,1033)
[4] Button "Terminal" center=(129,711)
[5] Button "New File" center=(1314,1013)
[6] Button "Commit" center=(967,372)
[7] Button "Open" center=(1538,791)

Instruction: run the code

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(856,301)
[1] Button "Pause" center=(404,285)
[2] Button "Shuffle" center=(1618,926)
[3] Slider "Volume" center=(856,131)
[4] Button "Play" center=(82,912)

Instruction: tap volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1547,865)
[1] Button "Sign in" center=(675,965)
[2] Link "Forgot password" center=(1442,657)
[3] TextField "Email" center=(308,407)
[4] TextField "Password" center=(1402,138)
[5] CheckBox "Remember me" center=(1744,416)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1297,230)
[1] Button "Open" center=(929,685)
[2] Button "Save" center=(912,135)
[3] Button "Run" center=(122,565)
[4] Button "Format" center=(739,1036)
[5] Button "Find" center=(989,732)
[6] Button "New File" center=(499,703)

Instruction: tap open

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(202,968)
[1] Button "Checkout" center=(1681,529)
[2] Button "Buy now" center=(655,192)
[3] Button "Add to cart" center=(301,740)
[4] SearchField "Search products" center=(655,901)
[5] Button "Filters" center=(549,575)

Instruction: search for something

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(1614,555)
[1] Button "Downloads" center=(73,156)
[2] SearchField "Search or enter address" center=(1556,717)
[3] Button "Reload" center=(987,223)
[4] Button "New tab" center=(1650,226)
[5] Button "Bookmark" center=(1766,442)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1694,311)
[1] Button "Pause" center=(739,1029)
[2] Button "Shuffle" center=(1824,869)
[3] Button "Previous" center=(423,256)
[4] Slider "Volume" center=(419,120)
[5] Button "Fullscreen" center=(1569,658)
[6] Button "Next" center=(571,932)

Instruction: click previous

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(1687,349)
[1] Link "Sign in" center=(219,168)
[2] SearchField "Search or enter address" center=(316,54)
[3] Button "New tab" center=(430,154)
[4] Button "Extensions" center=(1711,56)
[5] Button "Downloads" center=(1465,1033)
[6] Button "Bookmark" center=(425,919)
[7] Link "Pricing" center=(94,254)
[8] Button "Reload" center=(1025,704)
[9] Button "Go back" center=(337,705)
[10] Button "Go forward" center=(535,486)

Instruction: reload

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(1005,60)
[1] Button "Commit" center=(1800,500)
[2] Button "Find" center=(1238,288)
[3] Button "Open" center=(452,341)
[4] Button "Format" center=(1206,889)
[5] Button "Save" center=(1747,441)
[6] Button "Run" center=(144,381)
[7] Button "Terminal" center=(56,145)

Instruction: tap run

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1035,565)
[1] Button "Pause" center=(1250,782)
[2] Button "Play" center=(1292,997)
[3] Button "Next" center=(1429,966)
[4] Button "Shuffle" center=(1712,928)
[5] Button "Fullscreen" center=(1039,277)

Instruction: click shuffle

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(1363,395)
[1] Button "Terminal" center=(958,754)
[2] Button "Save" center=(933,61)
[3] Button "Find" center=(370,334)
[4] Button "Run" center=(1527,236)
[5] TextField "Search files" center=(1001,392)
[6] Button "New File" center=(937,277)
[7] Button "Open" center=(1079,264)
[8] Button "Format" center=(1243,970)

Instruction: press open

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1677,1034)
[1] TextField "Password" center=(1769,205)
[2] Button "Sign in" center=(1313,252)
[3] Link "Forgot password" center=(618,747)
[4] Button "Create account" center=(1574,128)
[5] CheckBox "Remember me" center=(1406,534)

Instruction: type into password

[Assistant] {"action":"type","id":1,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(136,58)
[1] Button "Displays" center=(774,942)
[2] CheckBox "Wi-Fi" center=(574,405)
[3] CheckBox "Do Not Disturb" center=(1426,202)
[4] Button "Sound" center=(397,887)
[5] Button "Network" center=(1800,955)
[6] Button "Privacy & Security" center=(1071,72)
[7] CheckBox "Bluetooth" center=(1692,105)
[8] Slider "Brightness" center=(731,480)

Instruction: click do not disturb

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(587,957)
[1] Button "Add to cart" center=(349,913)
[2] Button "Filters" center=(1453,934)
[3] SearchField "Search products" center=(1095,658)
[4] Button "Checkout" center=(1411,146)
[5] Button "Wishlist" center=(1486,91)
[6] Button "Buy now" center=(936,858)

Instruction: click buy now

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(1531,339)
[1] Link "Docs" center=(1490,492)
[2] Button "Bookmark" center=(1600,558)
[3] Button "Downloads" center=(1060,790)
[4] SearchField "Search or enter address" center=(1822,141)
[5] Link "Pricing" center=(355,702)
[6] Button "New tab" center=(784,382)

Instruction: tap reload

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(584,421)
[1] TextField "Search files" center=(1504,812)
[2] Button "Format" center=(889,409)
[3] Button "Find" center=(1329,720)
[4] Button "New File" center=(542,195)

Instruction: tap new file

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(1617,50)
[1] CheckBox "Bluetooth" center=(1509,865)
[2] CheckBox "Wi-Fi" center=(1099,1006)
[3] PopUpButton "Appearance" center=(182,126)
[4] Button "Sound" center=(732,362)
[5] Slider "Brightness" center=(862,830)

Instruction: tap displays

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(56,886)
[1] Button "Go forward" center=(612,961)
[2] Button "Downloads" center=(1120,665)
[3] Button "Run inference" center=(340,757)
[4] Link "Pricing" center=(867,496)
[5] Link "Docs" center=(1358,250)
[6] Button "Bookmark" center=(1857,597)
[7] Button "New tab" center=(1318,847)
[8] Button "Go back" center=(1306,707)
[9] Button "Reload" center=(945,754)

Instruction: press bookmark

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(477,135)
[1] Link "Forgot password" center=(1087,605)
[2] TextField "Password" center=(679,573)
[3] Button "Sign in" center=(737,815)
[4] CheckBox "Remember me" center=(1341,203)
[5] TextField "Email" center=(87,68)

Instruction: select remember me

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(137,814)
[1] Button "Network" center=(1373,1003)
[2] CheckBox "Bluetooth" center=(295,499)
[3] CheckBox "Do Not Disturb" center=(650,626)
[4] Button "Sound" center=(1549,45)

Instruction: press network

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(255,375)
[1] Button "Create account" center=(1097,636)
[2] Link "Forgot password" center=(434,882)
[3] TextField "Password" center=(1634,409)
[4] CheckBox "Remember me" center=(970,140)

Instruction: type into password

[Assistant] {"action":"type","id":3,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(796,733)
[1] Button "Find" center=(1117,452)
[2] Button "Commit" center=(1244,612)
[3] Button "Format" center=(1102,837)
[4] Button "Terminal" center=(848,932)

Instruction: tap format

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1482,565)
[1] Button "New File" center=(1807,581)
[2] Button "Run" center=(281,751)
[3] Button "Find" center=(475,506)
[4] Button "Format" center=(1009,73)
[5] Button "Open" center=(1568,200)

Instruction: click run

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1110,196)
[1] Button "Checkout" center=(281,420)
[2] Button "Buy now" center=(664,423)
[3] SearchField "Search products" center=(661,341)
[4] Button "Add to cart" center=(1035,68)

Instruction: tap checkout

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1025,626)
[1] Button "Run inference" center=(674,619)
[2] Button "Downloads" center=(966,961)
[3] Link "Sign in" center=(731,840)
[4] SearchField "Search or enter address" center=(812,526)
[5] Link "Docs" center=(1748,777)
[6] Button "Go back" center=(1747,660)
[7] Link "Pricing" center=(311,820)
[8] Button "Reload" center=(323,556)
[9] Button "Bookmark" center=(1343,511)
[10] Button "Extensions" center=(1599,500)

Instruction: select downloads

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1362,130)
[1] Link "Pricing" center=(1813,209)
[2] Button "New tab" center=(1488,65)
[3] Button "Go back" center=(1211,685)
[4] Button "Reload" center=(1670,830)
[5] Button "Bookmark" center=(1488,356)
[6] Button "Go forward" center=(1189,700)
[7] Link "Sign in" center=(293,699)
[8] Link "Docs" center=(1185,127)

Instruction: open the docs

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(603,845)
[1] Button "Checkout" center=(461,291)
[2] Button "Filters" center=(1739,762)
[3] Button "Add to cart" center=(1758,948)
[4] Button "Wishlist" center=(1226,494)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(935,224)
[1] Link "Account" center=(889,64)
[2] Button "Buy now" center=(1800,580)
[3] Button "Add to cart" center=(689,431)
[4] SearchField "Search products" center=(566,133)
[5] Button "Wishlist" center=(274,141)

Instruction: press buy now

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(95,770)
[1] PopUpButton "Appearance" center=(159,345)
[2] CheckBox "Wi-Fi" center=(838,141)
[3] Button "Privacy & Security" center=(695,185)
[4] Button "Network" center=(885,954)
[5] CheckBox "Do Not Disturb" center=(703,304)

Instruction: click sound

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(691,515)
[1] CheckBox "Bluetooth" center=(1172,456)
[2] CheckBox "Wi-Fi" center=(174,535)
[3] Button "Privacy & Security" center=(99,457)
[4] Button "Sound" center=(1337,279)

Instruction: select do not disturb

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1203,456)
[1] Button "Filters" center=(953,125)
[2] Button "Checkout" center=(1773,691)
[3] Button "Buy now" center=(1064,248)
[4] Button "Add to cart" center=(1643,1031)
[5] Link "Account" center=(331,422)

Instruction: press filters

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(632,708)
[1] Button "Buy now" center=(382,40)
[2] Link "Account" center=(163,995)
[3] Button "Add to cart" center=(1027,319)
[4] Button "Filters" center=(1264,296)
[5] SearchField "Search products" center=(1759,736)
[6] Button "Wishlist" center=(818,399)

Instruction: select buy now

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(471,665)
[1] Button "Downloads" center=(1073,1020)
[2] Button "Extensions" center=(1682,365)
[3] SearchField "Search or enter address" center=(1850,738)
[4] Link "Docs" center=(741,690)
[5] Button "Bookmark" center=(575,810)
[6] Button "Go back" center=(1489,503)
[7] Button "New tab" center=(599,291)
[8] Button "Go forward" center=(1396,369)
[9] Button "Run inference" center=(739,255)
[10] Link "Sign in" center=(981,1022)

Instruction: press pricing

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1178,956)
[1] Button "Next" center=(1466,760)
[2] Button "Shuffle" center=(1041,820)
[3] Button "Fullscreen" center=(844,591)
[4] Button "Previous" center=(206,846)
[5] Slider "Volume" center=(1835,704)

Instruction: click previous

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(402,899)
[1] Button "Fullscreen" center=(1612,871)
[2] Button "Pause" center=(1467,48)
[3] Button "Next" center=(295,513)
[4] Button "Play" center=(798,998)
[5] Button "Shuffle" center=(435,495)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(519,545)
[1] Button "Create account" center=(783,77)
[2] TextField "Email" center=(614,544)
[3] TextField "Password" center=(1481,763)
[4] Button "Sign in" center=(1281,254)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(300,305)
[1] SearchField "Search or enter address" center=(1207,916)
[2] Button "Go back" center=(506,942)
[3] Button "Downloads" center=(1111,786)
[4] Link "Docs" center=(1642,320)
[5] Button "Go forward" center=(432,908)

Instruction: select go forward

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(925,904)
[1] Button "Fullscreen" center=(873,320)
[2] Button "Next" center=(289,736)
[3] Button "Shuffle" center=(1101,517)
[4] Button "Play" center=(1054,410)
[5] Button "Previous" center=(1099,817)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1452,473)
[1] Button "Run inference" center=(192,979)
[2] SearchField "Search or enter address" center=(551,241)
[3] Button "Reload" center=(376,377)
[4] Button "Go forward" center=(676,104)
[5] Button "Bookmark" center=(1322,680)
[6] Button "Go back" center=(238,68)
[7] Button "Downloads" center=(1022,482)

Instruction: select bookmark

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1628,654)
[1] Button "Displays" center=(1308,489)
[2] Button "Network" center=(1604,137)
[3] Button "Sound" center=(1320,497)
[4] CheckBox "Wi-Fi" center=(1225,167)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(248,733)
[1] Button "Run inference" center=(154,612)
[2] SearchField "Search or enter address" center=(146,341)
[3] Link "Sign in" center=(1780,430)
[4] Link "Pricing" center=(862,138)
[5] Button "Bookmark" center=(1304,810)

Instruction: click pricing

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1099,376)
[1] Button "Format" center=(1243,237)
[2] Button "Open" center=(261,145)
[3] Button "Save" center=(65,649)
[4] Button "Find" center=(1500,109)
[5] Button "Run" center=(1254,848)
[6] TextField "Search files" center=(1761,238)
[7] Button "Terminal" center=(1213,169)

Instruction: search

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(301,359)
[1] Button "Next" center=(1237,486)
[2] Button "Play" center=(1090,1022)
[3] Button "Fullscreen" center=(1374,812)
[4] Button "Pause" center=(772,358)

Instruction: press pause

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(526,559)
[1] Button "Network" center=(1717,895)
[2] CheckBox "Wi-Fi" center=(1185,919)
[3] Button "Privacy & Security" center=(151,472)
[4] Button "Sound" center=(744,240)

Instruction: click network

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1301,346)
[1] CheckBox "Remember me" center=(897,376)
[2] Button "Sign in" center=(372,147)
[3] Link "Forgot password" center=(970,937)
[4] Button "Create account" center=(1656,972)
[5] TextField "Password" center=(126,547)

Instruction: type into email

[Assistant] {"action":"type","id":0,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(1024,792)
[1] Button "Play" center=(1613,1015)
[2] Button "Next" center=(1708,379)
[3] Button "Previous" center=(1693,797)
[4] Button "Fullscreen" center=(809,250)

Instruction: select volume

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(100,243)
[1] Button "New tab" center=(445,330)
[2] SearchField "Search or enter address" center=(396,264)
[3] Link "Docs" center=(1853,308)
[4] Button "Bookmark" center=(1512,218)

Instruction: press bookmark

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1381,846)
[1] PopUpButton "Appearance" center=(1852,153)
[2] Button "Sound" center=(733,998)
[3] Button "Displays" center=(470,352)
[4] Slider "Brightness" center=(974,285)
[5] CheckBox "Do Not Disturb" center=(797,868)
[6] CheckBox "Bluetooth" center=(802,716)
[7] CheckBox "Wi-Fi" center=(390,1003)

Instruction: click bluetooth

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1602,670)
[1] Button "Go back" center=(1242,233)
[2] Button "New tab" center=(1789,588)
[3] Link "Sign in" center=(642,848)
[4] Link "Pricing" center=(344,375)

Instruction: press new tab

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Checkout" center=(1081,437)
[1] Link "Account" center=(1578,943)
[2] Button "Filters" center=(1361,809)
[3] Button "Add to cart" center=(1082,423)
[4] Button "Buy now" center=(1061,571)

Instruction: select filters

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Add to cart" center=(1659,637)
[1] Button "Checkout" center=(1370,77)
[2] Button "Buy now" center=(464,264)
[3] SearchField "Search products" center=(1872,247)
[4] Button "Filters" center=(965,923)
[5] Button "Wishlist" center=(520,445)
[6] Link "Account" center=(487,353)

Instruction: find a product

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(1102,400)
[1] Button "Create account" center=(511,637)
[2] CheckBox "Remember me" center=(124,516)
[3] TextField "Password" center=(1089,676)
[4] TextField "Email" center=(728,476)
[5] Link "Forgot password" center=(963,518)

Instruction: tap remember me

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(583,700)
[1] TextField "Password" center=(195,407)
[2] Link "Forgot password" center=(208,341)
[3] TextField "Email" center=(1765,553)
[4] Button "Create account" center=(1090,599)

Instruction: click forgot password

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(1485,658)
[1] Link "Pricing" center=(322,915)
[2] SearchField "Search or enter address" center=(1580,773)
[3] Button "New tab" center=(1555,596)
[4] Link "Docs" center=(574,48)
[5] Button "Extensions" center=(1617,346)
[6] Button "Reload" center=(617,882)
[7] Button "Bookmark" center=(749,841)

Instruction: click reload

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(518,568)
[1] Button "Save" center=(1242,844)
[2] Button "Terminal" center=(1171,815)
[3] Button "Find" center=(999,779)
[4] Button "Open" center=(1707,512)
[5] Button "Run" center=(737,850)
[6] Button "Commit" center=(215,360)
[7] Button "New File" center=(1772,232)

Instruction: make a commit

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1699,714)
[1] Button "Filters" center=(1748,403)
[2] Button "Wishlist" center=(735,270)
[3] SearchField "Search products" center=(898,475)
[4] Button "Buy now" center=(1444,810)
[5] Button "Checkout" center=(866,821)

Instruction: select buy now

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1203,684)
[1] Button "Sign in" center=(1676,657)
[2] Button "Create account" center=(1284,627)
[3] CheckBox "Remember me" center=(1815,440)
[4] TextField "Email" center=(820,249)
[5] Link "Forgot password" center=(840,105)

Instruction: submit the login

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1172,221)
[1] SearchField "Search products" center=(105,886)
[2] Button "Filters" center=(1451,657)
[3] Button "Add to cart" center=(1040,67)
[4] Button "Checkout" center=(422,243)
[5] Link "Account" center=(182,973)
[6] Button "Wishlist" center=(830,330)

Instruction: click filters

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(266,172)
[1] Button "Run" center=(1138,468)
[2] Button "Commit" center=(1019,800)
[3] Button "New File" center=(180,438)
[4] Button "Open" center=(1812,251)
[5] TextField "Search files" center=(306,931)

Instruction: tap new file

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(952,105)
[1] Button "Privacy & Security" center=(1728,448)
[2] CheckBox "Wi-Fi" center=(1278,866)
[3] CheckBox "Do Not Disturb" center=(545,578)
[4] Slider "Brightness" center=(713,972)
[5] Button "Sound" center=(1205,940)

Instruction: click do not disturb

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1295,883)
[1] Button "Pause" center=(1698,910)
[2] Slider "Volume" center=(398,465)
[3] Button "Shuffle" center=(550,223)
[4] Button "Previous" center=(1305,790)
[5] Button "Next" center=(924,508)

Instruction: skip to next track

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1050,461)
[1] TextField "Search files" center=(1600,606)
[2] Button "Terminal" center=(288,417)
[3] Button "Find" center=(196,813)
[4] Button "New File" center=(255,556)
[5] Button "Open" center=(1609,838)

Instruction: find in file

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(743,803)
[1] Button "Buy now" center=(857,916)
[2] Button "Checkout" center=(438,446)
[3] SearchField "Search products" center=(117,50)
[4] Button "Filters" center=(553,157)
[5] Button "Wishlist" center=(1492,979)

Instruction: tap wishlist

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1763,205)
[1] Button "Format" center=(389,898)
[2] Button "Commit" center=(76,661)
[3] Button "Run" center=(245,221)
[4] TextField "Search files" center=(642,167)
[5] Button "Find" center=(1134,799)
[6] Button "Save" center=(840,212)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(110,954)
[1] Slider "Volume" center=(1422,585)
[2] Button "Play" center=(482,492)
[3] Button "Fullscreen" center=(1466,88)
[4] Button "Previous" center=(1753,690)
[5] Button "Pause" center=(1701,575)
[6] Button "Next" center=(1352,718)

Instruction: tap shuffle

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(551,423)
[1] Button "Pause" center=(528,244)
[2] Button "Shuffle" center=(227,56)
[3] Slider "Volume" center=(903,68)
[4] Button "Play" center=(69,385)

Instruction: go fullscreen

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(700,615)
[1] Button "Previous" center=(1323,234)
[2] Slider "Volume" center=(1319,252)
[3] Button "Play" center=(1149,444)
[4] Button "Fullscreen" center=(916,163)
[5] Button "Shuffle" center=(515,258)

Instruction: press volume

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(129,108)
[1] Button "New tab" center=(1853,263)
[2] Button "Reload" center=(881,348)
[3] Link "Sign in" center=(1611,653)
[4] Link "Pricing" center=(1063,144)
[5] Button "Bookmark" center=(1499,589)
[6] Button "Go forward" center=(1660,984)
[7] Button "Extensions" center=(1781,622)
[8] Button "Go back" center=(391,297)
[9] Button "Downloads" center=(234,654)

Instruction: press go forward

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1286,973)
[1] CheckBox "Remember me" center=(1238,499)
[2] TextField "Email" center=(823,563)
[3] Button "Create account" center=(1351,799)
[4] Button "Sign in" center=(516,555)

Instruction: i forgot my password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(321,423)
[1] TextField "Email" center=(1061,343)
[2] Button "Sign in" center=(453,195)
[3] CheckBox "Remember me" center=(809,249)
[4] Button "Create account" center=(1354,78)

Instruction: tap create account

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1661,464)
[1] Button "Play" center=(1367,872)
[2] Button "Previous" center=(630,808)
[3] Slider "Volume" center=(88,274)
[4] Button "Fullscreen" center=(1774,790)
[5] Button "Pause" center=(718,390)
[6] Button "Next" center=(536,250)

Instruction: press pause

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(945,684)
[1] Slider "Volume" center=(52,106)
[2] Button "Next" center=(1821,543)
[3] Button "Previous" center=(183,1016)
[4] Button "Pause" center=(434,93)

Instruction: press volume

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1524,518)
[1] CheckBox "Wi-Fi" center=(677,208)
[2] Slider "Brightness" center=(1517,242)
[3] Button "Sound" center=(541,474)
[4] PopUpButton "Appearance" center=(401,660)
[5] Button "Privacy & Security" center=(549,350)
[6] Button "Displays" center=(624,630)
[7] CheckBox "Do Not Disturb" center=(958,1030)

Instruction: tap appearance

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(1600,430)
[1] Button "Go forward" center=(915,914)
[2] Button "Go back" center=(210,817)
[3] Button "Extensions" center=(654,649)
[4] Button "New tab" center=(1140,196)

Instruction: select go forward

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1504,520)
[1] Button "Open" center=(440,198)
[2] Button "Save" center=(428,44)
[3] TextField "Search files" center=(66,922)
[4] Button "Commit" center=(1065,787)
[5] Button "New File" center=(1525,423)

Instruction: type into search files

[Assistant] {"action":"type","id":3,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(828,994)
[1] Button "Terminal" center=(1198,206)
[2] Button "Open" center=(437,821)
[3] Button "Save" center=(933,1020)
[4] Button "Find" center=(562,829)

Instruction: tap open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1018,75)
[1] Slider "Volume" center=(1705,410)
[2] Button "Pause" center=(1332,763)
[3] Button "Previous" center=(1425,68)
[4] Button "Play" center=(1468,721)

Instruction: tap pause

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1359,768)
[1] Button "Shuffle" center=(827,447)
[2] Button "Play" center=(1486,640)
[3] Slider "Volume" center=(204,570)
[4] Button "Fullscreen" center=(426,937)

Instruction: select volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(882,810)
[1] Button "Pause" center=(1834,986)
[2] Button "Fullscreen" center=(1530,930)
[3] Button "Previous" center=(222,151)
[4] Button "Shuffle" center=(1342,626)
[5] Slider "Volume" center=(1630,856)
[6] Button "Next" center=(745,873)

Instruction: press volume

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(238,111)
[1] Button "Pause" center=(1624,753)
[2] Button "Fullscreen" center=(895,577)
[3] Button "Play" center=(893,710)
[4] Button "Next" center=(630,790)

Instruction: select pause

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(451,100)
[1] Button "Privacy & Security" center=(1575,227)
[2] Button "Sound" center=(1697,817)
[3] CheckBox "Wi-Fi" center=(432,305)
[4] Button "Displays" center=(1825,458)
[5] CheckBox "Bluetooth" center=(942,681)

Instruction: press appearance

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(59,102)
[1] Button "Downloads" center=(584,809)
[2] Button "Bookmark" center=(54,721)
[3] Link "Sign in" center=(1232,848)
[4] Button "Run inference" center=(125,139)
[5] Button "Reload" center=(472,574)

Instruction: sign in

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1712,285)
[1] Button "Open" center=(779,721)
[2] TextField "Search files" center=(856,427)
[3] Button "Commit" center=(1290,98)
[4] Button "New File" center=(769,934)
[5] Button "Run" center=(716,865)
[6] Button "Find" center=(245,43)

Instruction: run the code

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(46,693)
[1] SearchField "Search or enter address" center=(1704,369)
[2] Link "Docs" center=(74,688)
[3] Button "Downloads" center=(777,934)
[4] Button "Extensions" center=(133,1032)
[5] Link "Pricing" center=(1693,793)
[6] Button "Reload" center=(706,734)
[7] Button "Bookmark" center=(949,279)

Instruction: click reload

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1283,239)
[1] TextField "Password" center=(1498,186)
[2] Button "Sign in" center=(1821,572)
[3] TextField "Email" center=(1834,625)
[4] CheckBox "Remember me" center=(1647,917)
[5] Button "Create account" center=(1507,556)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(1582,1006)
[1] Button "Run inference" center=(1391,756)
[2] Button "Downloads" center=(223,148)
[3] Button "Go forward" center=(930,772)
[4] Link "Docs" center=(99,204)
[5] Button "Go back" center=(708,964)
[6] Button "Bookmark" center=(825,824)

Instruction: navigate back

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1219,824)
[1] Slider "Volume" center=(1270,106)
[2] Button "Fullscreen" center=(1358,391)
[3] Button "Previous" center=(812,972)
[4] Button "Pause" center=(1509,800)

Instruction: tap previous

[Assistant] {"action":"click","id":3}