[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(249,732)
[1] Slider "Volume" center=(1556,953)
[2] Button "Shuffle" center=(1156,129)
[3] Button "Pause" center=(1249,472)
[4] Button "Next" center=(105,70)
[5] Button "Fullscreen" center=(231,263)
[6] Button "Play" center=(516,557)

Instruction: next song

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sound" center=(480,1020)
[1] CheckBox "Do Not Disturb" center=(1603,384)
[2] PopUpButton "Appearance" center=(249,134)
[3] Button "Network" center=(818,139)
[4] CheckBox "Bluetooth" center=(775,907)
[5] CheckBox "Wi-Fi" center=(744,658)

Instruction: tap sound

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(433,761)
[1] Button "Buy now" center=(182,86)
[2] Link "Account" center=(1394,273)
[3] Button "Checkout" center=(1623,336)
[4] Button "Filters" center=(203,915)
[5] Button "Wishlist" center=(516,927)
[6] Button "Add to cart" center=(246,429)

Instruction: select wishlist

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(1533,290)
[1] Button "Network" center=(374,513)
[2] Button "Displays" center=(817,316)
[3] CheckBox "Wi-Fi" center=(1350,744)
[4] CheckBox "Do Not Disturb" center=(1180,264)
[5] Slider "Brightness" center=(1442,372)
[6] CheckBox "Bluetooth" center=(1766,826)
[7] PopUpButton "Appearance" center=(1629,97)
[8] Button "Sound" center=(509,881)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1851,976)
[1] Button "Run" center=(1356,509)
[2] Button "Format" center=(332,311)
[3] Button "Commit" center=(325,292)
[4] Button "New File" center=(1565,614)
[5] TextField "Search files" center=(1143,309)
[6] Button "Terminal" center=(1569,638)
[7] Button "Open" center=(917,959)

Instruction: type into search files

[Assistant] {"action":"type","id":5,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1261,105)
[1] Button "Displays" center=(828,430)
[2] PopUpButton "Appearance" center=(1260,519)
[3] CheckBox "Bluetooth" center=(1123,297)
[4] Button "Sound" center=(1173,921)
[5] Button "Privacy & Security" center=(63,736)
[6] CheckBox "Wi-Fi" center=(1516,157)
[7] Slider "Brightness" center=(1436,946)

Instruction: select sound

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(1763,694)
[1] Link "Sign in" center=(1079,663)
[2] Button "Extensions" center=(447,196)
[3] Button "Go forward" center=(805,820)
[4] Button "Run inference" center=(370,592)
[5] SearchField "Search or enter address" center=(1634,984)
[6] Button "Downloads" center=(1126,980)
[7] Button "Go back" center=(41,653)
[8] Link "Pricing" center=(703,540)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(1538,537)
[1] Link "Forgot password" center=(1711,110)
[2] Button "Sign in" center=(1597,585)
[3] CheckBox "Remember me" center=(1608,168)
[4] TextField "Email" center=(302,715)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Wi-Fi" center=(1099,502)
[1] Button "Network" center=(287,293)
[2] Button "Sound" center=(500,105)
[3] CheckBox "Do Not Disturb" center=(732,61)
[4] Slider "Brightness" center=(1244,607)
[5] PopUpButton "Appearance" center=(511,642)
[6] CheckBox "Bluetooth" center=(491,47)
[7] Button "Privacy & Security" center=(185,764)

Instruction: click do not disturb

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1646,524)
[1] Link "Docs" center=(1693,456)
[2] Button "Reload" center=(429,136)
[3] Link "Pricing" center=(238,714)
[4] SearchField "Search or enter address" center=(922,402)
[5] Button "Run inference" center=(907,460)
[6] Button "Extensions" center=(996,924)
[7] Button "New tab" center=(1533,95)

Instruction: go to docs

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(904,227)
[1] Button "Create account" center=(610,513)
[2] CheckBox "Remember me" center=(551,935)
[3] Link "Forgot password" center=(194,493)
[4] TextField "Password" center=(1694,922)

Instruction: sign in

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(839,311)
[1] Button "Downloads" center=(1645,843)
[2] Button "New tab" center=(971,332)
[3] Button "Go back" center=(906,753)
[4] Link "Sign in" center=(1536,842)

Instruction: take me back

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1786,583)
[1] CheckBox "Bluetooth" center=(362,98)
[2] Button "Privacy & Security" center=(1080,122)
[3] Button "Displays" center=(1783,230)
[4] Slider "Brightness" center=(180,649)
[5] CheckBox "Do Not Disturb" center=(179,731)
[6] Button "Sound" center=(1805,280)
[7] PopUpButton "Appearance" center=(866,162)
[8] CheckBox "Wi-Fi" center=(1863,623)

Instruction: click network

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1415,700)
[1] Button "Find" center=(654,508)
[2] Button "Run" center=(687,991)
[3] Button "Terminal" center=(1579,998)
[4] Button "Commit" center=(188,49)
[5] Button "Open" center=(978,676)

Instruction: save the file

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(363,488)
[1] Link "Account" center=(1747,596)
[2] Button "Wishlist" center=(1480,349)
[3] Button "Add to cart" center=(1292,866)
[4] Button "Filters" center=(1379,581)
[5] Button "Buy now" center=(56,723)

Instruction: click wishlist

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Privacy & Security" center=(471,774)
[1] CheckBox "Do Not Disturb" center=(742,248)
[2] Button "Network" center=(1447,689)
[3] Slider "Brightness" center=(1786,310)
[4] PopUpButton "Appearance" center=(1075,540)
[5] Button "Sound" center=(554,967)
[6] CheckBox "Wi-Fi" center=(1772,92)
[7] CheckBox "Bluetooth" center=(229,689)
[8] Button "Displays" center=(907,889)

Instruction: open privacy settings

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(194,1007)
[1] Button "Play" center=(1848,747)
[2] Button "Fullscreen" center=(345,598)
[3] Slider "Volume" center=(113,894)
[4] Button "Previous" center=(796,636)

Instruction: press volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(524,925)
[1] Link "Docs" center=(372,1039)
[2] Link "Pricing" center=(1678,870)
[3] Button "Reload" center=(402,942)
[4] Button "Run inference" center=(884,65)
[5] Button "Go forward" center=(407,794)
[6] Link "Sign in" center=(720,841)

Instruction: i want to log in

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Brightness" center=(448,876)
[1] CheckBox "Bluetooth" center=(982,398)
[2] PopUpButton "Appearance" center=(665,880)
[3] Button "Network" center=(1668,932)
[4] CheckBox "Wi-Fi" center=(506,268)
[5] Button "Sound" center=(88,715)

Instruction: select appearance

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(262,650)
[1] Button "Bookmark" center=(929,393)
[2] Button "Extensions" center=(1532,845)
[3] Button "Go forward" center=(682,486)
[4] Button "Reload" center=(1281,563)

Instruction: refresh the page

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1400,982)
[1] Button "Next" center=(716,678)
[2] Button "Fullscreen" center=(682,719)
[3] Button "Pause" center=(1776,167)
[4] Slider "Volume" center=(1514,961)

Instruction: make it full screen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(1162,893)
[1] TextField "Search files" center=(40,351)
[2] Button "Terminal" center=(627,255)
[3] Button "Commit" center=(920,844)
[4] Button "Format" center=(1227,661)
[5] Button "Run" center=(1380,369)
[6] Button "Open" center=(992,492)
[7] Button "New File" center=(945,731)

Instruction: press new file

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(675,270)
[1] Button "Previous" center=(1691,243)
[2] Button "Fullscreen" center=(341,65)
[3] Button "Shuffle" center=(134,290)
[4] Slider "Volume" center=(1013,665)
[5] Button "Next" center=(1780,826)
[6] Button "Play" center=(189,506)

Instruction: next song

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New File" center=(1686,1020)
[1] Button "Terminal" center=(1465,570)
[2] TextField "Search files" center=(991,91)
[3] Button "Find" center=(1181,295)
[4] Button "Save" center=(1777,164)
[5] Button "Format" center=(974,176)

Instruction: select format

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(570,809)
[1] Button "Format" center=(546,900)
[2] Button "New File" center=(1345,323)
[3] Button "Open" center=(1608,836)
[4] Button "Run" center=(1107,536)
[5] Button "Find" center=(1323,284)
[6] Button "Terminal" center=(602,490)
[7] Button "Commit" center=(198,770)

Instruction: run the code

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(717,595)
[1] Button "Privacy & Security" center=(994,465)
[2] CheckBox "Wi-Fi" center=(167,251)
[3] Button "Displays" center=(1745,470)
[4] CheckBox "Bluetooth" center=(837,966)
[5] Button "Sound" center=(1616,638)
[6] Slider "Brightness" center=(1464,60)
[7] Button "Network" center=(1794,941)

Instruction: tap brightness

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search or enter address" center=(728,724)
[1] Button "Go back" center=(1430,857)
[2] Button "Extensions" center=(868,781)
[3] Link "Sign in" center=(378,900)
[4] Button "Reload" center=(997,981)
[5] Button "Go forward" center=(301,677)
[6] Link "Pricing" center=(1133,67)
[7] Button "Bookmark" center=(846,646)
[8] Button "Run inference" center=(1195,718)
[9] Link "Docs" center=(95,125)
[10] Button "New tab" center=(1356,478)

Instruction: press bookmark

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1003,59)
[1] Button "Extensions" center=(1574,592)
[2] Button "Go forward" center=(146,1014)
[3] Button "Run inference" center=(756,269)
[4] Link "Sign in" center=(1371,110)
[5] SearchField "Search or enter address" center=(1639,1020)

Instruction: press go back

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(383,660)
[1] Link "Sign in" center=(1283,1026)
[2] Button "Extensions" center=(1571,775)
[3] Button "Go forward" center=(274,836)
[4] Button "New tab" center=(1717,207)
[5] Button "Go back" center=(677,150)
[6] Button "Downloads" center=(1225,66)
[7] Link "Pricing" center=(678,629)
[8] SearchField "Search or enter address" center=(1427,969)
[9] Button "Bookmark" center=(808,446)

Instruction: click go forward

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1269,864)
[1] SearchField "Search products" center=(287,855)
[2] Button "Checkout" center=(1199,841)
[3] Button "Add to cart" center=(124,395)
[4] Button "Buy now" center=(1131,478)
[5] Button "Filters" center=(1394,419)
[6] Link "Account" center=(181,558)

Instruction: tap wishlist

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(1372,316)
[1] Button "Save" center=(1301,867)
[2] Button "Find" center=(1142,833)
[3] TextField "Search files" center=(1030,516)
[4] Button "Run" center=(932,885)
[5] Button "Format" center=(1537,646)
[6] Button "New File" center=(589,370)
[7] Button "Commit" center=(1784,291)
[8] Button "Open" center=(1741,997)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Displays" center=(766,856)
[1] CheckBox "Bluetooth" center=(569,388)
[2] Button "Sound" center=(612,941)
[3] PopUpButton "Appearance" center=(1260,758)
[4] Button "Network" center=(1842,322)
[5] CheckBox "Wi-Fi" center=(1178,50)

Instruction: toggle bluetooth

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(493,454)
[1] Button "New File" center=(1456,289)
[2] Button "Save" center=(667,719)
[3] Button "Find" center=(1231,417)
[4] Button "Commit" center=(1009,606)
[5] Button "Terminal" center=(1127,392)
[6] Button "Format" center=(911,803)

Instruction: select save

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(284,800)
[1] Button "Create account" center=(1137,1013)
[2] CheckBox "Remember me" center=(1600,746)
[3] TextField "Email" center=(419,236)
[4] TextField "Password" center=(483,796)
[5] Button "Sign in" center=(1031,323)

Instruction: type into password

[Assistant] {"action":"type","id":4,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(659,54)
[1] SearchField "Search products" center=(1490,586)
[2] Link "Account" center=(299,320)
[3] Button "Filters" center=(133,1037)
[4] Button "Buy now" center=(151,606)

Instruction: find a product

[Assistant] {"action":"type","id":1,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Docs" center=(1723,106)
[1] Button "Downloads" center=(860,543)
[2] SearchField "Search or enter address" center=(191,630)
[3] Button "Extensions" center=(1329,742)
[4] Button "Go forward" center=(149,195)
[5] Button "Reload" center=(345,870)

Instruction: type into search or enter address

[Assistant] {"action":"type","id":2,"text":"pioneer.ai"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(665,622)
[1] Button "Reload" center=(1311,101)
[2] SearchField "Search or enter address" center=(1288,1023)
[3] Button "Downloads" center=(1555,141)
[4] Link "Pricing" center=(1602,252)
[5] Button "New tab" center=(1321,256)
[6] Button "Run inference" center=(581,716)
[7] Button "Extensions" center=(206,200)
[8] Link "Docs" center=(531,217)
[9] Link "Sign in" center=(1170,116)
[10] Button "Go forward" center=(360,42)

Instruction: log in

[Assistant] {"action":"click","id":9}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(275,597)
[1] Slider "Brightness" center=(500,703)
[2] Button "Privacy & Security" center=(345,970)
[3] CheckBox "Bluetooth" center=(584,886)
[4] CheckBox "Wi-Fi" center=(331,113)
[5] Button "Network" center=(162,209)
[6] Button "Sound" center=(1663,354)
[7] PopUpButton "Appearance" center=(1258,806)

Instruction: turn on wi-fi

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(204,652)
[1] Button "Pause" center=(121,950)
[2] Slider "Volume" center=(924,792)
[3] Button "Fullscreen" center=(700,658)
[4] Button "Previous" center=(552,66)
[5] Button "Shuffle" center=(227,274)

Instruction: click previous

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1374,492)
[1] Button "Pause" center=(609,225)
[2] Button "Previous" center=(1238,486)
[3] Button "Play" center=(1340,873)
[4] Slider "Volume" center=(1047,1032)
[5] Button "Next" center=(226,521)
[6] Button "Fullscreen" center=(752,458)

Instruction: skip to next track

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(971,130)
[1] TextField "Password" center=(684,298)
[2] Link "Forgot password" center=(702,158)
[3] Button "Sign in" center=(1622,453)
[4] CheckBox "Remember me" center=(1811,566)

Instruction: type into email

[Assistant] {"action":"type","id":0,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(780,677)
[1] Button "Next" center=(1589,550)
[2] Button "Play" center=(1320,492)
[3] Button "Shuffle" center=(1596,92)
[4] Button "Previous" center=(456,313)
[5] Slider "Volume" center=(1164,174)
[6] Button "Fullscreen" center=(629,488)

Instruction: click shuffle

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(985,1009)
[1] Link "Sign in" center=(280,703)
[2] Button "Downloads" center=(1744,197)
[3] Button "Extensions" center=(1060,995)
[4] Link "Docs" center=(1507,338)
[5] Button "Go back" center=(1082,762)

Instruction: tap extensions

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(736,996)
[1] Button "Privacy & Security" center=(1653,559)
[2] CheckBox "Wi-Fi" center=(587,880)
[3] Button "Sound" center=(45,329)
[4] Button "Displays" center=(1526,345)
[5] CheckBox "Bluetooth" center=(1755,641)
[6] Button "Network" center=(1227,715)
[7] CheckBox "Do Not Disturb" center=(1042,926)

Instruction: select do not disturb

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1468,284)
[1] Button "Sign in" center=(1211,432)
[2] TextField "Password" center=(518,916)
[3] Button "Create account" center=(1628,460)
[4] CheckBox "Remember me" center=(129,365)

Instruction: make a new account

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(975,55)
[1] Button "Run" center=(1519,187)
[2] Button "Open" center=(879,931)
[3] TextField "Search files" center=(1380,1033)
[4] Button "Terminal" center=(356,116)
[5] Button "Commit" center=(1001,840)
[6] Button "Find" center=(582,386)
[7] Button "New File" center=(1316,749)
[8] Button "Save" center=(854,705)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(113,672)
[1] Button "Pause" center=(180,280)
[2] Button "Shuffle" center=(1332,740)
[3] Button "Next" center=(628,272)
[4] Button "Previous" center=(1569,132)
[5] Slider "Volume" center=(928,140)
[6] Button "Fullscreen" center=(1597,688)

Instruction: start playback

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run inference" center=(388,219)
[1] Link "Pricing" center=(201,664)
[2] Link "Docs" center=(1823,431)
[3] Link "Sign in" center=(1309,739)
[4] Button "Go forward" center=(533,549)
[5] Button "Bookmark" center=(1234,186)
[6] Button "Reload" center=(515,512)

Instruction: log in

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(975,905)
[1] Button "Open" center=(658,243)
[2] Button "Format" center=(827,913)
[3] TextField "Search files" center=(1029,149)
[4] Button "Terminal" center=(525,430)
[5] Button "Run" center=(1211,407)
[6] Button "Save" center=(1217,342)

Instruction: click format

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1628,858)
[1] Button "Pause" center=(511,661)
[2] Button "Shuffle" center=(1682,400)
[3] Slider "Volume" center=(488,691)
[4] Button "Play" center=(429,675)
[5] Button "Next" center=(553,734)

Instruction: skip to next track

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(787,789)
[1] Slider "Volume" center=(309,132)
[2] Button "Pause" center=(644,374)
[3] Button "Next" center=(1570,465)
[4] Button "Fullscreen" center=(399,245)
[5] Button "Shuffle" center=(310,845)
[6] Button "Play" center=(1144,937)

Instruction: make it full screen

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1584,1029)
[1] Button "Network" center=(502,920)
[2] CheckBox "Bluetooth" center=(1425,781)
[3] CheckBox "Wi-Fi" center=(1421,446)
[4] Button "Sound" center=(1772,863)
[5] CheckBox "Do Not Disturb" center=(1181,414)

Instruction: translate this to french

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(1422,279)
[1] Button "Run inference" center=(1005,65)
[2] Button "Extensions" center=(1308,945)
[3] Button "Bookmark" center=(1189,375)
[4] Button "Reload" center=(1289,266)
[5] Link "Docs" center=(1366,104)

Instruction: select bookmark

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(277,139)
[1] Slider "Volume" center=(520,948)
[2] Button "Fullscreen" center=(1140,178)
[3] Button "Shuffle" center=(835,504)
[4] Button "Pause" center=(799,726)
[5] Button "Play" center=(1561,753)

Instruction: press shuffle

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(948,495)
[1] TextField "Search files" center=(523,915)
[2] Button "Commit" center=(782,141)
[3] Button "Save" center=(1444,416)
[4] Button "New File" center=(1155,963)
[5] Button "Run" center=(1360,407)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(297,845)
[1] Button "Displays" center=(1196,250)
[2] Button "Network" center=(180,889)
[3] Button "Privacy & Security" center=(1607,607)
[4] PopUpButton "Appearance" center=(464,640)
[5] Button "Sound" center=(482,872)

Instruction: press network

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Filters" center=(265,716)
[1] Button "Wishlist" center=(1814,66)
[2] Link "Account" center=(309,55)
[3] SearchField "Search products" center=(773,848)
[4] Button "Buy now" center=(1654,283)

Instruction: press filters

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1015,499)
[1] CheckBox "Remember me" center=(1633,410)
[2] Button "Sign in" center=(1091,647)
[3] Button "Create account" center=(263,502)
[4] TextField "Password" center=(1071,266)

Instruction: select forgot password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(205,369)
[1] Button "New File" center=(1285,191)
[2] Button "Open" center=(174,169)
[3] Button "Terminal" center=(603,679)
[4] Button "Format" center=(1336,639)

Instruction: tap open

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1373,938)
[1] Link "Account" center=(1614,604)
[2] Button "Add to cart" center=(1516,926)
[3] Button "Filters" center=(480,480)
[4] Button "Wishlist" center=(964,949)
[5] Button "Checkout" center=(507,463)
[6] SearchField "Search products" center=(734,887)

Instruction: click filters

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(177,133)
[1] TextField "Email" center=(1743,127)
[2] TextField "Password" center=(230,482)
[3] Link "Forgot password" center=(237,802)
[4] Button "Sign in" center=(1553,421)
[5] Button "Create account" center=(1702,173)

Instruction: select sign in

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(1269,359)
[1] Button "Fullscreen" center=(760,146)
[2] Slider "Volume" center=(1223,559)
[3] Button "Shuffle" center=(475,198)
[4] Button "Next" center=(1384,533)
[5] Button "Previous" center=(499,907)

Instruction: next song

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(409,319)
[1] Button "Filters" center=(1479,820)
[2] Button "Checkout" center=(672,983)
[3] Button "Add to cart" center=(735,399)
[4] SearchField "Search products" center=(52,225)

Instruction: search for something

[Assistant] {"action":"type","id":4,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(362,418)
[1] CheckBox "Do Not Disturb" center=(678,778)
[2] Button "Privacy & Security" center=(704,835)
[3] Button "Network" center=(1202,650)
[4] Slider "Brightness" center=(213,944)
[5] Button "Displays" center=(147,199)
[6] Button "Sound" center=(362,812)

Instruction: click displays

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(599,260)
[1] TextField "Password" center=(1586,564)
[2] Button "Sign in" center=(273,393)
[3] Link "Forgot password" center=(920,153)
[4] CheckBox "Remember me" center=(620,734)
[5] Button "Create account" center=(1429,647)

Instruction: make a new account

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Terminal" center=(285,47)
[1] Button "Format" center=(1058,804)
[2] Button "Run" center=(921,219)
[3] Button "Find" center=(304,429)
[4] Button "Commit" center=(1130,760)
[5] Button "New File" center=(511,552)
[6] Button "Open" center=(1184,893)

Instruction: run the code

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(632,157)
[1] Button "Open" center=(869,61)
[2] Button "Format" center=(705,215)
[3] Button "Run" center=(1681,1010)
[4] Button "Find" center=(1305,511)
[5] Button "New File" center=(1743,746)
[6] TextField "Search files" center=(781,130)
[7] Button "Terminal" center=(934,904)

Instruction: type into search files

[Assistant] {"action":"type","id":6,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(722,837)
[1] Button "Checkout" center=(384,118)
[2] Button "Buy now" center=(1085,688)
[3] Button "Filters" center=(273,583)
[4] Button "Add to cart" center=(1084,238)

Instruction: select buy now

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(444,217)
[1] Button "Next" center=(1273,196)
[2] Button "Play" center=(1596,817)
[3] Slider "Volume" center=(1382,117)
[4] Button "Pause" center=(402,1014)
[5] Button "Shuffle" center=(1622,683)

Instruction: tap pause

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(940,109)
[1] Button "Pause" center=(1000,492)
[2] Button "Previous" center=(1333,350)
[3] Button "Next" center=(1670,321)
[4] Button "Fullscreen" center=(1251,97)

Instruction: start playback

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Reload" center=(1698,625)
[1] Button "Extensions" center=(1374,232)
[2] Link "Pricing" center=(698,659)
[3] Button "Bookmark" center=(1014,553)
[4] SearchField "Search or enter address" center=(348,1020)
[5] Link "Docs" center=(166,501)
[6] Button "Run inference" center=(251,870)
[7] Link "Sign in" center=(1754,391)
[8] Button "Go forward" center=(1502,126)
[9] Button "New tab" center=(1073,701)
[10] Button "Downloads" center=(393,80)
[11] Button "Go back" center=(547,764)

Instruction: i want to log in

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(174,377)
[1] CheckBox "Wi-Fi" center=(233,611)
[2] Button "Displays" center=(1429,435)
[3] Button "Privacy & Security" center=(621,298)
[4] Slider "Brightness" center=(1522,912)
[5] Button "Network" center=(1384,1027)
[6] CheckBox "Do Not Disturb" center=(1274,934)

Instruction: turn off bluetooth

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1184,425)
[1] TextField "Email" center=(1357,850)
[2] CheckBox "Remember me" center=(712,872)
[3] TextField "Password" center=(301,726)
[4] Button "Create account" center=(1479,888)
[5] Button "Sign in" center=(1553,741)

Instruction: reset password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1609,537)
[1] Button "Wishlist" center=(433,271)
[2] Link "Account" center=(321,198)
[3] SearchField "Search products" center=(198,342)
[4] Button "Filters" center=(1769,847)
[5] Button "Add to cart" center=(247,559)

Instruction: click add to cart

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1742,749)
[1] Button "Fullscreen" center=(1619,679)
[2] Button "Play" center=(1696,964)
[3] Button "Previous" center=(379,778)
[4] Button "Pause" center=(936,84)

Instruction: select fullscreen

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(793,734)
[1] Button "Next" center=(1208,491)
[2] Button "Shuffle" center=(985,827)
[3] Slider "Volume" center=(617,836)
[4] Button "Pause" center=(822,554)

Instruction: press next

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(623,125)
[1] CheckBox "Remember me" center=(1602,204)
[2] Link "Forgot password" center=(598,500)
[3] Button "Sign in" center=(1091,190)
[4] Button "Create account" center=(1741,487)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(807,269)
[1] Button "Open" center=(97,366)
[2] TextField "Search files" center=(242,899)
[3] Button "New File" center=(1503,705)
[4] Button "Commit" center=(726,850)

Instruction: search

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(78,302)
[1] Button "Play" center=(481,895)
[2] Button "Next" center=(346,601)
[3] Slider "Volume" center=(1529,663)
[4] Button "Previous" center=(1120,473)

Instruction: skip to next track

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Sign in" center=(636,479)
[1] Button "Downloads" center=(43,669)
[2] Button "Reload" center=(762,286)
[3] SearchField "Search or enter address" center=(1208,466)
[4] Button "New tab" center=(423,720)
[5] Button "Extensions" center=(1408,127)
[6] Button "Go back" center=(1112,1038)

Instruction: click new tab

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Open" center=(1654,109)
[1] TextField "Search files" center=(746,286)
[2] Button "Commit" center=(1541,712)
[3] Button "Find" center=(1326,146)
[4] Button "New File" center=(1620,635)
[5] Button "Run" center=(1545,814)
[6] Button "Format" center=(720,176)

Instruction: what time is it

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(1701,178)
[1] Link "Forgot password" center=(169,773)
[2] Button "Sign in" center=(1629,1006)
[3] Button "Create account" center=(977,77)
[4] TextField "Password" center=(640,246)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(599,167)
[1] Button "Extensions" center=(1677,416)
[2] SearchField "Search or enter address" center=(934,950)
[3] Button "Run inference" center=(859,801)
[4] Button "Go forward" center=(940,956)
[5] Button "Go back" center=(831,387)
[6] Button "Reload" center=(422,548)
[7] Link "Pricing" center=(1457,549)
[8] Link "Docs" center=(792,986)
[9] Link "Sign in" center=(1672,571)

Instruction: tap extensions

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Pricing" center=(953,657)
[1] Button "Go forward" center=(1509,476)
[2] Button "Bookmark" center=(381,746)
[3] Button "Extensions" center=(949,399)
[4] Button "Downloads" center=(955,83)
[5] Button "Go back" center=(1528,930)
[6] SearchField "Search or enter address" center=(762,669)
[7] Button "New tab" center=(930,321)
[8] Link "Sign in" center=(1349,850)

Instruction: tap go forward

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(298,108)
[1] CheckBox "Remember me" center=(523,837)
[2] Button "Create account" center=(1361,415)
[3] TextField "Email" center=(781,432)
[4] TextField "Password" center=(1201,73)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(855,361)
[1] Link "Forgot password" center=(1370,325)
[2] CheckBox "Remember me" center=(551,1023)
[3] TextField "Email" center=(272,66)
[4] Button "Create account" center=(1546,230)
[5] Button "Sign in" center=(1062,570)

Instruction: sign up

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(291,178)
[1] Button "Shuffle" center=(1786,115)
[2] Button "Fullscreen" center=(965,216)
[3] Button "Next" center=(1880,770)
[4] Slider "Volume" center=(951,129)

Instruction: go fullscreen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Wishlist" center=(1081,269)
[1] Button "Add to cart" center=(288,245)
[2] Button "Filters" center=(1662,182)
[3] Button "Buy now" center=(524,849)
[4] Link "Account" center=(1051,66)
[5] Button "Checkout" center=(779,607)

Instruction: press buy now

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(881,827)
[1] Button "Buy now" center=(878,883)
[2] SearchField "Search products" center=(1217,115)
[3] Button "Wishlist" center=(296,1035)
[4] Button "Checkout" center=(689,697)
[5] Button "Filters" center=(191,500)
[6] Button "Add to cart" center=(994,736)

Instruction: press buy now

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(700,997)
[1] Button "Sound" center=(1492,270)
[2] Button "Network" center=(748,1010)
[3] CheckBox "Bluetooth" center=(1102,956)
[4] Button "Displays" center=(621,906)
[5] Slider "Brightness" center=(201,296)

Instruction: select displays

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Do Not Disturb" center=(124,885)
[1] Button "Network" center=(1809,69)
[2] CheckBox "Wi-Fi" center=(206,86)
[3] Button "Privacy & Security" center=(1353,829)
[4] Slider "Brightness" center=(1221,311)
[5] Button "Sound" center=(1373,255)
[6] PopUpButton "Appearance" center=(1610,625)
[7] CheckBox "Bluetooth" center=(893,672)
[8] Button "Displays" center=(1348,71)

Instruction: select displays

[Assistant] {"action":"click","id":8}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(1450,101)
[1] Button "Fullscreen" center=(363,490)
[2] Slider "Volume" center=(891,535)
[3] Button "Shuffle" center=(991,248)
[4] Button "Pause" center=(736,661)

Instruction: select volume

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Run" center=(587,500)
[1] Button "Open" center=(547,184)
[2] Button "Terminal" center=(238,91)
[3] TextField "Search files" center=(634,1008)
[4] Button "Save" center=(826,924)

Instruction: press open

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Remember me" center=(1061,940)
[1] Button "Create account" center=(671,549)
[2] TextField "Email" center=(87,132)
[3] TextField "Password" center=(845,557)
[4] Button "Sign in" center=(976,1029)
[5] Link "Forgot password" center=(533,260)

Instruction: click remember me

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(922,177)
[1] TextField "Email" center=(1847,310)
[2] TextField "Password" center=(1529,414)
[3] CheckBox "Remember me" center=(1604,452)
[4] Link "Forgot password" center=(789,86)

Instruction: reset password

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1433,665)
[1] Button "Buy now" center=(283,171)
[2] Button "Filters" center=(238,443)
[3] SearchField "Search products" center=(804,853)
[4] Button "Wishlist" center=(734,611)
[5] Button "Add to cart" center=(788,813)
[6] Button "Checkout" center=(335,243)

Instruction: search for something

[Assistant] {"action":"type","id":3,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go back" center=(844,671)
[1] Button "Extensions" center=(1554,902)
[2] Button "Bookmark" center=(652,647)
[3] Link "Docs" center=(728,559)
[4] SearchField "Search or enter address" center=(1736,561)
[5] Button "New tab" center=(1130,541)
[6] Button "Go forward" center=(1491,616)

Instruction: click go forward

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1262,525)
[1] TextField "Password" center=(583,1030)
[2] CheckBox "Remember me" center=(1382,841)
[3] Button "Create account" center=(1629,632)
[4] TextField "Email" center=(1222,273)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(78,491)
[1] Button "Run" center=(682,468)
[2] TextField "Search files" center=(350,462)
[3] Button "Save" center=(1454,248)
[4] Button "Terminal" center=(880,553)
[5] Button "Format" center=(1627,666)

Instruction: click format

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(811,361)
[1] Button "Shuffle" center=(394,510)
[2] Button "Previous" center=(1131,390)
[3] Button "Next" center=(1158,402)
[4] Button "Fullscreen" center=(1424,830)
[5] Slider "Volume" center=(1816,778)
[6] Button "Pause" center=(1434,698)

Instruction: tap previous

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(689,531)
[1] CheckBox "Wi-Fi" center=(754,613)
[2] Button "Privacy & Security" center=(1668,778)
[3] Button "Sound" center=(600,334)
[4] CheckBox "Bluetooth" center=(289,627)
[5] CheckBox "Do Not Disturb" center=(1425,596)
[6] Slider "Brightness" center=(817,957)
[7] Button "Displays" center=(848,878)

Instruction: go to privacy and security

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(453,887)
[1] Button "Pause" center=(1143,317)
[2] Slider "Volume" center=(1191,752)
[3] Button "Shuffle" center=(596,180)
[4] Button "Previous" center=(263,670)

Instruction: press pause

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(1167,991)
[1] Button "New tab" center=(363,456)
[2] Link "Docs" center=(1377,997)
[3] Button "Downloads" center=(1843,939)
[4] Button "Reload" center=(1014,528)

Instruction: go to docs

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1573,1025)
[1] Button "Reload" center=(1826,988)
[2] Button "New tab" center=(632,937)
[3] Link "Pricing" center=(116,49)
[4] Link "Docs" center=(651,621)
[5] Link "Sign in" center=(1274,149)
[6] Button "Extensions" center=(726,331)

Instruction: sign in

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Network" center=(1511,230)
[1] Button "Sound" center=(68,794)
[2] Button "Displays" center=(730,849)
[3] Button "Privacy & Security" center=(644,621)
[4] CheckBox "Wi-Fi" center=(1421,815)
[5] CheckBox "Bluetooth" center=(435,219)
[6] CheckBox "Do Not Disturb" center=(1291,916)

Instruction: select privacy & security

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Extensions" center=(954,811)
[1] Button "Downloads" center=(586,1016)
[2] Button "New tab" center=(715,349)
[3] Button "Go back" center=(1232,780)
[4] Button "Run inference" center=(1213,51)
[5] Link "Pricing" center=(575,709)
[6] Button "Bookmark" center=(776,749)
[7] Link "Sign in" center=(523,103)

Instruction: sign in

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1298,266)
[1] Button "Pause" center=(488,1020)
[2] Button "Fullscreen" center=(313,529)
[3] Button "Play" center=(353,505)
[4] Button "Next" center=(1571,660)
[5] Button "Shuffle" center=(805,465)

Instruction: press play

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1737,600)
[1] Button "Network" center=(1075,247)
[2] Button "Privacy & Security" center=(1212,589)
[3] CheckBox "Wi-Fi" center=(346,208)
[4] CheckBox "Do Not Disturb" center=(711,914)

Instruction: press appearance

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(1091,465)
[1] Button "Pause" center=(975,617)
[2] Button "Fullscreen" center=(158,612)
[3] Button "Shuffle" center=(986,728)
[4] Button "Play" center=(1692,355)

Instruction: select shuffle

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(627,458)
[1] Link "Sign in" center=(408,827)
[2] Link "Docs" center=(317,824)
[3] Button "New tab" center=(1353,786)
[4] Link "Pricing" center=(1362,1026)

Instruction: tap new tab

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(405,590)
[1] Button "Run" center=(844,581)
[2] Button "New File" center=(300,785)
[3] Button "Terminal" center=(496,894)
[4] Button "Find" center=(47,814)

Instruction: call mom

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Save" center=(604,233)
[1] Button "New File" center=(1832,782)
[2] Button "Format" center=(1607,155)
[3] Button "Terminal" center=(105,868)
[4] Button "Open" center=(1392,469)
[5] Button "Run" center=(1299,824)

Instruction: save the file

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Go forward" center=(326,555)
[1] Button "Reload" center=(632,279)
[2] Link "Docs" center=(1710,978)
[3] Button "New tab" center=(1538,629)
[4] Button "Extensions" center=(693,631)

Instruction: open the docs

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(1251,940)
[1] Button "Play" center=(1545,973)
[2] Button "Shuffle" center=(398,681)
[3] Button "Next" center=(1430,477)
[4] Button "Previous" center=(1178,547)
[5] Button "Pause" center=(136,1035)
[6] Slider "Volume" center=(745,1004)

Instruction: select next

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Shuffle" center=(1009,286)
[1] Button "Play" center=(1779,270)
[2] Slider "Volume" center=(655,918)
[3] Button "Next" center=(1489,187)
[4] Button "Pause" center=(1690,514)
[5] Button "Fullscreen" center=(158,615)
[6] Button "Previous" center=(885,1031)

Instruction: press pause

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Format" center=(1523,889)
[1] Button "Terminal" center=(430,92)
[2] Button "Commit" center=(589,426)
[3] TextField "Search files" center=(1419,659)
[4] Button "Save" center=(1274,80)
[5] Button "New File" center=(1829,114)
[6] Button "Open" center=(425,863)
[7] Button "Find" center=(1599,641)
[8] Button "Run" center=(1523,727)

Instruction: search

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(536,752)
[1] CheckBox "Remember me" center=(1277,760)
[2] Button "Sign in" center=(458,446)
[3] Link "Forgot password" center=(530,605)
[4] TextField "Password" center=(698,834)
[5] TextField "Email" center=(619,429)

Instruction: type into email

[Assistant] {"action":"type","id":5,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(241,862)
[1] TextField "Password" center=(1519,520)
[2] Button "Create account" center=(1599,899)
[3] Link "Forgot password" center=(695,976)
[4] CheckBox "Remember me" center=(456,419)
[5] Button "Sign in" center=(680,463)

Instruction: order a pizza

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Slider "Volume" center=(715,275)
[1] Button "Shuffle" center=(817,623)
[2] Button "Fullscreen" center=(1746,291)
[3] Button "Previous" center=(1062,603)
[4] Button "Next" center=(1380,1006)

Instruction: tap fullscreen

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(326,777)
[1] Button "Pause" center=(1159,540)
[2] Button "Play" center=(416,978)
[3] Slider "Volume" center=(1861,595)
[4] Button "Previous" center=(1354,1020)

Instruction: delete my account permanently

[Assistant] {"action":"none"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "New tab" center=(487,557)
[1] Button "Bookmark" center=(973,422)
[2] Link "Pricing" center=(163,672)
[3] Button "Run inference" center=(1346,722)
[4] Button "Go forward" center=(1302,974)

Instruction: click run inference

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Bookmark" center=(259,980)
[1] Button "Go forward" center=(237,576)
[2] Link "Pricing" center=(344,286)
[3] Button "Run inference" center=(433,674)
[4] Button "Reload" center=(1118,298)
[5] Button "Downloads" center=(1709,402)
[6] Button "New tab" center=(587,854)

Instruction: tap pricing

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(812,605)
[1] Button "Open" center=(1014,97)
[2] Button "Terminal" center=(1344,49)
[3] Button "Find" center=(1479,215)
[4] Button "Save" center=(209,551)
[5] Button "Format" center=(1771,484)
[6] Button "New File" center=(1360,843)
[7] TextField "Search files" center=(718,619)

Instruction: click open

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] CheckBox "Bluetooth" center=(1781,362)
[1] Button "Displays" center=(72,1024)
[2] PopUpButton "Appearance" center=(464,641)
[3] Button "Privacy & Security" center=(335,810)
[4] Button "Sound" center=(1720,769)
[5] CheckBox "Wi-Fi" center=(1721,447)

Instruction: press appearance

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1523,120)
[1] Button "Add to cart" center=(1065,803)
[2] Button "Filters" center=(748,94)
[3] Button "Checkout" center=(240,487)
[4] Button "Wishlist" center=(515,898)
[5] SearchField "Search products" center=(195,388)
[6] Button "Buy now" center=(1279,827)

Instruction: tap wishlist

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1172,1003)
[1] CheckBox "Remember me" center=(808,482)
[2] TextField "Password" center=(420,736)
[3] Button "Create account" center=(1239,716)
[4] Button "Sign in" center=(818,127)
[5] TextField "Email" center=(1621,673)

Instruction: click remember me

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(777,819)
[1] Button "Go forward" center=(592,144)
[2] Button "Bookmark" center=(311,129)
[3] Button "Extensions" center=(423,481)
[4] Button "Go back" center=(959,610)
[5] Link "Pricing" center=(1176,564)
[6] Link "Sign in" center=(875,145)
[7] Button "Run inference" center=(93,131)
[8] SearchField "Search or enter address" center=(764,607)
[9] Link "Docs" center=(230,651)
[10] Button "Reload" center=(1264,843)
[11] Button "New tab" center=(704,924)

Instruction: go back

[Assistant] {"action":"click","id":4}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(324,315)
[1] Button "Previous" center=(656,314)
[2] Button "Next" center=(1049,189)
[3] Button "Shuffle" center=(169,211)
[4] Button "Fullscreen" center=(930,322)

Instruction: click shuffle

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(259,178)
[1] TextField "Email" center=(663,46)
[2] CheckBox "Remember me" center=(850,380)
[3] Button "Create account" center=(1763,675)
[4] TextField "Password" center=(818,863)
[5] Button "Sign in" center=(1805,377)

Instruction: press create account

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(855,1031)
[1] CheckBox "Remember me" center=(693,338)
[2] Button "Create account" center=(1545,751)
[3] TextField "Password" center=(1336,542)
[4] Link "Forgot password" center=(1219,840)

Instruction: type into password

[Assistant] {"action":"type","id":3,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1436,826)
[1] CheckBox "Remember me" center=(1166,67)
[2] Link "Forgot password" center=(1529,512)
[3] TextField "Email" center=(1490,254)
[4] Button "Sign in" center=(937,856)

Instruction: type into password

[Assistant] {"action":"type","id":0,"text":"hunter2"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(1656,426)
[1] Button "Save" center=(191,499)
[2] Button "Run" center=(1264,530)
[3] Button "Commit" center=(1232,450)
[4] Button "Open" center=(1134,558)
[5] TextField "Search files" center=(1821,748)
[6] Button "Format" center=(896,596)
[7] Button "New File" center=(115,849)

Instruction: save the file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Previous" center=(1670,480)
[1] Slider "Volume" center=(256,50)
[2] Button "Fullscreen" center=(241,304)
[3] Button "Play" center=(494,562)
[4] Button "Pause" center=(1569,571)
[5] Button "Next" center=(1182,634)

Instruction: tap volume

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Sign in" center=(254,342)
[1] TextField "Email" center=(891,982)
[2] Link "Forgot password" center=(215,158)
[3] TextField "Password" center=(1742,1003)
[4] CheckBox "Remember me" center=(1535,185)
[5] Button "Create account" center=(753,359)

Instruction: type into email

[Assistant] {"action":"type","id":1,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(177,347)
[1] Link "Account" center=(131,869)
[2] Button "Buy now" center=(1495,157)
[3] Button "Add to cart" center=(86,909)
[4] Button "Filters" center=(742,703)
[5] Button "Checkout" center=(263,733)

Instruction: check out

[Assistant] {"action":"click","id":5}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1329,228)
[1] Button "Add to cart" center=(421,695)
[2] Button "Filters" center=(1386,482)
[3] Button "Wishlist" center=(850,69)
[4] SearchField "Search products" center=(1550,668)
[5] Button "Checkout" center=(1848,242)

Instruction: tap wishlist

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(977,156)
[1] Button "Reload" center=(577,726)
[2] Button "Run inference" center=(1806,541)
[3] Link "Pricing" center=(1120,1029)
[4] Link "Docs" center=(1348,360)
[5] Button "Go back" center=(1262,437)
[6] Link "Sign in" center=(1291,441)
[7] Button "Go forward" center=(1242,156)

Instruction: press go forward

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Pause" center=(281,284)
[1] Button "Previous" center=(665,159)
[2] Button "Fullscreen" center=(414,422)
[3] Button "Play" center=(1460,184)
[4] Slider "Volume" center=(1086,437)
[5] Button "Next" center=(896,650)

Instruction: start playback

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Play" center=(423,362)
[1] Button "Next" center=(1764,501)
[2] Button "Previous" center=(1308,94)
[3] Button "Fullscreen" center=(1812,407)
[4] Button "Pause" center=(59,536)

Instruction: tap fullscreen

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Find" center=(638,73)
[1] Button "New File" center=(602,270)
[2] Button "Save" center=(1141,335)
[3] Button "Open" center=(384,839)
[4] TextField "Search files" center=(976,619)
[5] Button "Run" center=(1560,831)

Instruction: type into search files

[Assistant] {"action":"type","id":4,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Account" center=(1724,814)
[1] Button "Buy now" center=(125,818)
[2] Button "Wishlist" center=(1512,491)
[3] Button "Checkout" center=(1158,263)
[4] Button "Filters" center=(910,144)
[5] SearchField "Search products" center=(1547,889)
[6] Button "Add to cart" center=(1378,809)

Instruction: tap add to cart

[Assistant] {"action":"click","id":6}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(1700,795)
[1] CheckBox "Remember me" center=(1244,745)
[2] Link "Forgot password" center=(758,921)
[3] Button "Create account" center=(1495,664)
[4] TextField "Email" center=(1335,478)

Instruction: press remember me

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(1212,313)
[1] Slider "Brightness" center=(1797,566)
[2] Button "Network" center=(1431,874)
[3] CheckBox "Bluetooth" center=(235,178)
[4] Button "Privacy & Security" center=(1576,844)
[5] CheckBox "Do Not Disturb" center=(882,980)
[6] Button "Displays" center=(161,323)

Instruction: press brightness

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] PopUpButton "Appearance" center=(808,502)
[1] Slider "Brightness" center=(194,688)
[2] CheckBox "Wi-Fi" center=(1667,941)
[3] Button "Displays" center=(1614,135)
[4] Button "Network" center=(865,567)
[5] CheckBox "Bluetooth" center=(1575,324)
[6] Button "Privacy & Security" center=(1462,998)
[7] Button "Sound" center=(1836,417)

Instruction: select sound

[Assistant] {"action":"click","id":7}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] SearchField "Search products" center=(1669,587)
[1] Button "Buy now" center=(854,263)
[2] Button "Checkout" center=(919,884)
[3] Button "Wishlist" center=(473,546)
[4] Button "Filters" center=(589,370)

Instruction: check out

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Buy now" center=(1645,896)
[1] SearchField "Search products" center=(981,56)
[2] Link "Account" center=(294,991)
[3] Button "Add to cart" center=(1796,201)
[4] Button "Checkout" center=(943,507)
[5] Button "Wishlist" center=(41,1021)
[6] Button "Filters" center=(916,247)

Instruction: type into search products

[Assistant] {"action":"type","id":1,"text":"wireless headphones"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Create account" center=(208,420)
[1] CheckBox "Remember me" center=(1400,965)
[2] Link "Forgot password" center=(1368,209)
[3] TextField "Email" center=(146,445)
[4] Button "Sign in" center=(1321,352)
[5] TextField "Password" center=(1526,756)

Instruction: type into email

[Assistant] {"action":"type","id":3,"text":"alex@example.com"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Next" center=(311,647)
[1] Button "Pause" center=(503,571)
[2] Button "Previous" center=(1432,495)
[3] Slider "Volume" center=(59,48)
[4] Button "Fullscreen" center=(1464,391)

Instruction: press volume

[Assistant] {"action":"click","id":3}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Commit" center=(320,431)
[1] Button "New File" center=(1624,872)
[2] TextField "Search files" center=(1582,935)
[3] Button "Find" center=(312,696)
[4] Button "Format" center=(1426,186)
[5] Button "Save" center=(180,583)
[6] Button "Terminal" center=(1193,48)

Instruction: tap new file

[Assistant] {"action":"click","id":1}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Email" center=(217,958)
[1] Button "Sign in" center=(245,183)
[2] Button "Create account" center=(1581,166)
[3] Link "Forgot password" center=(1247,787)
[4] TextField "Password" center=(826,399)

Instruction: press create account

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(1776,907)
[1] Button "Create account" center=(1388,936)
[2] Button "Sign in" center=(657,250)
[3] TextField "Password" center=(1100,661)
[4] TextField "Email" center=(1083,233)

Instruction: log in

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Search files" center=(688,175)
[1] Button "Format" center=(832,921)
[2] Button "New File" center=(1758,622)
[3] Button "Save" center=(901,414)
[4] Button "Find" center=(1158,739)
[5] Button "Run" center=(393,821)
[6] Button "Terminal" center=(1004,1017)

Instruction: type into search files

[Assistant] {"action":"type","id":0,"text":"main.swift"}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Downloads" center=(1477,451)
[1] SearchField "Search or enter address" center=(208,695)
[2] Link "Sign in" center=(1138,590)
[3] Button "Reload" center=(1301,196)
[4] Link "Pricing" center=(608,125)
[5] Link "Docs" center=(674,120)
[6] Button "Extensions" center=(1087,983)
[7] Button "Bookmark" center=(456,1037)
[8] Button "New tab" center=(1694,199)
[9] Button "Go back" center=(1138,374)
[10] Button "Go forward" center=(840,640)

Instruction: click go forward

[Assistant] {"action":"click","id":10}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Link "Forgot password" center=(929,158)
[1] Button "Sign in" center=(968,666)
[2] TextField "Email" center=(1286,96)
[3] TextField "Password" center=(673,722)
[4] Button "Create account" center=(1558,713)

Instruction: i forgot my password

[Assistant] {"action":"click","id":0}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] Button "Fullscreen" center=(801,169)
[1] Button "Shuffle" center=(1575,839)
[2] Slider "Volume" center=(1828,315)
[3] Button "Next" center=(1541,947)
[4] Button "Previous" center=(281,829)

Instruction: tap volume

[Assistant] {"action":"click","id":2}


[System] You are a UI grounding model. Given a numbered list of on-screen elements (label + pixel center) and an instruction, pick the single best element. Reply with ONLY JSON: {"action":"click"|"type","id":<int>,"text":<string if typing>}. If nothing matches, reply {"action":"none"}.

[User]
Elements:
[0] TextField "Password" center=(972,563)
[1] CheckBox "Remember me" center=(777,96)
[2] Button "Create account" center=(1063,919)
[3] Button "Sign in" center=(1200,210)
[4] TextField "Email" center=(793,201)
[5] Link "Forgot password" center=(562,831)

Instruction: type into email

[Assistant] {"action":"type","id":4,"text":"alex@example.com"}