Update README.md

README.md

@@ -1,4 +1,4 @@
-[Github](https://github.com/showlab/ShowUI) | [Quick Start](#quickstart)
+[Github](https://github.com/showlab/ShowUI) | [Quick Start: UI-Grounding](#uigrounding) | [Quick Start: UI-Navigation](#uinavigation)
 
 ## ⭐ Quick Start {#quickstart}
 
@@ -34,7 +34,7 @@ max_pixels = 1344*28*28
 processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-2B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)
 ```
 
-2. Load screenshot and query
+2. **UI Grounding** {#uigrounding}
 ```python
 img_url = 'web_dbd7514b-9ca3-40cd-b09a-990f7b955da1.png'
 query = "Nahant"
@@ -82,3 +82,98 @@ draw_point(img_url, click_xy, 10)
 This will visualize the grounding results like (where the red points are [x,y])
 
 ![download](https://github.com/user-attachments/assets/8fe2783d-05b6-44e6-a26c-8718d02b56cb)
+
+3. **UI Navigation** {#uinavigation}
+- Set up system prompt.
+```python
+_NAV_SYSTEM = """You are an assistant trained to navigate the {_APP} screen. 
+Given a task instruction, a screen observation, and an action history sequence, 
+output the next action and wait for the next observation. 
+Here is the action space:
+{_ACTION_SPACE}
+"""
+
+_NAV_FORMAT = """
+Format the action as a dictionary with the following keys:
+{'action': 'ACTION_TYPE', 'value': 'element', 'position': [x,y]}
+
+If value or position is not applicable, set it as `None`.
+Position might be [[x1,y1], [x2,y2]] if the action requires a start and end position.
+Position represents the relative coordinates on the screenshot and should be scaled to a range of 0-1.
+"""
+
+action_map = {
+'web': """
+1. `CLICK`: Click on an element, value is not applicable and the position [x,y] is required. 
+2. `INPUT`: Type a string into an element, value is a string to type and the position [x,y] is required. 
+3. `SELECT`: Select a value for an element, value is not applicable and the position [x,y] is required. 
+4. `HOVER`: Hover on an element, value is not applicable and the position [x,y] is required.
+5. `ANSWER`: Answer the question, value is the answer and the position is not applicable.
+6. `ENTER`: Enter operation, value and position are not applicable.
+7. `SCROLL`: Scroll the screen, value is the direction to scroll and the position is not applicable.
+8. `SELECT_TEXT`: Select some text content, value is not applicable and position [[x1,y1], [x2,y2]] is the start and end position of the select operation.
+9. `COPY`: Copy the text, value is the text to copy and the position is not applicable.
+""",
+
+'phone': """
+1. `INPUT`: Type a string into an element, value is not applicable and the position [x,y] is required. 
+2. `SWIPE`: Swipe the screen, value is not applicable and the position [[x1,y1], [x2,y2]] is the start and end position of the swipe operation.
+3. `TAP`: Tap on an element, value is not applicable and the position [x,y] is required.
+4. `ANSWER`: Answer the question, value is the status (e.g., 'task complete') and the position is not applicable.
+5. `ENTER`: Enter operation, value and position are not applicable.
+"""
+}
+
+_NAV_USER = """{system}
+Task: {task}
+Observation: <|image_1|>
+Action History: {action_history}
+What is the next action?
+"""
+```
+
+```python
+img_url = 'chrome.png'
+split='web'
+system_prompt = _NAV_SYSTEM.format(_APP=split, _ACTION_SPACE=action_map[split])
+query = "Search the weather for the New York city."
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "text", "text": system_prompt},
+            {"type": "image", "image": img_url, "min_pixels": min_pixels, "max_pixels": max_pixels},
+            {"type": "text", "text": query}
+        ],
+    }
+]
+
+text = processor.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True,
+)
+image_inputs, video_inputs = process_vision_info(messages)
+inputs = processor(
+    text=[text],
+    images=image_inputs,
+    videos=video_inputs,
+    padding=True,
+    return_tensors="pt",
+)
+inputs = inputs.to("cuda")
+
+generated_ids = model.generate(**inputs, max_new_tokens=128)
+generated_ids_trimmed = [
+    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
+]
+output_text = processor.batch_decode(
+    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
+)[0]
+
+print(output_text)
+# {'action': 'CLICK', 'value': None, 'position': [0.49, 0.42]},
+# {'action': 'INPUT', 'value': 'weather for New York city', 'position': [0.49, 0.42]},
+# {'action': 'ENTER', 'value': None, 'position': None}
+```
+
+![download](https://github.com/user-attachments/assets/624097ea-06f2-4c8f-83f6-b6b9ee439c0c)