docs/tutorials/web_agent_tutorial_full.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Building a Browser Use Agent From Scratch and with Magentic-UI\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "You might have seen cool video demos online of AI agents taking control of a computer or a browser to perform tasks. This is a new category of agents referred to as Computer-Use-Agents (CUA) or Browser-Use-Agents (BUA). Examples of such CUA/BUA agents include [OpenAI's Operator](https://openai.com/index/introducing-operator/), [Claude Computer Use Model](https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool), [AutoGen's MultiModalWebSurfer](https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.agents.web_surfer.html), [Adept AI](https://www.adept.ai/blog/act-1), [Google's Project Mariner](https://deepmind.google/models/project-mariner/) and [Browser-Use](https://github.com/browser-use/browser-use/tree/main) among many others.\n",
    "\n",
    "\n",
    "## What is a Computer Use Agent?\n",
    "\n",
    "**Definition**: A computer or browser use agent is an agent that given a task, e.g., \"order a shawarma sandwich from BestShawarma for pickup now\", can programmatically control a computer or browser to autonomously complete the task. By \"control a browser\" we mean interacting with the browser in a similar way to how a human might control the browser: clicking on buttons, typing in fields, scrolling and so on. Note that a tool-use language model agent could complete this food ordering task if it had access to the restaurant API for instance, this would not make it a CUA agent as it is not _interacting_ with the browser to complete the task.\n",
    "\n",
    "To make this distinction more clear, here is another example task.\n",
    "Suppose we wanted to find the list of available Airbnbs in Miami from 6/18 to 6/20 for 2 guests.\n",
    "\n",
    "![airbnb_sc.png](airbnb_sc.png)\n",
    "\n",
    "How would a browser use agent solve this task:\n",
    "\n",
    "- **Step 1:** Visit airbnb.com\n",
    "- **Step 2:** Type \"Miami\" in the \"Where\" input box\n",
    "- **Step 3:** Select \"6/18\" in the \"Check in\" date box\n",
    "- **Step 4:** Select \"6/20\" in the \"Check out\" date box\n",
    "- **Step 5:** Click on the \"Who\" button\n",
    "- **Step 6:** Click \"+\" twice to add two guests\n",
    "- **Step 7:** Click \"Search\" button\n",
    "- **Step 8:** Summarize and extract listings from the webpage\n",
    "\n",
    "On the other hand, suppose we had an API for Airbnb that looks like: `find_listings(location, check_in, check_out, guests)`\n",
    "\n",
    "Then a tool-call agent would first need to generate a tool call: `find_listings(\"Miami\", 6/18, 6/20, 2)` and read out the result of the tool call.\n",
    "\n",
    "Clearly if we had an API for every website and everything on the computer, then it would be much simpler to perform this task. _But that is not the case currently_, many interfaces on the web cannot be accessed by an API and so the only way is through interacting with the website directly. While future interfaces might become more directly accessible to agents via APIs and MCP servers, for now we need to perform direct manipulation with the websites.\n",
    "\n",
    "## What Does This Tutorial Cover?\n",
    "\n",
    "In this tutorial, we will cover how to build a basic browser-use agent. The goal of this tutorial is to demystify such agents and show how we can build a simple version of them. The only thing we need is access to a large language model (LLM) that can perform tool calling or structured JSON outputs (GPT-4o, Qwen2.5-VL, Llama 3.1, ...). The LLM does not need to be vision capable, but a model capable of taking image input would improve performance significantly. The LLM also does not need to be trained previously for browser-use, out of the box LLMs can be turned into semi-capable browser-use agents following the recipe in this tutorial. At the end of the tutorial we will discuss further directions.\n",
    "\n",
    "We will cover three levels of building your browser use agent:\n",
    "\n",
    "- Level 1: From scratch using only the `playwright` python package.\n",
    "- Level 2: Using helpers from the `magentic-ui` package which simplifies building your agent.\n",
    "- Level 3: Using the WebSurfer Agent from the `magentic-ui` package directly.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial Prerequisites\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "You will need Python >3.10 to run this tutorial and the `magentic-ui` package. [Magentic-UI](https://github.com/microsoft/magentic-ui/tree/main) is a research prototype from Microsoft of a human-centered agentic interface. In this tutorial we will be using utilities and helpers from that package without using the Magentic-UI application itself.\n",
    "\n",
    "We recommend using a virtual environment to avoid conflicts with other packages.\n",
    "\n",
    "```bash\n",
    "python3 -m venv .venv\n",
    "source .venv/bin/activate\n",
    "pip install magentic-ui\n",
    "```\n",
    "\n",
    "Alternatively, if you use [`uv`](https://docs.astral.sh/uv/getting-started/installation/) for dependency management, you can install Magentic-UI with:\n",
    "\n",
    "```bash\n",
    "uv venv --python=3.12 .venv\n",
    ". .venv/bin/activate\n",
    "uv pip install magentic-ui\n",
    "```\n",
    "\n",
    "We also need to install the browsers that our agent will control with playwright:\n",
    "\n",
    "```bash\n",
    "playwright install --with-deps chromium\n",
    "```\n",
    "\n",
    "The other thing you need to set up is your LLM. The easiest way to follow this tutorial is to obtain an OpenAI API key and set it as an environment variable:\n",
    "\n",
    "```bash\n",
    "export OPENAI_API_KEY=<YOUR API KEY>\n",
    "```\n",
    "\n",
    "You can also use any open source model with [Ollama](https://ollama.com/) if you have a capable GPU at your disposal. We will be covering both using OpenAI and Ollama."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Level 1: Building a Browser Use Agent From Scratch\n",
    "\n",
    "For this level of building our browser use agent, we will only need the `playwright` and `openai` packages which are included in `magentic-ui` package.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Lauching a Browser\n",
    "\n",
    "The first step is to launch the browser that our agent will control. We will be using the [Playwright](https://github.com/microsoft/playwright-python) library that provides an API to control browsers.\n",
    "\n",
    "We can launch the browser in headless mode (we cannot see the actual browser on our machine) or non-headless where the browser will be launched locally.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from playwright.async_api import async_playwright\n",
    "\n",
    "headless = False  # Change to True to run the browser in headless mode\n",
    "\n",
    "# Launch and keep browser running\n",
    "p = await async_playwright().start()\n",
    "browser = await p.chromium.launch(headless=headless)\n",
    "# context is the browser window\n",
    "context = await browser.new_context()\n",
    "# page is the tab in the browser\n",
    "page = await context.new_page()\n",
    "print(\"Browser launched!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point you should see a browser launched locally, it will be pointing at a blank page:\n",
    "\n",
    "![blank_page.png](blank_page.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the playwright API to interact with this browser, for instance let us navigate to the bing homepage (give it a few seconds)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "await page.goto(\"https://www.bing.com\")\n",
    "print(\"Navigated to Bing homepage\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Represent the browser for the Agent using Set-Of-Marks Prompting.\n",
    "\n",
    "Our next challenge is how do we feed the browser as input to our agent so that it is able to perform actions on it.\n",
    "\n",
    "Using Playwright we can first take a screenshot of the browser as well as extract the text on the page.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image, display\n",
    "\n",
    "# Take a screenshot and store it in memory\n",
    "screenshot_bytes = await page.screenshot()\n",
    "\n",
    "# Display the screenshot\n",
    "display(Image(screenshot_bytes))\n",
    "\n",
    "# Get all the text on the page and print first 10 lines\n",
    "text = await page.evaluate(\"() => document.body.innerText\")\n",
    "print(\"\\nFirst 10 lines of text content:\")\n",
    "print(\"\\n\".join(text.split(\"\\n\")[:10]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now how do we get our agent to type in the search box and press search?\n",
    "\n",
    "The key is to extract all **interactive elements** in the page using Plawright. By interactive elements we mean elements on the page we can interact with including buttons, text boxes, dropdown menus among others. Each interactive element will have an ID that we can track on the page and if it is a visible element it will have the coordinates of bounding box of the element. We will also only look at the interactive elements that are currently visibile in the current viewport, some elements might be out of view and we'd need to scroll down to view them. For simplicity, we will ignore these elements and give our agent the ability to scroll down to view them later on.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dataclasses import dataclass\n",
    "from playwright.async_api import Page\n",
    "\n",
    "\n",
    "# A class to represent an interactive element on the page\n",
    "@dataclass\n",
    "class Element:\n",
    "    id: int  # The id of the element\n",
    "    aria_label: (\n",
    "        str  # The aria-label attribute is used to provide a label for an element\n",
    "    )\n",
    "    type: str  # The type of the element\n",
    "    bbox: dict  # The bounding box of the element\n",
    "    text: str  # The text content of the element\n",
    "\n",
    "\n",
    "# We will now go over the page and extract all interactive elements\n",
    "# We will also add a data attribute to the element with the element ID for later reference\n",
    "async def get_interactive_elements(page: Page) -> list[Element]:\n",
    "    elements: list[Element] = []\n",
    "    # Viewport size is a dict with keys 'width' and 'height'\n",
    "    viewport_size = page.viewport_size\n",
    "    print(f\"Viewport size: {viewport_size}\")\n",
    "\n",
    "    # For simplicity, we will only look at buttons, textboxes, and links. We can add more roles later on.\n",
    "    interactive_roles = [\"button\", \"textbox\", \"link\"]\n",
    "    i = 0\n",
    "    for role in interactive_roles:\n",
    "        print(f\"Getting {role} elements...\")\n",
    "        # We will use the Playwright API to get all elements with the given role\n",
    "        elements_with_role = await page.get_by_role(role).all()\n",
    "        for element in elements_with_role:\n",
    "            # Check if element is visible and in current viewport\n",
    "            bbox = await element.bounding_box()\n",
    "            if bbox:  # Element is visible if it has a bounding box\n",
    "                # Check if element is in current viewport (not scrolled out of view)\n",
    "                if 0 <= bbox[\"y\"] <= viewport_size[\"height\"]:\n",
    "                    # Set a data attribute with the element ID for later reference\n",
    "                    await element.evaluate(f\"el => el.setAttribute('data-element-id', '{i}')\")\n",
    "                    elements.append(\n",
    "                        Element(\n",
    "                            id=i,\n",
    "                            aria_label=await element.get_attribute(\"aria-label\")\n",
    "                            or await element.get_attribute(\"aria-role\")\n",
    "                            or \"\",\n",
    "                            type=role,\n",
    "                            bbox=bbox,\n",
    "                            text=await element.text_content() or \"\",\n",
    "                        )\n",
    "                    )\n",
    "                    i += 1\n",
    "    print(f\"Found {len(elements)} visible interactive elements in current viewport:\")\n",
    "    return elements\n",
    "\n",
    "\n",
    "elements = await get_interactive_elements(page)\n",
    "formatted_list_of_elements = \"\\n\".join(\n",
    "    [f\"Element {i}: {element}\" for i, element in enumerate(elements)]\n",
    ")\n",
    "print(formatted_list_of_elements)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first question, is how do we identify the search box element on the Bing page give these elements?\n",
    "We can try to read to figure this out by reading the list of elements, we can see that it is likely to be Element 19:\n",
    "\n",
    "Element(id=19, aria_label='0 characters out of 2000', type='textbox', bbox={'x': 193, 'y': 158, 'width': 843, 'height': 22}, text='')\n",
    "\n",
    "As this is the only texbox or searchbox element on the page.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find the search box\n",
    "search_box_id = None\n",
    "for element in elements:\n",
    "    if element.type == \"textbox\":\n",
    "        search_box_id = element.id\n",
    "        break\n",
    "print(f\"Search box id: {search_box_id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, we also have access to the page screenshot and the coordinates of each element. A neat idea would be to superimpose the bounding boxes on top of the screenshot to better understand what each element is. This technique is called Set-of-Mark Prompting (SoM) coined by Yang, Jianwei, et al. [1] to improve visual grounding.\n",
    "\n",
    "[1]: Yang, Jianwei, et al. \"Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.\" arXiv preprint arXiv:2310.11441 (2023). https://arxiv.org/pdf/2310.11441\n",
    "\n",
    "We're gonna now implement a simplified version of SoM prompting:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image, ImageDraw\n",
    "import io\n",
    "\n",
    "\n",
    "def get_som_screenshot(screenshot_bytes: bytes, elements: list[Element]) -> Image.Image:\n",
    "    screenshot = Image.open(io.BytesIO(screenshot_bytes))\n",
    "\n",
    "    # Create a drawing object\n",
    "    draw = ImageDraw.Draw(screenshot)\n",
    "\n",
    "    # Draw bounding boxes and element IDs for each element\n",
    "    for element in elements:\n",
    "        bbox = element.bbox\n",
    "        x = bbox[\"x\"]\n",
    "        y = bbox[\"y\"]\n",
    "        width = bbox[\"width\"]\n",
    "        height = bbox[\"height\"]\n",
    "\n",
    "        # Draw rectangle\n",
    "        draw.rectangle([(x, y), (x + width, y + height)], outline=\"red\", width=2)\n",
    "\n",
    "        # Draw element ID\n",
    "        draw.text((x, y - 15), f\"{element.id}\", fill=\"red\")\n",
    "\n",
    "    # Display the annotated screenshot\n",
    "    display(screenshot)\n",
    "    som_screenshot = screenshot.copy()\n",
    "    return som_screenshot\n",
    "\n",
    "\n",
    "screenshot_bytes = await page.screenshot()\n",
    "som_screenshot = get_som_screenshot(screenshot_bytes, elements)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This confirms what we previously found that Element with id=19 is in fact the searchbox!\n",
    "\n",
    "Let us now wrap what we just did in a helper function to prepare the page to be used as input to our agent:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def prepare_page_for_agent(page: Page) -> tuple[str, str, Image.Image]:\n",
    "    \"\"\"\n",
    "    Prepare the page for the agent.\n",
    "    Returns:\n",
    "        tuple[str, str, Image.Image]: The page text, the formatted list of elements, and the screenshot with bounding boxes.\n",
    "    \"\"\"\n",
    "    page_text = await page.evaluate(\"() => document.body.innerText\")\n",
    "    elements = await get_interactive_elements(page)\n",
    "    screenshot_bytes = await page.screenshot()\n",
    "    som_screenshot = get_som_screenshot(screenshot_bytes, elements)\n",
    "\n",
    "    formatted_list_of_elements = \"\\n\".join(\n",
    "        [f\"Element {i}: {element}\" for i, element in enumerate(elements)]\n",
    "    )\n",
    "\n",
    "    return page_text, formatted_list_of_elements, som_screenshot\n",
    "\n",
    "\n",
    "# page_text, formatted_list_of_elements, screenshot = await prepare_page_for_agent(page)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Define Agent Action Space\n",
    "\n",
    "Now that we have established how to represent the browser state for our Agent, it's time to define our Agent architecture. This section will cover the action space and execution flow that enables our agent to perform tasks using the browser.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Action Space Definition\n",
    "\n",
    "Our web agent operates with a carefully designed set of actions that cover the fundamental browser interactions needed for most web automation tasks:\n",
    "\n",
    "- **`goto(url)`**: Navigate to a specific URL\n",
    "- **`click(id)`**: Click on an element identified by its ID\n",
    "- **`type(id, text)`**: Input text into a form field or text element by ID\n",
    "- **`scroll(direction)`**: Scroll the page vertically (up/down)\n",
    "- **`stop_action(final_answer)`**: Complete the task and return the final result\n",
    "\n",
    "_Note: This is a simplfied action set designed for our initial prototype. It can be extended with additional actions like hover, select, wait, etc., as needed._\n",
    "\n",
    "### Agent Architecture Flow\n",
    "\n",
    "The following diagram illustrates how our web agent processes user queries and executes actions:\n",
    "\n",
    "```mermaid\n",
    "flowchart TD\n",
    "    A[\"Input: User Query\"] --> B[\"Initialize Agent\"]\n",
    "    B --> C[\"Capture Current Page State\"]\n",
    "    C --> D[\"Analyze Page & Query\"]\n",
    "    D --> E{Action Decision}\n",
    "    E -->|goto| F[\"Navigate to URL\"]\n",
    "    E -->|click| G[\"Click Element by ID\"]\n",
    "    E -->|type| H[\"Type Text in Element\"]\n",
    "    E -->|scroll| I[\"Scroll Page\"]\n",
    "    E -->|stop_action| J[\"Return Final Answer\"]\n",
    "    F --> K[\"Execute Action\"]\n",
    "    G --> K\n",
    "    H --> K\n",
    "    I --> K\n",
    "    K --> C\n",
    "    J --> L[\"Output: Final Answer\"]\n",
    "\n",
    "    style A fill:#e1f5fe\n",
    "    style L fill:#e8f5e8\n",
    "    style E fill:#fff3e0\n",
    "    style J fill:#ffebee\n",
    "```\n",
    "\n",
    "### Execution Flow Details\n",
    "\n",
    "1. **Input Processing**: The agent receives a user query describing the desired task\n",
    "2. **State Capture**: Current browser page state is captured and processed\n",
    "3. **Action Selection**: Based on the analysis, one of the five actions is chosen\n",
    "4. **Execution**: The selected action is executed in the browser. We append the feedback of the action into the chat history.\n",
    "5. **Loop Continuation**: The process repeats until `stop_action` is triggered\n",
    "\n",
    "The agent continues this loop until it determines the task is complete, at which point it executes `stop_action` with the final answer.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our first step is to create the prompt template for the model to decide on the correct action. Instead of using tool calling to decide on the action, we will JSON outputs for simplicity.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "AGENT_PROMPT = \"\"\"\n",
    "You are a helpful assistant that can navigate a web page and perform actions on it.\n",
    "\n",
    "The task we are trying to complete is:\n",
    "{task}\n",
    "\n",
    "The current visible text on the page is:\n",
    "{page_text}\n",
    "\n",
    "The current visible elements on the page are:\n",
    "{formatted_list_of_elements}\n",
    "\n",
    "You will need to decide on the next action to take.\n",
    "\n",
    "The action space is:\n",
    "- goto(url): navigate to a URL\n",
    "- click(id): click a button given it's ID\n",
    "- type(id, text): type \"text\" into element \"id\"\n",
    "- scroll(direction): scroll the page in direction up or down.\n",
    "- stop_action(final_answer): declare that we have finished the task and prepare a final_answer to return to the user.\n",
    "\n",
    "Output a JSON object with the following fields:\n",
    "{{\n",
    "    \"action\": \"goto\" | \"click\" | \"type\" | \"scroll\" | \"stop_action\",\n",
    "    \"action_args\": {{\n",
    "        \"url\": \"https://www.google.com\",\n",
    "        \"id\": \"123\",\n",
    "        \"text\": \"Hello\",\n",
    "        \"direction\": \"up\"\n",
    "    }}\n",
    "}}\n",
    "\n",
    "Only output the JSON object, no other text or comments.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now try this prompt with our LLM:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "import json\n",
    "import base64\n",
    "from PIL import Image\n",
    "import os\n",
    "\n",
    "# Prepare the page for the agent\n",
    "page_text, formatted_list_of_elements, som_screenshot = await prepare_page_for_agent(\n",
    "    page\n",
    ")\n",
    "task = \"Search for Magentic-UI\"\n",
    "# Now make the API call\n",
    "client = OpenAI(\n",
    "    api_key=os.getenv(\"OPENAI_API_KEY\")\n",
    ")  # you can use any other LLM client here\n",
    "image_data_url = f\"data:image/png;base64,{base64.b64encode((lambda b: (som_screenshot.save(b, format='PNG'), b.getvalue())[1])(io.BytesIO())).decode()}\"\n",
    "\n",
    "\n",
    "def get_llm_response(\n",
    "    client: OpenAI,  # OpenAI client\n",
    "    task: str,  # Task to complete\n",
    "    page_text: str,  # Page text\n",
    "    formatted_list_of_elements: str,  # Formatted list of elements\n",
    "    image_data_url: str,  # Image data URL\n",
    "    message_history: list[dict] = [],  # Message history\n",
    "    model: str = \"gpt-4o\",  # Model to use\n",
    ") -> dict:\n",
    "    response = client.chat.completions.create(\n",
    "        model=model,\n",
    "        messages=[\n",
    "            *message_history,\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": [\n",
    "                    {\n",
    "                        \"type\": \"text\",\n",
    "                        \"text\": AGENT_PROMPT.format(\n",
    "                            task=task,\n",
    "                            page_text=page_text,\n",
    "                            formatted_list_of_elements=formatted_list_of_elements,\n",
    "                        ),\n",
    "                    },\n",
    "                    {\n",
    "                        \"type\": \"image_url\",\n",
    "                        \"image_url\": {\"url\": image_data_url},\n",
    "                    },\n",
    "                ],\n",
    "            },\n",
    "        ],\n",
    "    )\n",
    "\n",
    "    # Parse the response\n",
    "    try:\n",
    "        action_decision = json.loads(response.choices[0].message.content)\n",
    "        print(\"Model's decision:\", json.dumps(action_decision, indent=2))\n",
    "    except json.JSONDecodeError:\n",
    "        # it starts with ```json\n",
    "        response_content = response.choices[0].message.content\n",
    "        response_content = response_content.replace(\"```json\", \"\").replace(\"```\", \"\")\n",
    "        action_decision = json.loads(response_content)\n",
    "        print(\"Model's decision:\", json.dumps(action_decision, indent=2))\n",
    "    except Exception as e:\n",
    "        raise e\n",
    "    return action_decision\n",
    "\n",
    "\n",
    "action_decision = get_llm_response(\n",
    "    client, task, page_text, formatted_list_of_elements, image_data_url\n",
    ")\n",
    "print(action_decision)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the model made the right decision given the task of \"Search for Magentic-UI\", the action is to type in the search box for \"Magentic-UI\"\n",
    "\n",
    "The last remaining piece before we put it all together is to now execute the action using Playwright."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Executing the actions with Playwright\n",
    "\n",
    "For each of the actions we have previously defined, we will now write code using Playwright to execute them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is mostly basic Playwright code, but we will use it to execute the actions.\n",
    "async def execute_action(action: str, action_args: dict, page: Page) -> str:\n",
    "    \"\"\"\n",
    "    Execute an action on the page.\n",
    "    \"\"\"\n",
    "    if action == \"goto\":\n",
    "        await page.goto(action_args[\"url\"])\n",
    "        return f\"I navigated to {action_args['url']}\"\n",
    "    elif action == \"click\":\n",
    "        # Get the element using the data attribute\n",
    "        await page.wait_for_selector(f\"[data-element-id='{action_args['id']}']\")\n",
    "        element = page.locator(f\"[data-element-id='{action_args['id']}']\")\n",
    "        if element:\n",
    "            await element.click()\n",
    "        else:\n",
    "            raise ValueError(f\"Element with ID {action_args['id']} not found\")\n",
    "        return f\"I clicked on {action_args['id']}\"\n",
    "    elif action == \"type\":\n",
    "        await page.wait_for_selector(f\"[data-element-id='{action_args['id']}']\")\n",
    "        element = page.locator(f\"[data-element-id='{action_args['id']}']\")\n",
    "        if element:\n",
    "            await element.fill(action_args[\"text\"])\n",
    "        else:\n",
    "            raise ValueError(f\"Element with ID {action_args['id']} not found\")\n",
    "        return f\"I typed {action_args['text']} into {action_args['id']}\"\n",
    "    elif action == \"scroll\":\n",
    "        await page.scroll(action_args[\"direction\"])\n",
    "        return f\"I scrolled {action_args['direction']}\"\n",
    "    elif action == \"stop_action\":\n",
    "        return action_args[\"final_answer\"]\n",
    "    else:\n",
    "        raise ValueError(f\"Invalid action: {action}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "await execute_action(action_decision[\"action\"], action_decision[\"action_args\"], page)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take a screenshot of the page\n",
    "screenshot = await page.screenshot()\n",
    "display(Image.open(io.BytesIO(screenshot)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Success! We can see that our agent was properly able to type \"Magentic-UI\" into the searchbox!\n",
    "\n",
    "The final step is to put it all together into our Agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "await browser.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Putting it all together into our Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "from playwright.async_api import Page\n",
    "from playwright.async_api import async_playwright\n",
    "from PIL import Image, ImageDraw\n",
    "import io\n",
    "import base64\n",
    "import json\n",
    "from dataclasses import dataclass\n",
    "from IPython.display import display\n",
    "\n",
    "@dataclass\n",
    "class Element:\n",
    "    id: int  # The id of the element\n",
    "    aria_label: (\n",
    "        str  # The aria-label attribute is used to provide a label for an element\n",
    "    )\n",
    "    type: str  # The type of the element\n",
    "    bbox: dict  # The bounding box of the element\n",
    "    text: str  # The text content of the element\n",
    "\n",
    "\n",
    "AGENT_PROMPT = \"\"\"\n",
    "You are a helpful assistant that can navigate a web page and perform actions on it.\n",
    "\n",
    "The task we are trying to complete is:\n",
    "{task}\n",
    "\n",
    "The current visible text on the page is:\n",
    "{page_text}\n",
    "\n",
    "The current visible elements on the page are:\n",
    "{formatted_list_of_elements}\n",
    "\n",
    "You will need to decide on the next action to take.\n",
    "\n",
    "The action space is:\n",
    "- goto(url): navigate to a URL\n",
    "- click(id): click a button given it's ID\n",
    "- type(id, text): type \"text\" into element \"id\"\n",
    "- scroll(direction): scroll the page in direction up or down.\n",
    "- stop_action(final_answer): declare that we have finished the task and prepare a final_answer to return to the user.\n",
    "\n",
    "Output a JSON object with the following fields:\n",
    "{{\n",
    "    \"action\": \"goto\" | \"click\" | \"type\" | \"scroll\" | \"stop_action\",\n",
    "    \"action_args\": {{\n",
    "        \"url\": \"https://www.google.com\",\n",
    "        \"id\": \"123\",\n",
    "        \"text\": \"Hello\",\n",
    "        \"direction\": \"up\"\n",
    "    }}\n",
    "}}\n",
    "\n",
    "Only output the JSON object, no other text or comments.\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "class BrowserUseAgent:\n",
    "    def __init__(\n",
    "        self,\n",
    "        client: OpenAI,\n",
    "        model: str = \"gpt-4o\",\n",
    "        headless: bool = False,\n",
    "        run_in_jupyter: bool = True,\n",
    "    ):\n",
    "        self.client = client\n",
    "        self.model = model\n",
    "        self.headless = headless\n",
    "        self.message_history: list[dict] = []\n",
    "        self.page: Page = None\n",
    "        self.run_in_jupyter = run_in_jupyter\n",
    "\n",
    "    async def _launch_browser(self) -> None:\n",
    "        p = await async_playwright().start()\n",
    "        self.browser = await p.chromium.launch(headless=self.headless)\n",
    "        # context is the browser window\n",
    "        self.context = await self.browser.new_context()\n",
    "        # page is the tab in the browser\n",
    "        self.page = await self.context.new_page()\n",
    "\n",
    "    async def execute_task(self, task: str) -> str:\n",
    "        \"\"\"\n",
    "        This is NEW! This is the main function that will be called to execute the task and implement our agent loop.\n",
    "        \"\"\"\n",
    "        # Step 1: Launch the browser if it's not already launched\n",
    "        if self.page is None:\n",
    "            await self._launch_browser()\n",
    "        # Our stop condition is when the LLM decides to output stop_action\n",
    "        should_stop = False\n",
    "        final_answer = None\n",
    "        i = 0\n",
    "        while not should_stop:\n",
    "            # Step 2: Prepare the page for the agent\n",
    "            (\n",
    "                page_text,\n",
    "                formatted_list_of_elements,\n",
    "                som_screenshot,\n",
    "            ) = await self._prepare_page_for_agent(self.page)\n",
    "            # Step 3: Get the LLM response\n",
    "            image_data_url = f\"data:image/png;base64,{base64.b64encode((lambda b: (som_screenshot.save(b, format='PNG'), b.getvalue())[1])(io.BytesIO())).decode()}\"\n",
    "            action_decision = self._get_llm_response(\n",
    "                self.client,\n",
    "                task,\n",
    "                page_text,\n",
    "                formatted_list_of_elements,\n",
    "                image_data_url,\n",
    "                self.message_history,\n",
    "                self.model,\n",
    "            )\n",
    "            print(f\"Action decision {i}: {action_decision}\")\n",
    "            # Add the action decision to the message history\n",
    "            self.message_history.append(\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": [{\"type\": \"text\", \"text\": json.dumps(action_decision)}],\n",
    "                }\n",
    "            )\n",
    "            # Step 4: Execute the action with some error handling\n",
    "            try:\n",
    "                action_feedback = await self._execute_action(\n",
    "                    action_decision[\"action\"], action_decision[\"action_args\"], self.page\n",
    "                )\n",
    "            except Exception as e:\n",
    "                print(f\"Error executing action {i}: {e}\")\n",
    "                action_feedback = f\"Error executing action {i}: {e}\"\n",
    "            print(f\"Action feedback {i}: {action_feedback}\")\n",
    "            # Sleep for 3 seconds to let the page load\n",
    "            await self.page.wait_for_timeout(3000)\n",
    "            # Update the message history with feedback on the action and the new page screenshot\n",
    "            new_page_screenshot = await self.page.screenshot()\n",
    "            self.message_history.append(\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": [\n",
    "                        {\"type\": \"text\", \"text\": action_feedback},\n",
    "                        {\n",
    "                            \"type\": \"image_url\",\n",
    "                            \"image_url\": {\n",
    "                                \"url\": f\"data:image/png;base64,{base64.b64encode(new_page_screenshot).decode()}\"\n",
    "                            },\n",
    "                        },\n",
    "                    ],\n",
    "                }\n",
    "            )\n",
    "            if self.run_in_jupyter:\n",
    "                display(Image.open(io.BytesIO(new_page_screenshot)))\n",
    "            # Check if the task is complete\n",
    "            should_stop = action_decision[\"action\"] == \"stop_action\"\n",
    "            if should_stop:\n",
    "                final_answer = action_decision[\"action_args\"][\"final_answer\"]\n",
    "            i += 1\n",
    "        return final_answer\n",
    "\n",
    "    async def _execute_action(self, action: str, action_args: dict, page: Page) -> str:\n",
    "        \"\"\"\n",
    "        Execute an action on the page.\n",
    "        \"\"\"\n",
    "        if action == \"goto\":\n",
    "            await page.goto(action_args[\"url\"])\n",
    "            return f\"I navigated to {action_args['url']}\"\n",
    "        elif action == \"click\":\n",
    "            # Get the element using the data attribute\n",
    "            await page.wait_for_selector(f\"[data-element-id='{action_args['id']}']\")\n",
    "            element = page.locator(f\"[data-element-id='{action_args['id']}']\")\n",
    "            if element:\n",
    "                await element.click()\n",
    "            else:\n",
    "                raise ValueError(f\"Element with ID {action_args['id']} not found\")\n",
    "            return f\"I clicked on {action_args['id']}\"\n",
    "        elif action == \"type\":\n",
    "            await page.wait_for_selector(f\"[data-element-id='{action_args['id']}']\")\n",
    "            element = page.locator(f\"[data-element-id='{action_args['id']}']\")\n",
    "            if element:\n",
    "                await element.fill(action_args[\"text\"])\n",
    "                # Press enter\n",
    "                await element.press(\"Enter\")\n",
    "            else:\n",
    "                raise ValueError(f\"Element with ID {action_args['id']} not found\")\n",
    "            return f\"I typed {action_args['text']} into {action_args['id']}\"\n",
    "        elif action == \"scroll\":\n",
    "            await page.scroll(action_args[\"direction\"])\n",
    "            return f\"I scrolled {action_args['direction']}\"\n",
    "        elif action == \"stop_action\":\n",
    "            return action_args[\"final_answer\"]\n",
    "        else:\n",
    "            raise ValueError(f\"Invalid action: {action}\")\n",
    "\n",
    "    def _get_llm_response(\n",
    "        self,\n",
    "        client: OpenAI,  # OpenAI client\n",
    "        task: str,  # Task to complete\n",
    "        page_text: str,  # Page text\n",
    "        formatted_list_of_elements: str,  # Formatted list of elements\n",
    "        image_data_url: str,  # Image data URL\n",
    "        message_history: list[dict] = [],  # Message history\n",
    "        model: str = \"gpt-4o\",  # Model to use\n",
    "    ) -> dict:\n",
    "        response = client.chat.completions.create(\n",
    "            model=model,\n",
    "            messages=[\n",
    "                *message_history,\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": [\n",
    "                        {\n",
    "                            \"type\": \"text\",\n",
    "                            \"text\": AGENT_PROMPT.format(\n",
    "                                task=task,\n",
    "                                page_text=page_text,\n",
    "                                formatted_list_of_elements=formatted_list_of_elements,\n",
    "                            ),\n",
    "                        },\n",
    "                        {\n",
    "                            \"type\": \"image_url\",\n",
    "                            \"image_url\": {\"url\": image_data_url},\n",
    "                        },\n",
    "                    ],\n",
    "                },\n",
    "            ],\n",
    "        )\n",
    "\n",
    "        # Parse the response\n",
    "        try:\n",
    "            action_decision = json.loads(response.choices[0].message.content)\n",
    "        except json.JSONDecodeError:\n",
    "            # it starts with ```json\n",
    "            response_content = response.choices[0].message.content\n",
    "            response_content = response_content.replace(\"```json\", \"\").replace(\n",
    "                \"```\", \"\"\n",
    "            )\n",
    "            action_decision = json.loads(response_content)\n",
    "        except Exception as e:\n",
    "            raise e\n",
    "        return action_decision\n",
    "\n",
    "    async def _prepare_page_for_agent(self, page: Page) -> tuple[str, str, Image.Image]:\n",
    "        \"\"\"\n",
    "        Prepare the page for the agent.\n",
    "        Returns:\n",
    "            tuple[str, str, Image.Image]: The page text, the formatted list of elements, and the screenshot with bounding boxes.\n",
    "        \"\"\"\n",
    "        page_text = await page.evaluate(\"() => document.body.innerText\")\n",
    "        elements = await self._get_interactive_elements(page)\n",
    "        screenshot_bytes = await page.screenshot()\n",
    "        som_screenshot = self._get_som_screenshot(screenshot_bytes, elements)\n",
    "\n",
    "        formatted_list_of_elements = \"\\n\".join(\n",
    "            [f\"Element {i}: {element}\" for i, element in enumerate(elements)]\n",
    "        )\n",
    "\n",
    "        return page_text, formatted_list_of_elements, som_screenshot\n",
    "\n",
    "    def _get_som_screenshot(\n",
    "        self, screenshot_bytes: bytes, elements: list[Element]\n",
    "    ) -> Image.Image:\n",
    "        screenshot = Image.open(io.BytesIO(screenshot_bytes))\n",
    "\n",
    "        # Create a drawing object\n",
    "        draw = ImageDraw.Draw(screenshot)\n",
    "\n",
    "        # Draw bounding boxes and element IDs for each element\n",
    "        for element in elements:\n",
    "            bbox = element.bbox\n",
    "            x = bbox[\"x\"]\n",
    "            y = bbox[\"y\"]\n",
    "            width = bbox[\"width\"]\n",
    "            height = bbox[\"height\"]\n",
    "\n",
    "            # Draw rectangle\n",
    "            draw.rectangle([(x, y), (x + width, y + height)], outline=\"red\", width=2)\n",
    "\n",
    "            # Draw element ID\n",
    "            draw.text((x, y - 15), f\"{element.id}\", fill=\"red\")\n",
    "\n",
    "        som_screenshot = screenshot.copy()\n",
    "        return som_screenshot\n",
    "\n",
    "    async def _get_interactive_elements(self, page: Page) -> list[Element]:\n",
    "        elements: list[Element] = []\n",
    "        # Viewport size is a dict with keys 'width' and 'height'\n",
    "        viewport_size = page.viewport_size\n",
    "\n",
    "        # For simplicity, we will only look at buttons, textboxes, and links. We can add more roles later on.\n",
    "        interactive_roles = [\"button\", \"textbox\", \"link\"]\n",
    "        i = 0\n",
    "        for role in interactive_roles:\n",
    "            # We will use the Playwright API to get all elements with the given role\n",
    "            elements_with_role = await page.get_by_role(role).all()\n",
    "            for element in elements_with_role:\n",
    "                # Check if element is visible and in current viewport\n",
    "                bbox = await element.bounding_box()\n",
    "                if bbox:  # Element is visible if it has a bounding box\n",
    "                    # Check if element is in current viewport (not scrolled out of view)\n",
    "                    if 0 <= bbox[\"y\"] <= viewport_size[\"height\"]:\n",
    "                        # Set a data attribute with the element ID for later reference\n",
    "                        await element.evaluate(\n",
    "                            f\"el => el.setAttribute('data-element-id', '{i}')\"\n",
    "                        )\n",
    "                        elements.append(\n",
    "                            Element(\n",
    "                                id=i,\n",
    "                                aria_label=await element.get_attribute(\"aria-label\")\n",
    "                                or await element.get_attribute(\"aria-role\")\n",
    "                                or \"\",\n",
    "                                type=role,\n",
    "                                bbox=bbox,\n",
    "                                text=await element.text_content() or \"\",\n",
    "                            )\n",
    "                        )\n",
    "                        i += 1\n",
    "        return elements\n",
    "\n",
    "    async def close(self) -> None:\n",
    "        if self.page is not None:\n",
    "            await self.page.close()\n",
    "        if self.context is not None:\n",
    "            await self.context.close()\n",
    "        if self.browser is not None:\n",
    "            await self.browser.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run the Agent on a sample task!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "import os\n",
    "openai_client = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n",
    "agent = BrowserUseAgent(openai_client)\n",
    "try:\n",
    "    final_answer = await agent.execute_task(\"find the open issues assigned to husseinmozannar on the microsoft/magentic-ui repo on github\")\n",
    "    print(final_answer)\n",
    "finally:\n",
    "    await agent.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sucess! Our agent was able to navigate to GitHub and filter the issues assigned to me. It ran into some issues but was able to debug and get to the right answer.\n",
    "\n",
    "To conclude, in this short tutorial, we showed how to build a browser use agent from scratch.\n",
    "\n",
    "The main ingredients were: set-of-marks prompting, playwright for browser automation and tool calling or structured JSON output ability of current LLMs. With these three ingredients we can build a semi-capable agent to navigate the web!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Level 2: Building a Browser Use Agent Using Magentic-UI\n",
    "\n",
    "While it was fun building the browser use agent from scratch, it was not easy. We had to figure out how to launch the browser, fiddle around with playwright to extract interactive elements, figure out how to execute actions on the page and so on.\n",
    "\n",
    "The `magentic-ui` library as we will see has many utilities that will make your life much easier when building browser use agents. We will now do the same steps as before but by using the helpers from the `magentic-ui` library.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Launching a Browser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Magentic-UI provides three different Playwright browser implementations, each designed for specific use cases:\n",
    "\n",
    "1. Local Playwright Browser (`LocalPlaywrightBrowser`)\n",
    "- **Purpose**: Runs Playwright directly on the local machine without Docker\n",
    "- **Use Case**: Development and testing environments where Docker isn't needed\n",
    "- **Features**: Lightweight, direct browser control, supports both headless and headed modes\n",
    "\n",
    "2. Headless Docker Playwright Browser (`HeadlessDockerPlaywrightBrowser`)  \n",
    "- **Purpose**: Runs a headless Playwright browser inside a Docker container\n",
    "- **Use Case**: Production environments, CI/CD pipelines, server-side automation\n",
    "- **Features**: Isolated execution, reproducible environment, no GUI overhead and more secure.\n",
    "- **Docker Image**: Uses Microsoft's official Playwright Docker image (`mcr.microsoft.com/playwright:v1.51.1-noble`)\n",
    "\n",
    "3. VNC Docker Playwright Browser (`VncDockerPlaywrightBrowser`)\n",
    "- **Purpose**: Runs Playwright in Docker with VNC support for visual interaction, you can interact with the browser on localhost.\n",
    "- **Use Case**: Debugging, development, and scenarios requiring visual browser inspection\n",
    "- **Features**: Programmatic control + visual access via noVNC web interface\n",
    "- **Docker Image**: Uses custom `magentic-ui-vnc-browser` image with VNC server. You need to run `magentic-ui --rebuild-docker` command to build it.\n",
    "\n",
    "How to Launch Each Browser:\n",
    "\n",
    "```python\n",
    "from pathlib import Path\n",
    "from magentic_ui.tools.playwright import HeadlessDockerPlaywrightBrowser, VncDockerPlaywrightBrowser, LocalPlaywrightBrowser\n",
    "\n",
    "# Direct instantiation examples\n",
    "async def launch_browsers():\n",
    "    # Headless Docker Browser\n",
    "    headless_browser = HeadlessDockerPlaywrightBrowser(\n",
    "        playwright_port=37367,\n",
    "        inside_docker=False\n",
    "    )\n",
    "    \n",
    "    # VNC Docker Browser  \n",
    "    vnc_browser = VncDockerPlaywrightBrowser(\n",
    "        bind_dir=Path(\"./workspace\"),\n",
    "        playwright_port=37367,\n",
    "        novnc_port=6080,\n",
    "        inside_docker=False\n",
    "    )\n",
    "    \n",
    "    # Local Browser\n",
    "    local_browser = LocalPlaywrightBrowser(headless=True)\n",
    "    \n",
    "```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity we will stick with the local playwright browser that we launched in Level 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from magentic_ui.tools.playwright import LocalPlaywrightBrowser\n",
    "browser = LocalPlaywrightBrowser(headless=False)\n",
    "# Start the browser\n",
    "await browser._start()\n",
    "# Get the browser context and start a new page\n",
    "context = browser.browser_context\n",
    "page = await context.new_page()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should now see a browser open to the blank page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Represent the browser for the Agent using Set-Of-Marks Prompting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the interactive elements on the page, we have done a lot of work for you in Magentic-UI to capture every posible interactive element type on the page including elements in the shadow-DOM [(see this javascript file if interested for more info)](https://github.com/microsoft/magentic-ui/blob/main/src/magentic_ui/tools/playwright/page_script.js).\n",
    "\n",
    "These utilities are wrapped in a helper class called the [`PlaywrightController`](https://github.com/microsoft/magentic-ui/blob/main/src/magentic_ui/tools/playwright/playwright_controller.py)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from magentic_ui.tools.playwright import PlaywrightController\n",
    "browser_controller = PlaywrightController(viewport_width=1280, viewport_height=720)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The PlaywrightController has a lot of convenience methods that have been debugged extensively so that we can perform actions on the browser more reliably and securily.\n",
    "\n",
    "There are methods to get the interactive elements, get the screenshot, click, type, scroll, manage tabs, hover, describe pages in markdown and much more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For now, let's navigate to Bing using our `browser_controller`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = await browser_controller.visit_page(page, \"https://www.bing.com\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visit_page method only returns when the page is fully loaded."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us get the set of interactive elements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactive_elements = await browser_controller.get_interactive_rects(page)\n",
    "# print the first 20 interactive elements\n",
    "i = 0\n",
    "for element in interactive_elements:\n",
    "    print(f\"Element {i}: id={element}, data={interactive_elements[element]}\")\n",
    "    i += 1\n",
    "    if i > 20:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that this ran much faster than using the Playwright script in Level 1 tutorial because here we are using javascript to extract the elements instead of going through the playwright API.\n",
    "\n",
    "Our searchbox is now Element id 22 and has the following data:\n",
    "\n",
    "\n",
    "    Element 12: id=22, data={'tag_name': 'textarea', 'role': 'textbox', 'aria_name': '0 characters out of 2000', 'v_scrollable': False, 'rects': [{'x': 193, 'y': 158, 'width': 843, 'height': 22, 'top': 158, 'right': 1036, 'bottom': 180, 'left': 193}]}\n",
    "\n",
    "To type in the searchbox we can use the fill_id method of the PlaywrightController:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "await browser_controller.fill_id(page, \"22\", \"Magentic-UI\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check if we are the right page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "import io\n",
    "from IPython.display import display\n",
    "\n",
    "screenshot = await browser_controller.get_screenshot(page)\n",
    "image = Image.open(io.BytesIO(screenshot))\n",
    "display(image)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also easily extract the search results using the get_page_markdown method that uses the [`markitdown`](https://github.com/microsoft/markitdown) package from our team at Microsoft Research."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "page_text = await browser_controller.get_page_markdown(page)\n",
    "print(page_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The final thing we need is to get the set-of-marks image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from magentic_ui.agents.web_surfer._set_of_mark import add_set_of_mark\n",
    "\n",
    "\n",
    "interactive_elements = await browser_controller.get_interactive_rects(page)\n",
    "screenshot = await browser_controller.get_screenshot(page)\n",
    "som_screenshot, visible_elements, elements_above, elements_below, _ = add_set_of_mark(\n",
    "    screenshot, interactive_elements, use_sequential_ids=True\n",
    ")\n",
    "\n",
    "display(som_screenshot)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The add_set_of_mark method returns the SoM screenshot in addition to elements visible on the viewport, elements above the viewport and elements below the viewport.\n",
    "\n",
    "We can see how much the `magentic-ui` makes our life easier with these tools, we are now ready to re-implement the agent from Level 1!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Putting it all together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the tools from the `magentic-ui` library now we can more easily implement our BrowserUseAgent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "from playwright.async_api import Page\n",
    "from playwright.async_api import async_playwright\n",
    "from PIL import Image, ImageDraw\n",
    "import io\n",
    "import base64\n",
    "import json\n",
    "from dataclasses import dataclass\n",
    "from IPython.display import display\n",
    "from magentic_ui.tools.playwright import LocalPlaywrightBrowser\n",
    "from magentic_ui.tools.playwright import PlaywrightController\n",
    "from magentic_ui.agents.web_surfer._set_of_mark import add_set_of_mark\n",
    "\n",
    "\n",
    "AGENT_PROMPT = \"\"\"\n",
    "You are a helpful assistant that can navigate a web page and perform actions on it.\n",
    "\n",
    "The task we are trying to complete is:\n",
    "{task}\n",
    "\n",
    "The current visible text on the page is:\n",
    "{page_text}\n",
    "\n",
    "The current visible elements on the page are:\n",
    "{formatted_list_of_elements}\n",
    "\n",
    "You will need to decide on the next action to take.\n",
    "\n",
    "The action space is:\n",
    "- goto(url): navigate to a URL\n",
    "- click(id): click a button given it's ID\n",
    "- type(id, text): type \"text\" into element \"id\"\n",
    "- scroll(direction): scroll the page in direction up or down.\n",
    "- stop_action(final_answer): declare that we have finished the task and prepare a final_answer to return to the user.\n",
    "\n",
    "Output a JSON object with the following fields:\n",
    "{{\n",
    "    \"action\": \"goto\" | \"click\" | \"type\" | \"scroll\" | \"stop_action\",\n",
    "    \"action_args\": {{\n",
    "        \"url\": \"https://www.google.com\",\n",
    "        \"id\": \"123\",\n",
    "        \"text\": \"Hello\",\n",
    "        \"direction\": \"up\"\n",
    "    }}\n",
    "}}\n",
    "\n",
    "Only output the JSON object, no other text or comments.\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "class BrowserUseAgent:\n",
    "    def __init__(\n",
    "        self,\n",
    "        client: OpenAI,\n",
    "        model: str = \"gpt-4o\",\n",
    "        headless: bool = False,\n",
    "        run_in_jupyter: bool = True,\n",
    "    ):\n",
    "        self.client = client\n",
    "        self.model = model\n",
    "        self.headless = headless\n",
    "        self.message_history: list[dict] = []\n",
    "        self.page: Page = None\n",
    "        self.run_in_jupyter = run_in_jupyter\n",
    "        self.browser_controller = PlaywrightController(\n",
    "            viewport_width=1280, viewport_height=720\n",
    "        )\n",
    "\n",
    "    async def _launch_browser(self) -> None:\n",
    "        self.browser = LocalPlaywrightBrowser(headless=False)\n",
    "        # Start the browser\n",
    "        await self.browser._start()\n",
    "        # Get the browser context and start a new page\n",
    "        self.context = self.browser.browser_context\n",
    "        self.page = await self.context.new_page()\n",
    "\n",
    "    async def execute_task(self, task: str) -> str:\n",
    "        \"\"\"\n",
    "        This is NEW! This is the main function that will be called to execute the task and implement our agent loop.\n",
    "        \"\"\"\n",
    "        # Step 1: Launch the browser if it's not already launched\n",
    "        if self.page is None:\n",
    "            await self._launch_browser()\n",
    "        # Our stop condition is when the LLM decides to output stop_action\n",
    "        should_stop = False\n",
    "        final_answer = None\n",
    "        i = 0\n",
    "        while not should_stop:\n",
    "            # Step 2: Prepare the page for the agent\n",
    "            (\n",
    "                page_text,\n",
    "                formatted_list_of_elements,\n",
    "                som_screenshot,\n",
    "            ) = await self._prepare_page_for_agent(self.page)\n",
    "            # Step 3: Get the LLM response\n",
    "            image_data_url = f\"data:image/png;base64,{base64.b64encode((lambda b: (som_screenshot.save(b, format='PNG'), b.getvalue())[1])(io.BytesIO())).decode()}\"\n",
    "            action_decision = self._get_llm_response(\n",
    "                self.client,\n",
    "                task,\n",
    "                page_text,\n",
    "                formatted_list_of_elements,\n",
    "                image_data_url,\n",
    "                self.message_history,\n",
    "                self.model,\n",
    "            )\n",
    "            print(f\"Action decision {i}: {action_decision}\")\n",
    "            # Add the action decision to the message history\n",
    "            self.message_history.append(\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": [{\"type\": \"text\", \"text\": json.dumps(action_decision)}],\n",
    "                }\n",
    "            )\n",
    "            # Step 4: Execute the action with some error handling\n",
    "            try:\n",
    "                action_feedback = await self._execute_action(\n",
    "                    action_decision[\"action\"], action_decision[\"action_args\"], self.page\n",
    "                )\n",
    "            except Exception as e:\n",
    "                print(f\"Error executing action {i}: {e}\")\n",
    "                action_feedback = f\"Error executing action {i}: {e}\"\n",
    "            print(f\"Action feedback {i}: {action_feedback}\")\n",
    "            # Sleep for 3 seconds to let the page load\n",
    "            await self.page.wait_for_timeout(3000)\n",
    "            # Update the message history with feedback on the action and the new page screenshot\n",
    "            new_page_screenshot = await self.page.screenshot()\n",
    "            self.message_history.append(\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": [\n",
    "                        {\"type\": \"text\", \"text\": action_feedback},\n",
    "                        {\n",
    "                            \"type\": \"image_url\",\n",
    "                            \"image_url\": {\n",
    "                                \"url\": f\"data:image/png;base64,{base64.b64encode(new_page_screenshot).decode()}\"\n",
    "                            },\n",
    "                        },\n",
    "                    ],\n",
    "                }\n",
    "            )\n",
    "            if self.run_in_jupyter:\n",
    "                display(Image.open(io.BytesIO(new_page_screenshot)))\n",
    "            # Check if the task is complete\n",
    "            should_stop = action_decision[\"action\"] == \"stop_action\"\n",
    "            if should_stop:\n",
    "                final_answer = action_decision[\"action_args\"][\"final_answer\"]\n",
    "            i += 1\n",
    "        return final_answer\n",
    "\n",
    "    async def _prepare_page_for_agent(self, page: Page) -> tuple[str, str, bytes]:\n",
    "        interactive_elements = await self.browser_controller.get_interactive_rects(page)\n",
    "        screenshot = await self.browser_controller.get_screenshot(page)\n",
    "        som_screenshot, visible_elements, elements_above, elements_below, _ = (\n",
    "            add_set_of_mark(screenshot, interactive_elements, use_sequential_ids=False)\n",
    "        )\n",
    "        visible_elements_formatted = \"\"\n",
    "        for element_id in visible_elements:\n",
    "            element_data = interactive_elements[element_id]\n",
    "            visible_elements_formatted += f\"{element_id}: {element_data}\\n\"\n",
    "\n",
    "        page_text = await self.browser_controller.get_page_markdown(page)\n",
    "        return page_text, visible_elements_formatted, som_screenshot\n",
    "    async def _execute_action(self, action: str, action_args: dict, page: Page) -> str:\n",
    "        if action == \"goto\":\n",
    "            await self.browser_controller.visit_page(page, action_args[\"url\"])\n",
    "            return f\"Visited {action_args['url']}\"\n",
    "        elif action == \"click\":\n",
    "            await self.browser_controller.click_id(self.context, page, action_args[\"id\"])\n",
    "            return f\"Clicked {action_args['id']}\"\n",
    "        elif action == \"type\":\n",
    "            await self.browser_controller.fill_id(page, action_args[\"id\"], action_args[\"text\"])\n",
    "            return f\"Typed {action_args['text']} into {action_args['id']}\"\n",
    "        elif action == \"scroll\":\n",
    "            if action_args[\"direction\"] == \"up\":\n",
    "                await self.browser_controller.page_up(page)\n",
    "            elif action_args[\"direction\"] == \"down\":\n",
    "                await self.browser_controller.page_down(page)\n",
    "            return f\"Scrolled {action_args['direction']}\"\n",
    "        elif action == \"stop_action\":\n",
    "            return action_args[\"final_answer\"]\n",
    "        else:\n",
    "            raise ValueError(f\"Invalid action: {action}\")\n",
    "\n",
    "    def _get_llm_response(\n",
    "        self,\n",
    "        client: OpenAI,  # OpenAI client\n",
    "        task: str,  # Task to complete\n",
    "        page_text: str,  # Page text\n",
    "        formatted_list_of_elements: str,  # Formatted list of elements\n",
    "        image_data_url: str,  # Image data URL\n",
    "        message_history: list[dict] = [],  # Message history\n",
    "        model: str = \"gpt-4o\",  # Model to use\n",
    "    ) -> dict:\n",
    "        response = client.chat.completions.create(\n",
    "            model=model,\n",
    "            messages=[\n",
    "                *message_history,\n",
    "                {\n",
    "                    \"role\": \"user\",\n",
    "                    \"content\": [\n",
    "                        {\n",
    "                            \"type\": \"text\",\n",
    "                            \"text\": AGENT_PROMPT.format(\n",
    "                                task=task,\n",
    "                                page_text=page_text,\n",
    "                                formatted_list_of_elements=formatted_list_of_elements,\n",
    "                            ),\n",
    "                        },\n",
    "                        {\n",
    "                            \"type\": \"image_url\",\n",
    "                            \"image_url\": {\"url\": image_data_url},\n",
    "                        },\n",
    "                    ],\n",
    "                },\n",
    "            ],\n",
    "        )\n",
    "\n",
    "        # Parse the response\n",
    "        try:\n",
    "            action_decision = json.loads(response.choices[0].message.content)\n",
    "        except json.JSONDecodeError:\n",
    "            # it starts with ```json\n",
    "            response_content = response.choices[0].message.content\n",
    "            response_content = response_content.replace(\"```json\", \"\").replace(\n",
    "                \"```\", \"\"\n",
    "            )\n",
    "            action_decision = json.loads(response_content)\n",
    "        except Exception as e:\n",
    "            raise e\n",
    "        return action_decision\n",
    "\n",
    "    async def close(self) -> None:\n",
    "        if self.page is not None:\n",
    "            await self.page.close()\n",
    "        if self.context is not None:\n",
    "            await self.context.close()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "import os\n",
    "openai_client = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n",
    "agent = BrowserUseAgent(openai_client)\n",
    "try:\n",
    "    final_answer = await agent.execute_task(\"find the open issues assigned to husseinmozannar on the microsoft/magentic-ui repo on github\")\n",
    "    print(final_answer)\n",
    "finally:\n",
    "    await agent.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Success! Our agent again performed the task correctly!\n",
    "\n",
    "With this tutorial, I hope to have convinced you that `magentic-ui` can help you build a browser-use agent more easily. You might be curious how to build the best browser-use agent possible given this, and we have already implemented one for you with many features that we haven't discussed previously in Magentic-UI which we will discuss next."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Level 3: Using the WebSurfer Agent from Magentic-UI\n",
    "\n",
    "We have a reference implementation of a capable browser use agent in Magentic-UI which we call the `WebSurfer` agent. I'll show you now how to use it. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "`WebSurfer` is an AutoGen AgentChat agent built using the tools we have seen previously to complete actions autonomously on the web. We have spent a lot of time fixing many many edge cases that arise on the web to arrive at a more reliable (but not perfect) browser use agent.\n",
    "This agent builds on the [`MultimodalWebSurfer`](https://microsoft.github.io/autogen/stable/reference/python/autogen_ext.agents.web_surfer.html) agent from AutoGen that we previously developed. \n",
    "\n",
    "Let's see now how to use it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from autogen_ext.models.openai import OpenAIChatCompletionClient\n",
    "from magentic_ui.agents import WebSurfer\n",
    "from magentic_ui.tools.playwright import (\n",
    "    LocalPlaywrightBrowser,\n",
    ")\n",
    "\n",
    "browser = LocalPlaywrightBrowser(headless=False)\n",
    "\n",
    "model_client = OpenAIChatCompletionClient(model=\"gpt-4o\")\n",
    "\n",
    "web_surfer = WebSurfer(\n",
    "    name=\"web_surfer\",\n",
    "    model_client=model_client, # Use any client from AutoGen!\n",
    "    animate_actions=True, # Set to True if you want to see the actions being animated!\n",
    "    max_actions_per_step=10, # Maximum number of actions to perform before returning\n",
    "    downloads_folder=\"debug\", # Where to save downloads\n",
    "    debug_dir=\"debug\", # Where to save debug files and screenshots\n",
    "    to_save_screenshots=False, # set to True if you want to save screenshots of the actions\n",
    "    browser=browser, # Use any browser from Magentic-UI!\n",
    "    multiple_tools_per_call=False, # Set to True if you want to use multiple tools per call\n",
    "    json_model_output=False, # Set to True if your model does not support tool calling\n",
    ")\n",
    "await web_surfer.lazy_init()\n",
    "\n",
    "task = \"find the open issues assigned to husseinmozannar on the microsoft/magentic-ui repo on github\"\n",
    "try:\n",
    "    messages = []\n",
    "    async for message in web_surfer.run_stream(task=task):\n",
    "        messages.append(message)\n",
    "        print(message)\n",
    "    print(\"########################################################\")\n",
    "    print(\"Final answer:\")\n",
    "    print(messages[-1].messages[-2].content)\n",
    "finally:\n",
    "    await web_surfer.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We encourage you to experiment using the sample code file [sample_web_surfer.py](https://github.com/microsoft/magentic-ui/blob/main/samples/sample_web_surfer.py) and to use the Magentic-UI application which provides a web UI to interact with the WebSurfer agent and launch multiple parallel tasks and more!\n",
    "\n",
    "Just run:\n",
    "\n",
    "```bash\n",
    "python3 -m venv .venv\n",
    "source .venv/bin/activate\n",
    "pip install magentic-ui\n",
    "# export OPENAI_API_KEY=<YOUR API KEY>\n",
    "magentic ui --port 8081\n",
    "```\n",
    "See [https://github.com/microsoft/magentic-ui](https://github.com/microsoft/magentic-ui) for the full instructions.\n",
    "\n",
    "![../img/magenticui_running.png](../img/magenticui_running.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What's next?\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Evaluation\n",
    "\n",
    "The first thing you might be curious about is how well does the WebSurfer agent perform?\n",
    "\n",
    "In Magentic-UI, we have built a small evaluation library [magentic-ui/eval](https://github.com/microsoft/magentic-ui/tree/main/src/magentic_ui/eval) that implements popular browser-use benchmarks and makes it easy to run evals. We will be building a bit on this library and will have a tutorial on how to use it.\n",
    "\n",
    " Magentic-UI has been tested against several benchmarks when running with o4-mini: [GAIA](https://huggingface.co/datasets/gaia-benchmark/GAIA) test set (42.52%), which assesses general AI assistants across reasoning, tool use, and web interaction tasks ; [AssistantBench](https://huggingface.co/AssistantBench) test set (27.60%), focusing on realistic, time-consuming web tasks; [WebVoyager](https://github.com/MinorJerry/WebVoyager) (82.2%), measuring end-to-end web navigation in real-world scenarios; and [WebGames](https://webgames.convergence.ai/) (45.5%), evaluating general-purpose web-browsing agents through interactive challenges.\n",
    "To reproduce these experimental results, please see the following [instructions](experiments/README.md).\n",
    "\n",
    "For reference, the current SOTA on WebVoyager is the [browser-use library](https://browser-use.com/posts/sota-technical-report) using GPT-4o achieving 89%. Note that the WebVoyager evaluation is not consistent across different systems as it relies on a mix of LLM-as-a-judge evaluation and human evaluation.\n",
    "\n",
    "\n",
    "## Limitations\n",
    "\n",
    "Using the Set-Of-Mark approach for building the Browser Use Agent has many limitations (note that both Magentic-UI and Browser Use library use SoM). For instance, any task that requires understanding coordinates on the screen our agent will fail on.\n",
    "Examples:\n",
    "\n",
    "- dragging an element from position A to position B\n",
    "- drawing on the screen\n",
    "- playing web games\n",
    "\n",
    "Moreover, it will not generalize to any Computer Use task where we might not have the DOM to obtain element coordinates. Therefore, we will need to have a model that can click on specific coordinates rather than using element IDs. The [UI-Tars](https://github.com/bytedance/UI-TARS) models have such an ability as well as the latest [compute-preview-api](https://platform.openai.com/docs/guides/tools-computer-use) from OpenAI. Another approach is to use a grounding or parsing model instead of the DOM such as [OmniParser](https://microsoft.github.io/OmniParser/) to obtain element IDs from any GUI interface combined with a tool-calling LLM.\n",
    "\n",
    "\n",
    "Another limitation is that these agents are not *real-time* and so tasks such as video-understanding or playing games become almost impossible natively as there multiple seconds delay between each agent action.\n",
    "\n",
    "## Safety\n",
    "\n",
    "Current LLMs are still very prone to adversarial attacks on the web, see these papers for how bad things can get it with current models even those tuned directly for CUA:\n",
    "\n",
    "- [Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attacks\n",
    "](https://arxiv.org/html/2502.08586v1)\n",
    "- [RedTeamCUA:\n",
    "Realistic Adversarial Testing of Computer-Use Agents in\n",
    "Hybrid Web-OS Environments](https://osu-nlp-group.github.io/RedTeamCUA/)\n",
    "\n",
    "We recommend to have guardrails built into the agent to allow the human to approve actions if needed. We call such guardrails \"ActionGuard\" in Magentic-UI and they allow you to define heuristics in addition to LLM judgmenet for when actions might need human approval.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you've made it this far I really appreciate you taking the time to read and hope you've enjoyed following along!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}