Image-Text-to-Text
Transformers
Safetensors
Korean
English
qwen3_5
gui-agent
web-agent
computer-use
browser
korean
conversational
Instructions to use KORMo-VL/KORMo-Web-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KORMo-VL/KORMo-Web-4B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="KORMo-VL/KORMo-Web-4B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("KORMo-VL/KORMo-Web-4B") model = AutoModelForMultimodalLM.from_pretrained("KORMo-VL/KORMo-Web-4B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use KORMo-VL/KORMo-Web-4B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KORMo-VL/KORMo-Web-4B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-4B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/KORMo-VL/KORMo-Web-4B
- SGLang
How to use KORMo-VL/KORMo-Web-4B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-4B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-4B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-4B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-4B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use KORMo-VL/KORMo-Web-4B with Docker Model Runner:
docker model run hf.co/KORMo-VL/KORMo-Web-4B
Quickstart works on vLLM and vllm-mlx: tool_choice, tool_calls restore, Mac 4B/9B serving + memory, fixes
Browse files- .gitattributes +1 -0
- HARNESS.md +11 -0
- README.md +37 -9
- assets/{gui_final_demo_maksed.mp4 โ gui_final_demo_masked.mp4} +0 -0
- examples/agent_playwright.py +33 -9
.gitattributes
CHANGED
|
@@ -39,3 +39,4 @@ assets/GUI_agent_dem.mp4 filter=lfs diff=lfs merge=lfs -text
|
|
| 39 |
assets/gui_final_demo_dubbed.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 40 |
assets/gui_final_demo_maksed.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 41 |
assets/setup_guide.mp4 filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 39 |
assets/gui_final_demo_dubbed.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 40 |
assets/gui_final_demo_maksed.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 41 |
assets/setup_guide.mp4 filter=lfs diff=lfs merge=lfs -text
|
| 42 |
+
assets/gui_final_demo_masked.mp4 filter=lfs diff=lfs merge=lfs -text
|
HARNESS.md
CHANGED
|
@@ -19,6 +19,17 @@ A runnable reference implementation is in [`examples/agent_playwright.py`](examp
|
|
| 19 |
| Sampling | [`generation_config.json`](generation_config.json) | `temperature 0.6 ยท top_p 0.95 ยท top_k 20` |
|
| 20 |
| Harness | **this file** | Message layout, parsing, execution, feedback |
|
| 21 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
---
|
| 23 |
|
| 24 |
## 2. Conversation layout
|
|
|
|
| 19 |
| Sampling | [`generation_config.json`](generation_config.json) | `temperature 0.6 ยท top_p 0.95 ยท top_k 20` |
|
| 20 |
| Harness | **this file** | Message layout, parsing, execution, feedback |
|
| 21 |
|
| 22 |
+
### Sending `tools` and `tool_choice`
|
| 23 |
+
|
| 24 |
+
Always pass the 22 tools as `tools=`; the chat template renders them into the prompt. What `tool_choice` to send depends on the server:
|
| 25 |
+
|
| 26 |
+
| Server | `tool_choice` | Where the call comes back |
|
| 27 |
+
|---|---|---|
|
| 28 |
+
| vLLM (no `--enable-auto-tool-choice`) | `"none"` | As text in `message.content` (`<think>โฆ</think>` + `<tool_call>โฆ</tool_call>`). Omitting `tool_choice` returns HTTP 400. |
|
| 29 |
+
| vllm-mlx (Apple silicon) | `"auto"` | In `message.tool_calls`. `"none"` would drop the tool schema from the prompt. |
|
| 30 |
+
|
| 31 |
+
When the call arrives in `message.tool_calls`, rebuild the `<tool_call><function=โฆ><parameter=โฆ>` text (ยง3) and keep that in the assistant turn, so the history looks exactly like training. `restore_tool_calls()` in the reference agent does this.
|
| 32 |
+
|
| 33 |
---
|
| 34 |
|
| 35 |
## 2. Conversation layout
|
README.md
CHANGED
|
@@ -42,7 +42,7 @@ tags:
|
|
| 42 |
|
| 43 |
<div id="kormo-web-card">
|
| 44 |
|
| 45 |
-
<video src="https://huggingface.co/KORMo-VL/KORMo-Web-4B/resolve/main/assets/
|
| 46 |
|
| 47 |
# KORMo-Web-4B
|
| 48 |
|
|
@@ -95,14 +95,29 @@ The fastest way to try KORMo-Web is the **[KORMo-Web Chrome extension](https://g
|
|
| 95 |
|
| 96 |
### Mac (Apple silicon)
|
| 97 |
|
| 98 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 99 |
|
| 100 |
```bash
|
| 101 |
uv venv --python 3.12 && source .venv/bin/activate
|
| 102 |
-
uv pip install vllm-mlx
|
| 103 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 104 |
```
|
| 105 |
|
|
|
|
|
|
|
| 106 |
### NVIDIA GPU (local PC or server)
|
| 107 |
|
| 108 |
Serve the model with vLLM. Much faster, and one server can be shared.
|
|
@@ -115,13 +130,13 @@ vllm serve KORMo-VL/KORMo-Web-4B --host 0.0.0.0 --port 8000 \
|
|
| 115 |
--gdn-prefill-backend triton
|
| 116 |
```
|
| 117 |
|
| 118 |
-
Check that it is up with `curl http://localhost:8010/v1/models` (Mac) or `curl http://<server>:8000/v1/models` (vLLM).
|
| 119 |
|
| 120 |
**2. Install the extension.** `git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git`, open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked** and choose the cloned folder. Open the **GUI ์์ด์ ํธ** side panel, type a task and press **์คํ** (Run).
|
| 121 |
|
| 122 |
**3. Point the extension at your model.** Open the panel's **๊ฐ๋ฐ์ ์ค์ ** (Developer settings).
|
| 123 |
|
| 124 |
-
- **Mac:**
|
| 125 |
- **NVIDIA GPU:** set the server URL to `http://<server>:8000/v1` (or `localhost`), the model name to `KORMo-Web-4B`, and `tool_choice` to `none`.
|
| 126 |
|
| 127 |
**4. Discord (optional).** Run `node relay/discord-relay.mjs` in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.
|
|
@@ -153,10 +168,20 @@ vllm serve ./KORMo-Web-4B --served-model-name KORMo-Web-4B \
|
|
| 153 |
|
| 154 |
```bash
|
| 155 |
python KORMo-Web-4B/examples/agent_playwright.py --model KORMo-Web-4B \
|
| 156 |
-
--
|
|
|
|
| 157 |
```
|
| 158 |
|
| 159 |
-
The agent
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 160 |
|
| 161 |
### Inference example (one step)
|
| 162 |
|
|
@@ -183,6 +208,7 @@ messages = [
|
|
| 183 |
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
| 184 |
out = client.chat.completions.create(
|
| 185 |
model="KORMo-Web-4B", messages=messages, tools=TOOLS,
|
|
|
|
| 186 |
temperature=0.6, top_p=0.95, max_tokens=16384,
|
| 187 |
extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
|
| 188 |
)
|
|
@@ -209,6 +235,8 @@ My plan: 1. Click the search box. 2. Type "KAIST". 3. Press Enter and open the o
|
|
| 209 |
</tool_call>
|
| 210 |
```
|
| 211 |
|
|
|
|
|
|
|
| 212 |
Coordinates in the tool call are normalized to 0โ1000. Convert them to pixels with `x / 1000 ร width` and `y / 1000 ร height`.
|
| 213 |
|
| 214 |
## How It Works
|
|
@@ -240,7 +268,7 @@ The exact message format, parsing rules and feedback strings are in **[HARNESS.m
|
|
| 240 |
| `chat_template.jinja` | Chat template (Qwen3.5 XML tool calls, thinking on) |
|
| 241 |
| `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema, verbatim from training |
|
| 242 |
| `HARNESS.md` | Full harness specification |
|
| 243 |
-
| `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). It does not execute `upload`. |
|
| 244 |
| `config.json` ยท `generation_config.json` ยท `processor_config.json` ยท `tokenizer*.json` | Model, sampling and processor configs |
|
| 245 |
|
| 246 |
## ํ๊ตญ์ด ์์ฝ
|
|
|
|
| 42 |
|
| 43 |
<div id="kormo-web-card">
|
| 44 |
|
| 45 |
+
<video src="https://huggingface.co/KORMo-VL/KORMo-Web-4B/resolve/main/assets/gui_final_demo_masked.mp4" controls autoplay muted loop playsinline width="100%"></video>
|
| 46 |
|
| 47 |
# KORMo-Web-4B
|
| 48 |
|
|
|
|
| 95 |
|
| 96 |
### Mac (Apple silicon)
|
| 97 |
|
| 98 |
+
KORMo-Web-4B (4-bit or full precision) and KORMo-Web-9B all run on a Mac with `vllm-mlx` 0.6.0 or later. Pick the one that fits your Mac's memory:
|
| 99 |
+
|
| 100 |
+
| Model | Weights | Memory in use | Recommended Mac memory |
|
| 101 |
+
|---|---:|---:|---|
|
| 102 |
+
| `KORMo-Web-4B-mlx-4bit` (4-bit) | 2.8 GB | about 3.8 GB | 16 GB or more |
|
| 103 |
+
| `KORMo-Web-4B` (bf16) | 9.6 GB | about 9.5 GB | 24 GB or more |
|
| 104 |
+
| `KORMo-Web-9B` (bf16) | 17.5 GB | about 19 GB | 48 GB or more (32 GB is tight) |
|
| 105 |
+
|
| 106 |
+
*Measured with vllm-mlx 0.6.0 after a few short requests (macOS units, 1 GB = 1024ยณ bytes). Long episodes add KV cache, about 1 GB per 30k tokens, and Chrome needs memory too. macOS lets the GPU use only about 2/3โ3/4 of RAM.*
|
| 107 |
|
| 108 |
```bash
|
| 109 |
uv venv --python 3.12 && source .venv/bin/activate
|
| 110 |
+
uv pip install "vllm-mlx>=0.6.0" huggingface_hub
|
| 111 |
+
export HF_HUB_DISABLE_XET=1 # only if large downloads stall
|
| 112 |
+
|
| 113 |
+
MODEL=KORMo-Web-4B # or KORMo-Web-4B-mlx-4bit / KORMo-Web-9B
|
| 114 |
+
hf download KORMo-VL/$MODEL --local-dir ~/models/$MODEL
|
| 115 |
+
vllm-mlx serve ~/models/$MODEL --port 8010 --served-model-name KORMo-VL/$MODEL \
|
| 116 |
+
--enable-auto-tool-choice --tool-call-parser auto
|
| 117 |
```
|
| 118 |
|
| 119 |
+
Keep this terminal open. `--served-model-name` is the name clients must request; without it the server registers the local folder path and requests by model name return 404. vllm-mlx 0.6.0 and later detect the vision model by itself; older versions need `--mllm`.
|
| 120 |
+
|
| 121 |
### NVIDIA GPU (local PC or server)
|
| 122 |
|
| 123 |
Serve the model with vLLM. Much faster, and one server can be shared.
|
|
|
|
| 130 |
--gdn-prefill-backend triton
|
| 131 |
```
|
| 132 |
|
| 133 |
+
Check that it is up with `curl http://localhost:8010/v1/models` (Mac) or `curl http://<server>:8000/v1/models` (vLLM). If you changed the port, use that port.
|
| 134 |
|
| 135 |
**2. Install the extension.** `git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git`, open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked** and choose the cloned folder. Open the **GUI ์์ด์ ํธ** side panel, type a task and press **์คํ** (Run).
|
| 136 |
|
| 137 |
**3. Point the extension at your model.** Open the panel's **๊ฐ๋ฐ์ ์ค์ ** (Developer settings).
|
| 138 |
|
| 139 |
+
- **Mac:** keep the server URL `http://localhost:8010/v1` and `tool_choice` `auto`. Set the model name to your `--served-model-name`, for example `KORMo-VL/KORMo-Web-4B`. The default model name `KORMo-VL/KORMo-Web-4B-mlx-4bit` already matches the 4-bit build.
|
| 140 |
- **NVIDIA GPU:** set the server URL to `http://<server>:8000/v1` (or `localhost`), the model name to `KORMo-Web-4B`, and `tool_choice` to `none`.
|
| 141 |
|
| 142 |
**4. Discord (optional).** Run `node relay/discord-relay.mjs` in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.
|
|
|
|
| 168 |
|
| 169 |
```bash
|
| 170 |
python KORMo-Web-4B/examples/agent_playwright.py --model KORMo-Web-4B \
|
| 171 |
+
--start-url https://www.danawa.com/ \
|
| 172 |
+
--task "๋ค๋์์์ 50๋ง ์ ์ดํ, 10์ธ์น ์ด์ ํ๋ธ๋ฆฟ 3๊ฐ๋ฅผ ์ฐพ์์ ๊ฐ๊ฒฉ์ ์๋ ค์ค"
|
| 173 |
```
|
| 174 |
|
| 175 |
+
The agent drives a headless Chromium, prints each action as it goes, and finishes with the model's answer. Add `--headful` to watch the browser window.
|
| 176 |
+
|
| 177 |
+
**On a Mac** (`vllm-mlx`, see *How to Start* above), point the script at that server and pass `--tool-choice auto`:
|
| 178 |
+
|
| 179 |
+
```bash
|
| 180 |
+
python KORMo-Web-4B/examples/agent_playwright.py --endpoint http://localhost:8010/v1 \
|
| 181 |
+
--model KORMo-VL/KORMo-Web-4B --tool-choice auto --headful \
|
| 182 |
+
--start-url https://www.danawa.com/ \
|
| 183 |
+
--task "๋ค๋์์์ 50๋ง ์ ์ดํ, 10์ธ์น ์ด์ ํ๋ธ๋ฆฟ 3๊ฐ๋ฅผ ์ฐพ์์ ๊ฐ๊ฒฉ์ ์๋ ค์ค"
|
| 184 |
+
```
|
| 185 |
|
| 186 |
### Inference example (one step)
|
| 187 |
|
|
|
|
| 208 |
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
| 209 |
out = client.chat.completions.create(
|
| 210 |
model="KORMo-Web-4B", messages=messages, tools=TOOLS,
|
| 211 |
+
tool_choice="none", # vLLM. On vllm-mlx send "auto" and read message.tool_calls (see below)
|
| 212 |
temperature=0.6, top_p=0.95, max_tokens=16384,
|
| 213 |
extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
|
| 214 |
)
|
|
|
|
| 235 |
</tool_call>
|
| 236 |
```
|
| 237 |
|
| 238 |
+
**`tool_choice` depends on the server.** On vLLM, send `"none"`: the tool schema is still rendered into the prompt and the call comes back as text in `message.content`, as above. Sending `tools` without `tool_choice` makes vLLM return HTTP 400 unless the server was started with `--enable-auto-tool-choice`. On vllm-mlx, `"none"` drops the tool schema from the prompt, so send `"auto"`; the call then arrives in `message.tool_calls`. [`examples/agent_playwright.py`](examples/agent_playwright.py) turns it back into the `<tool_call>` text above (`--tool-choice auto`).
|
| 239 |
+
|
| 240 |
Coordinates in the tool call are normalized to 0โ1000. Convert them to pixels with `x / 1000 ร width` and `y / 1000 ร height`.
|
| 241 |
|
| 242 |
## How It Works
|
|
|
|
| 268 |
| `chat_template.jinja` | Chat template (Qwen3.5 XML tool calls, thinking on) |
|
| 269 |
| `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema, verbatim from training |
|
| 270 |
| `HARNESS.md` | Full harness specification |
|
| 271 |
+
| `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). `--tool-choice none` for vLLM, `auto` for vllm-mlx; `--headful` shows the browser. It does not execute `upload`. |
|
| 272 |
| `config.json` ยท `generation_config.json` ยท `processor_config.json` ยท `tokenizer*.json` | Model, sampling and processor configs |
|
| 273 |
|
| 274 |
## ํ๊ตญ์ด ์์ฝ
|
assets/{gui_final_demo_maksed.mp4 โ gui_final_demo_masked.mp4}
RENAMED
|
File without changes
|
examples/agent_playwright.py
CHANGED
|
@@ -1,9 +1,16 @@
|
|
| 1 |
"""Minimal reference harness for KORMo-Web-4B (see ../HARNESS.md).
|
| 2 |
|
| 3 |
pip install openai playwright && playwright install chromium
|
| 4 |
-
vllm serve KORMo-VL/KORMo-Web-4B --
|
|
|
|
| 5 |
python agent_playwright.py --task "Search for 'KAIST' on Google and open the official homepage."
|
| 6 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
It reproduces the message layout the model was trained on:
|
| 8 |
system = system_prompt.md tools = tools.json (22 tools, rendered by chat_template.jinja)
|
| 9 |
user = <user_request>โฆ</user_request> + <observation> (+ screenshot on the first step)
|
|
@@ -66,6 +73,20 @@ async def observation(ctx, page, with_shot: bool) -> list[dict]:
|
|
| 66 |
{"type": "text", "text": "\n</observation>"}]
|
| 67 |
|
| 68 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
def drop_images(content: list[dict]) -> list[dict]:
|
| 70 |
"""Past turns keep the observation text but lose the screenshot."""
|
| 71 |
out = [c for c in content if c["type"] == "text"]
|
|
@@ -142,10 +163,11 @@ async def execute(ctx, page, name: str, a: dict):
|
|
| 142 |
return page, fb + (" Note: no visible navigation or new tab detected." if name.endswith("click") else "")
|
| 143 |
|
| 144 |
|
| 145 |
-
async def run(task: str, start_url: str, endpoint: str, model: str, max_steps: int
|
|
|
|
| 146 |
client = OpenAI(base_url=endpoint, api_key="EMPTY")
|
| 147 |
async with async_playwright() as pw:
|
| 148 |
-
browser = await pw.chromium.launch()
|
| 149 |
ctx = await browser.new_context(viewport={"width": 1280, "height": 720})
|
| 150 |
page = await ctx.new_page()
|
| 151 |
await page.goto(start_url)
|
|
@@ -153,10 +175,9 @@ async def run(task: str, start_url: str, endpoint: str, model: str, max_steps: i
|
|
| 153 |
first[0]["text"] = f"<user_request>\n{task}\n</user_request>\n" + first[0]["text"]
|
| 154 |
msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": first}]
|
| 155 |
for step in range(max_steps):
|
| 156 |
-
out = client.chat.completions.create(model=model, messages=msgs, tools=TOOLS,
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
text = f"<think>\n{out.choices[0].message.reasoning_content}\n</think>\n\n{text}"
|
| 160 |
msgs.append({"role": "assistant", "content": text}) # keep the full <think> + <tool_call>
|
| 161 |
try:
|
| 162 |
name, args = parse_call(text)
|
|
@@ -181,7 +202,10 @@ if __name__ == "__main__":
|
|
| 181 |
ap.add_argument("--task", required=True)
|
| 182 |
ap.add_argument("--start-url", default="https://www.google.com/")
|
| 183 |
ap.add_argument("--endpoint", default="http://localhost:8000/v1")
|
| 184 |
-
ap.add_argument("--model", default="KORMo-
|
| 185 |
ap.add_argument("--max-steps", type=int, default=50)
|
|
|
|
|
|
|
|
|
|
| 186 |
a = ap.parse_args()
|
| 187 |
-
asyncio.run(run(a.task, a.start_url, a.endpoint, a.model, a.max_steps))
|
|
|
|
| 1 |
"""Minimal reference harness for KORMo-Web-4B (see ../HARNESS.md).
|
| 2 |
|
| 3 |
pip install openai playwright && playwright install chromium
|
| 4 |
+
vllm serve KORMo-VL/KORMo-Web-4B --served-model-name KORMo-Web-4B \
|
| 5 |
+
--max-model-len 131072 --limit-mm-per-prompt '{"image":1,"video":0}'
|
| 6 |
python agent_playwright.py --task "Search for 'KAIST' on Google and open the official homepage."
|
| 7 |
|
| 8 |
+
vLLM : --tool-choice none (default). The tool schema is still rendered into the prompt
|
| 9 |
+
and the call comes back as text in message.content.
|
| 10 |
+
vllm-mlx : --tool-choice auto. vllm-mlx drops the schema on "none" and returns the call in
|
| 11 |
+
message.tool_calls; it is turned back into the <tool_call> XML below.
|
| 12 |
+
--headful : show the browser window (default: headless).
|
| 13 |
+
|
| 14 |
It reproduces the message layout the model was trained on:
|
| 15 |
system = system_prompt.md tools = tools.json (22 tools, rendered by chat_template.jinja)
|
| 16 |
user = <user_request>โฆ</user_request> + <observation> (+ screenshot on the first step)
|
|
|
|
| 73 |
{"type": "text", "text": "\n</observation>"}]
|
| 74 |
|
| 75 |
|
| 76 |
+
def restore_tool_calls(msg) -> str:
|
| 77 |
+
"""vllm-mlx returns the call in message.tool_calls; put it back as the <tool_call> XML the model wrote."""
|
| 78 |
+
text = msg.content or ""
|
| 79 |
+
if "<think>" not in text and "</think>" not in text and getattr(msg, "reasoning_content", None):
|
| 80 |
+
text = f"<think>\n{msg.reasoning_content}\n</think>\n\n{text}"
|
| 81 |
+
if getattr(msg, "tool_calls", None) and "<tool_call>" not in text:
|
| 82 |
+
for tc in msg.tool_calls:
|
| 83 |
+
args = json.loads(tc.function.arguments or "{}")
|
| 84 |
+
params = "".join(f"<parameter={k}>\n{v if isinstance(v, str) else json.dumps(v, ensure_ascii=False)}\n</parameter>\n"
|
| 85 |
+
for k, v in args.items())
|
| 86 |
+
text += f"\n\n<tool_call>\n<function={tc.function.name}>\n{params}</function>\n</tool_call>"
|
| 87 |
+
return text
|
| 88 |
+
|
| 89 |
+
|
| 90 |
def drop_images(content: list[dict]) -> list[dict]:
|
| 91 |
"""Past turns keep the observation text but lose the screenshot."""
|
| 92 |
out = [c for c in content if c["type"] == "text"]
|
|
|
|
| 163 |
return page, fb + (" Note: no visible navigation or new tab detected." if name.endswith("click") else "")
|
| 164 |
|
| 165 |
|
| 166 |
+
async def run(task: str, start_url: str, endpoint: str, model: str, max_steps: int,
|
| 167 |
+
tool_choice: str = "none", headful: bool = False):
|
| 168 |
client = OpenAI(base_url=endpoint, api_key="EMPTY")
|
| 169 |
async with async_playwright() as pw:
|
| 170 |
+
browser = await pw.chromium.launch(headless=not headful)
|
| 171 |
ctx = await browser.new_context(viewport={"width": 1280, "height": 720})
|
| 172 |
page = await ctx.new_page()
|
| 173 |
await page.goto(start_url)
|
|
|
|
| 175 |
first[0]["text"] = f"<user_request>\n{task}\n</user_request>\n" + first[0]["text"]
|
| 176 |
msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": first}]
|
| 177 |
for step in range(max_steps):
|
| 178 |
+
out = client.chat.completions.create(model=model, messages=msgs, tools=TOOLS,
|
| 179 |
+
tool_choice=tool_choice, **SAMPLING)
|
| 180 |
+
text = restore_tool_calls(out.choices[0].message)
|
|
|
|
| 181 |
msgs.append({"role": "assistant", "content": text}) # keep the full <think> + <tool_call>
|
| 182 |
try:
|
| 183 |
name, args = parse_call(text)
|
|
|
|
| 202 |
ap.add_argument("--task", required=True)
|
| 203 |
ap.add_argument("--start-url", default="https://www.google.com/")
|
| 204 |
ap.add_argument("--endpoint", default="http://localhost:8000/v1")
|
| 205 |
+
ap.add_argument("--model", default="KORMo-Web-4B", help="the server's --served-model-name")
|
| 206 |
ap.add_argument("--max-steps", type=int, default=50)
|
| 207 |
+
ap.add_argument("--tool-choice", default="none", choices=["none", "auto"],
|
| 208 |
+
help="none for vLLM, auto for vllm-mlx (Apple silicon)")
|
| 209 |
+
ap.add_argument("--headful", action="store_true", help="show the browser window")
|
| 210 |
a = ap.parse_args()
|
| 211 |
+
asyncio.run(run(a.task, a.start_url, a.endpoint, a.model, a.max_steps, a.tool_choice, a.headful))
|