File size: 15,215 Bytes
3f84c1c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
af8d263
 
 
 
3f84c1c
 
 
af8d263
3f84c1c
 
 
 
 
 
af8d263
 
3f84c1c
af8d263
3f84c1c
 
 
 
 
 
 
ca44759
588526b
 
3f84c1c
 
f95fbc9
 
af8d263
3f84c1c
588526b
bc2bf7b
588526b
 
 
bc2bf7b
f95fbc9
3f84c1c
f95fbc9
3f84c1c
f95fbc9
 
 
 
 
 
 
82e57e4
3f84c1c
 
82e57e4
3f84c1c
1abe989
3f84c1c
f95fbc9
 
 
 
 
 
 
3f84c1c
11c35b2
 
 
 
 
 
 
 
7e7e2f7
11c35b2
7e7e2f7
11c35b2
ca44759
 
 
 
 
 
 
 
 
11c35b2
7e7e2f7
 
ca44759
 
 
 
 
 
 
7e7e2f7
11c35b2
ca44759
 
7e7e2f7
11c35b2
7e7e2f7
 
 
 
 
 
 
 
 
 
ca44759
11c35b2
 
 
7e7e2f7
 
ca44759
7e7e2f7
11c35b2
 
 
 
 
3f84c1c
 
f95fbc9
 
4fcf1f1
 
f95fbc9
3f84c1c
 
f95fbc9
 
07d6bdb
3f84c1c
 
f95fbc9
 
 
 
 
 
 
 
 
 
 
ca44759
 
f95fbc9
 
ca44759
 
 
 
 
 
 
 
 
 
f95fbc9
4fcf1f1
 
 
3f84c1c
 
 
 
 
f95fbc9
 
3f84c1c
 
f95fbc9
 
3f84c1c
 
 
f95fbc9
3f84c1c
 
 
 
f95fbc9
3f84c1c
f95fbc9
ca44759
3f84c1c
f95fbc9
3f84c1c
4fcf1f1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3f84c1c
 
ca44759
 
f95fbc9
3f84c1c
f95fbc9
3f84c1c
f95fbc9
 
 
 
 
3f84c1c
 
f95fbc9
 
 
3f84c1c
f95fbc9
3f84c1c
f95fbc9
3f84c1c
f95fbc9
 
4528274
1d03290
3f84c1c
 
 
 
 
f95fbc9
 
 
ca44759
f95fbc9
3f84c1c
 
 
ca44759
3f84c1c
588526b
 
ca44759
588526b
 
0a9ef25
3f84c1c
 
d9f0dce
3f84c1c
b46d120
 
 
 
 
 
638b9ac
 
b46d120
 
 
3f84c1c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
---
license: apache-2.0
base_model:
- Qwen/Qwen3.5-9B
language:
- ko
- en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- gui-agent
- web-agent
- computer-use
- browser
- korean
---

<style>
#kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; }
#kormo-web-card h1 { margin: 0 0 12px; font-size: 30px; line-height: 1.25; letter-spacing: -0.018em; }
#kormo-web-card h2 { margin: 30px 0 10px; font-size: 22px; line-height: 1.3; letter-spacing: -0.012em; }
#kormo-web-card h3 { margin: 22px 0 8px; font-size: 17px; font-weight: 700; line-height: 1.4; }
#kormo-web-card h2 + h3 { margin-top: 14px; }
#kormo-web-card p { margin: 0 0 10px; }
#kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; }
#kormo-web-card li { margin: 4px 0; line-height: 1.55; }
#kormo-web-card table { width: 100%; margin: 6px 0 14px; border-collapse: collapse; font-size: 14px; line-height: 1.45; font-variant-numeric: tabular-nums; }
#kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; }
#kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; }
#kormo-web-card blockquote p { margin: 0; }
#kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; }
#kormo-web-card pre code { font-size: 13px; }
#kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; }
#kormo-web-card .pills { margin: 4px 0 18px; }
#kormo-web-card .pill { display: inline-block; margin: 0 6px 6px 0; padding: 3px 10px; border-radius: 999px; border: 1px solid rgba(127,127,127,.35); font-size: 13px; }
@media (max-width: 640px) {
  #kormo-web-card h1 { font-size: 26px; }
  #kormo-web-card table { font-size: 13px; }
  #kormo-web-card th, #kormo-web-card td { padding: 7px 6px; }
}
</style>

<div id="kormo-web-card">

<video src="https://huggingface.co/KORMo-VL/KORMo-Web-9B/resolve/main/assets/gui_final_demo_masked.mp4" controls autoplay muted loop playsinline width="100%"></video>


# KORMo-Web-9B

**KORMo-Web-9B** is a 9B vision-language agent that completes real browser tasks **from screenshots alone**. Give it a request in Korean or English; it plans, clicks, types and moves between sites and tabs until the task is done. In the same live-web evaluation, it beats every open-weight web agent we compared, **by +18.3 points in English and +27.9 points in Korean**.

On **Online-Mind2Web** it reaches **51.0% in English** and **48.1% in Korean**, which is **+26.3 / +31.4 points** over the base Qwen3.5-9B.

## What Makes KORMo-Web Different?

- **10M+ Atomic-Level Web Pretraining Samples.** Large-scale Korean and English web understanding pretraining teaches the model what individual UI elements **are, where they are, and what they do**โ€”covering text, location, appearance, layout, function, and state.
- **100K+ Web Interaction Trajectories.** Bilingual browser demonstrations teach the model to translate fine-grained web understanding into real-world, multi-step actions.
- **Open Models and Data.** Releasing model weights, large-scale bilingual web pretraining data, interaction trajectories, and evaluation benchmarks to support reproducible web agent research.

## Performance

All models below ran **the same 612 live-website tasks** (300 English, 312 Korean), each with its own prompt and action format, and were scored by **the same LLM judge**.

| Model | Size | Online-Mind2Web EN | Online-Mind2Web KO |
|---|---:|---:|---:|
| Qwen3.5-9B (base, no fine-tuning) | 9B | 24.7 | 16.7 |
| Fara-1.5-4B | 4B | 20.7 | 7.7 |
| UI-TARS-1.5-7B | 7B | 22.3 | 20.2 |
| OpenWebRL-4B | 4B | 32.7 | 15.4 |
| **KORMo-Web-9B (ours)** | **9B** | **51.0** | **48.1** |
| GPT-6-luna (proprietary) | โ€“ | 63.0 | 65.7 |


Success rate (%). The denominator is always the full task count, so crashed or empty runs count as failures.

KORMo-Web-9B outperforms open web agents of similar size in both English and Korean.

## Why KORMo-Web

- **Korean and English, natural requests.** Handles short commands as well as long, conversational instructions in either language.
- **Screenshot-only.** It needs no DOM or accessibility tree; it acts on what is on screen, like a person, and checks each action's result on the next screenshot.
- **Thinks, then acts.** A short reasoning step precedes every action, so its behaviour is easy to inspect and debug.
- **Multi-site, multi-tab workflows.** It can search on one site, verify on another and write the result into a third, managing tabs on its own.
- **Open and ready to run.** Apache-2.0 weights, a reference browser harness, the tool schema and the exact inference settings are all included.

## How to Start: Chrome Extension

The fastest way to try KORMo-Web is the **[KORMo-Web Chrome extension](https://github.com/MLP-Lab/KORMo-Web-Extension)**: type a request in Chrome's side panel and watch the agent work in your own browser. This video walks through the whole setup, from serving the model to running a task and controlling it from Discord.

<video src="https://huggingface.co/KORMo-VL/KORMo-Web-9B/resolve/main/assets/setup_guide.mp4" controls playsinline preload="metadata" width="100%"></video>

*Setup guide (about 15 min, Korean narration). Parts that showed account tokens have been blurred.*

**1. Serve the model** on a Mac or on an NVIDIA GPU.

### Mac (Apple silicon)

KORMo-Web-4B (4-bit or full precision) and KORMo-Web-9B all run on a Mac with `vllm-mlx` 0.6.0 or later. Pick the one that fits your Mac's memory:

| Model | Weights | Memory in use | Recommended Mac memory |
|---|---:|---:|---|
| `KORMo-Web-4B-mlx-4bit` (4-bit) | 2.8 GB | about 3.8 GB | 16 GB or more |
| `KORMo-Web-4B` (bf16) | 9.6 GB | about 9.5 GB | 24 GB or more |
| `KORMo-Web-9B` (bf16) | 17.5 GB | about 19 GB | 48 GB or more (32 GB is tight) |

*Measured with vllm-mlx 0.6.0 after a few short requests (macOS units, 1 GB = 1024ยณ bytes). Long episodes add KV cache, about 1 GB per 30k tokens, and Chrome needs memory too. macOS lets the GPU use only about 2/3โ€“3/4 of RAM.*

```bash
uv venv --python 3.12 && source .venv/bin/activate
uv pip install "vllm-mlx>=0.6.0" huggingface_hub
export HF_HUB_DISABLE_XET=1    # only if large downloads stall

MODEL=KORMo-Web-9B             # or KORMo-Web-4B / KORMo-Web-4B-mlx-4bit
hf download KORMo-VL/$MODEL --local-dir ~/models/$MODEL
vllm-mlx serve ~/models/$MODEL --port 8010 --served-model-name KORMo-VL/$MODEL \
  --enable-auto-tool-choice --tool-call-parser auto
```

Keep this terminal open. `--served-model-name` is the name clients must request; without it the server registers the local folder path and requests by model name return 404. vllm-mlx 0.6.0 and later detect the vision model by itself; older versions need `--mllm`.

### NVIDIA GPU (local PC or server)

Serve the model with vLLM. Much faster, and one server can be shared.

```bash
pip install vllm
vllm serve KORMo-VL/KORMo-Web-9B --host 0.0.0.0 --port 8000 \
  --served-model-name KORMo-Web-9B \
  --mm-encoder-tp-mode data --mm-processor-cache-type shm \
  --gdn-prefill-backend triton
```

Check that it is up with `curl http://localhost:8010/v1/models` (Mac) or `curl http://<server>:8000/v1/models` (vLLM). If you changed the port, use that port.

**2. Install the extension.** `git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git`, open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked** and choose the cloned folder. Open the **GUI ์—์ด์ „ํŠธ** side panel, type a task and press **์‹คํ–‰** (Run).

**3. Point the extension at your model.** Open the panel's **๊ฐœ๋ฐœ์ž ์„ค์ •** (Developer settings).

- **Mac:** keep the server URL `http://localhost:8010/v1` and `tool_choice` `auto`. Set the model name to your `--served-model-name`, for example `KORMo-VL/KORMo-Web-9B`. The default model name `KORMo-VL/KORMo-Web-4B-mlx-4bit` already matches the 4-bit build.
- **NVIDIA GPU:** set the server URL to `http://<server>:8000/v1` (or `localhost`), the model name to `KORMo-Web-9B`, and `tool_choice` to `none`.

**4. Discord (optional).** Run `node relay/discord-relay.mjs` in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.

Full step-by-step instructions (Korean) are in the [extension README](https://github.com/MLP-Lab/KORMo-Web-Extension).

## Quickstart

Tested with **vLLM 0.23.0** and **transformers 5.12.1**. The model runs on a single GPU (TP=1).

### Run the agent

**1. Install and download**

```bash
pip install -U "vllm>=0.23.0" openai playwright huggingface_hub
playwright install chromium
hf download KORMo-VL/KORMo-Web-9B --local-dir KORMo-Web-9B
```

**2. Start the model server**

```bash
vllm serve ./KORMo-Web-9B --served-model-name KORMo-Web-9B \
  --max-model-len 131072 --limit-mm-per-prompt '{"image":1,"video":0}'
```

**3. Run a browser task**

```bash
python KORMo-Web-9B/examples/agent_playwright.py --model KORMo-Web-9B \
  --start-url https://www.danawa.com/ \
  --task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"
```

The agent drives a headless Chromium, prints each action as it goes, and finishes with the model's answer. Add `--headful` to watch the browser window.

**On a Mac** (`vllm-mlx`, see *How to Start* above), point the script at that server and pass `--tool-choice auto`:

```bash
python KORMo-Web-9B/examples/agent_playwright.py --endpoint http://localhost:8010/v1 \
  --model KORMo-VL/KORMo-Web-9B --tool-choice auto --headful \
  --start-url https://www.danawa.com/ \
  --task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"
```

### Inference example (one step)

Send one screenshot with the request and get back one action:

```python
import base64, json
from openai import OpenAI

SYSTEM = open("KORMo-Web-9B/system_prompt.md").read().strip()
TOOLS = json.load(open("KORMo-Web-9B/tools.json"))
shot = base64.b64encode(open("screen.png", "rb").read()).decode()

obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
       "- Tab 0 (active): https://www.google.com/\n\nurl: https://www.google.com/\nscreenshot:\n")
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": [
        {"type": "text", "text": "<user_request>\n๊ตฌ๊ธ€์—์„œ KAIST ๋ฅผ ๊ฒ€์ƒ‰ํ•ด์„œ ๊ณต์‹ ํ™ˆํŽ˜์ด์ง€๋ฅผ ์—ด์–ด์ค˜.\n</user_request>\n" + obs},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}},
        {"type": "text", "text": "\n</observation>"},
    ]},
]
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
out = client.chat.completions.create(
    model="KORMo-Web-9B", messages=messages, tools=TOOLS,
    tool_choice="none",  # vLLM. On vllm-mlx send "auto" and read message.tool_calls (see below)
    temperature=0.6, top_p=0.95, max_tokens=16384,
    extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
)
print(out.choices[0].message.content)
```

The output looks like this (illustrative):

```text
<think>
I can see the Google home page with an empty search box in the center. I have not searched yet.
My plan: 1. Click the search box. 2. Type "KAIST". 3. Press Enter and open the official site.
</think>

<tool_call>
<function=click>
<parameter=x>
500
</parameter>
<parameter=y>
427
</parameter>
</function>
</tool_call>
```

**`tool_choice` depends on the server.** On vLLM, send `"none"`: the tool schema is still rendered into the prompt and the call comes back as text in `message.content`, as above. Sending `tools` without `tool_choice` makes vLLM return HTTP 400 unless the server was started with `--enable-auto-tool-choice`. On vllm-mlx, `"none"` drops the tool schema from the prompt, so send `"auto"`; the call then arrives in `message.tool_calls`. [`examples/agent_playwright.py`](examples/agent_playwright.py) turns it back into the `<tool_call>` text above (`--tool-choice auto`).

Coordinates in the tool call are normalized to 0โ€“1000. Convert them to pixels with `x / 1000 ร— width` and `y / 1000 ร— height`.

## How It Works

```
user       <user_request> task </user_request>  + observation (tabs, url, screenshot)
assistant  <think> what I see ยท what is done ยท plan </think>  <tool_call> one action </tool_call>
tool       action result  + new observation (newest screenshot only)
โ€ฆ          repeat until finished(answer)
```

- **Action space:** 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, `finished`, and others ([`tools.json`](tools.json)).
- **Memory:** the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
- **Recommended settings:** temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in `generation_config.json`.

The exact message format, parsing rules and feedback strings are in **[HARNESS.md](HARNESS.md)**. Follow it if you build your own harness.

## Evaluation Details

| | |
|---|---|
| Tasks | Online-Mind2Web: 300 English, 312 Korean |
| Judge | GPT-6 |

## Files

| File | Description |
|---|---|
| `chat_template.jinja` | Chat template (Qwen3.5 XML tool calls, thinking on) |
| `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema, verbatim from training |
| `HARNESS.md` | Full harness specification |
| `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). `--tool-choice none` for vLLM, `auto` for vllm-mlx; `--headful` shows the browser. It does not execute `upload`. |
| `config.json` ยท `generation_config.json` ยท `processor_config.json` ยท `tokenizer*.json` | Model, sampling and processor configs |

## ํ•œ๊ตญ์–ด ์š”์•ฝ

**KORMo-Web-9B**๋Š” ์‚ฌ๋žŒ์ฒ˜๋Ÿผ ์›น ํ™”๋ฉด์„ ๋ณด๊ณ  ํด๋ฆญยท์ž…๋ ฅยทํƒ์ƒ‰์„ ์ˆ˜ํ–‰ํ•˜๋Š” **9B ๊ทœ๋ชจ์˜ ํ•œ์˜ ์‹œ๊ฐ ๊ธฐ๋ฐ˜ ์›น ์—์ด์ „ํŠธ**์ž…๋‹ˆ๋‹ค.

- **๋Œ€๊ทœ๋ชจ ์›น ์‚ฌ์ „ํ•™์Šต:** 1,000๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ์˜ ์›น ํ•™์Šต ์ƒ˜ํ”Œ๋กœ UI ์š”์†Œ์˜ ์œ„์น˜ยท์˜๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋Šฅ ๋“ฑ์„ ์„ธ๋ฐ€ํ•˜๊ฒŒ ํ•™์Šต(Atomic-Level Pretraining)
- **์‹ค์ œ ์›น ํ–‰๋™ ํ•™์Šต:** 10๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ๊ตญ์ธ ํ•œ์˜ ์›น ์‚ฌ์šฉ ๊ถค์ ์„ ํ•™์Šตํ•ด ์—ฌ๋Ÿฌ ์›น์‚ฌ์ดํŠธ์™€ ํƒญ์„ ๋„˜๋‚˜๋“œ๋Š” ๋ณต์žกํ•œ ์ž‘์—… ์ˆ˜ํ–‰
- **๋›ฐ์–ด๋‚œ ์„ฑ๋Šฅ:** Online-Mind2Web์—์„œ ์˜์–ดยทํ•œ๊ตญ์–ด **51.0% ยท 48.1% ์„ฑ๊ณต๋ฅ **, ๋น„๊ตํ•œ ๊ณต๊ฐœ ๋ชจ๋ธ ์ค‘ ์ตœ๊ณ  ์„ฑ๋Šฅ
- **์‹œ๊ฐ ์ค‘์‹ฌ ์›น ์กฐ์ž‘:** ๋ณ„๋„์˜ DOM ์ •๋ณด ์—†์ด ์Šคํฌ๋ฆฐ์ƒท๋งŒ์œผ๋กœ ์›น์„ ์ดํ•ดํ•˜๊ณ , ํ–‰๋™ ๊ฒฐ๊ณผ๋ฅผ ํ™•์ธํ•˜๋ฉฐ ๋‹ค์Œ ์ž‘์—… ์ˆ˜ํ–‰
- **๋ชจ๋ธยท๋ฐ์ดํ„ฐ ๊ณต๊ฐœ:** Apache-2.0 ๋ชจ๋ธ ๊ฐ€์ค‘์น˜, ๋Œ€๊ทœ๋ชจ ํ•™์Šต ๋ฐ์ดํ„ฐ, ์›น ์‚ฌ์šฉ ๊ถค์ , ํ‰๊ฐ€ ๋ฒค์น˜๋งˆํฌ ๋ฐ ์‹คํ–‰ ์ฝ”๋“œ ๊ณต๊ฐœ
  
## Acknowledgements

Built on **[Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)**. Evaluated on **Online-Mind2Web** and its Korean counterpart.

## Contact
- KyungTae Lim, Professor at KAIST. `ktlim@kaist.ac.kr`

## Contributor
- Minkyung Cho (minkyung.cho@kaist.ac.kr)
- Yerin Nam (yerinnam@kaist.ac.kr)
- Hangyeol Yoo (hg.yoo@kaist.ac.kr)
- Minkyu Kim (minkyukim@kaist.ac.kr)
- Seungwoo Song (sswoo@kaist.ac.kr)
- Gyun Lee (gyunini@kaist.ac.kr)
- KyungTae Lim (ktlim@kaist.ac.kr)
</div>