KORMo-Web-9B / README.md
jungsin3's picture
Quickstart works on vLLM and vllm-mlx: tool_choice, tool_calls restore, Mac 4B/9B serving + memory, fixes
ca44759 verified
|
Raw History Blame Contribute Delete
15.2 kB
---
license: apache-2.0
base_model:
- Qwen/Qwen3.5-9B
language:
- ko
- en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- gui-agent
- web-agent
- computer-use
- browser
- korean
---
<style>
#kormo-web-card { font-size: 16px; line-height: 1.6; overflow-wrap: break-word; }
#kormo-web-card h1 { margin: 0 0 12px; font-size: 30px; line-height: 1.25; letter-spacing: -0.018em; }
#kormo-web-card h2 { margin: 30px 0 10px; font-size: 22px; line-height: 1.3; letter-spacing: -0.012em; }
#kormo-web-card h3 { margin: 22px 0 8px; font-size: 17px; font-weight: 700; line-height: 1.4; }
#kormo-web-card h2 + h3 { margin-top: 14px; }
#kormo-web-card p { margin: 0 0 10px; }
#kormo-web-card ul, #kormo-web-card ol { margin: 6px 0 12px; padding-left: 22px; }
#kormo-web-card li { margin: 4px 0; line-height: 1.55; }
#kormo-web-card table { width: 100%; margin: 6px 0 14px; border-collapse: collapse; font-size: 14px; line-height: 1.45; font-variant-numeric: tabular-nums; }
#kormo-web-card th, #kormo-web-card td { padding: 7px 10px; vertical-align: middle; }
#kormo-web-card blockquote { margin: 12px 0 16px; padding: 0 0 0 14px; font-size: 15px; }
#kormo-web-card blockquote p { margin: 0; }
#kormo-web-card pre { margin: 10px 0 14px; padding: 14px 16px; overflow-x: auto; line-height: 1.5; }
#kormo-web-card pre code { font-size: 13px; }
#kormo-web-card > p:has(> em:only-child) { margin: 4px 0 14px; font-size: 13px; }
#kormo-web-card .pills { margin: 4px 0 18px; }
#kormo-web-card .pill { display: inline-block; margin: 0 6px 6px 0; padding: 3px 10px; border-radius: 999px; border: 1px solid rgba(127,127,127,.35); font-size: 13px; }
@media (max-width: 640px) {
#kormo-web-card h1 { font-size: 26px; }
#kormo-web-card table { font-size: 13px; }
#kormo-web-card th, #kormo-web-card td { padding: 7px 6px; }
}
</style>
<div id="kormo-web-card">
<video src="https://huggingface.co/KORMo-VL/KORMo-Web-9B/resolve/main/assets/gui_final_demo_masked.mp4" controls autoplay muted loop playsinline width="100%"></video>
# KORMo-Web-9B
**KORMo-Web-9B** is a 9B vision-language agent that completes real browser tasks **from screenshots alone**. Give it a request in Korean or English; it plans, clicks, types and moves between sites and tabs until the task is done. In the same live-web evaluation, it beats every open-weight web agent we compared, **by +18.3 points in English and +27.9 points in Korean**.
On **Online-Mind2Web** it reaches **51.0% in English** and **48.1% in Korean**, which is **+26.3 / +31.4 points** over the base Qwen3.5-9B.
## What Makes KORMo-Web Different?
- **10M+ Atomic-Level Web Pretraining Samples.** Large-scale Korean and English web understanding pretraining teaches the model what individual UI elements **are, where they are, and what they do**โ€”covering text, location, appearance, layout, function, and state.
- **100K+ Web Interaction Trajectories.** Bilingual browser demonstrations teach the model to translate fine-grained web understanding into real-world, multi-step actions.
- **Open Models and Data.** Releasing model weights, large-scale bilingual web pretraining data, interaction trajectories, and evaluation benchmarks to support reproducible web agent research.
## Performance
All models below ran **the same 612 live-website tasks** (300 English, 312 Korean), each with its own prompt and action format, and were scored by **the same LLM judge**.
| Model | Size | Online-Mind2Web EN | Online-Mind2Web KO |
|---|---:|---:|---:|
| Qwen3.5-9B (base, no fine-tuning) | 9B | 24.7 | 16.7 |
| Fara-1.5-4B | 4B | 20.7 | 7.7 |
| UI-TARS-1.5-7B | 7B | 22.3 | 20.2 |
| OpenWebRL-4B | 4B | 32.7 | 15.4 |
| **KORMo-Web-9B (ours)** | **9B** | **51.0** | **48.1** |
| GPT-6-luna (proprietary) | โ€“ | 63.0 | 65.7 |
Success rate (%). The denominator is always the full task count, so crashed or empty runs count as failures.
KORMo-Web-9B outperforms open web agents of similar size in both English and Korean.
## Why KORMo-Web
- **Korean and English, natural requests.** Handles short commands as well as long, conversational instructions in either language.
- **Screenshot-only.** It needs no DOM or accessibility tree; it acts on what is on screen, like a person, and checks each action's result on the next screenshot.
- **Thinks, then acts.** A short reasoning step precedes every action, so its behaviour is easy to inspect and debug.
- **Multi-site, multi-tab workflows.** It can search on one site, verify on another and write the result into a third, managing tabs on its own.
- **Open and ready to run.** Apache-2.0 weights, a reference browser harness, the tool schema and the exact inference settings are all included.
## How to Start: Chrome Extension
The fastest way to try KORMo-Web is the **[KORMo-Web Chrome extension](https://github.com/MLP-Lab/KORMo-Web-Extension)**: type a request in Chrome's side panel and watch the agent work in your own browser. This video walks through the whole setup, from serving the model to running a task and controlling it from Discord.
<video src="https://huggingface.co/KORMo-VL/KORMo-Web-9B/resolve/main/assets/setup_guide.mp4" controls playsinline preload="metadata" width="100%"></video>
*Setup guide (about 15 min, Korean narration). Parts that showed account tokens have been blurred.*
**1. Serve the model** on a Mac or on an NVIDIA GPU.
### Mac (Apple silicon)
KORMo-Web-4B (4-bit or full precision) and KORMo-Web-9B all run on a Mac with `vllm-mlx` 0.6.0 or later. Pick the one that fits your Mac's memory:
| Model | Weights | Memory in use | Recommended Mac memory |
|---|---:|---:|---|
| `KORMo-Web-4B-mlx-4bit` (4-bit) | 2.8 GB | about 3.8 GB | 16 GB or more |
| `KORMo-Web-4B` (bf16) | 9.6 GB | about 9.5 GB | 24 GB or more |
| `KORMo-Web-9B` (bf16) | 17.5 GB | about 19 GB | 48 GB or more (32 GB is tight) |
*Measured with vllm-mlx 0.6.0 after a few short requests (macOS units, 1 GB = 1024ยณ bytes). Long episodes add KV cache, about 1 GB per 30k tokens, and Chrome needs memory too. macOS lets the GPU use only about 2/3โ€“3/4 of RAM.*
```bash
uv venv --python 3.12 && source .venv/bin/activate
uv pip install "vllm-mlx>=0.6.0" huggingface_hub
export HF_HUB_DISABLE_XET=1 # only if large downloads stall
MODEL=KORMo-Web-9B # or KORMo-Web-4B / KORMo-Web-4B-mlx-4bit
hf download KORMo-VL/$MODEL --local-dir ~/models/$MODEL
vllm-mlx serve ~/models/$MODEL --port 8010 --served-model-name KORMo-VL/$MODEL \
--enable-auto-tool-choice --tool-call-parser auto
```
Keep this terminal open. `--served-model-name` is the name clients must request; without it the server registers the local folder path and requests by model name return 404. vllm-mlx 0.6.0 and later detect the vision model by itself; older versions need `--mllm`.
### NVIDIA GPU (local PC or server)
Serve the model with vLLM. Much faster, and one server can be shared.
```bash
pip install vllm
vllm serve KORMo-VL/KORMo-Web-9B --host 0.0.0.0 --port 8000 \
--served-model-name KORMo-Web-9B \
--mm-encoder-tp-mode data --mm-processor-cache-type shm \
--gdn-prefill-backend triton
```
Check that it is up with `curl http://localhost:8010/v1/models` (Mac) or `curl http://<server>:8000/v1/models` (vLLM). If you changed the port, use that port.
**2. Install the extension.** `git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git`, open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked** and choose the cloned folder. Open the **GUI ์—์ด์ „ํŠธ** side panel, type a task and press **์‹คํ–‰** (Run).
**3. Point the extension at your model.** Open the panel's **๊ฐœ๋ฐœ์ž ์„ค์ •** (Developer settings).
- **Mac:** keep the server URL `http://localhost:8010/v1` and `tool_choice` `auto`. Set the model name to your `--served-model-name`, for example `KORMo-VL/KORMo-Web-9B`. The default model name `KORMo-VL/KORMo-Web-4B-mlx-4bit` already matches the 4-bit build.
- **NVIDIA GPU:** set the server URL to `http://<server>:8000/v1` (or `localhost`), the model name to `KORMo-Web-9B`, and `tool_choice` to `none`.
**4. Discord (optional).** Run `node relay/discord-relay.mjs` in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.
Full step-by-step instructions (Korean) are in the [extension README](https://github.com/MLP-Lab/KORMo-Web-Extension).
## Quickstart
Tested with **vLLM 0.23.0** and **transformers 5.12.1**. The model runs on a single GPU (TP=1).
### Run the agent
**1. Install and download**
```bash
pip install -U "vllm>=0.23.0" openai playwright huggingface_hub
playwright install chromium
hf download KORMo-VL/KORMo-Web-9B --local-dir KORMo-Web-9B
```
**2. Start the model server**
```bash
vllm serve ./KORMo-Web-9B --served-model-name KORMo-Web-9B \
--max-model-len 131072 --limit-mm-per-prompt '{"image":1,"video":0}'
```
**3. Run a browser task**
```bash
python KORMo-Web-9B/examples/agent_playwright.py --model KORMo-Web-9B \
--start-url https://www.danawa.com/ \
--task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"
```
The agent drives a headless Chromium, prints each action as it goes, and finishes with the model's answer. Add `--headful` to watch the browser window.
**On a Mac** (`vllm-mlx`, see *How to Start* above), point the script at that server and pass `--tool-choice auto`:
```bash
python KORMo-Web-9B/examples/agent_playwright.py --endpoint http://localhost:8010/v1 \
--model KORMo-VL/KORMo-Web-9B --tool-choice auto --headful \
--start-url https://www.danawa.com/ \
--task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"
```
### Inference example (one step)
Send one screenshot with the request and get back one action:
```python
import base64, json
from openai import OpenAI
SYSTEM = open("KORMo-Web-9B/system_prompt.md").read().strip()
TOOLS = json.load(open("KORMo-Web-9B/tools.json"))
shot = base64.b64encode(open("screen.png", "rb").read()).decode()
obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
"- Tab 0 (active): https://www.google.com/\n\nurl: https://www.google.com/\nscreenshot:\n")
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": [
{"type": "text", "text": "<user_request>\n๊ตฌ๊ธ€์—์„œ KAIST ๋ฅผ ๊ฒ€์ƒ‰ํ•ด์„œ ๊ณต์‹ ํ™ˆํŽ˜์ด์ง€๋ฅผ ์—ด์–ด์ค˜.\n</user_request>\n" + obs},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}},
{"type": "text", "text": "\n</observation>"},
]},
]
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
out = client.chat.completions.create(
model="KORMo-Web-9B", messages=messages, tools=TOOLS,
tool_choice="none", # vLLM. On vllm-mlx send "auto" and read message.tool_calls (see below)
temperature=0.6, top_p=0.95, max_tokens=16384,
extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
)
print(out.choices[0].message.content)
```
The output looks like this (illustrative):
```text
<think>
I can see the Google home page with an empty search box in the center. I have not searched yet.
My plan: 1. Click the search box. 2. Type "KAIST". 3. Press Enter and open the official site.
</think>
<tool_call>
<function=click>
<parameter=x>
500
</parameter>
<parameter=y>
427
</parameter>
</function>
</tool_call>
```
**`tool_choice` depends on the server.** On vLLM, send `"none"`: the tool schema is still rendered into the prompt and the call comes back as text in `message.content`, as above. Sending `tools` without `tool_choice` makes vLLM return HTTP 400 unless the server was started with `--enable-auto-tool-choice`. On vllm-mlx, `"none"` drops the tool schema from the prompt, so send `"auto"`; the call then arrives in `message.tool_calls`. [`examples/agent_playwright.py`](examples/agent_playwright.py) turns it back into the `<tool_call>` text above (`--tool-choice auto`).
Coordinates in the tool call are normalized to 0โ€“1000. Convert them to pixels with `x / 1000 ร— width` and `y / 1000 ร— height`.
## How It Works
```
user <user_request> task </user_request> + observation (tabs, url, screenshot)
assistant <think> what I see ยท what is done ยท plan </think> <tool_call> one action </tool_call>
tool action result + new observation (newest screenshot only)
โ€ฆ repeat until finished(answer)
```
- **Action space:** 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, `finished`, and others ([`tools.json`](tools.json)).
- **Memory:** the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
- **Recommended settings:** temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in `generation_config.json`.
The exact message format, parsing rules and feedback strings are in **[HARNESS.md](HARNESS.md)**. Follow it if you build your own harness.
## Evaluation Details
| | |
|---|---|
| Tasks | Online-Mind2Web: 300 English, 312 Korean |
| Judge | GPT-6 |
## Files
| File | Description |
|---|---|
| `chat_template.jinja` | Chat template (Qwen3.5 XML tool calls, thinking on) |
| `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema, verbatim from training |
| `HARNESS.md` | Full harness specification |
| `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). `--tool-choice none` for vLLM, `auto` for vllm-mlx; `--headful` shows the browser. It does not execute `upload`. |
| `config.json` ยท `generation_config.json` ยท `processor_config.json` ยท `tokenizer*.json` | Model, sampling and processor configs |
## ํ•œ๊ตญ์–ด ์š”์•ฝ
**KORMo-Web-9B**๋Š” ์‚ฌ๋žŒ์ฒ˜๋Ÿผ ์›น ํ™”๋ฉด์„ ๋ณด๊ณ  ํด๋ฆญยท์ž…๋ ฅยทํƒ์ƒ‰์„ ์ˆ˜ํ–‰ํ•˜๋Š” **9B ๊ทœ๋ชจ์˜ ํ•œ์˜ ์‹œ๊ฐ ๊ธฐ๋ฐ˜ ์›น ์—์ด์ „ํŠธ**์ž…๋‹ˆ๋‹ค.
- **๋Œ€๊ทœ๋ชจ ์›น ์‚ฌ์ „ํ•™์Šต:** 1,000๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ์˜ ์›น ํ•™์Šต ์ƒ˜ํ”Œ๋กœ UI ์š”์†Œ์˜ ์œ„์น˜ยท์˜๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋Šฅ ๋“ฑ์„ ์„ธ๋ฐ€ํ•˜๊ฒŒ ํ•™์Šต(Atomic-Level Pretraining)
- **์‹ค์ œ ์›น ํ–‰๋™ ํ•™์Šต:** 10๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ๊ตญ์ธ ํ•œ์˜ ์›น ์‚ฌ์šฉ ๊ถค์ ์„ ํ•™์Šตํ•ด ์—ฌ๋Ÿฌ ์›น์‚ฌ์ดํŠธ์™€ ํƒญ์„ ๋„˜๋‚˜๋“œ๋Š” ๋ณต์žกํ•œ ์ž‘์—… ์ˆ˜ํ–‰
- **๋›ฐ์–ด๋‚œ ์„ฑ๋Šฅ:** Online-Mind2Web์—์„œ ์˜์–ดยทํ•œ๊ตญ์–ด **51.0% ยท 48.1% ์„ฑ๊ณต๋ฅ **, ๋น„๊ตํ•œ ๊ณต๊ฐœ ๋ชจ๋ธ ์ค‘ ์ตœ๊ณ  ์„ฑ๋Šฅ
- **์‹œ๊ฐ ์ค‘์‹ฌ ์›น ์กฐ์ž‘:** ๋ณ„๋„์˜ DOM ์ •๋ณด ์—†์ด ์Šคํฌ๋ฆฐ์ƒท๋งŒ์œผ๋กœ ์›น์„ ์ดํ•ดํ•˜๊ณ , ํ–‰๋™ ๊ฒฐ๊ณผ๋ฅผ ํ™•์ธํ•˜๋ฉฐ ๋‹ค์Œ ์ž‘์—… ์ˆ˜ํ–‰
- **๋ชจ๋ธยท๋ฐ์ดํ„ฐ ๊ณต๊ฐœ:** Apache-2.0 ๋ชจ๋ธ ๊ฐ€์ค‘์น˜, ๋Œ€๊ทœ๋ชจ ํ•™์Šต ๋ฐ์ดํ„ฐ, ์›น ์‚ฌ์šฉ ๊ถค์ , ํ‰๊ฐ€ ๋ฒค์น˜๋งˆํฌ ๋ฐ ์‹คํ–‰ ์ฝ”๋“œ ๊ณต๊ฐœ
## Acknowledgements
Built on **[Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B)**. Evaluated on **Online-Mind2Web** and its Korean counterpart.
## Contact
- KyungTae Lim, Professor at KAIST. `ktlim@kaist.ac.kr`
## Contributor
- Minkyung Cho (minkyung.cho@kaist.ac.kr)
- Yerin Nam (yerinnam@kaist.ac.kr)
- Hangyeol Yoo (hg.yoo@kaist.ac.kr)
- Minkyu Kim (minkyukim@kaist.ac.kr)
- Seungwoo Song (sswoo@kaist.ac.kr)
- Gyun Lee (gyunini@kaist.ac.kr)
- KyungTae Lim (ktlim@kaist.ac.kr)
</div>