KORMo-Web-9B / README.md
jungsin3's picture
Quickstart works on vLLM and vllm-mlx: tool_choice, tool_calls restore, Mac 4B/9B serving + memory, fixes
ca44759 verified
|
Raw History Blame Contribute Delete
15.2 kB
metadata
license: apache-2.0
base_model:
  - Qwen/Qwen3.5-9B
language:
  - ko
  - en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
  - gui-agent
  - web-agent
  - computer-use
  - browser
  - korean

KORMo-Web-9B

KORMo-Web-9B is a 9B vision-language agent that completes real browser tasks from screenshots alone. Give it a request in Korean or English; it plans, clicks, types and moves between sites and tabs until the task is done. In the same live-web evaluation, it beats every open-weight web agent we compared, by +18.3 points in English and +27.9 points in Korean.

On Online-Mind2Web it reaches 51.0% in English and 48.1% in Korean, which is +26.3 / +31.4 points over the base Qwen3.5-9B.

What Makes KORMo-Web Different?

  • 10M+ Atomic-Level Web Pretraining Samples. Large-scale Korean and English web understanding pretraining teaches the model what individual UI elements are, where they are, and what they doโ€”covering text, location, appearance, layout, function, and state.
  • 100K+ Web Interaction Trajectories. Bilingual browser demonstrations teach the model to translate fine-grained web understanding into real-world, multi-step actions.
  • Open Models and Data. Releasing model weights, large-scale bilingual web pretraining data, interaction trajectories, and evaluation benchmarks to support reproducible web agent research.

Performance

All models below ran the same 612 live-website tasks (300 English, 312 Korean), each with its own prompt and action format, and were scored by the same LLM judge.

Model Size Online-Mind2Web EN Online-Mind2Web KO
Qwen3.5-9B (base, no fine-tuning) 9B 24.7 16.7
Fara-1.5-4B 4B 20.7 7.7
UI-TARS-1.5-7B 7B 22.3 20.2
OpenWebRL-4B 4B 32.7 15.4
KORMo-Web-9B (ours) 9B 51.0 48.1
GPT-6-luna (proprietary) โ€“ 63.0 65.7

Success rate (%). The denominator is always the full task count, so crashed or empty runs count as failures.

KORMo-Web-9B outperforms open web agents of similar size in both English and Korean.

Why KORMo-Web

  • Korean and English, natural requests. Handles short commands as well as long, conversational instructions in either language.
  • Screenshot-only. It needs no DOM or accessibility tree; it acts on what is on screen, like a person, and checks each action's result on the next screenshot.
  • Thinks, then acts. A short reasoning step precedes every action, so its behaviour is easy to inspect and debug.
  • Multi-site, multi-tab workflows. It can search on one site, verify on another and write the result into a third, managing tabs on its own.
  • Open and ready to run. Apache-2.0 weights, a reference browser harness, the tool schema and the exact inference settings are all included.

How to Start: Chrome Extension

The fastest way to try KORMo-Web is the KORMo-Web Chrome extension: type a request in Chrome's side panel and watch the agent work in your own browser. This video walks through the whole setup, from serving the model to running a task and controlling it from Discord.

Setup guide (about 15 min, Korean narration). Parts that showed account tokens have been blurred.

1. Serve the model on a Mac or on an NVIDIA GPU.

Mac (Apple silicon)

KORMo-Web-4B (4-bit or full precision) and KORMo-Web-9B all run on a Mac with vllm-mlx 0.6.0 or later. Pick the one that fits your Mac's memory:

Model Weights Memory in use Recommended Mac memory
KORMo-Web-4B-mlx-4bit (4-bit) 2.8 GB about 3.8 GB 16 GB or more
KORMo-Web-4B (bf16) 9.6 GB about 9.5 GB 24 GB or more
KORMo-Web-9B (bf16) 17.5 GB about 19 GB 48 GB or more (32 GB is tight)

Measured with vllm-mlx 0.6.0 after a few short requests (macOS units, 1 GB = 1024ยณ bytes). Long episodes add KV cache, about 1 GB per 30k tokens, and Chrome needs memory too. macOS lets the GPU use only about 2/3โ€“3/4 of RAM.

uv venv --python 3.12 && source .venv/bin/activate
uv pip install "vllm-mlx>=0.6.0" huggingface_hub
export HF_HUB_DISABLE_XET=1    # only if large downloads stall

MODEL=KORMo-Web-9B             # or KORMo-Web-4B / KORMo-Web-4B-mlx-4bit
hf download KORMo-VL/$MODEL --local-dir ~/models/$MODEL
vllm-mlx serve ~/models/$MODEL --port 8010 --served-model-name KORMo-VL/$MODEL \
  --enable-auto-tool-choice --tool-call-parser auto

Keep this terminal open. --served-model-name is the name clients must request; without it the server registers the local folder path and requests by model name return 404. vllm-mlx 0.6.0 and later detect the vision model by itself; older versions need --mllm.

NVIDIA GPU (local PC or server)

Serve the model with vLLM. Much faster, and one server can be shared.

pip install vllm
vllm serve KORMo-VL/KORMo-Web-9B --host 0.0.0.0 --port 8000 \
  --served-model-name KORMo-Web-9B \
  --mm-encoder-tp-mode data --mm-processor-cache-type shm \
  --gdn-prefill-backend triton

Check that it is up with curl http://localhost:8010/v1/models (Mac) or curl http://<server>:8000/v1/models (vLLM). If you changed the port, use that port.

2. Install the extension. git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git, open chrome://extensions, turn on Developer mode, click Load unpacked and choose the cloned folder. Open the GUI ์—์ด์ „ํŠธ side panel, type a task and press ์‹คํ–‰ (Run).

3. Point the extension at your model. Open the panel's ๊ฐœ๋ฐœ์ž ์„ค์ • (Developer settings).

  • Mac: keep the server URL http://localhost:8010/v1 and tool_choice auto. Set the model name to your --served-model-name, for example KORMo-VL/KORMo-Web-9B. The default model name KORMo-VL/KORMo-Web-4B-mlx-4bit already matches the 4-bit build.
  • NVIDIA GPU: set the server URL to http://<server>:8000/v1 (or localhost), the model name to KORMo-Web-9B, and tool_choice to none.

4. Discord (optional). Run node relay/discord-relay.mjs in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.

Full step-by-step instructions (Korean) are in the extension README.

Quickstart

Tested with vLLM 0.23.0 and transformers 5.12.1. The model runs on a single GPU (TP=1).

Run the agent

1. Install and download

pip install -U "vllm>=0.23.0" openai playwright huggingface_hub
playwright install chromium
hf download KORMo-VL/KORMo-Web-9B --local-dir KORMo-Web-9B

2. Start the model server

vllm serve ./KORMo-Web-9B --served-model-name KORMo-Web-9B \
  --max-model-len 131072 --limit-mm-per-prompt '{"image":1,"video":0}'

3. Run a browser task

python KORMo-Web-9B/examples/agent_playwright.py --model KORMo-Web-9B \
  --start-url https://www.danawa.com/ \
  --task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"

The agent drives a headless Chromium, prints each action as it goes, and finishes with the model's answer. Add --headful to watch the browser window.

On a Mac (vllm-mlx, see How to Start above), point the script at that server and pass --tool-choice auto:

python KORMo-Web-9B/examples/agent_playwright.py --endpoint http://localhost:8010/v1 \
  --model KORMo-VL/KORMo-Web-9B --tool-choice auto --headful \
  --start-url https://www.danawa.com/ \
  --task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"

Inference example (one step)

Send one screenshot with the request and get back one action:

import base64, json
from openai import OpenAI

SYSTEM = open("KORMo-Web-9B/system_prompt.md").read().strip()
TOOLS = json.load(open("KORMo-Web-9B/tools.json"))
shot = base64.b64encode(open("screen.png", "rb").read()).decode()

obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
       "- Tab 0 (active): https://www.google.com/\n\nurl: https://www.google.com/\nscreenshot:\n")
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": [
        {"type": "text", "text": "<user_request>\n๊ตฌ๊ธ€์—์„œ KAIST ๋ฅผ ๊ฒ€์ƒ‰ํ•ด์„œ ๊ณต์‹ ํ™ˆํŽ˜์ด์ง€๋ฅผ ์—ด์–ด์ค˜.\n</user_request>\n" + obs},
        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}},
        {"type": "text", "text": "\n</observation>"},
    ]},
]
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
out = client.chat.completions.create(
    model="KORMo-Web-9B", messages=messages, tools=TOOLS,
    tool_choice="none",  # vLLM. On vllm-mlx send "auto" and read message.tool_calls (see below)
    temperature=0.6, top_p=0.95, max_tokens=16384,
    extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
)
print(out.choices[0].message.content)

The output looks like this (illustrative):

<think>
I can see the Google home page with an empty search box in the center. I have not searched yet.
My plan: 1. Click the search box. 2. Type "KAIST". 3. Press Enter and open the official site.
</think>

<tool_call>
<function=click>
<parameter=x>
500
</parameter>
<parameter=y>
427
</parameter>
</function>
</tool_call>

tool_choice depends on the server. On vLLM, send "none": the tool schema is still rendered into the prompt and the call comes back as text in message.content, as above. Sending tools without tool_choice makes vLLM return HTTP 400 unless the server was started with --enable-auto-tool-choice. On vllm-mlx, "none" drops the tool schema from the prompt, so send "auto"; the call then arrives in message.tool_calls. examples/agent_playwright.py turns it back into the <tool_call> text above (--tool-choice auto).

Coordinates in the tool call are normalized to 0โ€“1000. Convert them to pixels with x / 1000 ร— width and y / 1000 ร— height.

How It Works

user       <user_request> task </user_request>  + observation (tabs, url, screenshot)
assistant  <think> what I see ยท what is done ยท plan </think>  <tool_call> one action </tool_call>
tool       action result  + new observation (newest screenshot only)
โ€ฆ          repeat until finished(answer)
  • Action space: 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, finished, and others (tools.json).
  • Memory: the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
  • Recommended settings: temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in generation_config.json.

The exact message format, parsing rules and feedback strings are in HARNESS.md. Follow it if you build your own harness.

Evaluation Details

Tasks Online-Mind2Web: 300 English, 312 Korean
Judge GPT-6

Files

File Description
chat_template.jinja Chat template (Qwen3.5 XML tool calls, thinking on)
system_prompt.md ยท tools.json System prompt and 22-tool schema, verbatim from training
HARNESS.md Full harness specification
examples/agent_playwright.py Reference browser agent (Playwright + OpenAI-compatible API). --tool-choice none for vLLM, auto for vllm-mlx; --headful shows the browser. It does not execute upload.
config.json ยท generation_config.json ยท processor_config.json ยท tokenizer*.json Model, sampling and processor configs

ํ•œ๊ตญ์–ด ์š”์•ฝ

KORMo-Web-9B๋Š” ์‚ฌ๋žŒ์ฒ˜๋Ÿผ ์›น ํ™”๋ฉด์„ ๋ณด๊ณ  ํด๋ฆญยท์ž…๋ ฅยทํƒ์ƒ‰์„ ์ˆ˜ํ–‰ํ•˜๋Š” 9B ๊ทœ๋ชจ์˜ ํ•œ์˜ ์‹œ๊ฐ ๊ธฐ๋ฐ˜ ์›น ์—์ด์ „ํŠธ์ž…๋‹ˆ๋‹ค.

  • ๋Œ€๊ทœ๋ชจ ์›น ์‚ฌ์ „ํ•™์Šต: 1,000๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ์˜ ์›น ํ•™์Šต ์ƒ˜ํ”Œ๋กœ UI ์š”์†Œ์˜ ์œ„์น˜ยท์˜๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋Šฅ ๋“ฑ์„ ์„ธ๋ฐ€ํ•˜๊ฒŒ ํ•™์Šต(Atomic-Level Pretraining)
  • ์‹ค์ œ ์›น ํ–‰๋™ ํ•™์Šต: 10๋งŒ ๊ฑด ์ด์ƒ์˜ ํ•œ๊ตญ์ธ ํ•œ์˜ ์›น ์‚ฌ์šฉ ๊ถค์ ์„ ํ•™์Šตํ•ด ์—ฌ๋Ÿฌ ์›น์‚ฌ์ดํŠธ์™€ ํƒญ์„ ๋„˜๋‚˜๋“œ๋Š” ๋ณต์žกํ•œ ์ž‘์—… ์ˆ˜ํ–‰
  • ๋›ฐ์–ด๋‚œ ์„ฑ๋Šฅ: Online-Mind2Web์—์„œ ์˜์–ดยทํ•œ๊ตญ์–ด 51.0% ยท 48.1% ์„ฑ๊ณต๋ฅ , ๋น„๊ตํ•œ ๊ณต๊ฐœ ๋ชจ๋ธ ์ค‘ ์ตœ๊ณ  ์„ฑ๋Šฅ
  • ์‹œ๊ฐ ์ค‘์‹ฌ ์›น ์กฐ์ž‘: ๋ณ„๋„์˜ DOM ์ •๋ณด ์—†์ด ์Šคํฌ๋ฆฐ์ƒท๋งŒ์œผ๋กœ ์›น์„ ์ดํ•ดํ•˜๊ณ , ํ–‰๋™ ๊ฒฐ๊ณผ๋ฅผ ํ™•์ธํ•˜๋ฉฐ ๋‹ค์Œ ์ž‘์—… ์ˆ˜ํ–‰
  • ๋ชจ๋ธยท๋ฐ์ดํ„ฐ ๊ณต๊ฐœ: Apache-2.0 ๋ชจ๋ธ ๊ฐ€์ค‘์น˜, ๋Œ€๊ทœ๋ชจ ํ•™์Šต ๋ฐ์ดํ„ฐ, ์›น ์‚ฌ์šฉ ๊ถค์ , ํ‰๊ฐ€ ๋ฒค์น˜๋งˆํฌ ๋ฐ ์‹คํ–‰ ์ฝ”๋“œ ๊ณต๊ฐœ

Acknowledgements

Built on Qwen3.5-9B. Evaluated on Online-Mind2Web and its Korean counterpart.

Contact

  • KyungTae Lim, Professor at KAIST. ktlim@kaist.ac.kr

Contributor