Instructions to use KORMo-VL/KORMo-Web-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KORMo-VL/KORMo-Web-9B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="KORMo-VL/KORMo-Web-9B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("KORMo-VL/KORMo-Web-9B") model = AutoModelForMultimodalLM.from_pretrained("KORMo-VL/KORMo-Web-9B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use KORMo-VL/KORMo-Web-9B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KORMo-VL/KORMo-Web-9B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/KORMo-VL/KORMo-Web-9B
- SGLang
How to use KORMo-VL/KORMo-Web-9B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-9B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-9B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use KORMo-VL/KORMo-Web-9B with Docker Model Runner:
docker model run hf.co/KORMo-VL/KORMo-Web-9B
Download README.md from KORMo-VL/KORMo-Web-9B: direct link, hf CLI and curl.
- Browser
- Download file 15.2 kB
-
https://huggingface.co/KORMo-VL/KORMo-Web-9B/resolve/main/README.md
- Command line
-
hf download hf://KORMo-VL/KORMo-Web-9B/README.md
-
curl -L -o README.md https://huggingface.co/KORMo-VL/KORMo-Web-9B/resolve/main/README.md
license: apache-2.0
base_model:
- Qwen/Qwen3.5-9B
language:
- ko
- en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- gui-agent
- web-agent
- computer-use
- browser
- korean
KORMo-Web-9B
KORMo-Web-9B is a 9B vision-language agent that completes real browser tasks from screenshots alone. Give it a request in Korean or English; it plans, clicks, types and moves between sites and tabs until the task is done. In the same live-web evaluation, it beats every open-weight web agent we compared, by +18.3 points in English and +27.9 points in Korean.
On Online-Mind2Web it reaches 51.0% in English and 48.1% in Korean, which is +26.3 / +31.4 points over the base Qwen3.5-9B.
What Makes KORMo-Web Different?
- 10M+ Atomic-Level Web Pretraining Samples. Large-scale Korean and English web understanding pretraining teaches the model what individual UI elements are, where they are, and what they doโcovering text, location, appearance, layout, function, and state.
- 100K+ Web Interaction Trajectories. Bilingual browser demonstrations teach the model to translate fine-grained web understanding into real-world, multi-step actions.
- Open Models and Data. Releasing model weights, large-scale bilingual web pretraining data, interaction trajectories, and evaluation benchmarks to support reproducible web agent research.
Performance
All models below ran the same 612 live-website tasks (300 English, 312 Korean), each with its own prompt and action format, and were scored by the same LLM judge.
| Model | Size | Online-Mind2Web EN | Online-Mind2Web KO |
|---|---|---|---|
| Qwen3.5-9B (base, no fine-tuning) | 9B | 24.7 | 16.7 |
| Fara-1.5-4B | 4B | 20.7 | 7.7 |
| UI-TARS-1.5-7B | 7B | 22.3 | 20.2 |
| OpenWebRL-4B | 4B | 32.7 | 15.4 |
| KORMo-Web-9B (ours) | 9B | 51.0 | 48.1 |
| GPT-6-luna (proprietary) | โ | 63.0 | 65.7 |
Success rate (%). The denominator is always the full task count, so crashed or empty runs count as failures.
KORMo-Web-9B outperforms open web agents of similar size in both English and Korean.
Why KORMo-Web
- Korean and English, natural requests. Handles short commands as well as long, conversational instructions in either language.
- Screenshot-only. It needs no DOM or accessibility tree; it acts on what is on screen, like a person, and checks each action's result on the next screenshot.
- Thinks, then acts. A short reasoning step precedes every action, so its behaviour is easy to inspect and debug.
- Multi-site, multi-tab workflows. It can search on one site, verify on another and write the result into a third, managing tabs on its own.
- Open and ready to run. Apache-2.0 weights, a reference browser harness, the tool schema and the exact inference settings are all included.
How to Start: Chrome Extension
The fastest way to try KORMo-Web is the KORMo-Web Chrome extension: type a request in Chrome's side panel and watch the agent work in your own browser. This video walks through the whole setup, from serving the model to running a task and controlling it from Discord.
Setup guide (about 15 min, Korean narration). Parts that showed account tokens have been blurred.
1. Serve the model on a Mac or on an NVIDIA GPU.
Mac (Apple silicon)
KORMo-Web-4B (4-bit or full precision) and KORMo-Web-9B all run on a Mac with vllm-mlx 0.6.0 or later. Pick the one that fits your Mac's memory:
| Model | Weights | Memory in use | Recommended Mac memory |
|---|---|---|---|
KORMo-Web-4B-mlx-4bit (4-bit) |
2.8 GB | about 3.8 GB | 16 GB or more |
KORMo-Web-4B (bf16) |
9.6 GB | about 9.5 GB | 24 GB or more |
KORMo-Web-9B (bf16) |
17.5 GB | about 19 GB | 48 GB or more (32 GB is tight) |
Measured with vllm-mlx 0.6.0 after a few short requests (macOS units, 1 GB = 1024ยณ bytes). Long episodes add KV cache, about 1 GB per 30k tokens, and Chrome needs memory too. macOS lets the GPU use only about 2/3โ3/4 of RAM.
uv venv --python 3.12 && source .venv/bin/activate
uv pip install "vllm-mlx>=0.6.0" huggingface_hub
export HF_HUB_DISABLE_XET=1 # only if large downloads stall
MODEL=KORMo-Web-9B # or KORMo-Web-4B / KORMo-Web-4B-mlx-4bit
hf download KORMo-VL/$MODEL --local-dir ~/models/$MODEL
vllm-mlx serve ~/models/$MODEL --port 8010 --served-model-name KORMo-VL/$MODEL \
--enable-auto-tool-choice --tool-call-parser auto
Keep this terminal open. --served-model-name is the name clients must request; without it the server registers the local folder path and requests by model name return 404. vllm-mlx 0.6.0 and later detect the vision model by itself; older versions need --mllm.
NVIDIA GPU (local PC or server)
Serve the model with vLLM. Much faster, and one server can be shared.
pip install vllm
vllm serve KORMo-VL/KORMo-Web-9B --host 0.0.0.0 --port 8000 \
--served-model-name KORMo-Web-9B \
--mm-encoder-tp-mode data --mm-processor-cache-type shm \
--gdn-prefill-backend triton
Check that it is up with curl http://localhost:8010/v1/models (Mac) or curl http://<server>:8000/v1/models (vLLM). If you changed the port, use that port.
2. Install the extension. git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git, open chrome://extensions, turn on Developer mode, click Load unpacked and choose the cloned folder. Open the GUI ์์ด์ ํธ side panel, type a task and press ์คํ (Run).
3. Point the extension at your model. Open the panel's ๊ฐ๋ฐ์ ์ค์ (Developer settings).
- Mac: keep the server URL
http://localhost:8010/v1andtool_choiceauto. Set the model name to your--served-model-name, for exampleKORMo-VL/KORMo-Web-9B. The default model nameKORMo-VL/KORMo-Web-4B-mlx-4bitalready matches the 4-bit build. - NVIDIA GPU: set the server URL to
http://<server>:8000/v1(orlocalhost), the model name toKORMo-Web-9B, andtool_choicetonone.
4. Discord (optional). Run node relay/discord-relay.mjs in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.
Full step-by-step instructions (Korean) are in the extension README.
Quickstart
Tested with vLLM 0.23.0 and transformers 5.12.1. The model runs on a single GPU (TP=1).
Run the agent
1. Install and download
pip install -U "vllm>=0.23.0" openai playwright huggingface_hub
playwright install chromium
hf download KORMo-VL/KORMo-Web-9B --local-dir KORMo-Web-9B
2. Start the model server
vllm serve ./KORMo-Web-9B --served-model-name KORMo-Web-9B \
--max-model-len 131072 --limit-mm-per-prompt '{"image":1,"video":0}'
3. Run a browser task
python KORMo-Web-9B/examples/agent_playwright.py --model KORMo-Web-9B \
--start-url https://www.danawa.com/ \
--task "๋ค๋์์์ 50๋ง ์ ์ดํ, 10์ธ์น ์ด์ ํ๋ธ๋ฆฟ 3๊ฐ๋ฅผ ์ฐพ์์ ๊ฐ๊ฒฉ์ ์๋ ค์ค"
The agent drives a headless Chromium, prints each action as it goes, and finishes with the model's answer. Add --headful to watch the browser window.
On a Mac (vllm-mlx, see How to Start above), point the script at that server and pass --tool-choice auto:
python KORMo-Web-9B/examples/agent_playwright.py --endpoint http://localhost:8010/v1 \
--model KORMo-VL/KORMo-Web-9B --tool-choice auto --headful \
--start-url https://www.danawa.com/ \
--task "๋ค๋์์์ 50๋ง ์ ์ดํ, 10์ธ์น ์ด์ ํ๋ธ๋ฆฟ 3๊ฐ๋ฅผ ์ฐพ์์ ๊ฐ๊ฒฉ์ ์๋ ค์ค"
Inference example (one step)
Send one screenshot with the request and get back one action:
import base64, json
from openai import OpenAI
SYSTEM = open("KORMo-Web-9B/system_prompt.md").read().strip()
TOOLS = json.load(open("KORMo-Web-9B/tools.json"))
shot = base64.b64encode(open("screen.png", "rb").read()).decode()
obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
"- Tab 0 (active): https://www.google.com/\n\nurl: https://www.google.com/\nscreenshot:\n")
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": [
{"type": "text", "text": "<user_request>\n๊ตฌ๊ธ์์ KAIST ๋ฅผ ๊ฒ์ํด์ ๊ณต์ ํํ์ด์ง๋ฅผ ์ด์ด์ค.\n</user_request>\n" + obs},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}},
{"type": "text", "text": "\n</observation>"},
]},
]
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
out = client.chat.completions.create(
model="KORMo-Web-9B", messages=messages, tools=TOOLS,
tool_choice="none", # vLLM. On vllm-mlx send "auto" and read message.tool_calls (see below)
temperature=0.6, top_p=0.95, max_tokens=16384,
extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
)
print(out.choices[0].message.content)
The output looks like this (illustrative):
<think>
I can see the Google home page with an empty search box in the center. I have not searched yet.
My plan: 1. Click the search box. 2. Type "KAIST". 3. Press Enter and open the official site.
</think>
<tool_call>
<function=click>
<parameter=x>
500
</parameter>
<parameter=y>
427
</parameter>
</function>
</tool_call>
tool_choice depends on the server. On vLLM, send "none": the tool schema is still rendered into the prompt and the call comes back as text in message.content, as above. Sending tools without tool_choice makes vLLM return HTTP 400 unless the server was started with --enable-auto-tool-choice. On vllm-mlx, "none" drops the tool schema from the prompt, so send "auto"; the call then arrives in message.tool_calls. examples/agent_playwright.py turns it back into the <tool_call> text above (--tool-choice auto).
Coordinates in the tool call are normalized to 0โ1000. Convert them to pixels with x / 1000 ร width and y / 1000 ร height.
How It Works
user <user_request> task </user_request> + observation (tabs, url, screenshot)
assistant <think> what I see ยท what is done ยท plan </think> <tool_call> one action </tool_call>
tool action result + new observation (newest screenshot only)
โฆ repeat until finished(answer)
- Action space: 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control,
finished, and others (tools.json). - Memory: the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
- Recommended settings: temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in
generation_config.json.
The exact message format, parsing rules and feedback strings are in HARNESS.md. Follow it if you build your own harness.
Evaluation Details
| Tasks | Online-Mind2Web: 300 English, 312 Korean |
| Judge | GPT-6 |
Files
| File | Description |
|---|---|
chat_template.jinja |
Chat template (Qwen3.5 XML tool calls, thinking on) |
system_prompt.md ยท tools.json |
System prompt and 22-tool schema, verbatim from training |
HARNESS.md |
Full harness specification |
examples/agent_playwright.py |
Reference browser agent (Playwright + OpenAI-compatible API). --tool-choice none for vLLM, auto for vllm-mlx; --headful shows the browser. It does not execute upload. |
config.json ยท generation_config.json ยท processor_config.json ยท tokenizer*.json |
Model, sampling and processor configs |
ํ๊ตญ์ด ์์ฝ
KORMo-Web-9B๋ ์ฌ๋์ฒ๋ผ ์น ํ๋ฉด์ ๋ณด๊ณ ํด๋ฆญยท์ ๋ ฅยทํ์์ ์ํํ๋ 9B ๊ท๋ชจ์ ํ์ ์๊ฐ ๊ธฐ๋ฐ ์น ์์ด์ ํธ์ ๋๋ค.
- ๋๊ท๋ชจ ์น ์ฌ์ ํ์ต: 1,000๋ง ๊ฑด ์ด์์ ํ์ ์น ํ์ต ์ํ๋ก UI ์์์ ์์นยท์๋ฏธยท๊ตฌ์กฐยท๊ธฐ๋ฅ ๋ฑ์ ์ธ๋ฐํ๊ฒ ํ์ต(Atomic-Level Pretraining)
- ์ค์ ์น ํ๋ ํ์ต: 10๋ง ๊ฑด ์ด์์ ํ๊ตญ์ธ ํ์ ์น ์ฌ์ฉ ๊ถค์ ์ ํ์ตํด ์ฌ๋ฌ ์น์ฌ์ดํธ์ ํญ์ ๋๋๋๋ ๋ณต์กํ ์์ ์ํ
- ๋ฐ์ด๋ ์ฑ๋ฅ: Online-Mind2Web์์ ์์ดยทํ๊ตญ์ด 51.0% ยท 48.1% ์ฑ๊ณต๋ฅ , ๋น๊ตํ ๊ณต๊ฐ ๋ชจ๋ธ ์ค ์ต๊ณ ์ฑ๋ฅ
- ์๊ฐ ์ค์ฌ ์น ์กฐ์: ๋ณ๋์ DOM ์ ๋ณด ์์ด ์คํฌ๋ฆฐ์ท๋ง์ผ๋ก ์น์ ์ดํดํ๊ณ , ํ๋ ๊ฒฐ๊ณผ๋ฅผ ํ์ธํ๋ฉฐ ๋ค์ ์์ ์ํ
- ๋ชจ๋ธยท๋ฐ์ดํฐ ๊ณต๊ฐ: Apache-2.0 ๋ชจ๋ธ ๊ฐ์ค์น, ๋๊ท๋ชจ ํ์ต ๋ฐ์ดํฐ, ์น ์ฌ์ฉ ๊ถค์ , ํ๊ฐ ๋ฒค์น๋งํฌ ๋ฐ ์คํ ์ฝ๋ ๊ณต๊ฐ
Acknowledgements
Built on Qwen3.5-9B. Evaluated on Online-Mind2Web and its Korean counterpart.
Contact
- KyungTae Lim, Professor at KAIST.
ktlim@kaist.ac.kr
Contributor
- Minkyung Cho (minkyung.cho@kaist.ac.kr)
- Yerin Nam (yerinnam@kaist.ac.kr)
- Hangyeol Yoo (hg.yoo@kaist.ac.kr)
- Minkyu Kim (minkyukim@kaist.ac.kr)
- Seungwoo Song (sswoo@kaist.ac.kr)
- Gyun Lee (gyunini@kaist.ac.kr)
- KyungTae Lim (ktlim@kaist.ac.kr)