Instructions to use KORMo-VL/KORMo-Web-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use KORMo-VL/KORMo-Web-4B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="KORMo-VL/KORMo-Web-4B") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("KORMo-VL/KORMo-Web-4B") model = AutoModelForMultimodalLM.from_pretrained("KORMo-VL/KORMo-Web-4B", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use KORMo-VL/KORMo-Web-4B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "KORMo-VL/KORMo-Web-4B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-4B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/KORMo-VL/KORMo-Web-4B
- SGLang
How to use KORMo-VL/KORMo-Web-4B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-4B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-4B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "KORMo-VL/KORMo-Web-4B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "KORMo-VL/KORMo-Web-4B", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use KORMo-VL/KORMo-Web-4B with Docker Model Runner:
docker model run hf.co/KORMo-VL/KORMo-Web-4B
KORMo-Web-4B
KORMo-Web-4B is a 4B vision-language agent that completes real browser tasks from screenshots alone. Give it a request in Korean or English; it plans, clicks, types and moves between sites and tabs until the task is done. In the same live-web evaluation, it beats every open-weight web agent we compared, by +9.6 points in English and +22.1 points in Korean.
On Online-Mind2Web it reaches 42.3% in English and 42.3% in Korean, which is +22.3 / +28.8 points over the base Qwen3.5-4B.
What Makes KORMo-Web Different?
- 10M+ Atomic-Level Web Pretraining Samples. Large-scale Korean and English web understanding pretraining teaches the model what individual UI elements are, where they are, and what they do—covering text, location, appearance, layout, function, and state.
- 100K+ Web Interaction Trajectories. Bilingual browser demonstrations teach the model to translate fine-grained web understanding into real-world, multi-step actions.
- Open Models and Data. Releasing model weights, large-scale bilingual web pretraining data, interaction trajectories, and evaluation benchmarks to support reproducible web agent research.
Performance
All models below ran the same 612 live-website tasks (300 English, 312 Korean), each with its own prompt and action format, and were scored by the same LLM judge.
| Model | Size | Online-Mind2Web EN | Online-Mind2Web KO |
|---|---|---|---|
| Qwen3.5-4B (base, no fine-tuning) | 4B | 20.0 | 13.5 |
| Fara-1.5-4B | 4B | 20.7 | 7.7 |
| UI-TARS-1.5-7B | 7B | 22.3 | 20.2 |
| OpenWebRL-4B | 4B | 32.7 | 15.4 |
| KORMo-Web-4B (ours) | 4B | 42.3 | 42.3 |
| KORMo-Web-9B (ours) | 9B | 51.0 | 48.1 |
Success rate (%). The denominator is always the full task count, so crashed or empty runs count as failures.
KORMo-Web-4B outperforms open web agents of similar size in both English and Korean.
Why KORMo-Web
- Korean and English, natural requests. Handles short commands as well as long, conversational instructions in either language.
- Screenshot-only. It needs no DOM or accessibility tree; it acts on what is on screen, like a person, and checks each action's result on the next screenshot.
- Thinks, then acts. A short reasoning step precedes every action, so its behaviour is easy to inspect and debug.
- Multi-site, multi-tab workflows. It can search on one site, verify on another and write the result into a third, managing tabs on its own.
- Open and ready to run. Apache-2.0 weights, a reference browser harness, the tool schema and the exact inference settings are all included.
Quickstart
Tested with vLLM 0.23.0 and transformers 5.12.1. The model runs on a single GPU (TP=1).
Run the agent
1. Install and download
pip install -U "vllm>=0.23.0" openai playwright huggingface_hub
playwright install chromium
huggingface-cli download KORMo-VL/KORMo-Web-4B --local-dir KORMo-Web-4B
2. Start the model server
vllm serve ./KORMo-Web-4B --served-model-name KORMo-Web-4B \
--max-model-len 131072 --limit-mm-per-prompt '{"image":1,"video":0}'
3. Run a browser task
python KORMo-Web-4B/examples/agent_playwright.py --model KORMo-Web-4B \
--task "다방에서 월평역 근처 보증금 천만원 이하로 설정한뒤 원룸 3개정도 찾아서 살펴보고 다하면 찾은 내용을 카이스트 메일로 mlplab@kaist.ac.kr 한테 정리해서 보내줘"
The agent opens Chromium, prints each action as it goes, and finishes with the model's answer.
Inference example (one step)
Send one screenshot with the request and get back one action:
import base64, json
from openai import OpenAI
SYSTEM = open("KORMo-Web-4B/system_prompt.md").read().strip()
TOOLS = json.load(open("KORMo-Web-4B/tools.json"))
shot = base64.b64encode(open("screen.png", "rb").read()).decode()
obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
"- Tab 0 (active): https://www.google.com/\n\nurl: https://www.google.com/\nscreenshot:\n")
messages = [
{"role": "system", "content": SYSTEM},
{"role": "user", "content": [
{"type": "text", "text": "<user_request>\n구글에서 KAIST 를 검색해서 공식 홈페이지를 열어줘.\n</user_request>\n" + obs},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{shot}"}},
{"type": "text", "text": "\n</observation>"},
]},
]
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
out = client.chat.completions.create(
model="KORMo-Web-4B", messages=messages, tools=TOOLS,
temperature=0.6, top_p=0.95, max_tokens=16384,
extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
)
print(out.choices[0].message.content)
The output looks like this (illustrative):
<think>
I can see the Google home page with an empty search box in the center. I have not searched yet.
My plan: 1. Click the search box. 2. Type "KAIST". 3. Press Enter and open the official site.
</think>
<tool_call>
<function=click>
<parameter=x>
500
</parameter>
<parameter=y>
427
</parameter>
</function>
</tool_call>
Coordinates in the tool call are normalized to 0–1000. Convert them to pixels with x / 1000 × width and y / 1000 × height.
How It Works
user <user_request> task </user_request> + observation (tabs, url, screenshot)
assistant <think> what I see · what is done · plan </think> <tool_call> one action </tool_call>
tool action result + new observation (newest screenshot only)
… repeat until finished(answer)
- Action space: 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control,
finished, and others (tools.json). - Memory: the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
- Recommended settings: temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in
generation_config.json.
The exact message format, parsing rules and feedback strings are in HARNESS.md. Follow it if you build your own harness.
Evaluation Details
| Tasks | Online-Mind2Web: 300 English, 312 Korean |
| Judge | GPT-6 |
Files
| File | Description |
|---|---|
chat_template.jinja |
Chat template (Qwen3.5 XML tool calls, thinking on) |
system_prompt.md · tools.json |
System prompt and 22-tool schema, verbatim from training |
HARNESS.md |
Full harness specification |
examples/agent_playwright.py |
Reference browser agent (Playwright + OpenAI-compatible API). It does not execute upload. |
config.json · generation_config.json · processor_config.json · tokenizer*.json |
Model, sampling and processor configs |
한국어 요약
KORMo-Web-4B는 사람처럼 웹 화면을 보고 클릭·입력·탐색을 수행하는 4B 규모의 한영 시각 기반 웹 에이전트입니다.
- 대규모 웹 사전학습: 1,000만 건 이상의 한영 웹 학습 샘플로 UI 요소의 위치·의미·구조·기능 등을 세밀하게 학습(Atomic-Level Pretraining)
- 실제 웹 행동 학습: 10만 건 이상의 한국인 한영 웹 사용 궤적을 학습해 여러 웹사이트와 탭을 넘나드는 복잡한 작업 수행
- 뛰어난 성능: Online-Mind2Web에서 영어·한국어 모두 42.3% 성공률, 비교한 공개 모델 중 최고 성능(기존 최고 대비 +9.6%p / +22.1%p)
- 시각 중심 웹 조작: 별도의 DOM 정보 없이 스크린샷만으로 웹을 이해하고, 행동 결과를 확인하며 다음 작업 수행
- 모델·데이터 공개: Apache-2.0 모델 가중치, 대규모 학습 데이터, 웹 사용 궤적, 평가 벤치마크 및 실행 코드 공개
Acknowledgements
Built on Qwen3.5-4B. Evaluated on Online-Mind2Web and its Korean counterpart.
Contact
- KyungTae Lim, Professor at KAIST.
ktlim@kaist.ac.kr
Contributor
- Minkyu Kim (minkyukim@kaist.ac.kr)
- Yerin Nam (yerinnam@kaist.ac.kr)
- Seungwoo Song (sswoo@kaist.ac.kr)
- Minkyung Cho (minkyung.cho@kaist.ac.kr)
- Hangyeol Yoo (hg.yoo@kaist.ac.kr)
- Gyun Lee (gyunini@kaist.ac.kr)
- KyungTae Lim (ktlim@kaist.ac.kr)
- Downloads last month
- 43