KORMo-Web-11B

One Model. Three Capabilities: LLM, VLM, and Web Agent.

KORMo-Web-11BλŠ” KAIST MLP 연ꡬ싀이 μ²˜μŒλΆ€ν„° κ΅¬μΆ•ν•œ KORMo νŒŒμš΄λ°μ΄μ…˜ λͺ¨λΈμ„ 기반으둜, LLMΒ·VLMΒ·GUI Agent λŠ₯λ ₯을 λͺ¨λ‘ κ°–μΆ˜ λ²”μš© λ©€ν‹°λͺ¨λ‹¬ AI λͺ¨λΈμž…λ‹ˆλ‹€.

  • ν•˜λ‚˜μ˜ λͺ¨λΈ, μ„Έ κ°€μ§€ λŠ₯λ ₯: 일반적인 μ§ˆλ¬Έμ—λŠ” LLM으둜 λ‹΅λ³€ν•˜κ³ , 이미지 이해가 ν•„μš”ν•œ 경우 VLM으둜, μ›Ή μ‘°μž‘μ΄ ν•„μš”ν•œ μš”μ²­μ—λŠ” GUI Agent둜 μž‘λ™ν•©λ‹ˆλ‹€.
  • λ…μž νŒŒμš΄λ°μ΄μ…˜ λͺ¨λΈ 기반: From-scratch둜 ν•™μŠ΅ν•œ KORMo-10Bλ₯Ό KORMo-VL둜 ν™•μž₯ν•˜κ³ , κΈ°μ‘΄ μ–Έμ–΄Β·μ‹œκ° 이해 λŠ₯λ ₯을 μœ μ§€ν•˜λ„λ‘ ν•™μŠ΅ν•œ μ›Ή μ—μ΄μ „νŠΈλ‘œ 생각<Think> λ˜ν•œ μœ μΌν•˜κ²Œ ν•œκ΅­μ–΄λ‘œ ν•˜λŠ” λͺ¨λΈμž…λ‹ˆλ‹€.
  • λŒ€κ·œλͺ¨ μ›Ή μ‚¬μ „ν•™μŠ΅: 1,000만 건 μ΄μƒμ˜ ν•œμ˜ μ›Ή ν•™μŠ΅ μƒ˜ν”Œλ‘œ UI μš”μ†Œμ˜ μœ„μΉ˜Β·μ˜λ―ΈΒ·κ΅¬μ‘°Β·κΈ°λŠ₯ 등을 μ„Έλ°€ν•˜κ²Œ ν•™μŠ΅ν–ˆμŠ΅λ‹ˆλ‹€.
  • μ‹€μ œ μ›Ή 행동 ν•™μŠ΅: 10만 건 μ΄μƒμ˜ ν•œμ˜ μ›Ή μ‚¬μš© ꢀ적을 ν™œμš©ν•΄ λ³΅μž‘ν•œ 닀단계 μ›Ή μž‘μ—…μ„ μˆ˜ν–‰ν•©λ‹ˆλ‹€.
  • μ™„μ „ 곡개 및 상업적 ν™œμš©: λͺ¨λΈ κ°€μ€‘μΉ˜μ™€ ν•™μŠ΅Β·ν‰κ°€ λ¦¬μ†ŒμŠ€λ₯Ό κ³΅κ°œν•˜λ©°, Apache-2.0 λΌμ΄μ„ μŠ€μ— 따라 상업적 이용과 μˆ˜μ •Β·μž¬λ°°ν¬κ°€ κ°€λŠ₯ν•©λ‹ˆλ‹€.

KORMo-Web-11B is a general-purpose bilingual AI model built on the KORMo foundation model family, developed by KAIST MLP Lab starting from a language model trained entirely from scratch. Unlike our Qwen-based KORMo-Web-4B and 9B models, which focus on maximizing web task performance, KORMo-Web-11B is designed to preserve its original LLM and VLM capabilities while learning to act as a GUI agent. It can understand Korean and English text, interpret images, and complete real browser tasks by visually navigating webpagesβ€”all within a single model.

What Makes KORMo-Web-11B Different?

  • Built on Our Own Foundation Model: Developed from the KORMo-10B language model, trained from scratch by KAIST MLP Lab, and extended through KORMo-VL into a web agent.
  • Three Capabilities in One Model: Designed to retain general LLM reasoning and VLM understanding while adding GUI interaction capabilities, rather than optimizing exclusively for web navigation.
  • 10M+ Atomic-Level Web Pretraining Samples: Large-scale Korean and English web understanding training, covering individual UI elements, their locations, layouts, functions, and states.
  • 100K+ Web Interaction Trajectories: Bilingual browser demonstrations for learning real-world, multi-step web tasks.
  • Open and Commercially Usable: Model weights and supporting resources are openly released under Apache-2.0, allowing commercial use, modification, and redistribution subject to the license terms.

From a Korean LLM to a General-Purpose Web Agent

KORMo-Web-11B extends our independently developed Korean–English foundation model through three stages:

Stage Model Capabilities
1. Language KORMo-10B Korean–English LLM trained from scratch with a Korean-optimized tokenizer
2. Vision KORMo-VL Multimodal understanding through a SigLIP vision encoder and KORMo language backbone
3. Web Agent KORMo-Web-11B Large-scale web pretraining and action learning for visual browser interaction

The goal is not just a better web agent, but a versatile model that can understand language, interpret visual information, and interact with the web.

Performance

We evaluated KORMo-Web-11B on 612 live-website tasks (300 English, 312 Korean) from Online-Mind2Web, using the same LLM judge across all evaluated models.

Model Size Online-Mind2Web EN Online-Mind2Web KO
Qwen3.5-9B (base) 9B 24.7 16.7
Fara-1.5-4B 4B 20.7 7.7
UI-TARS-1.5-7B 7B 22.3 20.2
OpenWebRL-4B 4B 32.7 15.4
KORMo-Web-11B (ours) 11B 29.0 30.1
GPT-6-luna (proprietary) – 63.0 65.7

Success rates are calculated over the full task set, including failed or incomplete runs. Each agent uses its own prompt and action format.

KORMo-Web-11B achieves 30.1% Korean task success, exceeding the strongest non-KORMo open-weight baseline listed above by 9.9 percentage points.

While our specialized KORMo-Web-4B and 9B models achieve higher web task success, KORMo-Web-11B prioritizes versatility across language, vision, and GUI interaction, rather than web-only optimization.

Why KORMo-Web-11B?

  • Korean from the Ground Up: Built upon our independently pretrained Korean–English foundation model.
  • General-Purpose Multimodal AI: Supports text-based language tasks, visual understanding, and browser interactions within one model.
  • Visual-First Web Interaction: Operates primarily through screenshots without requiring full DOM or accessibility trees.
  • Think, Then Act: Generates an inspectable reasoning step before executing browser actions.
  • Commercial Use Permitted: Released under Apache-2.0, supporting academic research, commercial applications, fine-tuning, and redistribution.
  • Open and Ready to Run: Includes model weights, Transformers 5 support, a vLLM inference path, browser harness, and reference inference settings.

KORMo-Web-11B β€” From a Korean Foundation Model to a General-Purpose Web Agent.

Quickstart

Tested with transformers 5.12.1 and vLLM 0.23.0. The model runs on a single GPU.

transformers does not ship the KORMo text model, so this repo includes it (_modeling_kormo.py) together with a small loader, kormo_load.py:

  • kormo_load.register(path) registers the KORMo text model. Call it before loading the processor or the model.
  • kormo_load.patch_vision(model) feeds the SigLIP features after post-layernorm to the projector, as in training. Without it the model still runs but misreads screenshots.

Inference example (one step, transformers)

pip install -U "transformers>=5.12" torch pillow huggingface_hub
huggingface-cli download KORMo-VL/KORMo-Web-11B --local-dir KORMo-Web-11B
import json, sys, torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

M = "KORMo-Web-11B"
sys.path.insert(0, M)
import kormo_load
kormo_load.register(M)
processor = AutoProcessor.from_pretrained(M)
model = AutoModelForImageTextToText.from_pretrained(M, dtype=torch.bfloat16).to("cuda").eval()
kormo_load.patch_vision(model)

SYSTEM = open(f"{M}/system_prompt.md").read().strip()
TOOLS = json.load(open(f"{M}/tools.json"))
obs = ("<observation>\nscreen size: 1280 x 720\nCurrent tab: 0\nAvailable tabs:\n"
       "- Tab 0 (active): https://www.aladin.co.kr/\n\nurl: https://www.aladin.co.kr/\nscreenshot:\n")
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": [
        {"type": "text", "text": "<user_request>\nμ•ŒλΌλ”˜ μ‚¬μ΄νŠΈμ—μ„œ 'ν•΄μ»€μŠ€ 토읡 RC'λ₯Ό κ²€μƒ‰ν•œ λ’€ κ°€μž₯ λ¨Όμ € ν‘œμ‹œλ˜λŠ” λ„μ„œμ˜ μΆœκ°„μΌμ„ μ•Œλ €μ€˜.\n</user_request>\n" + obs},
        {"type": "image", "image": Image.open("screen.png").convert("RGB")},
        {"type": "text", "text": "\n</observation>"},
    ]},
]
inputs = processor.apply_chat_template(messages, tools=TOOLS, add_generation_prompt=True,
                                       tokenize=True, return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=False))

A real first step from our Online-Mind2Web run (Aladin home page):

<think>I see the Aladin homepage, with the search field labeled β€œν†΅ν•©κ²€μƒ‰β€ near the top. I have not yet searched for the book, so I still need to enter β€œν•΄μ»€μŠ€ 토읡 RC,” inspect the first result’s publication date, and report that date. My plan:
1. Click the search field
2. Enter β€œν•΄μ»€μŠ€ 토읡 RC” and submit the search
3. Read the publication date of the first listed book
This click targets the visible search field and is a safe, relevant first action.
</think>

<tool_call>
<function=click>
<parameter=x>
482
</parameter>
<parameter=y>
212
</parameter>
</function>
</tool_call>

Coordinates are normalized to 0–1000. Convert them to pixels with x / 1000 Γ— width and y / 1000 Γ— height.

Run the agent (vLLM)

vLLM does not include the KORMo architecture either. The KORMo text layers have the Llama structure, so vllm/convert_to_vllm.py renames two parameter names and writes a folder vLLM loads natively. vllm/sitecustomize.py turns SigLIP post-layernorm back on inside vLLM.

1. Convert (once)

pip install -U "vllm>=0.23.0" openai playwright safetensors
playwright install chromium
python KORMo-Web-11B/vllm/convert_to_vllm.py KORMo-Web-11B KORMo-Web-11B-vllm

2. Start the model server

PYTHONPATH=$PWD/KORMo-Web-11B/vllm vllm serve ./KORMo-Web-11B-vllm --served-model-name KORMo-Web-11B \
  --max-model-len 65536 --limit-mm-per-prompt '{"image":1,"video":0}'

Check that the server log prints [kormo-vllm] SigLIP post_layernorm enabled.

3. Run a browser task

python KORMo-Web-11B/examples/agent_playwright.py --model KORMo-Web-11B \
  --task "λ‹€λ‚˜μ™€μ—μ„œ 50만 원 μ΄ν•˜, 10인치 이상 νƒœλΈ”λ¦Ώ 3개λ₯Ό μ°Ύμ•„μ„œ 가격을 μ•Œλ €μ€˜"

The agent opens Chromium, prints each action as it goes, and finishes with the model's answer.

How It Works

user       <user_request> task </user_request>  + observation (tabs, url, screenshot)
assistant  <think> what I see Β· what is done Β· plan </think>  <tool_call> one action </tool_call>
tool       action result  + new observation (newest screenshot only)
…          repeat until finished(answer)
  • Action space: 22 browser tools: click, type, hotkey, scroll, drag, goto, tab control, finished, and others (tools.json).
  • Memory: the full text history of thoughts, actions and observations is kept. Only the current screenshot is sent.
  • Recommended settings: temperature 0.6, top_p 0.95, top_k 20, thinking on, up to 16k tokens per step and 50 steps. These are already set in generation_config.json.

The exact message format, parsing rules and feedback strings are in HARNESS.md. Follow it if you build your own harness.

Evaluation Details

Tasks Online-Mind2Web: 300 English, 312 Korean
Judge GPT-6

Files

File Description
model.safetensors Β· config.json Weights and config (LLaVA-OneVision: SigLIP + KORMo-10B text model)
_configuration_kormo.py Β· _modeling_kormo.py KORMo text model code for transformers 5
kormo_load.py register() and patch_vision()
chat_template.jinja Chat template (XML tool calls, thinking on)
system_prompt.md Β· tools.json System prompt and 22-tool schema
HARNESS.md Full harness specification
examples/agent_playwright.py Reference browser agent (Playwright + OpenAI-compatible API). It does not execute upload.
vllm/convert_to_vllm.py Β· vllm/sitecustomize.py Convert for vLLM and the post-layernorm fix
generation_config.json Β· *processor_config.json Β· tokenizer*.json Β· special_tokens_map.json Sampling, processor and tokenizer configs

Acknowledgements

Built on KORMo-VL and KORMo-10B by KAIST MLP Lab. Evaluated on Online-Mind2Web and its Korean counterpart.

Contact

  • KyungTae Lim, Professor at KAIST. ktlim@kaist.ac.kr

Contributor

Downloads last month
26
Safetensors
Model size
11B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for KORMo-VL/KORMo-Web-11B

Finetuned
(1)
this model

Spaces using KORMo-VL/KORMo-Web-11B 2

Collection including KORMo-VL/KORMo-Web-11B