jungsin3 commited on
Commit
4be1449
ยท
verified ยท
1 Parent(s): 841f53f

Quickstart works on vLLM and vllm-mlx: tool_choice, tool_calls restore, Mac 4B/9B serving + memory, fixes

Browse files
.gitattributes CHANGED
@@ -39,3 +39,4 @@ assets/GUI_agent_dem.mp4 filter=lfs diff=lfs merge=lfs -text
39
  assets/gui_final_demo_dubbed.mp4 filter=lfs diff=lfs merge=lfs -text
40
  assets/gui_final_demo_maksed.mp4 filter=lfs diff=lfs merge=lfs -text
41
  assets/setup_guide.mp4 filter=lfs diff=lfs merge=lfs -text
 
 
39
  assets/gui_final_demo_dubbed.mp4 filter=lfs diff=lfs merge=lfs -text
40
  assets/gui_final_demo_maksed.mp4 filter=lfs diff=lfs merge=lfs -text
41
  assets/setup_guide.mp4 filter=lfs diff=lfs merge=lfs -text
42
+ assets/gui_final_demo_masked.mp4 filter=lfs diff=lfs merge=lfs -text
HARNESS.md CHANGED
@@ -19,6 +19,17 @@ A runnable reference implementation is in [`examples/agent_playwright.py`](examp
19
  | Sampling | [`generation_config.json`](generation_config.json) | `temperature 0.6 ยท top_p 0.95 ยท top_k 20` |
20
  | Harness | **this file** | Message layout, parsing, execution, feedback |
21
 
 
 
 
 
 
 
 
 
 
 
 
22
  ---
23
 
24
  ## 2. Conversation layout
 
19
  | Sampling | [`generation_config.json`](generation_config.json) | `temperature 0.6 ยท top_p 0.95 ยท top_k 20` |
20
  | Harness | **this file** | Message layout, parsing, execution, feedback |
21
 
22
+ ### Sending `tools` and `tool_choice`
23
+
24
+ Always pass the 22 tools as `tools=`; the chat template renders them into the prompt. What `tool_choice` to send depends on the server:
25
+
26
+ | Server | `tool_choice` | Where the call comes back |
27
+ |---|---|---|
28
+ | vLLM (no `--enable-auto-tool-choice`) | `"none"` | As text in `message.content` (`<think>โ€ฆ</think>` + `<tool_call>โ€ฆ</tool_call>`). Omitting `tool_choice` returns HTTP 400. |
29
+ | vllm-mlx (Apple silicon) | `"auto"` | In `message.tool_calls`. `"none"` would drop the tool schema from the prompt. |
30
+
31
+ When the call arrives in `message.tool_calls`, rebuild the `<tool_call><function=โ€ฆ><parameter=โ€ฆ>` text (ยง3) and keep that in the assistant turn, so the history looks exactly like training. `restore_tool_calls()` in the reference agent does this.
32
+
33
  ---
34
 
35
  ## 2. Conversation layout
README.md CHANGED
@@ -42,7 +42,7 @@ tags:
42
 
43
  <div id="kormo-web-card">
44
 
45
- <video src="https://huggingface.co/KORMo-VL/KORMo-Web-4B/resolve/main/assets/gui_final_demo_maksed.mp4" controls autoplay muted loop playsinline width="100%"></video>
46
 
47
  # KORMo-Web-4B
48
 
@@ -95,14 +95,29 @@ The fastest way to try KORMo-Web is the **[KORMo-Web Chrome extension](https://g
95
 
96
  ### Mac (Apple silicon)
97
 
98
- Run the 4-bit MLX build of KORMo-Web-4B with `vllm-mlx`. Simple, but slower. Keep this terminal open.
 
 
 
 
 
 
 
 
99
 
100
  ```bash
101
  uv venv --python 3.12 && source .venv/bin/activate
102
- uv pip install vllm-mlx
103
- vllm-mlx serve KORMo-VL/KORMo-Web-4B-mlx-4bit --port 8010 --mllm
 
 
 
 
 
104
  ```
105
 
 
 
106
  ### NVIDIA GPU (local PC or server)
107
 
108
  Serve the model with vLLM. Much faster, and one server can be shared.
@@ -115,13 +130,13 @@ vllm serve KORMo-VL/KORMo-Web-4B --host 0.0.0.0 --port 8000 \
115
  --gdn-prefill-backend triton
116
  ```
117
 
118
- Check that it is up with `curl http://localhost:8010/v1/models` (Mac) or `curl http://<server>:8000/v1/models` (vLLM).
119
 
120
  **2. Install the extension.** `git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git`, open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked** and choose the cloned folder. Open the **GUI ์—์ด์ „ํŠธ** side panel, type a task and press **์‹คํ–‰** (Run).
121
 
122
  **3. Point the extension at your model.** Open the panel's **๊ฐœ๋ฐœ์ž ์„ค์ •** (Developer settings).
123
 
124
- - **Mac:** nothing to change. The defaults (`http://localhost:8010/v1`, `KORMo-VL/KORMo-Web-4B-mlx-4bit`, `tool_choice` `auto`) match the command above.
125
  - **NVIDIA GPU:** set the server URL to `http://<server>:8000/v1` (or `localhost`), the model name to `KORMo-Web-4B`, and `tool_choice` to `none`.
126
 
127
  **4. Discord (optional).** Run `node relay/discord-relay.mjs` in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.
@@ -153,10 +168,20 @@ vllm serve ./KORMo-Web-4B --served-model-name KORMo-Web-4B \
153
 
154
  ```bash
155
  python KORMo-Web-4B/examples/agent_playwright.py --model KORMo-Web-4B \
156
- --task "๋‹ค๋ฐฉ์—์„œ ์›”ํ‰์—ญ ๊ทผ์ฒ˜ ๋ณด์ฆ๊ธˆ ์ฒœ๋งŒ์› ์ดํ•˜๋กœ ์„ค์ •ํ•œ๋’ค ์›๋ฃธ 3๊ฐœ์ •๋„ ์ฐพ์•„์„œ ์‚ดํŽด๋ณด๊ณ  ๋‹คํ•˜๋ฉด ์ฐพ์€ ๋‚ด์šฉ์„ ์นด์ด์ŠคํŠธ ๋ฉ”์ผ๋กœ mlplab@kaist.ac.kr ํ•œํ…Œ ์ •๋ฆฌํ•ด์„œ ๋ณด๋‚ด์ค˜"
 
157
  ```
158
 
159
- The agent opens Chromium, prints each action as it goes, and finishes with the model's answer.
 
 
 
 
 
 
 
 
 
160
 
161
  ### Inference example (one step)
162
 
@@ -183,6 +208,7 @@ messages = [
183
  client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
184
  out = client.chat.completions.create(
185
  model="KORMo-Web-4B", messages=messages, tools=TOOLS,
 
186
  temperature=0.6, top_p=0.95, max_tokens=16384,
187
  extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
188
  )
@@ -209,6 +235,8 @@ My plan: 1. Click the search box. 2. Type "KAIST". 3. Press Enter and open the o
209
  </tool_call>
210
  ```
211
 
 
 
212
  Coordinates in the tool call are normalized to 0โ€“1000. Convert them to pixels with `x / 1000 ร— width` and `y / 1000 ร— height`.
213
 
214
  ## How It Works
@@ -240,7 +268,7 @@ The exact message format, parsing rules and feedback strings are in **[HARNESS.m
240
  | `chat_template.jinja` | Chat template (Qwen3.5 XML tool calls, thinking on) |
241
  | `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema, verbatim from training |
242
  | `HARNESS.md` | Full harness specification |
243
- | `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). It does not execute `upload`. |
244
  | `config.json` ยท `generation_config.json` ยท `processor_config.json` ยท `tokenizer*.json` | Model, sampling and processor configs |
245
 
246
  ## ํ•œ๊ตญ์–ด ์š”์•ฝ
 
42
 
43
  <div id="kormo-web-card">
44
 
45
+ <video src="https://huggingface.co/KORMo-VL/KORMo-Web-4B/resolve/main/assets/gui_final_demo_masked.mp4" controls autoplay muted loop playsinline width="100%"></video>
46
 
47
  # KORMo-Web-4B
48
 
 
95
 
96
  ### Mac (Apple silicon)
97
 
98
+ KORMo-Web-4B (4-bit or full precision) and KORMo-Web-9B all run on a Mac with `vllm-mlx` 0.6.0 or later. Pick the one that fits your Mac's memory:
99
+
100
+ | Model | Weights | Memory in use | Recommended Mac memory |
101
+ |---|---:|---:|---|
102
+ | `KORMo-Web-4B-mlx-4bit` (4-bit) | 2.8 GB | about 3.8 GB | 16 GB or more |
103
+ | `KORMo-Web-4B` (bf16) | 9.6 GB | about 9.5 GB | 24 GB or more |
104
+ | `KORMo-Web-9B` (bf16) | 17.5 GB | about 19 GB | 48 GB or more (32 GB is tight) |
105
+
106
+ *Measured with vllm-mlx 0.6.0 after a few short requests (macOS units, 1 GB = 1024ยณ bytes). Long episodes add KV cache, about 1 GB per 30k tokens, and Chrome needs memory too. macOS lets the GPU use only about 2/3โ€“3/4 of RAM.*
107
 
108
  ```bash
109
  uv venv --python 3.12 && source .venv/bin/activate
110
+ uv pip install "vllm-mlx>=0.6.0" huggingface_hub
111
+ export HF_HUB_DISABLE_XET=1 # only if large downloads stall
112
+
113
+ MODEL=KORMo-Web-4B # or KORMo-Web-4B-mlx-4bit / KORMo-Web-9B
114
+ hf download KORMo-VL/$MODEL --local-dir ~/models/$MODEL
115
+ vllm-mlx serve ~/models/$MODEL --port 8010 --served-model-name KORMo-VL/$MODEL \
116
+ --enable-auto-tool-choice --tool-call-parser auto
117
  ```
118
 
119
+ Keep this terminal open. `--served-model-name` is the name clients must request; without it the server registers the local folder path and requests by model name return 404. vllm-mlx 0.6.0 and later detect the vision model by itself; older versions need `--mllm`.
120
+
121
  ### NVIDIA GPU (local PC or server)
122
 
123
  Serve the model with vLLM. Much faster, and one server can be shared.
 
130
  --gdn-prefill-backend triton
131
  ```
132
 
133
+ Check that it is up with `curl http://localhost:8010/v1/models` (Mac) or `curl http://<server>:8000/v1/models` (vLLM). If you changed the port, use that port.
134
 
135
  **2. Install the extension.** `git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git`, open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked** and choose the cloned folder. Open the **GUI ์—์ด์ „ํŠธ** side panel, type a task and press **์‹คํ–‰** (Run).
136
 
137
  **3. Point the extension at your model.** Open the panel's **๊ฐœ๋ฐœ์ž ์„ค์ •** (Developer settings).
138
 
139
+ - **Mac:** keep the server URL `http://localhost:8010/v1` and `tool_choice` `auto`. Set the model name to your `--served-model-name`, for example `KORMo-VL/KORMo-Web-4B`. The default model name `KORMo-VL/KORMo-Web-4B-mlx-4bit` already matches the 4-bit build.
140
  - **NVIDIA GPU:** set the server URL to `http://<server>:8000/v1` (or `localhost`), the model name to `KORMo-Web-4B`, and `tool_choice` to `none`.
141
 
142
  **4. Discord (optional).** Run `node relay/discord-relay.mjs` in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.
 
168
 
169
  ```bash
170
  python KORMo-Web-4B/examples/agent_playwright.py --model KORMo-Web-4B \
171
+ --start-url https://www.danawa.com/ \
172
+ --task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"
173
  ```
174
 
175
+ The agent drives a headless Chromium, prints each action as it goes, and finishes with the model's answer. Add `--headful` to watch the browser window.
176
+
177
+ **On a Mac** (`vllm-mlx`, see *How to Start* above), point the script at that server and pass `--tool-choice auto`:
178
+
179
+ ```bash
180
+ python KORMo-Web-4B/examples/agent_playwright.py --endpoint http://localhost:8010/v1 \
181
+ --model KORMo-VL/KORMo-Web-4B --tool-choice auto --headful \
182
+ --start-url https://www.danawa.com/ \
183
+ --task "๋‹ค๋‚˜์™€์—์„œ 50๋งŒ ์› ์ดํ•˜, 10์ธ์น˜ ์ด์ƒ ํƒœ๋ธ”๋ฆฟ 3๊ฐœ๋ฅผ ์ฐพ์•„์„œ ๊ฐ€๊ฒฉ์„ ์•Œ๋ ค์ค˜"
184
+ ```
185
 
186
  ### Inference example (one step)
187
 
 
208
  client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
209
  out = client.chat.completions.create(
210
  model="KORMo-Web-4B", messages=messages, tools=TOOLS,
211
+ tool_choice="none", # vLLM. On vllm-mlx send "auto" and read message.tool_calls (see below)
212
  temperature=0.6, top_p=0.95, max_tokens=16384,
213
  extra_body={"top_k": 20, "chat_template_kwargs": {"enable_thinking": True}},
214
  )
 
235
  </tool_call>
236
  ```
237
 
238
+ **`tool_choice` depends on the server.** On vLLM, send `"none"`: the tool schema is still rendered into the prompt and the call comes back as text in `message.content`, as above. Sending `tools` without `tool_choice` makes vLLM return HTTP 400 unless the server was started with `--enable-auto-tool-choice`. On vllm-mlx, `"none"` drops the tool schema from the prompt, so send `"auto"`; the call then arrives in `message.tool_calls`. [`examples/agent_playwright.py`](examples/agent_playwright.py) turns it back into the `<tool_call>` text above (`--tool-choice auto`).
239
+
240
  Coordinates in the tool call are normalized to 0โ€“1000. Convert them to pixels with `x / 1000 ร— width` and `y / 1000 ร— height`.
241
 
242
  ## How It Works
 
268
  | `chat_template.jinja` | Chat template (Qwen3.5 XML tool calls, thinking on) |
269
  | `system_prompt.md` ยท `tools.json` | System prompt and 22-tool schema, verbatim from training |
270
  | `HARNESS.md` | Full harness specification |
271
+ | `examples/agent_playwright.py` | Reference browser agent (Playwright + OpenAI-compatible API). `--tool-choice none` for vLLM, `auto` for vllm-mlx; `--headful` shows the browser. It does not execute `upload`. |
272
  | `config.json` ยท `generation_config.json` ยท `processor_config.json` ยท `tokenizer*.json` | Model, sampling and processor configs |
273
 
274
  ## ํ•œ๊ตญ์–ด ์š”์•ฝ
assets/{gui_final_demo_maksed.mp4 โ†’ gui_final_demo_masked.mp4} RENAMED
File without changes
examples/agent_playwright.py CHANGED
@@ -1,9 +1,16 @@
1
  """Minimal reference harness for KORMo-Web-4B (see ../HARNESS.md).
2
 
3
  pip install openai playwright && playwright install chromium
4
- vllm serve KORMo-VL/KORMo-Web-4B --max-model-len 131072 --limit-mm-per-prompt '{"image":1,"video":0}'
 
5
  python agent_playwright.py --task "Search for 'KAIST' on Google and open the official homepage."
6
 
 
 
 
 
 
 
7
  It reproduces the message layout the model was trained on:
8
  system = system_prompt.md tools = tools.json (22 tools, rendered by chat_template.jinja)
9
  user = <user_request>โ€ฆ</user_request> + <observation> (+ screenshot on the first step)
@@ -66,6 +73,20 @@ async def observation(ctx, page, with_shot: bool) -> list[dict]:
66
  {"type": "text", "text": "\n</observation>"}]
67
 
68
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
69
  def drop_images(content: list[dict]) -> list[dict]:
70
  """Past turns keep the observation text but lose the screenshot."""
71
  out = [c for c in content if c["type"] == "text"]
@@ -142,10 +163,11 @@ async def execute(ctx, page, name: str, a: dict):
142
  return page, fb + (" Note: no visible navigation or new tab detected." if name.endswith("click") else "")
143
 
144
 
145
- async def run(task: str, start_url: str, endpoint: str, model: str, max_steps: int):
 
146
  client = OpenAI(base_url=endpoint, api_key="EMPTY")
147
  async with async_playwright() as pw:
148
- browser = await pw.chromium.launch()
149
  ctx = await browser.new_context(viewport={"width": 1280, "height": 720})
150
  page = await ctx.new_page()
151
  await page.goto(start_url)
@@ -153,10 +175,9 @@ async def run(task: str, start_url: str, endpoint: str, model: str, max_steps: i
153
  first[0]["text"] = f"<user_request>\n{task}\n</user_request>\n" + first[0]["text"]
154
  msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": first}]
155
  for step in range(max_steps):
156
- out = client.chat.completions.create(model=model, messages=msgs, tools=TOOLS, **SAMPLING)
157
- text = out.choices[0].message.content or ""
158
- if "<think>" not in text and "</think>" not in text and getattr(out.choices[0].message, "reasoning_content", None):
159
- text = f"<think>\n{out.choices[0].message.reasoning_content}\n</think>\n\n{text}"
160
  msgs.append({"role": "assistant", "content": text}) # keep the full <think> + <tool_call>
161
  try:
162
  name, args = parse_call(text)
@@ -181,7 +202,10 @@ if __name__ == "__main__":
181
  ap.add_argument("--task", required=True)
182
  ap.add_argument("--start-url", default="https://www.google.com/")
183
  ap.add_argument("--endpoint", default="http://localhost:8000/v1")
184
- ap.add_argument("--model", default="KORMo-VL/KORMo-Web-4B")
185
  ap.add_argument("--max-steps", type=int, default=50)
 
 
 
186
  a = ap.parse_args()
187
- asyncio.run(run(a.task, a.start_url, a.endpoint, a.model, a.max_steps))
 
1
  """Minimal reference harness for KORMo-Web-4B (see ../HARNESS.md).
2
 
3
  pip install openai playwright && playwright install chromium
4
+ vllm serve KORMo-VL/KORMo-Web-4B --served-model-name KORMo-Web-4B \
5
+ --max-model-len 131072 --limit-mm-per-prompt '{"image":1,"video":0}'
6
  python agent_playwright.py --task "Search for 'KAIST' on Google and open the official homepage."
7
 
8
+ vLLM : --tool-choice none (default). The tool schema is still rendered into the prompt
9
+ and the call comes back as text in message.content.
10
+ vllm-mlx : --tool-choice auto. vllm-mlx drops the schema on "none" and returns the call in
11
+ message.tool_calls; it is turned back into the <tool_call> XML below.
12
+ --headful : show the browser window (default: headless).
13
+
14
  It reproduces the message layout the model was trained on:
15
  system = system_prompt.md tools = tools.json (22 tools, rendered by chat_template.jinja)
16
  user = <user_request>โ€ฆ</user_request> + <observation> (+ screenshot on the first step)
 
73
  {"type": "text", "text": "\n</observation>"}]
74
 
75
 
76
+ def restore_tool_calls(msg) -> str:
77
+ """vllm-mlx returns the call in message.tool_calls; put it back as the <tool_call> XML the model wrote."""
78
+ text = msg.content or ""
79
+ if "<think>" not in text and "</think>" not in text and getattr(msg, "reasoning_content", None):
80
+ text = f"<think>\n{msg.reasoning_content}\n</think>\n\n{text}"
81
+ if getattr(msg, "tool_calls", None) and "<tool_call>" not in text:
82
+ for tc in msg.tool_calls:
83
+ args = json.loads(tc.function.arguments or "{}")
84
+ params = "".join(f"<parameter={k}>\n{v if isinstance(v, str) else json.dumps(v, ensure_ascii=False)}\n</parameter>\n"
85
+ for k, v in args.items())
86
+ text += f"\n\n<tool_call>\n<function={tc.function.name}>\n{params}</function>\n</tool_call>"
87
+ return text
88
+
89
+
90
  def drop_images(content: list[dict]) -> list[dict]:
91
  """Past turns keep the observation text but lose the screenshot."""
92
  out = [c for c in content if c["type"] == "text"]
 
163
  return page, fb + (" Note: no visible navigation or new tab detected." if name.endswith("click") else "")
164
 
165
 
166
+ async def run(task: str, start_url: str, endpoint: str, model: str, max_steps: int,
167
+ tool_choice: str = "none", headful: bool = False):
168
  client = OpenAI(base_url=endpoint, api_key="EMPTY")
169
  async with async_playwright() as pw:
170
+ browser = await pw.chromium.launch(headless=not headful)
171
  ctx = await browser.new_context(viewport={"width": 1280, "height": 720})
172
  page = await ctx.new_page()
173
  await page.goto(start_url)
 
175
  first[0]["text"] = f"<user_request>\n{task}\n</user_request>\n" + first[0]["text"]
176
  msgs = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": first}]
177
  for step in range(max_steps):
178
+ out = client.chat.completions.create(model=model, messages=msgs, tools=TOOLS,
179
+ tool_choice=tool_choice, **SAMPLING)
180
+ text = restore_tool_calls(out.choices[0].message)
 
181
  msgs.append({"role": "assistant", "content": text}) # keep the full <think> + <tool_call>
182
  try:
183
  name, args = parse_call(text)
 
202
  ap.add_argument("--task", required=True)
203
  ap.add_argument("--start-url", default="https://www.google.com/")
204
  ap.add_argument("--endpoint", default="http://localhost:8000/v1")
205
+ ap.add_argument("--model", default="KORMo-Web-4B", help="the server's --served-model-name")
206
  ap.add_argument("--max-steps", type=int, default=50)
207
+ ap.add_argument("--tool-choice", default="none", choices=["none", "auto"],
208
+ help="none for vLLM, auto for vllm-mlx (Apple silicon)")
209
+ ap.add_argument("--headful", action="store_true", help="show the browser window")
210
  a = ap.parse_args()
211
+ asyncio.run(run(a.task, a.start_url, a.endpoint, a.model, a.max_steps, a.tool_choice, a.headful))