jungsin3 commited on
Commit
a15232d
ยท
verified ยท
1 Parent(s): 93cd567

How to Start: separate Mac and NVIDIA GPU instructions per model

Browse files
Files changed (1) hide show
  1. README.md +26 -16
README.md CHANGED
@@ -91,30 +91,40 @@ The fastest way to try KORMo-Web is the **[KORMo-Web Chrome extension](https://g
91
 
92
  *Setup guide (about 15 min, Korean narration). Parts that showed account tokens have been blurred.*
93
 
94
- **1. Serve the model.** Pick one.
95
 
96
- - **Mac (Apple silicon):** run the 4-bit MLX build with `vllm-mlx`. Simple, but slower.
97
 
98
- ```bash
99
- uv venv --python 3.12 && source .venv/bin/activate
100
- uv pip install vllm-mlx
101
- vllm-mlx serve KORMo-VL/KORMo-Web-4B-mlx-4bit --port 8010 --mllm
102
- ```
103
 
104
- - **GPU server:** serve this model with vLLM. Much faster, and one server can be shared.
 
 
 
 
 
 
 
 
105
 
106
- ```bash
107
- vllm serve KORMo-VL/KORMo-Web-4B --host 0.0.0.0 --port 8000 \
108
- --served-model-name KORMo-Web-4B \
109
- --mm-encoder-tp-mode data --mm-processor-cache-type shm \
110
- --gdn-prefill-backend triton
111
- ```
112
 
113
- Check it with `curl http://localhost:8010/v1/models` (use your server address and port for vLLM).
 
 
 
 
 
 
 
 
114
 
115
  **2. Install the extension.** `git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git`, open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked** and choose the cloned folder. Open the **GUI ์—์ด์ „ํŠธ** side panel, type a task and press **์‹คํ–‰** (Run).
116
 
117
- **3. Using a GPU server?** In the panel's **๊ฐœ๋ฐœ์ž ์„ค์ •** (Developer settings), set the server URL (ending in `/v1`), the model name (`KORMo-Web-4B`), and `tool_choice` to `none`. The Mac default needs no change.
 
 
 
118
 
119
  **4. Discord (optional).** Run `node relay/discord-relay.mjs` in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.
120
 
 
91
 
92
  *Setup guide (about 15 min, Korean narration). Parts that showed account tokens have been blurred.*
93
 
94
+ **1. Serve the model** on a Mac or on an NVIDIA GPU.
95
 
96
+ ### Mac (Apple silicon)
97
 
98
+ Run the model with `vllm-mlx`. Keep this terminal open.
 
 
 
 
99
 
100
+ ```bash
101
+ uv venv --python 3.12 && source .venv/bin/activate
102
+ uv pip install vllm-mlx
103
+ vllm-mlx serve KORMo-VL/KORMo-Web-4B --port 8010 --mllm
104
+ ```
105
+
106
+ For a lighter, faster model on a Mac with less memory, use the 4-bit MLX build `KORMo-VL/KORMo-Web-4B-mlx-4bit` instead (the extension's default).
107
+
108
+ ### NVIDIA GPU (local PC or server)
109
 
110
+ Serve the model with vLLM. Much faster, and one server can be shared.
 
 
 
 
 
111
 
112
+ ```bash
113
+ pip install vllm
114
+ vllm serve KORMo-VL/KORMo-Web-4B --host 0.0.0.0 --port 8000 \
115
+ --served-model-name KORMo-Web-4B \
116
+ --mm-encoder-tp-mode data --mm-processor-cache-type shm \
117
+ --gdn-prefill-backend triton
118
+ ```
119
+
120
+ Check that it is up with `curl http://localhost:8010/v1/models` (Mac) or `curl http://<server>:8000/v1/models` (vLLM).
121
 
122
  **2. Install the extension.** `git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git`, open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked** and choose the cloned folder. Open the **GUI ์—์ด์ „ํŠธ** side panel, type a task and press **์‹คํ–‰** (Run).
123
 
124
+ **3. Point the extension at your model.** Open the panel's **๊ฐœ๋ฐœ์ž ์„ค์ •** (Developer settings).
125
+
126
+ - **Mac:** keep the server URL `http://localhost:8010/v1` and set the model name to `KORMo-VL/KORMo-Web-4B`. `tool_choice` stays `auto`.
127
+ - **NVIDIA GPU:** set the server URL to `http://<server>:8000/v1` (or `localhost`), the model name to `KORMo-Web-4B`, and `tool_choice` to `none`.
128
 
129
  **4. Discord (optional).** Run `node relay/discord-relay.mjs` in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.
130