jungsin3 commited on
Commit
7e7e2f7
ยท
verified ยท
1 Parent(s): 07d6bdb

How to Start: separate Mac and NVIDIA GPU instructions per model

Browse files
Files changed (1) hide show
  1. README.md +24 -16
README.md CHANGED
@@ -91,30 +91,38 @@ The fastest way to try KORMo-Web is the **[KORMo-Web Chrome extension](https://g
91
 
92
  *Setup guide (about 15 min, Korean narration). Parts that showed account tokens have been blurred.*
93
 
94
- **1. Serve the model.** Pick one.
95
 
96
- - **Mac (Apple silicon):** run the 4-bit MLX build with `vllm-mlx`. Simple, but slower.
97
 
98
- ```bash
99
- uv venv --python 3.12 && source .venv/bin/activate
100
- uv pip install vllm-mlx
101
- vllm-mlx serve KORMo-VL/KORMo-Web-4B-mlx-4bit --port 8010 --mllm
102
- ```
103
 
104
- - **GPU server:** serve this model with vLLM. Much faster, and one server can be shared.
 
 
 
 
105
 
106
- ```bash
107
- vllm serve KORMo-VL/KORMo-Web-9B --host 0.0.0.0 --port 8000 \
108
- --served-model-name KORMo-Web-9B \
109
- --mm-encoder-tp-mode data --mm-processor-cache-type shm \
110
- --gdn-prefill-backend triton
111
- ```
112
 
113
- Check it with `curl http://localhost:8010/v1/models` (use your server address and port for vLLM).
 
 
 
 
 
 
 
 
 
 
114
 
115
  **2. Install the extension.** `git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git`, open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked** and choose the cloned folder. Open the **GUI ์—์ด์ „ํŠธ** side panel, type a task and press **์‹คํ–‰** (Run).
116
 
117
- **3. Using a GPU server?** In the panel's **๊ฐœ๋ฐœ์ž ์„ค์ •** (Developer settings), set the server URL (ending in `/v1`), the model name (`KORMo-Web-9B`), and `tool_choice` to `none`. The Mac default needs no change.
 
 
 
118
 
119
  **4. Discord (optional).** Run `node relay/discord-relay.mjs` in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.
120
 
 
91
 
92
  *Setup guide (about 15 min, Korean narration). Parts that showed account tokens have been blurred.*
93
 
94
+ **1. Serve the model** on a Mac or on an NVIDIA GPU.
95
 
96
+ ### Mac (Apple silicon)
97
 
98
+ Run the model with `vllm-mlx`. Keep this terminal open.
 
 
 
 
99
 
100
+ ```bash
101
+ uv venv --python 3.12 && source .venv/bin/activate
102
+ uv pip install vllm-mlx
103
+ vllm-mlx serve KORMo-VL/KORMo-Web-9B --port 8010 --mllm
104
+ ```
105
 
106
+ ### NVIDIA GPU (local PC or server)
 
 
 
 
 
107
 
108
+ Serve the model with vLLM. Much faster, and one server can be shared.
109
+
110
+ ```bash
111
+ pip install vllm
112
+ vllm serve KORMo-VL/KORMo-Web-9B --host 0.0.0.0 --port 8000 \
113
+ --served-model-name KORMo-Web-9B \
114
+ --mm-encoder-tp-mode data --mm-processor-cache-type shm \
115
+ --gdn-prefill-backend triton
116
+ ```
117
+
118
+ Check that it is up with `curl http://localhost:8010/v1/models` (Mac) or `curl http://<server>:8000/v1/models` (vLLM).
119
 
120
  **2. Install the extension.** `git clone https://github.com/MLP-Lab/KORMo-Web-Extension.git`, open `chrome://extensions`, turn on **Developer mode**, click **Load unpacked** and choose the cloned folder. Open the **GUI ์—์ด์ „ํŠธ** side panel, type a task and press **์‹คํ–‰** (Run).
121
 
122
+ **3. Point the extension at your model.** Open the panel's **๊ฐœ๋ฐœ์ž ์„ค์ •** (Developer settings).
123
+
124
+ - **Mac:** keep the server URL `http://localhost:8010/v1` and set the model name to `KORMo-VL/KORMo-Web-9B`. `tool_choice` stays `auto`.
125
+ - **NVIDIA GPU:** set the server URL to `http://<server>:8000/v1` (or `localhost`), the model name to `KORMo-Web-9B`, and `tool_choice` to `none`.
126
 
127
  **4. Discord (optional).** Run `node relay/discord-relay.mjs` in the cloned folder and follow the prompts to send tasks to the agent by mentioning a bot in a channel.
128