agharsallah commited on
Commit ·
7cedfb2
1
Parent(s): 49aa5c4
fix: update endpoint URLs to reflect new app naming conventions
Browse files- modal/README.md +1 -1
- modal/app_nvidia.py +1 -1
- modal/client.py +6 -1
- modal/docs/deploying.md +15 -5
- modal/docs/openapi.md +11 -8
- modal/openapi.yaml +11 -8
modal/README.md
CHANGED
|
@@ -50,7 +50,7 @@ modal secret create huggingface-secret HF_TOKEN=hf_xxx # for gated repos
|
|
| 50 |
|
| 51 |
modal deploy modal/app_nvidia.py
|
| 52 |
python modal/client.py \
|
| 53 |
-
--base-url https://<workspace>--nemotron-3-nano-4b.modal.run/v1 \
|
| 54 |
--model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
|
| 55 |
--prompt "Hello from the wood."
|
| 56 |
```
|
|
|
|
| 50 |
|
| 51 |
modal deploy modal/app_nvidia.py
|
| 52 |
python modal/client.py \
|
| 53 |
+
--base-url https://<workspace>--nvidia-llms-nemotron-3-nano-4b.modal.run/v1 \
|
| 54 |
--model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
|
| 55 |
--prompt "Hello from the wood."
|
| 56 |
```
|
modal/app_nvidia.py
CHANGED
|
@@ -4,7 +4,7 @@ Deploy: modal deploy modal/app_nvidia.py
|
|
| 4 |
Serve dev: modal serve modal/app_nvidia.py
|
| 5 |
|
| 6 |
Each model gets its own OpenAI-compatible endpoint at
|
| 7 |
-
``https://<workspace>--<endpoint-name>.modal.run/v1`` (one per model in
|
| 8 |
``NVIDIA_MODELS``). Add or retune models in ``registry.py``.
|
| 9 |
"""
|
| 10 |
|
|
|
|
| 4 |
Serve dev: modal serve modal/app_nvidia.py
|
| 5 |
|
| 6 |
Each model gets its own OpenAI-compatible endpoint at
|
| 7 |
+
``https://<workspace>--nvidia-llms-<endpoint-name>.modal.run/v1`` (one per model in
|
| 8 |
``NVIDIA_MODELS``). Add or retune models in ``registry.py``.
|
| 9 |
"""
|
| 10 |
|
modal/client.py
CHANGED
|
@@ -2,10 +2,15 @@
|
|
| 2 |
|
| 3 |
Usage:
|
| 4 |
python modal/client.py \\
|
| 5 |
-
--base-url https://<workspace>--gemma-4-12b.modal.run/v1 \\
|
| 6 |
--model google/gemma-4-12B \\
|
| 7 |
--prompt "Describe a mossy ticket booth in the wood."
|
| 8 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 9 |
The endpoints speak the OpenAI REST API, so the official ``openai`` SDK works
|
| 10 |
unchanged — the engine can point ``OPENAI_BASE_URL`` at any of these URLs.
|
| 11 |
"""
|
|
|
|
| 2 |
|
| 3 |
Usage:
|
| 4 |
python modal/client.py \\
|
| 5 |
+
--base-url https://<workspace>--google-llms-gemma-4-12b.modal.run/v1 \\
|
| 6 |
--model google/gemma-4-12B \\
|
| 7 |
--prompt "Describe a mossy ticket booth in the wood."
|
| 8 |
|
| 9 |
+
The endpoint URL is https://<workspace>--<app-name>-<endpoint-name>.modal.run/v1,
|
| 10 |
+
where <app-name> is the modal.App (nvidia-llms / openbmb-llms / google-llms) and
|
| 11 |
+
<endpoint-name> is the per-model slug. --model is the served model id (the HF
|
| 12 |
+
repo id), NOT the URL slug.
|
| 13 |
+
|
| 14 |
The endpoints speak the OpenAI REST API, so the official ``openai`` SDK works
|
| 15 |
unchanged — the engine can point ``OPENAI_BASE_URL`` at any of these URLs.
|
| 16 |
"""
|
modal/docs/deploying.md
CHANGED
|
@@ -38,19 +38,29 @@ shadow it).
|
|
| 38 |
|
| 39 |
## Endpoints
|
| 40 |
|
| 41 |
-
Each model becomes its own OpenAI-compatible endpoint
|
| 42 |
-
`endpoint_name`:
|
| 43 |
|
| 44 |
```
|
| 45 |
-
https://<workspace>--<endpoint-name>.modal.run/v1
|
| 46 |
```
|
| 47 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 48 |
Standard routes: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, plus
|
| 49 |
`/docs` for the Swagger UI. Smoke-test one:
|
| 50 |
|
| 51 |
```bash
|
| 52 |
python modal/client.py \
|
| 53 |
-
--base-url https://<workspace>--gemma-4-12b.modal.run/v1 \
|
| 54 |
--model google/gemma-4-12B \
|
| 55 |
--prompt "Describe a mossy ticket booth in the wood."
|
| 56 |
```
|
|
@@ -130,7 +140,7 @@ Endpoints are OpenAI-compatible, so the engine talks to them through the OpenAI
|
|
| 130 |
SDK. Point a model role at a deployed endpoint:
|
| 131 |
|
| 132 |
```bash
|
| 133 |
-
export OPENAI_BASE_URL="https://<workspace>--nemotron-3-nano-4b.modal.run/v1"
|
| 134 |
export OPENAI_API_KEY="EMPTY" # or the configured VLLM_API_KEY
|
| 135 |
export MODEL_TINY="nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16"
|
| 136 |
```
|
|
|
|
| 38 |
|
| 39 |
## Endpoints
|
| 40 |
|
| 41 |
+
Each model becomes its own OpenAI-compatible endpoint. Modal builds the URL from
|
| 42 |
+
the `modal.App` name **and** the function's `endpoint_name`:
|
| 43 |
|
| 44 |
```
|
| 45 |
+
https://<workspace>--<app-name>-<endpoint-name>.modal.run/v1
|
| 46 |
```
|
| 47 |
|
| 48 |
+
`<app-name>` is `nvidia-llms`, `openbmb-llms`, or `google-llms` (one per provider
|
| 49 |
+
app); `<endpoint-name>` is the per-model slug. e.g. the Nemotron 4B endpoint is
|
| 50 |
+
`https://<workspace>--nvidia-llms-nemotron-3-nano-4b.modal.run/v1`.
|
| 51 |
+
|
| 52 |
+
> **Model id vs URL slug.** The `--model` value (and the `"model"` field in a raw
|
| 53 |
+
> request) is the *served model id* — the HF repo id, e.g.
|
| 54 |
+
> `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` — because `served_model_name` defaults to
|
| 55 |
+
> the repo `name`. It is **not** the URL slug (`nemotron-3-nano-4b`). Call
|
| 56 |
+
> `/v1/models` on any endpoint to see the exact id it serves.
|
| 57 |
+
|
| 58 |
Standard routes: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, plus
|
| 59 |
`/docs` for the Swagger UI. Smoke-test one:
|
| 60 |
|
| 61 |
```bash
|
| 62 |
python modal/client.py \
|
| 63 |
+
--base-url https://<workspace>--google-llms-gemma-4-12b.modal.run/v1 \
|
| 64 |
--model google/gemma-4-12B \
|
| 65 |
--prompt "Describe a mossy ticket booth in the wood."
|
| 66 |
```
|
|
|
|
| 140 |
SDK. Point a model role at a deployed endpoint:
|
| 141 |
|
| 142 |
```bash
|
| 143 |
+
export OPENAI_BASE_URL="https://<workspace>--nvidia-llms-nemotron-3-nano-4b.modal.run/v1"
|
| 144 |
export OPENAI_API_KEY="EMPTY" # or the configured VLLM_API_KEY
|
| 145 |
export MODEL_TINY="nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16"
|
| 146 |
```
|
modal/docs/openapi.md
CHANGED
|
@@ -8,8 +8,8 @@ the familiar OpenAI one. There are two sources of truth:
|
|
| 8 |
`/docs`:
|
| 9 |
|
| 10 |
```
|
| 11 |
-
https://<workspace>--<endpoint-name>.modal.run/openapi.json
|
| 12 |
-
https://<workspace>--<endpoint-name>.modal.run/docs
|
| 13 |
```
|
| 14 |
|
| 15 |
- **Checked-in spec** — [`../openapi.yaml`](../openapi.yaml) documents the
|
|
@@ -19,11 +19,14 @@ the familiar OpenAI one. There are two sources of truth:
|
|
| 19 |
## Base URL
|
| 20 |
|
| 21 |
```
|
| 22 |
-
https://<workspace>--<endpoint-name>.modal.run/v1
|
| 23 |
```
|
| 24 |
|
| 25 |
-
One server per model; `<endpoint-name>`
|
| 26 |
-
`
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
## Endpoints
|
| 29 |
|
|
@@ -60,7 +63,7 @@ otherwise. Clients pass the same token as their API key.
|
|
| 60 |
### curl
|
| 61 |
|
| 62 |
```bash
|
| 63 |
-
curl https://<workspace>--gemma-4-12b.modal.run/v1/chat/completions \
|
| 64 |
-H "Authorization: Bearer $LLM_API_KEY" \
|
| 65 |
-H "Content-Type: application/json" \
|
| 66 |
-d '{
|
|
@@ -76,7 +79,7 @@ curl https://<workspace>--gemma-4-12b.modal.run/v1/chat/completions \
|
|
| 76 |
from openai import OpenAI
|
| 77 |
|
| 78 |
client = OpenAI(
|
| 79 |
-
base_url="https://<workspace>--gemma-4-12b.modal.run/v1",
|
| 80 |
api_key=os.environ["LLM_API_KEY"], # any value when auth is off
|
| 81 |
)
|
| 82 |
resp = client.chat.completions.create(
|
|
@@ -96,5 +99,5 @@ the `LLM_API_KEY` environment variable.
|
|
| 96 |
openapi-generator-cli generate -i modal/openapi.yaml -g python -o ./gen
|
| 97 |
|
| 98 |
# ...or from a live endpoint's exact spec:
|
| 99 |
-
curl -s https://<workspace>--gemma-4-12b.modal.run/openapi.json -o openapi.json
|
| 100 |
```
|
|
|
|
| 8 |
`/docs`:
|
| 9 |
|
| 10 |
```
|
| 11 |
+
https://<workspace>--<app-name>-<endpoint-name>.modal.run/openapi.json
|
| 12 |
+
https://<workspace>--<app-name>-<endpoint-name>.modal.run/docs
|
| 13 |
```
|
| 14 |
|
| 15 |
- **Checked-in spec** — [`../openapi.yaml`](../openapi.yaml) documents the
|
|
|
|
| 19 |
## Base URL
|
| 20 |
|
| 21 |
```
|
| 22 |
+
https://<workspace>--<app-name>-<endpoint-name>.modal.run/v1
|
| 23 |
```
|
| 24 |
|
| 25 |
+
One server per model; the URL label is `<app-name>-<endpoint-name>` — the
|
| 26 |
+
`modal.App` (`nvidia-llms` / `openbmb-llms` / `google-llms`) plus the model's
|
| 27 |
+
`endpoint_name` from `registry.py` (e.g. `google-llms-gemma-4-12b`,
|
| 28 |
+
`nvidia-llms-nemotron-3-nano-4b`). The `model` you send is the *served id* (the
|
| 29 |
+
HF repo id), not this slug.
|
| 30 |
|
| 31 |
## Endpoints
|
| 32 |
|
|
|
|
| 63 |
### curl
|
| 64 |
|
| 65 |
```bash
|
| 66 |
+
curl https://<workspace>--google-llms-gemma-4-12b.modal.run/v1/chat/completions \
|
| 67 |
-H "Authorization: Bearer $LLM_API_KEY" \
|
| 68 |
-H "Content-Type: application/json" \
|
| 69 |
-d '{
|
|
|
|
| 79 |
from openai import OpenAI
|
| 80 |
|
| 81 |
client = OpenAI(
|
| 82 |
+
base_url="https://<workspace>--google-llms-gemma-4-12b.modal.run/v1",
|
| 83 |
api_key=os.environ["LLM_API_KEY"], # any value when auth is off
|
| 84 |
)
|
| 85 |
resp = client.chat.completions.create(
|
|
|
|
| 99 |
openapi-generator-cli generate -i modal/openapi.yaml -g python -o ./gen
|
| 100 |
|
| 101 |
# ...or from a live endpoint's exact spec:
|
| 102 |
+
curl -s https://<workspace>--google-llms-gemma-4-12b.modal.run/openapi.json -o openapi.json
|
| 103 |
```
|
modal/openapi.yaml
CHANGED
|
@@ -19,15 +19,18 @@ servers:
|
|
| 19 |
default: your-workspace
|
| 20 |
description: Your Modal workspace slug.
|
| 21 |
endpoint:
|
| 22 |
-
default: gemma-4-12b
|
| 23 |
-
description:
|
|
|
|
|
|
|
|
|
|
| 24 |
enum:
|
| 25 |
-
- nemotron-3-nano-30b
|
| 26 |
-
- nemotron-3-nano-4b
|
| 27 |
-
- minicpm-o-4-5
|
| 28 |
-
- minicpm-4-1-8b
|
| 29 |
-
- gemma-4-26b
|
| 30 |
-
- gemma-4-12b
|
| 31 |
security:
|
| 32 |
- bearerAuth: []
|
| 33 |
paths:
|
|
|
|
| 19 |
default: your-workspace
|
| 20 |
description: Your Modal workspace slug.
|
| 21 |
endpoint:
|
| 22 |
+
default: google-llms-gemma-4-12b
|
| 23 |
+
description: >
|
| 24 |
+
The Modal function label: <app-name>-<endpoint_name>, where app-name is
|
| 25 |
+
the modal.App (nvidia-llms / openbmb-llms / google-llms) and endpoint_name
|
| 26 |
+
is the per-model slug from registry.py.
|
| 27 |
enum:
|
| 28 |
+
- nvidia-llms-nemotron-3-nano-30b
|
| 29 |
+
- nvidia-llms-nemotron-3-nano-4b
|
| 30 |
+
- openbmb-llms-minicpm-o-4-5
|
| 31 |
+
- openbmb-llms-minicpm-4-1-8b
|
| 32 |
+
- google-llms-gemma-4-26b
|
| 33 |
+
- google-llms-gemma-4-12b
|
| 34 |
security:
|
| 35 |
- bearerAuth: []
|
| 36 |
paths:
|