agharsallah commited on
Commit
7cedfb2
·
1 Parent(s): 49aa5c4

fix: update endpoint URLs to reflect new app naming conventions

Browse files
modal/README.md CHANGED
@@ -50,7 +50,7 @@ modal secret create huggingface-secret HF_TOKEN=hf_xxx # for gated repos
50
 
51
  modal deploy modal/app_nvidia.py
52
  python modal/client.py \
53
- --base-url https://<workspace>--nemotron-3-nano-4b.modal.run/v1 \
54
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
55
  --prompt "Hello from the wood."
56
  ```
 
50
 
51
  modal deploy modal/app_nvidia.py
52
  python modal/client.py \
53
+ --base-url https://<workspace>--nvidia-llms-nemotron-3-nano-4b.modal.run/v1 \
54
  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
55
  --prompt "Hello from the wood."
56
  ```
modal/app_nvidia.py CHANGED
@@ -4,7 +4,7 @@ Deploy: modal deploy modal/app_nvidia.py
4
  Serve dev: modal serve modal/app_nvidia.py
5
 
6
  Each model gets its own OpenAI-compatible endpoint at
7
- ``https://<workspace>--<endpoint-name>.modal.run/v1`` (one per model in
8
  ``NVIDIA_MODELS``). Add or retune models in ``registry.py``.
9
  """
10
 
 
4
  Serve dev: modal serve modal/app_nvidia.py
5
 
6
  Each model gets its own OpenAI-compatible endpoint at
7
+ ``https://<workspace>--nvidia-llms-<endpoint-name>.modal.run/v1`` (one per model in
8
  ``NVIDIA_MODELS``). Add or retune models in ``registry.py``.
9
  """
10
 
modal/client.py CHANGED
@@ -2,10 +2,15 @@
2
 
3
  Usage:
4
  python modal/client.py \\
5
- --base-url https://<workspace>--gemma-4-12b.modal.run/v1 \\
6
  --model google/gemma-4-12B \\
7
  --prompt "Describe a mossy ticket booth in the wood."
8
 
 
 
 
 
 
9
  The endpoints speak the OpenAI REST API, so the official ``openai`` SDK works
10
  unchanged — the engine can point ``OPENAI_BASE_URL`` at any of these URLs.
11
  """
 
2
 
3
  Usage:
4
  python modal/client.py \\
5
+ --base-url https://<workspace>--google-llms-gemma-4-12b.modal.run/v1 \\
6
  --model google/gemma-4-12B \\
7
  --prompt "Describe a mossy ticket booth in the wood."
8
 
9
+ The endpoint URL is https://<workspace>--<app-name>-<endpoint-name>.modal.run/v1,
10
+ where <app-name> is the modal.App (nvidia-llms / openbmb-llms / google-llms) and
11
+ <endpoint-name> is the per-model slug. --model is the served model id (the HF
12
+ repo id), NOT the URL slug.
13
+
14
  The endpoints speak the OpenAI REST API, so the official ``openai`` SDK works
15
  unchanged — the engine can point ``OPENAI_BASE_URL`` at any of these URLs.
16
  """
modal/docs/deploying.md CHANGED
@@ -38,19 +38,29 @@ shadow it).
38
 
39
  ## Endpoints
40
 
41
- Each model becomes its own OpenAI-compatible endpoint, named after its
42
- `endpoint_name`:
43
 
44
  ```
45
- https://<workspace>--<endpoint-name>.modal.run/v1
46
  ```
47
 
 
 
 
 
 
 
 
 
 
 
48
  Standard routes: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, plus
49
  `/docs` for the Swagger UI. Smoke-test one:
50
 
51
  ```bash
52
  python modal/client.py \
53
- --base-url https://<workspace>--gemma-4-12b.modal.run/v1 \
54
  --model google/gemma-4-12B \
55
  --prompt "Describe a mossy ticket booth in the wood."
56
  ```
@@ -130,7 +140,7 @@ Endpoints are OpenAI-compatible, so the engine talks to them through the OpenAI
130
  SDK. Point a model role at a deployed endpoint:
131
 
132
  ```bash
133
- export OPENAI_BASE_URL="https://<workspace>--nemotron-3-nano-4b.modal.run/v1"
134
  export OPENAI_API_KEY="EMPTY" # or the configured VLLM_API_KEY
135
  export MODEL_TINY="nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16"
136
  ```
 
38
 
39
  ## Endpoints
40
 
41
+ Each model becomes its own OpenAI-compatible endpoint. Modal builds the URL from
42
+ the `modal.App` name **and** the function's `endpoint_name`:
43
 
44
  ```
45
+ https://<workspace>--<app-name>-<endpoint-name>.modal.run/v1
46
  ```
47
 
48
+ `<app-name>` is `nvidia-llms`, `openbmb-llms`, or `google-llms` (one per provider
49
+ app); `<endpoint-name>` is the per-model slug. e.g. the Nemotron 4B endpoint is
50
+ `https://<workspace>--nvidia-llms-nemotron-3-nano-4b.modal.run/v1`.
51
+
52
+ > **Model id vs URL slug.** The `--model` value (and the `"model"` field in a raw
53
+ > request) is the *served model id* — the HF repo id, e.g.
54
+ > `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` — because `served_model_name` defaults to
55
+ > the repo `name`. It is **not** the URL slug (`nemotron-3-nano-4b`). Call
56
+ > `/v1/models` on any endpoint to see the exact id it serves.
57
+
58
  Standard routes: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, plus
59
  `/docs` for the Swagger UI. Smoke-test one:
60
 
61
  ```bash
62
  python modal/client.py \
63
+ --base-url https://<workspace>--google-llms-gemma-4-12b.modal.run/v1 \
64
  --model google/gemma-4-12B \
65
  --prompt "Describe a mossy ticket booth in the wood."
66
  ```
 
140
  SDK. Point a model role at a deployed endpoint:
141
 
142
  ```bash
143
+ export OPENAI_BASE_URL="https://<workspace>--nvidia-llms-nemotron-3-nano-4b.modal.run/v1"
144
  export OPENAI_API_KEY="EMPTY" # or the configured VLLM_API_KEY
145
  export MODEL_TINY="nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16"
146
  ```
modal/docs/openapi.md CHANGED
@@ -8,8 +8,8 @@ the familiar OpenAI one. There are two sources of truth:
8
  `/docs`:
9
 
10
  ```
11
- https://<workspace>--<endpoint-name>.modal.run/openapi.json
12
- https://<workspace>--<endpoint-name>.modal.run/docs
13
  ```
14
 
15
  - **Checked-in spec** — [`../openapi.yaml`](../openapi.yaml) documents the
@@ -19,11 +19,14 @@ the familiar OpenAI one. There are two sources of truth:
19
  ## Base URL
20
 
21
  ```
22
- https://<workspace>--<endpoint-name>.modal.run/v1
23
  ```
24
 
25
- One server per model; `<endpoint-name>` is the model's `endpoint_name` from
26
- `registry.py` (e.g. `gemma-4-12b`, `nemotron-3-nano-4b`).
 
 
 
27
 
28
  ## Endpoints
29
 
@@ -60,7 +63,7 @@ otherwise. Clients pass the same token as their API key.
60
  ### curl
61
 
62
  ```bash
63
- curl https://<workspace>--gemma-4-12b.modal.run/v1/chat/completions \
64
  -H "Authorization: Bearer $LLM_API_KEY" \
65
  -H "Content-Type: application/json" \
66
  -d '{
@@ -76,7 +79,7 @@ curl https://<workspace>--gemma-4-12b.modal.run/v1/chat/completions \
76
  from openai import OpenAI
77
 
78
  client = OpenAI(
79
- base_url="https://<workspace>--gemma-4-12b.modal.run/v1",
80
  api_key=os.environ["LLM_API_KEY"], # any value when auth is off
81
  )
82
  resp = client.chat.completions.create(
@@ -96,5 +99,5 @@ the `LLM_API_KEY` environment variable.
96
  openapi-generator-cli generate -i modal/openapi.yaml -g python -o ./gen
97
 
98
  # ...or from a live endpoint's exact spec:
99
- curl -s https://<workspace>--gemma-4-12b.modal.run/openapi.json -o openapi.json
100
  ```
 
8
  `/docs`:
9
 
10
  ```
11
+ https://<workspace>--<app-name>-<endpoint-name>.modal.run/openapi.json
12
+ https://<workspace>--<app-name>-<endpoint-name>.modal.run/docs
13
  ```
14
 
15
  - **Checked-in spec** — [`../openapi.yaml`](../openapi.yaml) documents the
 
19
  ## Base URL
20
 
21
  ```
22
+ https://<workspace>--<app-name>-<endpoint-name>.modal.run/v1
23
  ```
24
 
25
+ One server per model; the URL label is `<app-name>-<endpoint-name>` the
26
+ `modal.App` (`nvidia-llms` / `openbmb-llms` / `google-llms`) plus the model's
27
+ `endpoint_name` from `registry.py` (e.g. `google-llms-gemma-4-12b`,
28
+ `nvidia-llms-nemotron-3-nano-4b`). The `model` you send is the *served id* (the
29
+ HF repo id), not this slug.
30
 
31
  ## Endpoints
32
 
 
63
  ### curl
64
 
65
  ```bash
66
+ curl https://<workspace>--google-llms-gemma-4-12b.modal.run/v1/chat/completions \
67
  -H "Authorization: Bearer $LLM_API_KEY" \
68
  -H "Content-Type: application/json" \
69
  -d '{
 
79
  from openai import OpenAI
80
 
81
  client = OpenAI(
82
+ base_url="https://<workspace>--google-llms-gemma-4-12b.modal.run/v1",
83
  api_key=os.environ["LLM_API_KEY"], # any value when auth is off
84
  )
85
  resp = client.chat.completions.create(
 
99
  openapi-generator-cli generate -i modal/openapi.yaml -g python -o ./gen
100
 
101
  # ...or from a live endpoint's exact spec:
102
+ curl -s https://<workspace>--google-llms-gemma-4-12b.modal.run/openapi.json -o openapi.json
103
  ```
modal/openapi.yaml CHANGED
@@ -19,15 +19,18 @@ servers:
19
  default: your-workspace
20
  description: Your Modal workspace slug.
21
  endpoint:
22
- default: gemma-4-12b
23
- description: A model's endpoint_name from registry.py.
 
 
 
24
  enum:
25
- - nemotron-3-nano-30b
26
- - nemotron-3-nano-4b
27
- - minicpm-o-4-5
28
- - minicpm-4-1-8b
29
- - gemma-4-26b
30
- - gemma-4-12b
31
  security:
32
  - bearerAuth: []
33
  paths:
 
19
  default: your-workspace
20
  description: Your Modal workspace slug.
21
  endpoint:
22
+ default: google-llms-gemma-4-12b
23
+ description: >
24
+ The Modal function label: <app-name>-<endpoint_name>, where app-name is
25
+ the modal.App (nvidia-llms / openbmb-llms / google-llms) and endpoint_name
26
+ is the per-model slug from registry.py.
27
  enum:
28
+ - nvidia-llms-nemotron-3-nano-30b
29
+ - nvidia-llms-nemotron-3-nano-4b
30
+ - openbmb-llms-minicpm-o-4-5
31
+ - openbmb-llms-minicpm-4-1-8b
32
+ - google-llms-gemma-4-26b
33
+ - google-llms-gemma-4-12b
34
  security:
35
  - bearerAuth: []
36
  paths: