feat: add Modal model-serving layer, one app per provider

Serve small models as serverless, OpenAI-compatible endpoints on Modal
(vLLM behind an autoscaling web server). One app per provider keeps
deploy/scale/blast-radius isolated; one reusable serving path keeps adding
a model to a single ModelConfig.

- service.py: ModelConfig + image/command builder + register_model
- registry.py: declarative catalogue grouped by provider
- app_{nvidia,openbmb,google}.py: one Modal app each (6 models, all <=32B)
- client.py: OpenAI-SDK smoke-test client
- docs/: deploy guide + in-repo Modal docs mirror; ADR-0005

Co-Authored-By: Codex <codex@openai.com>

README.md

@@ -47,6 +47,10 @@ docs/
 scripts/
   new_journal_entry.py      Creates dated build log entries
   snapshot_progress.py      Updates docs/blog/building-in-public.md from journal
+modal/
+  service.py                Reusable vLLM serving layer (OpenAI-compatible)
+  registry.py               Declarative model catalogue, grouped by provider
+  app_*.py                  One Modal app per provider (nvidia/openbmb/google)
 ```
 
 ## Development Loop

docs/adr/0005-modal-model-serving.md

@@ -0,0 +1,46 @@
+# ADR-0005: Serve Small Models on Modal, One App Per Provider
+
+## Status
+
+Accepted
+
+## Context
+
+The engine routes agent roles to small models through an OpenAI-compatible
+interface, but until now there was no hosted backend behind it — only the local
+deterministic stub. We need real small models (all under the 32B cap, with a
+≤4B Tiny Titan tier) served as APIs the engine can call, without coupling the
+engine to any single inference vendor.
+
+We want the serving layer to be scalable (autoscaling, pay-per-use), extensible
+(adding a model or provider should be trivial), and configurable per task (GPU,
+context length, concurrency, tool/reasoning parsers, multimodal limits).
+
+## Decision
+
+Add a `modal/` folder that serves models on Modal as serverless,
+OpenAI-compatible endpoints (vLLM behind an autoscaling web server).
+
+- **One Modal app per provider** (`nvidia-llms`, `openbmb-llms`, `google-llms`).
+  Providers deploy, scale, and fail independently.
+- **One reusable serving path** in `service.py` (`ModelConfig` + `register_model`)
+  shared by every app, so the vLLM/Modal best practices are written once.
+- **Configuration is data** in `registry.py`: a model is one `ModelConfig`; a
+  provider is one app file. This mirrors the project's "config, not code"
+  invariant (see ADR for declarative worlds).
+- Weights and the vLLM compile cache live in **shared Volumes**, so a model
+  pulled once is warm across every provider app.
+
+## Consequences
+
+- The engine talks to any endpoint via the OpenAI SDK by setting
+  `OPENAI_BASE_URL`; model roles (`MODEL_TINY/FAST/BALANCED/STRONG`) map to the
+  endpoint whose size fits the role.
+- Vendor isolation: a provider can be added, retuned, or removed without
+  touching the others or the engine.
+- Gated repos (Gemma, the Nemotron repos used here) require a Hugging Face token
+  in the `huggingface-secret` Modal Secret; ungated models deploy without it.
+- vLLM tool/reasoning parser names are version-specific and left conservative;
+  enable per model once verified against the deployed vLLM version.
+- Modal's docs index is mirrored at `modal/docs/modal-llms.txt` and refreshed
+  when the pinned vLLM/Modal versions change (ADR-0004, document-as-we-build).

modal/README.md

@@ -0,0 +1,65 @@
+# Modal model serving
+
+Serverless, OpenAI-compatible endpoints for small models, deployed on
+[Modal](https://modal.com). Each provider is an isolated Modal app; all of them
+share one battle-tested serving path (vLLM behind an autoscaling web server) so
+adding a model is a one-line config change.
+
+## Layout
+
+```text
+modal/
+  service.py        Reusable serving layer: ModelConfig, image + vllm command,
+                    register_model() (provider-agnostic).
+  registry.py       Declarative catalogue of every model, grouped by provider.
+  app_nvidia.py     App "nvidia-llms"  — Nemotron 3 Nano 30B + 4B.
+  app_openbmb.py    App "openbmb-llms" — MiniCPM-o 4.5 + MiniCPM4.1-8B.
+  app_google.py     App "google-llms"  — Gemma 4 26B + 12B.
+  client.py         OpenAI-SDK smoke-test client for any endpoint.
+  requirements.txt  Deploy/client tooling (vLLM lives in the container image).
+  docs/
+    deploying.md    Deploy, configure, auth, GPU sizing, engine integration.
+    modal-llms.txt  In-repo mirror of Modal's docs index, kept updated.
+```
+
+## Models
+
+| Provider | App            | Model                                   | Endpoint name         | GPU     |
+| -------- | -------------- | --------------------------------------- | --------------------- | ------- |
+| NVIDIA   | `nvidia-llms`  | NVIDIA-Nemotron-3-Nano-30B-A3B-BF16     | `nemotron-3-nano-30b` | H200:1  |
+| NVIDIA   | `nvidia-llms`  | NVIDIA-Nemotron-3-Nano-4B-BF16          | `nemotron-3-nano-4b`  | L4:1    |
+| OpenBMB  | `openbmb-llms` | MiniCPM-o-4_5 (omni)                    | `minicpm-o-4-5`       | L40S:1  |
+| OpenBMB  | `openbmb-llms` | MiniCPM4.1-8B                           | `minicpm-4-1-8b`      | L40S:1  |
+| Google   | `google-llms`  | gemma-4-26B-A4B-it                      | `gemma-4-26b`         | H200:1  |
+| Google   | `google-llms`  | gemma-4-12B                             | `gemma-4-12b`         | L40S:1  |
+
+Every endpoint stays under the hackathon's 32B cap; `nemotron-3-nano-4b` is the
+≤4B Tiny Titan tier.
+
+## Quick start
+
+```bash
+pip install -r modal/requirements.txt
+modal token new
+modal secret create huggingface-secret HF_TOKEN=hf_xxx   # for gated repos
+
+modal deploy modal/app_nvidia.py
+python modal/client.py \
+  --base-url https://<workspace>--nemotron-3-nano-4b.modal.run/v1 \
+  --model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 \
+  --prompt "Hello from the wood."
+```
+
+See [`docs/deploying.md`](docs/deploying.md) for configuration, auth, GPU
+sizing, and how to add models/providers or wire endpoints into the engine.
+
+## Why this shape
+
+- **Each provider in its own app** — independent deploy, scaling, and blast
+  radius; one provider's outage or redeploy never touches another.
+- **Scalable** — serverless autoscaling, input concurrency, a shared weight
+  cache (pull once, warm everywhere), and per-model `min_containers` warm pools.
+- **Extensible** — add a model = one `ModelConfig`; add a provider = one app
+  file. The serving path is written once in `service.py`.
+- **Configurable per task** — GPU, context length, concurrency, tool/reasoning
+  parsers, and multimodal limits are all data in `registry.py`.

modal/app_google.py

@@ -0,0 +1,20 @@
+"""Google model-serving app (Gemma family).
+
+Deploy:    modal deploy modal/app_google.py
+Serve dev: modal serve modal/app_google.py
+
+Gemma repos are gated: create the ``huggingface-secret`` (HF_TOKEN=...) and
+accept the model license on Hugging Face before deploying. Each model gets its
+own OpenAI-compatible endpoint (one per model in ``GOOGLE_MODELS``).
+"""
+
+from __future__ import annotations
+
+import modal
+
+from registry import GOOGLE_MODELS
+from service import register_all
+
+app = modal.App("google-llms")
+
+register_all(app, GOOGLE_MODELS)

modal/app_nvidia.py

@@ -0,0 +1,20 @@
+"""NVIDIA model-serving app (Nemotron family).
+
+Deploy:    modal deploy modal/app_nvidia.py
+Serve dev: modal serve modal/app_nvidia.py
+
+Each model gets its own OpenAI-compatible endpoint at
+``https://<workspace>--<endpoint-name>.modal.run/v1`` (one per model in
+``NVIDIA_MODELS``). Add or retune models in ``registry.py``.
+"""
+
+from __future__ import annotations
+
+import modal
+
+from registry import NVIDIA_MODELS
+from service import register_all
+
+app = modal.App("nvidia-llms")
+
+register_all(app, NVIDIA_MODELS)

modal/app_openbmb.py

@@ -0,0 +1,20 @@
+"""OpenBMB model-serving app (MiniCPM family).
+
+Deploy:    modal deploy modal/app_openbmb.py
+Serve dev: modal serve modal/app_openbmb.py
+
+Each model gets its own OpenAI-compatible endpoint (one per model in
+``OPENBMB_MODELS``). MiniCPM-o is omni-modal; see ``registry.py`` for the
+multimodal limits and the extra media backends baked into its image.
+"""
+
+from __future__ import annotations
+
+import modal
+
+from registry import OPENBMB_MODELS
+from service import register_all
+
+app = modal.App("openbmb-llms")
+
+register_all(app, OPENBMB_MODELS)

modal/client.py

@@ -0,0 +1,41 @@
+"""Tiny OpenAI-compatible client for smoke-testing a deployed endpoint.
+
+Usage:
+    python modal/client.py \\
+        --base-url https://<workspace>--gemma-4-12b.modal.run/v1 \\
+        --model google/gemma-4-12B \\
+        --prompt "Describe a mossy ticket booth in the wood."
+
+The endpoints speak the OpenAI REST API, so the official ``openai`` SDK works
+unchanged — the engine can point ``OPENAI_BASE_URL`` at any of these URLs.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--base-url", required=True, help="endpoint URL ending in /v1")
+    parser.add_argument("--model", required=True, help="served model id")
+    parser.add_argument("--prompt", default="Say hello in one sentence.")
+    parser.add_argument("--max-tokens", type=int, default=256)
+    args = parser.parse_args()
+
+    from openai import OpenAI
+
+    # vLLM accepts any token unless an API key was configured on the server.
+    client = OpenAI(base_url=args.base_url, api_key=os.environ.get("MODAL_LLM_KEY", "EMPTY"))
+
+    response = client.chat.completions.create(
+        model=args.model,
+        messages=[{"role": "user", "content": args.prompt}],
+        max_tokens=args.max_tokens,
+    )
+    print(response.choices[0].message.content)
+
+
+if __name__ == "__main__":
+    main()

modal/docs/deploying.md

@@ -0,0 +1,126 @@
+# Deploying & configuring the model-serving apps
+
+This guide covers prerequisites, deployment, configuration knobs, auth, GPU
+sizing, and wiring the endpoints into the engine.
+
+## Prerequisites
+
+```bash
+pip install -r modal/requirements.txt
+modal token new            # one-time auth with your Modal workspace
+```
+
+Gated repos (Gemma, and the Nemotron repos here) require a Hugging Face token.
+Accept each model's license on its Hugging Face page, then create the secret:
+
+```bash
+modal secret create huggingface-secret HF_TOKEN=hf_xxx
+```
+
+Only models with `gated=True` mount this secret; ungated models deploy without it.
+
+## Deploy
+
+Each provider is its own Modal app, deployed independently:
+
+```bash
+modal deploy modal/app_nvidia.py     # Nemotron 3 Nano 30B + 4B
+modal deploy modal/app_openbmb.py    # MiniCPM-o 4.5 + MiniCPM4.1-8B
+modal deploy modal/app_google.py     # Gemma 4 26B + 12B
+```
+
+Use `modal serve modal/app_<provider>.py` for a hot-reloading dev session.
+
+Run these from the repo root; the script's own directory (`modal/`) is on
+`sys.path`, so `from service import ...` / `from registry import ...` resolve,
+and `import modal` still binds the installed SDK (the folder name does not
+shadow it).
+
+## Endpoints
+
+Each model becomes its own OpenAI-compatible endpoint, named after its
+`endpoint_name`:
+
+```
+https://<workspace>--<endpoint-name>.modal.run/v1
+```
+
+Standard routes: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, plus
+`/docs` for the Swagger UI. Smoke-test one:
+
+```bash
+python modal/client.py \
+  --base-url https://<workspace>--gemma-4-12b.modal.run/v1 \
+  --model google/gemma-4-12B \
+  --prompt "Describe a mossy ticket booth in the wood."
+```
+
+## Configuring models (per task)
+
+All knobs live in `registry.py` as `ModelConfig` fields — no serving code
+changes needed:
+
+| Field                   | Purpose                                                        |
+| ----------------------- | -------------------------------------------------------------- |
+| `gpu`                   | Modal GPU spec, e.g. `H200:1`, `H100:2`, `L40S:1`, `L4:1`.     |
+| `tensor_parallel_size`  | Shard across GPUs; set equal to the GPU count in `gpu`.        |
+| `max_model_len`         | Cap context length to fit memory / tune throughput.            |
+| `max_concurrent_inputs` | Requests multiplexed onto one container before scaling out.    |
+| `scaledown_window`      | Idle seconds before a container stops (cold-start vs. cost).   |
+| `min_containers`        | Keep N warm to eliminate cold starts (always-on cost).         |
+| `reasoning_parser` / `tool_call_parser` / `enable_auto_tool_choice` | OpenAI tool/reasoning features. |
+| `multimodal` / `mm_limits` | Image/audio/video inputs and per-prompt caps.               |
+| `trust_remote_code`     | Required by MiniCPM / Nemotron custom modeling code.           |
+| `extra_vllm_args`       | Raw `vllm serve` flags appended verbatim (escape hatch).       |
+| `extra_pip` / `env`     | Extra image deps / container env (escape hatch).               |
+
+### Add a model
+
+Append one `ModelConfig` to the appropriate provider list in `registry.py`.
+
+### Add a provider
+
+1. Add a `<PROVIDER>_MODELS` list in `registry.py`.
+2. Create `app_<provider>.py` that builds `modal.App("<provider>-llms")` and
+   calls `register_all(app, <PROVIDER>_MODELS)`.
+
+## Auth
+
+Modal web endpoints are public by default. To require a bearer token, either:
+
+- Set `VLLM_API_KEY` on the container (via a `modal.Secret`) so vLLM enforces
+  `Authorization: Bearer <key>`; or
+- Front the endpoint with Modal Proxy Auth Tokens
+  (see `docs/modal-llms.txt` → Proxy Auth Tokens).
+
+## GPU sizing cheatsheet
+
+BF16 weights ≈ 2 bytes/param; leave headroom for the KV cache. MoE models load
+all expert weights even though only a slice activates per token, so size to the
+total parameter count.
+
+| Model                              | Params (total / active) | Starting GPU |
+| ---------------------------------- | ----------------------- | ------------ |
+| Nemotron-3-Nano-30B-A3B            | 30B / ~3B (MoE)         | `H200:1`     |
+| Nemotron-3-Nano-4B                 | 4B (Tiny Titan)         | `L4:1`       |
+| MiniCPM-o-4_5 (omni)               | ~8B + media encoders    | `L40S:1`     |
+| MiniCPM4.1-8B                      | 8B                      | `L40S:1`     |
+| Gemma-4-26B-A4B-it                 | 26B / ~4B (MoE)         | `H200:1`     |
+| Gemma-4-12B                        | 12B                     | `L40S:1`     |
+
+These are starting points. If a container OOMs, lower `max_model_len`, raise the
+GPU tier, or bump `tensor_parallel_size` (and the GPU count) for sharding.
+
+## Engine integration
+
+Endpoints are OpenAI-compatible, so the engine talks to them through the OpenAI
+SDK. Point a model role at a deployed endpoint:
+
+```bash
+export OPENAI_BASE_URL="https://<workspace>--nemotron-3-nano-4b.modal.run/v1"
+export OPENAI_API_KEY="EMPTY"   # or the configured VLLM_API_KEY
+export MODEL_TINY="nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16"
+```
+
+Map the engine's `MODEL_TINY/FAST/BALANCED/STRONG` tiers to the endpoints whose
+size fits each role.

modal/docs/modal-llms.txt

@@ -0,0 +1,100 @@
+# Modal llms.txt
+
+> Modal is a platform for running Python code in the cloud with minimal
+> configuration, especially for serving AI models and high-performance batch
+> processing. It supports fast prototyping, serverless APIs, scheduled jobs,
+> GPU inference, distributed volumes, and sandboxes.
+
+> Local mirror of Modal's llms.txt, kept in-repo so the serving layer stays
+> grounded in current Modal guidance. Refresh from https://modal.com/docs and
+> re-sync when bumping the vLLM/Modal versions in ../service.py.
+
+Important notes:
+
+- Modal's primitives are embedded in Python and tailored for AI/GPU use cases,
+  but they can be used for general-purpose cloud compute.
+- Modal is a serverless platform, meaning you are only billed for resources used
+  and can spin up containers on demand in seconds.
+
+You can sign up for free at [https://modal.com] and get $30/month of credits.
+
+## Patterns this repo relies on
+
+- OpenAI-compatible vLLM serving via `@modal.web_server(port=8000)` launching
+  `vllm serve` as a subprocess (see ../service.py).
+- `@modal.concurrent(max_inputs=...)` to multiplex requests onto one container.
+- Shared `modal.Volume`s for the Hugging Face weight cache and vLLM compile
+  cache, so weights are pulled once and warm across every provider app.
+- `modal.Secret` (`huggingface-secret`, key `HF_TOKEN`) for gated repos.
+- `scaledown_window` / `min_containers` for cold-start vs. cost trade-offs.
+
+## Guide
+
+- [Introduction](https://modal.com/docs/guide)
+- Custom container images
+  - [Defining Images](https://modal.com/docs/guide/images.md)
+  - [Using existing container images](https://modal.com/docs/guide/existing-images.md)
+  - [Fast pull from registry](https://modal.com/docs/guide/fast-pull-from-registry.md)
+- GPUs and other resources
+  - [GPU acceleration](https://modal.com/docs/guide/gpu.md)
+  - [Using CUDA on Modal](https://modal.com/docs/guide/cuda.md)
+  - [Configuring CPU, memory, and disk](https://modal.com/docs/guide/resources.md)
+- Scaling out
+  - [Scaling out](https://modal.com/docs/guide/scale.md)
+  - [Input concurrency](https://modal.com/docs/guide/concurrent-inputs.md)
+  - [Batch processing](https://modal.com/docs/guide/batch-processing.md)
+  - [Job queues](https://modal.com/docs/guide/job-queue.md)
+  - [Dynamic batching](https://modal.com/docs/guide/dynamic-batching.md)
+- Deployment
+  - [Apps, Functions, and entrypoints](https://modal.com/docs/guide/apps.md)
+  - [Managing deployments](https://modal.com/docs/guide/managing-deployments.md)
+  - [Invoking deployed functions](https://modal.com/docs/guide/trigger-deployed-functions.md)
+  - [Continuous deployment](https://modal.com/docs/guide/continuous-deployment.md)
+- Secrets and environment variables
+  - [Secrets](https://modal.com/docs/guide/secrets.md)
+  - [Environment variables](https://modal.com/docs/guide/environment_variables.md)
+- Web Functions
+  - [Web Functions](https://modal.com/docs/guide/webhooks.md)
+  - [Streaming endpoints](https://modal.com/docs/guide/streaming-endpoints.md)
+  - [Web Function URLs](https://modal.com/docs/guide/webhook-urls.md)
+  - [Request timeouts](https://modal.com/docs/guide/webhook-timeouts.md)
+  - [Proxy Auth Tokens](https://modal.com/docs/guide/webhook-proxy-auth.md)
+- Data sharing and storage
+  - [Volumes](https://modal.com/docs/guide/volumes.md)
+  - [Storing model weights](https://modal.com/docs/guide/model-weights.md)
+- Performance
+  - [Cold start performance](https://modal.com/docs/guide/cold-start.md)
+  - [Memory Snapshots](https://modal.com/docs/guide/memory-snapshots.md)
+  - [High-performance LLM inference](https://modal.com/docs/guide/high-performance-llm-inference.md)
+- Reliability and robustness
+  - [Failures and retries](https://modal.com/docs/guide/retries.md)
+  - [Timeouts](https://modal.com/docs/guide/timeouts.md)
+  - [GPU health](https://modal.com/docs/guide/gpu-health.md)
+
+## Examples
+
+- Large language models (LLMs)
+  - [Deploy an OpenAI-compatible LLM service with vLLM](https://modal.com/docs/examples/llm_inference.md)
+  - [Maximize tokens per second in batch processing with vLLM](https://modal.com/docs/examples/vllm_throughput.md)
+  - [Serve an ultra-low-latency chatbot with SGLang](https://modal.com/docs/examples/sglang_low_latency.md)
+  - [Deploy Nemotron 3](https://modal.com/docs/examples/nemotron_inference.md)
+  - [Run a multimodal RAG chatbot to answer questions about PDFs](https://modal.com/docs/examples/chat_with_pdf_vision.md)
+
+## API Reference
+
+- Application construction
+  - [`App`](https://modal.com/docs/reference/modal.App.md)
+  - [`App.function`](https://modal.com/docs/reference/modal.App.md)
+  - [`App.cls`](https://modal.com/docs/reference/modal.App.md)
+- Web integrations
+  - [`web_server`](https://modal.com/docs/reference/modal.web_server.md)
+  - [`fastapi_endpoint`](https://modal.com/docs/reference/modal.fastapi_endpoint.md)
+  - [`asgi_app`](https://modal.com/docs/reference/modal.asgi_app.md)
+- Function semantics
+  - [`concurrent`](https://modal.com/docs/reference/modal.concurrent.md)
+  - [`batched`](https://modal.com/docs/reference/modal.batched.md)
+- Container configuration
+  - [`Image`](https://modal.com/docs/reference/modal.Image.md)
+  - [`Secret`](https://modal.com/docs/reference/modal.Secret.md)
+- Data primitives
+  - [`Volume`](https://modal.com/docs/reference/modal.Volume.md)

modal/registry.py

@@ -0,0 +1,100 @@
+"""Declarative catalogue of every servable model, grouped by provider.
+
+This is the single place to add, remove, or retune a model. Each provider app
+imports only its own list, so providers stay isolated in separate Modal apps.
+
+GPU sizing notes (starting points — tune against real memory use):
+- BF16 weights ≈ 2 bytes/param. Leave headroom for the KV cache.
+- MoE models (A3B / A4B) load all expert weights but only activate a slice,
+  so size GPU memory to the *total* parameter count, not the active count.
+- Cap ``max_model_len`` to trade context length for KV-cache memory / throughput.
+
+Parser names (``reasoning_parser`` / ``tool_call_parser``) are vLLM-version
+specific. They are left conservative here; enable per model once verified
+against the deployed vLLM version, otherwise vLLM rejects an unknown parser.
+"""
+
+from __future__ import annotations
+
+from service import ModelConfig
+
+# --- NVIDIA --------------------------------------------------------------------
+
+NVIDIA_MODELS: list[ModelConfig] = [
+    ModelConfig(
+        name="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
+        endpoint_name="nemotron-3-nano-30b",
+        # 30B total params in BF16 (~60GB) though only ~3B activate per token.
+        gpu="H200:1",
+        max_model_len=32768,
+        trust_remote_code=True,
+        gated=True,
+        max_concurrent_inputs=64,
+    ),
+    ModelConfig(
+        name="nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16",
+        endpoint_name="nemotron-3-nano-4b",
+        # Tiny Titan tier (≤4B): comfortably fits a single 24GB L4.
+        gpu="L4:1",
+        max_model_len=16384,
+        trust_remote_code=True,
+        gated=True,
+        max_concurrent_inputs=32,
+    ),
+]
+
+# --- OpenBMB (MiniCPM) ---------------------------------------------------------
+
+OPENBMB_MODELS: list[ModelConfig] = [
+    ModelConfig(
+        name="openbmb/MiniCPM-o-4_5",
+        endpoint_name="minicpm-o-4-5",
+        # Omni-modal (text + vision + audio). Needs custom code and media backends.
+        gpu="L40S:1",
+        trust_remote_code=True,
+        multimodal=True,
+        mm_limits={"image": 4, "audio": 2, "video": 1},
+        # Audio/vision preprocessing backends pulled into the image.
+        extra_pip=("librosa", "soundfile", "timm"),
+        max_concurrent_inputs=16,
+    ),
+    ModelConfig(
+        name="openbmb/MiniCPM4.1-8B",
+        endpoint_name="minicpm-4-1-8b",
+        gpu="L40S:1",
+        max_model_len=32768,
+        trust_remote_code=True,
+        max_concurrent_inputs=48,
+    ),
+]
+
+# --- Google (Gemma) ------------------------------------------------------------
+
+GOOGLE_MODELS: list[ModelConfig] = [
+    ModelConfig(
+        name="google/gemma-4-26B-A4B-it",
+        endpoint_name="gemma-4-26b",
+        # MoE: ~26B total params (~4B active). Gated repo — needs an HF token.
+        gpu="H200:1",
+        max_model_len=32768,
+        gated=True,
+        reasoning_parser="gemma4",
+        tool_call_parser="gemma4",
+        enable_auto_tool_choice=True,
+        max_concurrent_inputs=64,
+    ),
+    ModelConfig(
+        name="google/gemma-4-12B",
+        endpoint_name="gemma-4-12b",
+        gpu="L40S:1",
+        max_model_len=32768,
+        gated=True,
+        reasoning_parser="gemma4",
+        tool_call_parser="gemma4",
+        enable_auto_tool_choice=True,
+        max_concurrent_inputs=48,
+    ),
+]
+
+# Convenience: every model across providers (handy for tooling / docs).
+ALL_MODELS: list[ModelConfig] = NVIDIA_MODELS + OPENBMB_MODELS + GOOGLE_MODELS

modal/requirements.txt

@@ -0,0 +1,4 @@
+# Tooling for deploying and calling the Modal serving apps.
+# vLLM and CUDA live inside the container image (see service.py), not here.
+modal>=1.0
+openai>=1.40  # only needed for client.py / engine integration

modal/service.py

@@ -0,0 +1,209 @@
+"""Reusable, OpenAI-compatible model-serving layer for Modal.
+
+This module is provider-agnostic. It knows how to take a single ``ModelConfig``
+and turn it into a serverless, autoscaling, OpenAI-compatible HTTP endpoint
+backed by vLLM. Each provider app (``app_nvidia.py``, ``app_openbmb.py``,
+``app_google.py``) imports :func:`register_model` and wires up its own models,
+so providers stay fully isolated in their own Modal apps while sharing one
+battle-tested serving path.
+
+Design goals:
+- **Extensible**: add a model by appending one ``ModelConfig`` to the registry.
+- **Scalable**: serverless autoscaling, input concurrency, shared weight cache.
+- **Configurable per task**: every knob (GPU, context length, parsers,
+  multimodal limits, extra flags) lives in data, not code.
+
+The served endpoints speak the OpenAI REST API (``/v1/chat/completions`,
+``/v1/completions``, ``/v1/models``), so any OpenAI-compatible client can call
+them by pointing ``base_url`` at the deployed URL.
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+
+import modal
+
+# --- Shared serving constants --------------------------------------------------
+
+# Pin the inference stack so deploys are reproducible. Bump deliberately.
+VLLM_VERSION = "0.21.0"
+CUDA_IMAGE = "nvidia/cuda:12.9.0-devel-ubuntu22.04"
+PYTHON_VERSION = "3.12"
+
+# The in-container port vLLM listens on; Modal maps it to a public HTTPS URL.
+VLLM_PORT = 8000
+
+# Cache paths inside the container, backed by shared Volumes (see below).
+HF_CACHE_PATH = "/root/.cache/huggingface"
+VLLM_CACHE_PATH = "/root/.cache/vllm"
+
+# Name of the Modal Secret that holds a Hugging Face token (key: HF_TOKEN).
+# Required only for gated repos (e.g. Gemma). Create it once with:
+#   modal secret create huggingface-secret HF_TOKEN=hf_...
+HF_SECRET_NAME = "huggingface-secret"
+
+# Weights and the vLLM compile cache are shared across every provider app, so a
+# model pulled once is warm for all subsequent deploys and containers.
+hf_cache_vol = modal.Volume.from_name("huggingface-cache", create_if_missing=True)
+vllm_cache_vol = modal.Volume.from_name("vllm-cache", create_if_missing=True)
+
+# Baseline image shared by every text model. Multimodal models extend it via
+# ``ModelConfig.extra_pip`` (see ``build_image``).
+_BASE_ENV = {
+    "HF_HUB_CACHE": HF_CACHE_PATH,
+    "HF_XET_HIGH_PERFORMANCE": "1",  # faster weight downloads
+    "VLLM_LOG_STATS_INTERVAL": "1",
+}
+
+
+# --- Model configuration -------------------------------------------------------
+
+
+@dataclass(frozen=True)
+class ModelConfig:
+    """Everything needed to serve one model as an OpenAI-compatible endpoint.
+
+    Add a new model by constructing one of these in ``registry.py``. Nothing
+    else needs to change.
+    """
+
+    # Identity
+    name: str  # Hugging Face repo id, e.g. "google/gemma-4-12B"
+    endpoint_name: str  # Modal function + URL slug, e.g. "gemma-4-12b"
+    served_model_name: str | None = None  # model id clients pass; defaults to `name`
+    revision: str | None = None  # pin a commit for reproducibility
+
+    # Hardware
+    gpu: str = "L40S:1"  # Modal GPU spec, e.g. "H200:1", "H100:2", "L4:1"
+    tensor_parallel_size: int = 1  # set to GPU count for multi-GPU sharding
+
+    # Inference shape
+    max_model_len: int | None = None  # cap context to fit memory / task
+    trust_remote_code: bool = False  # required by MiniCPM / Nemotron custom code
+
+    # OpenAI feature parsers (vLLM names; leave None if unsupported on the model)
+    reasoning_parser: str | None = None
+    tool_call_parser: str | None = None
+    enable_auto_tool_choice: bool = False
+
+    # Multimodal
+    multimodal: bool = False
+    mm_limits: dict[str, int] | None = None  # e.g. {"image": 4, "audio": 2}
+
+    # Scaling / lifecycle
+    max_concurrent_inputs: int = 64  # requests multiplexed onto one container
+    scaledown_window: int = 15 * 60  # idle seconds before a container stops
+    min_containers: int = 0  # keep N warm to remove cold starts (costs $)
+    startup_timeout: int = 30 * 60  # weight download + load can be slow
+    request_timeout: int = 30 * 60  # max seconds a single request may run
+
+    # Access
+    gated: bool = False  # repo needs a Hugging Face token
+
+    # Escape hatches
+    extra_vllm_args: tuple[str, ...] = ()  # raw flags appended verbatim
+    env: dict[str, str] = field(default_factory=dict)  # extra container env
+    extra_pip: tuple[str, ...] = ()  # extra deps (audio/vision backends, etc.)
+
+    @property
+    def served_name(self) -> str:
+        return self.served_model_name or self.name
+
+
+# --- Image + command construction ----------------------------------------------
+
+
+def build_image(cfg: ModelConfig) -> modal.Image:
+    """Build the container image for a model. Layers are cached and shared, so
+    text models that only differ in env reuse the same base layers."""
+    image = (
+        modal.Image.from_registry(CUDA_IMAGE, add_python=PYTHON_VERSION)
+        .entrypoint([])  # drop the CUDA image's default entrypoint
+        .uv_pip_install(f"vllm=={VLLM_VERSION}")
+        .env(_BASE_ENV)
+    )
+    if cfg.extra_pip:
+        image = image.uv_pip_install(*cfg.extra_pip)
+    if cfg.env:
+        image = image.env(cfg.env)
+    return image
+
+
+def build_command(cfg: ModelConfig) -> list[str]:
+    """Assemble the ``vllm serve`` argv for a model. Returned as a list so we can
+    launch with ``subprocess.Popen`` without a shell (no quoting pitfalls)."""
+    cmd: list[str] = [
+        "vllm",
+        "serve",
+        cfg.name,
+        "--host",
+        "0.0.0.0",
+        "--port",
+        str(VLLM_PORT),
+        "--served-model-name",
+        cfg.served_name,
+        "--tensor-parallel-size",
+        str(cfg.tensor_parallel_size),
+        "--uvicorn-log-level",
+        "info",
+    ]
+    if cfg.revision:
+        cmd += ["--revision", cfg.revision]
+    if cfg.max_model_len:
+        cmd += ["--max-model-len", str(cfg.max_model_len)]
+    if cfg.trust_remote_code:
+        cmd += ["--trust-remote-code"]
+    if cfg.reasoning_parser:
+        cmd += ["--reasoning-parser", cfg.reasoning_parser]
+    if cfg.enable_auto_tool_choice:
+        cmd += ["--enable-auto-tool-choice"]
+    if cfg.tool_call_parser:
+        cmd += ["--tool-call-parser", cfg.tool_call_parser]
+    if cfg.mm_limits:
+        cmd += ["--limit-mm-per-prompt", json.dumps(cfg.mm_limits)]
+    cmd += list(cfg.extra_vllm_args)
+    return cmd
+
+
+# --- Endpoint registration ------------------------------------------------------
+
+
+def register_model(app: modal.App, cfg: ModelConfig) -> modal.Function:
+    """Attach one model to ``app`` as an autoscaling, OpenAI-compatible endpoint.
+
+    The function is serialized (its prebuilt ``vllm serve`` argv is shipped to
+    the container), which lets us register many distinctly-named endpoints from
+    a simple loop without each needing a hand-written module-level function.
+    """
+    image = build_image(cfg)
+    cmd = build_command(cfg)
+    secrets = [modal.Secret.from_name(HF_SECRET_NAME)] if cfg.gated else []
+
+    @app.function(
+        name=cfg.endpoint_name,
+        image=image,
+        gpu=cfg.gpu,
+        volumes={HF_CACHE_PATH: hf_cache_vol, VLLM_CACHE_PATH: vllm_cache_vol},
+        secrets=secrets,
+        scaledown_window=cfg.scaledown_window,
+        min_containers=cfg.min_containers,
+        timeout=cfg.request_timeout,
+        serialized=True,
+    )
+    @modal.concurrent(max_inputs=cfg.max_concurrent_inputs)
+    @modal.web_server(port=VLLM_PORT, startup_timeout=cfg.startup_timeout)
+    def serve():
+        import subprocess
+
+        # vLLM serves the OpenAI REST API on VLLM_PORT; Modal exposes it publicly.
+        subprocess.Popen(cmd)
+
+    return serve
+
+
+def register_all(app: modal.App, configs: list[ModelConfig]) -> None:
+    """Register every model in ``configs`` onto ``app``."""
+    for cfg in configs:
+        register_model(app, cfg)