| # ADR-0014: Serve Small Models on Modal, One App Per Provider |
|
|
| ## Status |
|
|
| Accepted. **Amended by [ADR-0019](0019-single-model-catalogue-no-cloud-path.md):** |
| the model catalogue moved to the stdlib-only `modal/catalogue.py` (single source of |
| truth, shared with the engine); `registry.py` is now a back-compat re-export, and |
| the engine binds endpoints from the catalogue + `MODAL_WORKSPACE` rather than |
| `OPENAI_BASE_URL`. |
|
|
| ## Context |
|
|
| The engine routes agent roles to small models through an OpenAI-compatible |
| interface, but until now there was no hosted backend behind it — only the local |
| deterministic stub. We need real small models (all under the 32B cap, with a |
| ≤4B Tiny Titan tier) served as APIs the engine can call, without coupling the |
| engine to any single inference vendor. |
|
|
| We want the serving layer to be scalable (autoscaling, pay-per-use), extensible |
| (adding a model or provider should be trivial), and configurable per task (GPU, |
| context length, concurrency, tool/reasoning parsers, multimodal limits). |
|
|
| ## Decision |
|
|
| Add a `modal/` folder that serves models on Modal as serverless, |
| OpenAI-compatible endpoints (vLLM behind an autoscaling web server). |
|
|
| - **One Modal app per provider** (`nvidia-llms`, `openbmb-llms`, `google-llms`). |
| Providers deploy, scale, and fail independently. |
| - **One reusable serving path** in `service.py` (`ModelConfig` + `register_model`) |
| shared by every app, so the vLLM/Modal best practices are written once. |
| - **Configuration is data** in `registry.py`: a model is one `ModelConfig`; a |
| provider is one app file. This mirrors the project's "config, not code" |
| invariant (ADR-0011). |
| - Weights and the vLLM compile cache live in **shared Volumes**, so a model |
| pulled once is warm across every provider app. |
|
|
| ## Consequences |
|
|
| - The engine talks to any endpoint via the OpenAI SDK by setting |
| `OPENAI_BASE_URL`; model roles (`MODEL_TINY/FAST/BALANCED/STRONG`) map to the |
| endpoint whose size fits the role. |
| - Vendor isolation: a provider can be added, retuned, or removed without |
| touching the others or the engine. |
| - Gated repos (Gemma, the Nemotron repos used here) require a Hugging Face token |
| in the `huggingface-secret` Modal Secret; ungated models deploy without it. |
| - Endpoints are public by default; bearer-token auth is opt-in at deploy time |
| (`MODAL_LLM_REQUIRE_AUTH=1`) and supplied via the `llm-api-key` Secret as the |
| `VLLM_API_KEY` env var — secrets are never hard-coded. |
| - The API is OpenAI-compatible: each endpoint self-documents at `/docs` and |
| `/openapi.json`, and a checked-in `modal/openapi.yaml` (3.1) plus |
| `modal/docs/openapi.md` document the shared surface. |
| - vLLM tool/reasoning parser names are version-specific and left conservative; |
| enable per model once verified against the deployed vLLM version. |
| - Modal's docs index is mirrored at `modal/docs/modal-llms.txt` and refreshed |
| when the pinned vLLM/Modal versions change (ADR-0004, document-as-we-build). |
|
|
