docs/adr/0014-modal-model-serving.md

ADR-0014: Serve Small Models on Modal, One App Per Provider

Status

Accepted. Amended by ADR-0019: the model catalogue moved to the stdlib-only modal/catalogue.py (single source of truth, shared with the engine); registry.py is now a back-compat re-export, and the engine binds endpoints from the catalogue + MODAL_WORKSPACE rather than OPENAI_BASE_URL.

Context

The engine routes agent roles to small models through an OpenAI-compatible interface, but until now there was no hosted backend behind it — only the local deterministic stub. We need real small models (all under the 32B cap, with a ≤4B Tiny Titan tier) served as APIs the engine can call, without coupling the engine to any single inference vendor.

We want the serving layer to be scalable (autoscaling, pay-per-use), extensible (adding a model or provider should be trivial), and configurable per task (GPU, context length, concurrency, tool/reasoning parsers, multimodal limits).

Decision

Add a modal/ folder that serves models on Modal as serverless, OpenAI-compatible endpoints (vLLM behind an autoscaling web server).

• One Modal app per provider (nvidia-llms, openbmb-llms, google-llms). Providers deploy, scale, and fail independently.
• One reusable serving path in service.py (ModelConfig + register_model) shared by every app, so the vLLM/Modal best practices are written once.
• Configuration is data in registry.py: a model is one ModelConfig; a provider is one app file. This mirrors the project's "config, not code" invariant (ADR-0011).
• Weights and the vLLM compile cache live in shared Volumes, so a model pulled once is warm across every provider app.

Consequences

• The engine talks to any endpoint via the OpenAI SDK by setting OPENAI_BASE_URL; model roles (MODEL_TINY/FAST/BALANCED/STRONG) map to the endpoint whose size fits the role.
• Vendor isolation: a provider can be added, retuned, or removed without touching the others or the engine.
• Gated repos (Gemma, the Nemotron repos used here) require a Hugging Face token in the huggingface-secret Modal Secret; ungated models deploy without it.
• Endpoints are public by default; bearer-token auth is opt-in at deploy time (MODAL_LLM_REQUIRE_AUTH=1) and supplied via the llm-api-key Secret as the VLLM_API_KEY env var — secrets are never hard-coded.
• The API is OpenAI-compatible: each endpoint self-documents at /docs and /openapi.json, and a checked-in modal/openapi.yaml (3.1) plus modal/docs/openapi.md document the shared surface.
• vLLM tool/reasoning parser names are version-specific and left conservative; enable per model once verified against the deployed vLLM version.
• Modal's docs index is mirrored at modal/docs/modal-llms.txt and refreshed when the pinned vLLM/Modal versions change (ADR-0004, document-as-we-build).