# ADR-0014: Serve Small Models on Modal, One App Per Provider

## Status

Accepted. **Amended by [ADR-0019](0019-single-model-catalogue-no-cloud-path.md):**
the model catalogue moved to the stdlib-only `modal/catalogue.py` (single source of
truth, shared with the engine); `registry.py` is now a back-compat re-export, and
the engine binds endpoints from the catalogue + `MODAL_WORKSPACE` rather than
`OPENAI_BASE_URL`.

## Context

The engine routes agent roles to small models through an OpenAI-compatible
interface, but until now there was no hosted backend behind it — only the local
deterministic stub. We need real small models (all under the 32B cap, with a
≤4B Tiny Titan tier) served as APIs the engine can call, without coupling the
engine to any single inference vendor.

We want the serving layer to be scalable (autoscaling, pay-per-use), extensible
(adding a model or provider should be trivial), and configurable per task (GPU,
context length, concurrency, tool/reasoning parsers, multimodal limits).

## Decision

Add a `modal/` folder that serves models on Modal as serverless,
OpenAI-compatible endpoints (vLLM behind an autoscaling web server).

- **One Modal app per provider** (`nvidia-llms`, `openbmb-llms`, `google-llms`).
  Providers deploy, scale, and fail independently.
- **One reusable serving path** in `service.py` (`ModelConfig` + `register_model`)
  shared by every app, so the vLLM/Modal best practices are written once.
- **Configuration is data** in `registry.py`: a model is one `ModelConfig`; a
  provider is one app file. This mirrors the project's "config, not code"
  invariant (ADR-0011).
- Weights and the vLLM compile cache live in **shared Volumes**, so a model
  pulled once is warm across every provider app.

## Consequences

- The engine talks to any endpoint via the OpenAI SDK by setting
  `OPENAI_BASE_URL`; model roles (`MODEL_TINY/FAST/BALANCED/STRONG`) map to the
  endpoint whose size fits the role.
- Vendor isolation: a provider can be added, retuned, or removed without
  touching the others or the engine.
- Gated repos (Gemma, the Nemotron repos used here) require a Hugging Face token
  in the `huggingface-secret` Modal Secret; ungated models deploy without it.
- Endpoints are public by default; bearer-token auth is opt-in at deploy time
  (`MODAL_LLM_REQUIRE_AUTH=1`) and supplied via the `llm-api-key` Secret as the
  `VLLM_API_KEY` env var — secrets are never hard-coded.
- The API is OpenAI-compatible: each endpoint self-documents at `/docs` and
  `/openapi.json`, and a checked-in `modal/openapi.yaml` (3.1) plus
  `modal/docs/openapi.md` document the shared surface.
- vLLM tool/reasoning parser names are version-specific and left conservative;
  enable per model once verified against the deployed vLLM version.
- Modal's docs index is mirrored at `modal/docs/modal-llms.txt` and refreshed
  when the pinned vLLM/Modal versions change (ADR-0004, document-as-we-build).