feat: LiteLLM gateway routing profiles to Modal-served small models

Add LiteLLMProvider; ModelRouter binds each profile (tiny/fast/balanced/strong)
to its Modal vLLM endpoint (workspace env-templated, not hardcoded) and meters
real token + cost usage into the Governor, making hourly_budget_usd live. Env-
gated; offline keeps the deterministic stub. 212 passed, 1 skipped. ADR-0015.

Co-Authored-By: Codex <codex@openai.com>

.env.example

@@ -1,7 +1,8 @@
 # Copy to .env and fill in your values.
-# Without OPENAI_API_KEY the app runs on a deterministic local stub.
+# With no live binding configured (neither OPENAI_API_KEY nor MODAL_WORKSPACE)
+# the app runs on a deterministic local stub — fully offline, no network.
 
-# Required for live model inference
+# Required for live model inference via a generic OpenAI-compatible key
 OPENAI_API_KEY=sk-...
 
 # Optional: point at any OpenAI-compatible endpoint
@@ -23,6 +24,20 @@ MODEL_NAME=gpt-4o-mini
 #   MODEL_BALANCED=qwen2.5-14b-instruct
 #   MODEL_STRONG=qwen2.5-32b-instruct
 
+# LiteLLM gateway → models served on Modal (ADR-0015). Setting MODAL_WORKSPACE
+# activates the live path: config/models.yaml templates each profile's endpoint
+# URL as https://${MODAL_WORKSPACE}--<endpoint>.modal.run/v1 (so the workspace is
+# never hard-coded), routes through LiteLLM, and meters real per-call cost into
+# the Governor's hourly_budget_usd. MODAL_LLM_KEY is the endpoint key — a
+# self-served vLLM endpoint accepts any token (default "EMPTY").
+#   MODAL_WORKSPACE=your-modal-workspace
+#   MODAL_LLM_KEY=EMPTY
+# Alternatively, point a single OpenAI-compatible base URL (also activates live):
+#   MODAL_LLM_BASE_URL=https://your-workspace--gemma-4-12b.modal.run/v1
+MODAL_WORKSPACE=
+MODAL_LLM_KEY=
+MODAL_LLM_BASE_URL=
+
 # Durable event store backend (ADR-0014). When unset, the system uses the
 # in-memory ledger (fully offline). When set, the append-only ledger is persisted
 # through SQLAlchemy. Neon (managed Postgres) form:

config/models.yaml

@@ -1,29 +1,53 @@
-# Logical model profiles → concrete small models.
+# Logical model profiles → concrete small models served on Modal.
 #
 # offline:
-#   null  = auto (deterministic stub unless OPENAI_API_KEY is set)
+#   null  = auto (deterministic stub unless a live binding is configured —
+#           OPENAI_API_KEY, or MODAL_WORKSPACE/MODAL_LLM_BASE_URL for Modal)
 #   true  = always the deterministic stub (reproducible demos / CI)
 #   false = always live inference
 #
-# Every model here is ≤32B (the hackathon cap); `tiny` is ≤4B (Tiny Titan).
-# Swap any `model` to point a tier at a different small model — that is the
-# whole "small models, easily configurable" story in one file.  Env vars
-# MODEL_TINY / MODEL_FAST / MODEL_BALANCED / MODEL_STRONG override at runtime.
+# Live transport is the LiteLLM gateway (ADR-0015).  Each profile binds to one of
+# the OpenAI-compatible vLLM endpoints served on Modal (see modal/registry.py):
+#   tiny     → nemotron-3-nano-4b   (≤4B, Tiny Titan)
+#   fast     → minicpm-4-1-8b
+#   balanced → gemma-4-12b
+#   strong   → gemma-4-26b
+#
+# The LiteLLM model string for a custom OpenAI-compatible endpoint is
+# `openai/<served_model_id>` (the HF repo id) with `base_url` pointing at the
+# endpoint's /v1 URL.  Modal serves each endpoint at
+#   https://<workspace>--<endpoint_name>.modal.run/v1
+# so only the workspace is deploy-specific: it is templated from $MODAL_WORKSPACE
+# and never hard-coded.  $MODAL_LLM_KEY is the endpoint key (vLLM accepts any
+# token, default "EMPTY").  Unset templates expand to "" → the offline stub is
+# used and these live bindings are ignored.
+#
+# Swap any `model` to point a tier at a different small model — that is the whole
+# "small models, easily configurable" story in one file.  Env vars MODEL_TINY /
+# MODEL_FAST / MODEL_BALANCED / MODEL_STRONG still override the model name.
 offline: null
 profiles:
   tiny:
-    model: qwen2.5-3b-instruct
+    model: openai/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
+    base_url: https://${MODAL_WORKSPACE}--nemotron-3-nano-4b.modal.run/v1
+    api_key: ${MODAL_LLM_KEY}
     temperature: 0.7
     max_tokens: 160
   fast:
-    model: qwen2.5-7b-instruct
+    model: openai/openbmb/MiniCPM4.1-8B
+    base_url: https://${MODAL_WORKSPACE}--minicpm-4-1-8b.modal.run/v1
+    api_key: ${MODAL_LLM_KEY}
     temperature: 0.9
     max_tokens: 220
   balanced:
-    model: qwen2.5-14b-instruct
+    model: openai/google/gemma-4-12B
+    base_url: https://${MODAL_WORKSPACE}--gemma-4-12b.modal.run/v1
+    api_key: ${MODAL_LLM_KEY}
     temperature: 0.8
     max_tokens: 320
   strong:
-    model: qwen2.5-32b-instruct
+    model: openai/google/gemma-4-26B-A4B-it
+    base_url: https://${MODAL_WORKSPACE}--gemma-4-26b.modal.run/v1
+    api_key: ${MODAL_LLM_KEY}
     temperature: 0.6
     max_tokens: 480

docs/adr/0015-litellm-gateway-modal-models.md

@@ -0,0 +1,94 @@
+# ADR-0015: LiteLLM Gateway for Modal-Served Small Models
+
+## Status
+
+Accepted
+
+## Context
+
+Per-agent model routing (ADR-0010) resolves each agent's logical profile
+(`tiny`/`fast`/`balanced`/`strong`) to a concrete model behind an
+OpenAI-compatible interface, and ADR-0005 added a Modal serving layer that exposes
+small models (all ≤32B, with a ≤4B Tiny Titan tier) as autoscaling, vLLM-backed,
+OpenAI-compatible HTTP endpoints. Until now the live transport was a hand-rolled
+`openai` SDK call (`OpenAICompatProvider`) and there was no real cost signal: the
+Governor's `hourly_budget_usd` (ADR-0007, ADR-0013) could only ever see `0.0`.
+
+We want one gateway that (a) routes every profile through a single, idiomatic
+call, (b) reaches the self-served Modal/vLLM endpoints without per-vendor
+branching, and (c) reports the real per-call cost so spend caps become
+enforceable — while keeping the offline path fully deterministic and free of any
+new dependency.
+
+## Decision
+
+Introduce a **LiteLLM gateway** as the live *transport*. This replaces how a model
+is *called*, not the routing abstraction: `ModelRouter.for_profile(profile) ->
+ModelProvider` and `ManifestAgent`'s usage are unchanged.
+
+**Thin, standard provider.** `LiteLLMProvider(ModelProvider)`
+(`src/models/litellm_provider.py`) issues a single
+`litellm.completion(model=…, api_base=…, api_key=…, messages=[{system},{user}],
+temperature=…, max_tokens=…)` call. The call is deliberately idiomatic so a later
+layer can wrap it (e.g. `instructor.from_litellm(litellm.completion)`) without
+fighting this code. `litellm` is imported lazily inside `complete()`, so importing
+`src.models.*` (and `app`) never requires the package. On error it mirrors
+`OpenAICompatProvider`: zero the usage and return a `"[model error: …]"` string.
+
+**Profiles → Modal endpoints.** Each profile binds to one served endpoint from
+`modal/registry.py`:
+
+| Profile | Modal endpoint | Served model id |
+|---|---|---|
+| `tiny` | `nemotron-3-nano-4b` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` |
+| `fast` | `minicpm-4-1-8b` | `openbmb/MiniCPM4.1-8B` |
+| `balanced` | `gemma-4-12b` | `google/gemma-4-12B` |
+| `strong` | `gemma-4-26b` | `google/gemma-4-26B-A4B-it` |
+
+For an OpenAI-compatible custom endpoint the LiteLLM model string is
+`openai/<served_model_id>` with `api_base` set to the endpoint's `/v1` URL. A
+self-served vLLM endpoint accepts any token, so the key defaults to the
+conventional `"EMPTY"` when unset.
+
+**Workspace is not hard-coded.** Modal serves each endpoint at a distinct
+subdomain `https://<workspace>--<endpoint>.modal.run/v1`, so a single base URL
+cannot address all four. `config/models.yaml` templates only the deploy-specific
+workspace: `base_url: https://${MODAL_WORKSPACE}--<endpoint>.modal.run/v1` and
+`api_key: ${MODAL_LLM_KEY}`. `Registry.from_dir()` expands these on load
+(`_expand_env`); if any referenced var is unset the whole string collapses to `""`
+(an incomplete binding is *not configured* rather than a broken half-URL) and a
+validator nulls it. `ModelProfileConfig`/`ProfileSpec` gained `api_key` alongside
+the existing `base_url`.
+
+**Real cost → Governor.** The provider reads cost from
+`response._hidden_params["response_cost"]`, falling back to
+`litellm.completion_cost(response)` — both guarded, so an unpriced/self-served
+model simply yields `0.0` instead of raising. Cost is exposed on
+`last_usage["cost_usd"]` (and `last_cost`); `ManifestAgent` carries it on its
+`last_usage`, and the conductor passes it to `governor.record_call(tokens=…,
+cost_usd=…)`. `hourly_budget_usd` is now a real spend cap on the live path.
+
+**Env-gated, offline by default.** `has_live_credentials()` stays the single
+online/offline decision and now also treats `MODAL_WORKSPACE` /
+`MODAL_LLM_BASE_URL` as an activating signal (in addition to `OPENAI_API_KEY`).
+With none set, the router serves `DeterministicTinyModel` for every profile, live
+bindings are ignored, and cost is `0.0`. `litellm` is an optional `litellm` extra
+in `pyproject.toml`.
+
+## Consequences
+
+- A hosted deployment sets `MODAL_WORKSPACE` (and optionally `MODAL_LLM_KEY`) and
+  every profile routes through LiteLLM to its Modal endpoint, with real cost
+  metered into the Governor. Nothing else changes.
+- The offline path is the default and import-clean: the full suite passes with no
+  credentials, no network, and `litellm` not installed.
+- `OpenAICompatProvider` is retained for its role→system persona map (reused by the
+  gateway) and `has_live_credentials()`; it is no longer the live transport.
+- The standard `litellm.completion(...)` shape leaves the door open for the
+  follow-up Instructor change to wrap the same client for structured output.
+- Cost accuracy depends on LiteLLM's pricing database; self-served vLLM models are
+  unpriced and report `0.0`. Token caps (`max_total_tokens`) remain the budget
+  guard for those; attaching `custom_cost_per_token` per endpoint is a follow-up.
+- A single `MODAL_LLM_BASE_URL` activates the live path but points only one URL;
+  multi-endpoint routing uses `MODAL_WORKSPACE` templating. Keeping both is
+  intentional (one-endpoint smoke tests vs. the full four-tier cast).

docs/architecture/model-routing.md

@@ -23,30 +23,77 @@ manifest.model_profile  ──►  ModelRouter.for_profile(profile)  ──►
 
 `ManifestAgent._complete()` calls `router.for_profile(self.manifest.model_profile)`
 every turn and records the provider's `last_usage` so the conductor can meter
-tokens into the Governor.
+tokens — and, on the live path, real cost — into the Governor.
+
+## Transport: the LiteLLM gateway (live path)
+
+The router resolves *which* model; the provider is *how* it is called. On the
+live path that transport is the **LiteLLM gateway** (`LiteLLMProvider`, ADR-0015):
+a single idiomatic `litellm.completion(...)` call routes every profile, including
+self-served OpenAI-compatible endpoints. The routing abstraction is unchanged —
+only the transport moved off a hand-rolled SDK call.
+
+```
+ModelRouter._build(profile)
+  offline → DeterministicTinyModel(variant="stub:<profile>")
+  live    → LiteLLMProvider(model="openai/<hf id>", api_base=<modal url>, …)
+```
+
+Profiles map to the OpenAI-compatible vLLM endpoints served on Modal
+(`modal/registry.py`):
+
+| Profile | Modal endpoint | Served model id |
+|---|---|---|
+| `tiny` | `nemotron-3-nano-4b` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` |
+| `fast` | `minicpm-4-1-8b` | `openbmb/MiniCPM4.1-8B` |
+| `balanced` | `gemma-4-12b` | `google/gemma-4-12B` |
+| `strong` | `gemma-4-26b` | `google/gemma-4-26B-A4B-it` |
+
+The LiteLLM model string for an OpenAI-compatible custom endpoint is
+`openai/<served_model_id>` with `api_base` pointing at the endpoint's `/v1` URL.
+
+### Real cost → Governor
+
+LiteLLM prices each call (`response._hidden_params["response_cost"]`, falling back
+to `litellm.completion_cost(response)` — both guarded, so an unpriced self-served
+model yields `0.0`). The provider exposes it on `last_usage["cost_usd"]` (and
+`last_cost`); `ManifestAgent` carries it, and the conductor passes it to
+`governor.record_call(tokens=…, cost_usd=…)`. This makes `hourly_budget_usd` a real
+spend cap on the live path. Offline cost is always `0.0`.
 
 ## Configuration
 
-`config/models.yaml` binds each profile to a concrete model + decoding:
+`config/models.yaml` binds each profile to a concrete model + endpoint + decoding.
+Only the Modal workspace is deploy-specific, so it is templated from
+`$MODAL_WORKSPACE` and never hard-coded; `$MODAL_LLM_KEY` is the endpoint key:
 
 ```yaml
 offline: null            # null=auto, true=stub everywhere, false=always live
 profiles:
-  tiny:     { model: qwen2.5-3b-instruct,  temperature: 0.7, max_tokens: 160 }
-  fast:     { model: qwen2.5-7b-instruct,  temperature: 0.9, max_tokens: 220 }
-  balanced: { model: qwen2.5-14b-instruct, temperature: 0.8, max_tokens: 320 }
-  strong:   { model: qwen2.5-32b-instruct, temperature: 0.6, max_tokens: 480 }
+  tiny:
+    model: openai/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
+    base_url: https://${MODAL_WORKSPACE}--nemotron-3-nano-4b.modal.run/v1
+    api_key: ${MODAL_LLM_KEY}
+    temperature: 0.7
+    max_tokens: 160
+  # fast / balanced / strong follow the same shape (see the file)
 ```
 
-Runtime env overrides (highest priority): `MODEL_TINY`, `MODEL_FAST`,
-`MODEL_BALANCED`, `MODEL_STRONG`, then `MODEL_NAME` as a final fallback.
+`Registry.from_dir()` expands `${VAR}` references when it loads the file. If any
+referenced var is unset, that string collapses to `""` (a binding built from a
+missing workspace is *not configured*, not a half-templated URL) and the validator
+nulls it. Runtime env overrides for the model name (highest priority): `MODEL_TINY`,
+`MODEL_FAST`, `MODEL_BALANCED`, `MODEL_STRONG`, then `MODEL_NAME` (these feed the
+`from_env` default path; explicit `models.yaml` specs win on the registry path).
 
 ## Offline determinism
 
-With no `OPENAI_API_KEY`, the router serves a `DeterministicTinyModel` for every
-profile (variant tagged per profile).  Demos and the entire test suite run with
-zero inference and full reproducibility — the offline/online decision is made
-once in `has_live_credentials()`.
+With no live binding configured — neither `OPENAI_API_KEY` nor
+`MODAL_WORKSPACE`/`MODAL_LLM_BASE_URL` — the router serves a
+`DeterministicTinyModel` for every profile (variant tagged per profile). Demos and
+the entire test suite run with zero inference and full reproducibility — the
+offline/online decision is made once in `has_live_credentials()`, and `litellm` is
+imported lazily so it need not be installed at all offline.
 
 ## Mixing tiers in one cast
 
@@ -64,6 +111,9 @@ everything on the big model.
 ## Code
 
 - `src/models/router.py` — `ModelRouter`, `ProfileSpec`, `_PROFILE_DECODING`
+- `src/models/litellm_provider.py` — `LiteLLMProvider` (live transport, real cost)
 - `src/core/manifest.py` — `resolve_model()` (env → default name resolution)
+- `src/core/registry.py` — `build_router()`, `_expand_env()` (YAML env templating)
 - `src/models/provider.py` — `ModelProvider.last_usage`, `estimate_tokens()`
-- `src/models/openai_compat.py` — live provider, real usage capture
+- `src/models/openai_compat.py` — `has_live_credentials()`, role��system personas
+- `modal/registry.py` — the served endpoints each profile points at

pyproject.toml

@@ -23,6 +23,13 @@ store = [
   "sqlalchemy>=2.0",
   "psycopg[binary]>=3",
 ]
+# Model gateway (ADR-0015). Optional: routes live profiles through LiteLLM to the
+# OpenAI-compatible models served on Modal, with real per-call cost metering. The
+# system runs fully offline on the deterministic stub without this installed;
+# litellm is imported lazily so importing src.models.* never requires it.
+litellm = [
+  "litellm>=1.40",
+]
 
 [tool.ruff]
 line-length = 120

src/core/conductor.py

@@ -160,8 +160,10 @@ class Conductor:
             projection=projection,
             recent_events=self.ledger.events,
         )
-        tokens = int(getattr(agent, "last_usage", {}).get("total_tokens", 0) or 0)
-        self.governor.record_call(tokens=tokens)
+        usage = getattr(agent, "last_usage", {})
+        tokens = int(usage.get("total_tokens", 0) or 0)
+        cost_usd = float(usage.get("cost_usd", 0.0) or 0.0)
+        self.governor.record_call(tokens=tokens, cost_usd=cost_usd)
         self._append(event)
         projection.apply(event)
 

src/core/config.py

@@ -30,9 +30,26 @@ class ModelProfileConfig(BaseModel):
 
     model: str
     base_url: str | None = None
+    """OpenAI-compatible endpoint URL (ends in /v1).  Env-templatable in YAML via
+    ``${MODAL_LLM_BASE_URL}`` so the Modal workspace is never hard-coded."""
+
+    api_key: str | None = None
+    """Key for the endpoint (env-templatable, e.g. ``${MODAL_LLM_KEY}``).  vLLM
+    accepts any token unless the server enforces one."""
+
     temperature: float = 0.8
     max_tokens: int = 256
 
+    @model_validator(mode="after")
+    def _blank_to_none(self) -> "ModelProfileConfig":
+        # An unset ``${VAR}`` template expands to "" (see registry._expand_env);
+        # normalise empty bindings back to None so the live transport omits them.
+        if not self.base_url:
+            self.base_url = None
+        if not self.api_key:
+            self.api_key = None
+        return self
+
 
 class ModelsConfig(BaseModel):
     model_config = ConfigDict(extra="forbid")

src/core/registry.py

@@ -14,6 +14,7 @@ their manifest.
 from __future__ import annotations
 
 import os
+import re
 from dataclasses import dataclass, field
 from pathlib import Path
 
@@ -31,6 +32,32 @@ _REPO_ROOT = Path(__file__).resolve().parents[2]
 # alternate deployment point the registry at a different config tree.
 DEFAULT_CONFIG_DIR = Path(os.getenv("MAL_CONFIG_DIR") or _REPO_ROOT / "config")
 
+
+_ENV_REF = re.compile(r"\$\{(\w+)\}|\$(\w+)")
+
+
+def _expand_env(value):
+    """Recursively expand ``$VAR`` / ``${VAR}`` in a loaded-config tree.
+
+    Lets ``config/models.yaml`` point profiles at a Modal endpoint without
+    hard-coding the workspace URL or key — e.g.
+    ``base_url: https://${MODAL_WORKSPACE}--<endpoint>.modal.run/v1``.
+
+    If *any* referenced var in a string is unset/empty, the whole string collapses
+    to ``""`` — a binding built from a missing workspace is simply *not configured*
+    rather than a half-templated, broken URL.  The validator then nulls it, and the
+    offline path ignores live bindings entirely."""
+    if isinstance(value, str):
+        refs = _ENV_REF.findall(value)
+        if refs and any(not os.getenv(g1 or g2, "") for g1, g2 in refs):
+            return ""
+        return _ENV_REF.sub(lambda m: os.getenv(m.group(1) or m.group(2), ""), value)
+    if isinstance(value, dict):
+        return {k: _expand_env(v) for k, v in value.items()}
+    if isinstance(value, list):
+        return [_expand_env(v) for v in value]
+    return value
+
 # ── handler registry (behaviour bindings) ────────────────────────────────────────
 
 HANDLERS: dict[str, type[ManifestAgent]] = {}
@@ -82,7 +109,8 @@ class Registry:
         models = ModelsConfig()
         models_file = root / "models.yaml"
         if models_file.is_file():
-            models = ModelsConfig.model_validate(yaml.safe_load(models_file.read_text()) or {})
+            raw_models = _expand_env(yaml.safe_load(models_file.read_text()) or {})
+            models = ModelsConfig.model_validate(raw_models)
 
         return cls(agents=agents, scenarios=scenarios, models=models)
 

src/models/litellm_provider.py

@@ -0,0 +1,131 @@
+"""LiteLLM-backed provider — one gateway, every logical profile.
+
+This is the *transport* the :class:`~src.models.router.ModelRouter` uses on the
+live path: it replaces hand-rolled per-vendor SDK calls with a single idiomatic
+``litellm.completion(...)`` call.  Routing (profile → concrete model + endpoint)
+is unchanged and still lives in the router; this class only knows how to *call* a
+model and report what it cost.
+
+Two things it adds over the plain OpenAI-compatible provider:
+
+  * **Real cost.**  LiteLLM prices the call from its model database, so the
+    Governor's ``hourly_budget_usd`` becomes real on the live path.  Cost is
+    exposed on ``last_usage["cost_usd"]`` (and ``last_cost``); offline it is 0.
+  * **One model string for any endpoint.**  An OpenAI-compatible custom endpoint
+    (the Modal/vLLM servers in ``modal/``) is reached with the LiteLLM model
+    string ``openai/<served_model_id>`` plus an ``api_base`` — no per-vendor
+    branching.
+
+``litellm`` is imported lazily so ``import src.models.*`` (and ``import app``)
+work with the package not installed; offline never touches this class.  The call
+is kept thin and standard so a later layer can wrap it (e.g.
+``instructor.from_litellm(litellm.completion)``) without fighting this code.
+See ADR-0015.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+from src.models.openai_compat import OpenAICompatProvider
+from src.models.provider import ModelProvider
+
+
+@dataclass
+class LiteLLMProvider(ModelProvider):
+    """Route one logical profile through the LiteLLM gateway.
+
+    ``model`` is a LiteLLM model string.  For an OpenAI-compatible custom
+    endpoint (Modal/vLLM) it is ``openai/<served_model_id>`` and ``api_base``
+    points at the endpoint's ``/v1`` URL.  Decoding (``temperature`` /
+    ``max_tokens``) and the binding come from the router's per-profile spec.
+    """
+
+    model: str
+    api_base: str | None = None
+    api_key: str | None = None
+    temperature: float = 0.8
+    max_tokens: int = 256
+    _last_usage: dict = field(default_factory=dict, init=False, repr=False)
+    _last_cost: float = field(default=0.0, init=False, repr=False)
+
+    def complete(self, role: str, prompt: str) -> str:
+        from src.models.provider import estimate_tokens
+
+        try:
+            import litellm
+        except ImportError as exc:  # pragma: no cover - exercised only when unset
+            raise ImportError(
+                "litellm package is required for LiteLLMProvider. "
+                "Install it with: uv pip install litellm"
+            ) from exc
+
+        system = OpenAICompatProvider._system_for_role(role)
+        # A self-served vLLM endpoint accepts any token; default to the conventional
+        # placeholder so a configured custom endpoint never trips on a missing key.
+        api_key = self.api_key or ("EMPTY" if self.api_base else None)
+        try:
+            response = litellm.completion(
+                model=self.model,
+                api_base=self.api_base,
+                api_key=api_key,
+                messages=[
+                    {"role": "system", "content": system},
+                    {"role": "user", "content": prompt},
+                ],
+                temperature=self.temperature,
+                max_tokens=self.max_tokens,
+            )
+            text = (response.choices[0].message.content or "").strip()
+            usage = getattr(response, "usage", None)
+            if usage is not None:
+                prompt_tokens = int(getattr(usage, "prompt_tokens", 0) or 0)
+                completion_tokens = int(getattr(usage, "completion_tokens", 0) or 0)
+                total_tokens = int(getattr(usage, "total_tokens", 0) or 0) or (
+                    prompt_tokens + completion_tokens
+                )
+            else:
+                prompt_tokens, completion_tokens = estimate_tokens(prompt), estimate_tokens(text)
+                total_tokens = prompt_tokens + completion_tokens
+            cost = self._extract_cost(litellm, response)
+            self._last_cost = cost
+            self._last_usage = {
+                "prompt_tokens": prompt_tokens,
+                "completion_tokens": completion_tokens,
+                "total_tokens": total_tokens,
+                "cost_usd": cost,
+            }
+            return text
+        except Exception as exc:
+            self._last_cost = 0.0
+            self._last_usage = {
+                "prompt_tokens": 0,
+                "completion_tokens": 0,
+                "total_tokens": 0,
+                "cost_usd": 0.0,
+            }
+            return f"[model error: {exc}]"
+
+    @property
+    def last_cost(self) -> float:
+        """Metered USD cost of the most recent :meth:`complete` call (0.0 offline)."""
+        return self._last_cost
+
+    @staticmethod
+    def _extract_cost(litellm, response) -> float:
+        """Best-effort USD cost for *response*; 0.0 if the model is unpriced.
+
+        Prefers the value LiteLLM already attached during the call
+        (``_hidden_params["response_cost"]``); falls back to pricing the response
+        directly.  Both paths are guarded — an unknown/custom model (e.g. a
+        self-served vLLM endpoint) simply yields 0.0 rather than raising.
+        """
+        hidden = getattr(response, "_hidden_params", None)
+        if isinstance(hidden, dict):
+            cost = hidden.get("response_cost")
+            if isinstance(cost, (int, float)):
+                return float(cost)
+        try:
+            cost = litellm.completion_cost(completion_response=response)
+            return float(cost or 0.0)
+        except Exception:
+            return 0.0

src/models/openai_compat.py

@@ -120,13 +120,23 @@ class OpenAICompatProvider(ModelProvider):
 
 
 def has_live_credentials() -> bool:
-    """True when a usable API key is configured for live inference.
+    """True when a usable model binding is configured for live inference.
 
     Single source of truth for the online/offline decision, shared by
-    ``build_from_env`` and the ModelRouter so they never disagree.
+    ``build_from_env`` and the ModelRouter so they never disagree.  Two ways to go
+    live, either is sufficient:
+
+      * ``OPENAI_API_KEY`` — a generic OpenAI-compatible key; or
+      * ``MODAL_WORKSPACE`` / ``MODAL_LLM_BASE_URL`` — the binding for the models
+        served on Modal (ADR-0015): the workspace templates each profile's
+        endpoint URL in ``config/models.yaml``.  The gateway reaches those with
+        ``MODAL_LLM_KEY`` (default ``"EMPTY"``), so the workspace/base-url is the
+        activating signal.
     """
     api_key = os.getenv("OPENAI_API_KEY", "")
-    return bool(api_key) and api_key not in ("sk-stub", "your-key-here")
+    if bool(api_key) and api_key not in ("sk-stub", "your-key-here"):
+        return True
+    return bool(os.getenv("MODAL_WORKSPACE", "") or os.getenv("MODAL_LLM_BASE_URL", ""))
 
 
 def build_from_env() -> ModelProvider:

src/models/router.py

@@ -12,13 +12,18 @@ This is the single place per-agent model selection happens, so:
 Offline (no API key) the router serves a :class:`DeterministicTinyModel` for
 every profile, so demos and tests run with zero inference and full
 reproducibility.  See ADR-0010.
+
+On the live path the concrete transport is the :class:`LiteLLMProvider` gateway
+(ADR-0015): profiles point at the OpenAI-compatible Modal/vLLM endpoints in
+``modal/`` and the gateway reports real per-call cost into the Governor.  The
+routing abstraction here is unchanged — only how a model is *called* moved.
 """
 from __future__ import annotations
 
 from dataclasses import dataclass, field
 
 from src.core.manifest import ModelProfile, resolve_model
-from src.models.openai_compat import OpenAICompatProvider, has_live_credentials
+from src.models.openai_compat import has_live_credentials
 from src.models.provider import DeterministicTinyModel, ModelProvider
 
 # Decoding defaults per profile.  Smaller models stay cooler and shorter; the
@@ -37,6 +42,7 @@ class ProfileSpec:
 
     model: str
     base_url: str | None = None
+    api_key: str | None = None
     temperature: float = 0.8
     max_tokens: int = 256
 
@@ -77,10 +83,15 @@ class ModelRouter:
     def _build(self, profile: str) -> ModelProvider:
         if self.offline:
             return DeterministicTinyModel(variant=f"stub:{profile}")
+        # Live transport is the LiteLLM gateway (ADR-0015).  Lazy-import keeps the
+        # offline path free of the dependency.
+        from src.models.litellm_provider import LiteLLMProvider
+
         spec = self._spec_for(profile)
-        return OpenAICompatProvider(
+        return LiteLLMProvider(
             model=spec.model,
-            base_url=spec.base_url,
+            api_base=spec.base_url,
+            api_key=spec.api_key,
             temperature=spec.temperature,
             max_tokens=spec.max_tokens,
         )

tests/test_conductor.py

@@ -2,6 +2,10 @@ from __future__ import annotations
 
 
 from src.core.conductor import Conductor
+from src.core.events import Event
+from src.core.governor import Governor
+from src.core.manifest import AgentManifest, ScheduleConfig
+from src.scenarios.base import Scenario
 from src.scenarios.thousand_token_wood import build_scenario
 
 
@@ -90,3 +94,39 @@ class TestConductorProjection:
         c.reset("the wood wakes")
         proj = c.projection
         assert proj.seed == "the wood wakes" or "the wood wakes" in proj.current_scene
+
+
+class _CostingAgent:
+    """Minimal agent that reports a per-call cost — stands in for the live gateway."""
+
+    manifest = AgentManifest(
+        name="coster",
+        persona="p",
+        may_emit=["world.observed"],
+        schedule=ScheduleConfig(tick_every=1),
+    )
+
+    def __init__(self) -> None:
+        self.last_usage = {"prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15, "cost_usd": 0.002}
+
+    def act(self, run_id, turn, projection, recent_events) -> Event:
+        return Event(run_id=run_id, turn=turn, kind="world.observed", actor="coster", payload={"text": "x"})
+
+
+class TestConductorCostMetering:
+    def test_live_cost_reaches_governor(self):
+        # On the live path the agent carries real cost on last_usage; the conductor
+        # must plumb it into the Governor so hourly_budget_usd is enforceable.
+        scenario = Scenario(name="s", default_seed="seed", agents=(_CostingAgent(),))
+        c = Conductor(scenario=scenario, governor=Governor())
+        c.reset("seed")
+        c.step()
+        assert c.governor.stats["spend_usd"] > 0
+        assert c.governor.stats["total_tokens"] >= 15
+
+    def test_offline_cost_stays_zero(self):
+        # The deterministic stub reports no cost; spend must remain 0.
+        c = _conductor()
+        c.reset("seed")
+        c.step()
+        assert c.governor.stats["spend_usd"] == 0.0

tests/test_litellm_provider.py

@@ -0,0 +1,164 @@
+"""LiteLLM gateway tests — fully offline, litellm.completion monkeypatched.
+
+No network and no real credentials: a fake ``litellm`` module (and a fake
+response with ``.usage`` and a cost hook) is injected so we can assert the
+provider returns the text and captures tokens + real cost, and that the router
+builds a :class:`LiteLLMProvider` when live and the deterministic stub offline.
+"""
+from __future__ import annotations
+
+import sys
+import types
+from dataclasses import dataclass
+
+import pytest
+
+from src.models.litellm_provider import LiteLLMProvider
+from src.models.provider import DeterministicTinyModel
+from src.models.router import ModelRouter, ProfileSpec
+
+
+# ── fake litellm response objects ────────────────────────────────────────────
+
+
+@dataclass
+class _FakeUsage:
+    prompt_tokens: int = 11
+    completion_tokens: int = 7
+    total_tokens: int = 18
+
+
+class _FakeMessage:
+    def __init__(self, content: str) -> None:
+        self.content = content
+
+
+class _FakeChoice:
+    def __init__(self, content: str) -> None:
+        self.message = _FakeMessage(content)
+
+
+class _FakeResponse:
+    def __init__(self, content: str, *, hidden_cost: float | None = None) -> None:
+        self.choices = [_FakeChoice(content)]
+        self.usage = _FakeUsage()
+        self._hidden_params = {} if hidden_cost is None else {"response_cost": hidden_cost}
+
+
+def _install_fake_litellm(monkeypatch, *, response, cost_value=0.0, record=None):
+    """Inject a fake ``litellm`` module exposing completion + completion_cost."""
+    fake = types.ModuleType("litellm")
+
+    def _completion(**kwargs):
+        if record is not None:
+            record.update(kwargs)
+        if isinstance(response, Exception):
+            raise response
+        return response
+
+    def _completion_cost(completion_response=None, **_kwargs):
+        return cost_value
+
+    fake.completion = _completion
+    fake.completion_cost = _completion_cost
+    monkeypatch.setitem(sys.modules, "litellm", fake)
+    return fake
+
+
+# ── provider ─────────────────────────────────────────────────────────────────
+
+
+class TestLiteLLMProviderComplete:
+    def test_returns_text_and_captures_usage(self, monkeypatch):
+        _install_fake_litellm(monkeypatch, response=_FakeResponse("a mossy booth"), cost_value=0.0)
+        provider = LiteLLMProvider(model="openai/some/model", api_base="https://x/v1")
+        out = provider.complete("scene-whisperer", "grow the wood")
+        assert out == "a mossy booth"
+        assert provider.last_usage["prompt_tokens"] == 11
+        assert provider.last_usage["completion_tokens"] == 7
+        assert provider.last_usage["total_tokens"] == 18
+
+    def test_captures_cost_from_completion_cost(self, monkeypatch):
+        _install_fake_litellm(monkeypatch, response=_FakeResponse("hi"), cost_value=0.0123)
+        provider = LiteLLMProvider(model="openai/some/model", api_base="https://x/v1")
+        provider.complete("echo", "drop a pebble")
+        assert provider.last_usage["cost_usd"] == pytest.approx(0.0123)
+        assert provider.last_cost == pytest.approx(0.0123)
+
+    def test_prefers_hidden_params_cost(self, monkeypatch):
+        # When LiteLLM already attached a cost, use it without re-pricing.
+        _install_fake_litellm(
+            monkeypatch, response=_FakeResponse("hi", hidden_cost=0.05), cost_value=999.0
+        )
+        provider = LiteLLMProvider(model="openai/some/model", api_base="https://x/v1")
+        provider.complete("echo", "drop a pebble")
+        assert provider.last_usage["cost_usd"] == pytest.approx(0.05)
+
+    def test_calls_openai_style_for_custom_endpoint(self, monkeypatch):
+        record: dict = {}
+        _install_fake_litellm(monkeypatch, response=_FakeResponse("ok"), record=record)
+        provider = LiteLLMProvider(
+            model="openai/google/gemma-4-12B",
+            api_base="https://ws--gemma-4-12b.modal.run/v1",
+            api_key="EMPTY",
+            temperature=0.3,
+            max_tokens=99,
+        )
+        provider.complete("seedkeeper", "observe")
+        assert record["model"] == "openai/google/gemma-4-12B"
+        assert record["api_base"] == "https://ws--gemma-4-12b.modal.run/v1"
+        assert record["api_key"] == "EMPTY"
+        assert record["temperature"] == 0.3
+        assert record["max_tokens"] == 99
+        # Two messages: a role-derived system prompt, then the user prompt.
+        roles = [m["role"] for m in record["messages"]]
+        assert roles == ["system", "user"]
+        assert record["messages"][1]["content"] == "observe"
+
+    def test_defaults_api_key_for_custom_endpoint(self, monkeypatch):
+        record: dict = {}
+        _install_fake_litellm(monkeypatch, response=_FakeResponse("ok"), record=record)
+        provider = LiteLLMProvider(model="openai/m", api_base="https://x/v1")  # no api_key
+        provider.complete("echo", "x")
+        assert record["api_key"] == "EMPTY"
+
+    def test_error_returns_marker_and_zeroes_usage(self, monkeypatch):
+        _install_fake_litellm(monkeypatch, response=RuntimeError("boom"))
+        provider = LiteLLMProvider(model="openai/m", api_base="https://x/v1")
+        out = provider.complete("echo", "x")
+        assert out.startswith("[model error:")
+        assert provider.last_usage["total_tokens"] == 0
+        assert provider.last_usage["cost_usd"] == 0.0
+        assert provider.last_cost == 0.0
+
+
+# ── router integration ───────────────────────────────────────────────────────
+
+
+class TestRouterBuildsGateway:
+    def test_live_profile_builds_litellm_provider(self):
+        router = ModelRouter(
+            offline=False,
+            specs={
+                "fast": ProfileSpec(
+                    model="openai/openbmb/MiniCPM4.1-8B",
+                    base_url="https://ws--minicpm-4-1-8b.modal.run/v1",
+                    api_key="EMPTY",
+                )
+            },
+        )
+        provider = router.for_profile("fast")
+        assert isinstance(provider, LiteLLMProvider)
+        assert provider.model == "openai/openbmb/MiniCPM4.1-8B"
+        assert provider.api_base == "https://ws--minicpm-4-1-8b.modal.run/v1"
+
+    def test_offline_builds_deterministic_stub(self):
+        router = ModelRouter(offline=True)
+        assert isinstance(router.for_profile("fast"), DeterministicTinyModel)
+
+    def test_offline_usage_has_no_cost(self):
+        # The offline stub never reports cost; the conductor reads 0.0 for it.
+        router = ModelRouter(offline=True)
+        provider = router.for_profile("tiny")
+        provider.complete("scene-whisperer", "grow")
+        assert "cost_usd" not in provider.last_usage

tests/test_router.py

@@ -1,6 +1,6 @@
 from __future__ import annotations
 
-from src.models.openai_compat import OpenAICompatProvider
+from src.models.litellm_provider import LiteLLMProvider
 from src.models.provider import DeterministicTinyModel, ModelProvider, estimate_tokens
 from src.models.router import ModelRouter, ProfileSpec
 
@@ -45,11 +45,21 @@ class TestModelRouterOnline:
     def test_explicit_spec_used(self):
         router = ModelRouter(
             offline=False,
-            specs={"tiny": ProfileSpec(model="qwen2.5-3b-instruct", temperature=0.5, max_tokens=128)},
+            specs={
+                "tiny": ProfileSpec(
+                    model="openai/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16",
+                    base_url="https://ws--nemotron-3-nano-4b.modal.run/v1",
+                    api_key="EMPTY",
+                    temperature=0.5,
+                    max_tokens=128,
+                )
+            },
         )
         provider = router.for_profile("tiny")
-        assert isinstance(provider, OpenAICompatProvider)
-        assert provider.model == "qwen2.5-3b-instruct"
+        assert isinstance(provider, LiteLLMProvider)
+        assert provider.model == "openai/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16"
+        assert provider.api_base == "https://ws--nemotron-3-nano-4b.modal.run/v1"
+        assert provider.api_key == "EMPTY"
         assert provider.temperature == 0.5
         assert provider.max_tokens == 128