docs/architecture/model-routing.md

Model Routing

Per-agent model selection is the heart of the "small models, easily configurable" story. Each agent declares a logical profile; the ModelRouter resolves it to a concrete small model with its own decoding config. No agent code ever names a model.

The profiles

Profile	Param target	Role of thumb
tiny	≤4B	cheap, high-volume workers (Tiny Titan mode)
fast	≤7B	default workers
balanced	≤13B	local judges, salience-heavy roles
strong	≤32B	the global judge, reflection passes

How a turn resolves a model

manifest.model_endpoint or model_profile  ──►  ModelRouter.for_profile(key)  ──►  ModelProvider
        (the agent's "route key")                 (cached per key)              (concrete model)

ManifestAgent computes a route key — self._route_key, the explicit model_endpoint when set, else the model_profile tier — and calls router.for_profile(self._route_key) every turn, recording the provider's last_usage so the conductor can meter tokens (and, live, real cost) into the Governor. The router accepts either kind of key: a tier resolves to the profile default, a catalogue endpoint slug to that specific model's binding.

Pinning a specific model (model_endpoint)

Tiers are the default, but a manifest can pin one mind to a specific catalogue model by setting model_endpoint to an endpoint slug from modal/catalogue.py (e.g. minicpm-4-1-8b). This overrides the tier and is how a cast mixes concrete sponsor models — one worker on MiniCPM, the Judge on Nemotron Cascade — including the unbound specialist models that no tier defaults to. See ADR-0022.

ModelRouter._spec_for(key)
  key in specs                → that ProfileSpec (the four tiers from models.yaml)
  key is a catalogue endpoint → _catalogue_spec(key): binding_for(key) + the model's
                                tier decoding (unbound specialist → balanced defaults)
  unknown non-tier key        → degrade to the fast tier (never crash)

Offline this path is never reached — _build serves the deterministic stub for any key, with the key folded into the stub's variant, so picking a different model still varies the (reproducible) output. The Fishbowl Lab writes model_endpoint from its per-cast model picker, so the model you choose in the UI is the model that runs (see fishbowl-ui.md).

Transport: the LiteLLM gateway (live path)

The router resolves which model; the provider is how it is called. On the live path that transport is the LiteLLM gateway (LiteLLMProvider, ADR-0015): a single idiomatic litellm.completion(...) call routes every profile, including self-served OpenAI-compatible endpoints. The routing abstraction is unchanged — only the transport moved off a hand-rolled SDK call.

ModelRouter._build(profile)
  offline → DeterministicTinyModel(variant="stub:<profile>")
  live    → LiteLLMProvider(model="openai/<hf id>", api_base=<modal url>, …)

Profiles map to the OpenAI-compatible vLLM endpoints served on Modal (modal/catalogue.py):

Profile	Modal endpoint	Served model id
tiny	nemotron-3-nano-4b	nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
fast	minicpm-4-1-8b	openbmb/MiniCPM4.1-8B
balanced	gemma-4-12b	google/gemma-4-12B
strong	gemma-4-26b	google/gemma-4-26B-A4B-it

The LiteLLM model string for an OpenAI-compatible custom endpoint is openai/<served_model_id> with api_base pointing at the endpoint's /v1 URL.

Backends: Modal · Hugging Face · Local GPU

A backend is just a catalogue + a binding rule, unified behind one registry (src/models/inference.py, ADR-0024). Models are named by a backend-qualified key "<backend>:<raw>"; a bare key means Modal, so all existing config keeps working. Three backends ship today:

Backend	Prefix	Where it runs	Opt-in
Modal	(bare)	vLLM endpoints you deploy on Modal GPUs	MODAL_WORKSPACE / MODAL_LLM_BASE_URL
Hugging Face	hf:	serverless Inference Providers router	HF_TOKEN
Local GPU	local:	in-process transformers model on the host's own GPU	SPACES_ZERO_GPU / LOCAL_INFERENCE=1 / CUDA auto-detected

The local backend (ADR-0033, supersedes ADR-0032) runs a cast fully in-process — no HTTP server, no extra process, no token. It loads a small instruct model via transformers inside a @spaces.GPU function (LocalTransformersProvider, src/models/local_provider.py). The hardware path is transparent:

• ZeroGPU Space — @spaces.GPU allocates a GPU per call from the shared pool (~5 min/day free quota). Enabled when SPACES_ZERO_GPU is set.
• Dedicated-GPU Space (T4 / L4 / L40S / A100) — persistent GPU, no per-call quota. Enabled with LOCAL_INFERENCE=1.
• Local CUDA box — the same LOCAL_INFERENCE=1 flag, or CUDA auto-detected at startup.

Off ZeroGPU and without LOCAL_INFERENCE=1, @spaces.GPU is a no-op and the engine falls back to the deterministic stub — so the demo is always reproducible on CPU-only hosts. Pick "Local GPU" in the Lab's backend radio to opt in per run.

Available models (all ≤32B; select via local:<repo_id>). One sponsor family per tier, so a single cast spans four sponsors at once (the multi-track strategy):

Key	Model	Tier	Notes
local:nvidia/Nemotron-Mini-4B-Instruct	Nemotron Mini 4B	tiny	NVIDIA lane; Tiny-Titan ≤4B band; plain Nemotron-4 transformer (native, no kernels)
local:openbmb/MiniCPM5-1B	MiniCPM5 1B	fast	OpenBMB lane; native llama arch (built for transformers 5.x). The MiniCPM 4.x custom-code models mis-compute under the 5.x floor, so MiniCPM5 is used in-process instead
local:CohereLabs/aya-expanse-8b	Aya Expanse 8B	balanced	Cohere lane; gated repo — needs licence acceptance + HF_TOKEN
local:JetBrains/Mellum2-12B-A2.5B-Instruct	Mellum 2 (12B MoE, ~2.5B active)	strong	JetBrains lane; native MellumConfig (Instruct, not Base)

Each tier is tagged, so a cast's fast/balanced/strong seats route to different sponsor models. That cross-sponsor cast loads several multi-GB models per show — heavy on the free ZeroGPU ~5-min/day budget and host RAM; on a dedicated GPU there is no cap. For a quota-light demo, pin the whole cast to the tiny default. The tiny model is listed first, so any untagged fallback also lands on the cheapest tier.

Bind a tier to a local model with a qualified key:

profiles:
  tiny: { endpoint: "local:nvidia/Nemotron-Mini-4B-Instruct", temperature: 0.7, max_tokens: 192 }

Real cost → Governor

LiteLLM prices each call (response._hidden_params["response_cost"], falling back to litellm.completion_cost(response) — both guarded, so an unpriced self-served model yields 0.0). The provider exposes it on last_usage["cost_usd"] (and last_cost); ManifestAgent carries it, and the conductor passes it to governor.record_call(tokens=…, cost_usd=…). This makes hourly_budget_usd a real spend cap on the live path. Offline cost is always 0.0.

Configuration

config/models.yaml binds each profile to a model by its catalogue key — the slug in modal/catalogue.py, the single source of truth for what is deployed. The loader expands that key into the concrete binding, so the served id and endpoint URL live in exactly one place (no parallel YAML to keep in sync):

offline: null            # null=auto, true=stub everywhere, false=always live
profiles:
  tiny:
    endpoint: nemotron-3-nano-4b   # catalogue key (modal/catalogue.py)
    temperature: 0.7
    max_tokens: 192
  # fast / balanced / strong follow the same shape (see the file). Note the
  # balanced/strong tiers are reasoning models (gemma4 reasoning parser) and carry
  # a much larger max_tokens so their thinking + answer fit (ADR-0023).

Registry.from_dir() resolves each endpoint: against the catalogue (via src/models/modal_catalogue.py) and fills:

• model = openai/<served_model_id>
• base_url = https://${MODAL_WORKSPACE}--<app>-<endpoint>.modal.run/v1 (or $MODAL_LLM_BASE_URL if set; "" when neither → offline stub)
• api_key = $MODAL_LLM_KEY (a self-served vLLM endpoint accepts any token)

Only the workspace is deploy-specific, and it is never hard-coded. Adding/retuning a model is a one-line edit in modal/catalogue.py; re-casting a tier is a one-line endpoint: change here. Per-profile env overrides for the model string (highest priority): MODEL_TINY, MODEL_FAST, MODEL_BALANCED, MODEL_STRONG. You can also bind a profile explicitly with model: + base_url: instead of endpoint: (an escape hatch for non-catalogue endpoints).

Offline determinism

With no live binding configured — no MODAL_WORKSPACE and no MODAL_LLM_BASE_URL — the router serves a DeterministicTinyModel for every profile (variant tagged per profile). Demos and the entire test suite run with zero inference and full reproducibility — the offline/online decision is made once in has_live_credentials(), and litellm is imported lazily so it need not be installed at all offline. There is no generic cloud key: live inference is always against the small models you deploy on Modal.

Mixing tiers in one cast

This is the economic payoff. Mystery Roots runs three cheap workers and one strong verifier:

clue-gatherer    fast       hypothesis-former  balanced
devils-advocate  fast       mystery-judge      strong

Many weak proposers, one strong judge — at a fraction of the cost of running everything on the big model.

Code

• src/models/router.py — ModelRouter, ProfileSpec, _PROFILE_DECODING, _catalogue_spec() (endpoint key → binding)
• src/agents/base.py — ManifestAgent._route_key (endpoint-or-tier)
• src/core/registry.py — Registry.from_world() (a UI/LLM-composed run on the same path)
• src/models/litellm_provider.py — LiteLLMProvider (live transport, real cost)
• src/models/modal_catalogue.py — engine view of the catalogue (key → binding)
• src/models/inference.py — unified backend registry (Modal · HF · Local GPU); qualified keys
• src/models/local_catalogue.py — local model catalogue + capability gate (has_credentials)
• src/models/local_provider.py — LocalTransformersProvider: in-process @spaces.GPU inference
• src/core/manifest.py — resolve_model() (env → catalogue default)
• src/core/registry.py — build_router(), _resolve_model_endpoints(), _expand_env()
• src/models/provider.py — ModelProvider.last_usage, estimate_tokens()
• src/models/openai_compat.py — has_live_credentials(), role→system personas
• modal/catalogue.py — the single source of truth: every served model + provider app