feat: Enhance model routing and deployment flexibility
Browse files- Introduced `vllm_version` to allow per-model inference stack pinning, enabling models to opt for nightly builds when necessary.
- Updated image building logic to handle model-specific vLLM versions, ensuring compatibility with various architectures.
- Added `model_endpoint` to `AgentManifest` for explicit model binding, allowing agents to route to specific models instead of relying solely on profiles.
- Enhanced `ModelRouter` to resolve specific catalogue models, improving routing accuracy and flexibility.
- Updated UI components in the Fishbowl Lab to support new model selection features, ensuring only deployable models are presented.
- Improved tests to validate new model selection and routing behaviors, ensuring robustness and adherence to expected functionality.
- docs/adr/0022-per-agent-explicit-model-binding.md +88 -0
- docs/architecture/fishbowl-ui.md +16 -8
- docs/architecture/manifest-spec.md +10 -0
- docs/architecture/model-routing.md +33 -6
- docs/schema/agent-manifest.md +8 -0
- modal/docs/deploying.md +8 -0
- modal/service.py +10 -6
- src/agents/base.py +16 -3
- src/core/manifest.py +10 -0
- src/core/registry.py +23 -1
- src/models/router.py +37 -1
- src/ui/fishbowl/app.py +83 -6
- src/ui/fishbowl/lab.py +152 -75
- tests/test_fishbowl_lab.py +51 -39
- tests/test_manifest_agent.py +19 -0
- tests/test_registry.py +37 -0
- tests/test_router.py +39 -0
|
@@ -0,0 +1,88 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# ADR-0022: Per-Agent Explicit Model Binding (`model_endpoint`)
|
| 2 |
+
|
| 3 |
+
## Status
|
| 4 |
+
|
| 5 |
+
Accepted
|
| 6 |
+
|
| 7 |
+
## Context
|
| 8 |
+
|
| 9 |
+
ADR-0010 gave per-agent model selection through four **logical tiers**
|
| 10 |
+
(`tiny`/`fast`/`balanced`/`strong`), and `config/models.yaml` binds each tier to one
|
| 11 |
+
concrete catalogue model. That is the right default, but it has two limits:
|
| 12 |
+
|
| 13 |
+
1. **A cast can express at most four distinct models** — one per tier. Two agents on
|
| 14 |
+
the same tier always share a model.
|
| 15 |
+
2. **The unbound specialist models are unreachable.** Several catalogue entries have
|
| 16 |
+
`profile=None` (Nemotron Cascade 14B, Nemotron 30B, MiniCPM-o) precisely so they do
|
| 17 |
+
*not* displace a tier default — but then no manifest could ever cast them.
|
| 18 |
+
|
| 19 |
+
Both bite the hackathon strategy directly: the unfair advantage is running *different
|
| 20 |
+
sponsor models in one cast* (Judge → Nemotron, a worker → MiniCPM) to qualify for
|
| 21 |
+
multiple tracks from a single submission. And the Fishbowl Lab needed to let a user pick
|
| 22 |
+
concrete Modal-hosted models per cast member — with the pick actually driving the run
|
| 23 |
+
(the Lab's model controls were previously cosmetic: `on_summon` ignored
|
| 24 |
+
`collect_world_config` and always built the scenario's default cast).
|
| 25 |
+
|
| 26 |
+
## Decision
|
| 27 |
+
|
| 28 |
+
Add an optional, additive per-agent override that names a **specific catalogue model**,
|
| 29 |
+
leaving the tier system as the default and fallback.
|
| 30 |
+
|
| 31 |
+
- **Manifest.** `AgentManifest` gains `model_endpoint: str | None = None` — a
|
| 32 |
+
`modal/catalogue.py` endpoint slug. `None` → route by `model_profile` (unchanged).
|
| 33 |
+
- **Routing.** `ManifestAgent` routes by a **route key** —
|
| 34 |
+
`self._route_key = model_endpoint or model_profile` — and calls
|
| 35 |
+
`router.for_profile(self._route_key)`. The `ModelRouter` already accepts any key;
|
| 36 |
+
`_spec_for` now resolves a non-tier key against the catalogue (`_catalogue_spec`:
|
| 37 |
+
`modal_catalogue.binding_for(key)` for the live model string / endpoint URL / api key,
|
| 38 |
+
with decoding inherited from the model's tier — an unbound specialist → `balanced`).
|
| 39 |
+
An unknown non-tier key degrades to the `fast` tier rather than crashing. Offline this
|
| 40 |
+
path is never reached: `_build` serves the deterministic stub for any key, with the key
|
| 41 |
+
folded into the stub's `variant` so a different pick still varies (reproducible) output.
|
| 42 |
+
- **Composed runs.** `Registry.from_world(world)` builds an in-memory registry from a
|
| 43 |
+
validated `WorldConfig`, so a UI- (or LLM-) composed run flows through the same
|
| 44 |
+
`build_scenario` / `build_router` / `governor_for` path as a config-file run.
|
| 45 |
+
- **Fishbowl Lab.** The cast section is a `@gr.render` over the scenario: one model
|
| 46 |
+
`gr.Dropdown` per non-judge player (the Judge picks in §04), its choices sourced *only*
|
| 47 |
+
from `modal_catalogue.entries()`. Picks accumulate in a `cast_models` state;
|
| 48 |
+
`collect_world_config` maps each onto the agent's `model_endpoint` (re-checking the key
|
| 49 |
+
against the catalogue), and `Summon` runs the composed world. Only catalogue-hosted
|
| 50 |
+
models are offerable, and the selection is load-bearing.
|
| 51 |
+
|
| 52 |
+
## Consequences
|
| 53 |
+
|
| 54 |
+
- A cast can pin **any** catalogue model per agent, including the unbound specialists —
|
| 55 |
+
enabling genuine multi-sponsor-model casts from one engine, one submission.
|
| 56 |
+
- The tier abstraction (ADR-0010) is untouched: it remains the default, the decoding
|
| 57 |
+
source, the offline-variant tag, and the fallback. `model_endpoint` is purely additive,
|
| 58 |
+
so every existing manifest, scenario, and test is byte-identical (defaults to `None`).
|
| 59 |
+
- The Lab's model picker is now functional, not cosmetic: the model you choose is the
|
| 60 |
+
model that speaks (offline → the deterministic stub, demo still reproducible). A bad
|
| 61 |
+
compose degrades to the scenario's default cast, so Summon never breaks the demo.
|
| 62 |
+
- A run cannot point at an undeployed model: the UI offers only catalogue entries, and
|
| 63 |
+
`collect_world_config` re-validates the key, dropping anything out-of-band or stale.
|
| 64 |
+
- Offline determinism is preserved end-to-end (the route key, not just the tier, seeds
|
| 65 |
+
the stub).
|
| 66 |
+
|
| 67 |
+
## Alternatives considered
|
| 68 |
+
|
| 69 |
+
- **Per-tier rebinding** (let the run choose which catalogue model backs each of the four
|
| 70 |
+
tiers): zero engine change, but still capped at four distinct models and still cannot
|
| 71 |
+
reach the unbound specialists. Rejected as too weak for the multi-sponsor goal.
|
| 72 |
+
- **Widening `ModelProfile` to an arbitrary `str`**: would dissolve the tier contract that
|
| 73 |
+
drives decoding defaults, the `MODEL_<TIER>` env overrides, and the offline variant.
|
| 74 |
+
Rejected in favour of a separate, additive field that keeps both concepts crisp.
|
| 75 |
+
|
| 76 |
+
## Code
|
| 77 |
+
|
| 78 |
+
- `src/core/manifest.py` — `AgentManifest.model_endpoint`
|
| 79 |
+
- `src/agents/base.py` — `ManifestAgent._route_key`
|
| 80 |
+
- `src/models/router.py` — `ModelRouter._spec_for` / `_catalogue_spec`
|
| 81 |
+
- `src/core/registry.py` — `Registry.from_world`
|
| 82 |
+
- `src/ui/fishbowl/lab.py` — `model_choices`, `_cast_defaults`, `_judge_manifest`, the
|
| 83 |
+
cast `gr.render`, `collect_world_config`
|
| 84 |
+
- `src/ui/fishbowl/app.py` — `_compose_session`, the `Summon` wiring
|
| 85 |
+
|
| 86 |
+
See also: ADR-0010 (logical-profile routing), ADR-0011 (declarative validatable config),
|
| 87 |
+
ADR-0015 (LiteLLM gateway to Modal models), ADR-0019 (single model catalogue), ADR-0021
|
| 88 |
+
(Fishbowl Gradio presenter).
|
|
@@ -47,14 +47,22 @@ play-head, so concurrent visitors never share a world.
|
|
| 47 |
### The Lab — compose a run
|
| 48 |
|
| 49 |
`lab.py` (`build_lab`) is the full interactive composer (ADR-0021, decision 4): a
|
| 50 |
-
scenario grid, premise / seed / world text fields, a narrator `gr.Dropdown`,
|
| 51 |
-
**
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
|
| 59 |
### The Show — watch it unfold
|
| 60 |
|
|
|
|
| 47 |
### The Lab — compose a run
|
| 48 |
|
| 49 |
`lab.py` (`build_lab`) is the full interactive composer (ADR-0021, decision 4): a
|
| 50 |
+
scenario grid, premise / seed / world text fields, a narrator `gr.Dropdown`, a
|
| 51 |
+
**per-cast model picker**, a judge `gr.Group`, a tools `gr.CheckboxGroup`, and budget
|
| 52 |
+
`gr.Number`/`gr.Slider` controls.
|
| 53 |
+
|
| 54 |
+
The cast picker is a `@gr.render` over the scenario: one row per player (name + a model
|
| 55 |
+
`gr.Dropdown`), re-rendered as the cast changes. Crucially, **every dropdown offers only
|
| 56 |
+
the models actually hosted on Modal** — its choices come from `modal_catalogue.entries()`
|
| 57 |
+
(the catalogue is the single source of truth and loads offline), so you can't cast a model
|
| 58 |
+
that isn't deployable. Each pick writes the chosen endpoint slug into a `cast_models`
|
| 59 |
+
`gr.State` (`{agent_name: endpoint_key}`); the Judge gets its own catalogue dropdown in
|
| 60 |
+
§04. "Surprise me" rerolls a cast; **"Summon" makes the choice real**: `collect_world_config`
|
| 61 |
+
maps each selection onto the agent's `model_endpoint` (ADR-0022), runs the per-run
|
| 62 |
+
`WorldConfig` through `validate_world()` (ADR-0011), and `Registry.from_world()` builds a
|
| 63 |
+
`Conductor` on the exact same engine path as a config-file run — so the model you pick is
|
| 64 |
+
the model that speaks (offline → the deterministic stub, demo still reproducible). A bad
|
| 65 |
+
compose degrades to the scenario's default cast, so Summon never breaks the demo.
|
| 66 |
|
| 67 |
### The Show — watch it unfold
|
| 68 |
|
|
@@ -25,6 +25,7 @@ class AgentManifest(BaseModel):
|
|
| 25 |
|
| 26 |
# Model
|
| 27 |
model_profile: ModelProfile # tiny | fast | balanced | strong
|
|
|
|
| 28 |
|
| 29 |
# Memory
|
| 30 |
memory: MemoryConfig # window, use_salience, salience_top_k, reflection_threshold
|
|
@@ -98,6 +99,15 @@ Logical model tier. Resolved to a concrete model name at runtime:
|
|
| 98 |
|
| 99 |
The pattern: workers use `fast` or `tiny`; the judge and reflector use `balanced` or `strong`.
|
| 100 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 101 |
### `memory.window`
|
| 102 |
Number of recent visible events to include in every prompt (recency-window mode).
|
| 103 |
Default: 8. Reduce to 4–5 for very small models.
|
|
|
|
| 25 |
|
| 26 |
# Model
|
| 27 |
model_profile: ModelProfile # tiny | fast | balanced | strong
|
| 28 |
+
model_endpoint: str | None # optional: pin ONE catalogue model, overriding the tier
|
| 29 |
|
| 30 |
# Memory
|
| 31 |
memory: MemoryConfig # window, use_salience, salience_top_k, reflection_threshold
|
|
|
|
| 99 |
|
| 100 |
The pattern: workers use `fast` or `tiny`; the judge and reflector use `balanced` or `strong`.
|
| 101 |
|
| 102 |
+
### `model_endpoint`
|
| 103 |
+
Optional escape hatch from tiers to a **specific served model**: a `modal/catalogue.py`
|
| 104 |
+
endpoint slug (e.g. `minicpm-4-1-8b`) the router binds this agent to, overriding
|
| 105 |
+
`model_profile`. `None` (default) → route by tier. Lets a cast mix concrete sponsor
|
| 106 |
+
models (one mind on MiniCPM, the Judge on Nemotron Cascade), including the *unbound
|
| 107 |
+
specialist* models no tier defaults to. Offline it folds into the deterministic stub like
|
| 108 |
+
any tier, so demos stay reproducible. This is what the Fishbowl Lab's per-cast model picker
|
| 109 |
+
writes. See [model-routing.md](model-routing.md) and ADR-0022.
|
| 110 |
+
|
| 111 |
### `memory.window`
|
| 112 |
Number of recent visible events to include in every prompt (recency-window mode).
|
| 113 |
Default: 8. Reduce to 4–5 for very small models.
|
|
@@ -17,13 +17,38 @@ names a model.
|
|
| 17 |
## How a turn resolves a model
|
| 18 |
|
| 19 |
```
|
| 20 |
-
manifest.model_profile ──► ModelRouter.for_profile(
|
| 21 |
-
(
|
| 22 |
```
|
| 23 |
|
| 24 |
-
`ManifestAgent
|
| 25 |
-
|
| 26 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
|
| 28 |
## Transport: the LiteLLM gateway (live path)
|
| 29 |
|
|
@@ -118,7 +143,9 @@ everything on the big model.
|
|
| 118 |
|
| 119 |
## Code
|
| 120 |
|
| 121 |
-
- `src/models/router.py` — `ModelRouter`, `ProfileSpec`, `_PROFILE_DECODING`
|
|
|
|
|
|
|
| 122 |
- `src/models/litellm_provider.py` — `LiteLLMProvider` (live transport, real cost)
|
| 123 |
- `src/models/modal_catalogue.py` — engine view of the catalogue (key → binding)
|
| 124 |
- `src/core/manifest.py` — `resolve_model()` (env → catalogue default)
|
|
|
|
| 17 |
## How a turn resolves a model
|
| 18 |
|
| 19 |
```
|
| 20 |
+
manifest.model_endpoint or model_profile ──► ModelRouter.for_profile(key) ──► ModelProvider
|
| 21 |
+
(the agent's "route key") (cached per key) (concrete model)
|
| 22 |
```
|
| 23 |
|
| 24 |
+
`ManifestAgent` computes a **route key** — `self._route_key`, the explicit
|
| 25 |
+
`model_endpoint` when set, else the `model_profile` tier — and calls
|
| 26 |
+
`router.for_profile(self._route_key)` every turn, recording the provider's
|
| 27 |
+
`last_usage` so the conductor can meter tokens (and, live, real cost) into the
|
| 28 |
+
Governor. The router accepts either kind of key: a tier resolves to the profile
|
| 29 |
+
default, a catalogue endpoint slug to that specific model's binding.
|
| 30 |
+
|
| 31 |
+
## Pinning a specific model (`model_endpoint`)
|
| 32 |
+
|
| 33 |
+
Tiers are the default, but a manifest can pin one mind to a **specific catalogue
|
| 34 |
+
model** by setting `model_endpoint` to an endpoint slug from `modal/catalogue.py`
|
| 35 |
+
(e.g. `minicpm-4-1-8b`). This overrides the tier and is how a cast mixes concrete
|
| 36 |
+
sponsor models — one worker on MiniCPM, the Judge on Nemotron Cascade — including the
|
| 37 |
+
*unbound specialist* models that no tier defaults to. See ADR-0022.
|
| 38 |
+
|
| 39 |
+
```
|
| 40 |
+
ModelRouter._spec_for(key)
|
| 41 |
+
key in specs → that ProfileSpec (the four tiers from models.yaml)
|
| 42 |
+
key is a catalogue endpoint → _catalogue_spec(key): binding_for(key) + the model's
|
| 43 |
+
tier decoding (unbound specialist → balanced defaults)
|
| 44 |
+
unknown non-tier key → degrade to the fast tier (never crash)
|
| 45 |
+
```
|
| 46 |
+
|
| 47 |
+
Offline this path is never reached — `_build` serves the deterministic stub for any
|
| 48 |
+
key, with the key folded into the stub's `variant`, so picking a different model still
|
| 49 |
+
varies the (reproducible) output. The **Fishbowl Lab** writes `model_endpoint` from
|
| 50 |
+
its per-cast model picker, so the model you choose in the UI is the model that runs
|
| 51 |
+
(see [fishbowl-ui.md](fishbowl-ui.md)).
|
| 52 |
|
| 53 |
## Transport: the LiteLLM gateway (live path)
|
| 54 |
|
|
|
|
| 143 |
|
| 144 |
## Code
|
| 145 |
|
| 146 |
+
- `src/models/router.py` — `ModelRouter`, `ProfileSpec`, `_PROFILE_DECODING`, `_catalogue_spec()` (endpoint key → binding)
|
| 147 |
+
- `src/agents/base.py` — `ManifestAgent._route_key` (endpoint-or-tier)
|
| 148 |
+
- `src/core/registry.py` — `Registry.from_world()` (a UI/LLM-composed run on the same path)
|
| 149 |
- `src/models/litellm_provider.py` — `LiteLLMProvider` (live transport, real cost)
|
| 150 |
- `src/models/modal_catalogue.py` — engine view of the catalogue (key → binding)
|
| 151 |
- `src/core/manifest.py` — `resolve_model()` (env → catalogue default)
|
|
@@ -28,6 +28,8 @@ schedule:
|
|
| 28 |
|
| 29 |
# Model (resolved to a concrete small model by the ModelRouter)
|
| 30 |
model_profile: fast # tiny ≤4B | fast ≤7B | balanced ≤13B | strong ≤32B
|
|
|
|
|
|
|
| 31 |
|
| 32 |
# Memory (a view over the ledger, not separate state)
|
| 33 |
memory:
|
|
@@ -52,6 +54,12 @@ output_extra_fields: [] # extra payload fields the model is asked for, e.
|
|
| 52 |
reactive, periodic, or both. Cadence is per-agent; scenarios don't schedule.
|
| 53 |
- **`model_profile`** never names a model; the router (config/env) does. Mix
|
| 54 |
tiers freely across a cast.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 55 |
- **`handler`** stays `null` for the common case (the generic `ManifestAgent`).
|
| 56 |
Set it to a key registered via `@register_handler` for agents that call tools or
|
| 57 |
need custom prompt logic; the YAML still supplies all declarative fields.
|
|
|
|
| 28 |
|
| 29 |
# Model (resolved to a concrete small model by the ModelRouter)
|
| 30 |
model_profile: fast # tiny ≤4B | fast ≤7B | balanced ≤13B | strong ≤32B
|
| 31 |
+
model_endpoint: null # optional: pin ONE specific catalogue model (modal/catalogue.py
|
| 32 |
+
# endpoint slug, e.g. minicpm-4-1-8b), overriding the tier above
|
| 33 |
|
| 34 |
# Memory (a view over the ledger, not separate state)
|
| 35 |
memory:
|
|
|
|
| 54 |
reactive, periodic, or both. Cadence is per-agent; scenarios don't schedule.
|
| 55 |
- **`model_profile`** never names a model; the router (config/env) does. Mix
|
| 56 |
tiers freely across a cast.
|
| 57 |
+
- **`model_endpoint`** is the escape hatch from tiers to a *specific* served model:
|
| 58 |
+
a `modal/catalogue.py` endpoint slug the router resolves to that model's live
|
| 59 |
+
binding (overriding `model_profile`). `null` → route by tier. This is how a cast
|
| 60 |
+
pins concrete sponsor models — one mind on MiniCPM, the Judge on Nemotron — and what
|
| 61 |
+
the Fishbowl Lab's per-cast model picker writes. Offline it folds into the
|
| 62 |
+
deterministic stub like any tier, so demos stay reproducible. See ADR-0022.
|
| 63 |
- **`handler`** stays `null` for the common case (the generic `ManifestAgent`).
|
| 64 |
Set it to a key registered via `@register_handler` for agents that call tools or
|
| 65 |
need custom prompt logic; the YAML still supplies all declarative fields.
|
|
@@ -81,9 +81,17 @@ changes needed:
|
|
| 81 |
| `reasoning_parser` / `tool_call_parser` / `enable_auto_tool_choice` | OpenAI tool/reasoning features. |
|
| 82 |
| `multimodal` / `mm_limits` | Image/audio/video inputs and per-prompt caps. |
|
| 83 |
| `trust_remote_code` | Required by MiniCPM / Nemotron custom modeling code. |
|
|
|
|
| 84 |
| `extra_vllm_args` | Raw `vllm serve` flags appended verbatim (escape hatch). |
|
| 85 |
| `extra_pip` / `env` | Extra image deps / container env (escape hatch). |
|
| 86 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
### Add a model
|
| 88 |
|
| 89 |
Append one `ModelConfig` to the appropriate provider list in `catalogue.py` (tag
|
|
|
|
| 81 |
| `reasoning_parser` / `tool_call_parser` / `enable_auto_tool_choice` | OpenAI tool/reasoning features. |
|
| 82 |
| `multimodal` / `mm_limits` | Image/audio/video inputs and per-prompt caps. |
|
| 83 |
| `trust_remote_code` | Required by MiniCPM / Nemotron custom modeling code. |
|
| 84 |
+
| `vllm_version` | Per-model inference-stack pin (escape hatch); `None` = the default `VLLM_VERSION`, `"nightly"` = latest nightly wheel, else a pinned version. |
|
| 85 |
| `extra_vllm_args` | Raw `vllm serve` flags appended verbatim (escape hatch). |
|
| 86 |
| `extra_pip` / `env` | Extra image deps / container env (escape hatch). |
|
| 87 |
|
| 88 |
+
> **Per-model vLLM version.** The image pins `VLLM_VERSION` (see `service.py`) for
|
| 89 |
+
> reproducible deploys. A single model can override it via `vllm_version` when the
|
| 90 |
+
> pinned release can't serve its architecture — this is scoped to that model's image,
|
| 91 |
+
> so one model's bump never touches another provider's app. The Gemma 4 entries set
|
| 92 |
+
> `vllm_version="nightly"` (plus `transformers>=5.10.2` and `VLLM_USE_FLASHINFER_SAMPLER=0`)
|
| 93 |
+
> because the `gemma4_unified` architecture is unservable on the pinned release.
|
| 94 |
+
|
| 95 |
### Add a model
|
| 96 |
|
| 97 |
Append one `ModelConfig` to the appropriate provider list in `catalogue.py` (tag
|
|
@@ -86,12 +86,16 @@ _BASE_ENV = {
|
|
| 86 |
def build_image(cfg: ModelConfig) -> modal.Image:
|
| 87 |
"""Build the container image for a model. Layers are cached and shared, so
|
| 88 |
text models that only differ in env reuse the same base layers."""
|
| 89 |
-
image = (
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
if cfg.extra_pip:
|
| 96 |
image = image.uv_pip_install(*cfg.extra_pip)
|
| 97 |
if cfg.env:
|
|
|
|
| 86 |
def build_image(cfg: ModelConfig) -> modal.Image:
|
| 87 |
"""Build the container image for a model. Layers are cached and shared, so
|
| 88 |
text models that only differ in env reuse the same base layers."""
|
| 89 |
+
image = modal.Image.from_registry(CUDA_IMAGE, add_python=PYTHON_VERSION).entrypoint(
|
| 90 |
+
[]
|
| 91 |
+
) # drop the CUDA image's default entrypoint
|
| 92 |
+
# vLLM version is per-model (defaults to the pinned VLLM_VERSION). A model can
|
| 93 |
+
# opt into a nightly wheel when the pinned release can't serve its architecture.
|
| 94 |
+
if cfg.vllm_version == "nightly":
|
| 95 |
+
image = image.uv_pip_install("vllm", pre=True, extra_index_url="https://wheels.vllm.ai/nightly")
|
| 96 |
+
else:
|
| 97 |
+
image = image.uv_pip_install(f"vllm=={cfg.vllm_version or VLLM_VERSION}")
|
| 98 |
+
image = image.env(_BASE_ENV)
|
| 99 |
if cfg.extra_pip:
|
| 100 |
image = image.uv_pip_install(*cfg.extra_pip)
|
| 101 |
if cfg.env:
|
|
@@ -16,6 +16,7 @@ Backward compatibility: Phase-0/1 agents extend Agent directly and are
|
|
| 16 |
unaffected. The conductor checks ``getattr(agent, "manifest", None)`` to
|
| 17 |
decide whether manifest-based routing applies.
|
| 18 |
"""
|
|
|
|
| 19 |
from __future__ import annotations
|
| 20 |
|
| 21 |
from abc import ABC, abstractmethod
|
|
@@ -42,6 +43,7 @@ _REFLECTION_KIND = "agent.reflected"
|
|
| 42 |
|
| 43 |
# ── minimal interface ─────────────────────────────────────────────────────────
|
| 44 |
|
|
|
|
| 45 |
class Agent(ABC):
|
| 46 |
name: str
|
| 47 |
|
|
@@ -58,6 +60,7 @@ class Agent(ABC):
|
|
| 58 |
|
| 59 |
# ── manifest-driven base ──────────────────────────────────────────────────────
|
| 60 |
|
|
|
|
| 61 |
class ManifestAgent(Agent):
|
| 62 |
"""Base class for manifest-driven agents.
|
| 63 |
|
|
@@ -141,6 +144,16 @@ class ManifestAgent(Agent):
|
|
| 141 |
|
| 142 |
# ── model routing ─────────────────────────────────────────────────────────
|
| 143 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
def _resolve_payload(
|
| 145 |
self,
|
| 146 |
role: str,
|
|
@@ -157,7 +170,7 @@ class ManifestAgent(Agent):
|
|
| 157 |
instruction and run the tolerant parser as before. Token/cost usage is
|
| 158 |
recorded from the provider in both paths.
|
| 159 |
"""
|
| 160 |
-
provider = self.router.for_profile(self.
|
| 161 |
if hasattr(provider, "complete_structured"):
|
| 162 |
model = build_output_model(allowed, extra_fields)
|
| 163 |
try:
|
|
@@ -176,7 +189,7 @@ class ManifestAgent(Agent):
|
|
| 176 |
|
| 177 |
def _complete(self, role: str, prompt: str) -> str:
|
| 178 |
"""Route to the provider for this agent's profile and record token usage."""
|
| 179 |
-
provider = self.router.for_profile(self.
|
| 180 |
raw = provider.complete(role, prompt)
|
| 181 |
self.last_usage = dict(provider.last_usage)
|
| 182 |
return raw
|
|
@@ -206,7 +219,7 @@ class ManifestAgent(Agent):
|
|
| 206 |
f"RECENT MEMORY (events you witnessed)\n{memory}\n\n"
|
| 207 |
"TASK\nSynthesise the above into ONE short, high-level belief about yourself or the "
|
| 208 |
"world. It will replace raw memories in your future context.\n\n"
|
| 209 |
-
|
| 210 |
'{"kind": "agent.reflected", "text": "<one-sentence belief>"}'
|
| 211 |
)
|
| 212 |
raw = self._complete(self.manifest.name + "-reflect", prompt)
|
|
|
|
| 16 |
unaffected. The conductor checks ``getattr(agent, "manifest", None)`` to
|
| 17 |
decide whether manifest-based routing applies.
|
| 18 |
"""
|
| 19 |
+
|
| 20 |
from __future__ import annotations
|
| 21 |
|
| 22 |
from abc import ABC, abstractmethod
|
|
|
|
| 43 |
|
| 44 |
# ── minimal interface ─────────────────────────────────────────────────────────
|
| 45 |
|
| 46 |
+
|
| 47 |
class Agent(ABC):
|
| 48 |
name: str
|
| 49 |
|
|
|
|
| 60 |
|
| 61 |
# ── manifest-driven base ──────────────────────────────────────────────────────
|
| 62 |
|
| 63 |
+
|
| 64 |
class ManifestAgent(Agent):
|
| 65 |
"""Base class for manifest-driven agents.
|
| 66 |
|
|
|
|
| 144 |
|
| 145 |
# ── model routing ─────────────────────────────────────────────────────────
|
| 146 |
|
| 147 |
+
@property
|
| 148 |
+
def _route_key(self) -> str:
|
| 149 |
+
"""Router key for this agent: the explicit ``model_endpoint`` catalogue key
|
| 150 |
+
when set (a specific served model), else the logical ``model_profile`` tier.
|
| 151 |
+
|
| 152 |
+
The router accepts either — a catalogue key resolves to that model's live
|
| 153 |
+
binding, a tier to the profile default — so an agent can be pinned to one
|
| 154 |
+
concrete Modal model without the engine naming a model anywhere (ADR-0022)."""
|
| 155 |
+
return self.manifest.model_endpoint or self.manifest.model_profile
|
| 156 |
+
|
| 157 |
def _resolve_payload(
|
| 158 |
self,
|
| 159 |
role: str,
|
|
|
|
| 170 |
instruction and run the tolerant parser as before. Token/cost usage is
|
| 171 |
recorded from the provider in both paths.
|
| 172 |
"""
|
| 173 |
+
provider = self.router.for_profile(self._route_key)
|
| 174 |
if hasattr(provider, "complete_structured"):
|
| 175 |
model = build_output_model(allowed, extra_fields)
|
| 176 |
try:
|
|
|
|
| 189 |
|
| 190 |
def _complete(self, role: str, prompt: str) -> str:
|
| 191 |
"""Route to the provider for this agent's profile and record token usage."""
|
| 192 |
+
provider = self.router.for_profile(self._route_key)
|
| 193 |
raw = provider.complete(role, prompt)
|
| 194 |
self.last_usage = dict(provider.last_usage)
|
| 195 |
return raw
|
|
|
|
| 219 |
f"RECENT MEMORY (events you witnessed)\n{memory}\n\n"
|
| 220 |
"TASK\nSynthesise the above into ONE short, high-level belief about yourself or the "
|
| 221 |
"world. It will replace raw memories in your future context.\n\n"
|
| 222 |
+
"OUTPUT FORMAT\nReply with a single JSON object and nothing else: "
|
| 223 |
'{"kind": "agent.reflected", "text": "<one-sentence belief>"}'
|
| 224 |
)
|
| 225 |
raw = self._complete(self.manifest.name + "-reflect", prompt)
|
|
@@ -104,6 +104,16 @@ class AgentManifest(BaseModel):
|
|
| 104 |
"""Logical profile: resolved to a concrete model name by the provider.
|
| 105 |
tiny=<=4B, fast=<=7B, balanced=<=13B, strong=<=32B."""
|
| 106 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
# Memory
|
| 108 |
memory: MemoryConfig = Field(default_factory=MemoryConfig)
|
| 109 |
|
|
|
|
| 104 |
"""Logical profile: resolved to a concrete model name by the provider.
|
| 105 |
tiny=<=4B, fast=<=7B, balanced=<=13B, strong=<=32B."""
|
| 106 |
|
| 107 |
+
model_endpoint: str | None = None
|
| 108 |
+
"""Optional catalogue key (``modal/catalogue.py`` endpoint slug, e.g.
|
| 109 |
+
``"minicpm-4-1-8b"``) binding this agent to one *specific* served model,
|
| 110 |
+
overriding the tier default in ``model_profile``. None → route by profile.
|
| 111 |
+
|
| 112 |
+
This is how a cast mixes concrete sponsor models — one mind on MiniCPM, the
|
| 113 |
+
Judge on Nemotron — while ``model_profile`` stays the decoding/offline fallback.
|
| 114 |
+
The router resolves the key against the live catalogue (ADR-0022); offline it
|
| 115 |
+
folds into the deterministic stub like any profile, so demos stay reproducible."""
|
| 116 |
+
|
| 117 |
# Memory
|
| 118 |
memory: MemoryConfig = Field(default_factory=MemoryConfig)
|
| 119 |
|
|
@@ -22,7 +22,14 @@ from pathlib import Path
|
|
| 22 |
import yaml
|
| 23 |
|
| 24 |
from src.agents.base import Agent, ManifestAgent
|
| 25 |
-
from src.core.config import
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 26 |
from src.core.governor import Governor
|
| 27 |
from src.core.manifest import AgentManifest
|
| 28 |
from src.models.router import ModelRouter, ProfileSpec
|
|
@@ -156,6 +163,21 @@ class Registry:
|
|
| 156 |
|
| 157 |
return cls(agents=agents, scenarios=scenarios, models=models)
|
| 158 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
# ── building ───────────────────────────────────────────────────────────────
|
| 160 |
|
| 161 |
def build_router(self) -> ModelRouter:
|
|
|
|
| 22 |
import yaml
|
| 23 |
|
| 24 |
from src.agents.base import Agent, ManifestAgent
|
| 25 |
+
from src.core.config import (
|
| 26 |
+
GovernorConfig,
|
| 27 |
+
ModelsConfig,
|
| 28 |
+
ScenarioConfig,
|
| 29 |
+
WorldConfig,
|
| 30 |
+
validate_agent,
|
| 31 |
+
validate_scenario,
|
| 32 |
+
)
|
| 33 |
from src.core.governor import Governor
|
| 34 |
from src.core.manifest import AgentManifest
|
| 35 |
from src.models.router import ModelRouter, ProfileSpec
|
|
|
|
| 163 |
|
| 164 |
return cls(agents=agents, scenarios=scenarios, models=models)
|
| 165 |
|
| 166 |
+
@classmethod
|
| 167 |
+
def from_world(cls, world: WorldConfig) -> "Registry":
|
| 168 |
+
"""Build an in-memory registry from a composed, validated :class:`WorldConfig`.
|
| 169 |
+
|
| 170 |
+
The in-memory mirror of :meth:`from_dir`: agents, scenarios, and model
|
| 171 |
+
bindings come straight off the world object instead of ``config/``. So a
|
| 172 |
+
run composed by the Lab (or an LLM) flows through the exact same
|
| 173 |
+
``build_scenario`` / ``build_router`` / ``governor_for`` path as a
|
| 174 |
+
config-file run — emit a world, validate it, run it. See ADR-0011 / ADR-0022."""
|
| 175 |
+
return cls(
|
| 176 |
+
agents={a.name: a for a in world.agents},
|
| 177 |
+
scenarios={s.name: s for s in world.scenarios},
|
| 178 |
+
models=world.models,
|
| 179 |
+
)
|
| 180 |
+
|
| 181 |
# ── building ───────────────────────────────────────────────────────────────
|
| 182 |
|
| 183 |
def build_router(self) -> ModelRouter:
|
|
@@ -101,13 +101,49 @@ class ModelRouter:
|
|
| 101 |
def _spec_for(self, profile: str) -> ProfileSpec:
|
| 102 |
if profile in self.specs:
|
| 103 |
return self.specs[profile]
|
| 104 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 105 |
return ProfileSpec(
|
| 106 |
model=resolve_model(profile), # type: ignore[arg-type]
|
| 107 |
temperature=float(decoding["temperature"]),
|
| 108 |
max_tokens=int(decoding["max_tokens"]),
|
| 109 |
)
|
| 110 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 111 |
# ── factory ─────────────────────────────────────────────────────────────
|
| 112 |
|
| 113 |
@classmethod
|
|
|
|
| 101 |
def _spec_for(self, profile: str) -> ProfileSpec:
|
| 102 |
if profile in self.specs:
|
| 103 |
return self.specs[profile]
|
| 104 |
+
# A key that is not one of the four tiers names a *specific* catalogue model
|
| 105 |
+
# (an agent's ``model_endpoint``): resolve it to that model's live binding so
|
| 106 |
+
# a cast can pin concrete Modal models (ADR-0022). Only reached on the live
|
| 107 |
+
# path — offline, ``_build`` serves the stub before this runs.
|
| 108 |
+
if profile not in _PROFILE_DECODING:
|
| 109 |
+
spec = self._catalogue_spec(profile)
|
| 110 |
+
if spec is not None:
|
| 111 |
+
return spec
|
| 112 |
+
# Unknown non-tier key with no catalogue match → degrade to the fast tier
|
| 113 |
+
# rather than crash ``resolve_model`` on a key it does not recognise.
|
| 114 |
+
profile = "fast"
|
| 115 |
+
decoding = _PROFILE_DECODING[profile]
|
| 116 |
return ProfileSpec(
|
| 117 |
model=resolve_model(profile), # type: ignore[arg-type]
|
| 118 |
temperature=float(decoding["temperature"]),
|
| 119 |
max_tokens=int(decoding["max_tokens"]),
|
| 120 |
)
|
| 121 |
|
| 122 |
+
def _catalogue_spec(self, key: str) -> ProfileSpec | None:
|
| 123 |
+
"""Build a :class:`ProfileSpec` from a catalogue endpoint *key*, or None.
|
| 124 |
+
|
| 125 |
+
The model string / endpoint URL / api key come from the catalogue + env
|
| 126 |
+
(``modal_catalogue.binding_for``); decoding inherits the model's tier default
|
| 127 |
+
(an unbound specialist model → the balanced tier). Returns None when the key
|
| 128 |
+
is not in the catalogue, so the caller can fall back gracefully."""
|
| 129 |
+
try:
|
| 130 |
+
from src.models import modal_catalogue
|
| 131 |
+
|
| 132 |
+
entry = modal_catalogue.entry_by_key(key)
|
| 133 |
+
if entry is None:
|
| 134 |
+
return None
|
| 135 |
+
binding = modal_catalogue.binding_for(key)
|
| 136 |
+
except Exception: # pragma: no cover - defensive: catalogue unavailable
|
| 137 |
+
return None
|
| 138 |
+
decoding = _PROFILE_DECODING.get(entry.get("profile") or "balanced", _PROFILE_DECODING["balanced"])
|
| 139 |
+
return ProfileSpec(
|
| 140 |
+
model=binding["model"],
|
| 141 |
+
base_url=binding["base_url"] or None,
|
| 142 |
+
api_key=binding["api_key"] or None,
|
| 143 |
+
temperature=float(decoding["temperature"]),
|
| 144 |
+
max_tokens=int(decoding["max_tokens"]),
|
| 145 |
+
)
|
| 146 |
+
|
| 147 |
# ── factory ─────────────────────────────────────────────────────────────
|
| 148 |
|
| 149 |
@classmethod
|
|
@@ -527,6 +527,16 @@ def _wire(
|
|
| 527 |
summon_btn = _h(lab_handles, "summon", "summon_btn", "launch", "start")
|
| 528 |
scenario_in = _h(lab_handles, "scenario", "scenario_select", "scenario_dropdown")
|
| 529 |
seed_in = _h(lab_handles, "seed", "seed_in", "world_seed")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 530 |
|
| 531 |
def _scenario_title(value) -> str:
|
| 532 |
"""Resolve a Lab scenario selection (title or internal name) to a title key."""
|
|
@@ -537,11 +547,67 @@ def _wire(
|
|
| 537 |
return title
|
| 538 |
return _DEFAULT_TITLE
|
| 539 |
|
| 540 |
-
|
| 541 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 542 |
title = _scenario_title(scenario_value)
|
| 543 |
name = SCENARIOS.get(title, "")
|
| 544 |
-
session =
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 545 |
if session is not None:
|
| 546 |
session.reset((seed_value or "").strip())
|
| 547 |
k = session.head
|
|
@@ -555,6 +621,14 @@ def _wire(
|
|
| 555 |
summon_inputs = [
|
| 556 |
scenario_in if scenario_in is not None else scenario_state,
|
| 557 |
seed_in if seed_in is not None else blank_state, # empty seed when no widget
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 558 |
layout_state,
|
| 559 |
mind_reader_state,
|
| 560 |
]
|
|
@@ -568,30 +642,33 @@ def _wire(
|
|
| 568 |
# Without this, switching to a new world (e.g. the spy game) leaves the premise,
|
| 569 |
# seed, cast table, and narrator showing the previous scenario — and Summon would
|
| 570 |
# genesis with the wrong seed. Refreshing them makes "compose a spy game" one click.
|
|
|
|
|
|
|
| 571 |
_scenario_fields = [
|
| 572 |
(_h(lab_handles, "premise"), "premise"),
|
| 573 |
(seed_in, "seed"),
|
| 574 |
(_h(lab_handles, "world"), "world"),
|
| 575 |
-
(_h(lab_handles, "cast"), "cast"),
|
| 576 |
(_h(lab_handles, "narrator"), "narrator"),
|
|
|
|
| 577 |
]
|
| 578 |
_present_fields = [(handle, key) for handle, key in _scenario_fields if handle is not None]
|
| 579 |
|
| 580 |
if scenario_in is not None and _present_fields:
|
| 581 |
|
| 582 |
def on_scenario_change(scenario_value):
|
| 583 |
-
from src.ui.fishbowl.lab import
|
| 584 |
|
| 585 |
cfg = _registry.scenarios.get(SCENARIOS.get(_scenario_title(scenario_value), ""))
|
| 586 |
if cfg is None:
|
| 587 |
return tuple(gr.update() for _ in _present_fields)
|
| 588 |
seeds = list(cfg.example_seeds) or [cfg.default_seed]
|
|
|
|
| 589 |
updates = {
|
| 590 |
"premise": gr.update(value=cfg.goal),
|
| 591 |
"seed": gr.update(choices=seeds, value=cfg.default_seed),
|
| 592 |
"world": gr.update(value=cfg.genesis_text or ""),
|
| 593 |
-
"cast": gr.update(value=_cast_rows_for(cfg)),
|
| 594 |
"narrator": gr.update(value=scenario_voice(cfg.name)),
|
|
|
|
| 595 |
}
|
| 596 |
return tuple(updates[key] for _handle, key in _present_fields)
|
| 597 |
|
|
|
|
| 527 |
summon_btn = _h(lab_handles, "summon", "summon_btn", "launch", "start")
|
| 528 |
scenario_in = _h(lab_handles, "scenario", "scenario_select", "scenario_dropdown")
|
| 529 |
seed_in = _h(lab_handles, "seed", "seed_in", "world_seed")
|
| 530 |
+
# Composer inputs — the per-cast Modal model picks (cast_models) and the run knobs;
|
| 531 |
+
# all looked up defensively so the shell still runs if the Lab omits a widget.
|
| 532 |
+
premise_in = _h(lab_handles, "premise")
|
| 533 |
+
cast_models_in = _h(lab_handles, "cast_models")
|
| 534 |
+
judge_model_in = _h(lab_handles, "judge_model")
|
| 535 |
+
judge_policy_in = _h(lab_handles, "judge_policy")
|
| 536 |
+
judge_strictness_in = _h(lab_handles, "judge_strictness")
|
| 537 |
+
tools_in = _h(lab_handles, "tools")
|
| 538 |
+
tokens_in = _h(lab_handles, "tokens")
|
| 539 |
+
max_rounds_in = _h(lab_handles, "max_rounds")
|
| 540 |
|
| 541 |
def _scenario_title(value) -> str:
|
| 542 |
"""Resolve a Lab scenario selection (title or internal name) to a title key."""
|
|
|
|
| 547 |
return title
|
| 548 |
return _DEFAULT_TITLE
|
| 549 |
|
| 550 |
+
def _compose_session(name, **knobs):
|
| 551 |
+
"""Build a session for a Lab-composed run: the selected Modal models drive the
|
| 552 |
+
cast (ADR-0022). The composed WorldConfig flows through ``Registry.from_world``
|
| 553 |
+
onto the exact same engine path as a config-file run. Any compose/validate error
|
| 554 |
+
degrades to the scenario's default cast so Summon always yields a runnable show
|
| 555 |
+
(and with no credentials the deterministic stub drives it, demo reproducible)."""
|
| 556 |
+
from src.core.registry import Registry
|
| 557 |
+
from src.ui.fishbowl.lab import collect_world_config
|
| 558 |
+
|
| 559 |
+
try:
|
| 560 |
+
world = collect_world_config(
|
| 561 |
+
scenario=name,
|
| 562 |
+
premise=knobs.get("premise") or "",
|
| 563 |
+
seed=knobs.get("seed") or "",
|
| 564 |
+
cast_models=knobs.get("cast_models") if isinstance(knobs.get("cast_models"), dict) else {},
|
| 565 |
+
judge_policy=knobs.get("judge_policy") or "Majority Vote",
|
| 566 |
+
judge_model=knobs.get("judge_model") or "",
|
| 567 |
+
judge_strictness=knobs.get("judge_strictness")
|
| 568 |
+
if isinstance(knobs.get("judge_strictness"), (int, float))
|
| 569 |
+
else 50,
|
| 570 |
+
tools=knobs.get("tools") if isinstance(knobs.get("tools"), list) else [],
|
| 571 |
+
tokens=knobs.get("tokens") if isinstance(knobs.get("tokens"), (int, float)) else None,
|
| 572 |
+
max_rounds=knobs.get("max_rounds") if isinstance(knobs.get("max_rounds"), (int, float)) else None,
|
| 573 |
+
)
|
| 574 |
+
return FishbowlSession(name, registry=Registry.from_world(world), tools=_tools)
|
| 575 |
+
except Exception:
|
| 576 |
+
return _new_session(name) # bad compose → default cast; Summon never breaks
|
| 577 |
+
|
| 578 |
+
# ── Summon: build a fresh session from the composed run, reset, jump to the Show ──
|
| 579 |
+
def on_summon(
|
| 580 |
+
scenario_value,
|
| 581 |
+
seed_value,
|
| 582 |
+
premise,
|
| 583 |
+
cast_models,
|
| 584 |
+
judge_model,
|
| 585 |
+
judge_policy,
|
| 586 |
+
judge_strictness,
|
| 587 |
+
tools,
|
| 588 |
+
tokens,
|
| 589 |
+
max_rounds,
|
| 590 |
+
layout,
|
| 591 |
+
mind_reader,
|
| 592 |
+
):
|
| 593 |
title = _scenario_title(scenario_value)
|
| 594 |
name = SCENARIOS.get(title, "")
|
| 595 |
+
session = (
|
| 596 |
+
_compose_session(
|
| 597 |
+
name,
|
| 598 |
+
premise=premise,
|
| 599 |
+
seed=seed_value,
|
| 600 |
+
cast_models=cast_models,
|
| 601 |
+
judge_model=judge_model,
|
| 602 |
+
judge_policy=judge_policy,
|
| 603 |
+
judge_strictness=judge_strictness,
|
| 604 |
+
tools=tools,
|
| 605 |
+
tokens=tokens,
|
| 606 |
+
max_rounds=max_rounds,
|
| 607 |
+
)
|
| 608 |
+
if name
|
| 609 |
+
else None
|
| 610 |
+
)
|
| 611 |
if session is not None:
|
| 612 |
session.reset((seed_value or "").strip())
|
| 613 |
k = session.head
|
|
|
|
| 621 |
summon_inputs = [
|
| 622 |
scenario_in if scenario_in is not None else scenario_state,
|
| 623 |
seed_in if seed_in is not None else blank_state, # empty seed when no widget
|
| 624 |
+
premise_in if premise_in is not None else blank_state,
|
| 625 |
+
cast_models_in if cast_models_in is not None else blank_state,
|
| 626 |
+
judge_model_in if judge_model_in is not None else blank_state,
|
| 627 |
+
judge_policy_in if judge_policy_in is not None else blank_state,
|
| 628 |
+
judge_strictness_in if judge_strictness_in is not None else blank_state,
|
| 629 |
+
tools_in if tools_in is not None else blank_state,
|
| 630 |
+
tokens_in if tokens_in is not None else blank_state,
|
| 631 |
+
max_rounds_in if max_rounds_in is not None else blank_state,
|
| 632 |
layout_state,
|
| 633 |
mind_reader_state,
|
| 634 |
]
|
|
|
|
| 642 |
# Without this, switching to a new world (e.g. the spy game) leaves the premise,
|
| 643 |
# seed, cast table, and narrator showing the previous scenario — and Summon would
|
| 644 |
# genesis with the wrong seed. Refreshing them makes "compose a spy game" one click.
|
| 645 |
+
# The cast picker re-seeds itself (it is a gr.render over the scenario), so it is not
|
| 646 |
+
# in this list; we re-seed the static fields plus the Judge's default Modal model.
|
| 647 |
_scenario_fields = [
|
| 648 |
(_h(lab_handles, "premise"), "premise"),
|
| 649 |
(seed_in, "seed"),
|
| 650 |
(_h(lab_handles, "world"), "world"),
|
|
|
|
| 651 |
(_h(lab_handles, "narrator"), "narrator"),
|
| 652 |
+
(_h(lab_handles, "judge_model"), "judge_model"),
|
| 653 |
]
|
| 654 |
_present_fields = [(handle, key) for handle, key in _scenario_fields if handle is not None]
|
| 655 |
|
| 656 |
if scenario_in is not None and _present_fields:
|
| 657 |
|
| 658 |
def on_scenario_change(scenario_value):
|
| 659 |
+
from src.ui.fishbowl.lab import _default_model_key, _judge_manifest
|
| 660 |
|
| 661 |
cfg = _registry.scenarios.get(SCENARIOS.get(_scenario_title(scenario_value), ""))
|
| 662 |
if cfg is None:
|
| 663 |
return tuple(gr.update() for _ in _present_fields)
|
| 664 |
seeds = list(cfg.example_seeds) or [cfg.default_seed]
|
| 665 |
+
judge = _judge_manifest(cfg)
|
| 666 |
updates = {
|
| 667 |
"premise": gr.update(value=cfg.goal),
|
| 668 |
"seed": gr.update(choices=seeds, value=cfg.default_seed),
|
| 669 |
"world": gr.update(value=cfg.genesis_text or ""),
|
|
|
|
| 670 |
"narrator": gr.update(value=scenario_voice(cfg.name)),
|
| 671 |
+
"judge_model": gr.update(value=_default_model_key(judge) if judge else None),
|
| 672 |
}
|
| 673 |
return tuple(updates[key] for _handle, key in _present_fields)
|
| 674 |
|
|
@@ -23,7 +23,9 @@ from __future__ import annotations
|
|
| 23 |
import gradio as gr
|
| 24 |
|
| 25 |
from src.core.config import ScenarioConfig, validate_scenario, validate_world
|
|
|
|
| 26 |
from src.core.registry import default_registry
|
|
|
|
| 27 |
from src.ui.fishbowl.adapter import VOICES, scenario_voice
|
| 28 |
|
| 29 |
# ── design vocabulary (mirrors ui/raw/lab.jsx) ──────────────────────────────────
|
|
@@ -37,9 +39,6 @@ JUDGE_POLICIES: list[str] = [
|
|
| 37 |
"Judge's Whim",
|
| 38 |
]
|
| 39 |
|
| 40 |
-
# Logical model profiles the engine understands (manifest.ModelProfile).
|
| 41 |
-
MODEL_PROFILES: list[str] = ["tiny", "fast", "balanced", "strong"]
|
| 42 |
-
|
| 43 |
# MCP tool grants the cast may reach for (friendly label, stored id). Mirrors
|
| 44 |
# ui/raw/lab.jsx:MCP_TOOLS; the value is what we store on the run.
|
| 45 |
TOOL_CHOICES: list[tuple[str, str]] = [
|
|
@@ -51,9 +50,6 @@ TOOL_CHOICES: list[tuple[str, str]] = [
|
|
| 51 |
("tts.speak · give it a voice", "tts.speak"),
|
| 52 |
]
|
| 53 |
|
| 54 |
-
# Cast Dataframe columns (editable in the Lab).
|
| 55 |
-
CAST_COLUMNS: list[str] = ["name", "archetype", "model_profile", "temp"]
|
| 56 |
-
|
| 57 |
# The scenario we lead with — the hackathon's north-star world.
|
| 58 |
_PREFERRED_SCENARIO = "thousand-token-wood"
|
| 59 |
|
|
@@ -77,25 +73,62 @@ def _scenario_by_title(title: str) -> ScenarioConfig | None:
|
|
| 77 |
return None
|
| 78 |
|
| 79 |
|
| 80 |
-
def
|
| 81 |
-
"""
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 82 |
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
|
|
|
| 87 |
registry = default_registry()
|
| 88 |
-
|
| 89 |
for agent_name in scenario.cast:
|
| 90 |
manifest = registry.agents.get(agent_name)
|
| 91 |
-
if manifest is None:
|
| 92 |
-
# A scenario referencing an unknown agent shouldn't happen (validate_world
|
| 93 |
-
# guards it) but degrade gracefully rather than crash the form.
|
| 94 |
-
rows.append([agent_name, f"the {agent_name}", "fast", 0.8])
|
| 95 |
continue
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
|
|
|
| 99 |
|
| 100 |
|
| 101 |
def _voice_choices() -> list[tuple[str, str]]:
|
|
@@ -113,8 +146,9 @@ def build_lab() -> dict[str, gr.components.Component]:
|
|
| 113 |
Wires no callbacks and imports no sibling render/show module — the app shell
|
| 114 |
(Unit 9) binds ``summon_btn`` to the session. Returns a dict of every handle a
|
| 115 |
caller needs to read the composed run (keys: ``scenario, premise, seed, world,
|
| 116 |
-
narrator,
|
| 117 |
-
max_rounds, seed_num, cadence, summon_btn, surprise_btn``).
|
|
|
|
| 118 |
"""
|
| 119 |
scenarios = _ordered_scenarios()
|
| 120 |
first = scenarios[0]
|
|
@@ -160,25 +194,64 @@ def build_lab() -> dict[str, gr.components.Component]:
|
|
| 160 |
lines=2,
|
| 161 |
)
|
| 162 |
|
| 163 |
-
# 03 — The Cast
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 164 |
with gr.Group():
|
| 165 |
gr.Markdown(
|
| 166 |
-
"**03 · The Cast** — bind
|
| 167 |
-
"(
|
| 168 |
-
)
|
| 169 |
-
handles["cast"] = gr.Dataframe(
|
| 170 |
-
value=_cast_rows_for(first),
|
| 171 |
-
headers=CAST_COLUMNS,
|
| 172 |
-
datatype=["str", "str", "str", "number"],
|
| 173 |
-
column_count=len(CAST_COLUMNS),
|
| 174 |
-
# The grid re-seeds (name/archetype/model) when the scenario changes; it shows
|
| 175 |
-
# the player roster — a scenario's judge/host is configured under §04, mirroring
|
| 176 |
-
# the prototype's 4-player spy cast (ui/raw/data.js).
|
| 177 |
-
row_count=len(first.cast),
|
| 178 |
-
interactive=True,
|
| 179 |
-
label="Cast",
|
| 180 |
)
|
| 181 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 182 |
# 04 — The Judge + 05 — Tools (side by side)
|
| 183 |
with gr.Row():
|
| 184 |
with gr.Group():
|
|
@@ -189,9 +262,10 @@ def build_lab() -> dict[str, gr.components.Component]:
|
|
| 189 |
label="Policy preset",
|
| 190 |
)
|
| 191 |
handles["judge_model"] = gr.Dropdown(
|
| 192 |
-
choices=
|
| 193 |
-
value=
|
| 194 |
-
label="
|
|
|
|
| 195 |
)
|
| 196 |
handles["judge_strictness"] = gr.Slider(
|
| 197 |
minimum=0,
|
|
@@ -244,7 +318,7 @@ def collect_world_config(
|
|
| 244 |
scenario: str,
|
| 245 |
premise: str,
|
| 246 |
seed: str,
|
| 247 |
-
|
| 248 |
judge_policy: str,
|
| 249 |
judge_model: str,
|
| 250 |
judge_strictness: float,
|
|
@@ -255,25 +329,29 @@ def collect_world_config(
|
|
| 255 |
"""Assemble + validate a per-run world from the Lab's form values.
|
| 256 |
|
| 257 |
Returns the validated :class:`WorldConfig` (raising ``pydantic.ValidationError``
|
| 258 |
-
on an incoherent run). This is the bridge
|
| 259 |
-
composed run.
|
| 260 |
-
|
| 261 |
-
The base scenario (selected by its display *title*
|
| 262 |
-
|
| 263 |
-
|
| 264 |
-
|
| 265 |
-
``
|
| 266 |
-
|
| 267 |
-
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
|
| 271 |
-
|
| 272 |
-
(
|
| 273 |
-
|
| 274 |
-
|
| 275 |
-
|
| 276 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 277 |
"""
|
| 278 |
registry = default_registry()
|
| 279 |
base = _scenario_by_title(scenario)
|
|
@@ -283,28 +361,27 @@ def collect_world_config(
|
|
| 283 |
if base is None:
|
| 284 |
raise ValueError(f"unknown scenario {scenario!r} (have: {sorted(registry.scenarios)})")
|
| 285 |
|
| 286 |
-
#
|
| 287 |
-
#
|
| 288 |
-
|
| 289 |
-
|
| 290 |
-
|
| 291 |
-
continue
|
| 292 |
-
name = str(row[0]).strip()
|
| 293 |
-
patch: dict = {}
|
| 294 |
-
if len(row) > 1 and row[1] not in (None, ""):
|
| 295 |
-
patch["archetype"] = str(row[1])
|
| 296 |
-
if len(row) > 2 and str(row[2]).strip() in MODEL_PROFILES:
|
| 297 |
-
patch["model_profile"] = str(row[2]).strip()
|
| 298 |
-
edits[name] = patch
|
| 299 |
|
| 300 |
-
# Build the per-run cast: every manifest the scenario references, with
|
|
|
|
|
|
|
| 301 |
agents = []
|
| 302 |
cast_names: list[str] = []
|
| 303 |
for agent_name in base.cast:
|
| 304 |
manifest = registry.agents.get(agent_name)
|
| 305 |
if manifest is None:
|
| 306 |
raise ValueError(f"scenario {base.name!r} references undefined agent {agent_name!r}")
|
| 307 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 308 |
agents.append(manifest.model_dump(mode="python"))
|
| 309 |
cast_names.append(manifest.name)
|
| 310 |
|
|
|
|
| 23 |
import gradio as gr
|
| 24 |
|
| 25 |
from src.core.config import ScenarioConfig, validate_scenario, validate_world
|
| 26 |
+
from src.core.manifest import AgentManifest
|
| 27 |
from src.core.registry import default_registry
|
| 28 |
+
from src.models import modal_catalogue
|
| 29 |
from src.ui.fishbowl.adapter import VOICES, scenario_voice
|
| 30 |
|
| 31 |
# ── design vocabulary (mirrors ui/raw/lab.jsx) ──────────────────────────────────
|
|
|
|
| 39 |
"Judge's Whim",
|
| 40 |
]
|
| 41 |
|
|
|
|
|
|
|
|
|
|
| 42 |
# MCP tool grants the cast may reach for (friendly label, stored id). Mirrors
|
| 43 |
# ui/raw/lab.jsx:MCP_TOOLS; the value is what we store on the run.
|
| 44 |
TOOL_CHOICES: list[tuple[str, str]] = [
|
|
|
|
| 50 |
("tts.speak · give it a voice", "tts.speak"),
|
| 51 |
]
|
| 52 |
|
|
|
|
|
|
|
|
|
|
| 53 |
# The scenario we lead with — the hackathon's north-star world.
|
| 54 |
_PREFERRED_SCENARIO = "thousand-token-wood"
|
| 55 |
|
|
|
|
| 73 |
return None
|
| 74 |
|
| 75 |
|
| 76 |
+
def model_choices() -> list[tuple[str, str]]:
|
| 77 |
+
"""Dropdown choices for the Modal-hosted catalogue: ``(friendly label, endpoint key)``.
|
| 78 |
+
|
| 79 |
+
The single source of truth is ``modal/catalogue.py`` (read through the engine's
|
| 80 |
+
``modal_catalogue`` view), so the Lab can *only* offer models that are actually
|
| 81 |
+
deployable — and the list loads offline (the catalogue is a plain stdlib file), so
|
| 82 |
+
the picker is populated even with no API key. Empty list → a stripped deployment
|
| 83 |
+
with no catalogue, in which case the cast falls back to the deterministic stub."""
|
| 84 |
+
choices: list[tuple[str, str]] = []
|
| 85 |
+
for entry in modal_catalogue.entries():
|
| 86 |
+
served = entry["served_model_id"].split("/")[-1]
|
| 87 |
+
params = f"{entry['params_b']:g}B" if entry.get("params_b") else "?"
|
| 88 |
+
tier = entry["profile"] or "specialist"
|
| 89 |
+
provider = entry["provider"].title()
|
| 90 |
+
choices.append((f"{served} · {params} · {tier} · {provider}", entry["key"]))
|
| 91 |
+
return choices
|
| 92 |
+
|
| 93 |
+
|
| 94 |
+
def _default_model_key(manifest: AgentManifest) -> str | None:
|
| 95 |
+
"""Catalogue key a cast row defaults to: the manifest's explicit ``model_endpoint``,
|
| 96 |
+
else the catalogue's default model for its tier, else the first catalogue model (or
|
| 97 |
+
None when the catalogue is empty)."""
|
| 98 |
+
if manifest.model_endpoint:
|
| 99 |
+
return manifest.model_endpoint
|
| 100 |
+
tiered = modal_catalogue.default_key_for_profile(manifest.model_profile)
|
| 101 |
+
if tiered:
|
| 102 |
+
return tiered
|
| 103 |
+
entries = modal_catalogue.entries()
|
| 104 |
+
return entries[0]["key"] if entries else None
|
| 105 |
+
|
| 106 |
+
|
| 107 |
+
def _judge_manifest(scenario: ScenarioConfig) -> AgentManifest | None:
|
| 108 |
+
"""The scenario's judge agent (first ``role == "judge"`` in the cast), or None."""
|
| 109 |
+
registry = default_registry()
|
| 110 |
+
for agent_name in scenario.cast:
|
| 111 |
+
manifest = registry.agents.get(agent_name)
|
| 112 |
+
if manifest is not None and manifest.role == "judge":
|
| 113 |
+
return manifest
|
| 114 |
+
return None
|
| 115 |
+
|
| 116 |
|
| 117 |
+
def _cast_defaults(scenario: ScenarioConfig) -> dict[str, str]:
|
| 118 |
+
"""Default model selection for a scenario's *non-judge* cast (name → endpoint key).
|
| 119 |
+
|
| 120 |
+
The Judge is bound under §04, so it is excluded here. Used to seed (and re-seed on
|
| 121 |
+
scenario change) the ``cast_models`` state the picker writes into."""
|
| 122 |
registry = default_registry()
|
| 123 |
+
defaults: dict[str, str] = {}
|
| 124 |
for agent_name in scenario.cast:
|
| 125 |
manifest = registry.agents.get(agent_name)
|
| 126 |
+
if manifest is None or manifest.role == "judge":
|
|
|
|
|
|
|
|
|
|
| 127 |
continue
|
| 128 |
+
key = _default_model_key(manifest)
|
| 129 |
+
if key:
|
| 130 |
+
defaults[agent_name] = key
|
| 131 |
+
return defaults
|
| 132 |
|
| 133 |
|
| 134 |
def _voice_choices() -> list[tuple[str, str]]:
|
|
|
|
| 146 |
Wires no callbacks and imports no sibling render/show module — the app shell
|
| 147 |
(Unit 9) binds ``summon_btn`` to the session. Returns a dict of every handle a
|
| 148 |
caller needs to read the composed run (keys: ``scenario, premise, seed, world,
|
| 149 |
+
narrator, cast_models, judge_policy, judge_model, judge_strictness, tools, tokens,
|
| 150 |
+
max_rounds, seed_num, cadence, summon_btn, surprise_btn``). ``cast_models`` is a
|
| 151 |
+
``gr.State`` holding ``{agent_name: catalogue_endpoint_key}`` for the non-judge cast.
|
| 152 |
"""
|
| 153 |
scenarios = _ordered_scenarios()
|
| 154 |
first = scenarios[0]
|
|
|
|
| 194 |
lines=2,
|
| 195 |
)
|
| 196 |
|
| 197 |
+
# 03 — The Cast: one Modal-hosted model per mind. The picker offers ONLY models in
|
| 198 |
+
# the catalogue (modal/catalogue.py), and the choice drives the run (ADR-0022). A
|
| 199 |
+
# gr.render keeps one row per player as the scenario (and its cast size) changes; each
|
| 200 |
+
# row's dropdown writes the chosen endpoint key into the cast_models state below.
|
| 201 |
+
cast_models = gr.State(_cast_defaults(first))
|
| 202 |
+
handles["cast_models"] = cast_models
|
| 203 |
+
catalogue = model_choices()
|
| 204 |
with gr.Group():
|
| 205 |
gr.Markdown(
|
| 206 |
+
"**03 · The Cast** — bind each mind to a model **hosted on Modal** "
|
| 207 |
+
"(the only models you can pick); the Judge is set in §04"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 208 |
)
|
| 209 |
|
| 210 |
+
@gr.render(inputs=[handles["scenario"]])
|
| 211 |
+
def _render_cast(scenario_value, _choices=catalogue):
|
| 212 |
+
scenario = _scenario_by_title(scenario_value) or default_registry().scenarios.get(scenario_value)
|
| 213 |
+
if scenario is None:
|
| 214 |
+
gr.Markdown("_No scenario selected._")
|
| 215 |
+
return
|
| 216 |
+
registry = default_registry()
|
| 217 |
+
shown = 0
|
| 218 |
+
for agent_name in scenario.cast:
|
| 219 |
+
manifest = registry.agents.get(agent_name)
|
| 220 |
+
if manifest is None or manifest.role == "judge":
|
| 221 |
+
continue # the Judge is configured under §04
|
| 222 |
+
shown += 1
|
| 223 |
+
with gr.Row():
|
| 224 |
+
gr.Markdown(
|
| 225 |
+
f"**{manifest.name}**<br/>"
|
| 226 |
+
f"<span style='opacity:.7'>{manifest.archetype or f'the {manifest.role}'}</span>"
|
| 227 |
+
)
|
| 228 |
+
picker = gr.Dropdown(
|
| 229 |
+
choices=_choices,
|
| 230 |
+
value=_default_model_key(manifest),
|
| 231 |
+
label="model · Modal",
|
| 232 |
+
interactive=bool(_choices),
|
| 233 |
+
scale=2,
|
| 234 |
+
)
|
| 235 |
+
|
| 236 |
+
# Capture the agent name per row; the dropdown writes its key into the
|
| 237 |
+
# shared cast_models dict the Summon handler reads.
|
| 238 |
+
def _set_model(key, state, _name=manifest.name):
|
| 239 |
+
return {**(state or {}), _name: key}
|
| 240 |
+
|
| 241 |
+
picker.change(_set_model, inputs=[picker, cast_models], outputs=[cast_models])
|
| 242 |
+
if not shown:
|
| 243 |
+
gr.Markdown("_This scenario has no selectable players._")
|
| 244 |
+
elif not _choices:
|
| 245 |
+
gr.Markdown("_No Modal models in the catalogue — the cast runs the deterministic stub._")
|
| 246 |
+
|
| 247 |
+
# Switching scenarios re-seeds the model picks to the new cast's defaults so a stale
|
| 248 |
+
# override from the previous world never leaks into the run.
|
| 249 |
+
def _reset_cast_models(scenario_value):
|
| 250 |
+
scn = _scenario_by_title(scenario_value) or default_registry().scenarios.get(scenario_value)
|
| 251 |
+
return _cast_defaults(scn) if scn else {}
|
| 252 |
+
|
| 253 |
+
handles["scenario"].change(_reset_cast_models, inputs=[handles["scenario"]], outputs=[cast_models])
|
| 254 |
+
|
| 255 |
# 04 — The Judge + 05 — Tools (side by side)
|
| 256 |
with gr.Row():
|
| 257 |
with gr.Group():
|
|
|
|
| 262 |
label="Policy preset",
|
| 263 |
)
|
| 264 |
handles["judge_model"] = gr.Dropdown(
|
| 265 |
+
choices=catalogue,
|
| 266 |
+
value=_default_model_key(_judge_manifest(first)) if _judge_manifest(first) else None,
|
| 267 |
+
label="Judge model · Modal",
|
| 268 |
+
interactive=bool(catalogue),
|
| 269 |
)
|
| 270 |
handles["judge_strictness"] = gr.Slider(
|
| 271 |
minimum=0,
|
|
|
|
| 318 |
scenario: str,
|
| 319 |
premise: str,
|
| 320 |
seed: str,
|
| 321 |
+
cast_models: dict[str, str] | None,
|
| 322 |
judge_policy: str,
|
| 323 |
judge_model: str,
|
| 324 |
judge_strictness: float,
|
|
|
|
| 329 |
"""Assemble + validate a per-run world from the Lab's form values.
|
| 330 |
|
| 331 |
Returns the validated :class:`WorldConfig` (raising ``pydantic.ValidationError``
|
| 332 |
+
on an incoherent run). This is the bridge the app shell uses to build a Conductor
|
| 333 |
+
from a composed run via :meth:`Registry.from_world`.
|
| 334 |
+
|
| 335 |
+
The base scenario (selected by its display *title* or internal name) supplies the
|
| 336 |
+
cast roster and agent manifests. Model selection binds each mind to a *specific*
|
| 337 |
+
Modal-hosted model: ``cast_models`` maps ``{agent_name: catalogue_endpoint_key}`` for
|
| 338 |
+
the players and ``judge_model`` is the Judge's endpoint key (§04). Each becomes that
|
| 339 |
+
agent's ``model_endpoint`` (ADR-0022) — non-destructively via ``model_copy``, so the
|
| 340 |
+
shared registry is untouched. Only keys that exist in ``modal_catalogue`` are honoured
|
| 341 |
+
(the picker offers nothing else; we re-check so a stale key can't reach the run); an
|
| 342 |
+
agent with no/blank/unknown selection keeps its manifest tier. The premise overrides
|
| 343 |
+
the scenario goal, ``seed`` becomes its ``default_seed``, and the budget knobs feed the
|
| 344 |
+
governor.
|
| 345 |
+
|
| 346 |
+
The judge knobs (``judge_policy`` / ``judge_strictness``) and the ``tools`` grant are
|
| 347 |
+
accepted and shape-checked, but the deeper synthesis (policy preset → judge behaviour,
|
| 348 |
+
tool grants onto worker manifests) is deferred and TODO'd below — every dict this builds
|
| 349 |
+
still passes through ``validate_scenario`` / ``validate_world`` before return, so the
|
| 350 |
+
contract ("emit, validate, run") holds. See ADR-0011 / ADR-0022.
|
| 351 |
+
|
| 352 |
+
TODO: map ``judge_policy`` / ``judge_strictness`` to concrete judge behaviour and wire
|
| 353 |
+
the selected ``tools`` onto each worker's manifest ``tools`` grant once those contracts
|
| 354 |
+
land in the live path.
|
| 355 |
"""
|
| 356 |
registry = default_registry()
|
| 357 |
base = _scenario_by_title(scenario)
|
|
|
|
| 361 |
if base is None:
|
| 362 |
raise ValueError(f"unknown scenario {scenario!r} (have: {sorted(registry.scenarios)})")
|
| 363 |
|
| 364 |
+
# Only catalogue-hosted models may be cast: the picker offers nothing else, and we
|
| 365 |
+
# re-check here so an out-of-band or stale key can never reach the run.
|
| 366 |
+
valid_keys = {e["key"] for e in modal_catalogue.entries()}
|
| 367 |
+
selections = dict(cast_models or {})
|
| 368 |
+
judge_key = (judge_model or "").strip()
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 369 |
|
| 370 |
+
# Build the per-run cast: every manifest the scenario references, with its chosen
|
| 371 |
+
# Modal model pinned via model_endpoint. Non-destructive: model_copy, never mutate
|
| 372 |
+
# the cached registry instance.
|
| 373 |
agents = []
|
| 374 |
cast_names: list[str] = []
|
| 375 |
for agent_name in base.cast:
|
| 376 |
manifest = registry.agents.get(agent_name)
|
| 377 |
if manifest is None:
|
| 378 |
raise ValueError(f"scenario {base.name!r} references undefined agent {agent_name!r}")
|
| 379 |
+
# The Judge's model comes from §04; every other mind from the cast picker.
|
| 380 |
+
chosen = judge_key if manifest.role == "judge" else selections.get(agent_name)
|
| 381 |
+
patch: dict = {}
|
| 382 |
+
if chosen and chosen in valid_keys:
|
| 383 |
+
patch["model_endpoint"] = chosen
|
| 384 |
+
manifest = manifest.model_copy(update=patch)
|
| 385 |
agents.append(manifest.model_dump(mode="python"))
|
| 386 |
cast_names.append(manifest.name)
|
| 387 |
|
|
@@ -3,7 +3,8 @@
|
|
| 3 |
Cover both surfaces: ``build_lab`` returns the expected handles inside a Blocks, and
|
| 4 |
``collect_world_config`` assembles a real scenario's data into a validated WorldConfig
|
| 5 |
(round-tripping through ``validate_world`` / ``validate_scenario``) without mutating the
|
| 6 |
-
shared registry.
|
|
|
|
| 7 |
"""
|
| 8 |
|
| 9 |
from __future__ import annotations
|
|
@@ -13,6 +14,7 @@ import pytest
|
|
| 13 |
|
| 14 |
from src.core.config import WorldConfig
|
| 15 |
from src.core.registry import default_registry
|
|
|
|
| 16 |
from src.ui.fishbowl import lab
|
| 17 |
|
| 18 |
EXPECTED_HANDLE_KEYS = {
|
|
@@ -21,7 +23,7 @@ EXPECTED_HANDLE_KEYS = {
|
|
| 21 |
"seed",
|
| 22 |
"world",
|
| 23 |
"narrator",
|
| 24 |
-
"
|
| 25 |
"judge_policy",
|
| 26 |
"judge_model",
|
| 27 |
"judge_strictness",
|
|
@@ -34,6 +36,9 @@ EXPECTED_HANDLE_KEYS = {
|
|
| 34 |
"surprise_btn",
|
| 35 |
}
|
| 36 |
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
def test_build_lab_returns_expected_handles():
|
| 39 |
with gr.Blocks():
|
|
@@ -45,44 +50,55 @@ def test_build_lab_returns_expected_handles():
|
|
| 45 |
assert isinstance(handles["seed"], gr.Dropdown)
|
| 46 |
assert handles["seed"].allow_custom_value is True
|
| 47 |
assert isinstance(handles["narrator"], gr.Dropdown)
|
| 48 |
-
|
| 49 |
-
assert handles["
|
| 50 |
assert isinstance(handles["tools"], gr.CheckboxGroup)
|
| 51 |
assert isinstance(handles["judge_strictness"], gr.Slider)
|
| 52 |
assert isinstance(handles["summon_btn"], gr.Button)
|
| 53 |
assert isinstance(handles["surprise_btn"], gr.Button)
|
| 54 |
|
| 55 |
|
| 56 |
-
def
|
| 57 |
-
registry = default_registry()
|
| 58 |
-
real_titles = {s.title or s.name for s in registry.scenarios.values()}
|
| 59 |
with gr.Blocks():
|
| 60 |
handles = lab.build_lab()
|
| 61 |
-
|
| 62 |
-
|
|
|
|
|
|
|
| 63 |
|
| 64 |
|
| 65 |
-
def
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
| 71 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
|
| 73 |
|
| 74 |
-
def
|
| 75 |
registry = default_registry()
|
| 76 |
scenario = registry.scenarios["thousand-token-wood"]
|
| 77 |
-
|
|
|
|
|
|
|
| 78 |
|
| 79 |
world = lab.collect_world_config(
|
| 80 |
scenario=scenario.title,
|
| 81 |
premise="A new whimsical premise for the wood.",
|
| 82 |
seed=scenario.default_seed,
|
| 83 |
-
|
| 84 |
judge_policy="Majority Vote",
|
| 85 |
-
judge_model=
|
| 86 |
judge_strictness=60,
|
| 87 |
tools=["dice.roll", "vote.tally"],
|
| 88 |
tokens=120_000,
|
|
@@ -90,46 +106,42 @@ def test_collect_world_config_validates_real_scenario():
|
|
| 90 |
)
|
| 91 |
|
| 92 |
assert isinstance(world, WorldConfig)
|
| 93 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 94 |
out = world.scenarios[0]
|
| 95 |
assert out.name == scenario.name
|
| 96 |
assert out.goal == "A new whimsical premise for the wood."
|
| 97 |
assert out.cast == list(scenario.cast)
|
| 98 |
-
# cross-reference check: every cast name resolves to a defined agent
|
| 99 |
-
assert {a.name for a in world.agents} >= set(out.cast)
|
| 100 |
assert out.governor is not None
|
| 101 |
assert out.governor.max_turns == 25
|
| 102 |
assert out.governor.max_total_tokens == 120_000
|
| 103 |
|
| 104 |
|
| 105 |
-
def
|
| 106 |
registry = default_registry()
|
| 107 |
scenario = registry.scenarios["thousand-token-wood"]
|
| 108 |
-
|
| 109 |
-
original_profile = registry.agents[first_agent].model_profile
|
| 110 |
-
|
| 111 |
-
rows = lab._cast_rows_for(scenario)
|
| 112 |
-
# flip the first agent's profile in the edited rows
|
| 113 |
-
new_profile = "strong" if original_profile != "strong" else "tiny"
|
| 114 |
-
rows[0][2] = new_profile
|
| 115 |
|
| 116 |
world = lab.collect_world_config(
|
| 117 |
scenario=scenario.name, # also accept name, not just title
|
| 118 |
premise="",
|
| 119 |
seed="",
|
| 120 |
-
|
| 121 |
judge_policy="Judge's Whim",
|
| 122 |
-
judge_model="
|
| 123 |
judge_strictness=10,
|
| 124 |
tools=[],
|
| 125 |
tokens=None,
|
| 126 |
max_rounds=None,
|
| 127 |
)
|
| 128 |
|
| 129 |
-
|
| 130 |
-
assert
|
| 131 |
-
# registry untouched
|
| 132 |
-
assert registry.agents[first_agent].model_profile == original_profile
|
| 133 |
# blank premise/seed fall back to the scenario's own
|
| 134 |
assert world.scenarios[0].goal == scenario.goal
|
| 135 |
assert world.scenarios[0].default_seed == scenario.default_seed
|
|
@@ -141,9 +153,9 @@ def test_collect_world_config_unknown_scenario_raises():
|
|
| 141 |
scenario="not-a-real-world",
|
| 142 |
premise="",
|
| 143 |
seed="",
|
| 144 |
-
|
| 145 |
judge_policy="Majority Vote",
|
| 146 |
-
judge_model="
|
| 147 |
judge_strictness=50,
|
| 148 |
tools=[],
|
| 149 |
tokens=None,
|
|
|
|
| 3 |
Cover both surfaces: ``build_lab`` returns the expected handles inside a Blocks, and
|
| 4 |
``collect_world_config`` assembles a real scenario's data into a validated WorldConfig
|
| 5 |
(round-tripping through ``validate_world`` / ``validate_scenario``) without mutating the
|
| 6 |
+
shared registry. Model selection is constrained to the Modal catalogue and pins each
|
| 7 |
+
agent's ``model_endpoint`` (ADR-0022).
|
| 8 |
"""
|
| 9 |
|
| 10 |
from __future__ import annotations
|
|
|
|
| 14 |
|
| 15 |
from src.core.config import WorldConfig
|
| 16 |
from src.core.registry import default_registry
|
| 17 |
+
from src.models import modal_catalogue
|
| 18 |
from src.ui.fishbowl import lab
|
| 19 |
|
| 20 |
EXPECTED_HANDLE_KEYS = {
|
|
|
|
| 23 |
"seed",
|
| 24 |
"world",
|
| 25 |
"narrator",
|
| 26 |
+
"cast_models",
|
| 27 |
"judge_policy",
|
| 28 |
"judge_model",
|
| 29 |
"judge_strictness",
|
|
|
|
| 36 |
"surprise_btn",
|
| 37 |
}
|
| 38 |
|
| 39 |
+
# A couple of real catalogue keys to cast (the catalogue loads offline — a plain file).
|
| 40 |
+
_CATALOGUE_KEYS = [e["key"] for e in modal_catalogue.entries()]
|
| 41 |
+
|
| 42 |
|
| 43 |
def test_build_lab_returns_expected_handles():
|
| 44 |
with gr.Blocks():
|
|
|
|
| 50 |
assert isinstance(handles["seed"], gr.Dropdown)
|
| 51 |
assert handles["seed"].allow_custom_value is True
|
| 52 |
assert isinstance(handles["narrator"], gr.Dropdown)
|
| 53 |
+
# The cast picker is a gr.render writing into this state (one dropdown per player).
|
| 54 |
+
assert isinstance(handles["cast_models"], gr.State)
|
| 55 |
assert isinstance(handles["tools"], gr.CheckboxGroup)
|
| 56 |
assert isinstance(handles["judge_strictness"], gr.Slider)
|
| 57 |
assert isinstance(handles["summon_btn"], gr.Button)
|
| 58 |
assert isinstance(handles["surprise_btn"], gr.Button)
|
| 59 |
|
| 60 |
|
| 61 |
+
def test_judge_model_dropdown_offers_only_catalogue_models():
|
|
|
|
|
|
|
| 62 |
with gr.Blocks():
|
| 63 |
handles = lab.build_lab()
|
| 64 |
+
assert isinstance(handles["judge_model"], gr.Dropdown)
|
| 65 |
+
values = {c[1] for c in handles["judge_model"].choices}
|
| 66 |
+
assert values <= set(_CATALOGUE_KEYS)
|
| 67 |
+
assert values, "judge model dropdown should list the catalogue"
|
| 68 |
|
| 69 |
|
| 70 |
+
def test_model_choices_are_all_catalogue_keys():
|
| 71 |
+
choices = lab.model_choices()
|
| 72 |
+
# Every selectable value is a real catalogue endpoint key — nothing else is offerable.
|
| 73 |
+
assert {key for _label, key in choices} == set(_CATALOGUE_KEYS)
|
| 74 |
+
# Labels are human-readable and name the served model.
|
| 75 |
+
assert all(" · " in label for label, _ in choices)
|
| 76 |
+
|
| 77 |
+
|
| 78 |
+
def test_cast_defaults_cover_non_judge_cast_with_catalogue_keys():
|
| 79 |
+
registry = default_registry()
|
| 80 |
+
scenario = registry.scenarios["thousand-token-wood"]
|
| 81 |
+
defaults = lab._cast_defaults(scenario)
|
| 82 |
+
judge_names = {n for n in scenario.cast if (registry.agents.get(n) and registry.agents[n].role == "judge")}
|
| 83 |
+
non_judge = [n for n in scenario.cast if n not in judge_names]
|
| 84 |
+
assert set(defaults) == set(non_judge) # judge excluded (set under §04)
|
| 85 |
+
assert all(v in _CATALOGUE_KEYS for v in defaults.values())
|
| 86 |
|
| 87 |
|
| 88 |
+
def test_collect_world_config_pins_selected_models_as_endpoints():
|
| 89 |
registry = default_registry()
|
| 90 |
scenario = registry.scenarios["thousand-token-wood"]
|
| 91 |
+
worker = next(n for n in scenario.cast if registry.agents[n].role != "judge")
|
| 92 |
+
judge = next(n for n in scenario.cast if registry.agents[n].role == "judge")
|
| 93 |
+
worker_key, judge_key = _CATALOGUE_KEYS[0], _CATALOGUE_KEYS[-1]
|
| 94 |
|
| 95 |
world = lab.collect_world_config(
|
| 96 |
scenario=scenario.title,
|
| 97 |
premise="A new whimsical premise for the wood.",
|
| 98 |
seed=scenario.default_seed,
|
| 99 |
+
cast_models={worker: worker_key},
|
| 100 |
judge_policy="Majority Vote",
|
| 101 |
+
judge_model=judge_key,
|
| 102 |
judge_strictness=60,
|
| 103 |
tools=["dice.roll", "vote.tally"],
|
| 104 |
tokens=120_000,
|
|
|
|
| 106 |
)
|
| 107 |
|
| 108 |
assert isinstance(world, WorldConfig)
|
| 109 |
+
by_name = {a.name: a for a in world.agents}
|
| 110 |
+
assert by_name[worker].model_endpoint == worker_key
|
| 111 |
+
assert by_name[judge].model_endpoint == judge_key # §04 binds the judge
|
| 112 |
+
# registry untouched (non-destructive model_copy)
|
| 113 |
+
assert registry.agents[worker].model_endpoint is None
|
| 114 |
+
assert registry.agents[judge].model_endpoint is None
|
| 115 |
+
|
| 116 |
out = world.scenarios[0]
|
| 117 |
assert out.name == scenario.name
|
| 118 |
assert out.goal == "A new whimsical premise for the wood."
|
| 119 |
assert out.cast == list(scenario.cast)
|
|
|
|
|
|
|
| 120 |
assert out.governor is not None
|
| 121 |
assert out.governor.max_turns == 25
|
| 122 |
assert out.governor.max_total_tokens == 120_000
|
| 123 |
|
| 124 |
|
| 125 |
+
def test_collect_world_config_ignores_unknown_or_blank_model_keys():
|
| 126 |
registry = default_registry()
|
| 127 |
scenario = registry.scenarios["thousand-token-wood"]
|
| 128 |
+
worker = next(n for n in scenario.cast if registry.agents[n].role != "judge")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
|
| 130 |
world = lab.collect_world_config(
|
| 131 |
scenario=scenario.name, # also accept name, not just title
|
| 132 |
premise="",
|
| 133 |
seed="",
|
| 134 |
+
cast_models={worker: "not-a-real-endpoint"}, # bogus key → ignored
|
| 135 |
judge_policy="Judge's Whim",
|
| 136 |
+
judge_model="", # blank → judge keeps its tier
|
| 137 |
judge_strictness=10,
|
| 138 |
tools=[],
|
| 139 |
tokens=None,
|
| 140 |
max_rounds=None,
|
| 141 |
)
|
| 142 |
|
| 143 |
+
by_name = {a.name: a for a in world.agents}
|
| 144 |
+
assert by_name[worker].model_endpoint is None # stale/unknown key dropped
|
|
|
|
|
|
|
| 145 |
# blank premise/seed fall back to the scenario's own
|
| 146 |
assert world.scenarios[0].goal == scenario.goal
|
| 147 |
assert world.scenarios[0].default_seed == scenario.default_seed
|
|
|
|
| 153 |
scenario="not-a-real-world",
|
| 154 |
premise="",
|
| 155 |
seed="",
|
| 156 |
+
cast_models={},
|
| 157 |
judge_policy="Majority Vote",
|
| 158 |
+
judge_model="",
|
| 159 |
judge_strictness=50,
|
| 160 |
tools=[],
|
| 161 |
tokens=None,
|
|
@@ -71,6 +71,25 @@ class TestManifestAgentEmits:
|
|
| 71 |
agent.act("r", 1, StageProjection(), ())
|
| 72 |
assert router.for_profile("tiny").variant == "stub:tiny"
|
| 73 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
|
| 75 |
class TestManifestAgentSalience:
|
| 76 |
def test_salience_path_runs(self):
|
|
|
|
| 71 |
agent.act("r", 1, StageProjection(), ())
|
| 72 |
assert router.for_profile("tiny").variant == "stub:tiny"
|
| 73 |
|
| 74 |
+
def test_model_endpoint_overrides_profile(self):
|
| 75 |
+
# An explicit catalogue endpoint pins a specific model: _route_key prefers it
|
| 76 |
+
# over the tier, and the agent routes there (ADR-0022).
|
| 77 |
+
class _Pinned(ManifestAgent):
|
| 78 |
+
manifest = AgentManifest(
|
| 79 |
+
name="scene-whisperer",
|
| 80 |
+
role="worker",
|
| 81 |
+
persona="You speak on a specific model.",
|
| 82 |
+
may_emit=["world.observed"],
|
| 83 |
+
model_profile="tiny",
|
| 84 |
+
model_endpoint="minicpm-4-1-8b",
|
| 85 |
+
)
|
| 86 |
+
|
| 87 |
+
agent = _Pinned(router := _router())
|
| 88 |
+
assert agent._route_key == "minicpm-4-1-8b"
|
| 89 |
+
agent.act("r", 1, StageProjection(), ())
|
| 90 |
+
# the routed (cached) provider is the endpoint's stub, not the tiny tier's
|
| 91 |
+
assert router.for_profile("minicpm-4-1-8b").variant == "stub:minicpm-4-1-8b"
|
| 92 |
+
|
| 93 |
|
| 94 |
class TestManifestAgentSalience:
|
| 95 |
def test_salience_path_runs(self):
|
|
@@ -56,6 +56,43 @@ class TestDefaultRegistry:
|
|
| 56 |
reg.build_scenario("no-such-scenario")
|
| 57 |
|
| 58 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
class TestHandlerBinding:
|
| 60 |
def test_handler_class_used_when_named(self):
|
| 61 |
@register_handler("test-handler")
|
|
|
|
| 56 |
reg.build_scenario("no-such-scenario")
|
| 57 |
|
| 58 |
|
| 59 |
+
class TestFromWorld:
|
| 60 |
+
"""A composed WorldConfig (e.g. from the Lab) builds a registry on the same path."""
|
| 61 |
+
|
| 62 |
+
def _world(self):
|
| 63 |
+
from src.core.config import validate_world
|
| 64 |
+
|
| 65 |
+
reg = default_registry()
|
| 66 |
+
base = reg.scenarios["thousand-token-wood"]
|
| 67 |
+
# Pin one agent to a specific catalogue model, the rest by tier.
|
| 68 |
+
agents = []
|
| 69 |
+
for name in base.cast:
|
| 70 |
+
m = reg.agents[name]
|
| 71 |
+
if name == "pocket-actor":
|
| 72 |
+
m = m.model_copy(update={"model_endpoint": "minicpm-4-1-8b"})
|
| 73 |
+
agents.append(m.model_dump(mode="python"))
|
| 74 |
+
return validate_world(
|
| 75 |
+
{
|
| 76 |
+
"agents": agents,
|
| 77 |
+
"scenarios": [base.model_dump(mode="python")],
|
| 78 |
+
}
|
| 79 |
+
)
|
| 80 |
+
|
| 81 |
+
def test_builds_scenario_and_router_from_world(self):
|
| 82 |
+
reg = Registry.from_world(self._world())
|
| 83 |
+
assert set(reg.agents) >= {"scene-whisperer", "pocket-actor", "echo"}
|
| 84 |
+
sc = reg.build_scenario("thousand-token-wood")
|
| 85 |
+
assert len(sc.agents) == 4
|
| 86 |
+
pocket = next(a for a in sc.agents if a.name == "pocket-actor")
|
| 87 |
+
assert pocket.manifest.model_endpoint == "minicpm-4-1-8b"
|
| 88 |
+
assert pocket._route_key == "minicpm-4-1-8b"
|
| 89 |
+
|
| 90 |
+
def test_governor_threaded_from_world(self):
|
| 91 |
+
reg = Registry.from_world(self._world())
|
| 92 |
+
gov = reg.governor_for("thousand-token-wood")
|
| 93 |
+
assert gov.max_turns == 60 # carried from the scenario's governor
|
| 94 |
+
|
| 95 |
+
|
| 96 |
class TestHandlerBinding:
|
| 97 |
def test_handler_class_used_when_named(self):
|
| 98 |
@register_handler("test-handler")
|
|
@@ -71,6 +71,45 @@ class TestModelRouterOnline:
|
|
| 71 |
assert provider.max_tokens == 320 # _PROFILE_DECODING["balanced"]
|
| 72 |
|
| 73 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
class TestFromEnv:
|
| 75 |
def test_offline_without_binding(self, monkeypatch):
|
| 76 |
# No Modal binding (and no stray cloud key) → deterministic offline stub.
|
|
|
|
| 71 |
assert provider.max_tokens == 320 # _PROFILE_DECODING["balanced"]
|
| 72 |
|
| 73 |
|
| 74 |
+
class TestModelRouterCatalogueEndpoint:
|
| 75 |
+
"""A non-tier router key names a specific catalogue model (manifest.model_endpoint)."""
|
| 76 |
+
|
| 77 |
+
def test_offline_endpoint_key_serves_distinct_stub(self):
|
| 78 |
+
# A concrete catalogue key routes like any profile offline: the deterministic
|
| 79 |
+
# stub, with the key folded into its variant so the choice still varies output.
|
| 80 |
+
router = ModelRouter(offline=True)
|
| 81 |
+
provider = router.for_profile("minicpm-4-1-8b")
|
| 82 |
+
assert isinstance(provider, DeterministicTinyModel)
|
| 83 |
+
assert "minicpm-4-1-8b" in provider.variant
|
| 84 |
+
assert provider.variant != router.for_profile("gemma-4-12b").variant
|
| 85 |
+
|
| 86 |
+
def test_online_endpoint_key_resolves_to_catalogue_binding(self, monkeypatch):
|
| 87 |
+
monkeypatch.setenv("MODAL_WORKSPACE", "demo-ws")
|
| 88 |
+
monkeypatch.setenv("MODAL_LLM_KEY", "EMPTY")
|
| 89 |
+
monkeypatch.delenv("MODEL_BALANCED", raising=False)
|
| 90 |
+
router = ModelRouter(offline=False)
|
| 91 |
+
provider = router.for_profile("gemma-4-12b")
|
| 92 |
+
assert isinstance(provider, LiteLLMProvider)
|
| 93 |
+
assert provider.model == "openai/google/gemma-4-12B"
|
| 94 |
+
assert "gemma-4-12b" in provider.api_base
|
| 95 |
+
assert provider.max_tokens == 320 # balanced tier decoding
|
| 96 |
+
|
| 97 |
+
def test_unbound_specialist_uses_balanced_decoding(self, monkeypatch):
|
| 98 |
+
# nemotron-cascade-14b-thinking has profile=None → balanced decoding defaults.
|
| 99 |
+
monkeypatch.setenv("MODAL_WORKSPACE", "demo-ws")
|
| 100 |
+
router = ModelRouter(offline=False)
|
| 101 |
+
provider = router.for_profile("nemotron-cascade-14b-thinking")
|
| 102 |
+
assert isinstance(provider, LiteLLMProvider)
|
| 103 |
+
assert provider.max_tokens == 320
|
| 104 |
+
|
| 105 |
+
def test_unknown_key_degrades_to_fast_tier(self, monkeypatch):
|
| 106 |
+
monkeypatch.setenv("MODEL_FAST", "fallback-model")
|
| 107 |
+
router = ModelRouter(offline=False)
|
| 108 |
+
provider = router.for_profile("not-a-real-endpoint")
|
| 109 |
+
assert provider.model == "fallback-model"
|
| 110 |
+
assert provider.max_tokens == 220 # fast tier decoding
|
| 111 |
+
|
| 112 |
+
|
| 113 |
class TestFromEnv:
|
| 114 |
def test_offline_without_binding(self, monkeypatch):
|
| 115 |
# No Modal binding (and no stray cloud key) → deterministic offline stub.
|