agharsallah commited on
Commit ·
a71301e
1
Parent(s): 6091937
feat: implement per-event model attribution for enhanced event tracking
Browse files- docs/adr/0028-session-stamped-events-and-scoped-recall.md +30 -4
- src/agents/base.py +17 -0
- src/core/events.py +9 -0
- src/core/memory_index.py +4 -0
- src/core/sqlalchemy_ledger.py +10 -0
- src/core/sqlite_ledger.py +19 -7
- src/models/provider.py +10 -0
- src/ui/fishbowl/adapter.py +14 -0
- src/ui/fishbowl/assets/styles.css +21 -9
- src/ui/fishbowl/render/mindcard.py +16 -4
- src/ui/fishbowl/view_model.py +11 -1
- tests/test_model_attribution.py +90 -0
docs/adr/0028-session-stamped-events-and-scoped-recall.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# ADR-0028: Session-Stamped Events and Run-Scoped
|
| 2 |
|
| 3 |
## Status
|
| 4 |
|
|
@@ -24,6 +24,12 @@ and scoped the **UI** to one run. Three gaps remained on the data/context side:
|
|
| 24 |
Additionally, the session id originates in `localStorage` — untrusted client input —
|
| 25 |
and reached the ledger unvalidated.
|
| 26 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
## Decision
|
| 28 |
|
| 29 |
- **`Event.session_id: str | None` on the envelope.** Stamped by the Conductor at
|
|
@@ -46,12 +52,32 @@ and reached the ledger unvalidated.
|
|
| 46 |
search, free for callers). `session_id` rides in index metadata for forensics.
|
| 47 |
- **RunIndex prefers the envelope.** `RunSummary.session_id` folds from
|
| 48 |
`event.session_id` first, payload copy second.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
|
| 50 |
## Consequences
|
| 51 |
|
| 52 |
-
- Every action is directly filterable by session
|
| 53 |
-
|
| 54 |
-
envelope
|
|
|
|
|
|
|
|
|
|
| 55 |
- Prompts are hermetic per run: neither the episodic window, nor salience ranking,
|
| 56 |
nor semantic recall can surface another run's text. Verified by probe-agent and
|
| 57 |
scoped-search tests.
|
|
|
|
| 1 |
+
# ADR-0028: Session-Stamped, Model-Attributed Events and Run-Scoped Recall
|
| 2 |
|
| 3 |
## Status
|
| 4 |
|
|
|
|
| 24 |
Additionally, the session id originates in `localStorage` — untrusted client input —
|
| 25 |
and reached the ledger unvalidated.
|
| 26 |
|
| 27 |
+
4. **No per-event model attribution.** `run.started.cast` recorded the *intended*
|
| 28 |
+
binding per agent, but no event recorded the model that *actually* produced it —
|
| 29 |
+
so "which model said this line" (and "which model won") was an inference from the
|
| 30 |
+
cast map, not a fact, and env overrides / endpoint fallbacks were invisible after
|
| 31 |
+
the fact.
|
| 32 |
+
|
| 33 |
## Decision
|
| 34 |
|
| 35 |
- **`Event.session_id: str | None` on the envelope.** Stamped by the Conductor at
|
|
|
|
| 52 |
search, free for callers). `session_id` rides in index metadata for forensics.
|
| 53 |
- **RunIndex prefers the envelope.** `RunSummary.session_id` folds from
|
| 54 |
`event.session_id` first, payload copy second.
|
| 55 |
+
- **Per-event model attribution.** `Event.model_profile` (the route key the agent
|
| 56 |
+
asked for — tier or endpoint key) and `Event.model_id` (the concrete model that
|
| 57 |
+
ran). The agent captures both when it resolves its provider
|
| 58 |
+
(`ModelProvider.model_id` unifies the live gateway's `model` and the stub's
|
| 59 |
+
`variant`) and stamps them on the event it emits; handler subclasses inherit it
|
| 60 |
+
via `super().act()`. Scenario/genesis/lifecycle events stay `None`. mem0 metadata
|
| 61 |
+
carries them; the Show's cast card prefers the *actual* model once a line is
|
| 62 |
+
spoken (`short_model_name`), falling back to the intended binding.
|
| 63 |
+
- **DB structure: columns + indexes, no new tables.** The queryable envelope facts
|
| 64 |
+
(`session_id`, `model_profile`, `model_id`) are typed columns, not buried JSON —
|
| 65 |
+
the right denormalization for an *immutable* log. We add a composite
|
| 66 |
+
`(run_id, offset)` index (the hottest read — `events_for_run` ordered by offset)
|
| 67 |
+
plus single indexes on `session_id` and `model_id`. We deliberately **do not**
|
| 68 |
+
add normalized/lookup tables or foreign keys: an append-only log never mutates,
|
| 69 |
+
so FKs buy no integrity and only cost joins. A materialized `runs` read-model
|
| 70 |
+
remains a deferred option (rebuildable cache) for *when* folding events is
|
| 71 |
+
measurably slow or real accounts arrive — not now.
|
| 72 |
|
| 73 |
## Consequences
|
| 74 |
|
| 75 |
+
- Every action is directly filterable by session **and by model** in SQL
|
| 76 |
+
(`WHERE session_id = ?`, `WHERE model_id = ?`, both indexed), in mem0 metadata,
|
| 77 |
+
and in exported traces (the JSONL dump inherits the envelope fields for free).
|
| 78 |
+
- "Which model won / spoke this line" is now a recorded fact, not an inference —
|
| 79 |
+
useful for sponsor-track receipts (which model played which part) and the demo's
|
| 80 |
+
per-card model badge.
|
| 81 |
- Prompts are hermetic per run: neither the episodic window, nor salience ranking,
|
| 82 |
nor semantic recall can surface another run's text. Verified by probe-agent and
|
| 83 |
scoped-search tests.
|
src/agents/base.py
CHANGED
|
@@ -115,6 +115,11 @@ class ManifestAgent(Agent):
|
|
| 115 |
self.memory_index = memory_index
|
| 116 |
self._reflection_tracker: ReflectionTracker | None = None
|
| 117 |
self.last_usage: dict[str, int] = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
|
| 119 |
@property
|
| 120 |
def name(self) -> str: # type: ignore[override]
|
|
@@ -175,6 +180,8 @@ class ManifestAgent(Agent):
|
|
| 175 |
kind=parsed["kind"],
|
| 176 |
actor=self.manifest.name,
|
| 177 |
payload={k: v for k, v in parsed.items() if k != "kind"},
|
|
|
|
|
|
|
| 178 |
)
|
| 179 |
|
| 180 |
@staticmethod
|
|
@@ -237,6 +244,7 @@ class ManifestAgent(Agent):
|
|
| 237 |
"""
|
| 238 |
wants_thought = bool(extra_fields and "thought" in extra_fields)
|
| 239 |
provider = self.router.for_profile(self._route_key)
|
|
|
|
| 240 |
with obs.span("agent.resolve", **{"mal.agent": role, "mal.profile": self._route_key}):
|
| 241 |
if hasattr(provider, "complete_structured"):
|
| 242 |
model = build_output_model(allowed, extra_fields)
|
|
@@ -308,11 +316,18 @@ class ManifestAgent(Agent):
|
|
| 308 |
def _complete(self, role: str, prompt: str) -> str:
|
| 309 |
"""Route to the provider for this agent's profile and record token usage."""
|
| 310 |
provider = self.router.for_profile(self._route_key)
|
|
|
|
| 311 |
raw = provider.complete(role, prompt)
|
| 312 |
self.last_usage = dict(provider.last_usage)
|
| 313 |
self._guard_model_error(role, raw)
|
| 314 |
return raw
|
| 315 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 316 |
# ── memory ──────────────────────────────────────────────────────────────
|
| 317 |
|
| 318 |
def _recall(self, turn: int, projection: StageProjection, recent_events: tuple[Event, ...]) -> str:
|
|
@@ -349,6 +364,8 @@ class ManifestAgent(Agent):
|
|
| 349 |
kind=_REFLECTION_KIND,
|
| 350 |
actor=self.manifest.name,
|
| 351 |
payload={k: v for k, v in parsed.items() if k != "kind"},
|
|
|
|
|
|
|
| 352 |
)
|
| 353 |
|
| 354 |
# ── output authority ──────────────────────────────────────────────────────
|
|
|
|
| 115 |
self.memory_index = memory_index
|
| 116 |
self._reflection_tracker: ReflectionTracker | None = None
|
| 117 |
self.last_usage: dict[str, int] = {"prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0}
|
| 118 |
+
# The model behind the most recent generation, captured when the provider is
|
| 119 |
+
# resolved and stamped onto the event in act()/reflection — so each line in the
|
| 120 |
+
# ledger records the model that actually produced it, not just the intended one.
|
| 121 |
+
self.last_model_profile: str | None = None
|
| 122 |
+
self.last_model_id: str | None = None
|
| 123 |
|
| 124 |
@property
|
| 125 |
def name(self) -> str: # type: ignore[override]
|
|
|
|
| 180 |
kind=parsed["kind"],
|
| 181 |
actor=self.manifest.name,
|
| 182 |
payload={k: v for k, v in parsed.items() if k != "kind"},
|
| 183 |
+
model_profile=self.last_model_profile,
|
| 184 |
+
model_id=self.last_model_id,
|
| 185 |
)
|
| 186 |
|
| 187 |
@staticmethod
|
|
|
|
| 244 |
"""
|
| 245 |
wants_thought = bool(extra_fields and "thought" in extra_fields)
|
| 246 |
provider = self.router.for_profile(self._route_key)
|
| 247 |
+
self._record_model(provider)
|
| 248 |
with obs.span("agent.resolve", **{"mal.agent": role, "mal.profile": self._route_key}):
|
| 249 |
if hasattr(provider, "complete_structured"):
|
| 250 |
model = build_output_model(allowed, extra_fields)
|
|
|
|
| 316 |
def _complete(self, role: str, prompt: str) -> str:
|
| 317 |
"""Route to the provider for this agent's profile and record token usage."""
|
| 318 |
provider = self.router.for_profile(self._route_key)
|
| 319 |
+
self._record_model(provider)
|
| 320 |
raw = provider.complete(role, prompt)
|
| 321 |
self.last_usage = dict(provider.last_usage)
|
| 322 |
self._guard_model_error(role, raw)
|
| 323 |
return raw
|
| 324 |
|
| 325 |
+
def _record_model(self, provider) -> None:
|
| 326 |
+
"""Remember the model behind the current generation so the emitted event can
|
| 327 |
+
record it (the route key asked for + the concrete model that actually ran)."""
|
| 328 |
+
self.last_model_profile = self._route_key
|
| 329 |
+
self.last_model_id = getattr(provider, "model_id", "") or None
|
| 330 |
+
|
| 331 |
# ── memory ──────────────────────────────────────────────────────────────
|
| 332 |
|
| 333 |
def _recall(self, turn: int, projection: StageProjection, recent_events: tuple[Event, ...]) -> str:
|
|
|
|
| 364 |
kind=_REFLECTION_KIND,
|
| 365 |
actor=self.manifest.name,
|
| 366 |
payload={k: v for k, v in parsed.items() if k != "kind"},
|
| 367 |
+
model_profile=self.last_model_profile,
|
| 368 |
+
model_id=self.last_model_id,
|
| 369 |
)
|
| 370 |
|
| 371 |
# ── output authority ──────────────────────────────────────────────────────
|
src/core/events.py
CHANGED
|
@@ -104,6 +104,15 @@ class Event(BaseModel):
|
|
| 104 |
# headless/legacy events. An OPTIONAL envelope field — additive, so
|
| 105 |
# schema_version stays 1 and old rows load with session_id=None.
|
| 106 |
session_id: str | None = None
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
|
| 108 |
@field_validator("kind")
|
| 109 |
@classmethod
|
|
|
|
| 104 |
# headless/legacy events. An OPTIONAL envelope field — additive, so
|
| 105 |
# schema_version stays 1 and old rows load with session_id=None.
|
| 106 |
session_id: str | None = None
|
| 107 |
+
# Which model produced this event, set by the agent at generation time:
|
| 108 |
+
# model_profile — the route key the agent asked for (tiny/fast/balanced/
|
| 109 |
+
# strong, or an explicit catalogue endpoint key, ADR-0022)
|
| 110 |
+
# model_id — the concrete model that actually ran (e.g.
|
| 111 |
+
# "openai/openbmb/MiniCPM4.1-8B", or "stub:fast" offline)
|
| 112 |
+
# Both None for events with no model behind them (genesis, user.injected,
|
| 113 |
+
# run.started/finished). Additive envelope fields — schema_version stays 1.
|
| 114 |
+
model_profile: str | None = None
|
| 115 |
+
model_id: str | None = None
|
| 116 |
|
| 117 |
@field_validator("kind")
|
| 118 |
@classmethod
|
src/core/memory_index.py
CHANGED
|
@@ -351,6 +351,8 @@ def _event_metadata(event: Event) -> dict:
|
|
| 351 |
"created_at": event.created_at.isoformat(),
|
| 352 |
"schema_version": event.schema_version,
|
| 353 |
"session_id": event.session_id,
|
|
|
|
|
|
|
| 354 |
}
|
| 355 |
|
| 356 |
|
|
@@ -368,6 +370,8 @@ def _event_from_metadata(metadata: dict | None) -> Event | None:
|
|
| 368 |
payload=dict(metadata.get("payload") or {}),
|
| 369 |
schema_version=int(metadata.get("schema_version", 1)),
|
| 370 |
session_id=metadata.get("session_id") or None,
|
|
|
|
|
|
|
| 371 |
)
|
| 372 |
except (KeyError, ValueError, TypeError): # pragma: no cover - defensive
|
| 373 |
return None
|
|
|
|
| 351 |
"created_at": event.created_at.isoformat(),
|
| 352 |
"schema_version": event.schema_version,
|
| 353 |
"session_id": event.session_id,
|
| 354 |
+
"model_profile": event.model_profile,
|
| 355 |
+
"model_id": event.model_id,
|
| 356 |
}
|
| 357 |
|
| 358 |
|
|
|
|
| 370 |
payload=dict(metadata.get("payload") or {}),
|
| 371 |
schema_version=int(metadata.get("schema_version", 1)),
|
| 372 |
session_id=metadata.get("session_id") or None,
|
| 373 |
+
model_profile=metadata.get("model_profile") or None,
|
| 374 |
+
model_id=metadata.get("model_id") or None,
|
| 375 |
)
|
| 376 |
except (KeyError, ValueError, TypeError): # pragma: no cover - defensive
|
| 377 |
return None
|
src/core/sqlalchemy_ledger.py
CHANGED
|
@@ -70,6 +70,7 @@ class SqlAlchemyLedger(Ledger):
|
|
| 70 |
from sqlalchemy import (
|
| 71 |
Column,
|
| 72 |
DateTime,
|
|
|
|
| 73 |
Integer,
|
| 74 |
MetaData,
|
| 75 |
String,
|
|
@@ -97,6 +98,11 @@ class SqlAlchemyLedger(Ledger):
|
|
| 97 |
Column("created_at", DateTime(timezone=True), nullable=False),
|
| 98 |
Column("schema_version", Integer, nullable=False, server_default="1"),
|
| 99 |
Column("session_id", String, nullable=True, index=True),
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 100 |
)
|
| 101 |
self._metadata.create_all(self._engine)
|
| 102 |
|
|
@@ -124,6 +130,8 @@ class SqlAlchemyLedger(Ledger):
|
|
| 124 |
created_at=_aware(event.created_at),
|
| 125 |
schema_version=event.schema_version,
|
| 126 |
session_id=event.session_id,
|
|
|
|
|
|
|
| 127 |
)
|
| 128 |
)
|
| 129 |
self._cache.append(event)
|
|
@@ -243,6 +251,8 @@ class SqlAlchemyLedger(Ledger):
|
|
| 243 |
created_at=created_at,
|
| 244 |
schema_version=row["schema_version"],
|
| 245 |
session_id=row.get("session_id"),
|
|
|
|
|
|
|
| 246 |
)
|
| 247 |
|
| 248 |
|
|
|
|
| 70 |
from sqlalchemy import (
|
| 71 |
Column,
|
| 72 |
DateTime,
|
| 73 |
+
Index,
|
| 74 |
Integer,
|
| 75 |
MetaData,
|
| 76 |
String,
|
|
|
|
| 98 |
Column("created_at", DateTime(timezone=True), nullable=False),
|
| 99 |
Column("schema_version", Integer, nullable=False, server_default="1"),
|
| 100 |
Column("session_id", String, nullable=True, index=True),
|
| 101 |
+
# Which model produced the event (set by the agent; ADR-0028).
|
| 102 |
+
Column("model_profile", String, nullable=True),
|
| 103 |
+
Column("model_id", String, nullable=True, index=True),
|
| 104 |
+
# Composite index for the hottest read: events of one run, by offset.
|
| 105 |
+
Index("ix_events_run_offset", "run_id", "offset"),
|
| 106 |
)
|
| 107 |
self._metadata.create_all(self._engine)
|
| 108 |
|
|
|
|
| 130 |
created_at=_aware(event.created_at),
|
| 131 |
schema_version=event.schema_version,
|
| 132 |
session_id=event.session_id,
|
| 133 |
+
model_profile=event.model_profile,
|
| 134 |
+
model_id=event.model_id,
|
| 135 |
)
|
| 136 |
)
|
| 137 |
self._cache.append(event)
|
|
|
|
| 251 |
created_at=created_at,
|
| 252 |
schema_version=row["schema_version"],
|
| 253 |
session_id=row.get("session_id"),
|
| 254 |
+
model_profile=row.get("model_profile"),
|
| 255 |
+
model_id=row.get("model_id"),
|
| 256 |
)
|
| 257 |
|
| 258 |
|
src/core/sqlite_ledger.py
CHANGED
|
@@ -57,12 +57,17 @@ class SQLiteLedger(Ledger):
|
|
| 57 |
payload TEXT NOT NULL,
|
| 58 |
created_at TEXT NOT NULL,
|
| 59 |
schema_version INTEGER NOT NULL DEFAULT 1,
|
| 60 |
-
session_id TEXT
|
|
|
|
|
|
|
| 61 |
);
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
CREATE INDEX IF NOT EXISTS
|
|
|
|
|
|
|
| 65 |
CREATE INDEX IF NOT EXISTS idx_session_id ON events(session_id);
|
|
|
|
| 66 |
""")
|
| 67 |
self._conn.commit()
|
| 68 |
|
|
@@ -74,8 +79,9 @@ class SQLiteLedger(Ledger):
|
|
| 74 |
try:
|
| 75 |
self._conn.execute(
|
| 76 |
"INSERT INTO events "
|
| 77 |
-
"(id, run_id, turn, kind, actor, payload, created_at, schema_version,
|
| 78 |
-
"
|
|
|
|
| 79 |
(
|
| 80 |
event.id,
|
| 81 |
event.run_id,
|
|
@@ -86,6 +92,8 @@ class SQLiteLedger(Ledger):
|
|
| 86 |
event.created_at.isoformat(),
|
| 87 |
event.schema_version,
|
| 88 |
event.session_id,
|
|
|
|
|
|
|
| 89 |
),
|
| 90 |
)
|
| 91 |
self._conn.commit()
|
|
@@ -122,7 +130,9 @@ class SQLiteLedger(Ledger):
|
|
| 122 |
ledger = cls(path)
|
| 123 |
return ledger
|
| 124 |
|
| 125 |
-
_SELECT_COLS =
|
|
|
|
|
|
|
| 126 |
|
| 127 |
@staticmethod
|
| 128 |
def _row_to_event(row: tuple) -> Event:
|
|
@@ -142,6 +152,8 @@ class SQLiteLedger(Ledger):
|
|
| 142 |
created_at=created_at,
|
| 143 |
schema_version=row[7],
|
| 144 |
session_id=row[8],
|
|
|
|
|
|
|
| 145 |
)
|
| 146 |
|
| 147 |
def _load_cache(self) -> None:
|
|
|
|
| 57 |
payload TEXT NOT NULL,
|
| 58 |
created_at TEXT NOT NULL,
|
| 59 |
schema_version INTEGER NOT NULL DEFAULT 1,
|
| 60 |
+
session_id TEXT,
|
| 61 |
+
model_profile TEXT,
|
| 62 |
+
model_id TEXT
|
| 63 |
);
|
| 64 |
+
-- Composite (run_id, offset) serves the hottest read — events_for_run
|
| 65 |
+
-- ordered by offset — from one index, instead of a run_id seek + sort.
|
| 66 |
+
CREATE INDEX IF NOT EXISTS idx_run_offset ON events(run_id, offset);
|
| 67 |
+
CREATE INDEX IF NOT EXISTS idx_kind ON events(kind);
|
| 68 |
+
CREATE INDEX IF NOT EXISTS idx_actor ON events(actor);
|
| 69 |
CREATE INDEX IF NOT EXISTS idx_session_id ON events(session_id);
|
| 70 |
+
CREATE INDEX IF NOT EXISTS idx_model_id ON events(model_id);
|
| 71 |
""")
|
| 72 |
self._conn.commit()
|
| 73 |
|
|
|
|
| 79 |
try:
|
| 80 |
self._conn.execute(
|
| 81 |
"INSERT INTO events "
|
| 82 |
+
"(id, run_id, turn, kind, actor, payload, created_at, schema_version, "
|
| 83 |
+
"session_id, model_profile, model_id) "
|
| 84 |
+
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
| 85 |
(
|
| 86 |
event.id,
|
| 87 |
event.run_id,
|
|
|
|
| 92 |
event.created_at.isoformat(),
|
| 93 |
event.schema_version,
|
| 94 |
event.session_id,
|
| 95 |
+
event.model_profile,
|
| 96 |
+
event.model_id,
|
| 97 |
),
|
| 98 |
)
|
| 99 |
self._conn.commit()
|
|
|
|
| 130 |
ledger = cls(path)
|
| 131 |
return ledger
|
| 132 |
|
| 133 |
+
_SELECT_COLS = (
|
| 134 |
+
"id, run_id, turn, kind, actor, payload, created_at, schema_version, session_id, model_profile, model_id"
|
| 135 |
+
)
|
| 136 |
|
| 137 |
@staticmethod
|
| 138 |
def _row_to_event(row: tuple) -> Event:
|
|
|
|
| 152 |
created_at=created_at,
|
| 153 |
schema_version=row[7],
|
| 154 |
session_id=row[8],
|
| 155 |
+
model_profile=row[9],
|
| 156 |
+
model_id=row[10],
|
| 157 |
)
|
| 158 |
|
| 159 |
def _load_cache(self) -> None:
|
src/models/provider.py
CHANGED
|
@@ -41,6 +41,16 @@ class ModelProvider:
|
|
| 41 |
def complete(self, role: str, prompt: str) -> str:
|
| 42 |
raise NotImplementedError
|
| 43 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 44 |
@property
|
| 45 |
def last_usage(self) -> dict[str, int]:
|
| 46 |
"""Token usage of the most recent complete() call.
|
|
|
|
| 41 |
def complete(self, role: str, prompt: str) -> str:
|
| 42 |
raise NotImplementedError
|
| 43 |
|
| 44 |
+
@property
|
| 45 |
+
def model_id(self) -> str:
|
| 46 |
+
"""The concrete model this provider runs — for per-event attribution.
|
| 47 |
+
|
| 48 |
+
Uniform across backends: the live gateway sets ``self.model`` (e.g.
|
| 49 |
+
``openai/openbmb/MiniCPM4.1-8B``); the offline stub sets ``self.variant``
|
| 50 |
+
(e.g. ``stub:fast``). Empty string when neither is set.
|
| 51 |
+
"""
|
| 52 |
+
return str(getattr(self, "model", None) or getattr(self, "variant", None) or "")
|
| 53 |
+
|
| 54 |
@property
|
| 55 |
def last_usage(self) -> dict[str, int]:
|
| 56 |
"""Token usage of the most recent complete() call.
|
src/ui/fishbowl/adapter.py
CHANGED
|
@@ -98,6 +98,20 @@ def model_label(endpoint: str) -> str:
|
|
| 98 |
return short or endpoint
|
| 99 |
|
| 100 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 101 |
def agent_model(manifest) -> str:
|
| 102 |
"""The model an agent is actually running, for the Show's model badge.
|
| 103 |
|
|
|
|
| 98 |
return short or endpoint
|
| 99 |
|
| 100 |
|
| 101 |
+
def short_model_name(model_id: str) -> str:
|
| 102 |
+
"""Prettify a *concrete* resolved model string for the card's model badge.
|
| 103 |
+
|
| 104 |
+
Unlike :func:`model_label` (which resolves a catalogue *key*), this takes the
|
| 105 |
+
model that actually ran, recorded on the event envelope (ADR-0028): a served id
|
| 106 |
+
like ``openai/openbmb/MiniCPM4.1-8B`` → ``MiniCPM4.1-8B``, or the offline
|
| 107 |
+
``stub:fast`` left as-is. Empty input degrades to ``""`` so callers fall back.
|
| 108 |
+
"""
|
| 109 |
+
text = (model_id or "").strip()
|
| 110 |
+
if not text or text.startswith("stub:"):
|
| 111 |
+
return text
|
| 112 |
+
return text.rsplit("/", 1)[-1] or text
|
| 113 |
+
|
| 114 |
+
|
| 115 |
def agent_model(manifest) -> str:
|
| 116 |
"""The model an agent is actually running, for the Show's model badge.
|
| 117 |
|
src/ui/fishbowl/assets/styles.css
CHANGED
|
@@ -677,14 +677,29 @@ footer { display: none !important; }
|
|
| 677 |
.fishbowl .mind.rattled .mind-front { border-color: var(--coral); animation: cardShake 0.4s ease-in-out infinite; }
|
| 678 |
@keyframes cardShake { 0%,100% { transform: translateX(0); } 25% { transform: translateX(-2px); } 75% { transform: translateX(2px); } }
|
| 679 |
|
| 680 |
-
|
|
|
|
| 681 |
.fishbowl .mind-id { min-width: 0; }
|
| 682 |
-
.fishbowl .mind-name { font-size:
|
| 683 |
.fishbowl .mic { color: var(--coral); font-size: 9px; animation: livePulse 0.9s ease-in-out infinite; }
|
| 684 |
-
.fishbowl .mind-arch { font-size: 10.5px; color: var(--ink-dim); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
|
| 685 |
-
|
| 686 |
-
|
| 687 |
-
.fishbowl .mind-mood {
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 688 |
|
| 689 |
.fishbowl .bubbles { display: flex; flex-direction: column; gap: 8px; padding-top: 11px; }
|
| 690 |
.fishbowl .bubble { border-radius: var(--r); padding: 9px 11px 10px; position: relative; }
|
|
@@ -914,9 +929,6 @@ footer { display: none !important; }
|
|
| 914 |
0%, 100% { box-shadow: 0 0 18px color-mix(in oklab, var(--ac) 32%, transparent); }
|
| 915 |
50% { box-shadow: 0 0 30px color-mix(in oklab, var(--ac) 55%, transparent); }
|
| 916 |
}
|
| 917 |
-
/* tighten the head so name + model never feel cramped at card width */
|
| 918 |
-
.fishbowl .mind-name { line-height: 1.2; }
|
| 919 |
-
.fishbowl .mind-arch { margin-top: 2px; }
|
| 920 |
.fishbowl .bubble.said { box-shadow: inset 0 0 0 1px var(--line-soft); }
|
| 921 |
|
| 922 |
/* ---- SPLIT (Lab Readout): let the omniscient table flow full-width ---- */
|
|
|
|
| 677 |
.fishbowl .mind.rattled .mind-front { border-color: var(--coral); animation: cardShake 0.4s ease-in-out infinite; }
|
| 678 |
@keyframes cardShake { 0%,100% { transform: translateX(0); } 25% { transform: translateX(-2px); } 75% { transform: translateX(2px); } }
|
| 679 |
|
| 680 |
+
/* head: avatar · identity · the live mood pill (the one datum that changes turn to turn) */
|
| 681 |
+
.fishbowl .mind-head { display: grid; grid-template-columns: auto 1fr auto; gap: 11px; align-items: center; padding-bottom: 10px; border-bottom: 1px solid var(--line-soft); }
|
| 682 |
.fishbowl .mind-id { min-width: 0; }
|
| 683 |
+
.fishbowl .mind-name { font-size: 14.5px; font-weight: 700; letter-spacing: 0.04em; color: var(--ink); line-height: 1.15; display: flex; align-items: center; gap: 7px; }
|
| 684 |
.fishbowl .mic { color: var(--coral); font-size: 9px; animation: livePulse 0.9s ease-in-out infinite; }
|
| 685 |
+
.fishbowl .mind-arch { margin-top: 3px; font-size: 10.5px; color: var(--ink-dim); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
|
| 686 |
+
|
| 687 |
+
/* mood pill — colour-coded by mood, with a glowing status dot; reads at a glance */
|
| 688 |
+
.fishbowl .mind-mood-pill { display: inline-flex; align-items: center; gap: 6px; padding: 3px 9px 3px 8px; border-radius: 999px; border: 1px solid color-mix(in oklab, var(--mc) 50%, transparent); background: color-mix(in oklab, var(--mc) 13%, transparent); white-space: nowrap; align-self: start; }
|
| 689 |
+
.fishbowl .mmp-dot { width: 6px; height: 6px; border-radius: 50%; background: var(--mc); box-shadow: 0 0 7px var(--mc); flex: none; }
|
| 690 |
+
.fishbowl .mmp-label { font-family: var(--font-display); font-size: 8px; font-weight: 600; letter-spacing: 0.12em; text-transform: uppercase; color: var(--mc); }
|
| 691 |
+
/* a panicking mind throbs its pill so the leak is impossible to miss */
|
| 692 |
+
@media (prefers-reduced-motion: no-preference) {
|
| 693 |
+
.fishbowl .mind.rattled .mmp-dot { animation: livePulse 0.7s ease-in-out infinite; }
|
| 694 |
+
}
|
| 695 |
+
|
| 696 |
+
/* model meta-strip — provenance gets its own labelled row, so a long served id stays legible */
|
| 697 |
+
.fishbowl .mind-meta-strip { display: flex; align-items: center; justify-content: space-between; gap: 8px; margin-top: 10px; padding: 6px 10px; border: 1px solid var(--line-soft); border-radius: var(--r); background: rgba(3,12,18,0.55); }
|
| 698 |
+
.fishbowl .mms-model { display: flex; align-items: center; gap: 7px; min-width: 0; }
|
| 699 |
+
.fishbowl .mms-model .tier-dot { width: 7px; height: 7px; }
|
| 700 |
+
.fishbowl .mms-name { font-family: var(--font-display); font-size: 10px; letter-spacing: 0.01em; color: var(--ink-mid); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
|
| 701 |
+
.fishbowl .mms-tier { font-family: var(--font-display); font-size: 8px; letter-spacing: 0.16em; text-transform: uppercase; color: var(--ink-faint); flex: none; }
|
| 702 |
+
.fishbowl .mms-tier:empty { display: none; }
|
| 703 |
|
| 704 |
.fishbowl .bubbles { display: flex; flex-direction: column; gap: 8px; padding-top: 11px; }
|
| 705 |
.fishbowl .bubble { border-radius: var(--r); padding: 9px 11px 10px; position: relative; }
|
|
|
|
| 929 |
0%, 100% { box-shadow: 0 0 18px color-mix(in oklab, var(--ac) 32%, transparent); }
|
| 930 |
50% { box-shadow: 0 0 30px color-mix(in oklab, var(--ac) 55%, transparent); }
|
| 931 |
}
|
|
|
|
|
|
|
|
|
|
| 932 |
.fishbowl .bubble.said { box-shadow: inset 0 0 0 1px var(--line-soft); }
|
| 933 |
|
| 934 |
/* ---- SPLIT (Lab Readout): let the omniscient table flow full-width ---- */
|
src/ui/fishbowl/render/mindcard.py
CHANGED
|
@@ -12,7 +12,7 @@ from __future__ import annotations
|
|
| 12 |
|
| 13 |
import html
|
| 14 |
|
| 15 |
-
from src.ui.fishbowl.adapter import TIER_COLOR
|
| 16 |
from src.ui.fishbowl.render.avatar import agent_color, agent_color_dim, render_avatar
|
| 17 |
|
| 18 |
__all__ = ["render_mindcard"]
|
|
@@ -74,8 +74,13 @@ def render_mindcard(
|
|
| 74 |
# to the profile tier name when it routes purely by profile.
|
| 75 |
model = html.escape(str(card.get("model") or card.get("model_profile", "")))
|
| 76 |
mood_label = html.escape(str(card.get("mood_label", "")))
|
|
|
|
| 77 |
tier = card.get("tier", "mid")
|
| 78 |
tier_color = TIER_COLOR.get(tier, "var(--cyan)")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
|
| 80 |
mic = '<span class="mic">●</span>' if speaking else ""
|
| 81 |
|
|
@@ -104,11 +109,18 @@ def render_mindcard(
|
|
| 104 |
f'<div class="mind-name disp">{name}{mic}</div>'
|
| 105 |
f'<div class="mind-arch">{archetype}</div>'
|
| 106 |
"</div>"
|
| 107 |
-
'<div class="mind-
|
| 108 |
-
|
| 109 |
-
f'<span class="
|
| 110 |
"</div>"
|
| 111 |
"</div>"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
'<div class="bubbles">'
|
| 113 |
'<div class="bubble said">'
|
| 114 |
'<span class="bub-tag">said aloud</span>'
|
|
|
|
| 12 |
|
| 13 |
import html
|
| 14 |
|
| 15 |
+
from src.ui.fishbowl.adapter import TIER_COLOR, mood_color
|
| 16 |
from src.ui.fishbowl.render.avatar import agent_color, agent_color_dim, render_avatar
|
| 17 |
|
| 18 |
__all__ = ["render_mindcard"]
|
|
|
|
| 74 |
# to the profile tier name when it routes purely by profile.
|
| 75 |
model = html.escape(str(card.get("model") or card.get("model_profile", "")))
|
| 76 |
mood_label = html.escape(str(card.get("mood_label", "")))
|
| 77 |
+
mood_col = mood_color(mood)
|
| 78 |
tier = card.get("tier", "mid")
|
| 79 |
tier_color = TIER_COLOR.get(tier, "var(--cyan)")
|
| 80 |
+
# The profile label (tiny/fast/balanced/strong) sits beside the model name as the
|
| 81 |
+
# human-chosen tier; the concrete served id rides the tooltip so a long name stays short.
|
| 82 |
+
profile = html.escape(str(card.get("model_profile", "")))
|
| 83 |
+
model_title = html.escape(str(card.get("model_id") or card.get("model") or profile))
|
| 84 |
|
| 85 |
mic = '<span class="mic">●</span>' if speaking else ""
|
| 86 |
|
|
|
|
| 109 |
f'<div class="mind-name disp">{name}{mic}</div>'
|
| 110 |
f'<div class="mind-arch">{archetype}</div>'
|
| 111 |
"</div>"
|
| 112 |
+
f'<div class="mind-mood-pill" style="--mc:{mood_col}">'
|
| 113 |
+
'<span class="mmp-dot"></span>'
|
| 114 |
+
f'<span class="mmp-label">{mood_label}</span>'
|
| 115 |
"</div>"
|
| 116 |
"</div>"
|
| 117 |
+
f'<div class="mind-meta-strip" title="{model_title}">'
|
| 118 |
+
'<span class="mms-model">'
|
| 119 |
+
f'<span class="tier-dot" style="background:{tier_color}"></span>'
|
| 120 |
+
f'<span class="mms-name">{model}</span>'
|
| 121 |
+
"</span>"
|
| 122 |
+
f'<span class="mms-tier">{profile}</span>'
|
| 123 |
+
"</div>"
|
| 124 |
'<div class="bubbles">'
|
| 125 |
'<div class="bubble said">'
|
| 126 |
'<span class="bub-tag">said aloud</span>'
|
src/ui/fishbowl/view_model.py
CHANGED
|
@@ -27,6 +27,7 @@ from src.ui.fishbowl.adapter import (
|
|
| 27 |
mood_label,
|
| 28 |
normalize_mood,
|
| 29 |
scenario_voice,
|
|
|
|
| 30 |
)
|
| 31 |
from src.ui.fishbowl.cast_state import derive_cast_state
|
| 32 |
|
|
@@ -65,6 +66,13 @@ def view_model_at(
|
|
| 65 |
names = [m.name for m in cast]
|
| 66 |
states = derive_cast_state(prefix, names)
|
| 67 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
speaking_id: str | None = None
|
| 69 |
if k > 0:
|
| 70 |
head = events[k - 1]
|
|
@@ -82,7 +90,9 @@ def view_model_at(
|
|
| 82 |
"hue": agent_hue(m),
|
| 83 |
"role": m.role,
|
| 84 |
"model_profile": m.model_profile,
|
| 85 |
-
|
|
|
|
|
|
|
| 86 |
"model_endpoint": getattr(m, "model_endpoint", None),
|
| 87 |
"tier": agent_tier(m),
|
| 88 |
"said": st.said,
|
|
|
|
| 27 |
mood_label,
|
| 28 |
normalize_mood,
|
| 29 |
scenario_voice,
|
| 30 |
+
short_model_name,
|
| 31 |
)
|
| 32 |
from src.ui.fishbowl.cast_state import derive_cast_state
|
| 33 |
|
|
|
|
| 66 |
names = [m.name for m in cast]
|
| 67 |
states = derive_cast_state(prefix, names)
|
| 68 |
|
| 69 |
+
# The model that *actually* produced each actor's most recent line (envelope,
|
| 70 |
+
# ADR-0028) — overrides the card's intended binding once a line has been spoken.
|
| 71 |
+
actual_model: dict[str, str] = {}
|
| 72 |
+
for e in prefix:
|
| 73 |
+
if e.model_id and e.actor in names:
|
| 74 |
+
actual_model[e.actor] = e.model_id
|
| 75 |
+
|
| 76 |
speaking_id: str | None = None
|
| 77 |
if k > 0:
|
| 78 |
head = events[k - 1]
|
|
|
|
| 90 |
"hue": agent_hue(m),
|
| 91 |
"role": m.role,
|
| 92 |
"model_profile": m.model_profile,
|
| 93 |
+
# Prefer the model that actually ran; fall back to the intended binding.
|
| 94 |
+
"model": short_model_name(actual_model[m.name]) if m.name in actual_model else agent_model(m),
|
| 95 |
+
"model_id": actual_model.get(m.name),
|
| 96 |
"model_endpoint": getattr(m, "model_endpoint", None),
|
| 97 |
"tier": agent_tier(m),
|
| 98 |
"said": st.said,
|
tests/test_model_attribution.py
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Per-event model attribution (ADR-0028): each agent line records the model that
|
| 2 |
+
actually produced it — the route key it asked for (``model_profile``) and the
|
| 3 |
+
concrete model that ran (``model_id``) — and that survives the SQL round-trip and
|
| 4 |
+
surfaces on the Show's cast cards.
|
| 5 |
+
|
| 6 |
+
No mocks: the deterministic stub drives the cast, so ``model_id`` reads ``stub:<tier>``
|
| 7 |
+
offline — the same envelope a live Modal/HF run fills with the served model id.
|
| 8 |
+
"""
|
| 9 |
+
|
| 10 |
+
from __future__ import annotations
|
| 11 |
+
|
| 12 |
+
import pytest
|
| 13 |
+
|
| 14 |
+
from src.core.ledger_factory import make_ledger
|
| 15 |
+
from src.core.registry import default_registry
|
| 16 |
+
from src.ui.fishbowl.adapter import short_model_name
|
| 17 |
+
from src.ui.fishbowl.session import FishbowlSession
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
@pytest.fixture
|
| 21 |
+
def shared_db(monkeypatch, tmp_path):
|
| 22 |
+
monkeypatch.setenv("DATABASE_URL", f"sqlite:///{tmp_path / 'events.db'}")
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def _first_scenario() -> str:
|
| 26 |
+
return next(iter(default_registry().scenarios))
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def _run_with_lines(session_id: str = "u1") -> FishbowlSession:
|
| 30 |
+
session = FishbowlSession(_first_scenario())
|
| 31 |
+
session.reset("seed", session_id=session_id)
|
| 32 |
+
for _ in range(session.autoplay_tick_cap):
|
| 33 |
+
events = session.events
|
| 34 |
+
if sum(1 for e in events if e.model_id) >= 2:
|
| 35 |
+
break
|
| 36 |
+
try:
|
| 37 |
+
if not session.step_one():
|
| 38 |
+
break
|
| 39 |
+
except Exception:
|
| 40 |
+
break
|
| 41 |
+
return session
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
class TestShortModelName:
|
| 45 |
+
def test_strips_org_prefix(self):
|
| 46 |
+
assert short_model_name("openai/openbmb/MiniCPM4.1-8B") == "MiniCPM4.1-8B"
|
| 47 |
+
assert short_model_name("google/gemma-4-12B") == "gemma-4-12B"
|
| 48 |
+
|
| 49 |
+
def test_leaves_stub_and_empty_alone(self):
|
| 50 |
+
assert short_model_name("stub:fast") == "stub:fast"
|
| 51 |
+
assert short_model_name("") == ""
|
| 52 |
+
assert short_model_name(None) == "" # type: ignore[arg-type]
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
class TestEventModelAttribution:
|
| 56 |
+
def test_agent_events_record_profile_and_model(self, shared_db):
|
| 57 |
+
session = _run_with_lines()
|
| 58 |
+
produced = [e for e in session.events if e.model_id]
|
| 59 |
+
assert produced, "stub cast should have produced at least one model-backed line"
|
| 60 |
+
for e in produced:
|
| 61 |
+
# Offline, the route key is a tier and the model is its stub.
|
| 62 |
+
assert e.model_profile # the route key the agent asked for
|
| 63 |
+
assert e.model_id == f"stub:{e.model_profile}" or e.model_id.startswith("stub:")
|
| 64 |
+
|
| 65 |
+
def test_scenario_and_genesis_events_have_no_model(self, shared_db):
|
| 66 |
+
session = _run_with_lines()
|
| 67 |
+
for e in session.events:
|
| 68 |
+
if e.kind in ("run.started", "run.finished") or e.actor == "conductor":
|
| 69 |
+
assert e.model_id is None and e.model_profile is None
|
| 70 |
+
|
| 71 |
+
def test_model_attribution_survives_sql_round_trip(self, shared_db):
|
| 72 |
+
session = _run_with_lines()
|
| 73 |
+
run_id = session.conductor.run_id
|
| 74 |
+
# A fresh ledger connection re-reads rows from disk — envelope must persist.
|
| 75 |
+
reread = make_ledger().events_for_run(run_id)
|
| 76 |
+
produced = [e for e in reread if e.model_id]
|
| 77 |
+
assert produced
|
| 78 |
+
assert all(e.model_profile for e in produced)
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
class TestCardSurfacesActualModel:
|
| 82 |
+
def test_card_model_reflects_the_model_that_ran(self, shared_db):
|
| 83 |
+
session = _run_with_lines()
|
| 84 |
+
vm = session.snapshot()
|
| 85 |
+
spoken_actors = {e.actor for e in session.events if e.model_id}
|
| 86 |
+
cards = {c["id"]: c for c in vm["cast"]}
|
| 87 |
+
# Every actor that produced a line shows its actual (stub) model on the card.
|
| 88 |
+
for actor in spoken_actors & cards.keys():
|
| 89 |
+
assert cards[actor]["model_id"] is not None
|
| 90 |
+
assert cards[actor]["model"] == short_model_name(cards[actor]["model_id"])
|