docs/adr/0016-instructor-structured-output.md

# ADR-0016: Validated Structured Output on the Live Path

## Status

Accepted

## Context

Structured output (`src/core/structured.py`) constrains agents to a
`{kind, text, …}` event payload by appending a JSON `OUTPUT FORMAT` block to the
prompt and parsing the model's reply with `parse_agent_output`. The parser has
three tiers: strict JSON, a regex that extracts an embedded `{…}` block, and a
final **`_raw_fallback`** that wraps whatever prose the model returned as a valid
event under the agent's first allowed kind.

That last tier is a correctness hazard. The ledger is append-only and
event-sourced (ADR-0001): every appended event is permanent history that
projections and memory replay. When a small model drifts into prose, the
`_raw_fallback` path *silently* admits non-compliant text as a structured event —
corrupting the ledger with a payload the model never intended as that kind, and
masking the failure as a normal turn. The migration note in
`docs/architecture/structured-output.md` already anticipated replacing
prompt-and-parse with enforced structured output once the transport supported it.

The LiteLLM gateway (ADR-0015) made that transport available: it issues a single,
idiomatic `litellm.completion(...)` deliberately shaped so a layer could wrap it
with `instructor.from_litellm(litellm.completion)` for validated output.

## Decision

Add **validated structured output on the live path**, keeping the tolerant parser
as the **offline** path. The event schema and the agent's emitted-event contract
are unchanged (ADR-0009): this improves how a payload is *produced*, not what an
event *is*.

**Dynamic, constrained output model.** `build_output_model(allowed_kinds,
extra_fields)` (in `structured.py`) builds a Pydantic model whose `kind` is a
`Literal` over the agent's `may_emit` grant (reflection excluded) and whose `text`
plus any `output_extra_fields` are required strings. A model literally cannot
validate with a kind it is not authorised to emit — the same `may_emit` boundary
the parser enforced, now enforced by the type. It is pure Pydantic: no provider,
no network, independently testable, and importable with `instructor` not
installed.

**Structured capability on the gateway.** `LiteLLMProvider.complete_structured(
role, prompt, response_model)` wraps the *same* `litellm.completion` with
`instructor.from_litellm(...)` and calls `create_with_completion(...,
response_model=…, max_retries=…)`. Instructor re-prompts on validation failure;
on success it returns both the validated instance and the raw completion, so
tokens and cost are read from that completion exactly as `complete()` does
(`last_usage["cost_usd"]` / `last_cost`). The plain `complete()` is retained
unchanged. `instructor` is imported lazily inside the method.

**Capability-checked wiring.** `ManifestAgent.act()` delegates to
`_resolve_payload(...)`: if the routed provider exposes `complete_structured`
(`hasattr`), it builds the constrained model, calls it, and returns
`result.model_dump()` — a validated payload with **no `_raw_fallback`**. The stub
has no such method, so offline takes the existing `json_instruction` +
`parse_agent_output` path untouched. If a live structured call raises
(validation exhausted or transport error), the agent falls back to the parser
path so a turn still produces an event rather than dropping. Token/cost usage is
recorded from the provider in every branch, so the conductor's
`governor.record_call(...)` (ADR-0013, ADR-0015) is unaffected.

**Dependency.** `instructor` is a new optional `instructor` extra in
`pyproject.toml`. Lazy imports keep `import src.*` and `import app` working with
it not installed.

## Refinement: guided decoding, not tool calling (2026-06)

The first cut left Instructor on its default `Mode.TOOLS`, which encodes the
schema as an OpenAI **function/tool call**. That only validates on a served model
whose vLLM deployment has tool calling enabled with a *matching* parser. The
`fast` tier (`minicpm-4-1-8b`, ADR-0022 catalogue) has neither: MiniCPM4.1 emits a
custom `<|tool_call_start|> … <|tool_call_end|>` format for which vLLM 0.21.0 ships
no parser, so every structured call returned **`400 Bad Request`** (rejected at
request validation, ~40 ms, no generation) and degraded to the prose fallback —
turning the `fast` tier's fast validated-JSON path into a ~7 s prose round-trip
every turn, and feeding the `clean_clue` over-filter that dropped first-person
clues (the `spy-bex` "no usable line" failure).

`LiteLLMProvider.structured_mode` now defaults to **`json_schema`** — vLLM
**guided decoding** via `response_format`, which constrains output to the schema
*without* a tool-call parser, so it is correct for every served model regardless of
tool support (Gemma/Nemotron keep validating; MiniCPM now validates instead of
400ing). It is a per-provider field (an `instructor.Mode` member name): `json`
(plain `json_object` + schema-in-prompt) is the fallback if a backend rejects
`json_schema`, and `tools` restores the old behaviour for a model that prefers it.
No redeploy is needed — the change is entirely client-side on the request shape.

## Consequences

- On the live path, agent output is schema-valid and kind-constrained by
  construction; malformed prose is retried, not admitted. The `_raw_fallback`
  corruption cannot enter the ledger when structured output is active.
- The offline path is the default and unchanged: deterministic stub +
  `parse_agent_output`, including the `_raw_fallback` tier and its tests. The full
  suite passes with no network, no credentials, and neither `instructor` nor
  `litellm` installed.
- `build_output_model` is the single source of the output contract, shared by the
  constraint and (implicitly) the parser's `{kind, text}` shape, so the two paths
  stay aligned.
- The live structured call is two messages and one or more model round-trips
  (retries); cost is metered per the underlying completion. Retries are bounded by
  `max_retries` (default 2).
- A live structured failure degrades to the parser path. This preserves liveness
  but means a persistently failing structured call can still reach the
  `_raw_fallback` tier; the `_raw_fallback` flag remains the signal that a prompt
  or model needs attention.
- Follow-up: thread `max_retries` through the router's per-profile spec so a
  scenario can tune it per tier, and surface the structured-vs-parser path in the
  stats panel alongside the `_raw_fallback` rate.