docs/adr/0024-observability.md

ADR-0024: Observability — Structured Logging, Tracing, In-App Monitoring

Status

Accepted

Context

The engine had no cohesive observability. Each module made its own logging.getLogger(__name__) with almost no calls (conductor.py, memory.py), and the only structured output was the dependency-free JSON formatter for the vLLM subprocess (modal/vllm_logging.py). There was no way to see, in one place, the prompts passed to each model and the memory each agent had access to — the two things that matter most when debugging a multi-agent run — nor any traces or metrics across the api-call → inference → memory → core-loop path.

We want a complete, modular log of the application that foregrounds agent behaviour, data, and LLM/API calls, readable both in the terminal and live in the Gradio app, leveraging OpenTelemetry for logging, tracing, and basic monitoring.

Decision

Introduce a single src/observability/ package — a thin facade over OpenTelemetry plus an in-memory store — that every layer imports as from src import observability as obs.

• One stable facade. configure(), get_logger(), log(event, **fields), span(name, **attrs), add_span_attrs(), incr()/observe() + named helpers (record_llm_call, record_agent_turn, record_governor_trip), bind()/set_context()/current_context(), and telemetry_store(). Call sites never touch the OTEL SDK directly, so instrumentation stays a one-liner and the wiring lives in one module.

• Correlation by context, not by parameter. Run / turn / agent are carried in contextvars and stamped onto every log record and span automatically. The conductor binds the run and turn; agents bind their name. A single llm.call line therefore says which agent, which turn, which run, with no threading.

• Real OTEL tracing, self-contained backend. A real TracerProvider makes spans nest by context: run → turn → agent.turn → {memory.recall → memory.index.search} + {llm.call | llm.structured} + tool.call. Finished spans and all log records flow into bounded in-memory ring buffers (TelemetryStore); a console exporter can also print spans. No Jaeger / Prometheus / Grafana to deploy — the whole monitoring story lives in-process and in the live demo. Metrics (LLM calls, tokens, cost, agent-turn latency, governor trips) are in-process counters/observations feeding the UI charts.

• In-app surface. A Gradio "Telemetry" tab reads from telemetry_store(): a filterable structured-log feed, metric charts, and a per-turn trace/timeline where selecting a span reveals the actual prompt + memory the agent saw.

• Env-driven, no secrets. MAL_LOG_LEVEL (DEBUG surfaces full prompts + memory), MAL_LOG_FORMAT (text|json), MAL_TRACING (off|console|memory|both, default memory). API keys are never logged or attached as span attributes; full prompts/memory are captured at DEBUG and truncated in stored snapshots for the UI.

OpenTelemetry (opentelemetry-api, -sdk, -semantic-conventions) is a first-class core dependency. LLM spans use GenAI semantic-convention attributes (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens / output_tokens) alongside engine-specific ones (llm.cost_usd, llm.prompt, llm.completion, llm.reasoning).

Consequences

• Any module gets full structured logging, tracing, and metrics by importing the facade — instrumentation is consistent and decoupled from the SDK.
• The live demo can show logs, traces, and charts with nothing extra to run; the in-memory buffers are bounded, so memory stays flat over a long session.
• OTEL is now a required dependency (the prior offline/no-new-deps stance is relaxed for observability, by decision); when MAL_TRACING=off, the API's no-op tracer keeps overhead negligible.
• setup_logging configures the root logger (the engine had none); it only manages handlers it tags _mal, leaving third-party handlers intact and making re-configuration idempotent.
• Future work (not in this ADR): optional OTLP export to an external collector, and mirroring the in-process metrics to an OTEL MeterProvider.