backend/app/interactive/playback/README.md

Live-play playback subsystem

The live-play surface (POST /v1/interactive/play/sessions/{sid}/chat

• GET /pending) runs the real-time scene-generation loop: chat arrives, the planner composes a scene, a render job goes to the video backend, and the clip streams back to the InteractivePlayer in the frontend.

Module map

File	Responsibility
scene_memory.py	Rolling context snapshot (persona, mood, last N turns, synopsis). Pure, synchronous.
scene_planner.py	plan_next_scene (sync heuristic) + plan_next_scene_async (LLM first, heuristic fallback).
llm_composer.py	LLM-backed composer — app.llm.chat_ollama with strict JSON mode.
video_job.py	ix_scene_queue CRUD + render_now (sync stub) + render_now_async (real-first, stub fallback).
render_adapter.py	Thin async wrapper over app.comfy.run_workflow + app.asset_registry.register_asset.
asset_urls.py	resolve_asset_url — registry lookup for the frontend <video src>.
schema.py	Defensive CREATE IF NOT EXISTS for ix_scene_queue.
playback_config.py	Env-flag + tuning-knob loader.

Enabling Phase-2 in production

All flags default off so upgrading is a no-op until you opt in.

# LLM scene composer — uses the configured Ollama / OpenAI-compatible backend
export INTERACTIVE_PLAYBACK_LLM=1
export INTERACTIVE_PLAYBACK_LLM_TIMEOUT_S=12        # optional, default 12
export INTERACTIVE_PLAYBACK_LLM_MAX_TOKENS=350      # optional, default 350
export INTERACTIVE_PLAYBACK_LLM_TEMPERATURE=0.65    # optional, default 0.65

# Video renderer — submits scene prompts to ComfyUI via run_workflow
export INTERACTIVE_PLAYBACK_RENDER=1
export INTERACTIVE_PLAYBACK_RENDER_WORKFLOW=animate   # optional, default 'animate'
export INTERACTIVE_PLAYBACK_RENDER_TIMEOUT_S=180      # optional, default 180

# Ollama + Comfy endpoints use the app-level config (unchanged)
export OLLAMA_BASE_URL=http://localhost:11434
export OLLAMA_MODEL=llama3.1:8b-instruct
export COMFY_BASE_URL=http://localhost:8188

Flip one or both flags per deployment — for example staging can run the LLM composer with a stub renderer while the real video backend warms up.

Failure modes (graceful by design)

Failure	Behaviour
INTERACTIVE_PLAYBACK_LLM=1 but the LLM is unreachable / slow / malformed JSON	Falls back to the heuristic composer. No user-visible error. Structured log lines: playback_llm_timeout, playback_llm_error, playback_llm_malformed_json.
INTERACTIVE_PLAYBACK_RENDER=1 but ComfyUI is down / times out / no output	Falls back to the phase-1 stub asset id. Job still lands ready. Logs: playback_render_timeout, playback_render_error, playback_render_no_output.
asset_registry.get_asset throws	resolve_asset_url returns None; frontend shows the mood backdrop.
Policy gate (check_free_input) blocks the message	/chat returns {"status": "blocked", …} without invoking the planner or render pipeline.

Each fallback keeps the player's polling loop in a terminal state, so the UI never hangs on a half-resolved turn.

Observability

All structured logs share an extra={"session_id": …} field for correlation. Grep-friendly event names:

• playback_llm_timeout / playback_llm_error / playback_llm_malformed_json / playback_llm_empty_content / playback_llm_missing_fields
• playback_render_timeout / playback_render_error / playback_render_no_output
• playback_asset_register_failed
• playback_asset_lookup_failed

Job-id stamps in ix_scene_queue.job_id carry a prefix so you can separate the two render paths at a glance:

• stub-* — phase-1 fallback path used this request
• live-* — phase-2 real render was attempted

Test coverage

The subsystem ships with 60+ tests across 7 files:

tests/test_interactive_playback_memory.py        rolling context + synopsis
tests/test_interactive_playback_planner.py       heuristic composer
tests/test_interactive_playback_planner_async.py LLM-first dispatch
tests/test_interactive_playback_llm_composer.py  LLM parsing + guardrails
tests/test_interactive_playback_jobs.py          scene queue + state machine
tests/test_interactive_playback_jobs_async.py    render_now_async fallback
tests/test_interactive_playback_render_adapter.py ComfyUI adapter contract
tests/test_interactive_playback_chat.py          /chat route end-to-end
tests/test_interactive_playback_asset_urls.py    URL resolver
tests/test_interactive_playback_config.py        env flag loader

Every test monkey-patches the real external call (chat_ollama, run_workflow, register_asset) so CI never reaches for a network.