docs/immersive_experience_execution_plan.md

# Immersive Time-Travel Experience Execution Plan

Date: 2026-06-13

## Product Direction

Build a surreal time-travel spectacle first. The user should not feel like they opened a voice chatbot; they should feel like they activated a machine, crossed a portal, landed in another time, and met a person who belongs there.

The core demo sequence:

1. User chooses coordinates and presses **Launch**.
2. The cockpit begins a dramatic portal sequence.
3. A year counter rewinds or fast-forwards from the current year into the destination year.
4. The portal hides model latency while world, character, and optional narration assets are prepared.
5. The destination fades in slowly: first atmosphere, then world image, then character portrait.
6. A narrator voice introduces the world and character in a cinematic tone.
7. The narrator voice stops. The character voice is clearly different.
8. Live conversation begins inside the revealed world.
9. The user can ask for an artifact/souvenir and receive a visual object from that world.

## Hackathon Constraints

- The app remains a Gradio app hosted as a Hugging Face Space.
- The 32B parameter limit is treated as **per model used**, not cumulative across all models.
- Every model used for core functionality must have an individually defensible parameter count.
- Cloud APIs are allowed unless a specific bonus badge requires local-only operation. We are optimizing for wow and reliability, not Off-the-Grid.
- Current model choices remain valid:
  - LLM: Qwen/Qwen3-8B, 8B.
  - STT: nvidia/nemotron-3.5-asr-streaming-0.6b, 0.6B.
  - TTS: ResembleAI/chatterbox or Chatterbox Turbo family, about 0.5B.
  - Image: black-forest-labs/FLUX.1-schnell, 12B.

## Architecture Strategy

Keep the existing ports-and-adapters architecture.

Add immersion as typed events and UI state, not as one-off Gradio hacks:

- `ImmersiveScene`: generated/fallback scene metadata.
- `CharacterPortrait`: generated/fallback portrait metadata.
- `Narration`: text, audio path, narrator voice profile.
- `Artifact`: visual souvenir image plus markdown description.
- `ImmersiveEvent`: event emitted during launch/reveal.

The backend remains responsible for encounter state. The frontend owns visual animation and audio mixing.

## Phase 1: Portal Shell And State Plumbing

Goal: Make launch feel dramatic before any new model integration.

Build:

- Portal stage in `cockpit.html`.
- Year-travel counter that animates from current year to destination year.
- Dramatic launch states: dormant, launching, traversing, resolving, revealing, conversation ready.
- World reveal container with fallback gradient/texture.
- Character portrait aperture with fallback silhouette.
- Narration caption slot.
- JS function to ingest a hidden JSON UI payload and update the portal.
- Preserve existing live voice controls and transcript.

No external APIs.
No image generation.
No narration audio yet.

Validation:

- User clicks Launch and sees a cinematic portal sequence.
- Year counter moves in the correct direction for past/future.
- Destination/persona data appears in the immersive shell.
- Existing voice controls still work.
- The app still runs with fixture profile.

## Phase 2: Immersive Backend Contracts

Goal: Add backend contracts that can carry scene, portrait, narration, and artifact data without forcing UI rewrites.

Build:

- Domain models:
  - `ImmersiveScene`
  - `CharacterPortrait`
  - `Narration`
  - `VisualArtifact`
- Domain events:
  - `SceneEvent`
  - `PortraitEvent`
  - `NarrationEvent`
  - `VisualArtifactEvent`
- UI render state fields for immersive payload JSON.
- Fixture adapter data for at least two hero encounters.
- Tests for event ordering and serialization.

Validation:

- Fixture encounters populate scene/portrait/artifact placeholders.
- Hidden JSON payload updates after launch.
- No real image/audio APIs are required.

## Phase 3: Image Generation With Together FLUX

Goal: Generate the world and character from the destination/persona.

Build:

- `ImageGenerationPort`.
- Together adapter using `black-forest-labs/FLUX.1-schnell`.
- Generate scene and portrait in parallel after destination/persona are available.
- Cache base64 images per encounter.
- Fallback to generated CSS visual if image generation fails.
- Add `image` model entry to `config/models.yaml` with 12B parameter count.
- Update model-budget validation to check each enabled model individually against 32B.

Validation:

- Scene and portrait appear during or shortly after portal traversal.
- Slow image generation does not block the visual launch sequence.
- Failure leaves the demo usable.

## Phase 4: Cinematic Narration

Goal: Add a movie-opening narration that is distinct from the character voice.

Build:

- Narration prompt template:
  - 15-25 seconds max.
  - Describes year, place, atmosphere, and the person being contacted.
  - Uses third-person cinematic narration.
  - Avoids speaking as the character.
- Narrator voice profile:
  - Stable, lower-energy, documentary/cinematic style.
  - Separate from character `VoiceProfile`.
- Generate narration audio during portal animation.
- Play narration once after world reveal.
- Duck ambient audio under narration.
- Only open live voice after narration finishes or user skips.

Validation:

- Narrator and character voices are obviously different.
- Narration improves immersion without delaying interaction too much.
- Skip/continue control exists if audio generation is slow.

## Phase 5: Ambient Sound And Reactive Mixing

Goal: Make the world feel alive before the character speaks.

Build:

- Bundle small CC0 ambient loops:
  - temporal storm / portal hum
  - rain
  - market/crowd murmur
  - fire/hearth
  - wind
  - machinery
  - insects/night
  - water/harbor
- Ambient selector based on destination atmosphere, mode, and visual motifs.
- Web Audio mixer:
  - portal hum during traversal
  - crossfade to destination ambience
  - duck ambience under narration and character speech
  - optional stereo panning/small reverb

Validation:

- Launch has sound even before voice response.
- Ambience does not fight speech.
- Audio remains optional and browser-safe.

## Phase 6: Artifact Reveal

Goal: Create a memorable "the character handed me something" moment.

Build:

- Extend souvenir generation into a visual artifact:
  - letter
  - map fragment
  - newspaper clipping
  - receipt
  - sketch
  - warning poster
  - tool tag
- Generate artifact prompt from destination, persona, and conversation.
- Reveal artifact in the portal/world UI with a physical animation.
- Save artifact with encounter trace.

Validation:

- Asking for a souvenir creates a visual object, not just markdown.
- Artifact matches the era and persona.
- Artifact is shareable in screenshots.

## Phase 7: Hero Demo Polish

Goal: Make the submission judge-ready.

Build:

- Two golden paths:
  - Surreal past encounter.
  - Strange/future encounter.
- A deterministic demo script:
  - Launch.
  - Listen to narration.
  - Ask what is happening.
  - Ask about one visible object.
  - Ask for a souvenir.
  - Challenge with a modern concept and watch the character react in-world.
- Field Notes doc.
- Shared traces.
- Screenshot/video-friendly layout.
- HF Space deployment check.

Validation:

- Demo works cold.
- Demo video can be recorded in one take.
- The app visibly qualifies for Off-Brand, Field Notes, and Sharing Is Caring.

## Deferred Until After Core Wow Works

- Full 3D avatar.
- Wav2Lip/SadTalker/LivePortrait realtime animation.
- Three.js room navigation.
- Gaussian splat viewer.
- WebXR.
- Fully live image generation for every conversation turn.

These are visually attractive but risky. A generated scene, portrait, ambient sound, narration, and artifact reveal provide most of the perceived immersion with much lower demo risk.

## Agent Implementation Model

Use agents by phase with disjoint ownership:

- Frontend worker: `cockpit.html`, `cockpit.css`, `cockpit.js`.
- Backend contracts worker: domain models/events, view model payload.
- Adapter worker: image generation port and Together adapter.
- Audio worker: narration/ambient audio integration.
- Test worker: focused unit/smoke tests and fixture updates.

At the end of each phase:

1. Run focused tests.
2. Start the app if needed.
3. Validate UI behavior.
4. Stop and collect feedback before proceeding.