docs/immersive_experience_execution_plan.md

Immersive Time-Travel Experience Execution Plan

Date: 2026-06-13

Product Direction

Build a surreal time-travel spectacle first. The user should not feel like they opened a voice chatbot; they should feel like they activated a machine, crossed a portal, landed in another time, and met a person who belongs there.

The core demo sequence:

1. User chooses coordinates and presses Launch.
2. The cockpit begins a dramatic portal sequence.
3. A year counter rewinds or fast-forwards from the current year into the destination year.
4. The portal hides model latency while world, character, and optional narration assets are prepared.
5. The destination fades in slowly: first atmosphere, then world image, then character portrait.
6. A narrator voice introduces the world and character in a cinematic tone.
7. The narrator voice stops. The character voice is clearly different.
8. Live conversation begins inside the revealed world.
9. The user can ask for an artifact/souvenir and receive a visual object from that world.

Hackathon Constraints

• The app remains a Gradio app hosted as a Hugging Face Space.
• The 32B parameter limit is treated as per model used, not cumulative across all models.
• Every model used for core functionality must have an individually defensible parameter count.
• Cloud APIs are allowed unless a specific bonus badge requires local-only operation. We are optimizing for wow and reliability, not Off-the-Grid.
• Current model choices remain valid:
  • LLM: Qwen/Qwen3-8B, 8B.
  • STT: nvidia/nemotron-3.5-asr-streaming-0.6b, 0.6B.
  • TTS: ResembleAI/chatterbox or Chatterbox Turbo family, about 0.5B.
  • Image: black-forest-labs/FLUX.1-schnell, 12B.

Architecture Strategy

Keep the existing ports-and-adapters architecture.

Add immersion as typed events and UI state, not as one-off Gradio hacks:

• ImmersiveScene: generated/fallback scene metadata.
• CharacterPortrait: generated/fallback portrait metadata.
• Narration: text, audio path, narrator voice profile.
• Artifact: visual souvenir image plus markdown description.
• ImmersiveEvent: event emitted during launch/reveal.

The backend remains responsible for encounter state. The frontend owns visual animation and audio mixing.

Phase 1: Portal Shell And State Plumbing

Goal: Make launch feel dramatic before any new model integration.

Build:

• Portal stage in cockpit.html.
• Year-travel counter that animates from current year to destination year.
• Dramatic launch states: dormant, launching, traversing, resolving, revealing, conversation ready.
• World reveal container with fallback gradient/texture.
• Character portrait aperture with fallback silhouette.
• Narration caption slot.
• JS function to ingest a hidden JSON UI payload and update the portal.
• Preserve existing live voice controls and transcript.

No external APIs. No image generation. No narration audio yet.

Validation:

• User clicks Launch and sees a cinematic portal sequence.
• Year counter moves in the correct direction for past/future.
• Destination/persona data appears in the immersive shell.
• Existing voice controls still work.
• The app still runs with fixture profile.

Phase 2: Immersive Backend Contracts

Goal: Add backend contracts that can carry scene, portrait, narration, and artifact data without forcing UI rewrites.

Build:

• Domain models:
  • ImmersiveScene
  • CharacterPortrait
  • Narration
  • VisualArtifact
• Domain events:
  • SceneEvent
  • PortraitEvent
  • NarrationEvent
  • VisualArtifactEvent
• UI render state fields for immersive payload JSON.
• Fixture adapter data for at least two hero encounters.
• Tests for event ordering and serialization.

Validation:

• Fixture encounters populate scene/portrait/artifact placeholders.
• Hidden JSON payload updates after launch.
• No real image/audio APIs are required.

Phase 3: Image Generation With Together FLUX

Goal: Generate the world and character from the destination/persona.

Build:

• ImageGenerationPort.
• Together adapter using black-forest-labs/FLUX.1-schnell.
• Generate scene and portrait in parallel after destination/persona are available.
• Cache base64 images per encounter.
• Fallback to generated CSS visual if image generation fails.
• Add image model entry to config/models.yaml with 12B parameter count.
• Update model-budget validation to check each enabled model individually against 32B.

Validation:

• Scene and portrait appear during or shortly after portal traversal.
• Slow image generation does not block the visual launch sequence.
• Failure leaves the demo usable.

Phase 4: Cinematic Narration

Goal: Add a movie-opening narration that is distinct from the character voice.

Build:

• Narration prompt template:
  • 15-25 seconds max.
  • Describes year, place, atmosphere, and the person being contacted.
  • Uses third-person cinematic narration.
  • Avoids speaking as the character.
• Narrator voice profile:
  • Stable, lower-energy, documentary/cinematic style.
  • Separate from character VoiceProfile.
• Generate narration audio during portal animation.
• Play narration once after world reveal.
• Duck ambient audio under narration.
• Only open live voice after narration finishes or user skips.

Validation:

• Narrator and character voices are obviously different.
• Narration improves immersion without delaying interaction too much.
• Skip/continue control exists if audio generation is slow.

Phase 5: Ambient Sound And Reactive Mixing

Goal: Make the world feel alive before the character speaks.

Build:

• Bundle small CC0 ambient loops:
  • temporal storm / portal hum
  • rain
  • market/crowd murmur
  • fire/hearth
  • wind
  • machinery
  • insects/night
  • water/harbor
• Ambient selector based on destination atmosphere, mode, and visual motifs.
• Web Audio mixer:
  • portal hum during traversal
  • crossfade to destination ambience
  • duck ambience under narration and character speech
  • optional stereo panning/small reverb

Validation:

• Launch has sound even before voice response.
• Ambience does not fight speech.
• Audio remains optional and browser-safe.

Phase 6: Artifact Reveal

Goal: Create a memorable "the character handed me something" moment.

Build:

• Extend souvenir generation into a visual artifact:
  • letter
  • map fragment
  • newspaper clipping
  • receipt
  • sketch
  • warning poster
  • tool tag
• Generate artifact prompt from destination, persona, and conversation.
• Reveal artifact in the portal/world UI with a physical animation.
• Save artifact with encounter trace.

Validation:

• Asking for a souvenir creates a visual object, not just markdown.
• Artifact matches the era and persona.
• Artifact is shareable in screenshots.

Phase 7: Hero Demo Polish

Goal: Make the submission judge-ready.

Build:

• Two golden paths:
  • Surreal past encounter.
  • Strange/future encounter.
• A deterministic demo script:
  • Launch.
  • Listen to narration.
  • Ask what is happening.
  • Ask about one visible object.
  • Ask for a souvenir.
  • Challenge with a modern concept and watch the character react in-world.
• Field Notes doc.
• Shared traces.
• Screenshot/video-friendly layout.
• HF Space deployment check.

Validation:

• Demo works cold.
• Demo video can be recorded in one take.
• The app visibly qualifies for Off-Brand, Field Notes, and Sharing Is Caring.

Deferred Until After Core Wow Works

• Full 3D avatar.
• Wav2Lip/SadTalker/LivePortrait realtime animation.
• Three.js room navigation.
• Gaussian splat viewer.
• WebXR.
• Fully live image generation for every conversation turn.

These are visually attractive but risky. A generated scene, portrait, ambient sound, narration, and artifact reveal provide most of the perceived immersion with much lower demo risk.

Agent Implementation Model

Use agents by phase with disjoint ownership:

• Frontend worker: cockpit.html, cockpit.css, cockpit.js.
• Backend contracts worker: domain models/events, view model payload.
• Adapter worker: image generation port and Together adapter.
• Audio worker: narration/ambient audio integration.
• Test worker: focused unit/smoke tests and fixture updates.

At the end of each phase:

1. Run focused tests.
2. Start the app if needed.
3. Validate UI behavior.
4. Stop and collect feedback before proceeding.