# Immersive AI Time Machine — Research & Implementation Plan ## The Vision Transform the AI Time Machine from a voice-chat-with-text-panels into a **portal experience** — when you click "Launch," the screen dissolves into the destination world, a character appears before you, and you converse face-to-face while the ambient sounds of that era surround you. --- ## Current State Assessment Your app currently renders everything as Gradio markdown panels. The `Destination` model already has rich data (`atmosphere`, `visual_motifs`, `visual_preset_key`) that's **never used visually** — it's just displayed as text. The `Persona` has `speaking_style`, `voice_theory`, etc., but there's no visual character representation. You already have great bones — the steampunk cockpit CSS, the live voice WebSocket channel, the streaming TTS. What's missing is the **sensory layer**. ``` Current: [Text Description] → [Voice Chat] → [Text Transcript] Target: [360° Scene + Ambient Audio] → [Talking Avatar] → [Immersive UI] ``` --- ## User Review Required > [!IMPORTANT] > This plan is organized into **3 tiers** — each tier is independently shippable and adds progressively more immersion. You can ship Tier 1 alone for a massive upgrade, then layer on Tier 2 and 3 later. **Which tier(s) do you want to pursue?** > [!WARNING] > **Hackathon budget considerations**: Some of these tools have free tiers, some don't. I've flagged costs for each. The AI image generation approach (Tier 1) is the most cost-effective since you already have a Together AI API key. 3D avatars (Tier 2+) add monthly API costs. --- ## Tier 1: Visual World & Character (Highest Impact, Lowest Complexity) This tier alone transforms the experience dramatically. No new backend services needed — just AI-generated images + frontend magic. ### 1A. AI-Generated Scene Backgrounds **What**: When a Destination is generated, use an image generation API to create a matching scene that fills the screen. | Tool | How It Works | Pricing | Best For | |------|-------------|---------|----------| | **Blockade Labs Skybox AI** ⭐ | Text → 360° equirectangular panorama + 3D mesh. Up to 8K resolution | ~$20/mo (100 credits, 3-6 per gen) | Full 360° immersion in Three.js | | **Flux 2 (via Together AI)** ⭐ | Text → image via the same Together AI you're already using for LLM | ~$0.02-0.05/image, same API key | **Zero new API signup** | | **Google Imagen 4** | Text → high-quality scene image via Vertex AI | $0.02-0.06/image ($300 GCP trial) | High quality, Google ecosystem | | **Stability AI (SD 3.5)** | Text → scene image. Open-source models available for self-hosting | 25 free credits. ~$0.01-0.08/image | Self-hosting option for unlimited free gen | | **OpenAI GPT Image 2** | Text → image. Exceptional prompt adherence | $0.005-0.21/image | Complex scene descriptions | > [!TIP] > **Recommended: Flux 2 via Together AI** for scene backgrounds. You're already using Together AI for your LLM (`TIME_MACHINE_LLM_API_KEY`). Their image API supports Flux models with the same API key. Zero friction. For maximum wow, **Blockade Labs** gives you true 360° panoramas that users can look around in. **Implementation approach**: - After `generate_destination()`, call image API with a prompt built from `destination.atmosphere` + `destination.place` + `destination.year` - Return image URL/bytes as part of the `DestinationEvent` - Frontend: fade-in the scene as a full-bleed background behind the UI, with a vignette overlay ### 1B. AI-Generated Character Portraits **What**: When a Persona is generated, create a portrait of that character. | Tool | Approach | Key Advantage | |------|----------|---------------| | **Flux 2 Kontext** ⭐ | Generate a portrait from persona description | **Best for character consistency** — can maintain features across multiple generations using reference images | | **Imagen 4** | Generate from persona + era description | Deep Google ecosystem integration | | **Stability AI** | Generate + can self-host for free | Open-source, most flexible | > [!TIP] > **Flux 2 Kontext** is specifically designed for maintaining character consistency across multiple generations — essential if you want the same character to look the same across scene changes or re-generations. **Implementation approach**: - After `generate_persona()`, call image API with a prompt built from persona description + destination era - Display the character portrait in a circular "aperture" frame (you already have the `.tm-cockpit-aperture` CSS!) - During conversation, the portrait is always visible — this is who you're talking to ### 1C. Immersive Frontend Overhaul **What**: Replace the current Gradio markdown panels with a cinematic, full-screen experience. | Effect | How | Impact | |--------|-----|--------| | **Full-bleed scene background** | Scene image as `background-image` with `background-size: cover`, dark vignette overlay | Instant world transportation | | **Cinematic fade-in** | GSAP-powered transition from cockpit → wormhole → destination | Transition magic | | **Parallax depth** | Scene background moves slightly on mouse movement, character portrait floats above | Depth and life | | **Atmospheric particles** | Canvas-based floating particles (embers for fire scenes, snow for winter, dust for ancient ruins) | Living world | | **Film grain + color grading** | CSS `filter` + animated noise overlay with `mix-blend-mode: overlay` (opacity 0.03-0.08) | Era authenticity | | **Era-specific color grading** | Sepia for ancient, blue-tint for future, warm amber for candlelit eras | Visual time-travel | | **Ambient glow pulse** | Character portrait pulses with soft glow when TTS audio is playing | Character feels alive | | **Floating dialogue text** | Conversation text fades in/out near the character, RPG-dialogue style | Immersive dialogue | > [!TIP] > **GSAP (GreenSock Animation Platform) is now 100% FREE** — including commercial use — after its Webflow acquisition. It's the industry gold standard for web animation timing and sequencing. Perfect for cinematic launch transitions. ### 1D. Ambient Soundscapes **What**: Background audio that matches the destination (marketplace chatter, ocean waves, wind, etc.) | Tool | How | Pricing | Key Feature | |------|-----|---------|-------------| | **ElevenLabs Sound Effects API** ⭐ | Text → sound effect. "Ancient Roman marketplace at dusk" → audio | 10K free credits/mo (non-commercial) | **Spatial awareness** — understands "cathedral echo" vs "open field". Supports `seamless_loop` parameter | | **Freesound.org API** | Search and stream CC-licensed ambient sounds | Free (API key required) | Massive library of real recordings | | **Pre-built ambient library** | Bundle 10-15 ambient loops, select by `visual_preset_key` | Free, zero latency | Instant, no API call needed | | **Web Audio API** | Procedural ambient + reverb effects | Free, built into browser | `ConvolverNode` with impulse responses for environment-matched reverb (cathedral IR, cave IR, etc.) | > [!TIP] > **Recommended hybrid approach**: Use **ElevenLabs SFX API** to generate destination-specific ambient audio on launch (it understands spatial context!), cache the results, and enhance with **Web Audio API** reverb/effects. Fall back to a **pre-built ambient library** for instant audio while the AI-generated soundscape loads. --- ## Tier 2: Animated Talking Character (Medium Complexity) This tier adds a character that **moves and speaks** — their lips sync to the audio, they have facial expressions. ### 2A. Real-Time Talking Avatar APIs (Cloud) | Tool | What It Does | Pricing | Latency | Integration | |------|-------------|---------|---------|-------------| | **Simli** ⭐ | Image → animated talking head with lip sync via WebRTC. "Trinity" facial model with full expressions | $10 free credits + 50 free min/mo. Then ~$0.05/min | Sub-300ms | JS SDK, WebRTC. **Best real-time option** | | **D-ID Agents** | Image → real-time avatar via WebRTC. Parallelized diffusion pipeline | 14-day trial (3 min, watermarked). From ~$4.70/mo | Sub-200ms, up to 100 FPS | Agents SDK with pre-built widget or full SDK | | **HeyGen LiveAvatar** | Highest visual fidelity. "Avatar V" with fluid micro-expressions | Pay-as-you-go API credits (starts $5) | Low-latency streaming | WebRTC-based LiveAvatar API | > [!IMPORTANT] > **Recommended: Simli** for real-time lip-synced avatars. It's (a) affordable with a real free tier, (b) truly real-time via WebRTC at sub-300ms, (c) takes a static image as input (feed it the AI-generated portrait from Tier 1), and (d) has a "Trinity" facial model with full expressions, not just basic lip movement. Your existing WebSocket architecture maps directly to this. **How Simli integrates with your architecture**: ``` Your TTS audio → Simli WebRTC API → Lip-synced video frames →