docs/immersive_experience_hackathon_plan.md

# Immersive AI Time Machine — Hackathon-Aligned Plan

This document filters the [full immersive experience research](immersive_experience_research.md) through the lens of the **Build Small Hackathon** constraints and judging criteria, identifying what to build, what to skip, and what earns bonus points.

---

## Hackathon Constraints Summary

| Constraint | Requirement | Impact on Plan |
|------------|-------------|----------------|
| **Small models** | ≤ 32B parameters **per model** | ✅ No blocker. Image gen models (Flux ~12B, SDXL ~6.6B) are well under. LLM (Qwen3-8B), STT (Nemotron 0.6B), TTS (Chatterbox) all individually under 32B |
| **Must be Gradio** | App must be a Gradio application | ⚠️ **Cannot switch to a custom frontend.** Must work within Gradio's `gr.HTML()`, `gr.Image()`, `gr.Audio()` injection approach. This limits Tier 2/3 complexity |
| **Hugging Face Space** | Must be deployable as an HF Space | ✅ Already compatible. Need to ensure any new assets (ambient audio, GSAP CDN) work in HF Space environment |
| **Working product** | Functional, polished, usable by non-technical users | ✅ Focus on reliable, delightful experience over bleeding-edge tech |

---

## Track Alignment: **Track 2 — An Adventure in Thousand Token Wood** ⭐

The AI Time Machine is a natural fit for Track 2. Here's how each judging criterion maps:

### Delight ✅
> *"Genuinely fun, memorable experiences, worth sharing"*

| Criterion | How We Score | What to Build |
|-----------|-------------|---------------|
| Genuinely fun | Transporting to another era and talking to someone from that time is inherently fun | AI-generated scene + character portrait make it *visually* fun, not just text-fun |
| Memorable | The "Launch" moment — screen transforms from cockpit to destination — is the memory | Cinematic fade-in transition, ambient sounds starting, character appearing |
| Worth sharing | "Look at this person I met in 1920s jazz-age Harlem" + screenshot/souvenir | The generated scene + portrait + souvenir are inherently shareable |

### AI Is Essential ✅
> *"Require AI to function, AI as core mechanic, not bolted on"*

| AI Component | Role | Essential? |
|-------------|------|-----------|
| LLM (Qwen3-8B) | Generates destinations, personas, conversations, souvenirs | ✅ Core — the entire experience is AI-generated |
| TTS (Chatterbox) | Character speaks with a unique voice | ✅ Core — voice is how you "meet" the person |
| STT (Nemotron) | You speak to the character naturally | ✅ Core — enables voice conversation |
| **Image Gen (Flux/Imagen)** | Generates scene + character portrait | ✅ Core — **this is the new "wow" layer** that makes AI create the entire world, not just the text |

> [!TIP]
> Adding AI-generated visuals actually **strengthens the AI-essential score** — now AI creates the *entire sensory experience*: the world you see, the person you see, the voice you hear, and the conversation you have. Every sense is AI-generated.

### Originality ✅
> *"Unusual ideas, novel interactions, avoid common chatbot wrappers"*

| What Makes It Original | Why |
|----------------------|-----|
| Time travel as interaction paradigm | Not a chatbot — it's a **portal** to another era |
| Ordinary people, not famous figures | You don't talk to "Einstein" — you talk to a baker in 1920s Paris |
| Multi-modal AI synthesis | Scene + portrait + voice + conversation all generated together from one destination |
| Souvenir system | You bring back an artifact from your trip — a tangible memento of an AI experience |

### Gradio App Polish ✅
> *"Feel complete, thoughtful UX, cohesive experience"*

This is where the immersive upgrades pay off most for judging:

| Current State | After Tier 1 | Judge Impact |
|---------------|-------------|-------------|
| Text-only destination description | Full-screen AI-generated scene background | Night and day difference |
| No character visual | AI-generated character portrait in aperture | "You're actually meeting someone" |
| Silent between voice clips | Ambient soundscape matching the era | Cohesive world-building |
| Standard Gradio form layout | Steampunk cockpit with particles, film grain, glow effects | **Off-Brand bonus badge** |

---

## Bonus Merit Badges — What We Can Earn

### ✅ Off-Brand (Custom UI) — **Already Partially Earned, Tier 1 Maxes It Out**
> *"Go beyond default Gradio styling. Custom layouts, advanced theming, branded UI systems, rich interaction patterns."*

We already have custom CSS (steampunk cockpit, brass theme, custom fonts). Tier 1 additions (scene backgrounds, particles, film grain, cinematic transitions) would make this an **exceptional** Off-Brand submission.

### ✅ Sharing Is Caring — **Already Earned**
> *"Publish and share agent traces."*

The app already has `JsonlTraceSink` that records all events. Just need to expose/publish the traces.

### ✅ Field Notes — **Easy to Earn**
> *"Document discoveries, experiments, lessons learned."*

This research document and the implementation decisions are themselves field notes. Write up a blog-style doc about the build process.

### ❌ Off the Grid — **Cannot Earn (We Use Cloud APIs)**
Requires no cloud AI APIs. We use Together AI for LLM and Modal for STT/TTS.

### ❓ Well-Tuned — **Possible But Not Planned**
Would require publishing a fine-tuned model on HF. Not in scope for immersive upgrades.

### ❓ Llama Champion — **Not Currently Applicable**
Would require switching to llama.cpp runtime. Not relevant to visual immersion work.

---

## What to Build (Hackathon-Optimized)

### ✅ BUILD — Tier 1 Features (All Hackathon-Compatible)

Everything in Tier 1 works within Gradio and strengthens hackathon scores:

| Feature | Hackathon Value | Implementation Constraint |
|---------|----------------|--------------------------|
| **AI scene background** (Flux via Together AI) | Massive visual wow, strengthens AI-essential + delight | Use `gr.Image()` or base64 in `gr.HTML()`. Per-model ≤32B: Flux is ~12B ✅ |
| **AI character portrait** (Flux via Together AI) | Character feels real, strengthens originality | Same approach as scene. Displayed in existing aperture |
| **Cinematic launch transition** (CSS/JS) | Memorable "Launch" moment, strengthens delight + polish | Pure CSS/JS in `gr.HTML()` injection. No dependencies |
| **Atmospheric particles** (Canvas JS) | Living world feel, strengthens Off-Brand badge | Canvas element in `gr.HTML()`. Lightweight |
| **Film grain + era color grading** (CSS) | Era authenticity, strengthens polish | CSS filters + overlay in existing cockpit.css |
| **Ambient soundscapes** (pre-built library) | Immersive audio, strengthens delight | `<audio>` element in `gr.HTML()`, bundled CC0 loops |
| **Portrait glow pulse on speech** (CSS/JS) | Character feels alive, strengthens delight | CSS animation triggered by audio events in cockpit.js |
| **GSAP transitions** (CDN) | Professional animation quality | Load from CDN in Gradio `head`. Free library |

### ⚠️ DEFER — Tier 2 Features (Risky for Hackathon)

| Feature | Why Defer |
|---------|----------|
| **Simli real-time avatar** | Adds external API dependency, WebRTC complexity, may break in HF Space sandboxing. Risk vs. reward not worth it for hackathon |
| **TalkingHead 3D avatar** | Requires loading Three.js + GLB models inside Gradio — fragile. Heavy JS that may conflict with Gradio's DOM management |
| **Wawa-Lipsync** | Needs npm package bundling or CDN load inside Gradio. Adds complexity without guaranteed stability |

> [!NOTE]
> **The "audio-reactive portrait" approach from Tier 1 (glow pulse + breathing animation) gives 80% of the "alive character" feeling at 5% of the Tier 2 complexity.** This is the right trade-off for a hackathon.

### ❌ SKIP — Tier 3 Features (Too Complex for Hackathon)

| Feature | Why Skip |
|---------|---------|
| **360° Three.js panorama** | Heavy 3D rendering inside Gradio is fragile. HF Space may have GPU/memory constraints for client-side WebGL |
| **WebXR / VR** | Niche audience, adds complexity, doesn't help judging |
| **3D character models** | MetaPerson API + Three.js rendering in Gradio is too brittle |

---

## Hackathon-Optimized Implementation Plan

### Phase 1: Image Generation (Backend) — ~4 hours

1. **New port**: `SceneGenerator` interface with `generate_scene()` and `generate_portrait()` methods
2. **New adapter**: Together AI Flux implementation (reuse existing API key, ≤32B per model ✅)
3. **Modify encounter_service**: Call image gen after destination + persona generation. Run both image calls in parallel using existing `ThreadPoolExecutor` pattern
4. **New events**: `SceneImageEvent`, `PortraitImageEvent` carrying base64 image data
5. **Model updates**: Add image data fields to `EncounterSession`

### Phase 2: Immersive Frontend (CSS/JS/HTML) — ~6 hours

1. **Scene background**: Full-bleed background image container behind cockpit, with vignette radial-gradient overlay
2. **Character portrait**: Circular portrait in the existing `.tm-cockpit-aperture`, with idle breathing CSS animation
3. **Launch transition**: CSS keyframe animation — cockpit dims → radial wipe/dissolve → scene fades in
4. **Particles**: Lightweight canvas-based particle system. Map `visual_preset_key` to particle type (embers/snow/dust/fireflies)
5. **Film grain**: SVG noise overlay with `mix-blend-mode: overlay`, opacity 0.04
6. **Era color grading**: CSS `filter` on scene container (sepia for old, blue-shift for future, warm amber for candlelit)
7. **Audio-reactive portrait**: Analyze TTS audio amplitude → pulse CSS glow intensity on portrait border
8. **Floating dialogue**: Conversation text near character portrait, fade-in/out with CSS transitions

### Phase 3: Ambient Audio — ~2 hours

1. **Bundle 8-10 CC0 ambient loops**: marketplace, ocean, forest, desert, rain, fire, wind, nighttime-crickets
2. **Keyword matcher**: Map destination atmosphere/place text to best ambient loop
3. **Audio element**: `<audio>` in cockpit HTML, controlled by cockpit.js, crossfade on destination change
4. **Web Audio reverb**: Optional `ConvolverNode` for environment-matched reverb (indoor vs outdoor)

### Phase 4: Gradio Integration — ~2 hours

1. **Wire images into view_models**: Handle `SceneImageEvent` and `PortraitImageEvent`, pass base64 to frontend
2. **Update cockpit.js**: Listen for Gradio state changes containing image data, update DOM
3. **Update realtime.py**: Forward image events over WebSocket for live voice mode
4. **Update container.py**: Wire `SceneGenerator` adapter into all profiles (fixture + modal + dev)

### Phase 5: Polish & Test — ~2 hours

1. Test across coordinate modes (past, future, strange, surprise)
2. Test mobile viewport responsiveness
3. Add `prefers-reduced-motion` for accessibility
4. Verify HF Space deployment compatibility
5. Run existing test suite (`pytest`)

**Total estimated effort: ~16 hours**

---

## Model Budget Check (Per-Model ≤ 32B)

| Role | Model | Parameters | Under 32B? |
|------|-------|-----------|-----------|
| LLM | Qwen/Qwen3-8B | 8B | ✅ |
| STT | nvidia/nemotron-3.5-asr-streaming-0.6b | 0.6B | ✅ |
| TTS | ResembleAI/chatterbox-turbo | ~0.3B | ✅ |
| **Image Gen** | **Flux 2 (Schnell)** | **~12B** | **✅** |

All models individually well under the 32B per-model limit.

---

## Judging Score Projection

| Criterion | Before Immersive | After Tier 1 | Why |
|-----------|-----------------|-------------|-----|
| **Delight** | 6/10 — fun concept, but text-heavy | **9/10** — visual + audio immersion, cinematic launch | Scene + portrait + ambient audio = multi-sensory delight |
| **AI Essential** | 8/10 — LLM + STT + TTS are core | **10/10** — AI generates *everything*: world, character, voice, conversation | Adding image gen makes AI create the complete experience |
| **Originality** | 8/10 — unique concept | **9/10** — now a full "portal" experience | Not just a chatbot — it's a time travel simulator |
| **Polish** | 6/10 — functional but text-panel UI | **9/10** — cinematic, branded, immersive | Particles, film grain, transitions, ambient audio |
| **Off-Brand Badge** | 7/10 — custom CSS theme | **10/10** — completely custom visual experience | Barely recognizable as Gradio |

---

## Risk Assessment

| Risk | Likelihood | Mitigation |
|------|-----------|-----------|
| Image gen API slow (>5s) | Medium | Generate scene + portrait in parallel. Show "Temporal engines charging" animation while loading |
| Image gen API fails | Low | Graceful fallback — app works without images (current behavior). Show cockpit background as fallback |
| Ambient audio files too large for HF Space | Low | Compress to 64kbps OGG, ~30KB per 30-second loop. Total ~300KB for 10 loops |
| GSAP CDN blocked in HF Space | Low | Fall back to CSS keyframe animations (slightly less smooth, but functional) |
| Gradio DOM conflicts with injected HTML | Medium | Test thoroughly. Use isolated DOM containers with unique IDs. Avoid Gradio-managed elements |

---

## Files to Create/Modify

### New Files
- `src/time_machine/ports/image_generation.py` — Port interface
- `src/time_machine/adapters/image_gen/` — Together AI Flux adapter
- `src/time_machine/ui/assets/ambience/` — Bundled ambient audio loops (CC0)

### Modified Files
- `src/time_machine/application/encounter_service.py` — Image gen calls after dest/persona
- `src/time_machine/domain/models.py` — Image data on session
- `src/time_machine/domain/events.py` — New image event types
- `src/time_machine/application/container.py` — Wire image gen adapter
- `src/time_machine/ui/view_models.py` — Handle image events
- `src/time_machine/ui/realtime.py` — Forward image events over WebSocket
- `src/time_machine/ui/assets/cockpit.html` — Scene/portrait/particle/audio containers
- `src/time_machine/ui/assets/cockpit.css` — Immersive visual styles
- `src/time_machine/ui/assets/cockpit.js` — Scene loading, particles, audio-reactive portrait, ambient audio