docs/immersive_experience_hackathon_plan.md

Immersive AI Time Machine — Hackathon-Aligned Plan

This document filters the full immersive experience research through the lens of the Build Small Hackathon constraints and judging criteria, identifying what to build, what to skip, and what earns bonus points.

---

Hackathon Constraints Summary

Constraint	Requirement	Impact on Plan
Small models	≤ 32B parameters per model	✅ No blocker. Image gen models (Flux ~12B, SDXL ~6.6B) are well under. LLM (Qwen3-8B), STT (Nemotron 0.6B), TTS (Chatterbox) all individually under 32B
Must be Gradio	App must be a Gradio application	⚠️ Cannot switch to a custom frontend. Must work within Gradio's gr.HTML(), gr.Image(), gr.Audio() injection approach. This limits Tier 2/3 complexity
Hugging Face Space	Must be deployable as an HF Space	✅ Already compatible. Need to ensure any new assets (ambient audio, GSAP CDN) work in HF Space environment
Working product	Functional, polished, usable by non-technical users	✅ Focus on reliable, delightful experience over bleeding-edge tech

---

Track Alignment: Track 2 — An Adventure in Thousand Token Wood ⭐

The AI Time Machine is a natural fit for Track 2. Here's how each judging criterion maps:

Delight ✅

"Genuinely fun, memorable experiences, worth sharing"

Criterion	How We Score	What to Build
Genuinely fun	Transporting to another era and talking to someone from that time is inherently fun	AI-generated scene + character portrait make it visually fun, not just text-fun
Memorable	The "Launch" moment — screen transforms from cockpit to destination — is the memory	Cinematic fade-in transition, ambient sounds starting, character appearing
Worth sharing	"Look at this person I met in 1920s jazz-age Harlem" + screenshot/souvenir	The generated scene + portrait + souvenir are inherently shareable

AI Is Essential ✅

"Require AI to function, AI as core mechanic, not bolted on"

AI Component	Role	Essential?
LLM (Qwen3-8B)	Generates destinations, personas, conversations, souvenirs	✅ Core — the entire experience is AI-generated
TTS (Chatterbox)	Character speaks with a unique voice	✅ Core — voice is how you "meet" the person
STT (Nemotron)	You speak to the character naturally	✅ Core — enables voice conversation
Image Gen (Flux/Imagen)	Generates scene + character portrait	✅ Core — this is the new "wow" layer that makes AI create the entire world, not just the text

Adding AI-generated visuals actually strengthens the AI-essential score — now AI creates the entire sensory experience: the world you see, the person you see, the voice you hear, and the conversation you have. Every sense is AI-generated.

Originality ✅

"Unusual ideas, novel interactions, avoid common chatbot wrappers"

What Makes It Original	Why
Time travel as interaction paradigm	Not a chatbot — it's a portal to another era
Ordinary people, not famous figures	You don't talk to "Einstein" — you talk to a baker in 1920s Paris
Multi-modal AI synthesis	Scene + portrait + voice + conversation all generated together from one destination
Souvenir system	You bring back an artifact from your trip — a tangible memento of an AI experience

Gradio App Polish ✅

"Feel complete, thoughtful UX, cohesive experience"

This is where the immersive upgrades pay off most for judging:

Current State	After Tier 1	Judge Impact
Text-only destination description	Full-screen AI-generated scene background	Night and day difference
No character visual	AI-generated character portrait in aperture	"You're actually meeting someone"
Silent between voice clips	Ambient soundscape matching the era	Cohesive world-building
Standard Gradio form layout	Steampunk cockpit with particles, film grain, glow effects	Off-Brand bonus badge

---

Bonus Merit Badges — What We Can Earn

✅ Off-Brand (Custom UI) — Already Partially Earned, Tier 1 Maxes It Out

"Go beyond default Gradio styling. Custom layouts, advanced theming, branded UI systems, rich interaction patterns."

We already have custom CSS (steampunk cockpit, brass theme, custom fonts). Tier 1 additions (scene backgrounds, particles, film grain, cinematic transitions) would make this an exceptional Off-Brand submission.

✅ Sharing Is Caring — Already Earned

"Publish and share agent traces."

The app already has JsonlTraceSink that records all events. Just need to expose/publish the traces.

✅ Field Notes — Easy to Earn

"Document discoveries, experiments, lessons learned."

This research document and the implementation decisions are themselves field notes. Write up a blog-style doc about the build process.

❌ Off the Grid — Cannot Earn (We Use Cloud APIs)

Requires no cloud AI APIs. We use Together AI for LLM and Modal for STT/TTS.

❓ Well-Tuned — Possible But Not Planned

Would require publishing a fine-tuned model on HF. Not in scope for immersive upgrades.

❓ Llama Champion — Not Currently Applicable

Would require switching to llama.cpp runtime. Not relevant to visual immersion work.

---

What to Build (Hackathon-Optimized)

✅ BUILD — Tier 1 Features (All Hackathon-Compatible)

Everything in Tier 1 works within Gradio and strengthens hackathon scores:

Feature	Hackathon Value	Implementation Constraint
AI scene background (Flux via Together AI)	Massive visual wow, strengthens AI-essential + delight	Use gr.Image() or base64 in gr.HTML(). Per-model ≤32B: Flux is ~12B ✅
AI character portrait (Flux via Together AI)	Character feels real, strengthens originality	Same approach as scene. Displayed in existing aperture
Cinematic launch transition (CSS/JS)	Memorable "Launch" moment, strengthens delight + polish	Pure CSS/JS in gr.HTML() injection. No dependencies
Atmospheric particles (Canvas JS)	Living world feel, strengthens Off-Brand badge	Canvas element in gr.HTML(). Lightweight
Film grain + era color grading (CSS)	Era authenticity, strengthens polish	CSS filters + overlay in existing cockpit.css
Ambient soundscapes (pre-built library)	Immersive audio, strengthens delight	<audio> element in gr.HTML(), bundled CC0 loops
Portrait glow pulse on speech (CSS/JS)	Character feels alive, strengthens delight	CSS animation triggered by audio events in cockpit.js
GSAP transitions (CDN)	Professional animation quality	Load from CDN in Gradio head. Free library

⚠️ DEFER — Tier 2 Features (Risky for Hackathon)

Feature	Why Defer
Simli real-time avatar	Adds external API dependency, WebRTC complexity, may break in HF Space sandboxing. Risk vs. reward not worth it for hackathon
TalkingHead 3D avatar	Requires loading Three.js + GLB models inside Gradio — fragile. Heavy JS that may conflict with Gradio's DOM management
Wawa-Lipsync	Needs npm package bundling or CDN load inside Gradio. Adds complexity without guaranteed stability

The "audio-reactive portrait" approach from Tier 1 (glow pulse + breathing animation) gives 80% of the "alive character" feeling at 5% of the Tier 2 complexity. This is the right trade-off for a hackathon.

❌ SKIP — Tier 3 Features (Too Complex for Hackathon)

Feature	Why Skip
360° Three.js panorama	Heavy 3D rendering inside Gradio is fragile. HF Space may have GPU/memory constraints for client-side WebGL
WebXR / VR	Niche audience, adds complexity, doesn't help judging
3D character models	MetaPerson API + Three.js rendering in Gradio is too brittle

---

Hackathon-Optimized Implementation Plan

Phase 1: Image Generation (Backend) — ~4 hours

1. New port: SceneGenerator interface with generate_scene() and generate_portrait() methods
2. New adapter: Together AI Flux implementation (reuse existing API key, ≤32B per model ✅)
3. Modify encounter_service: Call image gen after destination + persona generation. Run both image calls in parallel using existing ThreadPoolExecutor pattern
4. New events: SceneImageEvent, PortraitImageEvent carrying base64 image data
5. Model updates: Add image data fields to EncounterSession

Phase 2: Immersive Frontend (CSS/JS/HTML) — ~6 hours

1. Scene background: Full-bleed background image container behind cockpit, with vignette radial-gradient overlay
2. Character portrait: Circular portrait in the existing .tm-cockpit-aperture, with idle breathing CSS animation
3. Launch transition: CSS keyframe animation — cockpit dims → radial wipe/dissolve → scene fades in
4. Particles: Lightweight canvas-based particle system. Map visual_preset_key to particle type (embers/snow/dust/fireflies)
5. Film grain: SVG noise overlay with mix-blend-mode: overlay, opacity 0.04
6. Era color grading: CSS filter on scene container (sepia for old, blue-shift for future, warm amber for candlelit)
7. Audio-reactive portrait: Analyze TTS audio amplitude → pulse CSS glow intensity on portrait border
8. Floating dialogue: Conversation text near character portrait, fade-in/out with CSS transitions

Phase 3: Ambient Audio — ~2 hours

1. Bundle 8-10 CC0 ambient loops: marketplace, ocean, forest, desert, rain, fire, wind, nighttime-crickets
2. Keyword matcher: Map destination atmosphere/place text to best ambient loop
3. Audio element: <audio> in cockpit HTML, controlled by cockpit.js, crossfade on destination change
4. Web Audio reverb: Optional ConvolverNode for environment-matched reverb (indoor vs outdoor)

Phase 4: Gradio Integration — ~2 hours

1. Wire images into view_models: Handle SceneImageEvent and PortraitImageEvent, pass base64 to frontend
2. Update cockpit.js: Listen for Gradio state changes containing image data, update DOM
3. Update realtime.py: Forward image events over WebSocket for live voice mode
4. Update container.py: Wire SceneGenerator adapter into all profiles (fixture + modal + dev)

Phase 5: Polish & Test — ~2 hours

1. Test across coordinate modes (past, future, strange, surprise)
2. Test mobile viewport responsiveness
3. Add prefers-reduced-motion for accessibility
4. Verify HF Space deployment compatibility
5. Run existing test suite (pytest)

Total estimated effort: ~16 hours

---

Model Budget Check (Per-Model ≤ 32B)

Role	Model	Parameters	Under 32B?
LLM	Qwen/Qwen3-8B	8B	✅
STT	nvidia/nemotron-3.5-asr-streaming-0.6b	0.6B	✅
TTS	ResembleAI/chatterbox-turbo	~0.3B	✅
Image Gen	Flux 2 (Schnell)	~12B	✅

All models individually well under the 32B per-model limit.

---

Judging Score Projection

Criterion	Before Immersive	After Tier 1	Why
Delight	6/10 — fun concept, but text-heavy	9/10 — visual + audio immersion, cinematic launch	Scene + portrait + ambient audio = multi-sensory delight
AI Essential	8/10 — LLM + STT + TTS are core	10/10 — AI generates everything: world, character, voice, conversation	Adding image gen makes AI create the complete experience
Originality	8/10 — unique concept	9/10 — now a full "portal" experience	Not just a chatbot — it's a time travel simulator
Polish	6/10 — functional but text-panel UI	9/10 — cinematic, branded, immersive	Particles, film grain, transitions, ambient audio
Off-Brand Badge	7/10 — custom CSS theme	10/10 — completely custom visual experience	Barely recognizable as Gradio

---

Risk Assessment

Risk	Likelihood	Mitigation
Image gen API slow (>5s)	Medium	Generate scene + portrait in parallel. Show "Temporal engines charging" animation while loading
Image gen API fails	Low	Graceful fallback — app works without images (current behavior). Show cockpit background as fallback
Ambient audio files too large for HF Space	Low	Compress to 64kbps OGG, ~30KB per 30-second loop. Total ~300KB for 10 loops
GSAP CDN blocked in HF Space	Low	Fall back to CSS keyframe animations (slightly less smooth, but functional)
Gradio DOM conflicts with injected HTML	Medium	Test thoroughly. Use isolated DOM containers with unique IDs. Avoid Gradio-managed elements

---

Files to Create/Modify

New Files

• src/time_machine/ports/image_generation.py — Port interface
• src/time_machine/adapters/image_gen/ — Together AI Flux adapter
• src/time_machine/ui/assets/ambience/ — Bundled ambient audio loops (CC0)

Modified Files

• src/time_machine/application/encounter_service.py — Image gen calls after dest/persona
• src/time_machine/domain/models.py — Image data on session
• src/time_machine/domain/events.py — New image event types
• src/time_machine/application/container.py — Wire image gen adapter
• src/time_machine/ui/view_models.py — Handle image events
• src/time_machine/ui/realtime.py — Forward image events over WebSocket
• src/time_machine/ui/assets/cockpit.html — Scene/portrait/particle/audio containers
• src/time_machine/ui/assets/cockpit.css — Immersive visual styles
• src/time_machine/ui/assets/cockpit.js — Scene loading, particles, audio-reactive portrait, ambient audio