docs/tech_stack_decision.md

AI Time Machine Tech Stack Decision

Date: 2026-06-05 (Updated: 2026-06-09)

Purpose

This document records the current technical direction for the AI Time Machine hackathon project. It is intended to be handoff-ready for another coding agent.

The project is a surreal, voice-first time machine experience for the Hugging Face Build Small Hackathon, Track 2: An Adventure in Thousand Token Wood. The user should feel like they launch a strange laboratory machine, arrive at an impossible coordinate in the past or future, and speak aloud with an ordinary person from that world.

Primary Decision

Build a polished Gradio app hosted on a Hugging Face Space. Use a modular voice-agent architecture with custom frontend staging, streaming ASR, a small instruction LLM, character-oriented TTS, and a souvenir generator.

Main priority: the most polished recorded demo possible within hackathon constraints.

Secondary priority: keep the architecture modular so higher-risk voice/avatar experiments can be tried without endangering the MVP.

Hard Constraints

Hackathon constraints:

• The app must be a Gradio application.
• The submitted app must be hosted as a Hugging Face Space.
• Total AI model parameters must stay at or below 32B.
• Core functionality must use small models naturally suited to the task.
• The result must be a working, demonstrable product.

Product constraints:

• Voice-first experience.
• Default flow is Surprise Me.
• Visual theme starts as strange laboratory.
• The theme must be easy to reskin later.
• The project should connect the user to ordinary people, not famous figures.
• The app should prioritize theatrical believability over strict historical accuracy.

Budget context:

• Hugging Face credit: USD 20.
• Modal.com credit: USD 250.
• We do not need to spend the credits, but we should use them if they make the demo materially better.

Final MVP Stack

MVP target:

• App: Gradio Blocks.
• Host: Hugging Face Space with GPU.
• Language: Python.
• Frontend: custom HTML/CSS/JavaScript embedded in Gradio.
• LLM: Qwen3-8B via Together AI API (structured JSON mode built-in).
• STT: NVIDIA Nemotron 3.5 ASR Streaming 0.6B.
• TTS primary candidate: Qwen3-TTS 1.7B VoiceDesign/CustomVoice.
• TTS fallback: NVIDIA Magpie TTS Multilingual 357M.
• Emergency TTS fallback: Kokoro 82M.
• State/output style: structured JSON between generation steps.

Approximate MVP parameter budget:

• LLM: 4B to 8B.
• STT: 0.6B.
• TTS primary: 1.7B.
• Total: roughly 6.3B to 10.3B.

This is comfortably under the 32B cap and leaves room for experimentation.

Hosting Strategy

The required Gradio application lives on Hugging Face Spaces. Inference is split across three services to keep dependencies clean and costs manageable.

Decided architecture:

• HF Spaces: Gradio UI (CPU tier is sufficient — no local model inference).
• Together AI: LLM inference (Qwen3-8B with JSON mode).
• Modal: Audio models only (Nemotron STT, Qwen3-TTS).

Budget:

• $250 Modal credits available for audio GPU compute.
• $5 Together AI free credits for dev testing.

Key insight: separating LLM and audio dependencies avoids Python dependency conflicts that arise from bundling everything in one environment.

Tradeoff:

• Calling external services means the app is not local-only and should not pursue the Off the Grid badge.
• This is acceptable because polish matters more than bonus badges for this submission.

All external services remain behind narrow Python interfaces so they can be swapped.

Application Architecture

The app should be organized as five replaceable layers.

1. Cockpit UI

Technology:

• Gradio Blocks.
• Custom HTML/CSS/JavaScript.
• CSS variables for theme tokens.
• Small JavaScript state machine for visual transitions.

Responsibilities:

• Strange laboratory cockpit.
• Large windshield/portal.
• Launch button.
• Status/signal panel.
• Voice controls.
• Transcript/radio panel.
• Souvenir display.
• Animated states: dormant, launch, tunnel, turbulence, landing, smoke clear, destination reveal, signal lock, conversation, souvenir.

Implementation guidance:

• The strange laboratory look should be a skin, not hard-coded into app logic.
• Use CSS variables for color, glow, material, portal palette, warning states, and typography.
• Use data-state attributes or equivalent simple state flags for animations.
• Do not make the default UI look like a basic Gradio form.

2. World And Persona Engine

Technology:

• Python orchestration.
• Qwen3-class instruction LLM.
• Structured JSON outputs.

Responsibilities:

• Generate a destination profile.
• Generate visual motifs that map to frontend presets.
• Generate an ordinary-person persona card.
• Maintain tone, era/future context, character constraints, and safety.

Important output fields:

• destination year.
• destination place.
• destination mode: past, future, or strange.
• atmosphere.
• visual preset key.
• visual motif list.
• character name or local identifier.
• character role/occupation.
• immediate situation.
• daily concern.
• secret/fear/desire.
• worldview constraints.
• theory about the user's voice.
• speaking style.

3. Voice Input

Primary STT:

• NVIDIA Nemotron 3.5 ASR Streaming 0.6B.

Why:

• 600M parameters.
• Designed for streaming ASR.
• Supports multilingual transcription across many language-locales.
• Supports punctuation and capitalization.
• Has configurable chunk sizes from low-latency to higher-accuracy operation.
• Better fit than clip-only ASR for a voice-first cockpit experience.

Recommended MVP behavior:

• Start with microphone clip transcription if Gradio streaming wiring takes too long.
• Upgrade to streaming ASR once the basic loop works.
• Use streaming partials in the UI as signal decoding text when available.

Fallback STT:

• NVIDIA Nemotron Speech Streaming English 0.6B for English-only simplicity.
• Distil-Whisper if NeMo integration blocks progress.

4. Conversation And Souvenir Engine

Technology:

• Same Qwen3-class instruction LLM used for world/persona generation.

Responsibilities:

• Respond in character.
• Keep replies short enough for voice playback.
• Preserve the persona's worldview and misunderstanding of the time signal.
• Ask occasional questions back.
• Generate a final temporal souvenir.

Conversation prompt rules:

• The character is an ordinary person.
• The character should not know modern facts unless implied by the destination.
• The character may interpret the user as a spirit, official, dream, machine voice, ancestor, descendant, customer, omen, or strange weather.
• The response should contain sensory detail and personality.
• Avoid turning real suffering into spectacle.
• Prefer vivid, humane, surprising moments over encyclopedia-style history.

Souvenir output:

• Destination.
• Contact.
• Quote.
• Artifact.
• Stamp name.
• Short encounter summary.

5. Voice Output

Primary TTS candidate:

• Qwen3-TTS 1.7B VoiceDesign/CustomVoice.

Why:

• Best fit for theatrical character variety.
• Supports natural-language voice control.
• Supports emotion, prosody, timbre, and speaking-style instructions.
• Supports streaming generation.
• Supports custom voice and voice-design workflows.
• Apache 2.0 license.
• Lets the app create a more distinct voice per generated persona.

Recommended Qwen3-TTS workflow:

1. Generate persona.
2. Generate a short voice design instruction from the persona.
3. Use Qwen3-TTS VoiceDesign or CustomVoice to create the character voice.
4. Cache/reuse that voice setup for the encounter.
5. Generate each character line with short text and explicit emotion/prosody hints.

Fallback TTS:

• NVIDIA Magpie TTS Multilingual 357M.

Why:

• NVIDIA-backed.
• NeMo-compatible.
• Small.
• Commercial-ready.
• Multiple speaker options and multilingual support.
• Good reliability candidate if Qwen3-TTS integration is unstable.

Emergency fallback:

• Kokoro 82M.

Why:

• Tiny.
• Apache 2.0.
• Fast and inexpensive.
• Good enough to keep the demo working if stronger TTS options fail.

Do not make TTS a single hard dependency. Put it behind a small interface, for example:

• synthesize(text, voice_profile) -> audio_path
• prepare_voice(persona) -> voice_profile

TTS Comparison

Qwen3-TTS

Decision: primary TTS bet.

Strengths:

• Best character fit.
• VoiceDesign and CustomVoice match our generated-persona concept.
• Natural-language control over voice style.
• Streaming support.
• Clean Apache 2.0 license.
• 0.6B and 1.7B variants provide scaling options.

Risks:

• Newer stack.
• May require FlashAttention 2 and GPU/runtime tuning.
• Needs proof of latency and reliability in our deployment environment.

NVIDIA Magpie TTS 357M

Decision: first fallback.

Strengths:

• Small and practical.
• NVIDIA-backed.
• Works well with the broader NVIDIA speech stack.
• Multiple voices and nine languages.
• Good voice-agent fit.

Risks:

• Less dynamic character control than Qwen3-TTS.
• Fewer English voices than ideal for many different ordinary people.
• NeMo dependency still needs validation in the Space.

Parler-TTS Mini

Decision: optional fallback or comparison spike, not MVP default.

Strengths:

• Apache 2.0.
• Prompt-controllable voice features.
• 34 named speakers.
• Simple conceptual model for voice descriptions.

Risks:

• English-only.
• Older and less compelling than Qwen3-TTS for this project.
• Around 0.9B params, larger than Kokoro and less flexible than Qwen3-TTS.

Kokoro 82M

Decision: emergency fallback.

Strengths:

• Very small.
• Fast.
• Apache 2.0.
• Easy to deploy and inexpensive to run.

Risks:

• Less theatrical.
• Less expressive character control.
• Better as a backup than the product-defining voice.

Voxtral TTS 4B

Note: The user referred to this as Vostral; the model appears to be Mistral's Voxtral TTS.

Decision: stretch spike, not MVP default.

Strengths:

• Expressive, low-latency voice-agent TTS.
• 20 preset voices.
• Voice adaptation from reference audio.
• vLLM-Omni serving path.
• Runs on a single GPU with at least 16GB VRAM.

Risks:

• CC BY-NC 4.0 license.
• 4B params just for TTS.
• Heavier runtime than Qwen3-TTS, Magpie, or Kokoro.
• Better suited to a Modal experiment than the first Space implementation.

Stretch Technologies

AVTR-1 Realtime Avatar

Decision: design the UI with an avatar-ready portal slot, but do not put AVTR-1 on the MVP critical path.

Why it is compelling:

• Real-time talking-head avatar.
• Uses a portrait plus dual-stream audio.
• Can render speaking and active listening behavior.
• Could make the portal feel like a person is truly present.

Why it is risky:

• Gated model access.
• Requires CUDA 12.x, TensorRT 10.x, and Ampere+ GPU.
• Requires building TensorRT engines.
• Hugging Face model card reports L4 near the edge of real-time performance.
• Licensing includes noncommercial pieces and consent-sensitive avatar/deepfake restrictions.
• Parameter count needs verification before hackathon use.

How to prepare for it:

• Build the portal area as a replaceable component.
• MVP should show stylized silhouette, portrait card, waveform, static, and environmental animation.
• Later, AVTR-1 can replace or augment the portal visual.

Recommended spike:

• Run AVTR-1 separately on Modal.
• Use only generated or clearly consent-safe reference portraits.
• Confirm licensing and parameter count before integrating into the submitted app.

PersonaPlex 7B

Decision: stretch experiment, not MVP.

Why it is compelling:

• Real-time spoken conversation.
• Persona-conditioned speech-to-speech interaction.
• Could make the app feel much more alive.

Why it is risky:

• Gated.
• More complex runtime.
• Likely high-end GPU expectations.
• Could absorb time needed for cockpit polish, launch sequence, reliable voice loop, and demo flow.

Recommended spike:

• Build an isolated proof of concept after the MVP voice loop works.
• Compare latency and character quality against modular STT + LLM + TTS.
• Promote only if it clearly improves the demo without destabilizing delivery.

llama.cpp

Decision: future stretch for the Llama Champion badge, not MVP.

Rationale:

• The project should first optimize for a polished GPU-backed demo.
• GGUF/llama.cpp can be revisited after the app works end to end.

Frontend Strategy

The frontend should feel like a cinematic ride plus magical radio.

Default UI direction:

• Strange laboratory.
• Full or near-full cockpit.
• Portal/windshield as the visual center.
• Control panel with dials, meters, switches, and warning lights.
• Signal/status text that feels like the machine is decoding a temporal contact.
• Transcript/radio panel for voice clarity.

Core states:

• Dormant cockpit.
• Launch charging.
• Temporal tunnel.
• Turbulence.
• Landing.
• Smoke clearing.
• Destination reveal.
• Signal lock.
• Conversation.
• Souvenir.

MVP visual approach:

• Use CSS animations and small JavaScript state transitions.
• Use stylized world presets instead of generated images.
• Let the LLM emit visual motifs, then map them to known preset keys.
• Keep generated image and avatar work as stretch.

Example visual presets:

• Rainy lantern district.
• Future flooded market.
• Medieval port.
• Orbital repair bay.
• Desert archive.
• Underground signal room.
• Snowbound radio station.
• Solar eclipse observatory.

Build Order

Follow this order unless the user changes priorities.

1. Create Gradio app shell.
2. Build strange laboratory cockpit layout.
3. Add frontend state machine and launch/reveal animations.
4. Add static mock destination/persona data to validate UI flow.
5. Implement structured destination generation.
6. Implement structured persona generation.
7. Implement transcript-first conversation loop.
8. Add souvenir generation.
9. Add TTS interface with Kokoro or a stub so audio wiring exists early.
10. Integrate Qwen3-TTS as primary TTS candidate.
11. Add Magpie fallback if Qwen3-TTS is unstable.
12. Add STT interface with clip transcription first.
13. Integrate Nemotron 3.5 ASR Streaming.
14. Add voice-first UX polish: partial transcript, signal decoding, playback states.
15. Tune prompts, timing, and audio length for a crisp recorded demo.
16. Consider Modal offload for Qwen3-TTS or heavier components if Space performance is weak.
17. Run stretch spikes only after the core encounter works end to end.

Required Interfaces

Keep these boundaries clean so models can be swapped.

Destination Generator

Input:

• mode.
• optional user coordinate prompt.
• random seed.

Output:

• structured destination JSON.

Persona Generator

Input:

• destination JSON.

Output:

• structured persona JSON.

Conversation Engine

Input:

• destination JSON.
• persona JSON.
• conversation history.
• latest user message.

Output:

• short in-character response text.
• optional emotion/prosody hint for TTS.
• optional UI signal event.

STT Adapter

Input:

• audio file or audio stream chunk.

Output:

• transcript text.
• optional partial transcript.
• optional confidence/timing metadata.

TTS Adapter

Input:

• response text.
• persona voice profile.
• emotion/prosody hint.

Output:

• audio file path or audio bytes.

Souvenir Generator

Input:

• destination JSON.
• persona JSON.
• conversation highlights.

Output:

• structured souvenir JSON.

Risks And Mitigations

Risk: Qwen3-TTS integration takes too long

Mitigation:

• Keep TTS behind an adapter.
• Start with Kokoro or stub audio for wiring.
• Keep Magpie as first serious fallback.

Risk: Nemotron streaming ASR is hard to wire through Gradio

Mitigation:

• Start with clip-based microphone transcription.
• Add streaming partials after the main loop works.
• Use partial transcript display as polish, not as a hard MVP dependency.

Risk: Hugging Face Space GPU is not enough

Mitigation:

• Use Qwen3-4B instead of Qwen3-8B.
• Move TTS or LLM to Modal.
• Avoid image generation and avatar generation in MVP.

Risk: Custom frontend becomes fragile

Mitigation:

• Keep JavaScript small.
• Use explicit UI states.
• Keep business logic in Python.
• Avoid frontend-generated source-of-truth state.

Risk: Stretch tech distracts from the demo

Mitigation:

• AVTR-1, PersonaPlex, Voxtral, and llama.cpp are separate spikes.
• Do not block MVP on gated models, TensorRT builds, or alternate runtimes.
• Build the cockpit so these can be added later without rewiring the app.

Current Decisions To Preserve

• Optimize for the most polished demo.
• Use GPU if it helps quality.
• Make the app voice-first.
• Use Surprise Me as the default first action.
• Start with a strange laboratory visual theme.
• Make the visual theme easy to replace later.
• Use Nemotron 3.5 ASR Streaming for preferred STT.
• Use Qwen3-TTS as the primary TTS bet.
• Keep Magpie and Kokoro as fallbacks.
• Keep AVTR-1 and PersonaPlex as stretch experiments.
• Keep llama.cpp as a later badge-oriented stretch.
• Use Modal if it materially improves product quality or unlocks a hard component.

Open Questions For Implementation

• Which exact Qwen3 LLM checkpoint should be used first: 4B for reliability or 8B for quality? DECIDED: Qwen3-8B via Together AI API.
• Can Qwen3-TTS run reliably inside the selected Hugging Face Space GPU? DECIDED: Qwen3-TTS runs on Modal, not in HF Space.
• Should Qwen3-TTS run in the Space or on Modal? DECIDED: Modal.
• What Space GPU tier should be used for the demo? DECIDED: CPU tier is sufficient (no local model inference).
• How much true streaming does Gradio need in MVP versus polished clip-turn interaction? DECIDED: Walk phase uses sync/text. Sprint phase adds streaming if time permits.
• Which generated voice attributes should be allowed for safety and consistency? (Still open.)
• Should the MVP include multilingual character speech, or keep speech output English-first? DECIDED: English-first for MVP.

References

Local docs:

• docs/hackathon_details.md
• docs/ai_time_machine_idea.md

Model and platform references:

• Hugging Face Build Small Hackathon: https://huggingface.co/build-small-hackathon
• Gradio Spaces documentation: https://huggingface.co/docs/hub/spaces-sdks-gradio
• NVIDIA Nemotron Speech collection: https://huggingface.co/collections/nvidia/nemotron-speech
• NVIDIA Nemotron 3.5 ASR Streaming 0.6B: https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b
• NVIDIA Nemotron Speech Streaming English 0.6B: https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b
• NVIDIA Magpie TTS Multilingual 357M: https://huggingface.co/nvidia/magpie_tts_multilingual_357m
• Qwen3-TTS CustomVoice: https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice
• Qwen3-TTS VoiceDesign: https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign
• Kokoro 82M: https://huggingface.co/hexgrad/Kokoro-82M
• Parler-TTS Mini v1: https://huggingface.co/parler-tts/parler-tts-mini-v1
• Voxtral TTS 4B: https://huggingface.co/mistralai/Voxtral-4B-TTS-2603
• AVTR-1 realtime avatar: https://huggingface.co/avaturn-live/avtr-1
• NVIDIA PersonaPlex 7B: https://huggingface.co/nvidia/personaplex-7b-v1
• Modal: https://modal.com