future_mobile_app_considerations.md

Future Mobile App Considerations

Purpose

This note captures future mobile deployment guidance for ReadBookMom. The current hackathon plan should remain a Hugging Face Spaces demo, with mobile treated as a later product direction.

Recommendation

Do not plan the first mobile version as a fully on-device app running all models locally. A hybrid mobile app is more realistic for reasonable latency, battery life, interruption behavior, and device coverage.

For the mobile app, use the phone as the recording, playback, interruption, and story interaction surface. Keep cloned-voice TTS and Q&A generation on a GPU backend until the models and runtime stack are proven on target devices.

The mobile version should inherit the current PRD direction: interruptible chunked streaming with a tap-to-Ask flow. The first mobile product should not attempt always-listening voice barge-in.

Model Fit For Mobile

Model	Mobile Fit	Notes
Qwen2.5-3B-Instruct	Possible on high-end phones only	With 4-bit quantization, it may fit on flagship devices, but latency, RAM pressure, battery drain, and thermal throttling will vary by device and runtime.
Whisper-small	Borderline	Better suited to server or desktop-class environments. For mobile latency, prefer native iOS/Android speech APIs, Whisper-tiny, or Whisper-base.
QWEN-TTS-0.6B	Hardest to deploy locally	Voice cloning TTS adds audio pipeline cost beyond parameter count. On-device latency may be poor unless a mobile-optimized runtime path is available.

Mobile Interaction Model

The mobile app should behave like a smart audiobook player:

Story chunk is playing
-> child taps Ask
-> app pauses local playback immediately
-> question is captured as text or voice
-> backend generates a short grounded answer
-> answer audio plays in the cloned voice
-> story resumes from the saved position

Button-based interruption is the recommended first version. True voice barge-in, where the child talks over narration and the app detects it automatically, should remain a later research item because it requires echo cancellation, voice activity detection, full-duplex audio handling, and careful privacy UX.

Mobile Architecture

Layer	Responsibility	Recommendation
Mobile app	Recording, playback, Ask interruption, local state, local cache	Native iOS/Android app or React Native/Flutter with native audio modules.
Backend API	Voice profile session, story chunk generation, Q&A answer generation, answer TTS	GPU backend using the same model stack as the Space demo.
Streaming/control channel	Request chunks, pause/resume, cancel queued work, receive generation status	WebSocket for bidirectional control; REST plus polling is acceptable for narration-only prototyping but will not support the full interrupt/answer/resume loop reliably.
Audio delivery	Stream or download generated chunks	Prefer short chunk files with signed URLs and local prefetch cache.
Session state	Track story and interruption position	Store story_id, voice_session_id, current_chunk_index, playback state, and cache keys.

The phone should own playback responsiveness. The server can cancel or deprioritize queued generation, but the app should pause current audio locally without waiting for backend confirmation.

Deployment Options

Option 1: Cloud-Backed Mobile App

The mobile app handles recording, story selection, interruption, playback, and local chunk caching. ASR, Q&A, and cloned-voice TTS run on a GPU backend.

Best for: first production prototype, hackathon follow-up, broad device support.

Pros: fastest path to reliable model latency, easier model updates, works across more phones.

Cons: requires backend hosting, network connectivity, and clear privacy messaging.

Option 2: Hybrid Mobile App

The app uses native or lightweight on-device ASR for the child's question, while Q&A and voice cloning TTS run on a backend. The phone manages interruptible chunk playback, local pause/resume, and cached story chunks.

Best for: better perceived latency while keeping heavy voice generation off-device.

Pros: practical performance, less backend work for transcription, smoother UX, and faster interruption response.

Cons: still depends on server-side TTS and model hosting.

Option 3: Fully On-Device App

The phone runs ASR, Q&A, voice cloning TTS, chunk generation, and playback locally.

Best for: long-term privacy-first product direction on selected flagship devices.

Pros: strongest offline and privacy story.

Cons: highest engineering risk; likely issues with memory, heat, battery drain, app size, cold start, audio session complexity, and inconsistent latency across devices.

Suggested Mobile Stack

Capability	Near-Term Mobile Choice	Long-Term On-Device Candidate
Q&A	Backend Qwen/Qwen2.5-3B-Instruct	Quantized Qwen2.5-1.5B-Instruct or quantized Qwen2.5-3B-Instruct on flagship devices
ASR	Native iOS/Android speech APIs or backend Whisper	Whisper-tiny/base through a mobile-optimized runtime
Voice cloning TTS	Backend QWEN-TTS-0.6B	Mobile-optimized TTS/voice clone model if quality and latency are proven
Story narration	Backend chunk generation with mobile playback and local chunk cache	Pre-generate and store locally after first generation
Story Q&A audio	Generate short answers on demand	On-device Q&A plus server or local lightweight TTS
Interruption	Tap-to-Ask pauses local playback immediately	Voice barge-in after echo cancellation and VAD are proven

Latency Expectations

Flow	Target	Mobile Strategy
Story start	First chunk starts in 3-6 seconds on a good network	Generate the first chunk server-side, then prefetch the next chunks.
Narration interruption	Local playback pauses within 500ms after Ask tap	Pause on-device immediately; backend cancellation is best effort.
Child question transcription	1-2 seconds	Prefer native mobile speech APIs or a tiny ASR model.
Q&A text answer	1-3 seconds	Use short prompts, 1-2 sentence answers, and backend Qwen2.5-3B-Instruct.
Spoken Q&A answer	Starts in 4-8 seconds after question submit	Keep answer text short and generate TTS server-side.
Story resume	Under 1 second when next chunk is cached	Resume from saved chunk position; fetch missing chunks in the background.
Cache miss resume	2-5 seconds	Show a lightweight loading state while the next chunk is generated or downloaded. Play a short filler clip (“Hmm, let me find my place…”) pre-synthesized in the cloned voice during voice setup, so the child hears the parent’s voice instead of silence.

Mobile latency depends heavily on network quality. The app should show resilient playback states instead of assuming the backend will always meet desktop demo timing.

Mobile Constraints

Constraint	Product Impact	Recommendation
Network jitter	Chunks may arrive late or out of order.	Prefetch the next 1-2 chunks and keep a small playback buffer.
Battery and thermals	Continuous local ASR or on-device LLM/TTS can drain battery quickly.	Keep heavy models server-side for v1; avoid always-on microphone.
Audio session complexity	Recording while audio is playing is fragile on mobile.	Pause playback before recording the question; resume after answer playback.
Echo cancellation	Open-mic interruption can capture the story audio instead of the child.	Use tap-to-Ask first; research VAD and echo cancellation later.
App lifecycle	Incoming calls, lock screen, and backgrounding can interrupt playback.	Treat v1 as foreground-first; persist playback state and chunk cache.
Storage	Cached voice/story audio can grow quickly.	Cache by voice session, story ID, chunk index, and model version with TTL cleanup.
App size	A thin cloud-backed client is ~20 MB. A hybrid app with on-device ASR (Whisper-tiny) may reach 100–200 MB. A fully on-device build with all models could exceed 4 GB.	Start thin; gate on-device model downloads behind user opt-in.
Privacy	Family voice data and child questions are sensitive.	Make upload, retention, deletion, and cache behavior explicit.

Product Implications

• The first mobile app should be a hybrid app, not a full local AI runtime.
• The phone should own playback state and interruption responsiveness.
• Privacy messaging must be explicit: explain what audio is sent to the backend, how long it is retained, and whether generated voice samples are cached.
• If offline mode becomes a requirement, treat it as a separate research milestone, not a launch feature. As a conscious tradeoff, v1 has no offline fallback — even a cached-story-only offline mode would add significant complexity.
• Device qualification matters. Fully on-device AI should start with a short list of supported flagship phones.
• Chunk prefetching and caching are key to making bedtime story playback and resume feel instant.
• Always-listening barge-in should not be promised until echo cancellation, VAD, and mobile privacy UX are proven.

Future Research Tasks

1. Benchmark quantized Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct on representative iOS and Android devices.
2. Compare native speech APIs with Whisper-tiny/base for child speech accuracy and latency.
3. Test whether QWEN-TTS-0.6B has a viable mobile runtime path or whether server-side TTS remains required.
4. Prototype backend chunk generation and mobile prefetch cache keyed by story ID, voice profile, chunk index, and model version.
5. Prototype tap-to-Ask interruption with local pause, backend cancellation/deprioritization, answer playback, and story resume.
6. Measure behavior under poor mobile networks, app backgrounding, and audio session interruptions.
7. Define privacy and retention policy before any real family audio is stored.

Decision

For the next phase, keep the model stack server-side and design the mobile app as a polished hybrid client with native playback, local chunk cache, tap-to-Ask interruption, and resume state. Revisit fully on-device deployment only after benchmarking smaller Q&A models, mobile ASR options, mobile-ready voice cloning TTS, and reliable barge-in audio handling.