future_mobile_app_considerations.md

# Future Mobile App Considerations

## Purpose

This note captures future mobile deployment guidance for ReadBookMom. The current hackathon plan should remain a Hugging Face Spaces demo, with mobile treated as a later product direction.

## Recommendation

Do not plan the first mobile version as a fully on-device app running all models locally. A hybrid mobile app is more realistic for reasonable latency, battery life, interruption behavior, and device coverage.

For the mobile app, use the phone as the recording, playback, interruption, and story interaction surface. Keep cloned-voice TTS and Q&A generation on a GPU backend until the models and runtime stack are proven on target devices.

The mobile version should inherit the current PRD direction: interruptible chunked streaming with a tap-to-Ask flow. The first mobile product should not attempt always-listening voice barge-in.

## Model Fit For Mobile

| Model | Mobile Fit | Notes |
|---|---|---|
| Qwen2.5-3B-Instruct | Possible on high-end phones only | With 4-bit quantization, it may fit on flagship devices, but latency, RAM pressure, battery drain, and thermal throttling will vary by device and runtime. |
| Whisper-small | Borderline | Better suited to server or desktop-class environments. For mobile latency, prefer native iOS/Android speech APIs, Whisper-tiny, or Whisper-base. |
| QWEN-TTS-0.6B | Hardest to deploy locally | Voice cloning TTS adds audio pipeline cost beyond parameter count. On-device latency may be poor unless a mobile-optimized runtime path is available. |

## Mobile Interaction Model

The mobile app should behave like a smart audiobook player:

```text
Story chunk is playing
-> child taps Ask
-> app pauses local playback immediately
-> question is captured as text or voice
-> backend generates a short grounded answer
-> answer audio plays in the cloned voice
-> story resumes from the saved position
```

Button-based interruption is the recommended first version. True voice barge-in, where the child talks over narration and the app detects it automatically, should remain a later research item because it requires echo cancellation, voice activity detection, full-duplex audio handling, and careful privacy UX.

## Mobile Architecture

| Layer | Responsibility | Recommendation |
|---|---|---|
| Mobile app | Recording, playback, Ask interruption, local state, local cache | Native iOS/Android app or React Native/Flutter with native audio modules. |
| Backend API | Voice profile session, story chunk generation, Q&A answer generation, answer TTS | GPU backend using the same model stack as the Space demo. |
| Streaming/control channel | Request chunks, pause/resume, cancel queued work, receive generation status | WebSocket for bidirectional control; REST plus polling is acceptable for narration-only prototyping but will not support the full interrupt/answer/resume loop reliably. |
| Audio delivery | Stream or download generated chunks | Prefer short chunk files with signed URLs and local prefetch cache. |
| Session state | Track story and interruption position | Store `story_id`, `voice_session_id`, `current_chunk_index`, playback state, and cache keys. |

The phone should own playback responsiveness. The server can cancel or deprioritize queued generation, but the app should pause current audio locally without waiting for backend confirmation.

## Deployment Options

### Option 1: Cloud-Backed Mobile App

The mobile app handles recording, story selection, interruption, playback, and local chunk caching. ASR, Q&A, and cloned-voice TTS run on a GPU backend.

**Best for:** first production prototype, hackathon follow-up, broad device support.

**Pros:** fastest path to reliable model latency, easier model updates, works across more phones.

**Cons:** requires backend hosting, network connectivity, and clear privacy messaging.

### Option 2: Hybrid Mobile App

The app uses native or lightweight on-device ASR for the child's question, while Q&A and voice cloning TTS run on a backend. The phone manages interruptible chunk playback, local pause/resume, and cached story chunks.

**Best for:** better perceived latency while keeping heavy voice generation off-device.

**Pros:** practical performance, less backend work for transcription, smoother UX, and faster interruption response.

**Cons:** still depends on server-side TTS and model hosting.

### Option 3: Fully On-Device App

The phone runs ASR, Q&A, voice cloning TTS, chunk generation, and playback locally.

**Best for:** long-term privacy-first product direction on selected flagship devices.

**Pros:** strongest offline and privacy story.

**Cons:** highest engineering risk; likely issues with memory, heat, battery drain, app size, cold start, audio session complexity, and inconsistent latency across devices.

## Suggested Mobile Stack

| Capability | Near-Term Mobile Choice | Long-Term On-Device Candidate |
|---|---|---|
| Q&A | Backend `Qwen/Qwen2.5-3B-Instruct` | Quantized `Qwen2.5-1.5B-Instruct` or quantized `Qwen2.5-3B-Instruct` on flagship devices |
| ASR | Native iOS/Android speech APIs or backend Whisper | Whisper-tiny/base through a mobile-optimized runtime |
| Voice cloning TTS | Backend QWEN-TTS-0.6B | Mobile-optimized TTS/voice clone model if quality and latency are proven |
| Story narration | Backend chunk generation with mobile playback and local chunk cache | Pre-generate and store locally after first generation |
| Story Q&A audio | Generate short answers on demand | On-device Q&A plus server or local lightweight TTS |
| Interruption | Tap-to-Ask pauses local playback immediately | Voice barge-in after echo cancellation and VAD are proven |

## Latency Expectations

| Flow | Target | Mobile Strategy |
|---|---|---|
| Story start | First chunk starts in 3-6 seconds on a good network | Generate the first chunk server-side, then prefetch the next chunks. |
| Narration interruption | Local playback pauses within 500ms after Ask tap | Pause on-device immediately; backend cancellation is best effort. |
| Child question transcription | 1-2 seconds | Prefer native mobile speech APIs or a tiny ASR model. |
| Q&A text answer | 1-3 seconds | Use short prompts, 1-2 sentence answers, and backend Qwen2.5-3B-Instruct. |
| Spoken Q&A answer | Starts in 4-8 seconds after question submit | Keep answer text short and generate TTS server-side. |
| Story resume | Under 1 second when next chunk is cached | Resume from saved chunk position; fetch missing chunks in the background. |
| Cache miss resume | 2-5 seconds | Show a lightweight loading state while the next chunk is generated or downloaded. Play a short filler clip (“Hmm, let me find my place…”) pre-synthesized in the cloned voice during voice setup, so the child hears the parent’s voice instead of silence. |

Mobile latency depends heavily on network quality. The app should show resilient playback states instead of assuming the backend will always meet desktop demo timing.

## Mobile Constraints

| Constraint | Product Impact | Recommendation |
|---|---|---|
| Network jitter | Chunks may arrive late or out of order. | Prefetch the next 1-2 chunks and keep a small playback buffer. |
| Battery and thermals | Continuous local ASR or on-device LLM/TTS can drain battery quickly. | Keep heavy models server-side for v1; avoid always-on microphone. |
| Audio session complexity | Recording while audio is playing is fragile on mobile. | Pause playback before recording the question; resume after answer playback. |
| Echo cancellation | Open-mic interruption can capture the story audio instead of the child. | Use tap-to-Ask first; research VAD and echo cancellation later. |
| App lifecycle | Incoming calls, lock screen, and backgrounding can interrupt playback. | Treat v1 as foreground-first; persist playback state and chunk cache. |
| Storage | Cached voice/story audio can grow quickly. | Cache by voice session, story ID, chunk index, and model version with TTL cleanup. |
| App size | A thin cloud-backed client is ~20 MB. A hybrid app with on-device ASR (Whisper-tiny) may reach 100–200 MB. A fully on-device build with all models could exceed 4 GB. | Start thin; gate on-device model downloads behind user opt-in. |
| Privacy | Family voice data and child questions are sensitive. | Make upload, retention, deletion, and cache behavior explicit. |

## Product Implications

- The first mobile app should be a hybrid app, not a full local AI runtime.
- The phone should own playback state and interruption responsiveness.
- Privacy messaging must be explicit: explain what audio is sent to the backend, how long it is retained, and whether generated voice samples are cached.
- If offline mode becomes a requirement, treat it as a separate research milestone, not a launch feature. As a conscious tradeoff, v1 has no offline fallback — even a cached-story-only offline mode would add significant complexity.
- Device qualification matters. Fully on-device AI should start with a short list of supported flagship phones.
- Chunk prefetching and caching are key to making bedtime story playback and resume feel instant.
- Always-listening barge-in should not be promised until echo cancellation, VAD, and mobile privacy UX are proven.

## Future Research Tasks

1. Benchmark quantized Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct on representative iOS and Android devices.
2. Compare native speech APIs with Whisper-tiny/base for child speech accuracy and latency.
3. Test whether QWEN-TTS-0.6B has a viable mobile runtime path or whether server-side TTS remains required.
4. Prototype backend chunk generation and mobile prefetch cache keyed by story ID, voice profile, chunk index, and model version.
5. Prototype tap-to-Ask interruption with local pause, backend cancellation/deprioritization, answer playback, and story resume.
6. Measure behavior under poor mobile networks, app backgrounding, and audio session interruptions.
7. Define privacy and retention policy before any real family audio is stored.

## Decision

For the next phase, keep the model stack server-side and design the mobile app as a polished hybrid client with native playback, local chunk cache, tap-to-Ask interruption, and resume state. Revisit fully on-device deployment only after benchmarking smaller Q&A models, mobile ASR options, mobile-ready voice cloning TTS, and reliable barge-in audio handling.