Connect Dream QA voice input to Modal ASR

README.md

@@ -103,11 +103,12 @@ The public Space stays lightweight and can call private Modal endpoints through
 
 - `DREAM_CUSTOMS_TEXT_ENDPOINT`: Modal text route for `openbmb/MiniCPM5-1B`.
 - `DREAM_CUSTOMS_VISION_ENDPOINT`: Modal vision route for `openbmb/MiniCPM-V-4.6`.
+- `DREAM_CUSTOMS_ASR_ENDPOINT`: Modal ASR adapter route for voice-note transcription.
 - `DREAM_CUSTOMS_HOSTED_TOKEN`: shared bearer token checked by Modal and sent by the Space.
 
 Set these only as Hugging Face Space repository secrets or local shell variables. Do not store values in `.env`, docs, logs, screenshots, or git. Missing endpoints or route failures fall back to deterministic demo behavior.
 
-The Gradio UI defaults to `modal` for both text and vision backends, so a configured Space enters the private Modal MiniCPM endpoints first. The `demo` backend remains available in developer settings and as the deterministic fallback path when hosted routes are missing or fail.
+The Gradio UI defaults to `modal` for text, vision, and voice transcription backends, so a configured Space enters the private Modal routes first. The `demo` backend remains available in developer settings and as the deterministic fallback path when hosted routes are missing or fail.
 
 The Hugging Face Space may run on ZeroGPU for hackathon hardware eligibility. `dream_customs.zerogpu` registers a lightweight `@spaces.GPU` startup probe so ZeroGPU accepts the app, but real MiniCPM inference still happens on the private Modal backend.
 

docs/handoff.md

@@ -181,7 +181,7 @@ Hugging Face Space Gradio UI
   -> MiniCPM5-1B text generation and MiniCPM-V-4.6 visual clue extraction
 ```
 
-The Space defaults to `modal` for both text and vision backends, so a configured Space enters the private Modal MiniCPM endpoints first. The `model` auto route remains available for compatibility, and the `demo` backend remains available in developer settings as the deterministic fallback path.
+The Space defaults to `modal` for text, vision, and voice transcription backends, so a configured Space enters the private Modal routes first. The `model` auto route remains available for compatibility, and the `demo` backend remains available in developer settings as the deterministic fallback path.
 
 The public Space can run on ZeroGPU after `dream_customs.zerogpu` registers the lightweight `@spaces.GPU` startup probe. Modal credits still pay for the hidden GPU backend; the ZeroGPU probe is only there to make the HF hardware setting valid for this Gradio frontend.
 

docs/smoke/2026-06-10-modal-asr-voice-input-smoke.md

@@ -0,0 +1,19 @@
+# Modal ASR Voice Input Smoke - 2026-06-10
+
+## Scope
+
+Verify that Dream QA voice input uses a real Gradio audio component and that the backend can route recorded/uploaded audio to Modal ASR.
+
+No endpoint URLs, bearer tokens, Hugging Face tokens, Modal tokens, or secret values should be printed or recorded.
+
+## Evidence
+
+- Focused regression: `.venv/bin/python -m pytest tests/test_app_logic.py::test_defaults_use_modal_model_entrypoint tests/test_ui_actions.py::test_voice_input_uses_gradio_audio_file_for_modal_asr tests/test_modal_contract.py::test_decode_audio_payload_accepts_audio_key -q` -> 3 passed.
+- Full pytest: `.venv/bin/python -m pytest -q` -> 101 passed, 2 warnings.
+- Today Tip eval: `.venv/bin/python scripts/evaluate_today_tip_quality.py` -> 11 cases passed.
+- Whitespace check: `git diff --check` -> passed.
+- Local Gradio config: `Voice note` is a real `audio` component with `sources=["microphone", "upload"]`, `format="wav"`; `Voice input` default value is `modal`.
+- Modal deploy: `dream-customs-minicpm-backend` redeployed with a new `asr` web function. CLI output URLs were redacted.
+- Modal ASR auth smoke: route reachable without printing URL; unauthenticated request returned `status="unauthorized"`, confirming the route exists and requires the hosted token.
+- HF Space secret update attempt: API returned 403 for writing `DREAM_CUSTOMS_ASR_ENDPOINT`. The app now derives the ASR endpoint from the existing Modal text endpoint when the explicit ASR endpoint secret is missing, so no endpoint value was printed or stored.
+- Public Space check: pending.

docs/superpowers/plans/2026-06-10-modal-asr-voice-input.md

@@ -0,0 +1,35 @@
+# Modal ASR Voice Input Plan - 2026-06-10
+
+## Goal
+
+Replace the browser-only voice placeholder with a real Gradio audio recording/upload input that sends a filepath into the existing Dream QA intake, then connect voice transcription to a Modal-hosted ASR adapter.
+
+## Root Cause
+
+- The visible round microphone was custom JavaScript using browser speech recognition.
+- The backend `audio_input` was only `gr.State(None)`, so no recorded file reached `HostedASRClient`.
+- The Modal backend had text and vision routes but no `/asr` route.
+
+## Guardrails
+
+- ASR only transcribes audio; dream understanding remains MiniCPM-driven.
+- Do not store or print endpoint URLs, hosted tokens, HF tokens, Modal tokens, or secret values.
+- Keep text-only fallback working when audio permission, recording, upload, or hosted ASR fails.
+
+## Tasks
+
+- [x] Add failing tests for default Modal ASR, real Gradio audio input, and audio payload decoding.
+- [x] Change default ASR backend to `modal`.
+- [x] Replace the fake browser dictation input with `gr.Audio(sources=["microphone", "upload"], type="filepath")`.
+- [x] Add Modal `/asr` route with a small ASR adapter.
+- [x] Run full tests, eval, local UI smoke, and Modal deploy/smoke.
+- [ ] Run public Space verification after HF merge.
+- [ ] Commit, push, sync to Hugging Face Space, merge, and re-check Chrome.
+
+## Progress
+
+- Full pytest passes with 101 tests.
+- Today Tip eval passes 11 cases.
+- Local Gradio config confirms real audio input and Modal ASR default.
+- Modal deploy created the ASR route; unauthenticated route smoke confirmed the route is reachable and protected.
+- HF API cannot write Space secrets with the current token, so code derives the ASR endpoint from the existing Modal text endpoint when `DREAM_CUSTOMS_ASR_ENDPOINT` is absent.

dream_customs/app_logic.py

@@ -1,6 +1,7 @@
 import json
 import os
 from typing import Any, Tuple
+from urllib.parse import urlparse, urlunparse
 
 from dream_customs.models import (
     FakeASRClient,
@@ -12,7 +13,7 @@ from dream_customs.models import (
     OllamaTextClient,
     OllamaVisionClient,
 )
-from dream_customs.defaults import DEFAULT_TEXT_BACKEND, DEFAULT_VISION_BACKEND
+from dream_customs.defaults import DEFAULT_ASR_BACKEND, DEFAULT_TEXT_BACKEND, DEFAULT_VISION_BACKEND
 from dream_customs.pipeline import (
     add_evidence,
     answer_question,
@@ -75,6 +76,19 @@ def _as_int(value: Any, default: int) -> int:
         return default
 
 
+def _derive_modal_asr_endpoint(text_endpoint: str) -> str:
+    if not text_endpoint:
+        return ""
+    parsed = urlparse(text_endpoint.strip())
+    if parsed.netloc.endswith("-text.modal.run"):
+        netloc = f"{parsed.netloc.removesuffix('-text.modal.run')}-asr.modal.run"
+        return urlunparse(parsed._replace(netloc=netloc))
+    if parsed.path.rstrip("/").endswith("/text"):
+        path = f"{parsed.path.rstrip('/')[:-len('/text')]}/asr"
+        return urlunparse(parsed._replace(path=path))
+    return ""
+
+
 def _client_settings(
     text_endpoint: str = "",
     vision_endpoint: str = "",
@@ -88,15 +102,19 @@ def _client_settings(
     vision_temperature: Any = DEFAULT_VISION_TEMPERATURE,
     text_max_tokens: Any = DEFAULT_TEXT_MAX_TOKENS,
     vision_max_tokens: Any = DEFAULT_VISION_MAX_TOKENS,
-    asr_backend: str = "demo",
+    asr_backend: str = DEFAULT_ASR_BACKEND,
     asr_endpoint: str = "",
     asr_timeout_seconds: Any = DEFAULT_ASR_TIMEOUT_SECONDS,
     text_latency_budget_ms: Any = DEFAULT_TEXT_LATENCY_BUDGET_MS,
     vision_latency_budget_ms: Any = DEFAULT_VISION_LATENCY_BUDGET_MS,
     asr_latency_budget_ms: Any = DEFAULT_ASR_LATENCY_BUDGET_MS,
 ) -> dict:
+    resolved_text_endpoint = (text_endpoint or os.getenv("DREAM_CUSTOMS_TEXT_ENDPOINT", "")).strip()
+    resolved_asr_endpoint = (asr_endpoint or os.getenv("DREAM_CUSTOMS_ASR_ENDPOINT", "")).strip()
+    if not resolved_asr_endpoint:
+        resolved_asr_endpoint = _derive_modal_asr_endpoint(resolved_text_endpoint)
     return {
-        "text_endpoint": (text_endpoint or os.getenv("DREAM_CUSTOMS_TEXT_ENDPOINT", "")).strip(),
+        "text_endpoint": resolved_text_endpoint,
         "vision_endpoint": (vision_endpoint or os.getenv("DREAM_CUSTOMS_VISION_ENDPOINT", "")).strip(),
         "hosted_token": (hosted_token or os.getenv("DREAM_CUSTOMS_HOSTED_TOKEN", "")).strip(),
         "ollama_url": (ollama_url or os.getenv("DREAM_CUSTOMS_OLLAMA_URL", "http://localhost:11434")).strip(),
@@ -108,8 +126,8 @@ def _client_settings(
         "vision_temperature": max(0.0, min(_as_float(vision_temperature, DEFAULT_VISION_TEMPERATURE), 0.7)),
         "text_max_tokens": max(64, min(_as_int(text_max_tokens, DEFAULT_TEXT_MAX_TOKENS), 1200)),
         "vision_max_tokens": max(64, min(_as_int(vision_max_tokens, DEFAULT_VISION_MAX_TOKENS), 800)),
-        "asr_backend": (asr_backend or "demo").lower(),
-        "asr_endpoint": (asr_endpoint or os.getenv("DREAM_CUSTOMS_ASR_ENDPOINT", "")).strip(),
+        "asr_backend": (asr_backend or DEFAULT_ASR_BACKEND).lower(),
+        "asr_endpoint": resolved_asr_endpoint,
         "asr_timeout_seconds": max(1.0, _as_float(asr_timeout_seconds, DEFAULT_ASR_TIMEOUT_SECONDS)),
         "text_latency_budget_ms": max(0, _as_int(text_latency_budget_ms, DEFAULT_TEXT_LATENCY_BUDGET_MS)),
         "vision_latency_budget_ms": max(0, _as_int(vision_latency_budget_ms, DEFAULT_VISION_LATENCY_BUDGET_MS)),

dream_customs/defaults.py

@@ -1,2 +1,3 @@
 DEFAULT_TEXT_BACKEND = "modal"
 DEFAULT_VISION_BACKEND = "modal"
+DEFAULT_ASR_BACKEND = "modal"

dream_customs/ui/app.py

@@ -17,7 +17,7 @@ from dream_customs.app_logic import (
     DEFAULT_VISION_MODEL,
     DEFAULT_VISION_TEMPERATURE,
 )
-from dream_customs.defaults import DEFAULT_TEXT_BACKEND, DEFAULT_VISION_BACKEND
+from dream_customs.defaults import DEFAULT_ASR_BACKEND, DEFAULT_TEXT_BACKEND, DEFAULT_VISION_BACKEND
 from dream_customs.ui.actions import (
     answer_to_card_action,
     initial_mobile_state,
@@ -370,6 +370,10 @@ def _mic_html(language: str = DEFAULT_LANGUAGE) -> str:
 """.strip()
 
 
+def _voice_help_html(language: str = DEFAULT_LANGUAGE) -> str:
+    return f"<p class=\"dc-voice-help\">{escape(copy_for(language)['voice_help'])}</p>"
+
+
 def _field_tip_html(language: str = DEFAULT_LANGUAGE) -> str:
     return f"<p class=\"dc-field-tip\">{escape(copy_for(language)['field_tip'])}</p>"
 
@@ -403,7 +407,7 @@ def build_demo() -> gr.Blocks:
     initial = _load_view(initial_view)
     initial_copy = copy_for(DEFAULT_LANGUAGE)
 
-    with gr.Blocks(css=CSS, js=VOICE_JS, title=APP_TITLE) as demo:
+    with gr.Blocks(css=CSS, title=APP_TITLE) as demo:
         session_state = gr.State(initial_state)
         view_state = gr.State(initial_view)
 
@@ -423,8 +427,14 @@ def build_demo() -> gr.Blocks:
                                 value="",
                                 elem_classes=["dc-dream-text"],
                             )
-                            mic_html = gr.HTML(_mic_html(DEFAULT_LANGUAGE))
-                            audio_input = gr.State(None)
+                            voice_help_html = gr.HTML(_voice_help_html(DEFAULT_LANGUAGE))
+                            audio_input = gr.Audio(
+                                label=initial_copy["voice_label"],
+                                sources=["microphone", "upload"],
+                                type="filepath",
+                                format="wav",
+                                elem_classes=["dc-voice-input"],
+                            )
                             field_tip_html = gr.HTML(_field_tip_html(DEFAULT_LANGUAGE))
                         with gr.Row(elem_classes=["dc-submit-row"]):
                             example_button = gr.Button(initial_copy["example_button"], variant="secondary")
@@ -494,11 +504,11 @@ def build_demo() -> gr.Blocks:
                         asr_backend = gr.Dropdown(
                             label="Voice input",
                             choices=[
-                                ("Browser dictation now", "demo"),
-                                ("Modal ASR endpoint, planned", "modal"),
+                                ("Modal ASR endpoint", "modal"),
+                                ("Demo: mark voice received", "demo"),
                                 ("Hugging Face ASR endpoint, planned", "huggingface"),
                             ],
-                            value="demo",
+                            value=DEFAULT_ASR_BACKEND,
                         )
                         with gr.Accordion("Advanced endpoints", open=False, elem_classes=["dc-dev-advanced"]):
                             text_endpoint = gr.Textbox(label="Text endpoint", value="")
@@ -653,7 +663,7 @@ def build_demo() -> gr.Blocks:
                 _notice_html({"notice": copy["notice_record"], "status": "record"}),
                 _section_title_html(1, copy["dream_label"]),
                 gr.update(label=copy["dream_label"], placeholder=copy["dream_placeholder"]),
-                _mic_html(selected_language),
+                _voice_help_html(selected_language),
                 _field_tip_html(selected_language),
                 gr.update(value=copy["example_button"]),
                 gr.update(value=copy["submit_button"]),
@@ -682,7 +692,7 @@ def build_demo() -> gr.Blocks:
                 notice,
                 dream_section_html,
                 dream_text,
-                mic_html,
+                voice_help_html,
                 field_tip_html,
                 example_button,
                 submit_button,

dream_customs/ui/copy.py

@@ -23,6 +23,8 @@ APP_COPY = {
         "mic_listening": "Listening. Say the dream fragment when you are ready.",
         "mic_done": "Added to the dream note.",
         "mic_empty": "No speech detected. Tap again if you want to retry.",
+        "voice_label": "Voice note",
+        "voice_help": "Record or upload a short voice note. It is transcribed by the ASR adapter when you continue.",
         "field_tip": "People, places, feelings, colors, or the question you woke up with are all useful.",
         "example_button": "Try example",
         "submit_button": "Continue",
@@ -72,6 +74,8 @@ APP_COPY = {
         "mic_listening": "正在听。准备好后说出梦境片段。",
         "mic_done": "已加入梦境记录。",
         "mic_empty": "没有检测到语音。想重试的话，再点一次麦克风。",
+        "voice_label": "语音片段",
+        "voice_help": "可以录音或上传一小段语音。点击继续后，ASR 适配器会先转写它。",
         "field_tip": "可以补充人物、地点、情绪、颜色，或醒来后最在意的疑问。",
         "example_button": "试试示例",
         "submit_button": "继续解梦  →",

dream_customs/ui/styles.py

@@ -394,6 +394,26 @@ body,
   display: none !important;
 }
 
+.dc-voice-help {
+  color: var(--dc-muted);
+  font-size: 0.86rem;
+  line-height: 1.4;
+  margin: 12px 0 6px;
+}
+
+.dc-voice-input {
+  margin-top: 0 !important;
+}
+
+.dc-voice-input .wrap,
+.dc-voice-input .container,
+.dc-voice-input .input-container {
+  background: rgba(255, 253, 248, 0.9) !important;
+  border-color: var(--dc-line) !important;
+  border-radius: var(--dc-radius-sm) !important;
+  color: var(--dc-ink) !important;
+}
+
 .dc-mic-control {
   align-items: end;
   bottom: 68px;

modal_backend/contracts.py

@@ -54,6 +54,18 @@ def decode_image_payload(payload: Dict[str, Any]) -> bytes:
     return base64.b64decode(str(encoded))
 
 
+def decode_audio_payload(payload: Dict[str, Any]) -> tuple[bytes, str]:
+    encoded = payload.get("audio")
+    if not encoded and isinstance(payload.get("audios"), list) and payload["audios"]:
+        encoded = payload["audios"][0]
+    if not encoded:
+        raise ValueError("Missing audio payload.")
+    if isinstance(encoded, bytes):
+        encoded = encoded.decode("ascii")
+    filename = _clean_text(payload.get("filename")) or "dream-voice.wav"
+    return base64.b64decode(str(encoded)), filename
+
+
 def ensure_authorized(authorization_header: str, expected_token: str) -> None:
     expected_token = expected_token.strip()
     if not expected_token:

modal_backend/dream_customs_modal.py

@@ -9,6 +9,7 @@ from fastapi import Body, Header
 
 from modal_backend.contracts import (
     AuthError,
+    decode_audio_payload,
     decode_image_payload,
     ensure_authorized,
     normalize_text_payload,
@@ -19,6 +20,7 @@ from modal_backend.contracts import (
 APP_NAME = "dream-customs-minicpm-backend"
 TEXT_MODEL = "openbmb/MiniCPM5-1B"
 VISION_MODEL = "openbmb/MiniCPM-V-4.6"
+ASR_MODEL = "openai/whisper-tiny"
 MINUTES = 60
 
 app = modal.App(APP_NAME)
@@ -44,6 +46,8 @@ image = (
         "torch",
         "torchvision",
         "transformers>=4.56",
+        "librosa",
+        "soundfile",
     )
     .add_local_dir("modal_backend", remote_path="/root/modal_backend")
 )
@@ -57,6 +61,7 @@ secrets = [
 
 _TEXT_PIPE = None
 _VISION_PIPE = None
+_ASR_PIPE = None
 
 
 def _expected_token() -> str:
@@ -113,6 +118,20 @@ def _load_vision_pipe():
     return _VISION_PIPE
 
 
+def _load_asr_pipe():
+    global _ASR_PIPE
+    if _ASR_PIPE is None:
+        from transformers import pipeline
+
+        _ASR_PIPE = pipeline(
+            "automatic-speech-recognition",
+            model=os.getenv("DREAM_CUSTOMS_ASR_MODEL", ASR_MODEL),
+            device_map="auto",
+            torch_dtype="auto",
+        )
+    return _ASR_PIPE
+
+
 def _messages_from_payload(payload: Dict[str, Any], prompt: str) -> list[Dict[str, str]]:
     messages = payload.get("messages")
     if isinstance(messages, list) and messages:
@@ -299,6 +318,7 @@ def health() -> Dict[str, str]:
         "app": APP_NAME,
         "text_model": TEXT_MODEL,
         "vision_model": VISION_MODEL,
+        "asr_model": ASR_MODEL,
     }
 
 
@@ -390,3 +410,35 @@ async def vision(
         ]
         result = pipe(text=messages, max_new_tokens=int(payload.get("max_tokens", 320)))
     return response_payload(_stringify_pipeline_result(result))
+
+
+@app.function(
+    image=image,
+    gpu="L4",
+    timeout=10 * MINUTES,
+    scaledown_window=5 * MINUTES,
+    volumes={"/root/.cache/huggingface": hf_cache},
+    secrets=secrets,
+)
+@modal.fastapi_endpoint(method="POST", docs=True)
+async def asr(
+    payload: Dict[str, Any] = Body(...),
+    authorization: str = Header(""),
+):
+    try:
+        ensure_authorized(authorization, _expected_token())
+    except AuthError as exc:
+        return _json_error(str(exc), status="unauthorized")
+    try:
+        audio_bytes, filename = decode_audio_payload(payload)
+    except ValueError as exc:
+        return _json_error(str(exc))
+
+    suffix = os.path.splitext(filename)[1] or ".wav"
+    pipe = _load_asr_pipe()
+    with tempfile.NamedTemporaryFile(suffix=suffix) as temp_file:
+        temp_file.write(audio_bytes)
+        temp_file.flush()
+        result = pipe(temp_file.name)
+    transcript = _stringify_pipeline_result(result)
+    return {"status": "ok", "transcript": transcript, "response": transcript}

tests/test_app_logic.py

@@ -2,6 +2,7 @@ import json
 
 from dream_customs.app_logic import (
     _clients,
+    _client_settings,
     _debug_json,
     add_material_action,
     ask_another_question_action,
@@ -12,13 +13,14 @@ from dream_customs.app_logic import (
     seal_pact_action,
     start_declaration_action,
 )
-from dream_customs.defaults import DEFAULT_TEXT_BACKEND, DEFAULT_VISION_BACKEND
+from dream_customs.defaults import DEFAULT_ASR_BACKEND, DEFAULT_TEXT_BACKEND, DEFAULT_VISION_BACKEND
 from dream_customs.models import HostedASRClient, HostedMiniCPMTextClient, HostedMiniCPMVisionClient
 
 
 def test_defaults_use_modal_model_entrypoint():
     assert DEFAULT_TEXT_BACKEND == "modal"
     assert DEFAULT_VISION_BACKEND == "modal"
+    assert DEFAULT_ASR_BACKEND == "modal"
 
 
 def test_run_customs_once_generates_demo_outputs():
@@ -137,3 +139,12 @@ def test_developer_settings_configure_hosted_clients():
     assert vision_client.temperature == 0.22
     assert text_client.max_tokens == 256
     assert vision_client.max_tokens == 128
+
+
+def test_asr_endpoint_derives_from_modal_text_endpoint_when_secret_is_missing():
+    settings = _client_settings(
+        text_endpoint="https://workspace--dream-customs-minicpm-backend-text.modal.run",
+        asr_endpoint="",
+    )
+
+    assert settings["asr_endpoint"] == "https://workspace--dream-customs-minicpm-backend-asr.modal.run"

tests/test_modal_contract.py

@@ -2,6 +2,7 @@ import base64
 
 from modal_backend.contracts import (
     AuthError,
+    decode_audio_payload,
     decode_image_payload,
     ensure_authorized,
     normalize_text_payload,
@@ -39,6 +40,14 @@ def test_decode_image_payload_accepts_images_list():
     assert decode_image_payload({"images": [encoded]}) == b"fake-image-bytes"
 
 
+def test_decode_audio_payload_accepts_audio_key():
+    encoded = base64.b64encode(b"fake-audio-bytes").decode("ascii")
+    audio_bytes, filename = decode_audio_payload({"audio": encoded, "filename": "dream.wav"})
+
+    assert audio_bytes == b"fake-audio-bytes"
+    assert filename == "dream.wav"
+
+
 def test_response_payload_uses_existing_client_shape():
     assert response_payload("hello") == {"response": "hello"}
 

tests/test_ui_actions.py

@@ -22,6 +22,16 @@ def test_runtime_settings_are_collapsed_for_public_flow():
     assert 'gr.Accordion("Advanced", open=False' in source
 
 
+def test_voice_input_uses_gradio_audio_file_for_modal_asr():
+    source = inspect.getsource(ui_app.build_demo)
+
+    assert "audio_input = gr.Audio(" in source
+    assert 'sources=["microphone", "upload"]' in source
+    assert 'type="filepath"' in source
+    assert "audio_input = gr.State(None)" not in source
+    assert 'value=DEFAULT_ASR_BACKEND' in source
+
+
 def test_processing_note_is_story_copy_not_backend_jargon():
     lowered = PROCESSING_NOTE.lower()