Spaces:
Sleeping
Sleeping
Commit Β·
e6fcd38
1
Parent(s): b45f58a
docs: plan for TTS dash-strip + spoken opening + treasure align; refresh CLAUDE.md (voice/awareness/presence, new gotchas, mirror-first deploy)
Browse files
CLAUDE.md
CHANGED
|
@@ -14,10 +14,11 @@ A horror NPC chatbot for the **Build Small Hackathon** (HF / Gradio). Track:
|
|
| 14 |
## Stack (decided β do not reopen)
|
| 15 |
- **UI:** Gradio 6.x `gr.Blocks` + `gr.Chatbot` (messages format default, no `type` param). Pure CSS, **no JS**.
|
| 16 |
- **Model:** `Qwen/Qwen3-8B` via `transformers`+`@spaces.GPU(duration=90)` on Space; `qwen3:8b` via Ollama (`localhost:11434`) locally. Switch: `IS_SPACE = bool(os.environ.get("SPACE_ID"))`.
|
|
|
|
| 17 |
- **State:** plain dict in `gr.State` per session. **Parsing:** Pydantic v2 + fallback.
|
| 18 |
- **Art:** grayscale FLUX portraits (Fear & Hunger style) in `assets/`, base64 WebP. Audio synthesized in numpy at import (`sting.py`).
|
| 19 |
-
- **Pkg manager:** `uv` (`uv pip install`).
|
| 20 |
-
- **Rejected (don't reopen):** Gemma 4 12B (no fit
|
| 21 |
|
| 22 |
## Submission requirements (verified, Field Guide REQ-06)
|
| 23 |
- Space in the org β
. README frontmatter: track + badge tags + write-up β
(`track:wood`, `achievement:offgrid/offbrand/fieldnotes`). **No submission form β the Space in the org with a complete README IS the entry.**
|
|
@@ -26,19 +27,20 @@ A horror NPC chatbot for the **Build Small Hackathon** (HF / Gradio). Track:
|
|
| 26 |
|
| 27 |
## File structure
|
| 28 |
```
|
| 29 |
-
app.py # Gradio layout (3 cols: entity|chat|treasure); chat() generator;
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
sting.py #
|
| 37 |
-
styles.css # horror CSS
|
| 38 |
assets/ # hollow_{base,terror,almost,end,rage,peace}.webp (rage = almost copy placeholder; peace = real art)
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
|
|
|
| 42 |
```
|
| 43 |
|
| 44 |
## State shape (gr.State) β all NEW-field reads use `.get(...)` for pre-deploy sessions
|
|
@@ -55,6 +57,11 @@ tools/pacing_sim.py # plays scripted visitors through real Ollama,
|
|
| 55 |
"fragments_told": int, # how many OWN_FRAGMENTS surfaced (redemption path)
|
| 56 |
"ended": bool, # a finale fired β input dead
|
| 57 |
"ending": str | None, # "good"|"loop"|"bad" once fired
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 58 |
}
|
| 59 |
```
|
| 60 |
Dev seed: `HOLLOW_FAST_FINALE=good|loop|bad` (`1`/`neutral` aliasβ good/loop) β each ending fires on the 2nd message.
|
|
@@ -76,6 +83,11 @@ Single gate so no lower threshold can race a warm run:
|
|
| 76 |
- Tone bands in prompt (separate from ending routing): warm β₯15 / wounded β22..β1 / hostile β€β22 (injects wounds). Mimicry always on.
|
| 77 |
- **Measured pacing** (pacing_sim, real model): bad ~msg 6, good (warm) ~msg 10, loop (factual) ~msg 12, seeds msg 2. Warm play tone climbs to ~27; factual play tone stalls ~13 β clean gap at 20.
|
| 78 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
## Gotchas / lessons (DON'T re-learn these the hard way)
|
| 80 |
- **Verify CSS against the RUNNING app**, never the static harness β it lacks Gradio's cascade and has lied. Workflow: launch `HOLLOW_FAST_FINALE=good .venv/Scripts/python app.py` background β headless Chrome screenshot (`chrome.exe --headless --disable-gpu --screenshot=... --window-size=1900,950 --virtual-time-budget=12000 http://localhost:7860`) β Read the PNG β kill the port owner.
|
| 81 |
- **Gradio replaces the HTML DOM each update:** CSS *animations* restart (this IS the flash/screamer mechanism); CSS *transitions* DON'T animate across updates.
|
|
@@ -83,7 +95,7 @@ Single gate so no lower threshold can race a warm run:
|
|
| 83 |
- **Qwen emits `+2` (invalid JSON)** if examples lack positive anchors β every positive turn fell back to delta 0 (bond frozen). Fixed by sanitizer in `apply_update` + explicit "no leading +" instruction + dual examples. Keep both.
|
| 84 |
- **transformers 5.x:** `apply_chat_template(...,return_tensors="pt")` returns `BatchEncoding` (use `hasattr(t,"input_ids")`); use `dtype=` not deprecated `torch_dtype`. Qwen3 thinking OFF: `enable_thinking=False` (Space) / `"think": False` (Ollama).
|
| 85 |
- **One GPU acquisition per turn:** gen + extract inside a single `@spaces.GPU` call. ZeroGPU β8 s/turn, PRO β1500 s/day.
|
| 86 |
-
- **Portrait fills the arch:** art is square (1:1), arch is 5:6 β materialization is
|
| 87 |
- **HF rejects committed binaries (.wav):** synthesize audio at runtime (`sting.py`). Same would apply to any new binary asset that isn't an image.
|
| 88 |
- **Don't edit `.py` via PowerShell string ops** β it mangled UTF-8 (em-dash β mojibake) once. Use the Edit tool.
|
| 89 |
- **Ollama ROCm can corrupt** (`cudaMalloc failed` with free VRAM) β symptom is red Error on ALL components. Fix: restart Ollama (tray + server). The Space (transformers) is immune.
|
|
@@ -92,14 +104,22 @@ Single gate so no lower threshold can race a warm run:
|
|
| 92 |
- **NO threading around `run_turn`:** `@spaces.GPU` needs the main request context β a thread pool breaks ZeroGPU. The typing-dots animation is pure CSS for this reason (don't "fix" it with a worker thread).
|
| 93 |
- **Hide Gradio progress, animate our own:** `show_progress="hidden"` on the events kills the per-component "processing | Xs" overlay (CSS class-hiding was unreliable); we render our own `_PENDING` typing bubble. The HTML span + class survives Gradio's markdown sanitizer (verified) so Β§20 CSS animates it.
|
| 94 |
- **Live-verify via isolated probe:** to test a UI behavior that only appears mid-generation (typing dots, button states), spin a tiny throwaway Gradio app importing `styles.css`, screenshot it, delete it β don't guess CSS class names.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 95 |
|
| 96 |
## What's next
|
| 97 |
-
- **
|
| 98 |
-
- **
|
| 99 |
-
- **
|
| 100 |
-
- **
|
|
|
|
| 101 |
|
| 102 |
## Workflows
|
| 103 |
-
**Deploy** (from repo root
|
|
|
|
| 104 |
|
| 105 |
**Working style:** phase by phase; explain *why* before code; everything in **English** (code, prompts, UI, dialogue); no new deps/complexity without asking; push back on scope creep. Subagent-driven dev for plans (implementer + spec review + quality review per task).
|
|
|
|
| 14 |
## Stack (decided β do not reopen)
|
| 15 |
- **UI:** Gradio 6.x `gr.Blocks` + `gr.Chatbot` (messages format default, no `type` param). Pure CSS, **no JS**.
|
| 16 |
- **Model:** `Qwen/Qwen3-8B` via `transformers`+`@spaces.GPU(duration=90)` on Space; `qwen3:8b` via Ollama (`localhost:11434`) locally. Switch: `IS_SPACE = bool(os.environ.get("SPACE_ID"))`.
|
| 17 |
+
- **Voice:** Kokoro-82M TTS (`voice.py`), **CPU** β the child whispers (`af_nicole`, speed 0.88, "28% fog" numpy DSP). Import-guarded β **silent if unavailable** (local 3.13 `.venv` has no kokoro; works in 3.12 `.venv-tts` and on the Space at py3.11). Audio inlined as autoplay `<audio>` in a mounted-but-CSS-collapsed `voice_panel`. Plays on every Hollow reply + all finale lines.
|
| 18 |
- **State:** plain dict in `gr.State` per session. **Parsing:** Pydantic v2 + fallback.
|
| 19 |
- **Art:** grayscale FLUX portraits (Fear & Hunger style) in `assets/`, base64 WebP. Audio synthesized in numpy at import (`sting.py`).
|
| 20 |
+
- **Pkg manager:** `uv` (`uv pip install`). Space build also needs `packages.txt` (espeak-ng) + `python_version: "3.11"` (Kokoro needs <3.13).
|
| 21 |
+
- **Rejected (don't reopen):** Gemma 4 12B (no 8 GB VRAM fit); **Qwen3.5/3.6 (all multimodal β break the transformers loader; the field uses 1-8B, so 8B stays on-brand)**; LangGraph, Qdrant, cloud APIs, fine-tuning, sponsor-model swap (re-tuning risk for speculative prize).
|
| 22 |
|
| 23 |
## Submission requirements (verified, Field Guide REQ-06)
|
| 24 |
- Space in the org β
. README frontmatter: track + badge tags + write-up β
(`track:wood`, `achievement:offgrid/offbrand/fieldnotes`). **No submission form β the Space in the org with a complete README IS the entry.**
|
|
|
|
| 27 |
|
| 28 |
## File structure
|
| 29 |
```
|
| 30 |
+
app.py # thread-cap (OMP/OPENBLAS=4) at top; Gradio layout (3 cols: entity|chat|treasure + hidden voice_panel); chat() generator; 3 _play_finale_*; _PENDING bubble; _on_idle (gr.Timer); _toggle_voice (mute); _voice_html/_voice_and_pause; show_progress="hidden"
|
| 31 |
+
voice.py # speak(text)->base64 wav|None β Kokoro-82M whisper + locked DSP, CPU, import-guarded; thread-cap; _clean_for_tts (planned)
|
| 32 |
+
engine.py # run_turn(chat_messages, user_msg) -> (reply, raw_json); _EXTRACTION_SYSTEM; IS_SPACE switch; repetition_penalty on reply gen
|
| 33 |
+
character.py # build_system_prompt(... tone, wounds, ask_for_memory, own_fragment, style, aware_memory); tone bands; OWN_FRAGMENTS; OPENING_LINE
|
| 34 |
+
memory.py # get_tier, apply_update (+sanitize/Hollow-words filter), should_recall, decide_ending, style_signal (B3), pick_aware_memory (B2) β pure
|
| 35 |
+
finale.py # finale_steps (loop), finale_steps_bad (wound), finale_steps_good (redemption) β pure data; each has a "frenzy" convulse stage
|
| 36 |
+
render.py # render_entity(affinity, mode, seq) β modes: idle/flash/flash_strong/build/frenzy/convulse_good/convulse_loop/end/end_settle/rage/peace/peace_settle/peace_dissolve; materialization = fog+brightness (no blur); render_treasure(...)
|
| 37 |
+
sting.py # _wav + sting/chime/heartbeat/sigh/flatline _wav_bytes β numpy audio synth (HF rejects committed .wav)
|
| 38 |
+
styles.css # horror CSS Β§15-22 (22 = finale build pulse + soft convulse + restart + .voice-channel)
|
| 39 |
assets/ # hollow_{base,terror,almost,end,rage,peace}.webp (rage = almost copy placeholder; peace = real art)
|
| 40 |
+
requirements.txt / packages.txt # +kokoro/misaki[en]/soundfile/spacy/en_core_web_sm; espeak-ng (apt)
|
| 41 |
+
tests/ # 204 tests: test_{memory,render,finale,schemas,engine,character,app,sting,voice}.py
|
| 42 |
+
docs/superpowers/{specs,plans}/ # approved specs + plans (latest pending: 2026-06-13-tts-greeting-treasure-align)
|
| 43 |
+
tools/pacing_sim.py # scripted visitors through real Ollama β pacing tuning
|
| 44 |
```
|
| 45 |
|
| 46 |
## State shape (gr.State) β all NEW-field reads use `.get(...)` for pre-deploy sessions
|
|
|
|
| 57 |
"fragments_told": int, # how many OWN_FRAGMENTS surfaced (redemption path)
|
| 58 |
"ended": bool, # a finale fired β input dead
|
| 59 |
"ending": str | None, # "good"|"loop"|"bad" once fired
|
| 60 |
+
"voice_on": bool, # mute toggle (π/π)
|
| 61 |
+
"msg_lengths": list[int], # B3 β rolling word counts (cap 5) for style_signal
|
| 62 |
+
"last_aware_memory": str|None, # B2 β avoid re-surfacing the same memory
|
| 63 |
+
"last_activity": float, # idle clock; stamped by _start_turn + chat()
|
| 64 |
+
"idle_count": int, # how many idle nudges fired (tier + bond decay)
|
| 65 |
}
|
| 66 |
```
|
| 67 |
Dev seed: `HOLLOW_FAST_FINALE=good|loop|bad` (`1`/`neutral` aliasβ good/loop) β each ending fires on the 2nd message.
|
|
|
|
| 83 |
- Tone bands in prompt (separate from ending routing): warm β₯15 / wounded β22..β1 / hostile β€β22 (injects wounds). Mimicry always on.
|
| 84 |
- **Measured pacing** (pacing_sim, real model): bad ~msg 6, good (warm) ~msg 10, loop (factual) ~msg 12, seeds msg 2. Warm play tone climbs to ~27; factual play tone stalls ~13 β clean gap at 20.
|
| 85 |
|
| 86 |
+
## Voice + Presence (the child speaks & watches)
|
| 87 |
+
- **Voice:** every Hollow reply + all finale lines are spoken (Kokoro, CPU). Finale `_voice_and_pause` stretches each step's pause to the audio length (+0.15 s) so lines aren't cut. Recall + finale = the wow. Mute toggle (π/π) mutates `voice_on` in place + clears `voice_panel`.
|
| 88 |
+
- **Awareness (B1/B2/B3):** B3 `style_signal` (Hollow notices short/long messages); B2 `pick_aware_memory` (resurfaces an earlier memory unprompted, every 3rd non-recall turn); B1 `_on_idle` (`gr.Timer` 4 s) β after `_IDLE_AFTER=35 s` of silence Hollow speaks first, escalating pleadβhurtβhostile (`_IDLE_TIERS`) and each nudge drops affinity β3 / tone β2 (timer outputs `bond_panel` so the meter falls live).
|
| 89 |
+
- **Materialization:** the child is **sharp, veiled in fog + shadow** (brightness 0.4β1, fog/vig denser at low bond) β NOT blurred (blur read as a broken image).
|
| 90 |
+
|
| 91 |
## Gotchas / lessons (DON'T re-learn these the hard way)
|
| 92 |
- **Verify CSS against the RUNNING app**, never the static harness β it lacks Gradio's cascade and has lied. Workflow: launch `HOLLOW_FAST_FINALE=good .venv/Scripts/python app.py` background β headless Chrome screenshot (`chrome.exe --headless --disable-gpu --screenshot=... --window-size=1900,950 --virtual-time-budget=12000 http://localhost:7860`) β Read the PNG β kill the port owner.
|
| 93 |
- **Gradio replaces the HTML DOM each update:** CSS *animations* restart (this IS the flash/screamer mechanism); CSS *transitions* DON'T animate across updates.
|
|
|
|
| 95 |
- **Qwen emits `+2` (invalid JSON)** if examples lack positive anchors β every positive turn fell back to delta 0 (bond frozen). Fixed by sanitizer in `apply_update` + explicit "no leading +" instruction + dual examples. Keep both.
|
| 96 |
- **transformers 5.x:** `apply_chat_template(...,return_tensors="pt")` returns `BatchEncoding` (use `hasattr(t,"input_ids")`); use `dtype=` not deprecated `torch_dtype`. Qwen3 thinking OFF: `enable_thinking=False` (Space) / `"think": False` (Ollama).
|
| 97 |
- **One GPU acquisition per turn:** gen + extract inside a single `@spaces.GPU` call. ZeroGPU β8 s/turn, PRO β1500 s/day.
|
| 98 |
+
- **Portrait fills the arch:** art is square (1:1), arch is 5:6 β materialization is brightness/opacity/fog ONLY, never `scale` (left an ugly gap). **No CSS `blur`** β it read as a broken/loading image; the child is sharp, veiled in shadow+fog. `!important` img sizing (Gradio base reset wins otherwise).
|
| 99 |
- **HF rejects committed binaries (.wav):** synthesize audio at runtime (`sting.py`). Same would apply to any new binary asset that isn't an image.
|
| 100 |
- **Don't edit `.py` via PowerShell string ops** β it mangled UTF-8 (em-dash β mojibake) once. Use the Edit tool.
|
| 101 |
- **Ollama ROCm can corrupt** (`cudaMalloc failed` with free VRAM) β symptom is red Error on ALL components. Fix: restart Ollama (tray + server). The Space (transformers) is immune.
|
|
|
|
| 104 |
- **NO threading around `run_turn`:** `@spaces.GPU` needs the main request context β a thread pool breaks ZeroGPU. The typing-dots animation is pure CSS for this reason (don't "fix" it with a worker thread).
|
| 105 |
- **Hide Gradio progress, animate our own:** `show_progress="hidden"` on the events kills the per-component "processing | Xs" overlay (CSS class-hiding was unreliable); we render our own `_PENDING` typing bubble. The HTML span + class survives Gradio's markdown sanitizer (verified) so Β§20 CSS animates it.
|
| 106 |
- **Live-verify via isolated probe:** to test a UI behavior that only appears mid-generation (typing dots, button states), spin a tiny throwaway Gradio app importing `styles.css`, screenshot it, delete it β don't guess CSS class names.
|
| 107 |
+
- **Cap CPU math threads before torch loads** (`OMP/OPENBLAS/MKL_NUM_THREADS=4` at the top of `app.py`+`voice.py`). On many-core machines under memory pressure OpenBLAS spawns one buffer-hungry thread/core and the CPU allocator fails (`DefaultCPUAllocator: not enough memory` on a tiny alloc, with GBs free) β Kokoro silently disabled. NOT real RAM exhaustion.
|
| 108 |
+
- **`voice_panel` must stay MOUNTED** (`elem_classes="voice-channel"` + CSS collapse to 0 size), NOT `visible=False` β an invisible Gradio component can unmount so the `<audio>` never reaches the DOM and never plays. Voice audio goes here, NEVER in the entity panel (would break the heartbeat's byte-identical HTML).
|
| 109 |
+
- **The idle `gr.Timer` must not race the chat generator:** `_start_turn` outputs `state` (activity stamp persists across the event boundary) and `_on_idle` no-ops if a `_PENDING` "..." bubble is in flight β else it orphans the bubble / clobbers a streaming finale. Non-speaking ticks return `gr.update()` (never `""`) for chatbot/voice.
|
| 110 |
+
- **Browser autoplay blocks audio before the visitor's first gesture** β the opening line can't auto-speak on load; it's voiced on the first message instead. Same reason the chimes are silent until you interact.
|
| 111 |
+
- **Kokoro deps:** import-guarded (silent fallback, never crashes). Needs python <3.13, `misaki[en]`β`spacy`+`en_core_web_sm`, `espeak-ng` (apt). Local: 3.12 `.venv-tts` for voice; the 3.13 `.venv` runs silent (fine for tests, which mock `speak`). TTS verbalizes dashes β strip them (`_clean_for_tts`, planned).
|
| 112 |
+
- **Ollama 500 = the ROCm `cudaMalloc` corruption** (see above), NOT our code β a short prompt may pass while a real 300-token turn 500s. Symptom in-app: every reply is the "fog swallowed your words" fallback. Fix: restart Ollama (kill `ollama.exe` + `ollama app.exe`, reopen). The Space (transformers) is immune.
|
| 113 |
|
| 114 |
## What's next
|
| 115 |
+
- **Current work is on branch `feat/voice-awareness`, NOT deployed.** Both Spaces are frozen at the pre-voice commit (`4deeed5`) β the submission stays safe while we iterate. Voice + awareness + presence-polish are committed on the branch (204 tests green). Deploy is **mirror-first**: merge β push `space` β verify build+voice live β push `hackathon`.
|
| 116 |
+
- **Pending plan:** `docs/superpowers/plans/2026-06-13-tts-greeting-treasure-align.md` β strip dashes from TTS, speak the opening on the first message, align the Treasure panel with the entity frame.
|
| 117 |
+
- **`FIELD_NOTES.md`** lives in the repo (π badge content). Field Notes done; the demo video + LinkedIn post are Pablo's, link both from the README when ready.
|
| 118 |
+
- **Optional art:** `hollow_rage.webp` is still a copy of `hollow_almost` (works fine). Swap a PNG: `Image.open(png).convert("RGB").resize((512,512),LANCZOS).save(webp,"WEBP",quality=80,method=6)`.
|
| 119 |
+
- **Deliverables (Pablo):** demo video (horror-trailer), social post. **Mon June 15:** submit early + community vote.
|
| 120 |
|
| 121 |
## Workflows
|
| 122 |
+
**Deploy** (from repo root): **mirror-first** β `git push space main` β wait ~3-4 min (Kokoro/spacy build is heavy), **Ctrl+F5**, verify build+voice live β only then `git push hackathon main`. If the mirror build breaks, the submission stays untouched. Commit trailer: `Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>`.
|
| 123 |
+
**Local voice test:** the 3.13 `.venv` is silent (no kokoro); run from `.venv-tts` (py3.12) for voice, with Ollama running for real replies.
|
| 124 |
|
| 125 |
**Working style:** phase by phase; explain *why* before code; everything in **English** (code, prompts, UI, dialogue); no new deps/complexity without asking; push back on scope creep. Subagent-driven dev for plans (implementer + spec review + quality review per task).
|
docs/superpowers/plans/2026-06-13-tts-greeting-treasure-align.md
ADDED
|
@@ -0,0 +1,170 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# TTS Cleanup + Spoken Opening + Treasure Alignment β Implementation Plan
|
| 2 |
+
|
| 3 |
+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development to implement this plan phase-by-phase. Steps use checkbox (`- [ ]`). Stay on git branch `feat/voice-awareness`.
|
| 4 |
+
|
| 5 |
+
**Goal:** Three fixes. (1) The TTS pronounces dashes (`β`, `-`) as words β strip them from spoken text. (2) Speak the opening line at the start of the chat (greeting that asks for a memory). (3) Align the Treasure panel with the child's frame for visual symmetry.
|
| 6 |
+
|
| 7 |
+
**Context:** Branch `feat/voice-awareness`. Voice via Kokoro (`voice.py`, `speak()`), works in the 3.12 `.venv-tts`, silent/guarded in the 3.13 `.venv`. Voice plays on every Hollow reply and on finale lines, from a hidden-but-mounted `voice_panel`. Run tests: `.venv/Scripts/python -m pytest -q` (204 green β keep them green). Voice smoke test runs in `.venv-tts`.
|
| 8 |
+
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
## PHASE 1 β Strip dashes from TTS input
|
| 12 |
+
|
| 13 |
+
**Files:** Modify `voice.py`, `tests/test_voice.py`.
|
| 14 |
+
|
| 15 |
+
The displayed text keeps its stylistic dashes (e.g. "i don'tβ"); only the text sent to Kokoro is cleaned, so it stops saying "dash".
|
| 16 |
+
|
| 17 |
+
- [ ] **Step 1: Failing test** (append to `tests/test_voice.py`):
|
| 18 |
+
|
| 19 |
+
```python
|
| 20 |
+
def test_clean_for_tts_strips_dashes():
|
| 21 |
+
assert "β" not in voice._clean_for_tts("i don'tβ i don't remember")
|
| 22 |
+
assert "-" not in voice._clean_for_tts("a well-kept secret")
|
| 23 |
+
# collapses to single spaces, trims
|
| 24 |
+
assert voice._clean_for_tts("a β b") == "a , b"
|
| 25 |
+
```
|
| 26 |
+
|
| 27 |
+
- [ ] **Step 2: Run, verify fail** β `.venv/Scripts/python -m pytest tests/test_voice.py -k clean_for_tts -q`.
|
| 28 |
+
|
| 29 |
+
- [ ] **Step 3: Implement** in `voice.py`. Add `import re` near the top (after the `os` thread-cap block). Add the helper and use it in `speak()`:
|
| 30 |
+
|
| 31 |
+
```python
|
| 32 |
+
def _clean_for_tts(text: str) -> str:
|
| 33 |
+
"""Kokoro verbalizes dashes ('β', '-') as words. Replace em/en dashes with a
|
| 34 |
+
comma pause and hyphens with a space, then collapse whitespace."""
|
| 35 |
+
text = text.replace("β", ", ").replace("β", ", ").replace("-", " ")
|
| 36 |
+
return re.sub(r"\s+", " ", text).strip()
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
In `speak()`, change the synthesis loop to use the cleaned text:
|
| 40 |
+
```python
|
| 41 |
+
cleaned = _clean_for_tts(text)
|
| 42 |
+
if not cleaned:
|
| 43 |
+
return None
|
| 44 |
+
audio = None
|
| 45 |
+
for _, _, chunk in _PIPELINE(cleaned, voice=_VOICE, speed=_SPEED):
|
| 46 |
+
audio = chunk if audio is None else np.concatenate([audio, chunk])
|
| 47 |
+
```
|
| 48 |
+
(Keep the early `if _PIPELINE is None or not text or not text.strip(): return None` guard as-is at the top of `speak()`.)
|
| 49 |
+
|
| 50 |
+
- [ ] **Step 4: Run** `tests/test_voice.py -k clean_for_tts` then the full suite β green.
|
| 51 |
+
|
| 52 |
+
- [ ] **Step 5: Commit**
|
| 53 |
+
```bash
|
| 54 |
+
git add voice.py tests/test_voice.py
|
| 55 |
+
git commit -m "fix: strip dashes from TTS input (Kokoro was saying 'dash')
|
| 56 |
+
|
| 57 |
+
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
|
| 58 |
+
```
|
| 59 |
+
|
| 60 |
+
---
|
| 61 |
+
|
| 62 |
+
## PHASE 2 β Speak the opening line on the first turn
|
| 63 |
+
|
| 64 |
+
**Files:** Modify `app.py`, `tests/test_app.py`.
|
| 65 |
+
|
| 66 |
+
Browser autoplay can't fire before a user gesture, so the opening can't be spoken on page load. Instead: on the visitor's FIRST message (the first gesture), Hollow speaks the OPENING_LINE (it matches the opening text shown at the top of the chat), and that first reply is text-only so the greeting isn't cut. From the second turn on, replies are voiced as normal.
|
| 67 |
+
|
| 68 |
+
- [ ] **Step 1: Failing tests** (append to `tests/test_app.py`, class `TestVoice`):
|
| 69 |
+
|
| 70 |
+
```python
|
| 71 |
+
def test_first_turn_speaks_the_opening_line(self, monkeypatch):
|
| 72 |
+
captured = {}
|
| 73 |
+
def fake_speak(text):
|
| 74 |
+
captured["text"] = text
|
| 75 |
+
return "FAKEB64"
|
| 76 |
+
monkeypatch.setattr(app, "run_turn",
|
| 77 |
+
lambda *a, **k: ("a normal reply.", '{"affinity_delta":1}'))
|
| 78 |
+
monkeypatch.setattr(app.time, "sleep", lambda s: None)
|
| 79 |
+
monkeypatch.setattr(app, "should_recall", lambda s: (False, None))
|
| 80 |
+
monkeypatch.setattr(app, "speak", fake_speak)
|
| 81 |
+
state = _state(affinity=20, treasure=[], claimed=[], tone=0, turn=0)
|
| 82 |
+
outs = list(app.chat("hello", state, []))
|
| 83 |
+
assert "FAKEB64" in outs[-1][7] # the greeting is voiced
|
| 84 |
+
assert captured["text"] == OPENING_LINE # ...with the opening line
|
| 85 |
+
|
| 86 |
+
def test_second_turn_speaks_the_reply(self, monkeypatch):
|
| 87 |
+
captured = {}
|
| 88 |
+
monkeypatch.setattr(app, "run_turn",
|
| 89 |
+
lambda *a, **k: ("a normal reply.", '{"affinity_delta":1}'))
|
| 90 |
+
monkeypatch.setattr(app.time, "sleep", lambda s: None)
|
| 91 |
+
monkeypatch.setattr(app, "should_recall", lambda s: (False, None))
|
| 92 |
+
monkeypatch.setattr(app, "speak", lambda t: captured.setdefault("text", t) or "FAKEB64")
|
| 93 |
+
state = _state(affinity=20, treasure=[], claimed=[], tone=0, turn=3) # not first
|
| 94 |
+
list(app.chat("hi again", state, []))
|
| 95 |
+
assert captured["text"] == "a normal reply." # later turns voice the reply
|
| 96 |
+
```
|
| 97 |
+
|
| 98 |
+
- [ ] **Step 2: Run, verify fail** β both currently voice the reply.
|
| 99 |
+
|
| 100 |
+
- [ ] **Step 3: Implement** in `app.py` `chat()`. Near the TOP of the normal-turn path (before `state["turn"] += 1` runs), capture whether this is the first turn:
|
| 101 |
+
```python
|
| 102 |
+
is_first_turn = state["turn"] == 0
|
| 103 |
+
```
|
| 104 |
+
Then where the voice is computed for the final normal-turn yield, replace:
|
| 105 |
+
```python
|
| 106 |
+
voice = _voice_html(speak(reply)) if state.get("voice_on", True) else ""
|
| 107 |
+
```
|
| 108 |
+
with:
|
| 109 |
+
```python
|
| 110 |
+
# on the visitor's first message, Hollow speaks the opening line (its
|
| 111 |
+
# greeting/memory-ask, matching the text at the top of the chat); that
|
| 112 |
+
# first reply stays text-only so the greeting isn't cut. Browser autoplay
|
| 113 |
+
# can't fire before this first gesture, so this is the earliest it can play.
|
| 114 |
+
if state.get("voice_on", True):
|
| 115 |
+
voice = _voice_html(speak(OPENING_LINE if is_first_turn else reply))
|
| 116 |
+
else:
|
| 117 |
+
voice = ""
|
| 118 |
+
```
|
| 119 |
+
(`OPENING_LINE` is already imported in `app.py`.)
|
| 120 |
+
|
| 121 |
+
- [ ] **Step 4: Run** the Voice tests then the full suite β green.
|
| 122 |
+
|
| 123 |
+
- [ ] **Step 5: Commit**
|
| 124 |
+
```bash
|
| 125 |
+
git add app.py tests/test_app.py
|
| 126 |
+
git commit -m "feat: speak the opening line on the visitor's first message
|
| 127 |
+
|
| 128 |
+
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
|
| 129 |
+
```
|
| 130 |
+
|
| 131 |
+
---
|
| 132 |
+
|
| 133 |
+
## PHASE 3 β Align the Treasure panel with the child's frame (symmetry)
|
| 134 |
+
|
| 135 |
+
**Files:** Modify `styles.css` (and `app.py` only if a wrapper class is needed). Visual β verify against the running app.
|
| 136 |
+
|
| 137 |
+
The entity panel (left, the child in the stone arch) and the Treasure panel (right) should share the same top edge and read as a symmetric pair framing the chat. Today the Treasure panel sits slightly off.
|
| 138 |
+
|
| 139 |
+
- [ ] **Step 1: Observe the current misalignment.** Launch and screenshot:
|
| 140 |
+
- `HOLLOW_FAST_FINALE=good .venv/Scripts/python app.py` (background), headless Chrome screenshot (per the CLAUDE.md workflow), Read the PNG. Note the vertical offset between the top of the entity arch and the top of the Treasure box, and any height mismatch. Kill the port owner.
|
| 141 |
+
|
| 142 |
+
- [ ] **Step 2: Read the relevant CSS.** In `styles.css`, find the entity section (the `.entity-portal` / `.entity-frame` rules) and the treasure section (`.treasure-panel`). Note their top margins/padding. In `app.py`, the layout is a `gr.Row` of three `gr.Column`s: entity (scale 1), chat (scale 2), treasure (scale 1). Gradio columns add their own padding/gap.
|
| 143 |
+
|
| 144 |
+
- [ ] **Step 3: Align.** Adjust CSS so the Treasure panel's top aligns with the entity arch's top (and, if it reads better, match their heights). Likely a top-margin/padding tweak on `.treasure-panel` or its Gradio column, mirroring whatever offset the entity panel has. Keep the horror styling intact. Do NOT use fixed pixel heights that break on resize unless necessary β prefer matching the existing offsets.
|
| 145 |
+
|
| 146 |
+
- [ ] **Step 4: Verify (REQUIRED β CSS lies in the harness).** Re-launch, screenshot, Read the PNG: the entity arch and the Treasure box should now share a top edge and look symmetric. Iterate until aligned. Kill the port owner.
|
| 147 |
+
|
| 148 |
+
- [ ] **Step 5: Commit**
|
| 149 |
+
```bash
|
| 150 |
+
git add styles.css app.py
|
| 151 |
+
git commit -m "style: align the Treasure panel with the entity frame (symmetry)
|
| 152 |
+
|
| 153 |
+
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>"
|
| 154 |
+
```
|
| 155 |
+
|
| 156 |
+
---
|
| 157 |
+
|
| 158 |
+
## FINAL β verification
|
| 159 |
+
|
| 160 |
+
- [ ] **Step 1:** `.venv/Scripts/python -m pytest -q` β all green.
|
| 161 |
+
- [ ] **Step 2: Voice smoke** (`.venv-tts`): `python -c "import voice; print(voice._clean_for_tts('aβbβc')); print(bool(voice.speak('hello, i am here.')))"` β cleaned text (no dashes) + `True`.
|
| 162 |
+
- [ ] **Step 3: Live local test** (`.venv-tts`, Ollama running): the opening line is spoken on the first message; no "dash" pronounced anywhere; the Treasure box lines up with the child's frame.
|
| 163 |
+
- [ ] **Step 4: Deploy mirror-first** (orchestrator, with the user's go-ahead): merge `feat/voice-awareness` β `main`, push to `space` (mirror), verify the build + voice live, then push to `hackathon` (submission).
|
| 164 |
+
|
| 165 |
+
---
|
| 166 |
+
|
| 167 |
+
## Self-review notes
|
| 168 |
+
- Dash stripping is TTS-only β the displayed dialogue keeps its stylistic dashes.
|
| 169 |
+
- The opening voice is tied to the first SEND (the first gesture autoplay allows); if the visitor never sends, the idle "speak first" still covers presence.
|
| 170 |
+
- The treasure alignment is visual β never trust the static harness; screenshot the running app.
|