README.md

---
license: apache-2.0
language:
- en
tags:
- keyword-spotting
- wake-word-detection
- speech
- phoneme-recognition
- ctc
- esp32
- tinyml
- edge-ai
- on-device
library_name: pytorch
pipeline_tag: automatic-speech-recognition
---

# Wake Words Without Training — an open-vocabulary wake word engine for microcontrollers

**A wake word here is not a model. It is ~50 bytes of configuration.**

One small streaming phoneme recognizer is trained *once* on generic
speech and never sees a wake word. Any phrase — typed, invented,
whatever you like — becomes a detector in milliseconds via
grapheme-to-phoneme lookup, optionally sharpened by five spoken
examples. The whole runtime fits on an **ESP32-S3** at **340 KB INT8**
and **14 ms per 40 ms of audio**, leaving wake words changeable
without retraining, reflashing, or a cloud round-trip.

| | Every per-word-trained system | This |
|---|---|---|
| New wake word costs | TTS synthesis + GPU training (minutes–hours) | a dictionary lookup (ms) |
| Artifact per word | a 50–200 KB model | **~50 bytes** (phone ids + threshold) |
| N simultaneous words | N models | 1 shared model + N tiny decoders |
| Changing a word on device | reflash | send a few bytes |

---

## How it works

**Runtime lane (wake-word-agnostic, always on):**
`mic → 40-band log-mel + causal EMA normalization → streaming causal TCN
→ CTC phoneme posteriors every 20 ms → keyword-filler Viterbi decoder`

**Enrollment lane (per word, no training):**
`text → G2P → phone sequence` and/or `5 spoken examples → phoneme decode
→ pronunciation variants` → automatic per-variant threshold calibration
→ **config**, or an explicit *refusal* if the phrase cannot be separated
from ordinary speech.

The decoder is a filler-normalized Viterbi search with four gates, each
closing a failure mode we hit in practice:

1. **Phone-frame normalization** — score is normalized by frames spent in
   *emitting* states only. Normalizing by total duration lets degenerate
   paths idle in blank states (free in silence) and fire rhythmically on
   nothing. Fixing this moved a clean operating point from 65% recall
   @ 12 FA/h to **95% @ 1.25 FA/h**.
2. **Strong-evidence gate** — ≥50% of phone frames must have their phone
   within 1.5 nats of the frame's best class. Noisy audio yields flat
   posteriors that score fine *on average* while containing nothing;
   this cut false alarms on noise-degraded speech from ~100/h to ~3/h.
3. **Duration bounds** — 60–200 ms per phone. At a looser cap we observed
   1.5–2.3 s alignments crawling across background television.
4. **Acoustic energy veto** (on device) — the matched span must exceed the
   rolling noise floor by 5 dB.

Detections use only *within-frame logit differences*, so the deployed
engine never computes a softmax.

## Models

| File | Params | dev-clean PER | Common Voice dev PER | Use |
|---|---|---|---|---|
| `phoneme_tcn_student.pt` | 358,504 | **0.229** | **0.402** | deploy this |
| `phoneme_tcn_teacher.pt` | 3,337,768 | 0.150 | — | distillation teacher |

**Student architecture** — causal TCN: stride-2 stem (k=5), 8
depthwise-separable causal blocks (k=5, dilations 1,2,4,8 ×2, 192 ch,
BatchNorm+ReLU, residual), 1×1 head over 40 classes (39 stress-free
ARPAbet phones + CTC blank). 20 ms output frames, ~2.4 s receptive
field. Every op is BatchNorm-foldable conv / ReLU / residual — chosen so
post-training INT8 survives intact (**99.3%** frame-argmax agreement
with float).

**Training** — CTC over phonemized transcripts (CMUdict + neural G2P
fallback) on **LibriSpeech 960 h + 1.10 M Common Voice 17 English clips**
(~1,500 h, incl. 83k Indian-accented). On-GPU augmentation: synthetic
room impulse responses (T60 0.1–0.6 s), additive noise (5–30 dB SNR),
same-batch babble (10–25 dB), random gain, SpecAugment. The student was
then distilled from the teacher with **speech-weighted KD** — frames are
weighted by `1 − p(blank)` because ~75% of CTC frames are blank and
uniform KD otherwise spends its budget teaching silence (uniform KD
*degraded* the student; the weighted version improved it).

## Benchmarks

Ten phrases, Piper LibriTTS-R positives **verified by Whisper** (raw TTS
is unreliable: "tornado" → *"Pornado"*, "dakota" → *"Decoder"*), against
1.22 h LibriSpeech dev-clean + 0.82 h Common Voice dev negatives. Noisy
conditions degrade positives *and* negatives identically so operating
points stay condition-matched.

| Condition | Recall | Notes |
|---|---|---|
| Text-only, speaker-independent | median **0.38** @ 0 FA | hardest case: arbitrary phrase, arbitrary speaker, zero examples |
| Cross-speaker enrollment | repairs dictionary mismatch | tornado 0 → 0.35, "hey jarvis" 0.45 → 0.75 |
| Personal enrollment (5 examples) | mean **0.554**, median **0.667** @ 3.3 FA/h | synthetic renditions vary more than a self-consistent human |
| Universal auto-calibration | **83%** of arbitrary voices (`neptuno`) | 36 TTS voices, population-voted variants |
| On-device, single user | ~**14/15** utterances | ESP32-S3 + INMP441, live session |

**Safety property**: calibration *refuses* phrases it cannot separate.
"norman" (one phone from "normal") was rejected for 5/6 voices and
"dakota" for 6/6 — independently flagged by the text-only phrase scorer
(`score_phrase.py`) before any audio existed.

## Deployment

| Stage | Verification |
|---|---|
| BN folding + residual extraction | max err 7.6e-4 vs PyTorch |
| INT8 simulation vs float | 99.3% frame agreement |
| **C engine vs INT8 simulation** | **bit-exact (0.0)** |
| C mel frontend vs training frontend | ≤1e-3 log-mel |
| ESP32-S3 step time | **14.2 ms** / 40 ms frame (1 core @ 240 MHz) |

`engine_c/` is ~300 lines of dependency-free C with one INT8 ring buffer
per layer, so each frame costs only its own ~350k MACs — no window
recomputation. On ESP32 the binding constraint was **memory latency, not
math**: moving weights out of memory-mapped flash (28.6 → 15.9 ms) and
filling internal SRAM before PSRAM (→ 14.2 ms) mattered far more than
loop optimization.

## Files

```
phoneme_tcn_student.pt     deployable model (float, PyTorch state dict)
phoneme_tcn_teacher.pt     distillation teacher
model_int8.h               INT8 weights + layer table (C header, 340 KB)
act_scales.json            activation scales from PTQ calibration
frontend_data.h            mel filterbank + Hann window (exact training values)
engine_c/                  streaming INT8 engine + decoder + mel frontend (C)
phoneme_engine/            PyTorch model, decoder, enrollment, quantization, scorer
examples/                  a universal wake word config (~50 bytes of JSON)
```

## Usage

**Spot a typed phrase (Python):**

```python
import torch
from phoneme_engine.model import PhonemeTCN
from phoneme_engine.features import LogMel
from phoneme_engine.decoder import KeywordSpotter

model = PhonemeTCN().eval()
model.load_state_dict(torch.load("phoneme_tcn_student.pt",
                                 weights_only=True)["model"])
frontend = LogMel().eval()

spotter = KeywordSpotter("hey orbit")      # G2P -> HH EY AO R B AH T
with torch.no_grad():
    logp = torch.log_softmax(model(frontend(wav)).float(), 2)[0].numpy()
for frame, score in spotter.run(logp):
    print(f"detected at {frame * 0.02:.2f}s (score {score:.2f})")
```

**Score a candidate phrase before committing to it:**

```bash
python score_phrase.py "vucano"
# low reliability: /v/ is acoustically weak and often lost on small mics
```

**A wake word config, in full:**

```json
{"phrase": "neptuno",
 "spotters": [{"phones": ["N","EH","P","T","UW","N","OW"],
               "source": "dictionary", "threshold": -2.5}]}
```

**On a microcontroller:** compile `engine_c/` with `model_int8.h` and
`frontend_data.h`, feed it 40-band mel frames, and pass the logits to
`pww_spotter_step()`. Wake words are `uint8_t` arrays of phone ids plus a
float threshold — swap them at runtime.

## Limitations

- The 358k student is the binding constraint. Text-only speaker-independent
  recall is modest; 40% PER on real-world speech leaves little margin for
  accented, distant, or noisy input. Personal enrollment recovers much of it.
- Benchmarks are TTS-based (Whisper-verified, but synthetic). Human
  multi-speaker evaluation is future work.
- **Onset phonetics dominate word quality.** Nasals/stops/sibilants
  (N, M, K, T, S) work well; weak fricatives (V, F, TH, H) are often lost
  on small MEMS mics. The included scorer predicts this from text.
- False-alarm rates are measured against *continuous speech* — a worst case
  versus mostly-quiet rooms.
- Single-microphone. Far-field performance is a hardware question
  (beamforming arrays), not a decoder one.
- English only, though nothing in the architecture is language-bound beyond
  the phone inventory and G2P.

## License

Apache-2.0. Trained on LibriSpeech (CC BY 4.0) and Common Voice 17 (CC0).

## Citation

```bibtex
@misc{wakewordswithouttraining2026,
  title  = {Wake Words Without Training: Open-Vocabulary Wake Word
            Creation from Text and a Few Examples},
  author = {IOTEverythin},
  year   = {2026},
  url    = {https://huggingface.co/IOTEverythin/phoneme-wake-word}
}
```